noun
lexical unit; lexical token
Computing term for a basic element of text, such as a word or punctuation mark, treated as a single unit in lexical analysis or parsing.
字句単位でテキストを解析する。
Analyze the text in lexical units.
このパーサーは字句単位に分割してから構文解析を行う。
This parser first splits the input into lexical tokens, then performs syntactic analysis.
Compound of 字句 (characters and phrases) and 単位 (unit). A straightforward technical term with no obscure historical derivation.