字じ句く単たん位い

noun

lexical unit; lexical token

Computing term for a basic element of text, such as a word or punctuation mark, treated as a single unit in lexical analysis or parsing.

• 字じ句く単たん位いでテキストを解かい析せきする。

  Analyze the text in lexical units.

• このパーサーは字じ句く単たん位いに分ぶん割かつしてから構こう文ぶん解かい析せきを行おこなう。

  This parser first splits the input into lexical tokens, then performs syntactic analysis.