lexical unit; lexical token
Computing term for a basic element of text, such as a word or punctuation mark, treated as a single unit in lexical analysis or parsing.
Analyze the text in lexical units.
This parser first splits the input into lexical tokens, then performs syntactic analysis.
Compound of 字句 (characters and phrases) and 単位 (unit). A straightforward technical term with no obscure historical derivation.