lexical unit; lexical token
Computing term for a basic element of text, such as a word or punctuation mark, treated as a single unit in lexical analysis or parsing.
Analyze the text in lexical units.
This parser first splits the input into lexical tokens, then performs syntactic analysis.
トークン is the common loanword for 'token' in computing, while 字句単位 is a more formal Japanese term often used in academic or technical documentation.
形態素 (morpheme) is a linguistic unit of meaning, whereas 字句単位 is a broader term for any lexical token in text processing.
Compound of 字句 (characters and phrases) and 単位 (unit). A straightforward technical term with no obscure historical derivation.