noun
lexical analysis; tokenization
In computing, the process of converting a sequence of characters into a sequence of tokens. Often used in compilers and natural language processing.
コンパイラの字句解析は、ソースコードをトークンに分割する。
Lexical analysis in a compiler splits source code into tokens.
自然言語処理では、字句解析が最初のステップになることが多い。
In natural language processing, tokenization is often the first step.
字句解析 is lexical analysis (tokenization), while 構文解析 is syntactic analysis (parsing), which typically follows tokenization.
Compound of 字句 (characters and phrases) and 解析 (analysis). A technical term likely coined in the context of computer science.