lexical analysis; tokenization
In computing, the process of converting a sequence of characters into a sequence of tokens. Often used in compilers and natural language processing.
Lexical analysis in a compiler splits source code into tokens.
In natural language processing, tokenization is often the first step.
字句解析 is lexical analysis (tokenization), while 構文解析 is syntactic analysis (parsing), which typically follows tokenization.
Compound of 字句 (characters and phrases) and 解析 (analysis). A technical term likely coined in the context of computer science.