noun
lexical token
A technical term in computing and linguistics for a basic unit of lexical analysis, such as a word or punctuation mark.
コンパイラはソースコードを構文素に分割する。
The compiler splits the source code into lexical tokens.
トークン is the common loanword for 'token' in computing, often used interchangeably with 構文素 in technical contexts.
Compound of 構文 (syntax) and 素 (element), likely a calque from English 'lexical token' or similar computing terminology.