So,
And... a...Code:lexer == scanner == tokenizer
Scanner : Scan, its job to scan thing.A lexer is often organized as separate scanner and tokenizer functions
Tokenizer : Tokenize, its job to tokenize thing.
While a Tokenizer need a Scanner, without it, what thing to be tokenized?
And a Tokenizer is always a Scanner?Code:Scanner | Tokenizer
So, it means a Tokenizer == Lexer and Scanner != Lexer, but it's part of Lexer?
I think scanner and tokenizer are part of lexer...