I see the term 'token' and 'tokenizer' used a lot with programming, but haven't been able to find a decent explanation of what that means anywhere on the net. Can anyone help me understand it? Best I can figure is something to do with string parsing.
>Best I can figure is something to do with string parsing
Err.. .yeah. It's basically breaking something down into smaller seqments. A token could be a word in a sentence, for example.
When all else fails, read the instructions.
If you're posting code, use code tags: [code] /* insert code here */ [/code]
A tokenizer breaks a large chunk into pieces, called tokens. The most common implementation of one is a string tokenizer. You give it a delimeter character and it'll break apart the string you give it into an array, each seperated by the delimeter.
BTW, correct me if I'm wrong, but thats what it's called right? Delimeter?
Ex:
I walked the dog today.
Delimeter: " " (space)
token 1="I"
token 2="walked"
token 3="the"
token 4="dog."
Things that do this: VB's Split() function. Java's java.text.StringTokenizer class (or it might be in java.util....).
