Thread: Tokenizer

  1. #1
    Cheesy Poofs! PJYelton's Avatar
    Join Date
    Sep 2002


    I see the term 'token' and 'tokenizer' used a lot with programming, but haven't been able to find a decent explanation of what that means anywhere on the net. Can anyone help me understand it? Best I can figure is something to do with string parsing.

  2. #2
    End Of Line Hammer's Avatar
    Join Date
    Apr 2002
    >Best I can figure is something to do with string parsing
    Err.. .yeah. It's basically breaking something down into smaller seqments. A token could be a word in a sentence, for example.
    When all else fails, read the instructions.
    If you're posting code, use code tags: [code] /* insert code here */ [/code]

  3. #3
    Registered User
    Join Date
    Jan 2003
    A tokenizer breaks a large chunk into pieces, called tokens. The most common implementation of one is a string tokenizer. You give it a delimeter character and it'll break apart the string you give it into an array, each seperated by the delimeter.

    BTW, correct me if I'm wrong, but thats what it's called right? Delimeter?

    I walked the dog today.

    Delimeter: " " (space)

    token 1="I"
    token 2="walked"
    token 3="the"
    token 4="dog."

    Things that do this: VB's Split() function. Java's java.text.StringTokenizer class (or it might be in java.util....).

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Scanner? Lexical analyzer? Tokenizer?
    By audinue in forum A Brief History of
    Replies: 8
    Last Post: 12-23-2008, 11:32 PM
  2. string tokenizer
    By mbooka in forum C Programming
    Replies: 4
    Last Post: 02-15-2006, 06:00 PM
  3. C++ String Tokenizer
    By Annorax in forum Game Programming
    Replies: 10
    Last Post: 07-13-2005, 10:41 AM
  4. Tokenizer in C
    By Tarik in forum C Programming
    Replies: 21
    Last Post: 08-26-2004, 06:36 AM
  5. deriving ifstream class for tokenizer
    By djh000 in forum C++ Programming
    Replies: 0
    Last Post: 09-23-2001, 02:37 AM