Tokenizer

This is a discussion on Tokenizer within the C++ Programming forums, part of the General Programming Boards category; I see the term 'token' and 'tokenizer' used a lot with programming, but haven't been able to find a decent ...

  1. #1
    Cheesy Poofs! PJYelton's Avatar
    Join Date
    Sep 2002
    Location
    Boulder
    Posts
    1,728

    Tokenizer

    I see the term 'token' and 'tokenizer' used a lot with programming, but haven't been able to find a decent explanation of what that means anywhere on the net. Can anyone help me understand it? Best I can figure is something to do with string parsing.

  2. #2
    End Of Line Hammer's Avatar
    Join Date
    Apr 2002
    Posts
    6,231
    >Best I can figure is something to do with string parsing
    Err.. .yeah. It's basically breaking something down into smaller seqments. A token could be a word in a sentence, for example.
    When all else fails, read the instructions.
    If you're posting code, use code tags: [code] /* insert code here */ [/code]

  3. #3
    Registered User
    Join Date
    Jan 2003
    Posts
    648
    A tokenizer breaks a large chunk into pieces, called tokens. The most common implementation of one is a string tokenizer. You give it a delimeter character and it'll break apart the string you give it into an array, each seperated by the delimeter.

    BTW, correct me if I'm wrong, but thats what it's called right? Delimeter?

    Ex:
    I walked the dog today.

    Delimeter: " " (space)

    token 1="I"
    token 2="walked"
    token 3="the"
    token 4="dog."

    Things that do this: VB's Split() function. Java's java.text.StringTokenizer class (or it might be in java.util....).

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Scanner? Lexical analyzer? Tokenizer?
    By audinue in forum A Brief History of Cprogramming.com
    Replies: 8
    Last Post: 12-23-2008, 11:32 PM
  2. string tokenizer
    By mbooka in forum C Programming
    Replies: 4
    Last Post: 02-15-2006, 06:00 PM
  3. C++ String Tokenizer
    By Annorax in forum Game Programming
    Replies: 10
    Last Post: 07-13-2005, 11:41 AM
  4. Tokenizer in C
    By Tarik in forum C Programming
    Replies: 21
    Last Post: 08-26-2004, 07:36 AM
  5. deriving ifstream class for tokenizer
    By djh000 in forum C++ Programming
    Replies: 0
    Last Post: 09-23-2001, 03:37 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21