token in compiler

This is a discussion on token in compiler within the C Programming forums, part of the General Programming Boards category; what is token in compiler?...

  1. #1
    Registered User
    Join Date
    May 2008
    Posts
    9

    token in compiler

    what is token in compiler?

  2. #2
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    any "item" that the compiler cares about, such as keywords, names of variables and types, symbols (+, -, --, ++, *, &, &&, (, ), [, ], etc).

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  3. #3
    Captain Crash brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,274
    Quote Originally Posted by beginer View Post
    what is token in compiler?
    A token is a lexical atom.

    The input to a compiler is a stream of characters. Traditionally the compiler's front end is divided into two parts (at least conceptually): the "lexer" and the "parser." The lexer takes the input characters and separates them into pieces called "tokens" which become input to the parser.

    For instance the numeric literal "123", while composed of three distinct characters, comprises a single lexical element, i.e. a token. The parser phase of the compiler receives this element as an atomic unit, not caring about its composition of characters.

    More formally, the token is an abstract object representing an atomic unit of parser input -- the actual characters which comprise the token form a "lexeme."

    Tokens might not even correspond to actual pieces of input. Although this doesn't happen in C and C++, in Python for instance the lexer analyzes indentation and produces "INDENT" and "DEDENT" tokens when the indentation level changes. These tokens don't directly correspond to any particular character or sequence of characters in the input.
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Post...
    By maxorator in forum C++ Programming
    Replies: 12
    Last Post: 10-11-2005, 09:39 AM
  2. Dikumud
    By maxorator in forum C++ Programming
    Replies: 1
    Last Post: 10-01-2005, 07:39 AM
  3. Problem with Visual C++ Object-Oriented Programming Book.
    By GameGenie in forum C++ Programming
    Replies: 9
    Last Post: 08-30-2005, 12:21 AM
  4. Please Help - Problem with Compilers
    By toonlover in forum C++ Programming
    Replies: 5
    Last Post: 07-23-2005, 11:03 AM
  5. Windows using Dev-C++
    By Renegade in forum C++ Programming
    Replies: 15
    Last Post: 07-07-2005, 09:29 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21