Thread: token in compiler

  1. #1
    Registered User
    Join Date
    May 2008

    token in compiler

    what is token in compiler?

  2. #2
    Kernel hacker
    Join Date
    Jul 2007
    Farncombe, Surrey, England
    any "item" that the compiler cares about, such as keywords, names of variables and types, symbols (+, -, --, ++, *, &, &&, (, ), [, ], etc).

    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  3. #3
    Officially An Architect brewbuck's Avatar
    Join Date
    Mar 2007
    Portland, OR
    Quote Originally Posted by beginer View Post
    what is token in compiler?
    A token is a lexical atom.

    The input to a compiler is a stream of characters. Traditionally the compiler's front end is divided into two parts (at least conceptually): the "lexer" and the "parser." The lexer takes the input characters and separates them into pieces called "tokens" which become input to the parser.

    For instance the numeric literal "123", while composed of three distinct characters, comprises a single lexical element, i.e. a token. The parser phase of the compiler receives this element as an atomic unit, not caring about its composition of characters.

    More formally, the token is an abstract object representing an atomic unit of parser input -- the actual characters which comprise the token form a "lexeme."

    Tokens might not even correspond to actual pieces of input. Although this doesn't happen in C and C++, in Python for instance the lexer analyzes indentation and produces "INDENT" and "DEDENT" tokens when the indentation level changes. These tokens don't directly correspond to any particular character or sequence of characters in the input.
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Post...
    By maxorator in forum C++ Programming
    Replies: 12
    Last Post: 10-11-2005, 08:39 AM
  2. Dikumud
    By maxorator in forum C++ Programming
    Replies: 1
    Last Post: 10-01-2005, 06:39 AM
  3. Problem with Visual C++ Object-Oriented Programming Book.
    By GameGenie in forum C++ Programming
    Replies: 9
    Last Post: 08-29-2005, 11:21 PM
  4. Please Help - Problem with Compilers
    By toonlover in forum C++ Programming
    Replies: 5
    Last Post: 07-23-2005, 10:03 AM
  5. Windows using Dev-C++
    By Renegade in forum C++ Programming
    Replies: 15
    Last Post: 07-07-2005, 08:29 PM