Thread: reading in files token by token

  1. #1
    Registered User
    Join Date
    Oct 2001
    Posts
    25

    reading in files token by token

    hi, can someone help me with this (hopefully) minor problem...

    I want to open a C file and read it in word by word (or split it into tokens as is possible in Java). This is so that I can recognise keywords, function calls etc. The closest I have got to it is to read a file line by line. Is it possible to split the line down further?

    Thanks a lot,

    Bill


  2. #2
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    38,169
    If you want to split a line into words, then strtok might be the thing to go for.

    Code:
    #include <stdio.h>
    #include <string.h>
    
    int main ( ) {
      char test_msg[]="This is a test message";
      char *p;
      char *delim = " ";
      char *words[20];
      int i = 0, j;
    
      p = strtok(test_msg,delim);
      while ( p ) {
        words[i] = p;
        p = strtok(NULL,delim);
        i++;
      } 
    
      printf ( "%d\n", i );
      for ( j = 0 ; j < i ; j++ ){
        printf("'%s'\n", words[j]);
      }
    }
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  3. #3
    Registered User pinko_liberal's Avatar
    Join Date
    Oct 2001
    Posts
    284
    Originally posted by Salem
    If you want to split a line into words, then strtok might be the thing to go for.

    Code:
    #include <stdio.h>
    #include <string.h>
    
    int main ( ) {
      char test_msg[]="This is a test message";
      char *p;
      char *delim = " ";
      char *words[20];
      int i = 0, j;
    
      p = strtok(test_msg,delim);
      while ( p ) {
        words[i] = p;
        p = strtok(NULL,delim);
        i++;
      } 
    
      printf ( "%d\n", i );
      for ( j = 0 ; j < i ; j++ ){
        printf("'%s'\n", words[j]);
      }
    }
    the problem with strtok being that if the line
    char test_msg[]="This is a test message";
    were to be replaced by
    char *test_msg="This is a test message";
    there might be problem as strtok might try to modify a read only string , compiling with gcc leads to a problem but with bcc32 the program runs fine .

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Post...
    By maxorator in forum C++ Programming
    Replies: 12
    Last Post: 10-11-2005, 08:39 AM
  2. Dikumud
    By maxorator in forum C++ Programming
    Replies: 1
    Last Post: 10-01-2005, 06:39 AM
  3. C++ compilation issues
    By Rupan in forum C++ Programming
    Replies: 1
    Last Post: 08-22-2005, 05:45 AM
  4. Please Help - Problem with Compilers
    By toonlover in forum C++ Programming
    Replies: 5
    Last Post: 07-23-2005, 10:03 AM
  5. reading files
    By hiya in forum C++ Programming
    Replies: 7
    Last Post: 05-21-2005, 11:40 AM