Thread: Segfault with simple lexer

Hi all -

I'm trying to get this lexer to work but it gives a segfault at the moment. Here's the code -

Code:

/* lex.c */ 
/* This code is released to the public domain. */ 
/* "Share and enjoy......"  :)     */ 

#include <stdio.h>
#include <ctype.h>
#include <string.h>
#include <stdlib.h>


/* Array of our keywords in string form. */ 
char *kw_strings[10] = { 
   "select", "from", "where", "and", "or", "not", "in", "is", "null" 
    } ; 
   
    
/*  Search function to search the array of keywords. */ 
int search(char *arr[], int dim, char *str) { 
        
    int i;      
    int found_match;
    
    for (i=0; i<dim; i++) { 
        if ( !strcmp(arr[i], str ) )  {   
            found_match = 1;        
            break; 
    }   else found_match = 0;    
 }  /* For */     

    return found_match; 
}  /* search */ 


void print_token(char *str) {       
    char token[80];
    char *toktype;    
    int i=0;
    char c;      

while (*str != '\0') { 
    c = str[i];
              
    /* Keyword or identifier */ 
    if (isalpha(c) || c == '_') {        
       while (isalnum(c) || c == '_') { 
             token[i] = c; 
             str++;  i++;
       }
         
   if (search(kw_strings, 10, token) == 1 ) toktype = "Keyword";
   else toktype "Identifier" ;                                         
       printf("%s %s\n", toktype, token);            
    }    
    
    else if (ispunct(c)) { 
        toktype = "operator"; 
        printf("%s %s\n", toktype, token);          
        str++; i++;
    }    
    else if (isspace(c))  { 
       str++;  i++;    
    }      
   str++; i++;       
  }  // while    
                                               
}  // print_token 


int main()  { 
    
char *mystr = "select var1 from mytable ; " ; 
   
print_token(mystr);

return 0; 
}

If I can just get this cut-down lexer to work, getting the full version up and running should be no problem.
I'll put it on Github for others to use too.
Many thanks in advance for any help received!

Cheers -
Andy (latte123)

This while loop (below) is going to just go straight past the end of the string (str) because it never checks for the terminating '\0' character. As an added bonus because str is going out of bounds so will token[i] (most likely).

Code:

        /* Keyword or identifier */
        if (isalpha(c) || c == '_')
        {
            while (isalnum(c) || c == '_')
            {
                token[i] = c;
                str++;
                i++;
            }

Originally Posted by Hodor

This while loop (below) is going to just go straight past the end of the string (str) because it never checks for the terminating '\0' character. As an added bonus because str is going out of bounds so will token[i] (most likely).

Code:

        /* Keyword or identifier */
        if (isalpha(c) || c == '_')
        {
            while (isalnum(c) || c == '_')
            {
                token[i] = c;
                str++;
                i++;
            }

Hi - thanks Hodor and stahta01 - very helpful!

As an aside (and an experiment), I've also been trying the alternative approach below (which looks very promising). It would certainly be easier to maintain -

Code:

#include <stdio.h>
#include <ctype.h>
#include <string.h>
#include <stdlib.h>


void lex_kwident(char *str, int i)  { 
  printf("I am at pos. %d in string %s\n", i, str);     
}    


void lex_string(char *str, int i) { 
  printf("I am at pos. %d in string %s\n", i, str);        
}    


void lex_number(char *str, int i) { 
  printf("I am at pos. %d in string %s\n", i, str);        
}    
    

void lex_punct(char *str, int i) { 
  printf("I am at pos. %d in string %s\n", i, str);            
}     


void lex_space(char *str, int i) { 
  printf("I am at pos. %d in string %s\n", i, str);    
}     


void lex(char *str) { 
    int i=0; 
    char c; 

while (*str != '\0') { 
    c = str[i];
              
    /* Keyword or identifier */ 
    if (isspace(c))                   lex_space(str, i); 
    else if (isalpha(c) || c == '_')  lex_kwident(str, i); 
    else if ( c == '"')               lex_string(str, i); 
    else if (isdigit(c))              lex_number(str, i);
    else if (ispunct(c))              lex_punct(str, i);
                
    str++; i++;       
  }  // while    
                                               
}  // print_token 

 

int main()  { 
    
char *mystr = "select var1 from mytable ; " ; 
   
lex(mystr);

return 0; 

}

Originally Posted by stahta01

FYI:

Code:

c = str[i];

The code above implies str value is not being changed inside your print_token function.

While you are changing both "i" and "str" inside the function print_token.

Edit: The problem seems to be you are changing both at the same time inside the function.

Tim S.

Hi again -

Ahh..... I think I may be seeing what you mean. The code outputs positions up to 13, then it jumps to 17, 19 and 21.
If I think about it - we always want str[0] *anyway* when we use str++, so I should always keep c as str[0].

I've now got the function as below and have dropped the integer parameter to the lex_ functions -

Code:

void lex(char *str) {    
    char c; 

while (*str != '\0') { 
    c = str[0];
              
    /* Keyword or identifier */ 
    if (isspace(c))                   lex_space(str); 
    else if (isalpha(c) || c == '_')  lex_kwident(str); 
    else if ( c == '"')               lex_string(str); 
    else if (isdigit(c))              lex_number(str);
    else if (ispunct(c))              lex_punct(str);
                
     str++;       
  }  // while    
                                               
}  // print_token

The code now works much better - just as would be expected. Many thanks! My brain needed a good rattle and that's just what you've given.....

Cheers -
Andy

Hey stahta01 - a HUGE thanks for posting the fixed code! *Big* improvement! I've just run it and it's great!

Thanks again - cheers -
- Andy