Thread: Parsing and Tokens (strtok)

I am having difficulty correctly tokeninzing a file that is read in.

I am taking an Assembler class, but we are to write a scanner program in C. Each terminal needs to be recoginized and printed on a separate line. Terminals can include ( : ) , ; + - * / as well as words. I have been able to tokenize the file using the space or new line (\n) as a delimiter, but cannot figure out how to use the above symbols as the delimiters also. If I add them to the "sep[]" definition, they do not print.

How can I instruct the program to identify the terminals above, print them and go on to the next part of the file?

Any assistance or a point in the right direction would be greatly appreciated!

I have attached a text file that shows the program, the output and the original text file.

Thanks!

Sandy

Can you tell us the 'exact' text ( original ) version of you problem?

Here is the document that I didn't see attached. It shows the C program, the output and the original text file that was read. What I haven't been able to figure out was a way to print the tokens like ( or ) or , etc.

/* illustrates tokenizing */

#include <string.h>
#include <stdio.h>

FILE *f;
int main()
{
char string[1000], seps[] = " \n,( )";
char *p;

f=fopen("Test.dat","r");
if(!f)
return 1;

while(fgets( string, sizeof(string)-1, f) != NULL)
{

/* Break into tokens. */
p = string;
p = strtok( string, seps ); /* Find first token*/

while( p != NULL )
{
printf("Token: %s\n", p);
p = strtok( NULL, seps ); /* Find next token*/
}
}
return 0;
}
------------------------------------------------------------
Output

Token: PROGRAM
Token: STATS
Token: VAR
Token: SUM
Token: SUMSQ
Token: I
Token: VALUE
Token: MEAN
Token: VARIANCE
Token: :
Token: INTEGER
Token: BEGIN
Token: SUM
Token: :=
Token: 0;
Token: SUMSQ
Token: :=
Token: 0;
Token: FOR
Token: i
Token: :=
Token: 1
Token: TO
Token: 100
Token: DO
Token: BEGIN
Token: READ
Token: VALUE
Token: ;
Token: SUM
Token: :=
Token: SUM
Token: +
Token: VALUE;
Token: SUMSQ
Token: :=
Token: SUMSQ
Token: +
Token: VALUE
Token: *
Token: VALUE
Token: END;
Token: MEAN
Token: :=
Token: SUM
Token: DIV
Token: 100
Token: VARIANCE
Token: :=
Token: SUMSQ
Token: DIV
Token: 100
Token: -
Token: MEAN
Token: *
Token: MEAN;
Token: WRITE
Token: MEAN
Token: VARIANCE
---------------------------------------------------------------
Actual File

PROGRAM STATS
VAR
SUM, SUMSQ, I, VALUE, MEAN, VARIANCE : INTEGER
BEGIN
SUM := 0;
SUMSQ := 0;
FOR i := 1 TO 100 DO
BEGIN
READ (VALUE);
SUM := SUM + VALUE;
SUMSQ := SUMSQ + VALUE * VALUE
END;
MEAN := SUM DIV 100
VARIANCE := SUMSQ DIV 100 - MEAN * MEAN;
WRITE (MEAN, VARIANCE)

so thats the answer for you? it's pretty easy right?

Thanks Prelude. I did end up just using the space and the new line as the delimiter, because the tokenizing deletes the delimiter.

I have tried the sccanf, but have only been able it to work on one of the "special characters" This is what I have added:

while(p!=NULL)
{
if(sscanf(p, "( %s ",p)) printf("Token: (\n"); //to print (
printf("Token: %s\n",p); // to print the rest of this token
p = strtok(NULL, seps); //Find the next token

This prints the output okay, except the ) and , and ; are printed right next to the word in front of it.

I haven't been able to get any of the other versions of sscanf to work. I've tried
sscanf(p,"%s;",p) printf("Token: ;\n");
but it just prints the ; on every other line. I'm not sure what I'm doing wrong.

You mentioned analyzing the string. Is there another way to analyze each token?

Thanks for any help in advance.
Sandy