Thread: General question about undefined behavior

Originally Posted by ledow

In general, using strtok while passing out to other functions is generally a bad idea. It basically uses a "global variable" to keep track of its state, so any other code using strtok will wipe out your usage of it in the meantime. Tokenize the string, get the information you want, then act on those tokens without using strtok any more.

Additionally, conio.h is DOS-specific and nothing to do with C standards. It's "unexpected" behaviour you're experiencing.

That said, there's no reason why clrscr should interfere with strtok (I can't think of a sensible reason that clrscr would call it, but it might well be trashing RAM that affects strtok) but if you aren't properly initialising the conio functions, then you're likely to run into problems.

I would suggest you post the parts of your code that are having the problem. Don't "edit" them, don't show us something that doesn't error, show us the code that does something unexpected and your workaround. Chances are, almost certainly, that you weren't using strtok properly in the first place and that it's nothing to do with conio, clrscr or anything else.

Well the way that I design code before I implement it into a larger program, is that I test it alone - in this circumstance from other function calls the way that it is set up is like this :

Code:

 

#define PARSE_SIZE_LIMIT 81

// in a function call 

char x[PARSE_SIZE_LIMIT]; x[0] = '\0';
char y[PARSE_SIZE_LIMIT]; y[0] = '\0';
unsigned short int counta;

//lets assume x now has some string of text, in the char array and were going to pass it to string_parser

counta = string_parser(x,y,'#',-1);//find out how many '#' there are ... 

//based on counta  you will have 0,1, or more parts .......

if your text string was like :

#cabbage: red green #gofish:

it would find : two string parts:
#cabbage: red green
#gofish:

back to the code to show how the string is parsed :

Code:

string_parser(x,y,'#',1);//word now = 1 to initialize the first token search
string_parser(x,y,'#',2);//word now is greater than 1 to get the next token search

I can do this again and again as long as there is another '#' that exists in the string to find but thats what counta is for

let me post part 2 of this here
to show you how it works

Part 2 :

@ledow, et al

this is a ANSI C program that is portable so you can compile it if you wish

Code:

//re-testing parse_text for WATT

#include <stdio.h>
#include <string.h>

// original design setting:
#define PARSE_SIZE_LIMIT 81

short int string_parser(char *, char *, char, short int);//TEXT_PARSER
short int kfgets(char *);//KFGETS
short int ynreturn(void);//yes/no option

int main (void)
{
   char target[PARSE_SIZE_LIMIT]; target[0] = '\0';
   char worda[PARSE_SIZE_LIMIT]; worda[0] = '\0';
   char uword[PARSE_SIZE_LIMIT]; uword[0] = '\0';
   short int count,countb,x,y;

   do
   {

    do
    {
      printf("\n#>");
      kfgets(target);



      count = string_parser(target,worda,'#',-1);

      if(count == 0) { printf("you must enter something using \'#\'\n"); }

    } while (count == 0);

    string_parser(target,worda,'#',0);
    printf("you entered : %s\n",worda);


    //count # parts

    printf("found %d parts by \'#\'\n\n",count);

    for(x = 1;x <= count;x++)
    {
      string_parser(worda,uword,'#',x);

      printf("part#%d : %s\n",x,uword);
    }


    //count spaced parts

    count = string_parser(worda,uword,' ',-1);

    printf("found %d parts by \' \'\n\n",count);

    for(x = 1;x <= count;x++)
    {
      string_parser(worda,uword,' ',x);

      printf("part#%d : %s\n",x,uword);
    }

    printf("\ntest again (y/n)?");

   } while((ynreturn()) == 1);



   return 0;
}


short int string_parser(char *userstring, char *target, char magic, short int word)
{
  //subject must be static
  static char subject[PARSE_SIZE_LIMIT];//stores 80 chars +1 for the null character

  char *w = &magic;//assign the address of magic to w

  char a = 0,b = 0,c = 0,x = 0,z = 0;//temp variables

  short int length = 0,orig = 0;// get the length of the string
  short int offset = 0,y = 1;//segment control
  char *p;//used for assigning pointer address returned by strtok()

  //** format subject array of 'natural' garbage **

  for (x = 0;x < PARSE_SIZE_LIMIT; x++) { subject[x] = ' '; }
  x = 0;

  orig = strlen(userstring);//get size length target array

  //offset the copy of the userstring by 1 if the first character is not magic

  //if(userstring[0] != magic)
  //{
     //printf("offset detected\r\n");//temp
     //offset = 1;
     //subject[0] = magic;
  //}

  //copy and reformat the userstring into, 0 - PARSE_SIZE_LIMIT ; format.

  //for lines larger or equal than the size limit....
  if (orig >= (PARSE_SIZE_LIMIT - 1))
  {
    for (x = 0; x <= (PARSE_SIZE_LIMIT - 1); x++)
    { subject[x] = userstring[x]; }
  }

  //for lines less than the size limit....
  if (orig < (PARSE_SIZE_LIMIT - 1))
  {
    for (x = 0; x <= orig; x++)
    { subject[x] = userstring[x]; }
  }

  subject[x+1] = '\0';//add terminating null character

  //if (word == -1) { printf("<%s> ",subject); }//temp - keep

  length = strlen(subject);//get string length of subject[];

  //count where all the "magic" characters are
  for(x = 0;x < length;x++)
  {
     //if the subject character = "magic" character then count it
     if(subject[x] == magic)
     {
       a = x;//set a to x , to match where x is currently at.

       // check for no words between spaces if x > 0
       // if x = 0 then do nothing....
       if(x > 0)
       {
         c = a - b;// c = the new count (of x) - the old count (of x)

         //a magic character must have a difference between it's last
         //position (if there is one) and its new position greater
         //than 1 character to be counted as magic character.
         //Therefore, if "#" is a magic character and a string is,
         //"##magic:" it would be discounted. A string such "# #magic:"
         //would be counted as two magic characters

         if(c > 1) { y++; } /* if c > 1 then add 1 to y */
       }

       b = a;//set b to a , to save the previous count.
     }
  }

  //if the original length of the userstring is (PARSE_SIZE_LIMIT - 1)
  //then subtract 1
  if (orig >= (PARSE_SIZE_LIMIT - 1)) { y -= 1; }

  //return the number parts in the string
  if(word == -1)
  {
    //printf("= %d\r\n",y);//temp - keep
    return y;
  }

  //return the modified user string
  if(word == 0) { strcpy(target,subject); }

Before I had DJGPP - was using a really old ( obsolete ) compiler , Borland Turbo C++ for DOS v3.0 which used the code as an example on how to use strtok(); but it works even under DJGPP
so to continue the rest of my function listing ....

Code:

  //copy the first word in the string (must be done to find the next words)
  if(word == 1)
  {
    p = strtok(subject,w);
    strcpy(target,p);
  }

  //copy all the next words (sequencially) in the string
  if(word > 1)
  {
    p = strtok(NULL,w);
    strcpy(target,p);
  }

  return 0;
}

//generic string prompt - size was eliminated for TC++
short int kfgets(char *target)
{
   short int a;//temp. variables
   char line[81];//temp string storage

   //user must provide prompt for whatever information wanted from the user

   //format the strings
   for(a = 0;a < 81; a++) { line[a] = '\0';}

   fgets(line , 81, stdin);
   a = strlen(line);
   line[a-1] = '\0';//get rid of the newline character added by fgets

   strcpy(target,line);// copy line to the target array

   return 0;
}

short int ynreturn(void)
{

    short int result = 0;
    char x[3];

    while (fgets(x,3,stdin) != NULL && x[1] != '\n');

    switch(x[0])
    {
       case('Y'):
       case('y'):
       {
          result = 1;
          break;
       }

       case('N'):
       case('n'):
       {
          result = 0;
          break;
       }

       default:
       {
          result = 0;
          break;
       }
    }

    return result;
}

so what I am trying to do in my bigger program is break up strings by '#' and then by ' '.....

so in my big program a text line such as :

#clear:

...causes

#newline: 1

...to be parsed as :

#newline:

using the '#' as the token
before its parsed with ' ' as the token

The reason that I started this post as undefined behavior is that DJGPP was compiling my big program to allow parsing of the text line

#newline: 1

as

#newline: 1

other compiliations via DJGPP would do the previous behavior even though the only change
I was doing was commenting out cprintf statements that would show me what the value of
what was going on durring the process as a means of debugging my larger program.
So the way my bigger program works

is that in text file is that #newline: 1 is parsed as #newline: 1 if it comes before #clear: which
is parsed as #clear using '#' as the token. if the two lines are switched the other way around then
the problem occurs.

In the end, my bigger program compiles fine no matter what ... which makes trying to find what IS causing the undefined behavior to be troublesome

Im going to be posting my bigger program source code listing on sourceforge as soon I finish understanding git. I actually have a functioning program that I saved that was compiled with
the Borland compiler, that works.

Originally Posted by ZuK

Code:

 char *w = &magic;//assign the address of magic to w

Code:

p = strtok(subject,w);

w is just a pointer to a single char.
strtok expects a 0-terminated cstring deliminter

A simple case of wrong use of a library function.

Kurt

That's what I didn't understand about the syntax of strtok(); if it required a string or a character, long ago - but I've seen it used with both and since its strange that neither my Borland or DJGPP compiler didn't complain - I thought it was valid .

That is something no compiler can complain about. After all you passed a pointer to a chr to strtok() the compiler can't check that there isn't a \0 following.
Kurt

ISO/IEC 9899:1999 (E) ©ISO/IEC

7.21.5.8 The strtok function
Synopsis
1 #include <string.h>
char *strtok(char * restrict s1,const char * restrict s2);

Description
2 A sequence of calls to the strtok function breaks the string pointed
to by s1 into a sequence of tokens, each of which is delimited by a
character from the string pointed to by s2. The first call in the
sequence has a non-null first argument; subsequent calls in the
sequence have a null first argument. The separator string pointed to
by s2 may be different from call to call.
3 The first call in the sequence searches the string pointed to by s1
for the first character that is not contained in the current
separator string pointed to by s2. If no such character
is found, then there are no tokens in the string pointed to by s1 and
the strtok function returns a null pointer. If such a character is
found, it is the start of the first token.
4 The strtok function then searches from there for a character that is
contained in the current separator string. If no such character is
found, the current token extends to the end of the string pointed to
by s1, and subsequent searches for a token will return a null
pointer. If such a character is found, it is overwritten by a null
character, which terminates the current token. The strtok function
saves a pointer to the following character, from which the next
search for a token will start.
5 Each subsequent call, with a null pointer as the value of the first
argument, starts searching from the saved pointer and behaves as
described above.
6 The implementation shall behave as if no library function calls the
strtok function.

Returns
7 The strtok function returns a pointer to the first character of a
token, or a null pointer if there is no token.
8 EXAMPLE

Code:

#include <string.h>
static char str[] = "?a???b,,,#c";
char *t;
t = strtok(str, "?"); // t points to the token "a"
t = strtok(NULL, ","); // t points to the token "??b"
t = strtok(NULL, "#,"); // t points to the token "c"
t = strtok(NULL, "?"); // t is a null pointer

Ok - using the above text , I am not seeing a difference with what I am doing in conflict with what the standard dictates. Please highlight or underline what it is that I am not following, because I am confused, at this point.

Ok - using the above text , I am not seeing a difference with what I am doing in conflict with what the standard dictates. Please highlight or underline what it is that I am not following, because I am confused, at this point.

O_o

Post #17): strtok expects a 0-terminated cstring deliminter

We don't need to highlight the standard. You can see for yourself where the standard says "string" arguments are expected, but your confusion is born pointer and address syntax.

Do you know what makes a C string?

Do you understand the difference between:

Code:

char s = '\0';
// ...
char * s = "";

Soma

Originally Posted by phantomotap

O_o

Post #17): strtok expects a 0-terminated cstring deliminter

We don't need to highlight the standard. You can see for yourself where the standard says "string" arguments are expected, but your confusion is born pointer and address syntax.

Do you know what makes a C string?

Do you understand the difference between:

Code:

char s = '\0';
// ...
char * s = "";

Soma

The top statement is a char assigned the Null character
The bottom statement is a char pointer assigned to a empty char string

A C string is actually a array of characters such as :

Code:

char s[10] = "hello\0";

I think I understand now that its the null character that makes the difference in how strtok(); works. Even though there is no
provision in the C standard to check that a Null character is there or not depending on the first function call or the next function calls - it ends up producing the problem of the undefined behavior. I would call this a oversight in the standard.

Originally Posted by kjwilliams

Even though there is no provision in the C standard to check that a Null character is there or not depending on the first function call or the next function calls - it ends up producing the problem of the undefined behavior. I would call this a oversight in the standard.

It's just part of the standard, and if the terminating null character is supposed to be there, then the functions are supposed to be able to depend on that. There are alternative functions where you also specify the maximum numbers of characters, like strncmp().

Another "classic" method is p strings. Wiki articles:

C string handling - Wikipedia, the free encyclopedia

String (computer science) - Wikipedia, the free encyclopedia

C++ also has a string template class, where the string class keeps track of the size of the string and allows the size of the string to be changed.

String (C++) - Wikipedia, the free encyclopedia

Even though there is no provision in the C standard to check that a Null character is there or not depending on the first function call or the next function calls - it ends up producing the problem of the undefined behavior.

O_o

If I understand you, there can be no oversight in the standard in this regard; it is impossible, in C, to determine if a pointer passed to a function is a pointer to a single `char' or a null-terminated string. (You could loop until you found the terminator, but that would only tell you that you eventually found the terminator not that a string was offered.) Using null-terminated strings are just a part of working with standard string functions, I suggest you get used to the idea of strings as a protocol and not a type.

Soma

Code:

  char w2[1]; w2[0] = magic;
    ....
  if(word == 1)
  {
    p = strtok(subject,w2);
    strcpy(target,p);
  }

Still wrong. w2 is a single char array. ( no 0-termination )
Kurt

Originally Posted by kjwilliams

Code:

  /*
    note: strtok expects the token to be a cstring without a null character
          on the first use, and a token to be a cstring *with* a
          null character every use after the first.
  */

Simply not true

Kurt