File reading problem

This is a discussion on File reading problem within the C Programming forums, part of the General Programming Boards category; This program reads from a large file (1.5M records), and picks out my folding teammates, from among them. Counting merrily ...

  1. #1
    Registered User
    Join Date
    Sep 2006
    Posts
    8,868

    Thumbs down File reading problem

    This program reads from a large file (1.5M records), and picks out my folding teammates, from among them. Counting merrily along, and writes them out to a file, which it closes when finished.

    No problem.

    I open (or rewind) the very same file, and start printing out the records. I can use fgets() and go line by line, or I can use fscanf() and load the records into the array of structs - doesn't matter.

    Because the reading STOPS right in the middle of a name, either way.

    The file is has all the data, even checking it with a hex editor - and will load perfectly as long as I don't write it out first. (that is, if I open and read the same file created earlier by this program, and don't create a new file in the first part of this program)

    To be useful, it needs to create the file in this program.

    What I've tried:

    *changing the data types of points from unsigned long long int, to int
    *checking both files with a hex editor
    *reading the created file using fscanf() instead of fgets()
    *changing some arrays from local, to global
    *running a memory tester on the system
    *checking the temps of the cpu, and the speed
    *moved the program to another project and directory
    *ran it in debug mode. Sure enough, the exact same error!
    *removed the record it stopped on. Same problem, on the next record, now.
    *retyped the suspect record completely.
    *renamed the file.

    This is what the output prints on the bad load of the file:
    Name:  FileReadProblem2.PNG
Views: 332
Size:  15.3 KB

    The file however, has this at the end:
    Code:
    SLEEPER_V 10352
    Bukakke 10350
    Unscented 10350
    Shadi_Zumut 10341
    //This is record 2505, near the end of the file. Read stops at Hye
    Hyer 10325
    breadman 10320
    link1305 10299
    Michael_LFC 10289
    WhiteWulfe 10283
    JoshOohAh 10269
    Pyroryba 10261
    genguy 10176
    OCn00b 10170
    SAB47 10143
    PuerAeternus 10129
    Onmasses 10125
    RCtruckGUY 10093
    With_him_(www.overclockers.com) 10082
    Aaron 10067
    MameXP 10047
    stxmiddy4 10046
    JLSTurner 10041
    Deadly_Dude 10007
    This is an update (major) to this program. It's run for 3 years on Turbo C's compiler, (just 6 times per year), and I've never seen this problem before. I'm using Pelles C's compiler.

    Code:
    #include <stdio.h>       
    #include <stdlib.h>
    #include <string.h>
    #include <time.h>
    #include <limits.h> 
    
    #define Max 84 
    #define Nlength 64
    
    
    struct folder  {
       char nfield[Nlength];
       unsigned long long pfield;
       int milestone;
       int active;
    };
    
    
    struct folder A[3000];
    //char buff[Max];
    
    int main(void)  {
       
       int i, j, k, team, type, matesNew, matesOld, matesAll, loLevel, hiLevel;
       char buff[Max];
       char name[Nlength];
       //char oldname[Nlength];
    
       
       unsigned long long int credits, points, oldpoints;
    
       //struct folder* A;
       struct folder temp;
       FILE *in, *out;
       printf("\nNote: These are your highest values:\n for an unsigned long int: %29lu\n\
     for an unsigned long long int: %24llu \n",ULONG_MAX,ULLONG_MAX); 
                             
       //*** file for new records to t32_all.txt file ***
       if((in = fopen("fah_new.txt", "r")) == NULL)  {
          printf("\n Error - unable to open fah_new.txt file\n");
          printf("\n\n Press Enter to Quit");
          getchar();
          return 0;
       }
    
       if((out = fopen("t32_all.txt","wt")) == NULL)  {
          printf("\n Error - unable to open t32_dat.txt file\n");
          printf("\n\n Press Enter to Quit\n");
          getchar();
          return 0;
       }
    
       fgets(buff, Max-1, in);  //get run date and time
       fgets(buff, Max-1, in);  //get column headers 
    
       points=oldpoints=credits=0;
       team=matesNew=matesAll=0;
       //write out all teammates with > threshold points, to the t32_all.txt file
       while((fscanf(in, "%s %llu %llu %d", name, &points, &credits, &team)) != EOF)  {
          if(team == 32 && points > 10000) { 
             matesNew++;  
    //         if(matesNew > 2500) printf("%s %llu\n", name,points); 
             fprintf(out, "%s %llu\n", name, points);
          }
          team = 0;
          points = 0;
       }
       fclose(in);
       printf("\nThere are %d records in the newest data file\n",matesNew); //2524
       
       //load the new records from t32_all.txt
       in=fopen("t32_all.txt", "rt");
       if(!in) {
          printf("Error opening the t32_all.txt file!\n");
          exit(0);
       }
       printf("\n* loading data *\n");
       i = 0;                        
      
       //This is the read loop that fails after 2504 records (1 record per line) plus 3 chars.
       while((fgets(buff, sizeof(buff),in))!= NULL) {
          if(ferror(in)) {
             printf("file read error! #2\n");
             break;
          }
          if(i>2500) printf("i:%d  %s",i,buff);
          ++i;
       }
       
       fclose(in);
       printf("\n");
       return 0;
    }
    Last edited by Adak; 02-06-2013 at 02:17 AM.

  2. #2
    and the hat of wrongness Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,546
    > in=fopen("t32_all.txt", "rt");
    You're doing this without closing the out file.

    > if((out = fopen("t32_all.txt","wt")) == NULL)
    Where is the fclose()?
    You're trying to read a file where some of the file is still in some buffer somewhere, and not flushed to the file.

    This should be outside the while loop.
    Code:
       while((fgets(buff, sizeof(buff),in))!= NULL) {
          if(i>2500) printf("i:%d  %s",i,buff);
          ++i;
       }
       if(ferror(in)) {
           printf("file read error! #2\n");
       }
       if(feof(in)) {
           printf("file EOF\n");
       }
    If fgets() returns non-NULL, then there is NOTHING wrong with the stream, so testing for any file error inside the loop just won't work.
    You wait for fgets() to return NULL, then use feof() and ferror() to find out more.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  3. #3
    Registered User
    Join Date
    Sep 2008
    Posts
    200
    Quote Originally Posted by Adak View Post
    Code:
    SLEEPER_V 10352
    Bukakke 10350
    Unscented 10350
    Shadi_Zumut 10341
    //This is record 2505, near the end of the file. Read stops at Hye
    Hyer 10325
    breadman 10320
    link1305 10299
    Michael_LFC 10289
    WhiteWulfe 10283
    JoshOohAh 10269
    Pyroryba 10261
    genguy 10176
    OCn00b 10170
    SAB47 10143
    PuerAeternus 10129
    Onmasses 10125
    RCtruckGUY 10093
    With_him_(www.overclockers.com) 10082
    Aaron 10067
    MameXP 10047
    stxmiddy4 10046
    JLSTurner 10041
    Deadly_Dude 10007
    This is an update (major) to this program. It's run for 3 years on Turbo C's compiler, (just 6 times per year), and I've never seen this problem before. I'm using Pelles C's compiler.

    Code:
    #include <stdio.h>       
    #include <stdlib.h>
    #include <string.h>
    #include <time.h>
    #include <limits.h> 
    
    #define Max 84 
    #define Nlength 64
    
    
    struct folder  {
       char nfield[Nlength];
       unsigned long long pfield;
       int milestone;
       int active;
    };
    
    
    struct folder A[3000];
    //char buff[Max];
    
    int main(void)  {
       
       int i, j, k, team, type, matesNew, matesOld, matesAll, loLevel, hiLevel;
       char buff[Max];
       char name[Nlength];
       //char oldname[Nlength];
    
       
       unsigned long long int credits, points, oldpoints;
    
       //struct folder* A;
       struct folder temp;
       FILE *in, *out;
       printf("\nNote: These are your highest values:\n for an unsigned long int: %29lu\n\
     for an unsigned long long int: %24llu \n",ULONG_MAX,ULLONG_MAX); 
                             
       //*** file for new records to t32_all.txt file ***
       if((in = fopen("fah_new.txt", "r")) == NULL)  {
          printf("\n Error - unable to open fah_new.txt file\n");
          printf("\n\n Press Enter to Quit");
          getchar();
          return 0;
       }
    
       if((out = fopen("t32_all.txt","wt")) == NULL)  {
          printf("\n Error - unable to open t32_dat.txt file\n");
          printf("\n\n Press Enter to Quit\n");
          getchar();
          return 0;
       }
    
       fgets(buff, Max-1, in);  //get run date and time
       fgets(buff, Max-1, in);  //get column headers 
    
       points=oldpoints=credits=0;
       team=matesNew=matesAll=0;
       //write out all teammates with > threshold points, to the t32_all.txt file
       while((fscanf(in, "%s %llu %llu %d", name, &points, &credits, &team)) != EOF)  {
          if(team == 32 && points > 10000) { 
             matesNew++;  
    //         if(matesNew > 2500) printf("%s %llu\n", name,points); 
             fprintf(out, "%s %llu\n", name, points);
          }
          team = 0;
          points = 0;
       }
       fclose(in);
       printf("\nThere are %d records in the newest data file\n",matesNew); //2524
    
       // Try closing 'out', as some output may not be flushed.
       fclose(out);
       in = out = NULL; // I like to do this to be paranoid.
       
       //load the new records from t32_all.txt
       in=fopen("t32_all.txt", "rt");
       if(!in) {
          printf("Error opening the t32_all.txt file!\n");
          exit(0);
       }
       printf("\n* loading data *\n");
       i = 0;                        
      
       //This is the read loop that fails after 2504 records (1 record per line) plus 3 chars.
       while((fgets(buff, sizeof(buff),in))!= NULL) {
          if(ferror(in)) {
             printf("file read error! #2\n");
             break;
          }
          if(i>2500) printf("i:%d  %s",i,buff);
          ++i;
       }
       
       fclose(in);
       printf("\n");
       return 0;
    }
    Does this fix your problem?
    Programming and other random guff: cat /dev/thoughts > blogspot.com (previously prognix.blogspot.com)

    ~~~

    "The largest-scale pattern in the history of Unix is this: when and where Unix has adhered most closely to open-source practices, it has prospered. Attempts to proprietarize it have invariably resulted in stagnation and decline."

    Eric Raymond, The Art of Unix Programming

  4. #4
    Registered User
    Join Date
    Sep 2006
    Posts
    8,868
    Thanks, Salem! I typed in fclose(in) instead of fclose(out). That's a relief - and pretty funny! < ROFL >

    This is an example program I made for this post. The real program is way too large. Cunningly however I included the same line of bad code!

    Yes John, thanks very much -- problem solved.

  5. #5
    Registered User
    Join Date
    Jun 2005
    Posts
    6,301
    Generally it's not a good idea to mix calls of fgets() with calls of scanf() on the same file, but I doubt that is your problem.

    I suspect the main problem is that your code is not performing fclose(out) before attempting to open that same file for reading input.

    Other potential problems I can see on a quick look are

    1) Your data file contains a single string and a single integral value on each line, but your fscanf() call is reading a string, two unsigned values (points and credits), and a int (team). Therefore, unless the string on a new line contains a digit, fscanf() will never read values for the variables credits and team. The first thing after that is comparing team with 32, which will yield undefined behaviour (at least, on the first time through the loop - after which your code is setting team to zero).

    2) The character 't' in the options parameter for fopen() is not guaranteed to mean anything. At best, the behaviour is implementation-defined.

    3) fscanf() can return values other than EOF to indicate an error condition. You are not checking for values other than EOF.

    4) fprintf()'s return value is used to report errors, but there is no checking of that.

    5) fgets() is smart enough to cope with long lines, which can put your count out.
    Right 98% of the time, and don't care about the other 3%.

  6. #6
    Registered User
    Join Date
    Sep 2006
    Posts
    8,868
    I wanted to simplify the test program I posted here, so I replaced some rather complicated fscanf() code, with fgets().

    You are correct. The problem was I typed in fclose(in), instead of fclose(out), before re-opening the file.

    The data file has several fields and they're all read correctly. All my teammates are sent to the file, along with their current point totals.

    I added the "t" on the off chance that it might be the problem. The BIG data file is a Unix file, not DOS/Windows, but I'm on a Windows system.
    No other error checking is needed. The REAL program runs in about 15 seconds, and I know immediately if there has been an error in it.

    fgets() is not used in the real program. The data file is strictly formatted.

    Thanks, Grumpy! Although I really know my team, I will add a few more error codes to this version of the program.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Reading .Txt file problem
    By Dan05011991 in forum C Programming
    Replies: 12
    Last Post: 12-25-2010, 08:43 AM
  2. file reading problem
    By GrasshopperEsq in forum C++ Programming
    Replies: 6
    Last Post: 07-03-2008, 01:03 PM
  3. Problem reading file
    By coder_009 in forum C Programming
    Replies: 10
    Last Post: 01-15-2008, 12:22 PM
  4. file reading problem
    By chris285 in forum C++ Programming
    Replies: 5
    Last Post: 04-19-2005, 08:58 AM
  5. Problem reading a file
    By caduardo21 in forum C Programming
    Replies: 8
    Last Post: 04-18-2005, 11:51 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21