fseek failing - Error: Invalid argument

Hi all,

I have a file consisting of a huge array of floats written in binary mode (fwrite). Its actually a contiguous 2D array. I need to access various parts of the array and for this I use fseek to jump a different point and the fread to read in the data.

I noticed that some of my output coming out of the fread was arrays completely filled with only zeros. I could trace the problem to my fseek statement. For certain values of offset , my fseek just fails and checking errno gives me the error "Invalid argument". My code looks like this..

Code:

 

   int start_row=18375, n_cols =2960, n_rows_to_read =9000;

   float *array;

   fp = fopen(filename, rb);

   array = (float*)malloc(n_rows_to_read*n_cols*sizeof(float) );
  
   if( fseek(fp, start_row*n_cols*sizeof(float), SEEK_CUR) != 0)
   {
        perror("Fseekfailed");
  
        exit(0);
    }

   /** fread and write into outfile.. **/

For the exact start row that I have set here.. 181375, the code works fine. But beyond that, 18376 onwards fseek fails and the error, as I already said, is reported as "Invalid argument". I was not able to find any information on why fseek would give an invalid argument error.

When I read in 9000 rows starting from 181375, the output is fine for all the 9000 rows after it. So I know for a fact that the file isn;t blank beyond row 181375. But if I am not hitting end of file why else would fseek give this error??

Appreciate any help from you guys..

thanks,

Avinash

What is 4 * 2960 * 181376 ? Does that fit in a 32bit int?


Quzah.

Because when you go beyond row 181375, the offset becomes

181375 * 2960 * 4 = 2,147,480,000

... which is close to the limit 2,147,483,648 permissible.

Beyond that your calculation overflows, making the parameter negative integer... which is invalid.

Go find the fseek() function that can take 64-bit integer offsets.

Oh.. never thought about that. So the problem is that fseek defines its offset to be a signed integer. And my number is an invalid signed integer.. Makes sense.

I guess I ll ve to use a conditional and fseek in steps which will fit in an int.. Right?

Thanks a lot for the reply. Never thought in that direction..

cheers,

Avinash

Code:

fseek(fp, start_row*n_cols*sizeof(float), SEEK_CUR)

In addition to what others have said, SEEK_CUR is a seek relative to the current location in the file. Are you sure you don't want SEEK_SET?

And a negative number is not by definition invalid... depends if you seek off either end of the file. (Edit: But overflowing your integer is - take their advice.)

Damn!! SEEK_SET... I should have caught that.

I fixed the integer overflow and it works fine now.! Thanks for the help guys..
Posting my code here incase anyone is interested..

Code:

int max_fseekrow;

int start_row=18375, n_cols =2960, n_rows_to_read =9000;

   float *array;

   max_fseekrow = 2147483647 /n_cols*sizeof(float);

   fp = fopen(filename, rb);

   array = (float*)malloc(n_rows_to_read*n_cols*sizeof(float) );

   fseek_checkpt:

   if(start_pt > max_fseekrow)
   {

       fseek(fp,max_fseekrow*n_cols*sizeof(float),SEEK_CUR);
       start_pt = start_pt - max_fseekrow;
     
       goto fseek_checkpt;
   }

   else
   {
      fseek(fp, start_pt*ncols*sizeof(float), SEEK_CUR);
   }


   /** fread and printout **/

@Cactus: Yea.. actually, the first few lines of my file are comment/title lines. So in my code I have to fgets them out before my binary array begins and I can start to fseek. Ergo the SEEK_CUR.. Didnt seem connected thats why didnt put it in the sample code I put up.

Except that your
max_fseekrow = 2147483647 /n_cols*sizeof(float)

should be

max_fseekrow = 2147483647 / (n_cols * sizeof(float) )

Hi all,

Sorry to be reporting to an old thread. But I just realised that the problem that I announced as having been fixed above was actually not properly fixed. The issue was that I was unable to fseek beyond a certain number of rows in my file because the offset number was overflowing the max number that could be held in an int.

To fix this, i tried to do a stepped fseek as given in the above code. Basically, if my offset value was beyond the max possible int, I fseeked until the max possible row. And then fseek again to move over the remaining rows.. I thought that this had fixed the problem because I started to get some output. But i neglected to verify that output at that time. I now realised that though the filepointer moves to somepoint in the file.. Its not to the correct point.

I ve tried googling about but didnt turn up anything. Does anyone know if doing 2 fseeks one after the other is known to have any issues.

To test the code, I created a huge binary file in which each row is an array of floats filled with the row number. So by printing out the array which is read, I can know what is the array I am reading. This code is appended below

Code:

#include <stdio.h>

int main()
{

   FILE *fp;

   float *array;

   int i, j;

   int  n_records = 181500, mz_range = 2960; /** 181375 is the no of rows at which my offset over flows the max int value. So just picked an arbitrary value above that to be the no of rows **/

   
   if( (fp =fopen("fseek_test.txt","wb")) == NULL)
   {
     printf("Error.. can't open fseek_test.txt\n");
     exit(0);
   }


   array = (float*) malloc(mz_range*sizeof(float));

   fprintf(fp,"File created for the purpose of testing my stepped fseek code\n");


   for(i=0; i<n_records; i++)
   {
     
      for( j=0; j<mz_range; j++)
      {
        array[j] = i;
      }

    
      fwrite(array, sizeof(float), mz_range, fp);

   }
      

  fclose(fp);    
     
  printf("Test file created!!\n");

  return 0;

}

The code to test if my stepped fseek works is given below.

Code:

#include <stdio.h>
#include <string.h>

int main()
{
  
  FILE *fp;

  char szbuf[5000];

  int i, max_fseekrow, mz_range = 2960;
  
  int row = 181500, offset_row;

  float *array;


  if( (fp=fopen("fseek_test.txt","rb")) ==NULL)
  {
    printf("Error.. unable to open fseek_test.txt\n");
    
    exit(0);
  }


  array = (float*) malloc(mz_range*sizeof(float));

  fgets(szbuf,5000,fp); /** reading out the title line **/

  printf("%s\n\n",szbuf);


  max_fseekrow = (int) 2147483647 /( mz_range * sizeof(float) ); /** Computing max_fseekrow.. 2147483647 is the biggest possible 32 bit integer **/

   
  offset_row = row;


  printf("Offset row= %d\n",offset_row);

  fseek_checkpt:   
  if( offset_row > max_fseekrow)
    {

       fseek(fp, max_fseekrow*mz_range*sizeof(float), SEEK_CUR);

       printf("Done with the first fp movement\n");

       if( (fread(array, sizeof(float), mz_range, fp)) != mz_range)
       {

         printf("Possible read error in fread");

         exit(0);

       }

   
       for( i=0; i<10; i++)
       {

          printf("%f\t",array[i]);
       }



       offset_row = offset_row - max_fseekrow -1;

       goto fseek_checkpt;
    }

   else
   {

     if( ( fseek(fp, offset_row*mz_range*sizeof(float), SEEK_CUR)) != 0)
     {
       printf("Error.. fseek didnt work!\n");
       exit(0);
     } 

   } 


  
   
   if( (fread(array, sizeof(float), mz_range, fp)) != mz_range)
   {

     printf("Possible read error in fread");

     exit(0);

   }

   
   for( i=0; i<10; i++)
   {

      printf("%f\t",array[i]);
   }



  fclose(fp);

  return 0;
 }

As you can see, what I try in the code is just to calculate the max number of rows that I can fseek over without the offset value overflowing int. Then if the offset rows exceeds this max number of rows, then I first fseek over the max_fseekrows. And then to get to the point that i wanted to go to, I fseek over the rest of the file (offset_row - max_fseekrow). [Note: since I read in 1 row to check if the first fseek worked properly, I am setting offset_row to offset_row - max_fseekrow -1.]

The fread after the first fseek gives out the proper line. But the fread after the second fseek only prints out zeros. Irrespective of whether I want to offset it by offset it by offset_rows or just 1 row. It all gives out a zero value. Commenting out the second fseek and just doing freads hawever gives me proper output. So I am sure the file isn;t empty or anything. Does anyone have any ideas on what I am doing wrong.. I would be grateful for any advice..

best,

Avinash

You should be looking at the fseeko()/fseeko64() family of functions, which use an off_t/off64_t instead of an integer to specify the offset. Using certain macro definitions, you can cause the off_t type to become 64-bit, allowing seeks larger than 2^31-1.

Thanks for the quick reply brewbuck. I am doing my developing on Windows using VC++. Do you happen to know how to get fseeko64() in VC++ environment..? Or do u know of any alternatives that I can use on windows?

thanks

Originally Posted by avi2886

Thanks for the quick reply brewbuck. I am doing my developing on Windows using VC++. Do you happen to know how to get fseeko64() in VC++ environment..? Or do u know of any alternatives that I can use on windows?

thanks

The support for these (C99 standard!) functions is flaky in MSVC. In that case, look at fgetpos() and fsetpos(). They are less straighforward to use (they take an fpos_t *, which is actually a pointer to a structure) but by wrapping them in convenience functions you can abstract that away.

Since fsetpos() does not allow SEEK_CUR or SEEK_END-like behavior, you'll need to emulate them yourself (with unportable twiddling) -- SEEK_END can be emulated by using a normal fseek( fp, 0, SEEK_END ) to get to the end of the file (which works even for files larger than 2^31-1) followed by fgetpos() to get the fpos_t corresponding to the end of the file.

Yeah, it blows.

I haven't worked with the fpos_t datatype before. Does it store the position in bytes? I mean if I wanted to move to a certain bytes away from the current position, how should I increment the position?

I tried

Code:

 fgetpos(fp, &curr_pos);

   
  offset_pos = curr_pos + offset_row*mz_range*sizeof(float) ;

  fsetpos(fp, &offset_pos);

Where offset_row*mz_range*sizeof(float) is the number of bytes that I want to move away from the current position. but this doesn't seem to work either..

Originally Posted by avi2886

I haven't worked with the fpos_t datatype before. Does it store the position in bytes? I mean if I wanted to move to a certain bytes away from the current position, how should I increment the position?

fpos_t is intended to be an opaque type which you should not manipulate directly. That's why I referred to this as "unportable twiddling." Unfortunately, until MSVC provides better (i.e. standard) support for general large file seeks, this is the best solution I'm aware of.

You'll need to find where fpos_t is defined, look at it and figure out how to adjust the values contained therein to control the offset. I did this about a year ago, and if I remember it wasn't too difficult. Again, it's completely non-portable though.

Windows obviously supports large files, and large file seeking. But in typical MS fashion these features are only really accessible through non-standard APIs.

_fseeki64