Reading file error!

This is a discussion on Reading file error! within the C Programming forums, part of the General Programming Boards category; I save a binary file by "write": Code: id=creat("file",0666); write(id, variable, 512*512*4000*sizeof(float)); close(id); The above steo can successfully save my ...

  1. #1
    Registered User
    Join Date
    May 2009
    Posts
    4

    Reading file error!

    I save a binary file by "write":
    Code:
    id=creat("file",0666);
    write(id, variable, 512*512*4000*sizeof(float));
    close(id);
    The above steo can successfully save my data.

    But when I try to read it in, by:

    Code:
    id2=open("file",0);
    if(read(id2, input, 512*512*4000*sizeof(float)) != 512*512*4000*sizeof(float)) ;
    exit(-1);

    To here, it stopped. It looks this "read" has some problems. But when I use smaller data, like 256*256*2000*sizeof(float), it works.

    How can I fix this problem?

    Thank you.
    Last edited by fishwater00; 05-20-2009 at 12:05 PM.

  2. #2
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    The normal method would be to read the file in (relatively) small chunks, so you are doing something unusual here, and without knowing why you want to do it this way, it is hard to say whether there is really a problem/what that problem really is.

    What is "variable" and what is "input"? What are you doing with the file after you load it?
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  3. #3
    Registered User
    Join Date
    May 2009
    Posts
    4
    I want to creat a "file", to store my data "variable" which include dimension NX*NZ and total time "NT". In first step, I calculate wavefield in each time, then save each snapshot at each time to "variable", therefore the total "file" size is NX*NZ*NT*sizeof(float).

    After that, In second step, I want to read "file" information from disk, so I use:

    Code:
    id2=open("file",0);
    if (id2<=0)
    {printf("can not read file \n"), 
      exit(0);
    }
    
    if (read(id2, new_variable, nx*nz*nt*sizeof(float)) != nx*nz*nt*sizeof(float))
    /*"new_variable" can be considered in second step*/
    {printf("not same size! \n");
     exit(-1);
    }
    After I run it, it shows "not same size!". I am wonder if my "read" is wrong.

    Hope I explain more clear. Thanks.

  4. #4
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    Okay, I just wanted to make sure that it makes sense for you to be using a single variable to store an entire file, but it sounds like it does. Maybe -- is this just a bunch of floats strung together? If so, it may *not* make that much sense to do that.

    Anyway, you can still buffer the file in (and out) in chunks, which will at least make it easier to determine what is going wrong. Something like:
    Code:
    int all=nx*nz*nt*sizeof(float), part=nt*sizeof(float), got, total=0, i, j;
    unsigned char file[all];
    unsigned char buffer[part];
    while ((got=read(fd,buffer,part))>0) {
           j=0;
           for (i=total, i<total+got; i++) { file[i]=buffer[j]; j++; }
           total+=got;
    }
    The integrity of the file is preserved, and there is much more flexibility here to stick some printf's in to debug the problem.
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  5. #5
    Registered User
    Join Date
    May 2009
    Posts
    4
    Thanks.

    Sorry I forgot to mention one thing.

    I define array (not a variable, sorry) as:

    Code:
    float *array;   array=malloc(nx*nz*nt*sizeof(float));  /*"array" means "variable" in previous post*/
    then I can do "write" and "read" this "variable".

    Is there something wrong?

  6. #6
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    Quote Originally Posted by fishwater00 View Post
    Is there something wrong?
    No. If "file" in my example were a float* with enough memory you could read through it as an array, which makes good sense. Of course, you will have to cast it (void*).

    As long as you don't think that you have to always deal with it that way, or write/grab the file all at once. Read works byte by byte, and you could read (eg) exactly ten bytes at time, it doesn't matter. That is what it does anyway, if you look at the standard. Since no zero bytes (null terminators) are added by read, the data is identical.
    Last edited by MK27; 05-20-2009 at 01:22 PM.
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  7. #7
    Captain Crash brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,249
    512*512*4000*sizeof(float) == 4194304000, assuming sizeof(float) == 4.

    Attempting to write over 4 gigabytes in a single system call? That's ludicrous. I doubt you even have that much data loaded into memory, since the user VM space isn't even that large (unless you're on a 64-bit platform).
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  8. #8
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    I always look at the math part last
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  9. #9
    Registered User
    Join Date
    May 2009
    Posts
    4
    I think so. I think it is too large to read in memory. Because if I read smaller binary file, like 256*256*2500, it works.

    My data is arrange by single array, that means I store 3D data in 1D array.

    Code:
    for (k=0; k<nt, k++)
         for (i=0; i<nx; i++)
              for (j=0; j<nz; j++)
                    data is aranged as data[k*nx*nz+i*nx+j];
    How can I "read" my data "NX*NZ" at kth time (like from k*nx*nz to (k+1)*nx*nz), then after a cycle, I can can "read" from (k+1)*nx*nz to (k+2)*nx*nz, and so on.

    I am not good at IO, hope you guys can give me some hints.

    Thanks.


    Quote Originally Posted by brewbuck View Post
    512*512*4000*sizeof(float) == 4194304000, assuming sizeof(float) == 4.

    Attempting to write over 4 gigabytes in a single system call? That's ludicrous. I doubt you even have that much data loaded into memory, since the user VM space isn't even that large (unless you're on a 64-bit platform).

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Quantum Random Bit Generator
    By shawnt in forum C++ Programming
    Replies: 62
    Last Post: 06-18-2008, 10:17 AM
  2. Making C DLL using MSVC++ 2005
    By chico1st in forum C Programming
    Replies: 26
    Last Post: 05-28-2008, 01:17 PM
  3. more then 100errors in header
    By hallo007 in forum Windows Programming
    Replies: 20
    Last Post: 05-13-2007, 08:26 AM
  4. Using VC Toolkit 2003
    By Noobwaker in forum Windows Programming
    Replies: 8
    Last Post: 03-13-2006, 06:33 AM
  5. Couple C questions :)
    By Divx in forum C Programming
    Replies: 5
    Last Post: 01-28-2003, 12:10 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21