Printable View
I save a binary file by "write":
The above steo can successfully save my data.Code:id=creat("file",0666);
write(id, variable, 512*512*4000*sizeof(float));
close(id);
But when I try to read it in, by:
Code:id2=open("file",0);
if(read(id2, input, 512*512*4000*sizeof(float)) != 512*512*4000*sizeof(float)) ;
exit(-1);
To here, it stopped. It looks this "read" has some problems. But when I use smaller data, like 256*256*2000*sizeof(float), it works.
How can I fix this problem?
Thank you.
The normal method would be to read the file in (relatively) small chunks, so you are doing something unusual here, and without knowing why you want to do it this way, it is hard to say whether there is really a problem/what that problem really is.
What is "variable" and what is "input"? What are you doing with the file after you load it?
I want to creat a "file", to store my data "variable" which include dimension NX*NZ and total time "NT". In first step, I calculate wavefield in each time, then save each snapshot at each time to "variable", therefore the total "file" size is NX*NZ*NT*sizeof(float).
After that, In second step, I want to read "file" information from disk, so I use:
After I run it, it shows "not same size!". I am wonder if my "read" is wrong.Code:id2=open("file",0);
if (id2<=0)
{printf("can not read file \n"),
exit(0);
}
if (read(id2, new_variable, nx*nz*nt*sizeof(float)) != nx*nz*nt*sizeof(float))
/*"new_variable" can be considered in second step*/
{printf("not same size! \n");
exit(-1);
}
Hope I explain more clear. Thanks.
Okay, I just wanted to make sure that it makes sense for you to be using a single variable to store an entire file, but it sounds like it does. Maybe -- is this just a bunch of floats strung together? If so, it may *not* make that much sense to do that.
Anyway, you can still buffer the file in (and out) in chunks, which will at least make it easier to determine what is going wrong. Something like:
The integrity of the file is preserved, and there is much more flexibility here to stick some printf's in to debug the problem.Code:int all=nx*nz*nt*sizeof(float), part=nt*sizeof(float), got, total=0, i, j;
unsigned char file[all];
unsigned char buffer[part];
while ((got=read(fd,buffer,part))>0) {
j=0;
for (i=total, i<total+got; i++) { file[i]=buffer[j]; j++; }
total+=got;
}
Thanks.
Sorry I forgot to mention one thing.
I define array (not a variable, sorry) as:
then I can do "write" and "read" this "variable".Code:float *array; array=malloc(nx*nz*nt*sizeof(float)); /*"array" means "variable" in previous post*/
Is there something wrong?
No. If "file" in my example were a float* with enough memory you could read through it as an array, which makes good sense. Of course, you will have to cast it (void*).Quote:
Originally Posted by fishwater00
As long as you don't think that you have to always deal with it that way, or write/grab the file all at once. Read works byte by byte, and you could read (eg) exactly ten bytes at time, it doesn't matter. That is what it does anyway, if you look at the standard. Since no zero bytes (null terminators) are added by read, the data is identical.
512*512*4000*sizeof(float) == 4194304000, assuming sizeof(float) == 4.
Attempting to write over 4 gigabytes in a single system call? That's ludicrous. I doubt you even have that much data loaded into memory, since the user VM space isn't even that large (unless you're on a 64-bit platform).
I always look at the math part last ;)
I think so. I think it is too large to read in memory. Because if I read smaller binary file, like 256*256*2500, it works.
My data is arrange by single array, that means I store 3D data in 1D array.
How can I "read" my data "NX*NZ" at kth time (like from k*nx*nz to (k+1)*nx*nz), then after a cycle, I can can "read" from (k+1)*nx*nz to (k+2)*nx*nz, and so on.Code:for (k=0; k<nt, k++)
for (i=0; i<nx; i++)
for (j=0; j<nz; j++)
data is aranged as data[k*nx*nz+i*nx+j];
I am not good at IO, hope you guys can give me some hints.
Thanks.
Quote:
Originally Posted by brewbuck