Thread: Storing data in file

Here is the problem: I need a way to store large amounts of data (e.g. tables of doubles) in a file and to be able to read the data.
I created 2 types of save files.
One is a .txt file
benefits - its readable
The other is a file that uses fwrite() and fread() to copy an array to it.
benefits - 3 times smaller file size, no extra casting to decimal and rounding.

On small arrays, this worked fine, but when I tried these out on a 20,000x5 array I had problems.
The txt version worked ok, but when I load the 2nd type of file into an array(or maybe when I write the file, it writes gibberish), at some row at around 110-120 the data becomes gibberish. I can't find any problems with my code.
My questions:
Is there something wrong with my code?
Is there some better functions for storing data in files?
Is there some standard way that everyone uses to store a lot of data into files?
Thanks a lot for the help!

Code:

#include <stdio.h>
#include <stdlib.h>
#include <time.h>

#define TABLEROWS 20000
#define TABLECOLUMNS 5

int copytxttodarray(char filename[], double *array, int rows, int columns);
//loads a .txt file(created by copyarraytotxt()) into the array.

int copyfiletoarray(char filename[], double *array, int rows, int columns);
//copies data from a file(created by copyarraytofile()) into the array.

int copyarraytotxt(char filename[], double *array, int rows, int columns);
//copies data from array into a .txt file.

int copyarraytofile(char filename[], double *array, int rows, int columns);
//copies data from array into a file.


int main()
{
    int i,j;
    double array[TABLEROWS][TABLECOLUMNS];

    //here is some code to generate random numbers in the array if you need to test
    //    srand(time(NULL));
    //    for(i=0;i<TABLEROWS;i++)
    //    {
    //        for(j=0;j<TABLECOLUMNS;j++)
    //        {
    //            array[i][j] = rand();
    //        }
    //    }


    //here are the function calls, uncomment to test them

    //copytxttodarray("randinput.txt", array, TABLEROWS, TABLECOLUMNS);
    //copyarraytotxt("randoutput.txt", array, TABLEROWS, TABLECOLUMNS);
    //copyfiletoarray("randinput", array, TABLEROWS, TABLECOLUMNS);
    //copyarraytofile("randoutput", array, TABLEROWS, TABLECOLUMNS);

    //printing out the first 120 lines of the array to check for gibberish
    for(i=0;i<120;i++)
    {
        for(j=0;j<TABLECOLUMNS;j++)
        {
            printf("%15.3lf", array[i][j]);
        }
        printf("\n");
    }

    printf("\n\nHave a nice day!");
}

//the functions I made

int copytxttodarray(char filename[], double *array, int rows, int columns)
{
    FILE *fp;
    int i, stoppoint;

    fp = fopen(filename,"r");
    stoppoint = rows*columns;

    for(i=0;i<stoppoint;i++)
        fscanf(fp, "%lf", array++);
    fclose(fp);
    return 0;
}

int copyarraytotxt(char filename[], double *array, int rows, int columns)
{
    FILE *fp;
    int i, j;

    fp = fopen(filename,"w");

    for(i=0;i<rows;i++)
    {
        for(j=0;j<columns;j++)
        {
            fprintf(fp, "%.16e ", *(array++));
        }
        fprintf(fp, "\n");
    }

    fclose(fp);
    return 0;
}

int copyarraytofile(char filename[], double *array, int rows, int columns)
{
    FILE *fp;
    fp = fopen(filename, "w");
    fwrite(array, sizeof(double),rows*columns,fp);
    fclose(fp);
    return 0;
}

int copyfiletoarray(char filename[], double *array, int rows, int columns)
{
    FILE *fp;
    fp = fopen(filename, "r");
    fread(array,sizeof(double),rows*columns,fp);
    fclose(fp);
    return 0;
}

If your compiler is not complaining about the commented-out calls (once they're uncommented, of course), it's broken. You are passing a 2d array of double (which winds up being a pointer to an array of double) to a function that's expecting a pointer to double. These are not the same thing! Your compiler is required to issue a diagnostic.

The standard ways to store data are the two you've mentioned: either as text (which with floating point can be problematic, as you note) or as binary (which can be slightly painful if you're sharing the data between different systems). For the latter, you typically mandate byte-order and data format (which would certainly mean an IEEE float of some sort), and leave it up to the programmer to figure out how to convert that to the native format. If you're only ever going to read back data on the same system, of course, you can just write it out in its native form.

From my understanding, the name of an array itself is a pointer to its first number, and it is fine to put it into a function that wants a pointer.
I tried doing:
double *arr = array;
or
double *arr = &(array[0][0]);
and passing arr to the functions but it runs exactly the same.

What cas said... plus...

Code:

int copyarraytofile(char filename[], double *array, int rows, int columns)
{
    FILE *fp;
    fp = fopen(filename, "w");
    fwrite(array, sizeof(double),rows*columns,fp);
    fclose(fp);
    return 0;
}

You need to open the file as "wb" ... unless you want it adding carriage returns after 0s.

Originally Posted by purestr999

From my understanding, the name of an array itself is a pointer to its first number

Is a pointer to its first element. In the case of double [20000][5], this is an array of 20000 (arrays of 5 doubles), so the first element itself is not a number, but an array of 5 numbers.

Originally Posted by purestr999

From my understanding, the name of an array itself is a pointer to its first number, and it is fine to put it into a function that wants a pointer.
I tried doing:
double *arr = array;
or
double *arr = &(array[0][0]);
and passing arr to the functions but it runs exactly the same.

The name of an array can typically be treated as a pointer to its first member. In your case, the first member is an array. Thus you have a pointer to an array of double, as I said. Which, again, is not the same thing as a pointer to a double.

double *arr = &array[0][0] gets you a pointer to the first element of the first array; you are not allowed to go beyond the end of it. You certainly might be able to, but the code is wrong and a compiler is free to optimize in ways that assume you're not breaking the rules.

Since you have an array of array (which will effectively be a pointer to an array), why don't you just tell your functions to expect one? Why try shoehorning it into another type?

Code:

void f(double array[5][10])
{
}
...
double array[5][10];
f(array);

No tricks, no fuss. Looks just like you would expect it to.

Originally Posted by purestr999

This was the problem lol :P changing "w" to "wb" fixed it. What exactly does the b mean?

The usual method of finding these things out is to do what I did... look it up.

The "b" means binary mode... no line ends are inserted.

Originally Posted by purestr999

You guys are all saying that when I have an array ary[5][10] then ary is a array of 5 pointers, and each pointer is a pointer to an array of 10 items?
From what they taught us in school, and what I read in books, ary[5][10] is just one giant line of memory with 50 spaces.

What it is and how it is stored are two different things. You are guaranteed that the memory is all in a straight row. But that doesn't mean that ary[5][10] is the same as ary[50]. For instance, ary+1 will point to ary[1][0], not ary[0][1].

Originally Posted by purestr999

This was the problem lol :P changing "w" to "wb" fixed it. What exactly does the b mean?

Ok... one more try... When you open a file in "w" mode the library functions expect lines of text that correspond to strings and will format the file accordingly. That is, it's a text file. When you open in "wb" you are turning the text formatting off and now it will write only what you tell it to write.

You guys are all saying that when I have an array ary[5][10] then ary is a array of 5 pointers, and each pointer is a pointer to an array of 10 items?

It is an array arranged into 5 lines of 10 entries each.

From what they taught us in school, and what I read in books, ary[5][10] is just one giant line of memory with 50 spaces.

Well, it's that too. But only in the sense that it is stored as a continuous block of memory. The order of the elements may or may not be what you think they are...

This seems to be right from tests I did, and this program works when I copy my data into the array[0][0] and onwards. Just change "w" to "wb" and my code works.

On that compiler ... there's nothing guaranteeing other compilers will store/access the data in the same order.

I don't want to make my function only work for only one certain size of arrays, with my function, it will work for any array of any size, you just need to pass to the function the pointer to the first number, and how many rows and columns there are(to calculate the amount of numbers there are).

You would be far smarter to take charge of the order yourself... write to disk in loops --working with ari[i][j]-- that respect the array's dimensions. Then create a complimentary function to read it back from disk in the same manner. Then you will have something that stands a chance of working on future compilers and different machines.