Storing data in file

This is a discussion on Storing data in file within the C Programming forums, part of the General Programming Boards category; Here is the problem: I need a way to store large amounts of data (e.g. tables of doubles) in a ...

  1. #1
    Registered User
    Join Date
    Oct 2010
    Posts
    16

    Storing data in file

    Here is the problem: I need a way to store large amounts of data (e.g. tables of doubles) in a file and to be able to read the data.
    I created 2 types of save files.
    One is a .txt file
    benefits - its readable
    The other is a file that uses fwrite() and fread() to copy an array to it.
    benefits - 3 times smaller file size, no extra casting to decimal and rounding.

    On small arrays, this worked fine, but when I tried these out on a 20,000x5 array I had problems.
    The txt version worked ok, but when I load the 2nd type of file into an array(or maybe when I write the file, it writes gibberish), at some row at around 110-120 the data becomes gibberish. I can't find any problems with my code.
    My questions:
    Is there something wrong with my code?
    Is there some better functions for storing data in files?
    Is there some standard way that everyone uses to store a lot of data into files?
    Thanks a lot for the help!


    Code:
    #include <stdio.h>
    #include <stdlib.h>
    #include <time.h>
    
    #define TABLEROWS 20000
    #define TABLECOLUMNS 5
    
    int copytxttodarray(char filename[], double *array, int rows, int columns);
    //loads a .txt file(created by copyarraytotxt()) into the array.
    
    int copyfiletoarray(char filename[], double *array, int rows, int columns);
    //copies data from a file(created by copyarraytofile()) into the array.
    
    int copyarraytotxt(char filename[], double *array, int rows, int columns);
    //copies data from array into a .txt file.
    
    int copyarraytofile(char filename[], double *array, int rows, int columns);
    //copies data from array into a file.
    
    
    int main()
    {
        int i,j;
        double array[TABLEROWS][TABLECOLUMNS];
    
        //here is some code to generate random numbers in the array if you need to test
        //    srand(time(NULL));
        //    for(i=0;i<TABLEROWS;i++)
        //    {
        //        for(j=0;j<TABLECOLUMNS;j++)
        //        {
        //            array[i][j] = rand();
        //        }
        //    }
    
    
        //here are the function calls, uncomment to test them
    
        //copytxttodarray("randinput.txt", array, TABLEROWS, TABLECOLUMNS);
        //copyarraytotxt("randoutput.txt", array, TABLEROWS, TABLECOLUMNS);
        //copyfiletoarray("randinput", array, TABLEROWS, TABLECOLUMNS);
        //copyarraytofile("randoutput", array, TABLEROWS, TABLECOLUMNS);
    
        //printing out the first 120 lines of the array to check for gibberish
        for(i=0;i<120;i++)
        {
            for(j=0;j<TABLECOLUMNS;j++)
            {
                printf("%15.3lf", array[i][j]);
            }
            printf("\n");
        }
    
        printf("\n\nHave a nice day!");
    }
    
    //the functions I made
    
    int copytxttodarray(char filename[], double *array, int rows, int columns)
    {
        FILE *fp;
        int i, stoppoint;
    
        fp = fopen(filename,"r");
        stoppoint = rows*columns;
    
        for(i=0;i<stoppoint;i++)
            fscanf(fp, "%lf", array++);
        fclose(fp);
        return 0;
    }
    
    int copyarraytotxt(char filename[], double *array, int rows, int columns)
    {
        FILE *fp;
        int i, j;
    
        fp = fopen(filename,"w");
    
        for(i=0;i<rows;i++)
        {
            for(j=0;j<columns;j++)
            {
                fprintf(fp, "%.16e ", *(array++));
            }
            fprintf(fp, "\n");
        }
    
        fclose(fp);
        return 0;
    }
    
    int copyarraytofile(char filename[], double *array, int rows, int columns)
    {
        FILE *fp;
        fp = fopen(filename, "w");
        fwrite(array, sizeof(double),rows*columns,fp);
        fclose(fp);
        return 0;
    }
    
    int copyfiletoarray(char filename[], double *array, int rows, int columns)
    {
        FILE *fp;
        fp = fopen(filename, "r");
        fread(array,sizeof(double),rows*columns,fp);
        fclose(fp);
        return 0;
    }

  2. #2
    cas
    cas is offline
    Registered User
    Join Date
    Sep 2007
    Posts
    993
    If your compiler is not complaining about the commented-out calls (once they're uncommented, of course), it's broken. You are passing a 2d array of double (which winds up being a pointer to an array of double) to a function that's expecting a pointer to double. These are not the same thing! Your compiler is required to issue a diagnostic.

    The standard ways to store data are the two you've mentioned: either as text (which with floating point can be problematic, as you note) or as binary (which can be slightly painful if you're sharing the data between different systems). For the latter, you typically mandate byte-order and data format (which would certainly mean an IEEE float of some sort), and leave it up to the programmer to figure out how to convert that to the native format. If you're only ever going to read back data on the same system, of course, you can just write it out in its native form.

  3. #3
    Registered User
    Join Date
    Oct 2010
    Posts
    16
    From my understanding, the name of an array itself is a pointer to its first number, and it is fine to put it into a function that wants a pointer.
    I tried doing:
    double *arr = array;
    or
    double *arr = &(array[0][0]);
    and passing arr to the functions but it runs exactly the same.

  4. #4
    Banned
    Join Date
    Aug 2010
    Location
    Ontario Canada
    Posts
    9,547
    What cas said... plus...
    Code:
    int copyarraytofile(char filename[], double *array, int rows, int columns)
    {
        FILE *fp;
        fp = fopen(filename, "w");
        fwrite(array, sizeof(double),rows*columns,fp);
        fclose(fp);
        return 0;
    }
    You need to open the file as "wb" ... unless you want it adding carriage returns after 0s.

  5. #5
    Registered User
    Join Date
    Sep 2008
    Location
    Toronto, Canada
    Posts
    1,832
    Maybe your input file "randinput.txt" does not have enough data to fill TABLEROWS x TABLECOLUMNS.

    You should also check if the fopen() worked before going ahead with reads and writes to the file.
    Last edited by nonoob; 12-09-2010 at 05:56 PM.

  6. #6
    and the Hat of Guessing tabstop's Avatar
    Join Date
    Nov 2007
    Posts
    14,185
    Quote Originally Posted by purestr999 View Post
    From my understanding, the name of an array itself is a pointer to its first number
    Is a pointer to its first element. In the case of double [20000][5], this is an array of 20000 (arrays of 5 doubles), so the first element itself is not a number, but an array of 5 numbers.

  7. #7
    cas
    cas is offline
    Registered User
    Join Date
    Sep 2007
    Posts
    993
    Quote Originally Posted by purestr999 View Post
    From my understanding, the name of an array itself is a pointer to its first number, and it is fine to put it into a function that wants a pointer.
    I tried doing:
    double *arr = array;
    or
    double *arr = &(array[0][0]);
    and passing arr to the functions but it runs exactly the same.
    The name of an array can typically be treated as a pointer to its first member. In your case, the first member is an array. Thus you have a pointer to an array of double, as I said. Which, again, is not the same thing as a pointer to a double.

    double *arr = &array[0][0] gets you a pointer to the first element of the first array; you are not allowed to go beyond the end of it. You certainly might be able to, but the code is wrong and a compiler is free to optimize in ways that assume you're not breaking the rules.

    Since you have an array of array (which will effectively be a pointer to an array), why don't you just tell your functions to expect one? Why try shoehorning it into another type?
    Code:
    void f(double array[5][10])
    {
    }
    ...
    double array[5][10];
    f(array);
    No tricks, no fuss. Looks just like you would expect it to.

  8. #8
    Registered User
    Join Date
    Oct 2010
    Posts
    16
    Quote Originally Posted by CommonTater View Post
    You need to open the file as "wb" ... unless you want it adding carriage returns after 0s.
    This was the problem lol :P changing "w" to "wb" fixed it. What exactly does the b mean?

    You guys are all saying that when I have an array ary[5][10] then ary is a array of 5 pointers, and each pointer is a pointer to an array of 10 items?
    From what they taught us in school, and what I read in books, ary[5][10] is just one giant line of memory with 50 spaces.
    ary is a pointer to ary[0][0]
    ari[0][1] is *(ari + 1)
    ari[1][0] is *(ari + 10) (in this example)
    ari[3][4] is *(ari + 3*10 + 4)

    This seems to be right from tests I did, and this program works when I copy my data into the array[0][0] and onwards. Just change "w" to "wb" and my code works.

    Quote Originally Posted by cas View Post
    Since you have an array of array (which will effectively be a pointer to an array), why don't you just tell your functions to expect one? Why try shoehorning it into another type?
    I don't want to make my function only work for only one certain size of arrays, with my function, it will work for any array of any size, you just need to pass to the function the pointer to the first number, and how many rows and columns there are(to calculate the amount of numbers there are).
    Last edited by purestr999; 12-09-2010 at 06:30 PM.

  9. #9
    Banned
    Join Date
    Aug 2010
    Location
    Ontario Canada
    Posts
    9,547
    Quote Originally Posted by purestr999 View Post
    This was the problem lol :P changing "w" to "wb" fixed it. What exactly does the b mean?
    The usual method of finding these things out is to do what I did... look it up.

    The "b" means binary mode... no line ends are inserted.

  10. #10
    and the Hat of Guessing tabstop's Avatar
    Join Date
    Nov 2007
    Posts
    14,185
    Quote Originally Posted by purestr999 View Post
    You guys are all saying that when I have an array ary[5][10] then ary is a array of 5 pointers, and each pointer is a pointer to an array of 10 items?
    From what they taught us in school, and what I read in books, ary[5][10] is just one giant line of memory with 50 spaces.
    What it is and how it is stored are two different things. You are guaranteed that the memory is all in a straight row. But that doesn't mean that ary[5][10] is the same as ary[50]. For instance, ary+1 will point to ary[1][0], not ary[0][1].

  11. #11
    Banned
    Join Date
    Aug 2010
    Location
    Ontario Canada
    Posts
    9,547
    Quote Originally Posted by purestr999 View Post
    This was the problem lol :P changing "w" to "wb" fixed it. What exactly does the b mean?
    Ok... one more try... When you open a file in "w" mode the library functions expect lines of text that correspond to strings and will format the file accordingly. That is, it's a text file. When you open in "wb" you are turning the text formatting off and now it will write only what you tell it to write.

    You guys are all saying that when I have an array ary[5][10] then ary is a array of 5 pointers, and each pointer is a pointer to an array of 10 items?
    It is an array arranged into 5 lines of 10 entries each.

    From what they taught us in school, and what I read in books, ary[5][10] is just one giant line of memory with 50 spaces.
    Well, it's that too. But only in the sense that it is stored as a continuous block of memory. The order of the elements may or may not be what you think they are...

    This seems to be right from tests I did, and this program works when I copy my data into the array[0][0] and onwards. Just change "w" to "wb" and my code works.
    On that compiler ... there's nothing guaranteeing other compilers will store/access the data in the same order.

    I don't want to make my function only work for only one certain size of arrays, with my function, it will work for any array of any size, you just need to pass to the function the pointer to the first number, and how many rows and columns there are(to calculate the amount of numbers there are).
    You would be far smarter to take charge of the order yourself... write to disk in loops --working with ari[i][j]-- that respect the array's dimensions. Then create a complimentary function to read it back from disk in the same manner. Then you will have something that stands a chance of working on future compilers and different machines.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. parent in a binary search tree
    By roaan in forum C Programming
    Replies: 4
    Last Post: 08-26-2009, 08:08 PM
  2. Data Structure Eror
    By prominababy in forum C Programming
    Replies: 3
    Last Post: 01-06-2009, 09:35 AM
  3. xor linked list
    By adramalech in forum C Programming
    Replies: 23
    Last Post: 10-14-2008, 11:13 AM
  4. HELP with storing serial port data into txt file
    By inmaterichard in forum C Programming
    Replies: 2
    Last Post: 04-02-2008, 03:20 AM
  5. Writing and modifying data in a file
    By Micko in forum C Programming
    Replies: 2
    Last Post: 02-17-2005, 03:42 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21