Thread: problem with reading struct from file generated by hexdump

  1. #1
    Registered User
    Join Date
    Dec 2008
    Posts
    8

    problem with reading struct from file generated by hexdump

    Hi,
    I am new to the forum and also a rookie C-programmer.
    So please excuse if I will ask a typical beginner question to you but this problem drives me nuts for days now.

    Here is the problem:

    I want to read a struct from a file.
    When I do this as an experiment:
    #include <stdio.h>
    #include <stdlib.h>

    typedef struct
    {
    int one;
    } record ;

    record weather;
    record new_weather;
    FILE *fp;

    int i;

    int main (void)
    {
    record *weather_ptr,*weather_ptr_start,*new_weather_ptr;
    weather_ptr =&weather;
    weather_ptr_start =&weather;
    new_weather_ptr =&new_weather;


    for (i=1;i<=10;i++)
    { weather_ptr->one = i;
    weather_ptr++;
    }

    fp =fopen("test.txt","w");
    weather_ptr = weather_ptr_start;
    for (i=1;i<=10;i++)
    {
    fprintf(fp, "%u\n",weather_ptr->one);
    weather_ptr++;
    }
    fclose(fp);
    fp =fopen("test.txt","r");



    for (i=1;i<=10;i++)
    {
    fscanf(fp, "%u\n",&new_weather_ptr->one);
    printf("%u\n",new_weather_ptr->one);
    weather_ptr++;
    }



    return 0;
    }

    All goes well.

    So I think (I think!) I have understood the mechanics of pointers, structs and file IO.

    Now I have generated a file from a binary with hexdump (unix)
    The file when looked at with vi looks OK and similar to what I have generated with above test program.

    But if I run the read part of the above program I get a crash.
    It happens to crash when I try to read the second line of the text file.
    The first struct record is OK.

    What I see is that the first line gets read in correct and the file pointer jumps to the next line.
    The next fscanf will then generate the crash.

    Do I have a misunderstanding of how hexdump works?
    Is there any special character which I do not se and which will crash fscanf?

    Thanks for your help.

  2. #2
    Lurking whiteflags's Avatar
    Join Date
    Apr 2006
    Location
    United States
    Posts
    9,613
    Well in my opinion I think you are being a bit clever with your use of pointers. Some of the operations you do, like increment, don't make any sense in the program. You're not iterating over an array. If you were, it would be an acceptable use, because arrays are guaranteed to lay out their elements sequentially in memory. I think you should write this program again without pointers.

  3. #3
    Registered User
    Join Date
    Dec 2008
    Posts
    8
    please explain , why does incrementing a pointer makes no sense?
    I thought that if I increment a pointer it will point to the next struct.

    Honestly, I think it makes no sense to have a pointer which you can not increment.

    How would you point to the next structure?

    The final use of this program is to read a unquantified amount of data.
    which means it could be a couple of datasets but also hundred of thousands.
    It is a dynamical dataset.

  4. #4
    C++ Witch laserlight's Avatar
    Join Date
    Oct 2003
    Location
    Singapore
    Posts
    28,413
    Quote Originally Posted by wollek
    please explain , why does incrementing a pointer makes no sense?
    I thought that if I increment a pointer it will point to the next struct.
    If the pointer points to some object in an array, then indeed incrementing the pointer would make it point to the next object in the array (or one past the end, in the event the pointer points to the last object in the array).

    In this case you only have two separate record objects, so incrementing the pointer does not make sense. In fact, it looks like you are dereferencing a pointer that points to memory that you do not actually own, resulting in undefined behaviour.

    Quote Originally Posted by wollek
    Honestly, I think it makes no sense to have a pointer which you can not increment.
    Then you have forgotten that pointers can point to lone objects.
    Quote Originally Posted by Bjarne Stroustrup (2000-10-14)
    I get maybe two dozen requests for help with some sort of programming or design problem every day. Most have more sense than to send me hundreds of lines of code. If they do, I ask them to find the smallest example that exhibits the problem and send me that. Mostly, they then find the error themselves. "Finding the smallest program that demonstrates the error" is a powerful debugging tool.
    Look up a C++ Reference and learn How To Ask Questions The Smart Way

  5. #5
    Registered User
    Join Date
    Dec 2008
    Posts
    8
    Okay, looks I have a fundamental misunderstanding here.

    So what would be the correct way to address a element of a structured group with unknown array size?

    In other words, I have a set of records and the amount of records is unknown.

    How would I set up a dynamical read of it?

    Just to make it clear , I do not want to see any kind of predefined array size, it should be totally dynamical, the only practical limitation should be the size of memory my workstation has.

  6. #6
    Registered User
    Join Date
    Dec 2008
    Posts
    8
    Looks like you set me in the right direction and I will be able to answer my questions myself.

    Keyword is dynamical memory allocation.

    I just remembered about 20 years ago while doing pascal programming there was something like a heap pointer.

    I will google a bit on that.

  7. #7
    Lurking whiteflags's Avatar
    Join Date
    Apr 2006
    Location
    United States
    Posts
    9,613
    >> In other words, I have a set of records and the amount of records is unknown.

    >> How would I set up a dynamical read of it?

    Based on what you have posted before, is there any reason why you can't process one record at a time? I realize you attempted this, but allow me to explain.

    Pointers typically have two important uses. One, is so that you can refer to memory returned by malloc: This memory does not have a name provided by you, so it makes sense to "point" to it. Since you can name more pointers at the beginning of statement blocks, there should be no problem referring to such memory. Another factor to this is that you control the scope of pointed to objects. If you're using memory that was returned by malloc it will only be released by a subsequent free.

    Another important use in C for pointers is when you want to pass data to a function by reference. Rather than a function receiving a copy of some object, it receives the address instead, allowing the function to operate on the remotely stored value.

    If your program does not need either or both of these features then pointers are most likely unnecessary for the task at hand.

    As far as I can tell the only thing you need to do is print the value right after reading it. There is no reason why you can't do that in the same loop, using a simple variable. What are you really doing?

  8. #8
    Registered User
    Join Date
    Oct 2008
    Location
    TX
    Posts
    2,059
    As size of the file is unknown, dynamically allocate memory using malloc() and since you want to print it later setup a linked list.
    The FAQ has a tutorial on how to go about linked lists here http://faq.cprogramming.com/cgi-bin/...&id=1073086407

  9. #9
    Registered User
    Join Date
    Dec 2008
    Posts
    8
    As I said my program has to read a unspecified amount of data.

    The struct in my example as only one element but in fact it will be a dozen more .

    Reading the data in one at a time would mean that sorting the data and working with it would need a lot of disk access which would slow down the program runtime way to much.

    Therefore I want to have all the data in memory and do all my manipulations and calculations in RAM

    Right now I slowly remember my pascal times.
    I think back then I solved such problems with a record which had at the end of the record a pointer to the beginning of the new record. So it chained through the whole set of records.

    I am sorry but it's a long time ago that I wrote programs.

  10. #10
    Registered User
    Join Date
    Dec 2008
    Posts
    8
    Thanks itCbitC!

    That's what I just remembered.
    The link will help me to get back in the saddle, thanks!

  11. #11
    Lurking whiteflags's Avatar
    Join Date
    Apr 2006
    Location
    United States
    Posts
    9,613
    Quote Originally Posted by itCbitC View Post
    As size of the file is unknown, dynamically allocate memory using malloc() and since you want to print it later setup a linked list.
    The FAQ has a tutorial on how to go about linked lists here http://faq.cprogramming.com/cgi-bin/...&id=1073086407
    I'm not convinced he needs a collection of records at all.

    Quote Originally Posted by wollek View Post
    Reading the data in one at a time would mean that sorting the data and working with it would need a lot of disk access which would slow down the program runtime way to much.
    Well don't assume; know. Usually you would code first, and then measure the actual run time performance using a program called a profiler.

    Nevertheless, that's no justification for not processing one record at a time. You just haven't thought the algorithm through:
    Code:
    #include <stdio.h>
    
    struct record {
        int one;
    };
    
    int main()
    {
        struct record weather;
    
        /** make a file **/
        FILE * fp;
        int i;
    
        fp = fopen("test.txt", "w");
        for(i = 1; i <= 10; i++) {
            weather.one = i;
            fprintf(fp, "%d\n", weather.one);
        }
        fclose(fp);
    
        /** read the file **/
        fp = fopen("test.txt", "r");
        while(1 == fscanf(fp, "%d", &weather.one)) {
            /** process **/
            printf("weather.one == %d\n", weather.one);
        }
        fclose(fp);
    
        remove("test.txt"); /** removing the file from my HD **/
        return 0;
    }
    That should be enough, based on what I actually know about your real goals.

  12. #12
    C++ Witch laserlight's Avatar
    Join Date
    Oct 2003
    Location
    Singapore
    Posts
    28,413
    Quote Originally Posted by wollek
    I think back then I solved such problems with a record which had at the end of the record a pointer to the beginning of the new record. So it chained through the whole set of records.
    Depending on your usage patterns you may want to consider using a dynamically allocated array instead of a linked list.
    Quote Originally Posted by Bjarne Stroustrup (2000-10-14)
    I get maybe two dozen requests for help with some sort of programming or design problem every day. Most have more sense than to send me hundreds of lines of code. If they do, I ask them to find the smallest example that exhibits the problem and send me that. Mostly, they then find the error themselves. "Finding the smallest program that demonstrates the error" is a powerful debugging tool.
    Look up a C++ Reference and learn How To Ask Questions The Smart Way

  13. #13
    Registered User
    Join Date
    Dec 2008
    Posts
    8

    Talking

    thanks whiteflags and laserlight

    I will read through the link itCbitC has send and play around with your suggestions.

    At the beginning I was suspicious I had a problem with my hexdump output as my little example was running as I was expecting.
    So I got sidetracked from the real problem and that was the wrong setup of my pointer mechanics.

    @whiteflags

    I can run a profiler but I do not see how a harddisk access can be equal to a memory access in speed.

    My hardware is right now in a stage where it dumps binary data on the HD but I hope to fix this so that I can stream directly into my program. Therefor I do want to have the program ready for that.


    So thanks to you all, you helped me a lot.

    Thanks!

  14. #14
    Lurking whiteflags's Avatar
    Join Date
    Apr 2006
    Location
    United States
    Posts
    9,613
    I can run a profiler but I do not see how a harddisk access can be equal to a memory access in speed.
    I don't disagree, but that's not the point. If your just reading a file and dumping it on the screen, your program's performance is I/O bound anyway. Keeping the entire data set in memory for that alone is wasteful; especially if you're true to your word and the set is huge.

    Consider that your RAM might not be enough to hold a million records at once.

    I hope you understand why I don't think array processing helps you write cat.

  15. #15
    C++ Witch laserlight's Avatar
    Join Date
    Oct 2003
    Location
    Singapore
    Posts
    28,413
    Quote Originally Posted by whiteflags
    I hope you understand why I don't think array processing helps you write cat.
    The records will be sorted and presumably manipulated in other ways, so this is not just writing cat.
    Quote Originally Posted by Bjarne Stroustrup (2000-10-14)
    I get maybe two dozen requests for help with some sort of programming or design problem every day. Most have more sense than to send me hundreds of lines of code. If they do, I ask them to find the smallest example that exhibits the problem and send me that. Mostly, they then find the error themselves. "Finding the smallest program that demonstrates the error" is a powerful debugging tool.
    Look up a C++ Reference and learn How To Ask Questions The Smart Way

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Converting from C to C++
    By Taka in forum C++ Programming
    Replies: 5
    Last Post: 04-08-2009, 02:16 AM
  2. Post...
    By maxorator in forum C++ Programming
    Replies: 12
    Last Post: 10-11-2005, 08:39 AM
  3. Binary Search Trees Part III
    By Prelude in forum A Brief History of Cprogramming.com
    Replies: 16
    Last Post: 10-02-2004, 03:00 PM
  4. towers of hanoi problem
    By aik_21 in forum C Programming
    Replies: 1
    Last Post: 10-02-2004, 01:34 PM
  5. Request for comments
    By Prelude in forum A Brief History of Cprogramming.com
    Replies: 15
    Last Post: 01-02-2004, 10:33 AM