problem with reading struct from file generated by hexdump

Hi,
I am new to the forum and also a rookie C-programmer.
So please excuse if I will ask a typical beginner question to you but this problem drives me nuts for days now.

Here is the problem:

I want to read a struct from a file.
When I do this as an experiment:
#include <stdio.h>
#include <stdlib.h>

typedef struct
{
int one;
} record ;

record weather;
record new_weather;
FILE *fp;

int i;

int main (void)
{
record *weather_ptr,*weather_ptr_start,*new_weather_ptr;
weather_ptr =&weather;
weather_ptr_start =&weather;
new_weather_ptr =&new_weather;


for (i=1;i<=10;i++)
{ weather_ptr->one = i;
weather_ptr++;
}

fp =fopen("test.txt","w");
weather_ptr = weather_ptr_start;
for (i=1;i<=10;i++)
{
fprintf(fp, "%u\n",weather_ptr->one);
weather_ptr++;
}
fclose(fp);
fp =fopen("test.txt","r");



for (i=1;i<=10;i++)
{
fscanf(fp, "%u\n",&new_weather_ptr->one);
printf("%u\n",new_weather_ptr->one);
weather_ptr++;
}



return 0;
}

All goes well.

So I think (I think!) I have understood the mechanics of pointers, structs and file IO.

Now I have generated a file from a binary with hexdump (unix)
The file when looked at with vi looks OK and similar to what I have generated with above test program.

But if I run the read part of the above program I get a crash.
It happens to crash when I try to read the second line of the text file.
The first struct record is OK.

What I see is that the first line gets read in correct and the file pointer jumps to the next line.
The next fscanf will then generate the crash.

Do I have a misunderstanding of how hexdump works?
Is there any special character which I do not se and which will crash fscanf?

Thanks for your help.

Well in my opinion I think you are being a bit clever with your use of pointers. Some of the operations you do, like increment, don't make any sense in the program. You're not iterating over an array. If you were, it would be an acceptable use, because arrays are guaranteed to lay out their elements sequentially in memory. I think you should write this program again without pointers.

please explain , why does incrementing a pointer makes no sense?
I thought that if I increment a pointer it will point to the next struct.

Honestly, I think it makes no sense to have a pointer which you can not increment.

How would you point to the next structure?

The final use of this program is to read a unquantified amount of data.
which means it could be a couple of datasets but also hundred of thousands.
It is a dynamical dataset.

Quote:

---

Originally Posted by wollek

please explain , why does incrementing a pointer makes no sense?
I thought that if I increment a pointer it will point to the next struct.

---

If the pointer points to some object in an array, then indeed incrementing the pointer would make it point to the next object in the array (or one past the end, in the event the pointer points to the last object in the array).

In this case you only have two separate record objects, so incrementing the pointer does not make sense. In fact, it looks like you are dereferencing a pointer that points to memory that you do not actually own, resulting in undefined behaviour.

Quote:

---

Originally Posted by wollek

Honestly, I think it makes no sense to have a pointer which you can not increment.

---

Then you have forgotten that pointers can point to lone objects.

Okay, looks I have a fundamental misunderstanding here.

So what would be the correct way to address a element of a structured group with unknown array size?

In other words, I have a set of records and the amount of records is unknown.

How would I set up a dynamical read of it?

Just to make it clear , I do not want to see any kind of predefined array size, it should be totally dynamical, the only practical limitation should be the size of memory my workstation has.

Looks like you set me in the right direction and I will be able to answer my questions myself.

Keyword is dynamical memory allocation.

I just remembered about 20 years ago while doing pascal programming there was something like a heap pointer.

I will google a bit on that.

>> In other words, I have a set of records and the amount of records is unknown.

>> How would I set up a dynamical read of it?

Based on what you have posted before, is there any reason why you can't process one record at a time? I realize you attempted this, but allow me to explain.

Pointers typically have two important uses. One, is so that you can refer to memory returned by malloc: This memory does not have a name provided by you, so it makes sense to "point" to it. Since you can name more pointers at the beginning of statement blocks, there should be no problem referring to such memory. Another factor to this is that you control the scope of pointed to objects. If you're using memory that was returned by malloc it will only be released by a subsequent free.

Another important use in C for pointers is when you want to pass data to a function by reference. Rather than a function receiving a copy of some object, it receives the address instead, allowing the function to operate on the remotely stored value.

If your program does not need either or both of these features then pointers are most likely unnecessary for the task at hand.

As far as I can tell the only thing you need to do is print the value right after reading it. There is no reason why you can't do that in the same loop, using a simple variable. What are you really doing?

As size of the file is unknown, dynamically allocate memory using malloc() and since you want to print it later setup a linked list.
The FAQ has a tutorial on how to go about linked lists here http://faq.cprogramming.com/cgi-bin/...&id=1073086407

As I said my program has to read a unspecified amount of data.

The struct in my example as only one element but in fact it will be a dozen more .

Reading the data in one at a time would mean that sorting the data and working with it would need a lot of disk access which would slow down the program runtime way to much.

Therefore I want to have all the data in memory and do all my manipulations and calculations in RAM

Right now I slowly remember my pascal times.
I think back then I solved such problems with a record which had at the end of the record a pointer to the beginning of the new record. So it chained through the whole set of records.

I am sorry but it's a long time ago that I wrote programs.

Thanks itCbitC!

That's what I just remembered.
The link will help me to get back in the saddle, thanks!

Quote:

---

Originally Posted by itCbitC

As size of the file is unknown, dynamically allocate memory using malloc() and since you want to print it later setup a linked list.
The FAQ has a tutorial on how to go about linked lists here http://faq.cprogramming.com/cgi-bin/...&id=1073086407

---

I'm not convinced he needs a collection of records at all.

Quote:

---

Originally Posted by wollek

Reading the data in one at a time would mean that sorting the data and working with it would need a lot of disk access which would slow down the program runtime way to much.

---

Well don't assume; know. Usually you would code first, and then measure the actual run time performance using a program called a profiler.

Nevertheless, that's no justification for not processing one record at a time. You just haven't thought the algorithm through:

Code:

---

#include <stdio.h>

struct record {
    int one;
};

int main()
{
    struct record weather;

    /** make a file **/
    FILE * fp;
    int i;

    fp = fopen("test.txt", "w");
    for(i = 1; i <= 10; i++) {
        weather.one = i;
        fprintf(fp, "%d\n", weather.one);
    }
    fclose(fp);

    /** read the file **/
    fp = fopen("test.txt", "r");
    while(1 == fscanf(fp, "%d", &weather.one)) {
        /** process **/
        printf("weather.one == %d\n", weather.one);
    }
    fclose(fp);

    remove("test.txt"); /** removing the file from my HD **/
    return 0;
}

---

That should be enough, based on what I actually know about your real goals.

Quote:

---

Originally Posted by wollek

I think back then I solved such problems with a record which had at the end of the record a pointer to the beginning of the new record. So it chained through the whole set of records.

---

Depending on your usage patterns you may want to consider using a dynamically allocated array instead of a linked list.

thanks whiteflags and laserlight

I will read through the link itCbitC has send and play around with your suggestions.

At the beginning I was suspicious I had a problem with my hexdump output as my little example was running as I was expecting.
So I got sidetracked from the real problem and that was the wrong setup of my pointer mechanics.

@whiteflags

I can run a profiler but I do not see how a harddisk access can be equal to a memory access in speed.

My hardware is right now in a stage where it dumps binary data on the HD but I hope to fix this so that I can stream directly into my program. Therefor I do want to have the program ready for that.


So thanks to you all, you helped me a lot.

Thanks!

Quote:

---

I can run a profiler but I do not see how a harddisk access can be equal to a memory access in speed.

---

I don't disagree, but that's not the point. If your just reading a file and dumping it on the screen, your program's performance is I/O bound anyway. Keeping the entire data set in memory for that alone is wasteful; especially if you're true to your word and the set is huge.

Consider that your RAM might not be enough to hold a million records at once.

I hope you understand why I don't think array processing helps you write cat.

Quote:

---

Originally Posted by whiteflags

I hope you understand why I don't think array processing helps you write cat.

---

The records will be sorted and presumably manipulated in other ways, so this is not just writing cat.

Quote:

---

Originally Posted by whiteflags

I'm not convinced he needs a collection of records at all.

---

The OP doesn't know how many records are in that file, besides (s)he needs to print them out at the end. Doesn't that say linked list loud 'n clear.

Quote:

---

Originally Posted by itCbitC

The OP doesn't know how many records are in that file, besides (s)he needs to print them out at the end. Doesn't that say linked list loud 'n clear.

---

No, it does not. You certainly can deal with an indefinite number of records and print them all if you process them one at a time. What suggests to me that a container would be useful is the need to sort and perform some unspecified manipulations of the records.

Quote:

---

Originally Posted by laserlight

No, it does not. You certainly can deal with an indefinite number of records and print them all if you process them one at a time. What suggests to me that a container would be useful is the need to sort and perform some unspecified manipulations of the records.

---

No you don't need a linked list if you want to process each record in the input file and print it out at the same time, however if you want to store them for later printing or processing then you do need a linked list as a dynamically allocated array won't do in that case. That's what I was trying to get across.

Quote:

---

Originally Posted by itCbitC

No you don't need a linked list if you want to process each record in the input file and print it out at the same time, however if you want to store them for later printing or processing then you do need a linked list. That's what I was trying to get across.

---

You do not need a linked list, but a linked list certainly would be an option. I would argue that a dynamically allocated array should actually be the "default" to use here, not a linked list. Among other things, sorting would be somewhat easier since qsort() can be used.

I just want to point out that I completely understand when 'containers' are useful but that the OPs example was purposefully misleading. All he had to do was include some clue about a sorting step or one of those unspecified operations for me to understand.

Regardless of how wrong I am it appears the OP understands perfectly what went wrong. I'm surprised I did anything worth a thank you, but... confusion happens.

Quote:

---

Originally Posted by laserlight

You do not need a linked list, but a linked list certainly would be an option. I would argue that a dynamically allocated array should actually be the "default" to use here, not a linked list. Among other things, sorting would be somewhat easier since qsort() can be used.

---

So you're implying a malloc() followed by realloc()'s??

Quote:

---

Originally Posted by whiteflags

I just want to point out that I completely understand when 'containers' are useful but that the OPs example was purposefully misleading. All he had to do was include some clue about a sorting step or one of those unspecified operations for me to understand.

---

I'll give wollek the benefit of the doubt and say that it was merely accidentally misleading :D

Quote:

---

Originally Posted by itCbitC

So you're implying a malloc() followed by realloc()'s??

---

Yes, plus tracking of the size and capacity.

Quote:

---

Originally Posted by whiteflags

Consider that your RAM might not be enough to hold a million records at once.

---

Sounds like you are a PC based programmer :-)

The little testprogram I posted was just for demonstration.
The screen output is not needed.

The main program is to work with the data. Things like maxima,minima, std deviation , average over a certain time period, a whole bunch of std. mathematical analytics.
This is the part I do not have trouble. I just had encountered the problem that my little test program ran fine while when putting it to the test on a datablock it had failed.
As I said I was chasing the hexdump and had not realized that there was something wrong with my pointer mechanics .
That's when you think you can rest you programming skills for a decade or two and pick up where you have left. :-)

So thanks again for your help, you put me on the right track.