Binary data handling

[maverickbu]

I have a stream of binary data that I am pushing into structs and handling. The data sometimes, though, can contain variable length fields in between other fixed fields. I'm having trouble trying to decide the best way to handle structuring my structs to handle this. For example, lets say the "event1" data comes as:

- int block_type
- int block_length // includes type & length length
- char *block
- int id
- int other_data
- int block_type2
- int block_length2 // includes type & length length
- char *block2
- int another_value

The way I've handled it to this point in a piece by piece approach. As I'm dealing with a binary chunk of data and multiple variable length fields, I need know the sizeof() each section before moving on to the next. It works but is less organized and easy to look at than I'd like. For example, I've implemented the above as such:

struct _Block {
    int block_type;
    int block_length;
    char block[MAXBUFSIZE];
} __attribute__((__packed__));
typedef struct _Block Block;

and handling would be something like this:

// Handle event data chunk and return number of bytes processed so I can
// move onto the next one
int handle_data(char *rawdata) {
    Block *block, *block2;
    int *id;
    int *other_data;
    int *another_value;
    int offset = 0;

    block = (Block*)rawdata;
    // process block
    offset += block->block_length;

    id = (int*)(rawdata + offset);
    // process id
    offset += sizeof(int);

    other_data = (int*)(rawdata + offset);
    // process other_data
    offset += sizeof(int);

    block2 = (Block*)(rawdata + offset);
    // process block2
    offset += block2->block_length;

    another_value = (int*)(rawdata + offset);
    // process another_value
    offset += sizeof(int);

    return offset;
}

Size can slightly be reduced by by bundling a section like the id and other_data into another struct. Basically, is there a more elegant way to handle this? Is there a way to design my initial struct such that it contains all the fields? If so, I assume the variable length char* fields would have to be dynamically allocated?

[dwks]

Writing a structure to a file which contains pointers to dynamically allocated memory would not work. You'd have to set those members to something else when you read in the file, since the pointers would be invalid next time you read the data into memory (unless the same instance of the program was still running), which would defeat the purpose.

You could, however, do something like this:

struct string_t {
    size_t len;
    char *data;
};

Then you'd write that structure to the file followed by the data in data. When reading that structure in, you'd look at len and read that many more bytes into data.

Something like this:

struct string_t str = {3, "abc"};

fwrite(&str, sizeof(str), 1, fp);
fputs(str.data, fp);

/* ... */

fread(&str, sizeof(str), 1, fp);
read_bytes(str.len, &str.data);

where read_bytes() would read str.len bytes into str.data, which would be dynamically allocated. Maybe like this, without error checking.

void read_bytes(size_t len, char **data) {
    size_t x;
    *data = malloc(len + 1);
    for(x = 0; x < len; x ++) {
        data[x] = fgetc(fp);
    }
    data[x] = 0;
}