Bit Packing Issue

I've tried searching for this but I can't find a specific thread about what I'm seeing.

Code:

---

//FLAC METADATA_BLOCK_STREAMINFO
struct streaminfo {
        uint16_t min_block_size:16;    //16 bits, 0 bits left
        uint16_t max_block_size:16;    //32 bits, 0 bits left
        uint32_t min_frame_size:24;    //64 bits, 8 bits left
        uint32_t max_frame_size:24;    //96 bits, 16 bits left
        uint32_t sample_rate:20;      //128 bits, 28 bits left
        uint32_t num_channels:3;      //128 bits, 25 bits left
        uint32_t bits_per_sample:5;    //128 bits, 20 bits left
        uint64_t total_samples:36;    //192 bits, 48 bits left
        uint32_t md5_sig1:32;          //Why doesn't this fit in the 192?
        //uint16_t md5_sig2:16;
} __attribute__ ((packed));

---

Have I done my math right? Shouldn't that fit in the remainder of the allocated bits? I'm doing a sizeof() on my system and I'm getting 28 bytes (224 bits).

FYI I'm using gcc 4.8.0 (mingw32).

Thanks!

Edit: Should alignment be a concern? 192 bits (24 bytes) is divisible by 4 bytes, which I think is the default alignment.

Works fine here.

Code:

---

$ cat bar.c
#include <stdio.h>
#include <stdint.h>
#include <limits.h>

//FLAC METADATA_BLOCK_STREAMINFO
struct streaminfo {
    uint16_t min_block_size:16;    //16 bits, 0 bits left
    uint16_t max_block_size:16;    //32 bits, 0 bits left
    uint32_t min_frame_size:24;    //64 bits, 8 bits left
    uint32_t max_frame_size:24;    //96 bits, 16 bits left
    uint32_t sample_rate:20;      //128 bits, 28 bits left
    uint32_t num_channels:3;      //128 bits, 25 bits left
    uint32_t bits_per_sample:5;    //128 bits, 20 bits left
    uint64_t total_samples:36;    //192 bits, 48 bits left
    uint32_t md5_sig1:32;          //Why doesn't this fit in the 192?
    //uint16_t md5_sig2:16;
} __attribute__ ((packed));

int main()
{
  printf("%zd\n", sizeof(struct streaminfo) * CHAR_BIT );
  return 0;
}

$ gcc -v
gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3)

$ ./a.out
176

---

Except your comments are wrong.

Hmm, when I run your code, it gives 224. Maybe its a gcc/system difference?

What specifically is wrong about my comments?

Your whole notion of "bits left" makes no sense in a packed struct.
It seems to start off assuming the underlying storage unit is 16 bits, then it becomes 32 bits and finally 64 bits.

Code:

---

#include <stdio.h>
#include <stdint.h>
#include <limits.h>

struct foo {
  uint16_t  a:10;
  uint8_t  b:2;
};

struct bar {
  uint32_t  a:10;
  uint8_t  b:2;
};

struct baz {
  uint32_t  a:10;
  uint8_t  b:2;
} __attribute__ ((packed));


int main()
{
  printf("16-bit SU, size=%zd\n", sizeof(struct foo) * CHAR_BIT );
  printf("32-bit SU, size=%zd\n", sizeof(struct bar) * CHAR_BIT );
  printf("32-bit packed SU, size=%zd\n", sizeof(struct baz) * CHAR_BIT );
  return 0;
}

$ gcc bar.c
$ ./a.out
16-bit SU, size=16
32-bit SU, size=32
32-bit packed SU, size=16

---

Without pack, the compiler (your mileage may vary if it isn't gcc) will use the widest member type to choose the underlying storage unit(*). This makes bar 32-bits, even though only 12 are used (and 16 would have been enough).
With pack, it doesn't matter what the SU is, because you've told the compiler to use as few bits as possible by squeezing out all the dead space. The size of a packed struct is just the sum of all the bit-field widths, rounded up to the next multiple of CHAR_BIT.


(*)
Note that the standard doesn't say anything about whether every SU has to be the same size, but a consistent SU size would generally be simpler to deal with.

Quote:

---

9 An implementation may allocate any addressable storage unit large enough to hold a bit-
field. If enough space remains, a bit-field that immediately follows another bit-field in a
structure shall be packed into adjacent bits of the same unit. If insufficient space remains,
whether a bit-field that does not fit is put into the next unit or overlaps adjacent units is
implementation-defined. The order of allocation of bit-fields within a unit (high-order to
low-order or low-order to high-order) is implementation-defined. The alignment of the
addressable storage unit is unspecified.

---

Your code's going to be useless on platforms that don't support the 'packed' extension, use a different endianness to the FLAC spec, or don't provide the optional uint(16,32,64)_t types. If you provide more information about the desired portability of your code, what you're trying to do, and how you plan on going about it, you'll receive better advice.

Thanks for the explanation Salem. My assumption was what threw me off.

Quote:

---

Originally Posted by Barney McGrew

Your code's going to be useless on platforms that don't support the 'packed' extension, use a different endianness to the FLAC spec, or don't provide the optional uint(16,32,64)_t types. If you provide more information about the desired portability of your code, what you're trying to do, and how you plan on going about it, you'll receive better advice.

---

Simply put, all I want to do is read everything from the file into various structs, but it looks like best practice for this type of thing is to make special struct loading functions since these field sizes aren't too friendly in terms of memory packing/alignment. I'm not a software engineer by trade, but how is something like this typically done in the industry? I imagine that its such a simple/common thing that there must be a pretty easy or elegant solution.

Thanks for hanging with me guys.

Given

Code:

---

struct foo {
    uint8_t  a:4;
};

---

You don't know in advance (and have no control over) whether the compiler uses xxxxNNNN or NNNNxxxx (the low or high nibble) - that's the "The order of allocation of bit-fields" statement in the standard.

So if your external file format has something like
Byte 1 LSB - 4 bits indicating sub-type of file
Byte 1 MSB - 4 bits reserved, currently always 0
Byte 2-3 - 16 bit count of the number of blocks, stored in little-endian

And you try this, then all sorts of things can go wrong.

Code:

---

struct foo {
    uint8_t sub_type:4;
    uint8_t reserved:4;
    uint16_t block_count;
};

---

The first being as stated above. The 4 bits you hoped would line up with the sub-type in the file actually end up being the reserved bits.
Then there is the matter of Endianness to deal with as well. If your machine is a different endian to the file format, then you're going to have to deal with a lot of byte swapping on multi-byte quantities.

If your machine is the same endian as the file format, and you have a cooperative compiler that lays out bitfields how you want them, then bitfields can be a good short-term way of solving the problem.

But if you want widely portable code (or it's forced on you), then there is little alternative to picking the file apart yourself one byte at a time.

> I'm not a software engineer by trade, but how is something like this typically done in the industry? I imagine that its such a simple/common thing that there must be a pretty easy or elegant solution.
It depends on your industry. The communications industry uses ASN.1 quite a lot, and there is a compiler which can read a spec and generate code to deal with it. Another example of a serialisation standard is XML.

Back when C was invented (on a machine with only 64KB of memory), any saving of memory was a really good idea (same goes for unions in C). So long as they only exist as in-memory data, all the portability issues surrounding them do not apply.

Ultimately, I would suggest you forget about bit fields for representing external data.
Pretty much everything about them is implementation specific, which make them all but useless for writing portable code.

I suggest reading section 8.6 (Byte Order) and 9.1 (Formatting Data) of The Practice of Programming. You might also want to look at the source code for libflac (stream_encoder.c seems relevant).

EDIT: Sorry, since you're reading data, you'd want to look at stream_decoder.c.

Thanks guys. I was aware of the endianness conversion and was planning on some post-processing. Thanks for the explanations of the non-portability of this solution. I often do have to write code for my job (though this is a personal project), and having a completely different background doesn't provide me with best practice. I think while it may result in more code, it will be simpler and will aid readability as well, both of which I find more important than a quick, hacky solution that may only work on some architectures.

Thanks again for your help guys. I'll be sure to do some reading now!