file size with binary write larger than it should be?

Hi everyone.

I have what seemed to be a simple problem at the start (famous last words) but now I am pretty confused as to whats going wrong.

I need to write several structures to a binary file. Within these structures are more, smaller structures, and within those are the actual data types (unsigned shorts, unsigned chars) that I want to keep track of. The following are the 'base' structures':

Code:

---

struct Time
{
 unsigned short hour;
 unsigned char minute;
};
struct LogVersion
{
 unsigned char digit1;
 unsigned char digit2;
};
struct Date
{
 unsigned char month;
 unsigned char day;
 unsigned char year;
};
struct Counter
{
 unsigned short count;
};
struct BatterySlope
{
 unsigned char AverageSlope;
 unsigned short ComputationCount;
};
struct PowerParams
{
 unsigned char AverageSlope;
 unsigned char AverageOffset;
};
struct AverageValue
{
 unsigned char Average;
};

---

Now, when I try and write to file all of the structures, according to my math, I should have a file size near 550 bytes. However when I am done writing the file and go to check, the size is about 710. Big whoop you might say, but it is a requirement for this project that I get the file size to under 512 bytes, and it ought to be much closer than 710 right now. Keep in mind that I am only testing the application and do not have much, if any, manipulations to these data types and are being written as their default values of 0. So in trying to figure out what I am doing wrong, I wrote only one of the larger structures that has a collection of 3 sub structures, each with 5 ushort values:

Code:

---

struct mlog
{
  struct Counter Count1[5];
  struct Counter Count2[5];
  struct Counter Count3[5];
};

---

This ought to yield a file size of 30 bytes, and it does. I move onto the next one, which is larger:

Code:

---

struct glog
{
 struct LogVersion Version;
 struct Date CurrentDate;
 struct Counter BootCount[6];
 struct Counter LockCount[5];
 struct Counter ProgramCount[5];
 struct Counter StandbyCount[6];
 struct Counter RCVolumeCount;
 struct Counter RCSensitivityCount;
 struct Counter RCProgramCount;
 struct Counter RCZoomControlCount;
 struct Counter DuoPhoneCount;
 struct Counter ListeningCheckCount;
 struct Counter iComBTPhoneCount;
 struct Counter iComA2DPCount;
 struct Counter iComJackCount;
 struct Counter iComFMCount;
 struct Counter ErrorCount;
 struct Time ZoomCtlUseTime;
 struct Time DuoPhoneUseTime;
 struct Time iComStreamingUseTime;
};

---

Did a little counting, and based on what I have here, the file size ought to be 80 bytes, based on having 80 bytes worth of data scattered in this structure. I check the file size after write, and it is 88 bytes??? I have gone through and written the items individually to a file as well, and got the same result. I also have written just one item to file to be absolutely sure htat my items are the right size. For some reason if I try and write, for example, CurrentVersion.digit1, and then CurrentVersion.digit2 to file by themselves, the file size is 4 bytes when I have only written two 1-byte chars??? I assumed that this would bypass the whole 'structure padding' problem that I have read about as far as adding extra bytes to structures and putting those into a write, but it seems to have not done so if this ends up being the problem, and this is the point where I am seriously questioning my understanding of everything I am working with and its not just 'woops, I put in a short where I shoulda put in a char.'

I have a third structure that I won't list because its already getting a little wordier than I wanted, but this structure is even larger than 'glog', has more of this mystery error in file size, and to top it off I have an array of 5 of those high size structures, which creates the gobs of extra file space that I sorely need to remove. Can someone tell me if theres some kind of fundamental understanding I am missing between structures, writing them to files, or something else I've overlooked?

Look up "padding and alignment".

sizeof(struct BatterySlope) is almost certainly NOT going to be 3.

Quote:

---

Originally Posted by Salem

Look up "padding and alignment".

sizeof(struct BatterySlope) is almost certainly NOT going to be 3.

---

See that's another thing. My 'sizeof' function is not working as I expect it to. If I print out a 'sizeof' anything, it will tell me the amount of items it contains, as opposed to the amount of bytes contianed in the data. For example, BatterySlope's size, according to sizeof, is 2, and not 3, you are right. But in testing further I have written just a simple printf("%d", sizeof(unsigned short)) and I get 1 instead of 2 (bytes) as I expect I should.

Is the extra space really a padding/alignment problem though? Because I thought I would have bypassed that by accessing each individual item within a structure and writing just that to the file, but it doesn't seem to work. Perhaps I am getting an extra byte of space put into structures that contain more than one data type. For example struct BatterySlope having a slope and an offset will cause a single byte to be placed between them, increasing the size of the structure and therefore the size of the file?

I'll look into padding and alignment more, thank you.

What machine are you compiling for?

What value is CHAR_BIT on your machine?

Code:

---

#include <stdio.h>
#include <limits.h>
int main ( ) {
  printf("%d\n",CHAR_BIT);
  return 0;
}

---

Quote:

---

Originally Posted by Rath

See that's another thing. My 'sizeof' function is not working as I expect it to. If I print out a 'sizeof' anything, it will tell me the amount of items it contains, as opposed to the amount of bytes contianed in the data. For example, BatterySlope's size, according to sizeof, is 2, and not 3, you are right. But in testing further I have written just a simple printf("%d", sizeof(unsigned short)) and I get 1 instead of 2 (bytes) as I expect I should.

---

I wonder if you're just not doing something right. Like, sizeof returns a size_t, which is an unsigned integer type. But from the symptoms you describe, it's not the signedness of what you print that makes it wrong. It's more like you're making it off by one somehow. If you're curious enough, compile and run sys-sizes. It will tell you the various sizes of types for your machine correctly. If there are differences from what you've experienced, you definitely have a bug in your code somewhere.

After reading a link I found here regarding padding/alignment: Structure Member Alignment, Padding and Data Packing | GeeksforGeeks

I think I roughly understand a little bit of the problem I seem to be having. When I try and write a 'char' value to file (through fwrite if that makes a difference), it seems to always, no matter what I do, want to add another single byte with it. This is because the way the padding works between chars and other memory items, the memory is in 4 byte blocks, so that integers will have 0 bytes padded. Also, ushorts will have 0 bytes padded because it is possible to access memory in 2 byte chunks. However with 1 byte chars, there is no way to access this single byte of memory because there will always be an extra 1 byte of padding, as the minimum size of a single padded memory read is 2 bytes? Am I somewhat on base with this?

If all that is true, then my problem seems clear. I need to find a way to be able to get that single byte into a file, without the extra byte that's attached to it. A solution I found was to use #pragma pack, but it seems to be very taboo from what people say about it, causing unsafe memory accessing and the like. Is there anything else that you guys can think of that would get me to put a single byte into a file? Frankly, I don't care about the padding of the structures, it seems like a good way to keep everything straight. But I do need to be able to remove the padding when I go to write the char into my file.


EDIT: I just ran the "CHAR_BIT" code you wrote Salem, and it returned 16....... what the heck! Does this mean that all my chars are defined as 16 bit, i.e. 2 bytes, no matter what??? If that's the case, how would I even be able to have a 1 byte data type? I guess that would completely mess up my padding theory huh? Note: I am writing this for an embedded firmware application that uses a proprietary (though I believe gcc based) compiler.

Put #pragma pack(1) before you define the structures. It will get rid of padding.

Could you run this program please.

Code:

---

#include <stdio.h>

int main() {
    char str[7];
    printf("%d %d %d %d\n", sizeof(char), sizeof(short),
                            sizeof(int),  sizeof(str));
    return 0;
}

---

With a 16-bit char and 16-bit short, sizeof(struct glog) would be 44 without padding. So the compiler is doubling the size of that struct (from 88 bytes to 176 bytes).

What machine are you compiling for?

Could a macro somewhere be overriding his compiler's sizeof op?

Quote:

---

Originally Posted by oogabooga

Could you run this program please.

Code:

---

#include <stdio.h>

int main() {
    char str[7];
    printf("%d %d %d %d\n", sizeof(char), sizeof(short),
                            sizeof(int),  sizeof(str));
    return 0;
}

---

---

The result of this program yields:

1 1 1 7

:confused: So it would seem my sizeof function does not work in the same way as most of what I have read online seems to be suggesting.
I am compiling for a DSP Core, made by Ceva called Teaklite.

I am thinking I should try the #pragma pack(1) just to see what the file size results will be... if they are what I expected then its packing. If not, I have 16 bit char types and need to figure out something else.

In C, a char is the smallest data type that can have a unique address in memory. C only requires that CHAR_BIT be at least 8 (it could be more, as in your case).

This in itself isn't a problem (as far as the code is concerned), though it does mess about with your expectations.

All that really matters is that you can fwrite() out the structures and fread() them back in again, and be back where you started.

> Put #pragma pack(1) before you define the structures. It will get rid of padding.
But it won't change the underlying architecture of the machine, which has CHAR_BIT set to 16.
So char and short are both the same size, and there is no padding to squeeze out.

It looks like you may have 16-bit ints as well.
sizeof() actually doesn't return the number of bytes, but the number of chars.
One last piece of code. Please try this:

Code:

---

#include <stdio.h>
#include <limits.h>

int main() {
    printf("%d %d\n", INT_MIN, INT_MAX);
    return 0;
}

---

Quote:

---

Originally Posted by Salem

So char and short are both the same size, and there is no padding to squeeze out.

---

There may still be padding. struct glog should be 44 "chars" long. But instead he said sizeof gives 88. (Unless I've miscounted.)

Dunno - I read it as being "44" as reported by sizeof(), and "88" as reported by say "ls -l" on the file system, which may be counting bytes in the more traditional sense.

I just reread the initial post and you're right. That makes more sense!

If he really needs to save space, he could probably manually pack two characters into a single 16-bit char.

Quote:

---

Originally Posted by oogabooga

It looks like you may have 16-bit ints as well.
sizeof() actually doesn't return the number of bytes, but the number of chars.
One last piece of code. Please try this:

Code:

---

#include <stdio.h>
#include <limits.h>

int main() {
    printf("%d %d\n", INT_MIN, INT_MAX);
    return 0;
}

---

---

This code yields:

-32768 -1

So my Max int is -1? I may be a novice with C, and correct me if I'm wrong, but that seems like total nonsense.

Also, returning sizeof(struct glog) yields 44 like you say, yet its my actual file size that says 88 bytes (just a simple windows text file). However when I just count with my fingers and toes all the bytes that should be in the struct, I get 80.

But now that you have mentioned the 16 bit char being an architectural design, it makes sense. I have a total of 8 'char' types in my struct, so I was expecting those to only hold up 1 byte each. But it seems instead of the 80 I expect when I count, I get 88 in the final file size (i.e. the chars are double the size I expected as the theory is going).

Whether that makes sense or it doesn't, the problem still remains that I need to find a way to cut down on my data usage for this file I am writing. If its true that my char's are 16 bit types in my architecture, is there any way through C that I can possibly define a macro or something along those lines, to create a legitimate 8 bit type to replace my chars with?

Quote:

---

Originally Posted by oogabooga

If he really needs to save space, he could probably manually pack two characters into a single 16-bit char.

---

Could you elaborate on this idea a bit? I get it at a high level and I think if I come up with the right implementation, there should be no big problem with doing this (will be a lot more annoying than the original 'fwrite(glog)' I thought I was going to get away with). But how can something like cutting two 16 bit types in half and sticking them together work as far as code?

Quote:

---

Could you elaborate on this idea a bit?

---

Something like this:

Code:

---

typedef unsigned char uchar;
uchar a = 'b', b = 'b';
uchar packed;

// to pack
packed = a<<8 | b;

// to unpack
a = packed >> 8;
b = packed & 0xff;

---

Quote:

---

So my Max int is -1? I may be a novice with C, and correct me if I'm wrong, but that seems like total nonsense.

---

I agree.

Quote:

---

However when I just count with my fingers and toes all the bytes that should be in the struct, I get 80.

---

Considering that char, short, (AND int) are all 2 bytes on your target machine, it actually adds up to 44 16-bit chunks (which is what sizeof is reporting) or 88 bytes.

@Rath: Try using unsigned to print

Code:

---

printf("%u %u\n", INT_MIN, INT_MAX);

---

Quote:

---

Originally Posted by stahta01

@Rath: Try using unsigned to print

Code:

---

printf("%u %u\n", INT_MIN, INT_MAX);

---

---

This code yields:

32768 65535

so it does seem that yes, my integers are 16 bits in size whether they're signed or unsigned. Hm. I don't see why this architecture makes sense from a design standpoint, maybe it was designed to use assembly and get more low level? I would assume it would incorporate fairly traditional standard types though.

@oogabooga, thank you for that implementation. I have thought on it and it seems like if worst comes to worst and I absolutely cannot, for some reason, create a simple 1 byte data type easily on this architecture I can at the very least do a few quick shift operations before my write/read and not be eating up too many cycles on the system in the process. Just going to get a little hairy keeping track of things like 'the first 8 bits of my 16 bit char are for minutes of Time1, 2nd are Time2" etc for all these chars I have floating around :devil:

It doesn't look like you'll be able to create anything but 16-bit types on that architecture. It's a 16-bit digital signal processor, and that's that. The reason for the restriction is to simplify and thus speed up the hardware.

Perhaps the limit on your file size is unrealistic.

Quote:

---

Originally Posted by oogabooga

It doesn't look like you'll be able to create anything but 16-bit types on that architecture. It's a 16-bit digital signal processor, and that's that. The reason for the restriction is to simplify and thus speed up the hardware.

Perhaps the limit on your file size is unrealistic.

---

I believe so too, alas I am a mere intern and would prefer to get it to spec rather than ask for a change in requirements on my first project :tongue:

And I beleive you are rightabout the DSP core limitation. I have been scouring documentation for the past few hours to try and find a place where it lists in the spec that both chars and ints default to 16 bit types for this processor, and it says exactly that. Bitshifting solution it is. Thank you very much for your help.