efficient binary file read

Hello

I would like to read a binary file contain short (16 bit) ints into an array. However, I would like that
array to be of type float (32 bit). Presently I am simply reading the file into an int array using fread
and then doing a loop to assign the short ints to another array that is of type float. The problem is that
the arrays can be rather large and I would like to be memory efficient by avoiding allocating memory
for the both an int and a float array. Is there a straighforward way to read this file directly and quickly
into the float array?

Thanks

1) Have one short int.
2) Have one float array.
3) Have a loop where a short int is read, converted to a float and placed inside the float array.

In fact, 3 should be a simple assignment. Just store it directly into the float array.

You cannot read a 16-bit int directly as a float. You can't even read a 32-bit int directly as a 32-bit float. Floats in binary are usually represented in some very different fashion than ints.

Thanks for the replies

But these files are big. Reading an int at a time and making the assignment to the elements of a float array will take a lot more time than using fread to read the big block of data. Wont it?

Are you allowed to rewrite the file? Cause if reading it that way is a problem, then you should store floating point numbers in the file as well. Then you can just read them into memory at once.

If not, then what MacGyver suggested is as good as it's going to get it seems.

Quote:

---

Originally Posted by Marv

The problem is that the arrays can be rather large and I would like to be memory efficient by avoiding allocating memory for the both an int and a float array.

---

What you could do is read a block of ints at a time. That way your int array doesn't have to be so large.

#define SIZE ( BUFSIZ / sizeof short int )

short int inputBuffer[ SIZE ];

while ( (n=fread(inputBuffer, sizeof short int, SIZE, fp )) > 0 )

Now convert 'n' shorts into floats

Those have all been good suggestions. Unfortunately I do not have control (or little control) over how the file is initially written.

I think I will follow Swoopy and Salem's idea about reading a good size block of int with fread and then doing an assignment to the large float array (repeat and rinse). That is unless there is some
other clever way one of you might introduce me to.

What they suggested is the correct way of doing it unless you are under such severe memory restrictions that you can't afford to make the other array (which seems unlikely).

Thanks for the help everyone.

Quote:

---

Originally Posted by swoopy

What you could do is read a block of ints at a time. That way your int array doesn't have to be so large.

---

Wow, that's almost like... what stdio does!

Quote:

---

Originally Posted by MacGyver

You cannot read a 16-bit int directly as a float. You can't even read a 32-bit int directly as a 32-bit float. Floats in binary are usually represented in some very different fashion than ints.

---

Hi, I'd like to ask that how are the floats presented differently? I've experimented a little, and as a result the binary file made looks like this:
(Opened with binary editor)

1F 85 AB 3F AE 47 2D 42 CD 0C A6 43 D9 2A 13 45 ...?.G-B...C.*.E

That was the first line, I want to understand how this works. I kinda understood what happens with ints, but this is somewhat peculiar.

You might want to take a look at this:
http://en.wikipedia.org/wiki/IEEE_75...ecision_32_bit

There is obviously no way to know whether those numbers are integers, floats or some other data type(s). It would be possible, if we know that it MAY be float to make some searches for certain patterns and say "no, this is unlikely to be a float" or "it's likely that it is a sequence of float", but you can't know for certain.

I don't see anything indicating your numbers CAN'T be float - but they could just as well be anything else.

The difference between float and integer types is that the floating point numbers are "split into portions", a standard 32-bit float has one 1 bit of sign (bit 31) followed by 8 bits of exponent, and the remaining 23 bits reprsent the mantissa.

The number is essentially s*1.m*2^e (s = sign = 1 or -1).

More info here:
http://en.wikipedia.org/wiki/IEEE_754

--
Mats

Quote:

---

Originally Posted by Marv

Thanks for the replies

But these files are big. Reading an int at a time and making the assignment to the elements of a float array will take a lot more time than using fread to read the big block of data. Wont it?

---

Nope.

EDIT: Yikes. I'm contributing to the resurrection of a defunct thread. Let's just let this one die, eh?