Thread: Advice needed, how to encode large arrays

Hi, i need some advice,

I have an array of 5*10e10 integer values. Values span between 1 and 50000 they do not appear in nay particular order. Also i have an array of characters of the 3*10e9 size. I would like to store these two arrays on disc. if i store them as binary then since the integer array is encoded as long int i consume more memory then just writing the thing down as a regular txt file. In any case both the text and the binary are too large to be transported over the net. so my question is can anyone give my an advice on how to encode this particular case, the binary and character array so that i consume as less space on disc as possible.

thank you

baxy

P.S.

How to save an array of type long to disc as a type short

just an example

Code:

long array;

fwrite(&array,sizeof(short),NUMBER_OF_VALUES,fileIn)

Is something like this possible (i don't see how but i am not the best programmer out there)

Originally Posted by baxy

P.S.

How to save an array of type long to disc as a type short

just an example

Code:

long array;

fwrite(&array,sizeof(short),NUMBER_OF_VALUES,fileIn)

Is something like this possible (i don't see how but i am not the best programmer out there)

Yes, it's possible, but not that way. First, if short was a 16-bit type, you would need an unsigned short to hold values over 32,767. Second, short is guaranteed by the standard to be at least 16 bits, but it could be a larger type (e.g. 32-bits). I would recommend using fixed-size type, like uint16_t (make sure to #include <stdint.h>).

You would have to cast the elements of the original array to the type you want as you write it. That either means creating an array of uint16_t of a reasonable size (say 1024 or so) and filling that one-by-one from your giant array, casting each time:

Code:

for offset = 0 to 5e10
    for i from 0 to 1023
        small_array[i] = (uint16_t) big_array[offset + i)
    fwrite(small_array, sizeof(small_array), 1024, fileIn)
    offset += 1024

On a 64-bit machine, this could save you 75% alone in the binary file format, before compression.

> In any case both the text and the binary are too large to be transported over the net.
How many copies, over what distance, in what timeframe?
throughput and latency

> so my question is can anyone give my an advice on how to encode this particular case
In order to answer this, we would need to know how you arrived at so much data to begin with.
Are there any peculiar or special properties of the data which might lead to novel compression approaches.