I have a program which exchanges streams of data with a server across the Internet. It runs on Linux. Different users run it on a wide variety of flavors of Linux. The data streams are compressed. The application had been using bzip compression, but I am now changing it to use gzip. The application is working well with gzip now, but there is one thing about it which is less than satisfactory. These packets of data are stored in buffer variables and I need to make the buffers big enough to hold them. That means that when I compress data to send, I need to have an advance estimate of the compressed size to dynamically allocate a buffer for it, and when I uncompress data received, I need to have an estimate of the uncompressed size to allocate a buffer to store it. If I cannot do these things, then I will always have to use the largest size that any transmission might ever be, and that will be a big waste of memory.
I wish to emphasize that this data never gets stored into files. It is only stored in memory variables. I can estimate the compressed size before I compress data because the zlib library furnishes a method called CompressBound(), which takes the uncompressed size and returns an upper bound for what the compressed size might be. I use this size to allocate the buffer. However, for decompressing received packets, I can find no way to figure out in advance what the uncompressed size will be. I have seen vague hints that I could parse the gzip header, but nothing approaching an instruction as to how to do that. Does anyone know how to estimate the uncompressed size of a gzip compressed buffer???
Once again, this data is not in files, only in memory. Furthermore, this application rapidly exchanges large volumes of information with the server, and I cannot use the time or disk I/O to make some kind of temporary file for each packet. Nor can I shell out to give an operating system command like gunzip or something each time I receive a packet. This has to be handled with C++ calls alone.
Thanks in advance to any kind soul who can shed some illumination on this issue.