Arrays with NULL bytes

[Yawney]

Hello,

I'm working with arrays that might have NULL bytes in them and I'm wondering how to determine the length of the array or store it somewhere with the array (strlen() won't help because of the NULL, right?).

I've found advice like store the length of the array in the first byte of the array, but since sizeof(size_t) is 8 should I leave the first 8 bytes for the length?

Would it be better do define my own structure which would store the array and its length? What's the usual way these things are handled in practice?

Thank you.

[Salem]

Sure, I would suggest

struct rec {
    unsigned char *data;
    size_t allocatedSpace;
    size_t usedSpace;
};

[rcgldr]

If the array of characters is always 255 or less in size, you can store the size in the first character. This is called a p-string (p for pascal).

[Malcolm McLean]

You've got two options.

One is to define a two member structure, with a char * and a length member. The other is to pass about the length in a separate variable. Which you choose depends on whether you need lots of these objects (go for the structure), and if the code is small and simple enough to do without a structure.
If the reason you need nuls is that the array contains several strings, it's quite common to use two nuls to represent the terminating state.
You can store length in the buffer, and in fact you'll do this for serialisation, but to do it for passing data around internally is a hack, and best avoided, unless, as with the pstrings, you're integrating with another language.

[rcgldr]

You can store length in the buffer, and in fact you'll do this for serialisation, but to do it for passing data around internally is a hack, and best avoided, unless, as with the pstrings, you're integrating with another language.

I've used p-strings for text file sorting, replacing the newline characters with length counts. I read in the file at buffer +8, to leave room for the first count of the first record (at buffer +7). I'm also using an array of pointers to those strings, and it's the array of pointers that gets sorted. There's a bit of a speed advantage on the compares since I can use memcmp which will use the Intel X86 string compare instruction (repe cmpsb).

[Malcolm McLean]

I've used p-strings for text file sorting, replacing the newline characters with length counts. I read in the file at buffer +8, to leave room for the first count of the first record (at buffer +7). I'm also using an array of pointers to those strings, and it's the array of pointers that gets sorted. There's a bit of a speed advantage on the compares since I can use memcmp which will use the Intel X86 string compare instruction (repe cmpsb).

The other advantage over the structure method is that you only need one allocation. There are reasons for doing it. But I think it's a bit of a hack. Alignment is a big problem for the mem and str functions. If you can convince the compiler that you're aligned and a multiple of 4 or 8 bytes, you can get a huge speed up.

[rcgldr]

Alignment is a big problem for the mem and str functions.

That's somewhat masked by the way the outer cache works. A cache line is going to load or store on ram boundaries, and the boundary depends if the cpu is dual channel, tripple channel, or quad channel. If the process is somewhat sequential, such as a bottom up merge sort, then most of the cache loads end up being utilized.

[iMalc]

The other advantage over the structure method is that you only need one allocation. There are reasons for doing it. But I think it's a bit of a hack. Alignment is a big problem for the mem and str functions. If you can convince the compiler that you're aligned and a multiple of 4 or 8 bytes, you can get a huge speed up.

If you combine the allocated size and used size, i.e. the string doesn't need to grow, then it can be done as one allocation with the string tacked onto the end of the struct.

Then one could go a bit fuerther and basically reimplement the BSTR type window's uses.