”Converting” an array…

[guraknugen]

I know there are some array questions around, but I didn't find an answer to this particular one:

Is it possible to convert a one dimensional array to dimensional array by just some kind of ”type casting”? For example, I am trying to use an external library to read audio data (samples). For example, if the file is a stereo audio file, the samples end up like:

First  sample, left  channel → Audio[0]
First  sample, right channel → Audio[1]
Second sample, left  channel → Audio[2]
Second sample, right channel → Audio[3]
     ⁝

If I have a two dimensional array, like Audio[Sample][Channel], as I understand it, its data is arranged in the memory in a similar way, maybe something like:

Audio[0][0]
Audio[0][1]
Audio[1][0]
Audio[1][1]
   ⁝

This should make it fairly easy to just ”convert” between the two ”data arrengements”, shouldn't it?
Of course I know that I can just copy the data from one to the other in a loop, but if it was possible to just ”type cast” it, it would execute much faster, wouldn't it? Would be a nice shortcut…

[Tclausex]

I'm not even sure what a cast like that would look like. Have you tried?

You can always take advantage of the fact that multidimensional arrays are laid out linearly as a 1D array in memory.

#include <stdio.h>


#define ROWS 6
#define COLS 5


int main()
{
    int arr[ROWS*COLS] = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12};
    int i, j;
    int *p = arr;


    for( i = 0; i < ROWS; ++i ){
        for( j = 0; j < COLS; ++j )
            printf("%2d ", *(p + i*COLS + j));
        printf("\n");
    }


return 0;
}

[c99tutorial]

Of course I know that I can just copy the data from one to the other in a loop, but if it was possible to just ”type cast” it, it would execute much faster, wouldn't it? Would be a nice shortcut…

You can use memcpy for this if you know they are the same size in bytes and if you know they have the same memory layout.

memcpy(new_array, old_array, sizeof(new_array));

[AndiPersti]

A solution using type casting would be:

#include <stdio.h>
 
#define COLS 2 
// or whatever you need

int main(void)
{
    int arr1d[] = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12};
    int (*arr2d)[COLS] = (int (*)[COLS]) arr1d;
    int i, j;
    int a = 0;   // for indexing into arr1d
    int rows = sizeof(arr1d)/(sizeof(arr1d[0]) * COLS);

    puts("Nr\tarr2d\t    arr1d");
    for (i = 0; i < rows; ++i)
        for (j = 0; j < COLS; j++)
            printf("%2d   %p   %p\n", arr2d[i][j], (void *) &arr2d[i][j], (void *) &arr1d[a++]);
    return 0;
}

arr2d is a pointer to an array of COLS ints, i.e. it is a pointer to a row of your two dimensional array (with x rows and COLS columns).
The assignment on line 8 has the effect that it points to the beginning of your one dimensional array, thus you now treat the one dimensional array as an array with COLS columns and x rows (line 11 calculates the number of rows assuming that the one dimensional array has a multiple of COLS elements).

The rest is simple pointer arithmetic. arr2d[i] points to the ith row, thus arr2d[i+1] points to the ith + 1 row. Since arr2d is a pointer to an array with COLS ints, COLS elements of your one dimensional array are skipped.

The output will show you that you actually access the same element with both variables (arr1d and arr2d).

Bye, Andreas

[guraknugen]

I'm not even sure what a cast like that would look like. Have you tried?

Well, I have done some thinking, but I consider myself some kind of a beginner so I'm not surprised that I didn't come up with anything to try, except some lame experiments that failed, like declaring two arrays and assigning one to the other. The compilator, run in gnu99 mode, didn't like that, of course.

You can always take advantage of the fact that multidimensional arrays are laid out linearly as a 1D array in memory.

#include <stdio.h>


#define ROWS 6
#define COLS 5


int main()
{
    int arr[ROWS*COLS] = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12};
    int i, j;
    int *p = arr;


    for( i = 0; i < ROWS; ++i ){
        for( j = 0; j < COLS; ++j )
            printf("%2d ", *(p + i*COLS + j));
        printf("\n");
    }


return 0;
}

[guraknugen]

You can use memcpy for this if you know they are the same size in bytes and if you know they have the same memory layout.

memcpy(new_array, old_array, sizeof(new_array));

Looks like a shortcut in that sense that the there is not much code to type, but wouldn't this take as long to execute as a double loop (one inside the other) would?

[guraknugen]

This looks like what I was looking for! I'll take a closer look to it and do some tests later today. Thanks!

[c99tutorial]

Looks like a shortcut in that sense that the there is not much code to type, but wouldn't this take as long to execute as a double loop (one inside the other) would?

memcpy will copy each byte of the old_array to the new_array, so if old_array is very large and/or you do the memcpy very often in your algorithm, then it could impact performance. On the other hand, if the arrays are small then you can think of it as a constant-time operation.

[guraknugen]

memcpy will copy each byte of the old_array to the new_array, so if old_array is very large and/or you do the memcpy very often in your algorithm, then it could impact performance. On the other hand, if the arrays are small then you can think of it as a constant-time operation.

Well, I don't have my project completely defined yet, but I am pretty sure it's going to happen only once per session. Once I have my two dimension array, the program will work with it until the end. On the other hand, it's not small at all. It's audio samples, so it's size will depend on the audio file that it's going to process. A three minute stereo file at 44.1 kHz sampling rate means an array like AudioData[2][7938000]…

[c99tutorial]

A three minute stereo file at 44.1 kHz sampling rate means an array like AudioData[2][7938000]…

Ok, let's assume you use uint32_t for each element (4 bytes). Then your array is no larger than 4 * 2 * 8 = 64 million bytes. Typically computers can transfer billions of bytes in the memory per second. In other words, I don't think it will matter if you do this one time.

By the way, if you look at programs that handle uncompressed audio like Audacity you will see that they don't typically load the entire sample into memory at one time. They load chunks of it and keep almost all of it on disk. As your audio data becomes larger, this strategy will be more and more useful. You could theoretically load everything into dynamic memory and just let the operating system handle swapping it to disk for you when it gets too large, but this strategy usually leads to programs which are not responsive to user input.

[guraknugen]

Ok, let's assume you use uint32_t for each element (4 bytes). Then your array is no larger than 4 * 2 * 8 = 64 million bytes. Typically computers can transfer billions of bytes in the memory per second. In other words, I don't think it will matter if you do this one time.

Actually I am going to convert all the data to double (8 bytes) or long double (12 bytes), since I want to do some maths on them. I think long double will be overkill, so I guess double is what I'm going to use. The input file will in most cases be 24-bit FLACs, and I don't think I will support other formats, at least no lossy ones.

By the way, if you look at programs that handle uncompressed audio like Audacity you will see that they don't typically load the entire sample into memory at one time. They load chunks of it and keep almost all of it on disk. As your audio data becomes larger, this strategy will be more and more useful. You could theoretically load everything into dynamic memory and just let the operating system handle swapping it to disk for you when it gets too large, but this strategy usually leads to programs which are not responsive to user input.

You mean using buffers?
Since I'm not a very experienced programmer, I think that will be too complicated for me at the moment. The audio signal will be searched for certain things (peaks and stuff), and when something is found, there is going to be some processing around the signal. That means that if I find something at the very beginning of a buffer, I need to go back to last buffer and so on. So I think it will be very much easier if I work with the whole file loaded in memory, giving me freedom to go back and forth the way I need, I am mainly going to process 2-5 minute music stereo files, no big concerts with 5.1 sound or anything like that, so I don't think memory usage will be a big issue.

[guraknugen]

Done some tests now and yes, that was exactly what I was looking for.
I hope it works with dynamic arrays too. Well, I see no reason why it wouldn't.

The only thing I could hope for now is to find a way to transpose the array without copying things around, but I guess that is not possible…

[c99tutorial]

The only thing I could hope for now is to find a way to transpose the array without copying things around, but I guess that is not possible…

At least with the routines in Lapack, there is a parameter that lets you choose the format of your input array. In other words, they let you use it in whichever way it is in without having to transpose it to a specific format. So if you have that option, transposing might not be needed; you just tell the routine how it is and it carries out its job as is. However, unless you have an algorithm which is continuously transposing a large array, I don't see how transposing would lead to a performance problem.

[guraknugen]

Seems like I need to work with buffers after all.
Did a simple test with a 2D-array of Doubles:

double a[INDEXMAX][2];

When INDEXMAX was set to a little more than 5×10⁵, there was a segmentation fault. There is no exact value where this happens. Sometimes one value results in a segmentation fault, sometimes not, but the limit seems to be somewhere around 5.0×10⁵ ↔ 5.5×10⁵.

A double seems to be 64 bits on my machine. 8 bytes, that is, so 5×10⁵ ”rows” and 2 ”columns” should mean 8 MB, which is not that much. Seems like I have a lot more than that free.

$ free -m
             total       used       free     shared    buffers     cached
Mem:          2013       1881        131          0         98        806
-/+ buffers/cache:        976       1036
Swap:         1905         51       1854
$

Is there a way around this…?

Oh, and my simple test code looks like this:

#include <stdlib.h>

#define INDEXMAX 525000

int main(int argc, char *argv[])
{
    double a[INDEXMAX][2];
        
     for (size_t Sample=0; Sample<INDEXMAX; Sample++)
        for (size_t Ch=0; Ch<2; Ch++)
            a[Sample][Ch]=(double)Sample;

    return 0;
}

Command used to compile:

gcc -std=gnu99 -o Test Test.c

[AndiPersti]

int main(int argc, char *argv[])
{
    double a[INDEXMAX][2];

The array is stored on the stack, thus the command to check how much stack memory you are allowed to use is

ulimit -s

Is there a way around this…?

Create your array on the heap (using malloc())

Bye, Andreas