Thread: Newbie Question - store and manipulate large amounts of data in C

Hi.

I am trying to create a C program that will take a binary file containing over 7000000 different values and store them so that i can process the data. I already have a Matlab program that does this but my aim is to convert this to C to improve the speed of the data processing.

so far I have this that I got from looking on websites etc.

Code:

#include <conio.h>
#include <iostream>

using namespace std;

int main(int argc, char *argv[])
{
    
FILE *f;
unsigned char buffer[463488];
int n;

f = fopen("C:\\Dev-Cpp\\Project1\\test.fti", "rb");
if (f)
{
    n = fread(buffer, 463488, 1, f);
}

else
{
    // error opening file
}
 
    getch();
    return EXIT_SUCCESS;

}

I am only a beginner programmer so please excuse my limited knowledge but I want to learn.

Sorry, I don't see a question in your post - what are you after? The best strategy to do this? A review of your code? The more specific you can be, the more we can help you!

sorry, perhaps its best written as 2 questions, 1 more specific and 1 more general

1. Specifically I would like to know what is the best container (I think this is the correct term) to store and manipulate large amounts of data

2. Generally some input as to a good strategy to do this from experienced programmers would be helpful

I posted my code mainly to show that I have tried to do it myself before coming here

Originally Posted by predator887

1. Specifically I would like to know what is the best container (I think this is the correct term) to store and manipulate large amounts of data

2. Generally some input as to a good strategy to do this from experienced programmers would be helpful

I think both of these questions are best answered with the advice "do the simplest thing that could possibly work" - I can't remember who said that, but it's good advice.

The best container is the simplest one that will do what you want. It sounds like you want a static array, so just use that. If your data is a fixed size of 463488 bytes then use a 463488-byte char array, as you seem to have done (probably best to place it outside the main function though - then it goes on the heap and not on the stack). If it's not fixed size, try malloc()-ing it. If those don't work because of memory restrictions (or whatever), you'll have to start a more complicated scheme of treating the data as a stream or as smaller blocks of data.

As for input strategy - again, try the simplest thing, which is just reading it all into memory in one go as you're doing now. If you start getting memory allocation errors, look at a more complex one.

Originally Posted by predator887

Hi.

I am trying to create a C program that will take a binary file containing over 7000000 different values and store them so that i can process the data. I already have a Matlab program that does this but my aim is to convert this to C to improve the speed of the data processing.

I am only a beginner programmer so please excuse my limited knowledge but I want to learn.

First... what are the "values" in this file? Are they integers, floats, text, an image...???
Second... please post an excerpt from the beginning of the file, half a dozen lines will do.

The way you load the file will depend largely upon what is in the file.
For example:

Code:

01203434567948194869332846666292840689

requres a totally different strategy than

Code:

01 22 23434 607 00 954 22 11

So in this case, given your question stated and rephrased...the content of the file is actually more important than your first attempt at code (which was pretty good. btw)