Handling Large Amounts of Data

Hey Guys,

I was hoping to get some peoples opinion on something. I am not new to C++/C programming but by no means am I an expert. I am a physics student who uses it in his research. I have an extremely large Data set which I want to visualize. I have the graphics portion of the code done, I just need a way to handle reading all the data into the program. The data is organized into 100 different files (each file essentially represents a frame); within each file there is 1000 rows and 11 columns. Reading this data in, normalizing it and displaying it file by file is not working (Maybe I am just implementing it inefficently) and reading it all in at the begining seems counter productive and slow. Does anyone know of an efficent way to handle this much information within a program?

Depends on many factors, kas2002. You will have to be a little more precise on how the data is being displayed and what type of data are we talking about... strings, numbers, ... and probably also on what type of operations you are performing on that data.

Usually the best way nonetheless is to try and follow some data treatment and displaying scheme that allows you to use lazy evaluation techniques. But do provide more info.

Could you specify the things that the program has to do and the code that you have written for doing it.

> The data is organized into 100 different files
Well the simple test is to do

Code:

---

while ( f.getline(string) ) {
}

---

inside the loop which opens and closes all 100 files.

This will give you the benchmark for simply reading all the data.

At least you will have something to compare against if the next step doubles the time.

The data files are just numbers give as follows:

int int int float float float float float float float float float float float

First 3 integers are just coordinates of a small area of the system (called a bin). Within the simulation program (which is seperate from the visualization) we do specific types of averaging in these bins, and those floats in the data file are that output. What I want to do is read in the x,y,z coords, plot a box at that point with it's transluceny dependant on one of the floats.

As I said above each file represents a frame, or in otherwords a timestep in the simulation. What I need is an efficent way to read in this data and display it, and move on to the next timestep without lag or absolutly clogging memory. i.e. some sort of data structure which can either efficiently access the data on the fly OR store all of the data within the program in a concise easy to access manner. I have tried two dimensional vectors but that was memory hogging/counter accesible.

When I only needed to display one timestep at a time this is how I went about it (I use the Irrlicht game engine for rendering purposes).

1) I read in the file storing all the data in 11 different vectors.
2) Using Irrlicht and glut primitives I plot the boxes and render the scene.

> As I said above each file represents a frame,
So if you get say
2 3 4
in two different frame files, which obviously update the same part of the visualisation, what do you expect to happen?

From what you've said, you need
- the bins from summing all the previous frames
- the current row of data from the current file.

I don't think you need to read in the whole file in order to update the visualisation.

Quote:

---

Originally Posted by Salem

> As I said above each file represents a frame,
So if you get say
2 3 4
in two different frame files, which obviously update the same part of the visualisation, what do you expect to happen?

---

I expect the color of each bin to change according to the difference in values. i.e. 2 3 4 represents the same bin in each frame.....however one of the 8 corresponding floats will change with each timestep. It is that value which I am attempting to correlate to spatial properties.

and I do need to read in the whole file because each file defines the entire system for one timestep and I would like to see the entire system.

> and I do need to read in the whole file
I said, you only need to store the current row, not the entire file.

Code:

---

while ( f.getline() ) {
  // process line
}
// display final result

---

Vs.

Code:

---

while ( f.getline() ) {
  // store line in a vector
}
for ( each vector element ) {
  // process
}
// display final result

---

If your files are text you can also improve performance by writing a preprocessing step that will convert your text files into something like

Code:

---

struct element {
    int x,y,z;
    float value;
};

---

and then use ifstream.read()'s. if each line in the text file is exactly the same number of characters you can use some seekg()'s to skip to the next line faster.

SQLite seems appropriate for this...