Thread: what do you do to avoid loading large dataset again and again

hey, there,

At beginning of my code , I first read a large data file and then do analysis of the dataset. I never change the dataset. The datafile is large, >=300M. I frequently change the analysis part and run test. Everytime I need to load the large datafile again, which has been unbearable.

In Matlab, I can read the datafile at beginning of the session, and then do all kinds of analysis without need to reload the data. So I am thinking of establishing a separate process which reads the datafile, and stay there all day responding to data request from my analysis routine. but I am not sure how to do it in C++. or maybe there is a better method.

please kindly share with me how you cope with this situation. Thanks.

Michael

Originally Posted by matsp

It shouldn't take very long to read 300MB of data is you do it in relatively large blocks. Mats

Some times this is not possible so easily, for example if you recreate object sets using some serialization library.

Once I used boost::interprocess to hold the data set inside a designated process. The real reason though was to allow data processing by several processes without the need to load the data into every single process. But of course your problem was solved too. Depending an your kind of data this method can be more or less tricky. For tree structures for example you have to take care converting pointers and precisely manage memory layout inside your processes. But boost also has a bit support for that

Originally Posted by matsp

How are you reading the file? How long does it take? My guess would be that reading the file is much less of a problem than your METHOD of reading the file (or some other processing that is part of reading the file). It shouldn't take very long to read 300MB of data is you do it in relatively large blocks.

--
Mats

it is a very simple csv file with three fields:
time, double, double

but I have 11,737,015 lines. "wc" it on my machine takes more than 1 minutes. The reader uses getline() and the following subroutine to decompose it into fields.

Code:

void onecsvline(string linebuf, time_t& t, double& bid, double& ask) {
	string partbuf;
	istringstream s(linebuf);//turn this line into
	std::getline(s, partbuf, ',');
	istringstream ss(partbuf);
	ss >> t;
	std::getline(s, partbuf, ',');
	istringstream sss(partbuf);
	sss >> bid;
	std::getline(s, partbuf, ',');
	istringstream ssss(partbuf);
	ssss >> ask;
}

so for file, I will call this routine 11,737,015 times. I don't quite get what you mean read in large blocks. Maybe that's exactly what I shall do to improve efficiency. Please kindly show me a example. Thanks.

Originally Posted by anon

Run the tests with a smaller data set? (I assume you are testing changes to the program and it's not normal usage of the program to rewrite and recompile it...)

You are right. I used to test it on small set in the last a few weeks. At this time point, I need to test it over large set more often. And it becomes a pain on neck.

1 minute is not too bad, and I am not going to spend hours to squeeze a few seconds out of it. The problem is that I need to call the function many times (>=40?) during the day. And it is not fun to wait 1 minute or 50 seconds everytime. Maybe the Boost::interprocess mentioned early is a solution for me. Thanks guys!

thanks, reading binary file is much faster!

Originally Posted by matsp

One solution to "preload" the data is to use a memory mapped file (mmap in Linux/Unix, MapViewOfFile Function (Windows) in Windows).

--
Mats

Is it somehow possible to create in std fstream object to a memory mapped file? This would solve the problem with the serialization library.