Originally Posted by
matsp
How are you reading the file? How long does it take? My guess would be that reading the file is much less of a problem than your METHOD of reading the file (or some other processing that is part of reading the file). It shouldn't take very long to read 300MB of data is you do it in relatively large blocks.
--
Mats
it is a very simple csv file with three fields:
time, double, double
but I have 11,737,015 lines. "wc" it on my machine takes more than 1 minutes. The reader uses getline() and the following subroutine to decompose it into fields.
Code:
void onecsvline(string linebuf, time_t& t, double& bid, double& ask) {
string partbuf;
istringstream s(linebuf);//turn this line into
std::getline(s, partbuf, ',');
istringstream ss(partbuf);
ss >> t;
std::getline(s, partbuf, ',');
istringstream sss(partbuf);
sss >> bid;
std::getline(s, partbuf, ',');
istringstream ssss(partbuf);
ssss >> ask;
}
so for file, I will call this routine 11,737,015 times. I don't quite get what you mean read in large blocks. Maybe that's exactly what I shall do to improve efficiency. Please kindly show me a example. Thanks.