Thread: Reading 2 csv files and combining their content into an output

You shouldn't be using fread or strtok. They are C functions.
This can be done easily in C++.
One way I can think of is to use std::getline to read a full line. Then, you can construct a string stream from the string you read, and then construct an istream_iterator from that. You can then use std::copy to copy the items out of the line into a vector to hold your items using a back_insert operator (you can use std::back_inserter to make one) as the third parameter, the output iterator to std::copy.
Then write everything back into an output file from the strings in your vector, putting whitespace and newlines as you desire.

I could make an example, but I also want this to be a learning experience for you to read the documentation and get familiar with these features in C++.

one way you can help yourself out is by not writing C+ code

what exactly is the problem you are having? you are still mixing c with c++, Elysia showed you routes to the docs for many useful c++ features, you should read those, just browse the pages. Also your original theme of make a .csv into a .txt - is completely redundant - the data is simply data.

Do not mix C-style I/O with C++ I/O streams unless you have a good reason to do so. This:

Code:

FILE* fout = fopen(outfilename.c_str(), "w");
// ...
fprintf(fout, "%d\t", row);

should be:

Code:

std::ofstream fout(outfilename.c_str());
// ...
fout << row << '\t';

You should not use eof() to control a loop like that. I would expect something like:

Code:

    while (getline(ss, ftid, ':'))
    {
        if (!ftid.empty())
        {
            cout << ftid << "!";
        }

        if (getline(ss, ftvalue, ','))
        {
            if (!ftvalue.empty())
            {
                cout << ftvalue << ",";
            }
        }
        else
        {
            break;
        }
    }

1) Does it make sense to use C++ on such huge files? I was told that Python and MATLAB would take much longer.

I'm not completely sure I understand your file format. But consider the following python code:

Code:

def process_line(line):
    return '!'.join(line.split(':')).strip('\n');

Which can be called on all the lines of both files in something like:

Code:

for line, line2 in zip(file1, file2):

You can print output as you go, but this assumes that file 1 and file 2 have the same number of lines. All considered, I think this is something python can do efficiently.

Originally Posted by laserlight

Do not mix C-style I/O with C++ I/O streams unless you have a good reason to do so. This:

Code:

FILE* fout = fopen(outfilename.c_str(), "w");
// ...
fprintf(fout, "%d\t", row);

should be:

Code:

std::ofstream fout(outfilename.c_str());
// ...
fout << row << '\t';

Thanks! I've made the corresponding changes. I think I read on stackoverflow somewhere that the C variant is much faster. But if it causes compatibility issues, I'd be happy to omit it.

Do not assume C is faster in any way or form until you can prove it. Keep yourself in the domain of C++. Worry about possible optimizations only if you need it.

Originally Posted by Elysia

One way I can think of is to use std::getline to read a full line. Then, you can construct a string stream from the string you read, and then construct an istream_iterator from that. You can then use std::copy to copy the items out of the line into a vector to hold your items using a back_insert operator (you can use std::back_inserter to make one) as the third parameter, the output iterator to std::copy.
Then write everything back into an output file from the strings in your vector, putting whitespace and newlines as you desire.

Alright - I found a way to integrate stringstream, istream_operator and copy. Here's the current version of the code:

Code:

#include<iostream>
#include<sstream>
#include<fstream>
#include<iterator>
using namespace std;

template <class T>

class csv_istream_iterator: public iterator<input_iterator_tag, T>
{
istream * _input;
 char _delim;
string _value;
public:

 csv_istream_iterator( char delim = ',' ): _input( 0 ), _delim( delim ) {}

 csv_istream_iterator( istream & in, char delim = ',' ): _input( &in ), _delim( delim ) { ++*this; }

const T operator *() const {
        istringstream ss( _value );
       T value;
       ss >> value;
       return value;
    }
 istream & operator ++() {

       if( !( getline( *_input, _value, _delim ) ) )
        {
            _input = 0;
       }
       return *_input;
 }
booloperator !=( const csv_istream_iterator & rhs ) const {
   return _input != rhs._input;
    }
};
template <>
const string csv_istream_iterator<string>::operator *() const {

 return _value;
}
int main( int argc, char * args[] )
{
      ifstream ft( "input1.csv" );                                // declare ft stream


 
 
// 1) Get number of data points

string datapoints, dummy;
 getline(ft, datapoints, ',');               // read very first value: # data points

 cout << "datapoints: " << string(datapoints) << endl;// display number of datapoints

double count = atof(datapoints.c_str());

 getline(ft, dummy, '\n');                       // read and drop second value

 
　
// 2) Start writing into output file

 string outfilename = "out.txt";
 
 
ofstream fout(outfilename.c_str());
for (int row = 1; row <= count; row++)
 {
// enumerator

 fout << row << '\t';

   if( ft )
    {
          copy( csv_istream_iterator<string>( ft ),
                          csv_istream_iterator<string>(),
                          ostream_iterator<string>( cout, "," ) );
    }
}
　
fout.close();
return 0;
　
}
 

The output is as follows:

1 1:4,2:5,3:7,4:4,5:7
,
2 1:4,2:3,3:6,4:4,5:6
,
3 1:4,2:4,3:7,4:4,5:7
,
4 1:2,3:5,4:7,5:4,8:3
,
5 2:3,4:4,5:7,,
,
6 1:4,3:7,4:6,,
,
7 ,,,,
,
8 2:4,8:7,,,
,
9 3:1,4:4,6:7,,
,
10 1:5,2:8,,,
,
11 1:1,2:5,,,
,
12 3:4,4:5,6:8,,

However, there are a few things which are disturbing and must be mended: 1) All these extra commas. If there's no additional character, the commas must go away. 2) I have not made use of back_inserter yet. Where does that come in?

You didn't (and don't) have to write your own iterator. You can use ones already existing in the standard library. The one you have written just mimics istream_iterator anyway.
You don't need to manually close files. They close automatically when they go out of scope. Learn to rely on this behaviour. It is a cornerstone in C++ called RAII.
back_inserter creates a back_inserter iterator which you can use to push back items into a container. For example:

Code:

	std::ifstream in("myfile.txt");
	std::vector<std::string> Lines;
	std::copy(std::istream_iterator<std::string>(in), std::istream_iterator<std::string>(), std::back_inserter(Lines));
	std::copy(Lines.begin(), Lines.end(), std::ostream_iterator<std::string>(std::cout, "\n"));

A vector does not automatically contain storage, so you can't just write elements to it directly. So you can use a back inserter, which calls container.push_back(elem) for every element.