Reading 2 csv files and combining their content into an output

**in_ship** · 05-07-2013

Originally Posted by Elysia

You didn't (and don't) have to write your own iterator. You can use ones already existing in the standard library. The one you have written just mimics istream_iterator anyway.
You don't need to manually close files. They close automatically when they go out of scope. Learn to rely on this behaviour. It is a cornerstone in C++ called RAII.
back_inserter creates a back_inserter iterator which you can use to push back items into a container. For example:

Code:

    std::ifstream in("myfile.txt");
    std::vector<std::string> Lines;
    std::copy(std::istream_iterator<std::string>(in), std::istream_iterator<std::string>(), std::back_inserter(Lines));
    std::copy(Lines.begin(), Lines.end(), std::ostream_iterator<std::string>(std::cout, "\n"));

A vector does not automatically contain storage, so you can't just write elements to it directly. So you can use a back inserter, which calls container.push_back(elem) for every element.

Hey Elysia! Thanks again for the hints (will remove the close commands and the template). However, I am wondering whether it is really necessary to have two copy statements. Why not copy things into the output file right away? Plus, the main issue I am dealing with is still the commas and formatting...

**in_ship** · 05-07-2013

So - I have adapted the code a bit: 1) Managed to resolve the issue with the additional newline (by subtracting last character from string) and 2) Added back_inserter functionality.

Code:

#include <iostream>
#include <sstream>
#include <fstream>
#include <iterator>
#include <vector>

　
using namespace std;


　
int main( int argc, char * args[] )
{

    ifstream ft( "input1.csv" );                    // declare ft stream
    string ftline;

    // 1) Get number of data points

    string datapoints, dummy;

    getline(ft, datapoints, ',');                     // read very first value: # data points
    double count = atof(datapoints.c_str());
getline(ft, dummy, '\n');                         // read and drop second value


 // 2) Start writing into output file
    string outfilename = "output.txt";
    ofstream fout(outfilename.c_str());

    
    for (int row = 1; row <= count; row++)
    {

        // enumerator
        fout << row << '\t';

        
        // format features
        getline(ft, ftline, '\n');

        ftline = ftline.substr(0, ftline.size()-1);
        stringstream ss(ftline);

        vector<string> Line;
        copy(istream_iterator<string>(ss), istream_iterator<string>(), back_inserter( Line ));
        copy( Line.begin(), Line.end(), ostream_iterator<string>( fout, "\n" ) );

    }


return 0;
    
 }

Output file output.txt:

1 1:4,2:5,3:7,4:4,5:7
2 1:4,2:3,3:6,4:4,5:6
3 1:4,2:4,3:7,4:4,5:7
4 1:2,3:5,4:7,5:4,8:3
5 2:3,4:4,5:7,,
6 1:4,3:7,4:6,,
7 ,,,,
8 2:4,8:7,,,
9 3:1,4:4,6:7,,
10 1:5,2:8,,,
11 1:1,2:5,,,
12 3:4,4:5,6:8,

　
Now, the two remaining issues are still: 1) Do away with those extra commas and 2) Replace colon ':' with exclamation mark '!'.
　

　

**Elysia** · 05-07-2013

So the general idea here is that if you have a file such as:

token:token:token\n
token:token:token\n

Then you have two options:
Either you read token by token, then write it out token by token separated with a token separator you choose, or you read one row and replace every token separator in the read row with whatever other token separator you want, then write the entire line.

Note that getline read up to and including your stop reading at token as specified by the third argument. getline does not, however, put that token into the read data. So if you read a line, it will not store the newline into the read string.
If you read token by token, then you have to consider that a token might also be separated by a row (like in the above). In that case, it might be good to read a full row, then parse the tokens out of it.

**whiteflags** · 05-07-2013

Remember the key to actually merging the files is to process them both at once.

Now, the two remaining issues are still: 1) Do away with those extra commas and 2) Replace colon ':' with exclamation mark '!'.

You could also throw a regular expression at the problem. C++11 added a regex library to the standard. If you're using MSVC++ 2010 you have it. I managed to use this with lots of success:

Code:

#include <regex>
#include <string>
#include <sstream>
using namespace std;

string process_line(string line)
{
    regex tokenRegex("\\d+:\\d+,|\\d+:\\d+$|\\d+:\\d+\.\\d+,|\\d+:\\d+\.\\d+$");
    sregex_iterator tokensBegin(line.begin(), line.end(), tokenRegex);
    sregex_iterator tokensEnd;

    ostringstream builder;
    for ( sregex_iterator i = tokensBegin; i != tokensEnd; i++ )
    {
        string match = (*i).str();

        string::size_type where = match.find(':');
        match[where] = '!';
        builder << match;
    }

	
    string result = builder.str();
    if (!result.empty() && result.back() == ',')
        result.pop_back();

    return result;
}

Running it on both files, I managed to produce:

Code:

1	1!4,2!5,3!7,4!4,5!7	5!1.0
2	1!4,2!3,3!6,4!4,5!6	1!1.0,4!1.0
3	1!4,2!4,3!7,4!4,5!7	2!1.0,3!1.0
4	1!2,3!5,4!7,5!4,8!3	3!1.0,8!1.0
5	2!3,4!4,5!7	4!1.0
6	1!4,3!7,4!6	5!1.0
7		
8	2!4,8!7	2!1.0
9	3!1,4!4,6!7	3!1.0
10	1!5,2!8	4!1.0,7!1.0
11	1!1,2!5	3!1.0,4!1.0,5!1.0
12	3!4,4!5,6!8	1!1.0,5!1.0

Python also does regex and has for a long time...

**in_ship** · 05-09-2013

I think what I was trying to do initially was a total overkill. This piece of code got the work done in less than 10min.

Code:

/*
 * FileParser.cpp
 *
 *  Created on: May 5, 2013
 *      Author: in_ship
 */


#include <iostream>
#include <sstream>
#include <fstream>
#include <iterator>
#include <vector>
#include <algorithm>

using namespace std;



int main( int argc, char * args[] )
{

    ifstream ft("./input1.csv");        // declare ft stream
    ifstream lbl("./intput2.csv");     // declare lbl stream


    // 1) Get number of data points
    string datapoints, dummy;
    getline(ft, datapoints, ' ');                     // read very first value: # data points
    double count = atof(datapoints.c_str());
    getline(ft, dummy, '\n');                         // read and drop second value
    getline(lbl, dummy, '\n');                         // read and drop whole line


    // 2) Start writing into output file
    string outfilename = ".output.txt";
    ofstream fout(outfilename.c_str());

    for (int row = 1; row <= count; row++)
    {

        // enumerator
        fout << row << '\t';

        // format features
        string ftline;
        getline(ft, ftline, '\n');
        ftline = ftline.substr(0, ftline.size()-1);
        replace(ftline.begin(), ftline.end(), ':', '!');
        replace(ftline.begin(), ftline.end(), ' ', ',');
        fout << ftline << '\t';

        // format labels
        string lblline;
        getline(lbl, lblline, '\n');
        replace(lblline.begin(), lblline.end(), ':', '!');
        replace(lblline.begin(), lblline.end(), ' ', ',');
        fout << lblline << '\n';

    }

    cout << "done\n";
    return 0;


}

Thread: Reading 2 csv files and combining their content into an output

Thread Tools

Search Thread

Display

Similar Threads

Combining files together

GUIs and combining files

Combining files

Combining four hex characters into one floating point number for output?

Combining multiple wav files into one