parsing through a text file

I decided to try my hand at C++ in order to get this done.

I have a source text file that is formatted like this:

15:20:35.334 Race /myfolder/racenumber3234.rdc, /myfolder/racenumber3234.rdc, BUS ENGINE, VER TAT124
MORE RANDOM INFO
0: AC45 F678 00E4 65D2 BLOCK 4, 197 Words (15:20:35.334)
4: 0001 5476 F432 E456 3214 F83E F436 B23A 68C3 F84E EEE7 DF43 45CB 98CE 31BD B345 CAEF 43ED 4E5C 4556
24: 0001 5476 F432 E456 3214 F83E F436 B23A 68C3 F84E EEE7 DF43 45CB 98CE 31BD B345 CAEF 43ED 4E5C 4556
44: 0001 5476 F432 E456 3214 F83E F436 B23A 68C3 F84E EEE7 DF43 45CB 98CE 31BD B345 CAEF 43ED 4E5C 4556
64: 0001 5476 F432 E456 3214 F83E F436 B23A 68C3 F84E EEE7 DF43 45CB 98CE 31BD B345 CAEF 43ED 4E5C 4556
84: 0001 5476 F432 E456 3214 F83E F436 B23A 68C3 F84E EEE7 DF43 45CB 98CE 31BD B345 CAEF 43ED 4E5C 4556
104: 0001 5476 F432 E456 3214 F83E F436 B23A 68C3 F84E EEE7 DF43 45CB 98CE 31BD B345 CAEF 43ED 4E5C 4556
124: 0001 5476 F432 E456 3214 F83E F436 B23A 68C3 F84E EEE7 DF43 45CB 98CE 31BD B345 CAEF 43ED 4E5C 4556
144: 0001 5476 F432 E456 3214 F83E F436 B23A 68C3 F84E EEE7 DF43 45CB 98CE 31BD B345 CAEF 43ED 4E5C 4556
164: 0001 5476 F432 E456 3214 F83E F436 B23A 68C3 F84E EEE7 DF43 45CB 98CE 31BD B345 CAEF 43ED 4E5C 4556
184: 0001 5476 F432 E456 3214 F83E F436 B23A 68C3 F84E EEE7 DF43 45CB 98CE 31BD B345 CAEF 43ED 4E5C 4556
15:20:44.645 Race /myfolder/racenumber3234.rdc, /myfolder/racenumber3234.rdc, BUS ENGINE, VER TAT124
MORE RANDOM INFO
0: AC45 F678 00E4 65D2 BLOCK 4, 197 Words (15:20:44.645)
4: 0001 5476 F432 E456 3214 F83E F436 B23A 68C3 F84E EEE7 DF43 45CB 98CE 31BD B345 CAEF 43ED 4E5C 4556
24: 0001 5476 F432 E456 3214 F83E F436 B23A 68C3 F84E EEE7 DF43 45CB 98CE 31BD B345 CAEF 43ED 4E5C 4556
44: 0001 5476 F432 E456 3214 F83E F436 B23A 68C3 F84E EEE7 DF43 45CB 98CE 31BD B345 CAEF 43ED 4E5C 4556
64: 0001 5476 F432 E456 3214 F83E F436 B23A 68C3 F84E EEE7 DF43 45CB 98CE 31BD B345 CAEF 43ED 4E5C 4556
84: 0001 5476 F432 E456 3214 F83E F436 B23A 68C3 F84E EEE7 DF43 45CB 98CE 31BD B345 CAEF 43ED 4E5C 4556
104: 0001 5476 F432 E456 3214 F83E F436 B23A 68C3 F84E EEE7 DF43 45CB 98CE 31BD B345 CAEF 43ED 4E5C 4556
124: 0001 5476 F432 E456 3214 F83E F436 B23A 68C3 F84E EEE7 DF43 45CB 98CE 31BD B345 CAEF 43ED 4E5C 4556
144: 0001 5476 F432 E456 3214 F83E F436 B23A 68C3 F84E EEE7 DF43 45CB 98CE 31BD B345 CAEF 43ED 4E5C 4556
164: 0001 5476 F432 E456 3214 F83E F436 B23A 68C3 F84E EEE7 DF43 45CB 98CE 31BD B345 CAEF 43ED 4E5C 4556
184: 0001 5476 F432 E456 3214 F83E F436 B23A 68C3 F84E EEE7 DF43 45CB 98CE 31BD B345 CAEF 43ED 4E5C 4556

This sample might be too big, so let me know if I'm breaking some kind of rule.

It's not exactly like this, as it has hundreds of blocks like this instead of just 2, that are different. I just made one up and pasted it 2 times to demonstrate the algorithm that I want to use and need help refining.

The algorithm for extracting what I need is going to be like this:
The first 2 lines(the ones that have a timecode, file location info, bus, and "more random info". Can be discarded pretty much.

The first 4 hex codes starting at 0: can be discarded, as well as the first hex code immediately after the 4:

I figure i can extract the timecode I need from the one located in parenthesis.

Basically, after those entries I discussed are discarded, starting from the 6th hex code basically, I need to make it so that the 6th hex code, the 9th code, and the 12th code and so on are rearranged in a column grouped together. The 7th, 10th, and 13th in another, and the 8th 11th and 14th.

I figure I can use arrays to do this. But the problem I'm having is discarding the things that need discarded, while also parsing the hex codes into the arrays.

This is what I have so far code wise. I don't have much experience yet with this.

Code:

---

#include <iostream>
#include <sstream>
#include <fstream>

using namespace std;

int main()
{
    string line;
    ifstream myfile;
    ofstream myfile;
    myfile.open("input.txt");
    myfile2.open("output.txt");
   
    while (myfile.good() )
    {
          getline(myfile,line);
          myfile2<<line<<endl;
    }
    myfile.close();
    myfile2.close();

    return 0;
}

---

A general idea: grab each line with getline(), as you have done, make a stringstream out of the line and use ordinary >> to split the input into numbers, words, etc. You can also just use >> from the original file directly, but it's much harder to recover from errors that way and track line numbers, etc. A starting point:

Code:

---

std::ifstream file("input.txt");
std::string line;
while(std::getline(file, line)) {
    std::stringstream ss(line);

    // do whatever parsing you want... here I assume there's an int at the beginning, a colon, then a word.
    int x;
    char junk;  // to grab the ':'
    std::string firstword;
    if(ss >> x >> junk >> firstword) {
        std::cout << "success\n";
    }
    else { /* error */ }
}

---

I've used std:: front of stuff, you don't need that if you're going to have using namespace std. Notice how you can just check the return value of getline() to see whether it worked, without worrying about file.good(). It may still be worth checking file.is_open() though, to see whether the file could be opened or not. Also, files close themselves when the ifstream/whatever object gets destructed.

If you want more information, search around for C++ file tutorials. Also, picking apart something that has comma separators instead of whitespace separators might be a bit more work ... but you can do another getline on the stringstream with whatever separator you choose. read comma delimited text file into an a - C++ Forum

[edit] Also, if you want to parse hex codes you can use this:

Code:

---

#include <iomanip>  // for std::hex
int x;
ss >> std::hex >> x;  // assumes an integer in hex form

---

[/edit]

I think I understand the process here, but I think i'm stuck on how to properly parse. I can read in the time code that I need just fine, but when I go to pick out the hex codes I'm having some trouble:

Code:

---

#include <iostream>
#include <sstream>
#include <fstream>
#include <iomanip>

using namespace std;

int main()
{

    string line;
    ifstream myfile;
    ofstream myfile2, myfile3;
    myfile.open("input.txt");
    myfile2.open("output.txt");
    myfile3.open("logfile.txt");
    myfile2<<"column1\t\t"<<"column2\t\t"<<column3<<endl;

    while(getline(myfile,line)
    {
          stringstream ss(line);
          int x[20],y,z,hours,minutes,index;
          char colon;
          double seconds;
          string junk;
          string word1;

if(ss>> hours>>colon>>minutes>>colon>>seconds && colon==':')
{
myfile3<<"success reading time code\n";
myfile<<hours<<":"<<minutes<<":"<<seconds<<endl;
}

else if(ss>>index>>colon>>hex>>x[0]>>hex>>x[1]>>hex>>x[2]>>hex>>x[3]>>hex>>x[4]>>hex>>x[5]>>hex>>x[6]>>hex>>x[7]>>hex>>x[8]>>hex>>x[9]>>hex>>x[10]>>hex>>x[11]>>hex>>x[12]>>hex>>x[13]>>hex>>x[14]>>hex>>x[15]>>hex>>x[16]>>hex>>x[17]>>hex>>x[18]>>hex>>x[19] && colon==":")
{
myfile3<<"success reading hex code line\n";
myfile2<<x[0]<<endl;
}

else
myfile3<<"error"<<endl;

}

myfile.close();
myfile2.close();
myfile3.close();
return0

}

---

Forgive the formatting, but I can't copy and paste as this is a different computer than the one I have to program on.

This program compiles fine, but it only will print the time line (the first if statement) it's as if the 2nd if statement isn't being run even though getline should be fed through the loop again.

Is there something that I'm missing? The format of the text that I am reading from is in my first entry. Maybe my format is wrong, but it shouldn't be.

If the stringstream has failed you have to clear() the fail state, otherwise any attempt to read will fail.

Code:

---

if(ss>> hours>>colon>>minutes>>colon>>seconds && colon==':') {
    myfile3<<"success reading time code\n";
    myfile<<hours<<":"<<minutes<<":"<<seconds<<endl;
}
// reading has failed
// the next reads will fail as well
else if(ss>>index>>colon>>hex>>x[0]>> ...  && colon==":") {
    myfile3<<"success reading hex code line\n";
    myfile2<<x[0]<<endl;
}

---

use ss.clear() before reading the hex data.

Kurt

EDIT: Better would be to initialize another stringstream with line and extract the data from that one because the first two fields would already be extracted.

Is there a way to know that stringstream failed the ss call? Or do I just assume that it failed and clear it anyway?

And should I clear it after the if/else if/else?

Quote:

---

Originally Posted by oopsyourhead

Is there a way to know that stringstream failed..

---

Code:

---

if(!your_stream_object)
{
    //It failed
}

---

I don't think I'm properly grasping why this isn't working.

#include <iostream> #include <sstream> #include <fstream> #include <iomanip> using namespace std; int main() { string line; ifstream myfile; ofstream myfile2, myfile3; myfile.open("input.txt"); myfile2.open("output.txt"); myfile3.open("logfile.txt"); myfile2<<"column1\t\t"<<"column2\t\t"<<column3<<endl; while(getline(myfile,line) { stringstream ss(line); ss.clear(); int x[20],y,z,hours,minutes,index; char colon; double seconds; string junk; string word1; if(ss>> hours>>colon>>minutes>>colon>>seconds && colon==':') { myfile3<<"success reading time code\n"; myfile<<hours<<":"<<minutes<<":"<<seconds<<endl; ss<<""; ss.clear(); } else if(ss>>index>>colon>>hex>>x[0]>>hex>>x[1]>>hex>>x[2]>>hex>>x[3]>>hex>>x[4]>>hex>>x[5]>>hex>>x[6]>>hex>>x[7]>>hex>>x[8]>>hex>>x[9]>>hex>>x[10]>>hex>>x[11]>>hex>>x[12]>>hex>>x[13]>>hex>>x[14]>>hex>>x[15]>>hex>>x[16]>>hex>>x[17]>>hex>>x[18]>>hex>>x[19] && colon==":") { myfile3<<"success reading hex code line\n"; myfile2<<x[0]<<endl; ss<<""; ss.clear(); } else{ myfile3<<"error"<<endl; ss<<""; ss.clear(); } } myfile.close(); myfile2.close(); myfile3.close(); return0 }

I've added what I have found in a few guides, and put ss<<""; and ss.clear(); into every entry. And it still isn't working. Am I just not understanding the way this program is being read?

OK, so now I've gotten it to kind of discern between my if/else if/else, but now the else if isn't reading the right characters:

Code:

---

#include <iostream>
#include <sstream>
#include <fstream>
#include <iomanip>

using namespace std;

int main()
{

    string line;
    ifstream myfile;
    ofstream myfile2, myfile3;
    myfile.open("data.txt");
    myfile2.open("output.txt");
    myfile3.open("logfile.txt");
    myfile2<<"column1\t\t"<<"column2\t\t"<<"column3"<<endl;

    while(getline(myfile,line))
    {
        stringstream ss(line);
        int x[20],y,z,hours,minutes,index,index2;
        char colon;
        double seconds;
        string junk,junk2,junk3,junk4,junk5,junk6,junk7, junk8,junk9,junk10;
        string word1;
        //cout<<ss.str()<<endl;

        if(ss>> hours>>colon>>minutes>>colon>>seconds>>junk>>junk2>>junk3>>junk4>>junk5>>junk6>>junk7>>junk8>>junk9>>junk10 && colon==':')
        {
            cout<<ss.str()<<endl;
            myfile3<<"success reading time code"<<endl;
            myfile2<<hours<<":"<<minutes<<":"<<seconds<<endl;
            ss<<"";
            ss.clear();
        }



        else if(ss>>word1)
        {
            cout<<ss.str()<<endl;
            myfile3<<"success reading hex code line\n";
            myfile2<<word1<<endl;
            ss<<"";
            ss.clear();
        }

        else
        {
            myfile3<<"error"<<endl;
            ss<<"";
            ss.clear();
        }

    }

myfile.close();
myfile2.close();
myfile3.close();
return 0;

}

---

It looks like after the initial read on the time code line, it's skipping a bunch entries and starting on the 13th entry on the line.

Like the time code outputs fine, then it goes 45CB which it should read in 4: the way I have it written currently, can anyone guess what is happening?

The cout's in each if statement are to check what ss is read as. And they appear to output the correct line.

It skips the line that starts with 0:, but that's ok as I was going to discard the data from that line anyway.

This won't work correctly:

Code:

---

if(ss>> hours>>colon>>minutes>>colon>>seconds>>/*...*/) {}
else if(ss>>word1) {}

---

If the first separation into hours, colon, minutes, etc doesn't work, then the stream will be put into an error state and you won't be able to read into a word. In other words, the else if will never execute. As mentioned, you need to clear the error state with .clear(). Something like this, perhaps:

Code:

---

if(ss>>a>>b>>c) {
    // ...
}
else {
    ss.clear();
    if(ss>>word) {
        // ...
    }
    else {
        std::cout << "Error\n";
    }
}

---

Or use multiple stringstream objects. Or, if you're familiar with functions, you could use simple ones to encompass each line format.

Code:

---

bool isLineWithTime(const std::string &line) {
    std::istringstream ss(line);
    int a, b, c;
    if(ss >> a >> b >> c) {
        std::cout << "Success: " << a << b << c << std::endl;
        return true;
    }
    return false;  // it's not this type of line
}

// in the main function:
if(isLineWithTime(line)) {}
else if(isLineWithOtherStuff(line)) {}
// ...and so on.

---

[edit] Notice what ZuK said earlier:

Quote:

---

EDIT: Better would be to initialize another stringstream with line and extract the data from that one because the first two fields would already be extracted.

---

This is important. If you reuse the same stringstream object, the first read might have been partially successful in picking off some tokens before failing. Then if you try to read more, you won't start at the beginning of the string. This is why you're seeing AC45 or whatever instead of 4:. It also implies that you might be able to continue reading (at least string words) from a stringstream even if it has failed... so maybe clear() isn't necessary.

I think the best idea would be to start using multiple string stream objects and see what happens. [/edit]

It seems like if I decide to use a separate stringstream for each line, that using the loop to parse through it will not work.

While doing each line separately will work for a short file, the file I'm trying to read is much longer than the 2 blocks that I listed in my first entry. We're talking hundreds of blocks like the one I posted.

Is there a way to incorporate reading into a different stream for each line into what I've already done? Is it possible to stringstream into another stringstream?

I guess implementing the methods/functions would build in multiple stringstreams if I chose, but wouldn't those variables that I read in to be local to that function only?

You could try to decide what type of line it is before reading from the stringstream

Code:

---

    bool readOK = true;
    while(readOk && getline(myfile,line)) {

          stringstream ss(line);
          // check for LineType
          if ( line.find("BUS ENGINE" ) != string::npos ) {
              // read time
              readOK = ss>> hours>>colon>>minutes>>colon>>seconds && colon==':'
          }
          else if  ( line.find("RANDOM INFO" ) != string::npos ) {
              // nothing to do
          }
          else if  ( line.find("Words" ) != string::npos ) {
              // nothing to do
          }
          else {
              // must be a hex data line
              readOK = ss>>index>>colon>>hex>>x[0]>>hex>>x[1]>>hex>>x[2]>>hex>>x[3]>>.... && colon==":"
          }
}

---

This way you would have to initialize only one stringstream from the input line and no clear() is needed.
Kurt

This method seems to be much easier to understand, the one problem i'm haing is from the one where I try and read the hex codes into an array of ints.

Code:

---

//junk 1 is the discarded hex code
else if (line.find("4:) !=string::npos)
{
    readOK=(ss>>index>>colon>>junk1>>hex>>x[0]  && colon==':')
    myfile2<<hex<<x[0];
}

---

is there a way to get it to make the hex digits 4 numbers long no matter what? I know in C you can do like %4x and it will format it to 4 hex digits

Try something like this

Code:

---

else if (line.find("4:) !=string::npos)
{ 
    ss>>index>>colon>>junk1;
    readOK= ss && colon == ':';

    const int NDAT = 19;
    int data[NDAT];

    for( int idx =0; readOK && idx < NDAT; ++idx )
        readOK= ss>>hex>>data[idx];

    if ( readOK ) {
        for( int idx =0; idx < NDAT; ++idx )
            myfile2<< setw(4) << setfill('0') << hex << data[idx] << ' ';
    }
    else {
        cerr << "faild to parse data." << endl;
    }
}

---

Kurt

OK so now I have the correct output for the program I was writing. The problem, is is that I need some additional info from this output file.

ID IC TIME
15:23:43.867 /g/mydata/dataoutputfile.txt
0003 1233 abcd
0043 eb54 abf3
000f 0bb4 ac24
000a a325 ac75
0023 0043 ac91
0003 1233 abcd
0043 eb54 abf3
000f 0bb4 ac24
000a a325 ac75
0023 0043 ac91
0003 1233 abcd
0043 eb54 abf3
000f 0bb4 ac24
000a a325 ac75
0023 0043 ac91

Is kind of the output I have. The time resets every so often.

What I am doing now is making 2 additional columns in addition to the 3 i have in my example. The first column is the conversion of the ID column, into a translation into an understandable message. The second additional column will calculate the difference between each time code, except when the time code resets.

My logic is, is to read each column into an array so I can perform the necessary translations and operations.

I'm reading the lines in similar to my previous program. And I am focusing on getting the timecode differential first, as I think getting the translation will be a bit simpler.

The problem I'm having is getting the entries read into their matrices:

my code looks a bit like this:

Code:

---

while(readOK && getline(myfile,line))
{
stringstream ss(line);
string ident,IC,timehex,time,filelocation;
string junk1,junk2;
int ID[count];
int timecode[count2];
int idx=0;

if(line.find("ID") !=string::npos)
{
    readOK=ss>>ident>>IC>>timehex;
    myfile2<<ident<<"\t\t"<<IC<<"\t\t"<<timehex<<"\t\t"<<"ID Decoded"<<"\t\t"<<"DT"<<endl;
    myfile3<<"headers read"<<endl
}

else if(line.find("identifier") != string::npos)
{
    readOK=ss>>time>>filelocation;
    myfile3<<"time and location read";
    myfile2<<time<<"\t\t"<<filelocation<<endl;
}

else //this is for the hex code lines
{
    readOK=ss>>hex>>ID[idx]>>IC>>timecode[idx];

    if (readOK)
    {
          myfile2<<setw(4)<<setfill('0')<<hex<<ID[1000]<<"\t\t"<<IC<<"\t\t"<<timecode[1000]<<endl;
          myfile3<<"success reading info into arrays"<<endl;
    }

    else
    myfile3<<"error reading hex codes"<<endl;
}
idx++;
}

---

My gut is telling me that I'm calling the matrix entries too early and they haven't been filled yet, because if I cout number 1000, I get a 0 (i have well over 15000 lines in my input file and I have the boundaries of my arrays set dynamically in another part of my program). Am I on the right track here?

The way you're declaring the arrays (e.g. timecode[]) is very strange. First of all, assuming count2 is a variable then they're variable-length arrays. Not technically standard, but if your compiler supports them you can use them. But you're declaring the arrays anew each time the loop executes -- so that the arrays will only contain the results of the last read. Hence if timecode[1000] is ever read in, you may only see that printed once (for iteration 1000), and during all other iterations it will appear to be uninitialized.

I'm not sure of the exact semantics of variable-length arrays, but I think they may always be initialized to zero, which would explain why you're always seeing timecode[1000] and ID[1000] as zero. They do get set, once, but then the array gets zeroed in the next iteration. If you used statically-sized arrays, they may -- *may*! -- get allocated to the same memory over and over again and you could, by coincidence, see the values getting retained -- but this is undefined behaviour. Far safer to move the array declarations outside the while loop, where the array will exist for the duration of the while loop and afterwords, and then you should be able to check out element 1000 with no issues (and it will stay set once you've set it . . .).

P.S. You may not want to call a variable "time", since it appears you're using namespace std. There's a standard function called std::time() if you were to include <ctime> or <time.h>, and this could cause confusion. It shouldn't, since the function has global scope and your own function would simply hide it, but it's good to get into the habit of not reusing standard library function names. (If you had a function called time() you could encounter a compiler error that only manifested when you included <ctime> . . . .)