Thread: CSV read problem usng tellg()

Hi all,

Having a nightmare here with a CSV file, sounds simple, should be, but is giving me some major pain!

The problem is i want to use tellg() to get me a value that i can use in my progress bar by dividing against the filesize, this works fine in other parts of the program.

However, in a final operation i do a dedupe against the output file that has just been written to in the main working functions.

I want my progress bar to now show how far through the dedupe we are so i used this same method. The problem is that each time tellg() is called it is moving the filepointer ahead to the 'wrong' place and hence the next getline(ifstream, string) call picks up a partial string.
If i comment out the call to tellg() then the file lines are read perfectly.

I have tried with other files and the problem does not occur so i have to think something is wrong with the format i am creating my output strings in when creating the output file.
I have tried variations, \"field\",\"field\" and field,field
The present variation does not use any quotes in the field and simply goes for a comma delimiter between the strings, when this leaves tellg() closer to where it should start, but still ahead.
like so:

Code:

void ListBuilder::CreateOutputString(unsigned& match)
{
    string tempStr;
    size_t pos;
    char qot = '\"';
    char comma = ',';

    if(match == MATCH_SWAP)
    {
        sfName1.swap(sfName2);
    }

    RemoveQuotes(outputStr);

    pos = outputStr.find_first_of(',');
    if(pos != string::npos)
    tempStr = outputStr.substr(pos+1, outputStr.length());

    outputStr.assign(sfName1+","+sfName2+","+tempStr+",");

    //get the publication name from diver line
    pos = dvLine.find_last_of(',');
    if(pos != string::npos)
    tempStr = dvLine.substr(pos+1,dvLine.length());
    RemoveQuotes(tempStr);

    if((currentSite == "NL") || (currentSite == "TE"))
    {
        Set_NE_PubLabel(tempStr);
    }

    outputStr.append(tempStr);
}

The file writing lines:
outputFile here writes to a temp file

Code:

 if(CheckSite())         //test the site name matches current site..if not ignore test and
    {                       //update current site until it matches next in vector
        if(!CheckDupeCustomer())
        {                           
            outputStr = sfThisLine;  
            SeperateSFNames();
            unsigned matchType = TestDiverMatch();
            if(matchType != NO_MATCH)
            {
                outputFile << outputStr << "\n";
                count++;                             //increment the matches count
                countOutp->value(GetCountString(matchType));  //output as c style string
            }
                   }
        setNext = true;
    }

The relevant call to tellg() in (inefficient but ok for testing!) dedupe function:
(Here the stream currOutp opens the temp file written to in code fragment above and the outputFile stream is reused to write the final results file.

Code:

            while(!currOutput.eof())
            {
                dupe = false;
                getline(currOutput, fileLine);
                for(unsigned i = 0; i < testLine.size(); i++)
                {
                    if(fileLine == testLine[i])
                    {
                        dupe = true;
                        i = testLine.size(); //exit loop
                    }
                }
                if(!dupe)
                {
                    outputFile << fileLine << "\n";
                    testLine.push_back(fileLine);
                    count++;
                }
                tg = currOutput.tellg();
                progressPrg->value( tg / file_size);
            }

This kind of fits my problem, seems to have been reported as a known issue with gcc / mingw some years ago, i will have to try their boards to see about any update.Bug 13333 – [cygwin] fstream tell/seek bug with DOS-format text files