Tokenizing a string

I'm trying to parse a line from an input file as a hex number followed by four signed integers.

I found this to work beautifully:

Code:

---

trace_file >> std::hex >> pc;
trace_file >> std::dec >> op >> dest_reg >> src1_reg >> src2_reg;

---

except that EOF is not being detected for some reason and I am reading too far.

Maybe someone can help with that instead because that works so well (until I get to the end).

Here is how I am using it

Code:

---

void fetch()
{
    uint num_fetched = 0;

    InstructionPtr fetched_instr;

    while (!dispatch_queue->full() && num_fetched++ < depth && !trace_file.eof())
    {
        fetched_instr = fetchInstruction(trace_file);

        dispatch_queue->push(fetched_instr);

        rob->push(fetched_instr);

        fetched_instr->info.seq_no = fetched_instr->tag() - 1;
        fetched_instr->info.if_begin = current_cycle;
    }
}

InstructionPtr fetchInstruction(std::ifstream &trace_file)
{
    unsigned int pc;
    unsigned int op;
    unsigned int dest_reg;
    int src1_reg;
    int src2_reg;

    trace_file >> std::hex >> pc;
    trace_file >> std::dec >> op >> dest_reg >> src1_reg >> src2_reg;

    InstructionPtr instr(new Instruction(pc, op, dest_reg, src1_reg, src2_reg));

    return instr;
}

---

With that code I am reading too many lines and I segfault. My file ends with a char that is an int, no newline or space after it.

I have just tried looking into using stings and char*s and even boost's tokenizer. I believe the tokenizing works fine but strtol complains if I use the string function c_str and I was getting segfaults really early when I passed it a char*.

I've looked into stringstreams as well but could not find an example or details for what I need to do.

The problem is that the EOF condition is only reached when the read fails due to EOF being reached. So, after everything has been read trace_file.eof() is still false, until that next time when you try to read from trace_file to pc, but by then control is already in the loop body.

Now, I am guessing that fetched_instr really is a pointer, and that fetchInstruction() returns a null pointer if EOF is reached. If so, you could write:

Code:

---

while (!dispatch_queue->full() && num_fetched++ < depth && (fetched_instr = fetchInstruction(trace_file)))
{
    dispatch_queue->push(fetched_instr);

    rob->push(fetched_instr);

    fetched_instr->info.seq_no = fetched_instr->tag() - 1;
    fetched_instr->info.if_begin = current_cycle;
}

---

Excellent! Thanks once again.