Tokenizing a string

This is a discussion on Tokenizing a string within the C++ Programming forums, part of the General Programming Boards category; I'm trying to parse a line from an input file as a hex number followed by four signed integers. I ...

  1. #1
    Registered User
    Join Date
    Oct 2009
    Posts
    48

    Tokenizing a string

    I'm trying to parse a line from an input file as a hex number followed by four signed integers.

    I found this to work beautifully:
    Code:
    trace_file >> std::hex >> pc;
    trace_file >> std::dec >> op >> dest_reg >> src1_reg >> src2_reg;
    except that EOF is not being detected for some reason and I am reading too far.

    Maybe someone can help with that instead because that works so well (until I get to the end).

    Here is how I am using it
    Code:
    void fetch()
    {
        uint num_fetched = 0;
    
        InstructionPtr fetched_instr;
    
        while (!dispatch_queue->full() && num_fetched++ < depth && !trace_file.eof())
        {
            fetched_instr = fetchInstruction(trace_file);
    
            dispatch_queue->push(fetched_instr);
    
            rob->push(fetched_instr);
    
            fetched_instr->info.seq_no = fetched_instr->tag() - 1;
            fetched_instr->info.if_begin = current_cycle;
        }
    }
    
    InstructionPtr fetchInstruction(std::ifstream &trace_file)
    {
        unsigned int pc;
        unsigned int op;
        unsigned int dest_reg;
        int src1_reg;
        int src2_reg;
    
        trace_file >> std::hex >> pc;
        trace_file >> std::dec >> op >> dest_reg >> src1_reg >> src2_reg;
    
        InstructionPtr instr(new Instruction(pc, op, dest_reg, src1_reg, src2_reg));
    
        return instr;
    }
    With that code I am reading too many lines and I segfault. My file ends with a char that is an int, no newline or space after it.

    I have just tried looking into using stings and char*s and even boost's tokenizer. I believe the tokenizing works fine but strtol complains if I use the string function c_str and I was getting segfaults really early when I passed it a char*.

    I've looked into stringstreams as well but could not find an example or details for what I need to do.

  2. #2
    C++ Witch laserlight's Avatar
    Join Date
    Oct 2003
    Location
    Singapore
    Posts
    21,457
    The problem is that the EOF condition is only reached when the read fails due to EOF being reached. So, after everything has been read trace_file.eof() is still false, until that next time when you try to read from trace_file to pc, but by then control is already in the loop body.

    Now, I am guessing that fetched_instr really is a pointer, and that fetchInstruction() returns a null pointer if EOF is reached. If so, you could write:
    Code:
    while (!dispatch_queue->full() && num_fetched++ < depth && (fetched_instr = fetchInstruction(trace_file)))
    {
        dispatch_queue->push(fetched_instr);
    
        rob->push(fetched_instr);
    
        fetched_instr->info.seq_no = fetched_instr->tag() - 1;
        fetched_instr->info.if_begin = current_cycle;
    }
    C + C++ Compiler: MinGW port of GCC
    Version Control System: Bazaar

    Look up a C++ Reference and learn How To Ask Questions The Smart Way

  3. #3
    Registered User
    Join Date
    Oct 2009
    Posts
    48
    Excellent! Thanks once again.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Inheritance Hierarchy for a Package class
    By twickre in forum C++ Programming
    Replies: 7
    Last Post: 12-08-2007, 03:13 PM
  2. String issues
    By The_professor in forum C++ Programming
    Replies: 7
    Last Post: 06-12-2007, 09:11 AM
  3. Compile Error that i dont understand
    By bobthebullet990 in forum C++ Programming
    Replies: 5
    Last Post: 05-05-2006, 09:19 AM
  4. Classes inheretance problem...
    By NANO in forum C++ Programming
    Replies: 12
    Last Post: 12-09-2002, 02:23 PM
  5. Warnings, warnings, warnings?
    By spentdome in forum C Programming
    Replies: 25
    Last Post: 05-27-2002, 06:49 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21