Searching a file for a String Occurence

Okay, I can't read anymore about file I/O.

I have the follow class:

Code:

---

class Student
{
private:
    std::string studentName;
    int numberDaysPresent;
    int numberDaysAbsent;
    int totalDaysOfClass;
    double currentGrade;
}

---

I read the data in from a comma delimited text file, and create 543 Student objects that contain that data.

What I want to do now is search a different data file for all occurences of Student[x].studentName. But I do not want to load the other file completely in memory because it contains 11000 lines of data (and takes forever to load) in the following comma delimited format:
DATE OF CLASS, STUDENT NAME, CLASS PARTICIPATION GRADE, QUIZ GRADE.

I have been chewing on this one for several days, and haven't been able to figure it out. Any suggestions would be greatly appreciated.

Cheers!

EDIT:
I then want to grab just the data from the file that relates to the matched student and assign it to the following class:

Code:

---

class OldStudentData
{
    std::string dateOfClass;
    std::string studentName;
    double participationGrade;
    double quizGrade;
}

---

Open the file with ifstream, and chug on through with getline(), setting ',' as the delim. Read and check one field at a time -- or one line at a time.
Give a larger portion of the file and I can be more specific.

I would go with one line at a time. Read in a line into a string, type the magic word "find", and if successful then process the info.

Quote:

---

Originally Posted by CodeMonkey

Open the file with ifstream, and chug on through with getline(), setting ',' as the delim. Read and check one field at a time -- or one line at a time.
Give a larger portion of the file and I can be more specific.

---

Quote:

---

Originally Posted by tabstop

I would go with one line at a time. Read in a line into a string, type the magic word "find", and if successful then process the info.

---

Both of those make more sense, I had been loading all of the data into the second class, and then comparing the two classes, which takes an eternity. I'll try the getline and find.

Thanks!

Hmmm, delimma, and I just figured the problem that I don't know how to solve.

The second file isn't a comma delimited text file, it is a tab delimited binary file with the following struct:

Code:

---

struct ClassData
{
    char dateOfClass[15];
    char studentName[25];
    double partGrade;
    double quizGrade;
}

---

I thought it was a text file, but I finally found the program that I used to originally write the data file. How would I read the data and compare the class to the binary data?

Cheers!

tab-delimited binary? Really?

Well, whatever you used to write it out, do the opposite to read it in. (fwrite->fread, etc.) You will have a char[] instead of a string, and then compare on that char array.

Quote:

---

Originally Posted by tabstop

tab-delimited binary? Really?

Well, whatever you used to write it out, do the opposite to read it in. (fwrite->fread, etc.) You will have a char[] instead of a string, and then compare on that char array.

---

Then I would have to do what I don't want to do: read the entire file into memory then compare the class against the data. That was taking upwards of 30-40 minutes. I will probably just read the binary in and write it as a comma delimited text file.

One other question to throw out there, how in the world do you use a buffer to read from a file?

Thanks in advance!

Quote:

---

Originally Posted by Phyxashun

Then I would have to do what I don't want to do: read the entire file into memory then compare the class against the data.

---

I never suggested such a thing. Read in one "line" of data, just like you were planning to do with getline. Nothing actually changes, here.

Oh, and a buffer is where you put your data when you read it in. You read in one piece of data, and you put it somewhere, and you process it; and when you read the next piece of data in you put it in the same place.

Quote:

---

Originally Posted by tabstop

I never suggested such a thing. Read in one "line" of data, just like you were planning to do with getline. Nothing actually changes, here.

Oh, and a buffer is where you put your data when you read it in. You read in one piece of data, and you put it somewhere, and you process it; and when you read the next piece of data in you put it in the same place.

---

Hmmm, guess I need to go to sleep on this one, maybe I will be able to wrap my head around it tomorrow. Appreciate the help.

Cheers!

Quote:

---

Originally Posted by Phyxashun

Then I would have to do what I don't want to do: read the entire file into memory then compare the class against the data. That was taking upwards of 30-40 minutes. I will probably just read the binary in and write it as a comma delimited text file.

One other question to throw out there, how in the world do you use a buffer to read from a file?

Thanks in advance!

---

Just HOW large is this file. (Or are you running this on a 1980's computer?).

By the way, unless you have some good reason to believe that you will find the data in the first half of the file, you can expect that reading "one line" at a time will still average around 15-20 minutes, since on average you will read half the file, and worst case is still 30-40 minutes, since it will take that long to read the entire file (including when you search for something that you can't find).

Unless of course you find a more clever way of storing your data - for example, if you are searching for a name, and the file is sorted in name-order, you could (assuming all lines are the same size) use a binary search - which allows you to search 1 million entries in about 20 comparisons.

Databases uses a key<->index table that is sorted, so that you can search through for example "first name" to find "John" and then know where (at what index) you can find that data in the file. But that requires a fair bit more work.

--
Mats

Quote:

---

Originally Posted by matsp

Just HOW large is this file. (Or are you running this on a 1980's computer?).

By the way, unless you have some good reason to believe that you will find the data in the first half of the file, you can expect that reading "one line" at a time will still average around 15-20 minutes, since on average you will read half the file, and worst case is still 30-40 minutes, since it will take that long to read the entire file (including when you search for something that you can't find).

---

Like I said in the beginning, the file is approximately 11000 entries in the specified format, but they are sorted by class date not by name. I will continue hacking away at the problem, once a find the solution that works for me; I'll post it.

Thanks!

Can we see the code that you are using to load the entire file and then search it? Even for 11,000 entries, I can see this taking a few seconds, not a few minutes... There might be something wrong with the way you are loading it.

I think I figured it out:

Code:

---

ClassData searchRecords(ClassData &obj, long recordNum)
{
    ifstream fin(FILE_NAME2, ios::in | ios::binary);
    if(!fin)
    {
        cout << "Cannot open file.\n";
        exit(1);
    }

    fin.seekg(sizeof(ClassData) * recordNum, ios::beg);
    fin.read((char *) &obj, sizeof(ClassData));
    fin.close();

    if(!fin.good())
    {
        cout << "A file error occurred.\n";
        exit(1);
    }
    return obj;
}

---

I then used a for loop to search through everything read by the above function until I found matches then just stored that data in different member variables that I added to the original student class.

Thanks for the help!

Quote:

---

Originally Posted by Phyxashun

I think I figured it out:

Code:

---

ClassData searchRecords(ClassData &obj, long recordNum)
{
    ifstream fin(FILE_NAME2, ios::in | ios::binary);
    if(!fin)
    {
        cout << "Cannot open file.\n";
        exit(1);
    }

    fin.seekg(sizeof(ClassData) * recordNum, ios::beg);
    fin.read((char *) &obj, sizeof(ClassData));
    fin.close();

    if(!fin.good())
    {
        cout << "A file error occurred.\n";
        exit(1);
    }
    return obj;
}

---

I then used a for loop to search through everything read by the above function until I found matches then just stored that data in different member variables that I added to the original student class.

Thanks for the help!

---

It would of course help if you didn't open, seek, and close the file for EVERY record - although it will work that way, it makes it a tad inefficient - it would be better to open the file in the function that calls the searchData function, and then close it when you have finished searching. You should also consider not returning the object, since you are already passing it in by reference - so why also make a copy for the calling function to the copy it again.

--
Mats

I changed it:

Code:

---

void searchRecords(ClassData &obj, long recordNum)
{
    fin.seekg(sizeof(ClassData) * recordNum, ios::beg);
    fin.read((char *) &obj, sizeof(ClassData));
}

---

Thanks for the help!

One thing that I would like to do now is make the search not case sensitive, any ideas out there?

I think if I convert the camparing to upper or lower case before comparison; that will work, will it not?

Cheers!

If you convert both to monocase before comparing, that will work.

There is no standard case-insensitive comparison function, but most OSes have a nonstandard one of their own.

You could always implement your own version using toupper();

Quote:

---

Originally Posted by CodeMonkey

You could always implement your own version using toupper();

---

That's what I'm working on, problem is: I want a generic function to do so. The first class uses strings the second is a structure of char arrays.

ideas appreciated.

Use a template that accepts any object for which operator[] can be called.

*then termination might be tricky*

Quote:

---

Originally Posted by CodeMonkey

Use a template that accepts any object for which operator[] can be called.

*then termination might be tricky*

---

those are my thoughts as well, once I have something that quasi works, i'll post it to help get it refined. I usually go about accomplishing tasks the hard way:)

Thanks!

Come to think of it, I suggest you just overload the function. You could bank on std::string::operator[](this->size()) to be 0, as the standard demands for c_str() -- but the former is not always true. The obvious solution is NocaseComp(str1, str2, num), but that seems inelegent, especially if only two data types are seriously under consideration... and for such a small function.

Actually, is this ugly and slow?

Code:

---

template<class string_type>
bool strcompi(string_type & a, string_type & b)
{
    std::string one(a);
    std::string two(b);
    std::for_each(one.begin(), one.end(), std::toupper);
    std::for_each(two.begin(), two.end(), std::toupper);
    return one == two;
}

---

Quote:

---

Originally Posted by CodeMonkey

Actually, is this ugly and slow?

---

The use of std::toupper (as opposed to the global toupper) is problematic in this case, though the exact reason eludes me at the moment. (Type mismatch? I know this has been discussed before.) Josuttis suggests just defining a function that compares characters without regard for case, and then use say, equal(), e.g.,

Code:

---

bool noCaseEqualChar(char lhs, char rhs)
{
    return std::toupper(lhs) == std::toupper(rhs);
}

bool noCaseEqual(const std::string& lhs, const std::string& rhs)
{
    return lhs.size() == rhs.size()
        && std::equal(lhs.begin(), lhs.end(), rhs.begin(), noCaseEqualChar);
}

---

We could use function object template classes instead and thus not have to resort to the different function names.

It was actually much easier than that:

Code:

---

strupr((char *)studentName.c_str());  // This is the string member of the class
strupr((char *)obj.studentName);      // This is the char[] member of the struct

---

Then compare them, but since I don't want to change the original data I am going to copy them into temp variables, put them in uppercase, then compare them.

Is there any problem with doing it this way?

Thanks everyone for you help!

Quote:

---

Originally Posted by Phyxashun

Is there any problem with doing it this way?

---

Yes. The c_str() member function returns a const char* for a good reason: you are not allowed to modify the string via the pointer returned. Blatantly ignoring the const qualifier by means of a cast is just sticking your fingers in your ears to try and block out the fire alarm in the hope that the fire will not actually burn you.

Besides that, a potential problem is that strupr() is non-standard.

I'm not worried about strupr not being standard.

but is this correct?

Code:

---

char *pStudent = new char[sizeof(studentName)+1];
strcpy(pStudent, studentName.c_str());

char *pOStudent = new char[sizeof(obj.studentName)];
strcpy(pOStudent, obj.studentName);

strupr(pStudent);
strupr(pOStudent);
...
...
...
delete [] pStudent;
delete [] pOStudent;

---

it works, and doesn't modify the original.

Quote:

---

Originally Posted by Phyxashun

but is this correct?

---

Almost. If the second new[] throws std::bad_alloc then you would get a memory leak since you would not be able to use delete[] on the pointer returned by the first new[].

Why are you so resistant to using std::string and my suggestion?

EDIT:
Wait a minute. studentName is a std::string, so that use of sizeof is wrong. You would use studentName.size() (or studentName.length()) instead.

And why doesn't this logic hold?

Code:

---

if(pStudent == pOStudent)
...

---

There is no reason why haven't tried std::string yet. From what I can tell that will be the route I take. Right now I am just fiddling around with the function, to see what I can come up with.

Thanks!

Quote:

---

Originally Posted by Phyxashun

And why doesn't this logic hold?

---

As in it does not actually do a case insensitive comparison? That would be because pStudent and pOStudent are pointers, so you are merely comparing addresses. You would need to use strcmp() if you really want to use C-style strings.

Quote:

---

Originally Posted by laserlight

As in it does not actually do a case insensitive comparison? That would be because pStudent and pOStudent are pointers, so you are merely comparing addresses. You would need to use strcmp() if you really want to use C-style strings.

---

My mind isn't working right today, I shouldn't have woken up to watch the BCS Championship earlier this morning. I need some coffee! Or a nap :)

But it goes to show that I am reminded of something every day!

Code:

---

if(!strcmp(pStudent, pOStudent))
...

---

Thanks!

Now I will try the std::string route, once I have that figured out i'll post it.

Cheers!

Quote:

---

Originally Posted by Phyxashun

But it goes to show that I am reminded of something every day!

---

Personally, I would prefer to write:

Code:

---

if (strcmp(pStudent, pOStudent) == 0)

---

The ! make me think of "not equal" instead of "equal" :)

Quote:

---

Originally Posted by Phyxashun

Now I will try the std::string route, once I have that figured out i'll post it.

---

Incidentally, if you are trying to figure out my example, note that you need to #include <algorithm> for std::equal(). If you are trying to write a case insensitive "less than" comparator then the same idea applies, but you might use std::lexicographical_compare() instead of std::equal().

Quote:

---

Originally Posted by laserlight

The use of std::toupper (as opposed to the global toupper) is problematic in this case, though the exact reason eludes me at the moment. (Type mismatch? I know this has been discussed before.) Josuttis suggests just defining a function that compares characters without regard for case, and then use say, equal(), e.g.,

Code:

---

bool noCaseEqualChar(char lhs, char rhs)
{
    return std::toupper(lhs) == std::toupper(rhs);
}

bool noCaseEqual(const std::string& lhs, const std::string& rhs)
{
    return lhs.size() == rhs.size()
        && std::equal(lhs.begin(), lhs.end(), rhs.begin(), noCaseEqualChar);
}

---

We could use function object template classes instead and thus not have to resort to the different function names.

---

These functions are absolutely perfect for use when comparing std::strings.

Thanks laserlight!

Quote:

---

Originally Posted by laserlight

Josuttis suggests just defining a function that compares characters without regard for case, and then use say, equal(), e.g.,

---

What page does Josuttis talk about that. I can't find it.

Thanks in advance!

Quote:

---

Originally Posted by Phyxashun

What page does Josuttis talk about that. I can't find it.

---

In the 16th printing of The C++ Standard Library: A Tutorial and Reference, Josuttis talks about this case insensitive comparison on pages 499 to 501.

Once again, thanks.

:)

This might be useful to someone:

Code:

---

/*******************************************************************************
*  Usage:
*      to_lower(std::string); Converts std::string to lowercase
*      to_upper(std::string); Converts std::string to uppercase
*******************************************************************************/
#include <locale>
#include <string>

// to_lower(std::string)
template <typename Char, typename Traits>
    std::basic_string<Char, Traits> &to_lower(std::basic_string<Char, Traits>
    &str, std::locale loc = std::locale())
{
    typedef std::basic_string<Char, Traits> string;
    typedef std::ctype<Char> char_type;
    char_type const * the_type_ptr = &std::use_facet<char_type>(loc);
    for (typename string::size_type i = 0; i <str.size(); ++i)
    {
        str[i] = the_type_ptr->tolower(str[i]);
    }
    return (str);
}

// to_upper(std::string)
template <typename Char, typename Traits>
    std::basic_string<Char, Traits> &to_upper(std::basic_string<Char, Traits>
    &str, std::locale loc = std::locale())
{
    typedef std::basic_string<Char, Traits> string;
    typedef std::ctype<Char> char_type;
    char_type const * the_type_ptr = &std::use_facet<char_type>(loc);
    for (typename string::size_type i = 0; i <str.size(); ++i)
    {
        str[i] = the_type_ptr->toupper(str[i]);
    }
    return (str);
}

---

Cheers!

This is what I settled on:

Code:

---

void searchRecords(std::ifstream &fin, std::string &searchString,
                  OldStudentData &obj, long recordNum)
{
    fin.seekg(sizeof(OldStudentData) * recordNum, std::ios::beg);
    fin.read((char *) &obj, sizeof(OldStudentData));

    std::string personName(obj.studentname);

    if(noCaseEqual(searchString, personName))
    {
        // Do some stuff in here!
    }
}

bool noCaseEqual(const std::string &lhs, const std::string &rhs)
{
    return lhs.size() == rhs.size()
        && std::equal(lhs.begin(), lhs.end(), rhs.begin(), noCaseEqualChar);
}

bool noCaseEqualChar(char lhs, char rhs)
{
    return std::toupper(lhs) == std::toupper(rhs);
}

---

Thanks for all of the help! Any Ideas on refinement?