Multithreading File Reading

This is a discussion on Multithreading File Reading within the C++ Programming forums, part of the General Programming Boards category; I was reading up on multithreading and I came across things like Mutexes, Locks, Starvation and it's quite interesting. Here ...

  1. #1
    printf("Hello Cboard\n"); codeprada's Avatar
    Join Date
    Jan 2011
    Location
    In a little room at the back of your brain
    Posts
    68

    Multithreading File Reading

    I was reading up on multithreading and I came across things like Mutexes, Locks, Starvation and it's quite interesting. Here I'm using CRITICAL_SECTION in my code to prevent data corruption and so forth. I could have used Mutexex, Events, Semaphores, etc... I managed to read and add the contents of a 5MB text file in about 5.9 seconds with 7 threads. This is a big improvement since I had a program before that took a whole 10 seconds to read and add the contents of a 1MB file to a vector. This one uses maps and counts each word's/phrase occurrences. The speed is highly dependent on the placement of the critical sections. I shaved off about 4 seconds just by moving the EnterCriticalSection down one line!!

    I'm looking for other ways to optimize the CRITICAL_SECTIONS though. I want to get this down to at least 3 seconds if possible. xD

    Code:
    #include <windows.h>
    #include <process.h>
    #include <iostream>
    #include <string>
    #include <fstream>
    #include <map>
    #include <time.h>
    
    using namespace std;
    
    map <string, long> words;
    ifstream f;
    CRITICAL_SECTION c, c2;
    
    unsigned int __stdcall thread(void *)
    {
        string buffer;
        map <string, long>::iterator iter;
        while (f >> buffer) {
            EnterCriticalSection(&c);
            f >> buffer;
            LeaveCriticalSection(&c);
            iter = words.find(buffer);
            EnterCriticalSection(&c2);
            if (iter != words.end())
                iter->second++;
            else
                words.insert(make_pair(buffer, 1));
            LeaveCriticalSection(&c2);
    
        }
        return 0;
    }
    
    int main()
    {
        cout << "How much threads do you want to use? -> ";
        int thread_count;
        cin >> thread_count;
        f.open("file.txt", ios::in);
        clock_t t1, t2;
        if (f.is_open())
        {
            InitializeCriticalSection(&c);
            InitializeCriticalSection(&c2);
            t1 = clock();
            HANDLE * hThreads = (HANDLE *)malloc(sizeof(HANDLE) * thread_count);
            for(int i = 0; i < thread_count; i++)
                hThreads[i] = (HANDLE) _beginthreadex(0, 0, &thread, (void *)0, 0, 0);
            WaitForMultipleObjects(thread_count, hThreads, true, INFINITE);
            t2 = clock();
            for(int i = 0; i < thread_count; i++)
    	     CloseHandle(hThreads[i]);
            DeleteCriticalSection(&c2);
            DeleteCriticalSection(&c);
            f.close();
            map<string, long>::iterator iter = words.begin();
            //IF YOU WISH TO DISPLAY THE MAP
            /*for(; iter != words.end(); iter++)
                cout << iter->first.c_str() << " - " << iter->second << endl;*/
            cout << "Unique WOrD CoUNt -> " << words.size() << endl;
        }
        else
        {
            cout << "The file couldn't be opened" << endl;
        }
        cout << "Word count took " << (double(t2 - t1) / 1000) 
               << " second(s) with " << thread_count << " threads";
        words.clear();
        std::cin.ignore();
        std::cin.get();
        return 0;
    }
    Last edited by codeprada; 02-10-2011 at 07:31 PM.
    We shouldn't be quick to criticize unless we can do better.
    Code:
    public function __clone() { die ( "I'm one of a kind" ); }

  2. #2
    printf("Hello Cboard\n"); codeprada's Avatar
    Join Date
    Jan 2011
    Location
    In a little room at the back of your brain
    Posts
    68
    just about 9 threads you'll get the best performance and oh yea please change the file name before running it
    We shouldn't be quick to criticize unless we can do better.
    Code:
    public function __clone() { die ( "I'm one of a kind" ); }

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. opening empty file causes access violation
    By trevordunstan in forum C Programming
    Replies: 10
    Last Post: 10-22-2008, 12:19 AM
  2. Formatting a text file...
    By dagorsul in forum C Programming
    Replies: 12
    Last Post: 05-02-2008, 04:53 AM
  3. Formatting the contents of a text file
    By dagorsul in forum C++ Programming
    Replies: 2
    Last Post: 04-29-2008, 01:36 PM
  4. Possible circular definition with singleton objects
    By techrolla in forum C++ Programming
    Replies: 3
    Last Post: 12-26-2004, 10:46 AM
  5. System
    By drdroid in forum C++ Programming
    Replies: 3
    Last Post: 06-28-2002, 11:12 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21