Best way to multithread a loop?

This is a discussion on Best way to multithread a loop? within the C Programming forums, part of the General Programming Boards category; I have a for loop that reads an image, converts it, and then writes it to disk. This loop runs ...

  1. #1
    Registered User
    Join Date
    Jun 2009
    Posts
    97

    Best way to multithread a loop?

    I have a for loop that reads an image, converts it, and then writes it to disk. This loop runs in its own thread, and it works fine. However, I would like split this up so I can run this loop in say, the main thread and then spawn multiple threads to do the dirty work. This way, I can use multiple CPU cores to convert the images (since it would then be able to process as many images simultaneously as the number of cores in the machine).

    I am not quite sure what the best way is to do this. Should I create the loop in the main/separate thread, then spawn threads inside the loop up to the maximum number of CPU cores? What would be the best way to keep track of how many threads were already running? Just set a var that is incremented upon the thread starting, then decremented when it finishes, locking the latest thread if the max number of threads are already running?

    Thanks for any help!

  2. #2
    Captain Crash brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,158
    The best way is probably to spawn a pool of image conversion worker threads. You also create a thread-safe queue of conversion requests. The main thread will read image names and place the conversion requests in the queue. The worker threads will remove conversion requests from the queue and carry out the conversions.

    When all the images have been queued, place special commands on the queue which tell the worker threads to terminate. Then join with all the worker threads.

    Pseudocode:

    Code:
    imageQueue = Queue()
    workerThreads = []
    for n in range(0, numThreads):
        workerThreads.append( SpawnThread( WorkerThread, imageQueue ) )
    for img in images:
        imageQueue.Enqueue( img )
    for n in range(0, numThreads):
        imageQueue.Enqueue( QuitCommand )
    for n in range(0, numThreads):
        Join( workerThreads[n] )
    
    def WorkerThread( imageQueue ):
        while True:
            command = imageQueue.Dequeue()
            if command is QuitCommand:
                break
            ConvertImage( command )
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  3. #3
    and the hat of wrongness Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,334
    > I have a for loop that reads an image, converts it, and then writes it to disk.
    Unless your conversion is substantially slower than the time to read/write to disk, all your optimisation will achieve is multiple cores all waiting for the disk at the same time.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  4. #4
    Registered User
    Join Date
    Jun 2009
    Posts
    97
    Quote Originally Posted by Salem View Post
    > I have a for loop that reads an image, converts it, and then writes it to disk.
    Unless your conversion is substantially slower than the time to read/write to disk, all your optimisation will achieve is multiple cores all waiting for the disk at the same time.
    Yes, I should have mentioned that the machine is connected to a FC SAN with very high I/O throughput. Otherwise, the disk would be the bottleneck as you mentioned.

  5. #5
    Algorithm Dissector iMalc's Avatar
    Join Date
    Dec 2005
    Location
    New Zealand
    Posts
    6,261
    What kind of image conversion are you doing?
    The method of parallelising your conversions depends on whether you're doing something as simple as converting a TIFF file to greyscale, or something more complex such as converting a GIF to a JPEG.
    My homepage
    Advice: Take only as directed - If symptoms persist, please see your debugger

    Linus Torvalds: "But it clearly is the only right way. The fact that everybody else does it some other way only means that they are wrong"

  6. #6
    Registered User
    Join Date
    Jun 2009
    Posts
    97
    Quote Originally Posted by iMalc View Post
    What kind of image conversion are you doing?
    The method of parallelising your conversions depends on whether you're doing something as simple as converting a TIFF file to greyscale, or something more complex such as converting a GIF to a JPEG.
    I'm converting video files in YUV space to RGB images. The input is a video file, and the output is a BMP or TIFF, whatever. I have a loop that reads each frame of video and puts it into a struct, which I then pass to a function that does the actual conversion. I was thinking I could spawn this conversion function as multiple threads until reaching the machine's CPU core limit.

    Also, I am doing this in Objective C, which I don't know well. With pthreads, you pass params as a struct, but here I am using NSThread, which requires parameters be passed an array of Objective C objects. However, I need to pass pointers, which I can't get to work with the NSArray or NSData objects since you must "wrap" your pointer in an object.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Poll event loop
    By rogster001 in forum C++ Programming
    Replies: 2
    Last Post: 09-17-2009, 04:28 AM
  2. need help with a loop
    By Darkw1sh in forum C Programming
    Replies: 19
    Last Post: 09-13-2009, 09:46 PM
  3. funny-looking while loop
    By Aisthesis in forum C++ Programming
    Replies: 3
    Last Post: 08-30-2009, 11:54 PM
  4. nested loop, simple but i'm missing it
    By big_brother in forum C Programming
    Replies: 19
    Last Post: 10-23-2006, 10:21 PM
  5. loop issues
    By kristy in forum C Programming
    Replies: 3
    Last Post: 03-05-2005, 08:14 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21