Best way to multithread a loop?

**synthetix** · 04-08-2010

I have a for loop that reads an image, converts it, and then writes it to disk. This loop runs in its own thread, and it works fine. However, I would like split this up so I can run this loop in say, the main thread and then spawn multiple threads to do the dirty work. This way, I can use multiple CPU cores to convert the images (since it would then be able to process as many images simultaneously as the number of cores in the machine).

I am not quite sure what the best way is to do this. Should I create the loop in the main/separate thread, then spawn threads inside the loop up to the maximum number of CPU cores? What would be the best way to keep track of how many threads were already running? Just set a var that is incremented upon the thread starting, then decremented when it finishes, locking the latest thread if the max number of threads are already running?

Thanks for any help!

**brewbuck** · 04-08-2010

The best way is probably to spawn a pool of image conversion worker threads. You also create a thread-safe queue of conversion requests. The main thread will read image names and place the conversion requests in the queue. The worker threads will remove conversion requests from the queue and carry out the conversions.

When all the images have been queued, place special commands on the queue which tell the worker threads to terminate. Then join with all the worker threads.

Pseudocode:

Code:

imageQueue = Queue()
workerThreads = []
for n in range(0, numThreads):
    workerThreads.append( SpawnThread( WorkerThread, imageQueue ) )
for img in images:
    imageQueue.Enqueue( img )
for n in range(0, numThreads):
    imageQueue.Enqueue( QuitCommand )
for n in range(0, numThreads):
    Join( workerThreads[n] )

def WorkerThread( imageQueue ):
    while True:
        command = imageQueue.Dequeue()
        if command is QuitCommand:
            break
        ConvertImage( command )

**Salem** · 04-08-2010

> I have a for loop that reads an image, converts it, and then writes it to disk.
Unless your conversion is substantially slower than the time to read/write to disk, all your optimisation will achieve is multiple cores all waiting for the disk at the same time.

**synthetix** · 04-08-2010

Originally Posted by Salem

> I have a for loop that reads an image, converts it, and then writes it to disk.
Unless your conversion is substantially slower than the time to read/write to disk, all your optimisation will achieve is multiple cores all waiting for the disk at the same time.

Yes, I should have mentioned that the machine is connected to a FC SAN with very high I/O throughput. Otherwise, the disk would be the bottleneck as you mentioned.

**iMalc** · 04-09-2010

What kind of image conversion are you doing?
The method of parallelising your conversions depends on whether you're doing something as simple as converting a TIFF file to greyscale, or something more complex such as converting a GIF to a JPEG.

**synthetix** · 04-09-2010

Originally Posted by iMalc

What kind of image conversion are you doing?
The method of parallelising your conversions depends on whether you're doing something as simple as converting a TIFF file to greyscale, or something more complex such as converting a GIF to a JPEG.

I'm converting video files in YUV space to RGB images. The input is a video file, and the output is a BMP or TIFF, whatever. I have a loop that reads each frame of video and puts it into a struct, which I then pass to a function that does the actual conversion. I was thinking I could spawn this conversion function as multiple threads until reaching the machine's CPU core limit.

Also, I am doing this in Objective C, which I don't know well. With pthreads, you pass params as a struct, but here I am using NSThread, which requires parameters be passed an array of Objective C objects. However, I need to pass pointers, which I can't get to work with the NSArray or NSData objects since you must "wrap" your pointer in an object.

Thread: Best way to multithread a loop?

Thread Tools

Search Thread

Display

Best way to multithread a loop?

Similar Threads

Poll event loop

need help with a loop

funny-looking while loop

nested loop, simple but i'm missing it

loop issues