Parallel processing

Hi, I have a plugin program (dll) for another program and in some parts of the program I'd like to introduce multi core processing. I want to learn this. I have read up on some basic concepts and understand where I need it and how it logically would be implemented but don't have much idea how to set it up.

I need a simple as possible little module in my program that I can use to distribute task to different processors where needed inside functions. What do I need? Some library... what is most common? Example code... web resources, tutorials?

I also need the end result to be easially distributed with the plugin, preferrably no funky dependencies at all. The program is windows based only. I'm using Visual Studio 2008 pro.

Thanks

I have read some about CC++ (Compositional C++) in a book from 1995, is that a good thing to use? It's supposed to be an extension to C++ for parallel programming.

What platform? Windows? Linux?
You can use threads. Thousands of example in the internet. Basically, you have the main program, which is the main thread. That main thread calls other threads which run simultaneously. Each thread executes one function. You synchronize them with special functions depending on what library you want to use.

An easiest way is to use OpenMP http://en.wikipedia.org/wiki/OpenMP
With OpenMP you can do things easily. I don't remember the exact syntax, but lets say you want a for loop to run with all processors:

Code:

#pragma omp parallel for parameters
{
for (...)
{..}
}

easy as that, if it is logically correct

Originally Posted by C_ntua

What platform? Windows? Linux?
You can use threads. Thousands of example in the internet. Basically, you have the main program, which is the main thread. That main thread calls other threads which run simultaneously. Each thread executes one function. You synchronize them with special functions depending on what library you want to use.

An easiest way is to use OpenMP http://en.wikipedia.org/wiki/OpenMP
With OpenMP you can do things easily. I don't remember the exact syntax, but lets say you want a for loop to run with all processors:

Code:

#pragma omp parallel for parameters
{
for (...)
{..}
}

easy as that, if it is logically correct

Windows only. Thanks, looks like OpenMP could be what I need, seems really simple. So you mean it automatically splits up the for loop into 4 tasks if I have 4 processors? I'm guessing I will have to deal with a bit more synchronization though. Is it possible with OpenMP to explicitly set one processor to run one function and another to do something different at the same time?

Do I need any special libraries or additions to use this in Visual Studio 2008?

It doesn't do anything automatically for you. You have to make sure the code contains OpenMP directives and such. There should be tutorials...
And OpenMP is completely integrated in VS, so no need for anything extra.

You may also be thrilled to know that Visual Studio 2010 will ship with parallel processing libraries / stuff.

Ok, it seems I am able to access all the OpenMP functions by just including omp.h.

Tried running the example on the wiki site:

Code:

int th_id, nthreads;
   #pragma omp parallel private(th_id)
   {
     th_id = omp_get_thread_num();
     printf("Hello World from thread %d\n", th_id);
     #pragma omp barrier
     if ( th_id == 0 ) {
       nthreads = omp_get_num_threads();
       printf("There are %d threads\n",nthreads);
     }
   }

But it only prints once and says I have 1 thread. Running the function omp_get_max_threads() or omp_get_num_procs() says I have 4 though. Using omp_set_num_threads() doesn't change anything.

Then I turned on OpenMP Support in Properties\C/C++\Language\OpenMP Support and then my program crashes when running the above... Are there any perticular settings I need to use for it to work? The plugin is compiled as a Multi-threaded DLL (/MD).

Thanks Elysia for the tips.

From looking at this tutorial:

http://gamecreator.blogspot.com/2008...asy-multi.html

...it looks like you only need to include the header and enable OpenMP Support, which I did. Could be that the program I have the plugin for is not liking OpenMP for some reason. Will have to look into it.

If you have more tips please share.

You should set the number of processors. You say you do this, but where? If you included it before the #pragma ... like, void omp_set_num_threads(4), then cannot help you. Generally, this functions set the enviromental variable OMP_NUM_THREADS, so maybe a problem there?

I automatically creates threads for the processors I mean. Or something like that. It is much easier than threads, but you have much less control as well.

Probably is indeed some setting. Cannot help you though, have only done this with gcc with the -openmp (or -omp) flag.

If you would follow the example and set the number of cores you want, it should work. Theoretically anyway.
It may be some other unknown factor at work, like race conditions and stuff. OpenMP does not protected you against that.

You do have to turn on OpenMP support in Visual Studio or your #pragma omp directives will be ignored by the compiler. That would explain why you only had 1 thread inside your parallel block. For more information on OpenMP and Visual Studio, see MSDN.

Code:

Could be that the program I have the plugin for is not liking OpenMP for some reason.

OpenMP has nothing special by itself. The preprocessor use the directives to add code to your code to make it multithreaded. But with the information I have, I couldn't tell why your program is crashing at runtime.

Is it possible with OpenMP to explicitly set one processor to run one function and another to do something different at the same time?

Yes, look at the sections directive to achieve task parallelism.

Example code... web resources, tutorials?

I've just started learning parallel processing and IMHO, there just isn't too much useful info on the net concerning parallel processing in general and OpenMP in particular. I did find one useful example as follows:

Code:

// openmp.cpp : Defines the entry point for the console application.
//
#include <stdio.h>
#include <time.h>
#include <float.h>
#include <math.h>
#include <windows.h>
#ifdef _OPENMP
#include <omp.h>
#endif
#define BOOL _Bool
#define true 1
#define false 0

// Matrix size constants
// Be careful to set your shell's stacksize limit to a high value if you
// wish to increase the SIZE.
#define SIZE 160 // Must be a multiple of 8.
#define M SIZE/8
#define N SIZE/4
#define P SIZE/2
#define NTIMES 5 // product matrix calculations

int main(void)
{
    double a[M][N], b[N][P], c[M][P], walltime;
    printf("here\n");

    bool nthr_checked=FALSE;
    time_t start;

    int i, j, k, l, i1, i2, i3, k1, k2, k3, nthr=1;

    printf("Using time() for wall clock time\n");
    printf("Problem size: c(%d,%d) = a(%d,%d) * b(%d,%d)\n",
        M, P, M, N, N, P);
    printf("Calculating product %d time(s)\n", NTIMES);

    // a is identity matrix
    for (i=0; i<M; i++)
        for (j=0; j<N; j++)
            a[i][j] = 1.0;

    // each column of b is the sequence 1,2,...,N
    for (i=0; i<N; i++)
        for (j=0; j<P; j++)
            b[i][j] = i+1.;

    start = time(NULL);

#ifdef _OPENMP
#pragma omp parallel private(i,j,k)
#endif
    {
        for (l=0; l<NTIMES; l++) {
#ifdef _OPENMP
#pragma omp single nowait
#endif
            if (!nthr_checked) {
#ifdef _OPENMP
                nthr = omp_get_num_threads();
#endif
                printf( "\nWe are using %d thread(s)\n", nthr);
                nthr_checked = true;
            }

            // Initialize product matrix
#ifdef _OPENMP
#pragma omp for nowait
#endif
            for (i=0; i<M; i++)
                for (j=0; j<P; j++)
                    c[i][j] = 0.0;

            // Parallelize by row. The threads don't need to synchronize at
            // loop end, so "nowait" can be used.
#ifdef _OPENMP
#pragma omp for nowait
#endif
            for (i=0; i<M; i++) {
                for (k=0; k<N; k++) {
                    // Each element of the product is just the sum 1+2+...+n
                    for (j=0; j<P; j++) {
                        c[i][j] += a[i][k] * b[k][j];
                    }
                }
            }
        } // #pragma omp parallel private(i,j,k)
    } // l=0,...NTIMES-1

    walltime = time(NULL) - start;
    printf("\nFinished calculations.\n");
    printf("Matmul kernel wall clock time = %.2f sec\n", walltime);
    printf("Wall clock time/thread = %.2f sec\n", walltime/nthr);
    printf("MFlops = %f\n",
        (double)(NTIMES)*(double)(N*M*2)*(double)(P)/walltime/1.0e6);

    return 0;
}

I'm not sure where I found the above example. But IMHO, it's a good starting point to learn OpenMP.

Also, the best tutorials I have found so far are located at high performance computing link. I use the training material provided for the introduction to parallel processing and OpenMP. I'll be using this resource until I find something better.

Finally, the last time I checked, the MS Express edition compiler doesn't support parallel processing. You'll need the Professional edition or higher.

You need to indent any code you post.

Thanks for the replies, I figured out what was causing the crash. It was a version of printf that I'm using in the mother program that doesn't like to be called from several threads at the same time. For now it seems like I can start digging into this more.

Thanks.