How difficult is parallel programming in C?

I know some graduate physics students who are writing research programs in C. These programs involve massive calculations.

I am trying to choose a configuration for a new workstation for these students. They tell me, however, that they don't want a multicore processor because it is too difficult to write a program utilizing more than one processor, they don't want to write parallel.

This seems like a problem to me because I don't know any workstation config that doesn't use multiple processors. Is it really that hard to write a C program to utilize 2, 4 or 8 processors? (I've never tired it)

Thanks

Owning a multi-core processor, and writing code to utilize more than one core are unrelated. A single threaded C/C++ program will always run on just one core, regardless of how many cores you have.

The OS and other applications may appreciate the extra cores, even if the programmers don't.

gg

The C language, as far as I know, doesn't have any statement or anything that can help you learn parallel programming.

That does not mean you can't do parallel computing from C, but you have to use a library, for example. Useful keywords : threads, OpenMP, CUDA, etc. CUDA could be fun I guess.

Or maybe start there :
Parallel programming model - Wikipedia, the free encyclopedia

These are physics students, they really don't want to get into heavy programming. And I don't know if I have the time to help them.

They do run some of these programs on supercomputers, but they have someone there to utilize the 100+ parallel processors there.

I'll at least take a look at some libraries.

One thing to keep in mind: If their application is doing a bunch of calculations where each calculation relies on the result of the previous calculation, it can be pretty tough to refactor it such that parrallel programming provides any performance gains.

Parallel programming can both be easy and difficult.
If everything is done by its own, all different tasks, so to speak, then it's rather straight-forward. But if calculations depends on other calculations which depends on some other calculation, then it's going to grow complex pretty fast because you have to synchronize.
And that opens up the possibility of bugs and race conditions which are hard to debug.

Apparently at the supercomputer center there were lots of experts helping them run their programs on multiple processors, plus they used libraries. But they want to do some work on their own on a local workstation.

Also their professor was complaining that sometimes those special libraries were not adequate all the time.

Like I said before, I'll do some snooping around (I've done some programming before) but I think it will be too hard for those students to do it on their own.

If all you want is some speedup from the cores, and not necessarily a certain amount of speedup in a certain way, you might want to use an auto-parallelizing compiler. I don't think GCC has that option, but Sun's SunCC does. Check out the -xautopar option.

>> Also their professor was complaining that sometimes those special libraries were not adequate all the time.

Well, threading libraries are fairly standard (eg: pthreads) so I wouldn't necessarily call them "special". If they really can't handle it, though, I'd look into robwhit's suggestion.

Is there a list of stuff to follow, like use const etc

Most primitive kind of parallel processing, without needing to change the program - use the OS scheduler!

Can they run multiple instances of their program to work on different problems? (or different parts of a problem) Can they split the input into independent sections?

Yeah, OK. These physics students want to run a lot of calculations, but have a "it's just software" mentality and don't understand the mechanics of running their software

If the only hardware they can obtain has a multi-core processor the software will still only execute on one processor/core unless it is already designed in parts that can be run in parallel. That will mean one processor/core flat out, and the others idling. The output will be the same. If it isn't that means the software is broken in some way.

Other things they need to consider (some of which others have already said in this thread, some not) are;

1) The best opportunities for parallelising will come when there are no inter-dependencies between sets of calculations. If any of the output from run A needs to be fed input into run B, the opportunities for parallelising run A and run B are constrained.

2) A lot of physics-based simulations dump output at the end of a run, so that means no opportunity for parallelisation.

3) Physicists are required to actually understand the physics. If they do they should be able to describe the process that they are simulating on paper, and understand where the interdependencies of the process are. That bit of paper may be large, but it will probably be the best thing they can provide to the software specialists who implement the code.

4) If physicists are worrying about executing code on their own computer, that probably means they have limited understanding of the physics. Building that understanding would be the best application of their time. If they understand the physics, they can better describe things so the simulation can be implemented in software - or they can solve a problem another way.

5) I've seen horror stories in situations like this. For example, one physics PhD student spending all his time coordinating with computing centres for spare processing cycles, and setting up batch jobs to run the program with select inputs on different supercomputers. After receiving (literally!) truck loads of output, he noticed some anomalies in outputs but didn't understand them. Six months later (with more runs on the various supercomputers) he isolated the problem to a bug in some code he had modified. He then had to fix the code and run most of the computations again from scratch. His PhD project turned out to take seven years.

I've discussed this issue with the professor, and have been told the main graduate student in discussion is quite competent in writing code. So apparently he can write multithreaded code, but doesn't want to in this situation. (They write their own code, they don't have THAT much money). I've heard them complain a great deal about the difficulties they are have been running into over the years working with parallel systems at supercomputer centers (even with people helping them). Maybe they are facing some of the difficulties you guys are pointing out.

I've explained the inefficiency that running a single thread program would cause, particularly with the basic hardware designs they have today, and that it would be so much faster multi-threaded, but I'm being ignored. So hopefully they know what they are doing.

I will bring up all your points to them, however. It might help.

I'm guessing the programs will run rather slow (on just one of the 4 cores in a quad core processor, and not even using the other processor) but will have access to a huge amount of memory for neccessary data. Also it is possible some other physics students might be new at this and would want to avoid writing parallel code.

I too can see all kinds of trouble a physics programs can run into, considering how gigantic their calculations are. There are 'soft errors' caused by cosmic rays flipping bits, one or two errors a month. Then there are round off errors caused by that crazy 'floating point representation' in computer storage. And of course the infamous bug in the code. I am a bit concerned about bugs, since they are physicists first, programmers second, and even dedicated programmers have all kinds of problems with bugs.

I'm trying to design the computer workstation to take into account these problems (except for bugs), but this is my first workstation! Wish me luck.

Quote:

---

Originally Posted by darsunt

Then there are round off errors caused by that crazy 'floating point representation' in computer storage.

---

This is true, and frankly, the natives types are unless IMHO.
Thankfully, there are libraries out there with unlimited precision. But it comes at the price of extra computing power, of course.

Well, good luck anyway. Parallelism is not easy.

Quote:

---

Originally Posted by darsunt

I've discussed this issue with the professor, and have been told the main graduate student in discussion is quite competent in writing code. So apparently he can write multithreaded code, but doesn't want to in this situation. (They write their own code, they don't have THAT much money). I've heard them complain a great deal about the difficulties they are have been running into over the years working with parallel systems at supercomputer centers (even with people helping them). Maybe they are facing some of the difficulties you guys are pointing out.

---

I think you are little hardware crazied with your feeling that a single core processor is wrong to use no matter what, but here's a thought in support of your idea: They can write single threaded programs and run them in parallel to greater effect on a multi core system, since if you run two labour intensive things at the same time, the OS will give them a processor each. Unless there is only one thing they actually ever need to do at one time, which seems unlikely.

Quote:

---

I too can see all kinds of trouble a physics programs can run into, considering how gigantic their calculations are. There are 'soft errors' caused by cosmic rays flipping bits, one or two errors a month.

---

Wow. I guess nothing will prevent that but it is worth noting that the other advantage to running single threaded apps on a multi-core system is that the OS will always have resources available to it if they are leaving something running for weeks, which might make it all more stable.

>> I guess nothing will prevent that...
There's ECC memory - Dynamic random access memory - Wikipedia, the free encyclopedia

gg

Yes I hope to get hardware configured to run ECC memory. It will cause a performance hit, however. Also ECC memory is not perfect, some soft errors will probably slip through. If a cosmic ray particle flips 2 bits (unlikely but possible) ECC can't fix it, won't even know what happened. That could be enough to throw such huge calculations completely off.

My concern about using a single processor is that these programs are HUGE. One program will easily use up all memory resources (assuming I can find an op system that allows large process size). So they will be running one program at a time. Also the initial request they made to me was for speed, which is out the window if they use 1 out of 4 or 1 out of 8 cores.

The other cores will help cause the op system won't interrupt the main program running, but not a lot. However, again they said that is what they want.

Any 64-bit OS will allow you to use a lot of memory.
Also, have you considered that if you can't run a single calculation on multiple cores, that you can actually run several calculations simultaneously?

darsunt, I consider you're getting bedazzled with one aspect of the problem: the hardware.

The hardware is actually the cheapest factor in this whole setting. If the problems are able to be split across four cores, they are able to be split across four computers. And the cost of four computers is probably less than the cost of rearchitecting the software for parallelism. And make no mistake: rearchitecting a working software design so it can be run in parallel is a massive rearchitecture. The incidence of systematic errors (i.e. programmer errors) increases substantially with complexity of the system design and with the size of the code base - and parallelism increases both design complexity and code size. One practical rule of thumb: system rearchitecture is an extremely effective method to produce an expensive and unreliable system from a working system.

The hardware will also be routinely replaced. Parallelism, on the other hand, increases both the initial and maintenance cost of the software - because it increases the complexity of the software, making it harder to maintain. "Bang for buck" of hardware will keep going up (even if the software is not written specifically to exploit multiple cores) but that will not necessarily be true with a new software design.

I consider the concerns raised in this thread with floating point precision are irrelevant. The mathematics of floating point is not trivial, but nor is it a show stopper for large scale numerical calculations if it is accounted for in the software - specifically algorithm - design and when interpreting the program outputs. That's true whether the software is parallelised or not.

Similarly, the point about soft errors is irrelevant - even if we ignoring the fact that cosmic rays are only one contributor. Soft errors is a probabilistic effect that needs to be detected (if possible) and accounted for but - if anything - parallel software will be more likely to be adversely affected because parallel software involves use of additional hardware recources (eg memory) for synchronisation. That means a need for more machinery - software and hardware overhead - to detect the impacts following on from soft errors.

What I suspect these guys are looking for from you is optimisation of the code they have - i.e. fine tuning the algorithms - rather than a complete system rearchitecture. In terms of return on investment, I suspect they're right.

how can i Marge two .exe file together and how they can run both r simultaneously...
please replay me as soon as possible..
thanks...

"as soon as possible"