Program to run other processes
I've not worked with execve(), fork(), etc... in C yet. In fact, I'm not even sure if they are the correct functions for the job.
I am going to describe what I need to do and I'd appreciate some guidance from the more experienced.
I have a commandline program which is written in Python. It does not support multithreading but it is an extremely process-intensive program that involves searching through about 16GB of text files and performing CPU intensive calculations.
I need to parse some data from a few text files and feed this data into the python program as commandline arguments.
Problem #1: Why am I not just doing it in Python? Because Python doesn't really support multithreading, and I know C better.
Problem #2: Obviously, the Python engine will still be doing the processing, so I'm not sure that this C program I write will actually speed anything up. However, what I've been doing is manually running like 6-7 instances of Python in order to achieve something similar to multiprocessing/multithreading and this will at least save me that hassle.
So what I want to do is have a C program fire up these python programs in separate processes for me, make sure to only run like 6 of them at a time, wait for some to finish, and then fire up new processes again until the list of input data from the text files is exhausted.
There really doesn't need to be any "shared data" as each separate instance of the processes will be fed a separate data file.
High-level advice appreciated. Thanks.
Originally Posted by Asymptotic
