Thread: Program to run other processes

  1. #1
    Old Fashioned
    Join Date
    Nov 2016
    Posts
    136

    Post Program to run other processes

    I've not worked with execve(), fork(), etc... in C yet. In fact, I'm not even sure if they are the correct functions for the job.

    I am going to describe what I need to do and I'd appreciate some guidance from the more experienced.

    I have a commandline program which is written in Python. It does not support multithreading but it is an extremely process-intensive program that involves searching through about 16GB of text files and performing CPU intensive calculations.

    I need to parse some data from a few text files and feed this data into the python program as commandline arguments.

    Problem #1: Why am I not just doing it in Python? Because Python doesn't really support multithreading, and I know C better.

    Problem #2: Obviously, the Python engine will still be doing the processing, so I'm not sure that this C program I write will actually speed anything up. However, what I've been doing is manually running like 6-7 instances of Python in order to achieve something similar to multiprocessing/multithreading and this will at least save me that hassle.

    So what I want to do is have a C program fire up these python programs in separate processes for me, make sure to only run like 6 of them at a time, wait for some to finish, and then fire up new processes again until the list of input data from the text files is exhausted.

    There really doesn't need to be any "shared data" as each separate instance of the processes will be fed a separate data file.

    High-level advice appreciated. Thanks.
    If I was homeless and jobless, I would take my laptop to a wifi source and write C for fun all day. It's the same thing I enjoy now!

  2. #2
    C++ Witch laserlight's Avatar
    Join Date
    Oct 2003
    Location
    Singapore
    Posts
    27,550
    Quote Originally Posted by Asymptotic
    Problem #1: Why am I not just doing it in Python? Because Python doesn't really support multithreading, and I know C better.
    Python does support multithreading, but the global interpreter lock thing in CPython means there may be serious limitations on how useful multithreading in Python might be for you. On the other hand, Python also supports multiple processes, which is what execve and fork in C would do, and this could fit the bill very well since you mention that there's no shared data.
    Quote Originally Posted by Bjarne Stroustrup (2000-10-14)
    I get maybe two dozen requests for help with some sort of programming or design problem every day. Most have more sense than to send me hundreds of lines of code. If they do, I ask them to find the smallest example that exhibits the problem and send me that. Mostly, they then find the error themselves. "Finding the smallest program that demonstrates the error" is a powerful debugging tool.
    Look up a C++ Reference and learn How To Ask Questions The Smart Way

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. ++k and k++ are different processes, or not?
    By hefese in forum C++ Programming
    Replies: 2
    Last Post: 11-10-2013, 07:55 AM
  2. Parent and child processes program variables
    By tfarmer4 in forum Linux Programming
    Replies: 1
    Last Post: 03-01-2011, 05:36 PM
  3. Help for C program using processes and signals
    By enimas in forum Linux Programming
    Replies: 2
    Last Post: 12-14-2010, 11:43 AM
  4. Simple Processes Program
    By khdani in forum Linux Programming
    Replies: 9
    Last Post: 11-18-2008, 10:59 AM
  5. Replies: 5
    Last Post: 04-17-2003, 07:07 AM

Tags for this Thread