how to do batch processing with time out?

I have created an algorithm and I need to run it with different datasets, and then record the running time the algorithm took for each datasets.

I understand that shell script is one way to do it. But is it possible to have a time out option? For example, if the algorithm took more than an hour on a dataset and it had not finish running, can I set a time out and make the algorithm to process on the next dataset?

You could run the algorithm as a cronjob, and then send a killall an hour later to kill it. It's not the most elegant solution, but unless you can be interrupted because of a time limit you woulnd't be able to do it.

You could also add the timeout in the actual algorithm. Every time you finish processing a set/piece you could check to see how long it has passed. If more than an hour has passed, abort() or exit() the program.

but cronjob does not have a time out.

i found a code for shellscript on the internet:

Code:

---

my_command & sleep 5
  kill -0 $! && kill $!

---

seems like I can run my algorithm, put it to sleep, and then kill it. but what does the parameters for kill mean? does anyone knows?

Someone posted this code below and I seek help from other forum. You guys know what it means? Sorry, I'm a newbie in shell scipt... =)

Code:

---

#!/bin/bash

# Signal handler for SIGALRM.
#
on_timeout()
{
        echo "Time out! Exit..."
        exit 3
}

# Timer function.
#  - First argument:  PID to send SIGALRM to.
#  - Second argument: time-out in seconds.
#
timeout_timer()
{
        sleep $2
        kill -s SIGALRM $1 >/dev/null 2>&1
}

# Set signal-handler function
#
trap 'on_timeout' ALRM


# Start time-out timer in the background.
# Set time-out to 4 seconds.
#
timeout_timer $$ 4 &

# Main part of the script, which will time out.
# Here: just looping forever, or until time-out
#
SEC=0
while true ; do
        sleep 1
        SEC=$((SEC+1))
        echo "I'm PID# $$, and I'm alive for $SEC seconds now!"
done

# We never get here.

---

Why bother messing around in the shell when you can do it all in your C program.
http://linux.about.com/library/cmd/b..._setitimer.htm
Define an alarm for when you want to stop, and a signal handler to catch it.

Then arrange for your program to exit smoothly rather than just shooting it dead from the outside world.

thanks salem! following your help, i found a website which shows an example on how to use it http://www.quepublishing.com/article...seqNum=14&rl=1

Code:

---

#include <signal.h>
#include <stdio.h>
#include <string.h>
#include <sys/time.h>

void timer_handler (int signum)
{
 static int count = 0;
 printf ("timer expired %d times\n", ++count);
}

int main ()
{
 struct sigaction sa;
 struct itimerval timer;

 /* Install timer_handler as the signal handler for SIGVTALRM. */
 memset (&sa, 0, sizeof (sa));
 sa.sa_handler = &timer_handler;
 sigaction (SIGVTALRM, &sa, NULL);

 /* Configure the timer to expire after 250 msec... */
 timer.it_value.tv_sec = 0;
 timer.it_value.tv_usec = 250000;
 /* ... and every 250 msec after that. */
 timer.it_interval.tv_sec = 0;
 timer.it_interval.tv_usec = 250000;
 /* Start a virtual timer. It counts down whenever this process is
  executing. */
 setitimer (ITIMER_VIRTUAL, &timer, NULL);

 /* Do busy work. */
 while (1);
}

---

but it seems confusing to me, where do I add my code to execute my program? and what happens after the timer expire? below is an exmplae of my code on how i execute batch processing for the multiple datasets. Some of the input_dataset took more than an hour to run, and I need a timer to stop them if they took more than an hour.

Code:

---

for(i=0; i<num_of_input_dataset; i++) {
  sprintf(str_cmd, "./myprogram input_dataset%d", i);
  system(str_cmd);
}

---

Thanks for your help again!

> void timer_handler (int signum)
Well the aggressive answer would be to call say exit() to just drop you out of the program.

A better thing would be to set some global flag which is checked periodically by your main code.

Eg.

Code:

---

volatile int exitFlag = 0;
void timer_handler (int signum)
{
  exitFlag = 1;
}

int main ( ) {
  // stuff you've got
  while ( !exitFlag ) {
  }
}

---

will checking periodically slows down the running time of my program?

Pick your spot in the code - check as often or as little as you want.

If you have say 3 nested loop, then do something like

Code:

---

for ( i = 0 ; i < lim && !exitFlag ; i++ )
  for ( j = 0 ; j < lim ; j++ )
    for ( k = 0 ; k < lim ; k++ )

---

If you want a more responsive exit, then add the test to the inner loops as well.

As for checking time, no it wont (not in any measureable way)

I try using the the example source code that I had posted earlier and incorparate it into my program, but why doesnt it raise the event function timer_handler ?

Sorry guys, I kinda desperate because my deadline is approaching and I can't find any solutions. Any suggestions to improve this code is much appreciate! :D

Code:

---

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <time.h>
#include <sys/timeb.h>
#include <signal.h>
#include <iostream>
#include <sys/time.h>


#define DATA_LOCATION                "/home/codes/expt/graph/"
#define EXEC_LOCATION  "/home/codes/lcm4/"
#define MAX_FILENAME_LEN        200
#define MAX_LINE_LEN        500
#define MAX_COMD_LEN        500

void timer_handler (int signum)
{
 static int count = 0;
 char str_cmd[MAX_COMD_LEN];
 printf ("timer expired %d times\n", ++count);
 sprintf(str_cmd, "kill -0 $! && kill $!");
 system(str_cmd);
}

void LCM(char* dataset_name, int start_min_sup, int iterations, int inc_step, bool bout, int tnum)
{
  FILE* fpsum_file;
  char str_cmd[MAX_COMD_LEN];
  double running_time, total_time;
  int ntotal_pats, nmin_sup, i;
  struct timeb start, end, end1;

  fpsum_file = fopen("aaLCMtransaction.time.sum.txt", "at");
  if(fpsum_file==NULL)
    {
      printf("Error: cannot open file afopt.time.sum.txt for write\n");
      exit(-1);
    }

  if(fpsum_file==NULL)
    {
      printf("Error: cannot open file afopt.time.sum.txt for write\n");
      exit(-1);
    }

  for(i=0;i<iterations;i++)
    {
      nmin_sup = i*inc_step + start_min_sup;
      if(bout) {
        sprintf(str_cmd, "%slcm C %s%s %d ", EXEC_LOCATION, DATA_LOCATION, dataset_name, nmin_sup);
     
        printf("%s\n", str_cmd);
       
        ftime(&start);
        ntotal_pats = system(str_cmd);
        ftime(&end);
        total_time = running_time = end.time-start.time+(double)(end.millitm-start.millitm)/1000;
     
        fprintf(fpsum_file, "LCM\t");
        fprintf(fpsum_file, "%s\t", dataset_name);
        fprintf(fpsum_file, "%d\t", nmin_sup);
        //        fprintf(fpsum_file, "%d\t", ntotal_pats);
        fprintf(fpsum_file, "%f\n", running_time);

      }
    }

  fclose(fpsum_file);
}



void BatchExec(char* config_filename)
{
  FILE* fpconfig_file;
  char str_cmd[MAX_COMD_LEN], str_out_flag[10], dataset_name[MAX_FILENAME_LEN];
  int nmin_sup, inc_step, iteration, ntype, tnum;
  bool bout;
  char ch;

  fpconfig_file = fopen(config_filename, "rt");
  if(fpconfig_file==NULL)
    {
      printf("Erorr: cannot open file %s for read\n", config_filename);
      exit(-1);
    }

  fscanf(fpconfig_file, "%s ", str_cmd);
  while(!feof(fpconfig_file))
    {
      if(str_cmd[0]=='#')
        {
          ch = getc(fpconfig_file);
          while(!feof(fpconfig_file) && ch!='\n')
            ch = getc(fpconfig_file);
          fscanf(fpconfig_file, "%s\n", str_cmd);
          continue;
        }
     
      fscanf(fpconfig_file, "%s ", dataset_name);
      fscanf(fpconfig_file, "%d", &nmin_sup);
      fscanf(fpconfig_file, "%d", &iteration);
      fscanf(fpconfig_file, "%d", &inc_step);
      fscanf(fpconfig_file, "%s", str_out_flag);
      fscanf(fpconfig_file, "%d", &tnum);

      if(!strcmp(str_out_flag, "true"))
        bout = true;
      else
        bout = false;
     
      if(!strcmp(str_cmd, "lcm"))
        LCM(dataset_name, nmin_sup, iteration, inc_step, bout, tnum);
       
               
      fscanf(fpconfig_file, "%s ", str_cmd);
    }

  fclose(fpconfig_file);
}




int main(int argc, char* argv[])
{
  struct sigaction sa;
  struct itimerval timer;
  /* Install timer_handler as the signal handler for SIGVTALRM. */
  memset (&sa, 0, sizeof (sa));
  sa.sa_handler = &timer_handler;
  sigaction (SIGVTALRM, &sa, NULL);
 
  /* Configure the timer to expire after 250 msec... */
  timer.it_value.tv_sec = 0;
  timer.it_value.tv_usec = 250000;
  /* ... and every 250 msec after that. */
  timer.it_interval.tv_sec = 0;
  timer.it_interval.tv_usec = 250000;
  /* Start a virtual timer. It counts down whenever this process is
    executing. */
  setitimer (ITIMER_VIRTUAL, &timer, NULL);

  while (1) {
  if(argc==2)
    BatchExec(argv[1]);
  }

 
  return 0;
}

---

I give you a perfectly workable idea, and you retreat to your old hacks.

Here, read and digest.

Then choose whether you want x amount of CPU time or whether you want to watch the clock on the wall.

Code:

---

#include <signal.h>
#include <stdio.h>
#include <string.h>
#include <sys/time.h>
#include <unistd.h>

volatile int isExit = 0;
void timer_handler (int signum)
{
  isExit = signum;
}

#if defined(WORK_NOT_SLEEP)
#define FOO_SIGNAL  SIGVTALRM
#define FOO_TIMER  ITIMER_VIRTUAL
#else
#define FOO_SIGNAL  SIGALRM
#define FOO_TIMER  ITIMER_REAL
#endif

int main ()
{
  int count = 0;
  struct sigaction sa;
  struct itimerval timer, otimer;

  /* Install timer_handler as the signal handler for SIGVTALRM. */
  memset (&sa, 0, sizeof (sa));
  sa.sa_handler = timer_handler;
  if ( sigaction (FOO_SIGNAL, &sa, NULL) != 0 ) {
    perror("action");
  }

  /* Configure the timer to expire after 5 sec ... */
  timer.it_value.tv_sec = 5;
  timer.it_value.tv_usec = 0;
  timer.it_interval.tv_sec = 0; /* we only need the one */
  timer.it_interval.tv_usec = 0;

  /* Start a virtual timer. It counts down whenever this process is
  executing. */
  /* ITIMER_REAL for the wall clock */
  /* ITIMER_VIRTUAL for CPU time. */
  /* -- Doesn't work too well if process mostly sleeps */
  if ( setitimer (FOO_TIMER, &timer, &otimer) != 0 ) {
    perror("timer");
  }

  /* Do busy work. */
  while ( !isExit ) {
    printf ( "%03d", count++ );
    fflush ( stdout );
#if defined(WORK_NOT_SLEEP)
    {
      /* approx 1 second of work on my box */
      volatile int loop;
      for ( loop = 0 ; loop < 1E8 ; loop++ );
    }
#else
    sleep ( 1 );
#endif
    printf ( "\b\b\b" );
    fflush ( stdout );
  }
  printf ( "\nTime expired - program exiting\n" );
  return 0;
}


My results, with both timer types
$ gcc -W -Wall -DWORK_NOT_SLEEP  foo.c
$ ./a.out
006
Time expired - program exiting
$ gcc -W -Wall  foo.c
$ ./a.out
004
Time expired - program exiting

---

Oh, and while you're at it, read about feof() and control loops in the FAQ.

Signal handlers are more like interrupts rather than function calls. You can't willy nilly run any bit of code you like inside signal handlers without causing grief.

Dear Salem, thanks for your help! You have been very patient with me and I really appreciate that. :D

But the code that you provided has the same problem as mine. I edited some parts of your code shown below

Code:

---

/* Do busy work. */
  while ( !isExit ) {

    sprintf(str_cmd, "./myalgo inputdataset");  // inserted to do work
    system(str_cmd);                                        // inserted to do work

    printf ( "%03d", count++ );
    fflush ( stdout );
#if defined(WORK_NOT_SLEEP)
    {
      /* approx 1 second of work on my box */
      volatile int loop;
      for ( loop = 0 ; loop < 1E8 ; loop++ );
    }
#else
    sleep ( 1 );
#endif
    printf ( "\b\b\b" );
    fflush ( stdout );
  }

---

The problem function system calls myalgo to run and it will put the current program to sleep. Since the current program is the one that is doing the checking for timeout, and it is sleeping, the timeout will not be called.

Is there a way not to make it sleep?

sprintf(str_cmd, "./myalgo inputdataset"); // inserted to do work
system(str_cmd); // inserted to do work


Do you have the source code for myalgo, and can you edit it / rebuild it ?

If you can, then the timer code goes in there, not in any "shell" program you use to make it run with whatever data sets you have.

Besides, using 'C' just to string together a few system() calls isn't really the best approach.

I have got 3 algorithms for the experimentation comparisons. I do not have the source code for one of them.

Since using C is not possible, maybe shell scripting is possible?

Mmm, perhaps a different approach then

Code:

---

pid_t pid;
pid = fork ( );
if ( pid == 0 ) {
  char *args[10];
  args[0] = "./myprog";
  args[1] = "filename";
  args[2] = NULL;
  execvp( args[0], args );
} else {
  int i, status;
  for ( i = 0 ; i < 20 ; i++ ) {
    sleep ( 1 );
    if ( waitpid(pid,&status,WNOHANG) == pid ) { /* better checks */
      /* child has ended */
      break;
    }
  }
  if ( i == 20 ) {
    kill ( pid, SIGQUIT );  /* or whatever signal you want */
  }
}

---

I tried it, but it didn't work.... :confused:

Code:

---

#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>

main( int argc, char *argv[] )
{
 
 
    pid_t pid;
    pid = fork ( );
    if ( pid == 0 ) {
      char *args[10];
      args[0] = "./myprog";
      args[1] = "filename";
      args[2] = NULL;
      execvp( args[0], args );
    } else {
      int i, status;
      for ( i = 0 ; i < 20 ; i++ ) {
        sleep ( 1 );
        if ( waitpid(pid,&status,WNOHANG) == pid ) { /* better checks */
          /* child has ended */
          break;
        }
      }
      if ( i == 20 ) {
        kill ( pid, SIGQUIT );  /* or whatever signal you want */
      }
 
  }
}

---

args[0] = "./myprog";
args[1] = "filename";
You did actually substitute your real command and filenames here.

Also, check to see if exec() actually returns, which it will do if it can't run the program.