Need Help in program which hangs with large dataset

[viswas]

Hi All,

I have a decision tree program which constructs a decision tree. It works fine with various datasets. Now i am trying to get this program with microarray datsets (datasets with thousands of attributes). Now this program hangs after some 8-9 hours. I had to wait for that time and after all that wait..it hangs....

I had tried a similar program in JAVA (weka) with microarray daasets. It works fine if i increase heap memory for jave.exe ....

I am wondering if theres is a similar thing in C where i can allocate more memory for running this program. Experts please help me.

[Brad0407]

Are you dynamically allocating the arrays? Dynamically allocated arrays can hold alot more than arrays that have a predetermined size. For instance:

#define MAXSIZEA 10
#define MAXSIZEB 10

int ArrayA[MAXSIZEA];
int *ArrayB = malloc(sizeof(ArrayB) * MAXSIZEB);

You can set MAXSIZEB to a much higher value than MAXSIZEA because MAXSIZEB is the size of the dynamically allocated array.

[Salem]

How much memory a program is allowed to have is usually some configuration of the OS, not the program.

Tell us more about your environment.

> I am wondering if theres is a similar thing in C
Do you have such a program, or are you just guessing that a C program would be better, if it existed?

[viswas]

Thanks a lot for responding both Brad and Salem.

I faced a similar problem in java where i ran out of memory and i fixed it by increasing heap size for the executing program

C:\Program Files\Weka-3-5>
java -Xmx512m -classpath "C:\Program Files\Java\j2re1.4.2_03\lib\ext\QTJava.zip;weka.ja r" weka.gui.GUIChooser

Once i launch my program through this command, it succesfully reads large microarray datasets.

Now coming to C program, it works for many medium datasets and real problem occurs when it faces a large dataset.

I did some debugging and found my code breaks at this function

//
// Creates a new row object
//
row *createrow ()
{
    row *newrow;

    if ((newrow = (row*)malloc(sizeof(row))) == NULL) {
        printf("ERRORn: Unable to allocate memory!");
        freeitems(item_list);
		freetestitems(item_list_test);
        freetree(tree);
        exit(1);
    }
    newrow->no = 0;
    newrow->next = NULL;

    return newrow;
}

After processing the dataset for sometime ...prog stops at this line in my above code

 printf("ERRORn: Unable to allocate memory!");

For smaller datasets it doesnot even come into this if condition....
Please let me know your thoughts..i am not good in C, Appreciate your help

[Salem]

Which compiler are you using for C?

> -Xmx512m
I'm guessing this means 512 Megabytes.

As far as I know, the C runtime will continue to allocate memory until either the 2GB limit for user address space is used up, or the real+swap space is used up (whichever is first).

[Perspective]

I venture a guess that the C program has a memory leak... Check to see that unused memory is being free()ed.

[viswas]

hi Salem,

Thanks for your response. I am using visualstuido.net for compiling my c file and after that i am running myexe file in command prompt. I have 1 gb ram on windows xp. So i am wondering if i can give all 1 gb for running my exe file...also i am trying to run a dataset with 12000 attributes, so for each attribute the prog store realled info abt them. After reaching a certain number of attributes the program encounters my error code...unable to allocate memory...

Then i thought if i can give more memory to run the program i cann solve thi sprogram...like the ay i did it for the java program for the same dataset...

Please let me know your thoughts

Thanks

[Perspective]

Viswas, you need to understand that there is no -Xmx512m equivalent in C. What that command line option does in Java is signals the java virtual machine to allocate a 512 MB heap for the process about to run (you can also specify the stack size with -Xms256m for example).

In C there is no virtual machine, the code executes natively and allocates the amount of memory needed within the program.

If the Java application can load the dataset on your machine, then there is no reason that the C program should fail, other than programming error.

[viswas]

Thanks for the response and pointing me in right direction...is there a way i can check for these memory leaks....can u direct me how to fix these problems..actually prog stops at the above code. i am wondering how to get this working(works for small and medium datasets).

Any tips for fixing is appreciated