Need Help in program which hangs with large dataset

This is a discussion on Need Help in program which hangs with large dataset within the Tech Board forums, part of the Community Boards category; Hi All, I have a decision tree program which constructs a decision tree. It works fine with various datasets. Now ...

  1. #1
    Registered User
    Join Date
    Apr 2007
    Posts
    5

    Need Help in program which hangs with large dataset

    Hi All,

    I have a decision tree program which constructs a decision tree. It works fine with various datasets. Now i am trying to get this program with microarray datsets (datasets with thousands of attributes). Now this program hangs after some 8-9 hours. I had to wait for that time and after all that wait..it hangs....

    I had tried a similar program in JAVA (weka) with microarray daasets. It works fine if i increase heap memory for jave.exe ....

    I am wondering if theres is a similar thing in C where i can allocate more memory for running this program. Experts please help me.

  2. #2
    Captain - Lover of the C
    Join Date
    May 2005
    Posts
    341
    Are you dynamically allocating the arrays? Dynamically allocated arrays can hold alot more than arrays that have a predetermined size. For instance:
    Code:
    #define MAXSIZEA 10
    #define MAXSIZEB 10
    
    int ArrayA[MAXSIZEA];
    int *ArrayB = malloc(sizeof(ArrayB) * MAXSIZEB);
    You can set MAXSIZEB to a much higher value than MAXSIZEA because MAXSIZEB is the size of the dynamically allocated array.
    Don't quote me on that... ...seriously

  3. #3
    and the hat of wrongness Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,484
    How much memory a program is allowed to have is usually some configuration of the OS, not the program.

    Tell us more about your environment.

    > I am wondering if theres is a similar thing in C
    Do you have such a program, or are you just guessing that a C program would be better, if it existed?
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  4. #4
    Registered User
    Join Date
    Apr 2007
    Posts
    5

    Angry Hi

    Thanks a lot for responding both Brad and Salem.

    I faced a similar problem in java where i ran out of memory and i fixed it by increasing heap size for the executing program

    C:\Program Files\Weka-3-5>
    java -Xmx512m -classpath "C:\Program Files\Java\j2re1.4.2_03\lib\ext\QTJava.zip;weka.ja r" weka.gui.GUIChooser

    Once i launch my program through this command, it succesfully reads large microarray datasets.

    Now coming to C program, it works for many medium datasets and real problem occurs when it faces a large dataset.

    I did some debugging and found my code breaks at this function
    Code:
    //
    // Creates a new row object
    //
    row *createrow ()
    {
        row *newrow;
    
        if ((newrow = (row*)malloc(sizeof(row))) == NULL) {
            printf("ERRORn: Unable to allocate memory!");
            freeitems(item_list);
    		freetestitems(item_list_test);
            freetree(tree);
            exit(1);
        }
        newrow->no = 0;
        newrow->next = NULL;
    
        return newrow;
    }
    After processing the dataset for sometime ...prog stops at this line in my above code
    Code:
     printf("ERRORn: Unable to allocate memory!");
    For smaller datasets it doesnot even come into this if condition....
    Please let me know your thoughts..i am not good in C, Appreciate your help

  5. #5
    and the hat of wrongness Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,484
    Which compiler are you using for C?

    > -Xmx512m
    I'm guessing this means 512 Megabytes.

    As far as I know, the C runtime will continue to allocate memory until either the 2GB limit for user address space is used up, or the real+swap space is used up (whichever is first).
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  6. #6
    Crazy Fool Perspective's Avatar
    Join Date
    Jan 2003
    Location
    Canada
    Posts
    2,640
    I venture a guess that the C program has a memory leak... Check to see that unused memory is being free()ed.

  7. #7
    Registered User
    Join Date
    Apr 2007
    Posts
    5

    hi

    hi Salem,

    Thanks for your response. I am using visualstuido.net for compiling my c file and after that i am running myexe file in command prompt. I have 1 gb ram on windows xp. So i am wondering if i can give all 1 gb for running my exe file...also i am trying to run a dataset with 12000 attributes, so for each attribute the prog store realled info abt them. After reaching a certain number of attributes the program encounters my error code...unable to allocate memory...

    Then i thought if i can give more memory to run the program i cann solve thi sprogram...like the ay i did it for the java program for the same dataset...

    Please let me know your thoughts

    Thanks

  8. #8
    Crazy Fool Perspective's Avatar
    Join Date
    Jan 2003
    Location
    Canada
    Posts
    2,640
    Viswas, you need to understand that there is no -Xmx512m equivalent in C. What that command line option does in Java is signals the java virtual machine to allocate a 512 MB heap for the process about to run (you can also specify the stack size with -Xms256m for example).

    In C there is no virtual machine, the code executes natively and allocates the amount of memory needed within the program.

    If the Java application can load the dataset on your machine, then there is no reason that the C program should fail, other than programming error.

  9. #9
    Registered User
    Join Date
    Apr 2007
    Posts
    5
    Thanks for the response and pointing me in right direction...is there a way i can check for these memory leaks....can u direct me how to fix these problems..actually prog stops at the above code. i am wondering how to get this working(works for small and medium datasets).

    Any tips for fixing is appreciated

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. what do you do to avoid loading large dataset again and again
    By patiobarbecue in forum C++ Programming
    Replies: 12
    Last Post: 04-19-2009, 01:13 AM
  2. Need help with my program...
    By Noah in forum C Programming
    Replies: 2
    Last Post: 03-11-2006, 06:49 PM
  3. Simple copy program hangs?
    By curlious in forum Linux Programming
    Replies: 3
    Last Post: 07-24-2004, 05:00 PM
  4. My program, anyhelp
    By @licomb in forum C Programming
    Replies: 14
    Last Post: 08-14-2001, 10:04 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21