Thread: A question about memory allocation in Linux

  1. #1
    Registered User
    Join Date
    Aug 2009
    Posts
    1

    A question about memory allocation in Linux

    I have a programming question and hope to get your suggestions. I have a C++ program that reads files.

    From a preprocessing program, I know the size of the file and how much memory is needed.

    Step 1: I get the memory needed from reading a small configuration file and allocate the memory by calling new(). The total mem is around 800M since it is a large network.

    Step 2: I read the data file, process the data and write them to the memory.

    However, the program is very slow during the reading file and filling the data. The whole machine has 4G mem. It is CentOS and program is in C++, compiled by g++.

    I tried reading the file only, doing simple operations and not writing them into the mem. It is fast. Seems the memory allocation is slow?

    Any suggestion is appreciated.

    BBH

  2. #2
    and the Hat of Guessing tabstop's Avatar
    Join Date
    Nov 2007
    Posts
    14,336
    gprof might be able to get you the information you desire.

  3. #3
    Guest Sebastiani's Avatar
    Join Date
    Aug 2001
    Location
    Waterloo, Texas
    Posts
    5,708
    Quote Originally Posted by BBH View Post
    I have a programming question and hope to get your suggestions. I have a C++ program that reads files.

    From a preprocessing program, I know the size of the file and how much memory is needed.

    Step 1: I get the memory needed from reading a small configuration file and allocate the memory by calling new(). The total mem is around 800M since it is a large network.

    Step 2: I read the data file, process the data and write them to the memory.

    However, the program is very slow during the reading file and filling the data. The whole machine has 4G mem. It is CentOS and program is in C++, compiled by g++.

    I tried reading the file only, doing simple operations and not writing them into the mem. It is fast. Seems the memory allocation is slow?

    Any suggestion is appreciated.

    BBH

    You may have 4GB of total memory - but the question is: do you have 800MB of contiguous memory available? If not, then the memory will get swapped out to a page file, possibly leading to thrashing. Also, a lot of little allocations adds up to big overhead for bookkeeping, so if possible, allocate the entire block at once and divide amongst the different parts of the program "manually". If that doesn't work, you may have to break the job into small chunks and process it little by little. Depending on the application, that may or may not be feasable, of course.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. A question related to dynamic memory allocation
    By spiit231 in forum C Programming
    Replies: 2
    Last Post: 03-11-2008, 12:25 AM
  2. Understanding Memory Allocation
    By Ragsdale85 in forum C Programming
    Replies: 7
    Last Post: 10-31-2005, 08:36 AM
  3. Memory allocation and deallocation
    By Micko in forum C++ Programming
    Replies: 3
    Last Post: 08-19-2005, 06:45 PM
  4. Pointer's
    By xlordt in forum C Programming
    Replies: 13
    Last Post: 10-14-2003, 02:15 PM
  5. Memory Allocation :: C/C++
    By kuphryn in forum C++ Programming
    Replies: 4
    Last Post: 08-15-2002, 10:38 AM