large memory allocation problem

This is a discussion on large memory allocation problem within the C Programming forums, part of the General Programming Boards category; Hello Everybody; I need to read data about 3 x 80 Mb into memory so that I can process it ...

  1. #1
    Registered User
    Join Date
    Dec 2010
    Posts
    3

    large memory allocation problem

    Hello Everybody;

    I need to read data about 3 x 80 Mb into memory so that I can process it faster. What would be the best way to do this?

    I tried reading it into array (3 different arrays, each with 80 Mb), then it does not even compile; kind of freezes when compiling. I also have tried it by allocating memory (malloc, 3 pointers pointing to 80 Mb data), does not work too. It compiles, but the results are wrong. It gives duplicated results.

    Anybody has a clue what might be the problem?

    The computer I am using has 512 Mb RAM, and it runs Ubuntu 10.04.

  2. #2
    Epy
    Epy is offline
    Fortran lover Epy's Avatar
    Join Date
    Sep 2009
    Location
    California, USA
    Posts
    959
    With only 512 MB RAM, the system might not have 240 MB free for your program. Are you checking the result from malloc()? You might be getting a null (failed) result. Malloc() should work, I've used it to allocate up to 600 MB before.

  3. #3
    Registered User
    Join Date
    Sep 2006
    Posts
    8,868
    We could yak about it for awhile - I would bet you can't get that much RAM out of the stack, so malloc seems like the way to go. To do much beyond that, you'd have to post your code, and give a description of the algorithm or logic, it's using.

    I don't understand why you'd get duplicated results from using malloc. Off-hand, I'd say that's your error. That is a lot of RAM to require, for a rig with only 512Mb, though. If your OS is being forced by your request to use swap disk space for RAM, it may not be faster to have all that data in memory, after all. Would more RAM be possible for that system you're using?

    But describe what you're doing, and how you're trying to do it algorithmically, and post up your code, and we'll take a peek "under the hood", and see what's making ping, knock, and funny noise.

  4. #4
    Registered User
    Join Date
    Dec 2010
    Posts
    3
    Thank you guys! The problem solved.. Being not so experienced in C, I was suspicious that duplicate results might be due to memory shortage. But getting kinda confirmation from you, like you managed using malloc for even larger memory chunk, and the one, I don't understand why you'd get duplicated results from using malloc. Off-hand, I'd say that's your error., made me check for external causes. And, it turned out indeed there was many duplicated values in the file. Unbelievable!.. anyways, probably some corruption in the file due to instrument malfunction or sth..

  5. #5
    Registered User
    Join Date
    Jun 2005
    Posts
    6,255
    Not knowing what your code is doing, or what instruments it is relying on, we can only guess the meaning of your "duplicated results".

    First check that the "instruments" are behaving as expected (preferably without using your code). In particular, do they sample over time, and give repeat values if a reading is taken from them too frequently?

    If the instruments are behaving as expected, the problem is - by a process of elimination - in the logic of your code. You are using malloc() which means, to use the memory obtained, some form of pointer or array operations. If any of those operations are invalid (eg walking past the end of allocated memory) anything is allowed to happen.

    The problem is vastly more likely to be in your code that uses the memory, than in malloc() itself.

    It is very very unlikely that your problem is with malloc(). One very frequent phenomenon among novice (and even experienced) programmers is a belief "it can't be my code, so the problem must be with malloc()". The vast majority of people making such a claim have actually done something wrong in their code.
    Right 98% of the time, and don't care about the other 3%.

  6. #6
    Registered User
    Join Date
    Dec 2010
    Posts
    3

    Solved

    The program reads HDF files, which is taken by a satellite instrument. So, no way going up there and checking the instrument By writing a small code, I managed to get the indices of the duplicated values and then just looked the file by its graphical program. And there it was the duplication! Apparently the file for that date has been corrupted. Looking other files, no problems so far.

  7. #7
    Registered User
    Join Date
    Sep 2006
    Posts
    8,868
    Quote Originally Posted by emaitidude View Post
    The program reads HDF files, which are taken by a satellite instrument. So, no way going up there and checking the instrument
    I swear, the programmers today are SO lazy - in my day we ran to check those satellites, and back - uphill both ways, in our bare feet!

  8. #8
    Banned
    Join Date
    Aug 2010
    Location
    Ontario Canada
    Posts
    9,547
    Quote Originally Posted by adak View Post
    i swear, the programmers today are so lazy - in my day we ran to check those satellites, and back - uphill both ways, in our bare feet!
    rotflmao.....

  9. #9
    Registered User
    Join Date
    Jun 2005
    Posts
    6,255
    Quote Originally Posted by emaitidude View Post
    The program reads HDF files, which is taken by a satellite instrument. So, no way going up there and checking the instrument By writing a small code, I managed to get the indices of the duplicated values and then just looked the file by its graphical program. And there it was the duplication! Apparently the file for that date has been corrupted. Looking other files, no problems so far.
    Be that as it may.

    If your code relies on assumptions, and those assumptions are invalid, then that is a design problem with your code.

    In your particular case, it would be better if your program checked that the input file was valid, rather than assuming it is. If nothing else, it would allow reporting errors in a more meaningful way than with a program crash.
    Right 98% of the time, and don't care about the other 3%.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Memory allocation problem
    By ariko in forum C Programming
    Replies: 10
    Last Post: 07-24-2010, 09:09 PM
  2. Memory allocation problem, don't know what's wrong
    By Metalmurphy in forum C Programming
    Replies: 10
    Last Post: 04-22-2008, 04:32 AM
  3. Relate memory allocation in struct->variable
    By Niara in forum C Programming
    Replies: 4
    Last Post: 03-23-2007, 03:06 PM
  4. Memory allocation problem
    By Thantos in forum C Programming
    Replies: 6
    Last Post: 01-22-2004, 11:56 PM
  5. Pointer's
    By xlordt in forum C Programming
    Replies: 13
    Last Post: 10-14-2003, 02:15 PM

Tags for this Thread


1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21