Thread: large memory allocation problem

Hello Everybody;

I need to read data about 3 x 80 Mb into memory so that I can process it faster. What would be the best way to do this?

I tried reading it into array (3 different arrays, each with 80 Mb), then it does not even compile; kind of freezes when compiling. I also have tried it by allocating memory (malloc, 3 pointers pointing to 80 Mb data), does not work too. It compiles, but the results are wrong. It gives duplicated results.

Anybody has a clue what might be the problem?

The computer I am using has 512 Mb RAM, and it runs Ubuntu 10.04.

With only 512 MB RAM, the system might not have 240 MB free for your program. Are you checking the result from malloc()? You might be getting a null (failed) result. Malloc() should work, I've used it to allocate up to 600 MB before.

We could yak about it for awhile - I would bet you can't get that much RAM out of the stack, so malloc seems like the way to go. To do much beyond that, you'd have to post your code, and give a description of the algorithm or logic, it's using.

I don't understand why you'd get duplicated results from using malloc. Off-hand, I'd say that's your error. That is a lot of RAM to require, for a rig with only 512Mb, though. If your OS is being forced by your request to use swap disk space for RAM, it may not be faster to have all that data in memory, after all. Would more RAM be possible for that system you're using?

But describe what you're doing, and how you're trying to do it algorithmically, and post up your code, and we'll take a peek "under the hood", and see what's making ping, knock, and funny noise.

Thank you guys! The problem solved.. Being not so experienced in C, I was suspicious that duplicate results might be due to memory shortage. But getting kinda confirmation from you, like you managed using malloc for even larger memory chunk, and the one, I don't understand why you'd get duplicated results from using malloc. Off-hand, I'd say that's your error., made me check for external causes. And, it turned out indeed there was many duplicated values in the file. Unbelievable!.. anyways, probably some corruption in the file due to instrument malfunction or sth..

The program reads HDF files, which is taken by a satellite instrument. So, no way going up there and checking the instrument By writing a small code, I managed to get the indices of the duplicated values and then just looked the file by its graphical program. And there it was the duplication! Apparently the file for that date has been corrupted. Looking other files, no problems so far.

Originally Posted by emaitidude

The program reads HDF files, which are taken by a satellite instrument. So, no way going up there and checking the instrument

I swear, the programmers today are SO lazy - in my day we ran to check those satellites, and back - uphill both ways, in our bare feet!

Originally Posted by adak

i swear, the programmers today are so lazy - in my day we ran to check those satellites, and back - uphill both ways, in our bare feet!

rotflmao.....