large memory allocation problem

Hello Everybody;

I need to read data about 3 x 80 Mb into memory so that I can process it faster. What would be the best way to do this?

I tried reading it into array (3 different arrays, each with 80 Mb), then it does not even compile; kind of freezes when compiling. I also have tried it by allocating memory (malloc, 3 pointers pointing to 80 Mb data), does not work too. It compiles, but the results are wrong. It gives duplicated results.

Anybody has a clue what might be the problem?

The computer I am using has 512 Mb RAM, and it runs Ubuntu 10.04.

With only 512 MB RAM, the system might not have 240 MB free for your program. Are you checking the result from malloc()? You might be getting a null (failed) result. Malloc() should work, I've used it to allocate up to 600 MB before.

We could yak about it for awhile - I would bet you can't get that much RAM out of the stack, so malloc seems like the way to go. To do much beyond that, you'd have to post your code, and give a description of the algorithm or logic, it's using.

I don't understand why you'd get duplicated results from using malloc. Off-hand, I'd say that's your error. That is a lot of RAM to require, for a rig with only 512Mb, though. If your OS is being forced by your request to use swap disk space for RAM, it may not be faster to have all that data in memory, after all. Would more RAM be possible for that system you're using?

But describe what you're doing, and how you're trying to do it algorithmically, and post up your code, and we'll take a peek "under the hood", and see what's making ping, knock, and funny noise. :D

Thank you guys! The problem solved.. Being not so experienced in C, I was suspicious that duplicate results might be due to memory shortage. But getting kinda confirmation from you, like you managed using malloc for even larger memory chunk, and the one, I don't understand why you'd get duplicated results from using malloc. Off-hand, I'd say that's your error., made me check for external causes. And, it turned out indeed there was many duplicated values in the file. Unbelievable!.. anyways, probably some corruption in the file due to instrument malfunction or sth..

Not knowing what your code is doing, or what instruments it is relying on, we can only guess the meaning of your "duplicated results".

First check that the "instruments" are behaving as expected (preferably without using your code). In particular, do they sample over time, and give repeat values if a reading is taken from them too frequently?

If the instruments are behaving as expected, the problem is - by a process of elimination - in the logic of your code. You are using malloc() which means, to use the memory obtained, some form of pointer or array operations. If any of those operations are invalid (eg walking past the end of allocated memory) anything is allowed to happen.

The problem is vastly more likely to be in your code that uses the memory, than in malloc() itself.

It is very very unlikely that your problem is with malloc(). One very frequent phenomenon among novice (and even experienced) programmers is a belief "it can't be my code, so the problem must be with malloc()". The vast majority of people making such a claim have actually done something wrong in their code.

The program reads HDF files, which is taken by a satellite instrument. So, no way going up there and checking the instrument :) By writing a small code, I managed to get the indices of the duplicated values and then just looked the file by its graphical program. And there it was the duplication! Apparently the file for that date has been corrupted. Looking other files, no problems so far.

Quote:

---

Originally Posted by emaitidude

The program reads HDF files, which are taken by a satellite instrument. So, no way going up there and checking the instrument :)

---

I swear, the programmers today are SO lazy - in my day we ran to check those satellites, and back - uphill both ways, in our bare feet! :p :p :p

Quote:

---

Originally Posted by adak

i swear, the programmers today are so lazy - in my day we ran to check those satellites, and back - uphill both ways, in our bare feet! :p :p :p

---

rotflmao.....

Quote:

---

Originally Posted by emaitidude

The program reads HDF files, which is taken by a satellite instrument. So, no way going up there and checking the instrument :) By writing a small code, I managed to get the indices of the duplicated values and then just looked the file by its graphical program. And there it was the duplication! Apparently the file for that date has been corrupted. Looking other files, no problems so far.

---

Be that as it may.

If your code relies on assumptions, and those assumptions are invalid, then that is a design problem with your code.

In your particular case, it would be better if your program checked that the input file was valid, rather than assuming it is. If nothing else, it would allow reporting errors in a more meaningful way than with a program crash.