Thread: Ridiculous memory usage - can't find why

  1. #1
    C++まいる!Cをこわせ!
    Join Date
    Oct 2007
    Location
    Inside my computer
    Posts
    24,654

    Ridiculous memory usage - can't find why

    I seem to having trouble lately...

    I have some program that innocently allocates ~4500 objects which are 240 bytes in size. Unfortunately, it translates into the program using several gigabyte of memory.
    Clearly unacceptable, yet I cannot find the source. I don't know if it's these small objects that create the problems or some other data structures, such as vectors, queues and maps.
    I've also taken a look at vmap from sysinternals and it's showing me several allocations that are 16 MB in size, which is disturbing.

    I don't know what's causing this. Have anyone had any similar experiences? Are there any good tips or tools to allow one to see what's eating up all the memory?
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  2. #2
    Registered User
    Join Date
    May 2010
    Posts
    4,633
    What operating system are you using?

    Jim

  3. #3
    C++まいる!Cをこわせ!
    Join Date
    Oct 2007
    Location
    Inside my computer
    Posts
    24,654
    I think most regulars should know, but it's a fair question. All don't know, after all.
    I'm running Windows 8.1 right now. Compiler is Visual Studio 2012 (RTM).
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  4. #4
    Registered User
    Join Date
    Oct 2006
    Posts
    3,445
    is it possible that each allocation is allocating a new 4k page in memory? that still wouldn't account for it, as 4500 * 4096 = 18432000 bytes, but that at least partially explains the allocations of 16MB.
    What can this strange device be?
    When I touch it, it gives forth a sound
    It's got wires that vibrate and give music
    What can this thing be that I found?

  5. #5
    C++まいる!Cをこわせ!
    Join Date
    Oct 2007
    Location
    Inside my computer
    Posts
    24,654
    It allocates a total of 4500 objects before it consumes several order of gigabyte and crashes due to an unrelated bug. I see many 16 MB allocations in vmap. I don't think there are any significant memory leaks (I know there are some in code that isn't mine, but since that code is only executed once, I don't think it affects this). At least, no significant leaks that I can find with Intel Inspector.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  6. #6
    Registered User
    Join Date
    May 2010
    Posts
    4,633
    I'm running Windows 8.1 right now. Compiler is Visual Studio 2012 (RTM).
    Then have you looked into the Windows Performance Toolkit


    Jim

  7. #7
    C++まいる!Cをこわせ!
    Join Date
    Oct 2007
    Location
    Inside my computer
    Posts
    24,654
    No, I don't know how to use it. But I'll check it out anyway and see if I can gleem some insider information.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  8. #8
    C++まいる!Cをこわせ!
    Join Date
    Oct 2007
    Location
    Inside my computer
    Posts
    24,654
    Well, I found the problem. I am still not sure how to solve it, though, but that's another matter. I did some old-fashioned stepping through the code and noticed it began eating a lot of memory when looping in a loop, but it seemed only to "jump" at certain times and not others (each allocation was about 16 MB). Looking at the code, I guessed what the problem was, and I was right, and that was that.
    WPA didn't help at all. It provided some nice graph about how memory was increasing with time, but no other data. No tables, no showing what allocation took up how many bytes, or where (or I just suck at using it).
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  9. #9
    Registered User
    Join Date
    Jun 2005
    Posts
    6,815
    Seeing as you seem to have isolated the problem, how about creating a small sample that illustrates the symptom? Someone might then be able to offer an explanation. Or, in the process of creating the small sample, the reason may become obvious to you - so you can describe it.
    Right 98% of the time, and don't care about the other 3%.

    If I seem grumpy or unhelpful in reply to you, or tell you you need to demonstrate more effort before you can expect help, it is likely you deserve it. Suck it up, Buttercup, and read this, this, and this before posting again.

  10. #10
    C++まいる!Cをこわせ!
    Join Date
    Oct 2007
    Location
    Inside my computer
    Posts
    24,654
    I didn't do it because it probably wouldn't help. It's very specific to the problem I'm working on. But simplified, I can say this:
    It's a search tree, so it contains a lot of nodes (each node is ~240 bytes).
    It's a search tree, so sometimes I need to store extra information on the problem the search tree represents in a node. This "extra state" takes about 16 MB of memory. It does it every 8 nodes or so.
    So the problem was basically that the problem I was trying to solve endlessly tried to go deeper and deeper (hence 4500 nodes), and every 8 nodes or so, it would create another 16 MB extra state.
    I detected it when taking a look when the code explored the tree, as the memory seemed to "jump" every 8 or so nodes. So I took a look at the code exploring the tree and thought - wait, maybe the "extra state" is taking up all that memory. So I tried my hypothesis by running up that point, and monitoring memory usage while storing that extra state, and there you go.
    Would have been nice if I had had some tool that you point me to that right away instead of me having to fool around trying to find it.

    I was initially confused because if a DFS search on the tree was made, memory was all fine. But when running my own search, it ballooned.
    Later, I realized that DFS is performing a left-most search and my search was performing a right-most search and that turned out to be a bad idea for the problem.
    But that's alright. This is supposed to be a framework for others to use, so the implementation would appear to be "correct." It's up to the user to supply the correct parameters for the problem.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  11. #11
    Registered User VirtualAce's Avatar
    Join Date
    Aug 2001
    Posts
    9,607
    Wow search 'state' data requires 16MB? That is huge. Why is this 'state' stored in the tree?

  12. #12
    C++まいる!Cをこわせ!
    Join Date
    Oct 2007
    Location
    Inside my computer
    Posts
    24,654
    The tree represents different states of the problem, so when exploring the tree, or more exactly, when backtracking, the state of the problem must be recomputed, so the tree needs to periodically store this state because its function is to aid finding a solution to the problem. Admittedly, the problem was a little large, as you found out...
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  13. #13
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Ireland
    Posts
    8,446
    As VirtualAce hinted, you need to offload the state to disk. That design, as you may have realized by now, is unacceptable.

    I recently went through a similar issue on a small financial analysis program I was making to help the school I'm working on decide on optimum returns. Before I knew it, although not as big as yours, I was allocating large amounts of data, because I wanted to keep every projection in memory. Offloading this data, helped conserve the most scarce resource we have on our computers, while not damaging in the slightest the program performance, except for the initial moment when the data file is opened for the first time.
    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

  14. #14
    C++まいる!Cをこわせ!
    Join Date
    Oct 2007
    Location
    Inside my computer
    Posts
    24,654
    I suppose it is possible to offload state to disk that is stored at a large distance from the current position in the tree that is being explored, then read that back in when it gets close enough.
    But there are two issues here:
    - It's way too late to implement such a complicated systems and there are higher priority things that must be implemented and working. Right now, I'm only fixing very simple bugs and changing things that doesn't have a big impact on the design. The worst thing is that I have tests only for one problem and I don't really use a test framework (because I don't have time to find a good and put it to use), so I really don't want to do these kinds of things at this stage since it will mess up stuff already done (analysis, benchmarking), and I don't have time or the desire to redo it!
    - I'm not solving problems per se, I'm solving the problem that solves the problem, so to speak. I am responsible for finding the solutions to the problems, but there is no single problem that must work. It's simply up to the user to specify an efficient model. Problem is that I'm testing with sometimes very large problems (because small ones are too fast to be benchmarked).
    Anyway, the idea sounds like a good one and it also sounds like a fun challenge, but right now, I don't have time.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  15. #15
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Ireland
    Posts
    8,446
    Quote Originally Posted by Elysia View Post
    I suppose it is possible to offload state to disk that is stored at a large distance from the current position in the tree that is being explored, then read that back in when it gets close enough.
    Just thinking out loud here (I realize you aren't interested in a solution at this point)...


    • I don't know if there's another alternative to POSIX mmap on Windows, but you do have Win32 File mapping functions available to you that won't require so much effort in managing your tree. Like mmap, you get a pointer to the whole file, you won't upload the whole file to memory and all I/O operations are cached. Benchmarking would be required, but I suppose unless performance is a critical aspect to this portion of the code, you won't get much of a visual clue your tree is being traversed more slowly.
    • Don't forget however that the mother of all data mapping strategies is still available to you; A small, no-configuration database solution like SQLite for instance, can greatly speed and facilitate your tree IO.
    • This is one of those scenarios where hardware will definitely help. Faster HDDs (or SSDs) and a wider bus will trump almost any file access trick in the book.
    • I won't comment on the tree itself since I don't know the type of tree and what exactly it is being done with it, but threading would almost certainly increase your performance, if at least the whole file operations are kept on a separate thread.
    • 16MB per node is... ugh! However it's probably a candidate for rudimentary packing. If there's plenty of numeric data, I suppose you could pack most of it as byte arrays.
    Last edited by Mario F.; 08-01-2013 at 06:15 AM.
    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Memory / CPU Usage
    By javaeyes in forum C Programming
    Replies: 3
    Last Post: 02-27-2012, 06:19 PM
  2. Memory usage and memory leaks
    By vsanandan in forum C Programming
    Replies: 1
    Last Post: 05-03-2008, 05:45 AM
  3. Dialogs in memory? How ridiculous...
    By SMurf in forum Windows Programming
    Replies: 2
    Last Post: 08-13-2002, 02:12 PM
  4. Memory Usage
    By ghe1 in forum Linux Programming
    Replies: 0
    Last Post: 03-18-2002, 09:43 AM
  5. Memory usage
    By Razor Ice in forum C++ Programming
    Replies: 3
    Last Post: 01-08-2002, 03:16 PM