Thread: is a leak... or is it not?

  1. #1
    Registered User
    Join Date
    Apr 2007
    Posts
    13

    is a leak... or is it not?

    Hi all,

    For a while now i've been working on a video broadcasting server (namely VLC) under linux. Obviously the program runs (or should run) permanently as a daemon.

    My problem is when constantly running for several days the program crashes (segfault) as of OOM (memory full). Of course this indicates a memory leak somewhere.... the problem is where? or is it...?

    The program is naturally multithreaded and the sheer volume of mallocs and frees as well as their not always straightforward distribution is a bit of a pain...

    I must admit that the core dump has not helped much... Anyway... here's my bottom line:

    a) When the program crashes the memory (seen from 'top') is not freed.
    b) 'top' does not indicate any increase in resource usage from the particular pid. Also, the overall memory consumed is at any time much more than that procecess reportedly use (summed up). The overall used memory constantly increases but the process percentages are the same...
    c) Valgrind (i.e. memcheck - almost) does not detect any MAJOR leaks somewhere... There are bits leaked but only amount to a few Ks.

    So... my question is... is it possible that the program is killed while memory is leached from a different process altogether? Any ideas on how i can track what is causing the leak? I'm trying to make some use of the info under /proc (this should be linux specific) but no luck so far...

    any ideas how i can track such a leak down??

    many thanks everyone and sorry for the long and not very detailed report...

    Nik
    Last edited by nantonop; 04-20-2007 at 06:14 AM.

  2. #2
    Registered User
    Join Date
    Apr 2007
    Posts
    45
    c) Valgrind (i.e. memcheck - almost) does not detect any MAJOR leaks somewhere... There are bits leaked but only amount to a few Ks.
    Ks add up to megabytes!!
    we are one

  3. #3
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    This post should be on the developer mailing list for VLC.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  4. #4
    Registered User
    Join Date
    Apr 2007
    Posts
    13
    as a matter of fact this post IS on the VLC developer mailing list as well... but this is not the point. the question that i suppose suits this forum is "how to certify which process has the memory leak"...

    sorry if you think this is out of scope. you re probably right...

    thanks anyway...

  5. #5
    Registered User
    Join Date
    Apr 2007
    Posts
    13
    one more thing...

    when the program crashes the memory does NOT drop to normal levels. The memory leaked appears to be unfreed even when the process has been killed. Can this happen or does this mean that the memory consumption is due to a different process??

  6. #6
    Registered User
    Join Date
    Apr 2007
    Posts
    45
    Quote Originally Posted by nantonop View Post
    one more thing...

    when the program crashes the memory does NOT drop to normal levels. The memory leaked appears to be unfreed even when the process has been killed. Can this happen or does this mean that the memory consumption is due to a different process??
    You should find your swapfile usage drops to 0 (or close to it).

    Linux utilises all available RAM eventually and gives the appearance of not freeing it after processes have ended (really it's distributing it all equally back across all the other processes running ).
    we are one

  7. #7
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    So perhaps a more general technical question on how to debug larger applications then (moved to tech board).

    Is it perhaps consuming (and not releasing) shared memory (shared with other processes). Such memory isn't going to go away when a single process dies, but I don't know off hand where to look to get information as to who owns what, and how much.

    I do have a feeling though that there is a lot less shared memory than there is virtual memory allocatable to each process.

    Does it take the same amount of time for the memory to re-leak following a crash, or do you have to reboot the machine to get back to normal?
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  8. #8
    Registered User
    Join Date
    Apr 2007
    Posts
    13
    thanks & sorry again for misplacing the post,

    to answer ur question Salem, i need to reboot to get memory back to normal...

    Some more: it takes a substantial length of time (~1 week) to reach OOM. But there's a weird part... I've been developing and testing under Ubuntu Dapper and Edgy. The Dapper flavour has no problem - keeps on running many more days without raising consumed mem levels. It is with the Edgy Ubuntu that i get the problems...

    i suppose there's something different in memory handling in these two...

    I hope i'm not confusing everyone in the same way i have myself... But it's just too many things to consider and i can't really put my finger to smth in particular. Keep in mind that this is a streaming server doing 12 video streams + 4 audio unicast... You can imagine the sort of hell it is to debug this thing...

    thnx for ur input everyone...

  9. #9
    Registered User
    Join Date
    Oct 2001
    Posts
    2,934
    >to answer ur question Salem, i need to reboot to get memory back to normal...
    I think what Salem's wondering is: can you restart the program after crashing (without rebooting), and if so, does it crash sooner the second time?

  10. #10
    Officially An Architect brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,396
    Quote Originally Posted by nantonop View Post
    when the program crashes the memory does NOT drop to normal levels. The memory leaked appears to be unfreed even when the process has been killed. Can this happen or does this mean that the memory consumption is due to a different process??
    It could be that your program depends on some DLL which itself contains a memory leak. Windows has a bad habit of not unloading DLLs (this is why you almost always have to REBOOT after you update something). So if the DLL is buggy, that leaked memory may hang around.

  11. #11
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    > I've been developing and testing under Ubuntu Dapper and Edgy.
    Are these all using the same versions of the kernel, glibc, other libraries?

    Are they all running on the same hardware (the same memory, swap space) ?

    Are they all running the same set of standard services? Maybe the ones which fail are running something else which is also eroding resources.

    The kernel in particular has many build time and init time configuration options which affect the number of resources available. Maybe the Dapper system simply has more of some resource so it just takes longer for it to fail.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  12. #12
    Registered User
    Join Date
    Apr 2007
    Posts
    13
    well, the kernel is the same. glibc too. some libraries (it's a bit hard to know which) should not be the same. The H/W is the same... The thing is that the Dapper tests show that everything's normal! I mean there appears to be NO leaks at all... at least memory consumption as seen through 'top' is pretty much constant...

    I have a feeling this is not a forum kind of question... there's simply too many things that could go wrong and too little detail i can give u...

  13. #13
    Registered User
    Join Date
    Apr 2007
    Posts
    45
    I haven't used Valgrind yet (only recently switched to this craaaaaazy OS and not had the need for it so far), but is it no help in plugging the leaks you found?

    That would totally be my first port of call!
    we are one

  14. #14
    Officially An Architect brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,396
    In my previous response I assumed you were running Windows for some dumb reason. Sorry. Have you tried running "ipcs" as root? That will display all IPC resources. Perhaps some program crashed without destroying a shared memory segment.

  15. #15
    Registered User
    Join Date
    Apr 2007
    Posts
    13
    well... at the moment i'm trying to fix the leaks reported by valgrind and u can imagine some are not very easy to find... valgrind indicates the allocation spot but not where the leak actually happens and in a program of this nature it can get tedious locating where memory is leaked...

    >Have you tried running "ipcs" as root?
    actually i haven't. i'll have a go and let u know id i come up w/ smth...

    thanx guys!

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Detecting memory leak in complicating source code
    By mosxopul in forum C++ Programming
    Replies: 5
    Last Post: 03-30-2009, 11:41 AM
  2. Question regarding Memory Leak
    By clegs in forum C++ Programming
    Replies: 29
    Last Post: 12-07-2007, 01:57 AM
  3. Is this a memory leak?
    By cboard_member in forum C++ Programming
    Replies: 9
    Last Post: 07-20-2005, 01:15 PM
  4. Memory leak trackers
    By Darkness in forum A Brief History of Cprogramming.com
    Replies: 5
    Last Post: 11-30-2004, 02:03 PM
  5. Why is this a memory leak?
    By deadpoet in forum C++ Programming
    Replies: 2
    Last Post: 02-20-2004, 11:16 AM