Like Tree4Likes
  • 1 Post By anduril462
  • 1 Post By Salem
  • 1 Post By Codeplug
  • 1 Post By MK27

Expert opinions wanted!

This is a discussion on Expert opinions wanted! within the Linux Programming forums, part of the Platform Specific Boards category; This was inspired by a recent thread: http://cboard.cprogramming.com/linux...-examples.html Cross-posted to linuxquestions.org: http://www.linuxquestions.org/questi...63#post4492563 In the spring I was involved with a ...

  1. #1
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300

    Expert opinions wanted!

    This was inspired by a recent thread:

    Trying to use the libgtop examples

    Cross-posted to linuxquestions.org: http://www.linuxquestions.org/questi...63#post4492563

    In the spring I was involved with a web based project involving some technology new to me and everyone else concerned, and our memory usage turned out to be significantly more than was predicted, so we needed to scrutinize what was responsible for what. To that end, I wrote a simple process monitor that could be used to log statistics for individual process over a period of time (days or weeks). Ie, it is based on the proc filesystem, but it is not a re-invented "top".

    Since then I've refined the tool and it seems to me worth sharing, so I sat down to write a manual page. One of the things I wanted to do with that was explain some of the familiar memory measurements (RSS, VSZ, etc) from proc in a concise and concrete way not available elsewhere (although there are many discussions of these things available via online searching). While the tool serves its purpose for me based on my current understanding of these things, I'm a stickler for accuracy and do not want to spread erroneous information, so I'm looking for feedback about the following draft:

    Code:
           VirtualSz
                This is the total of all the memory regions (RAM and swap) used or
                partially used by the process, including all the shared libraries,
                which (generally) account for most of it.  Hence, this is not a mea‐
                sure of the load added by the process, since (generally)  much  of
                those shared libraries are already in play.
    
           ResidentSz
                Resident  Set Size (RSS). This is much closer to the load added by
                the process, however, it still includes  some  memory  shared  (or
                sharable) by other processes spawned from the same executable.  If
                this is the only instance running, it is an  accurate  measure  of
                how much memory the process has consumed.
    
           Shared_Mem
                This  is  not the total of all the shared libraries used, ie, sub‐
                tracting it from the virtual size will not give you  the  resident
                size.   It is a total of all the actual shared mappings (eg, those
                parts of shared libraries actually in use).
    
           Data+Stack
                This is the bulk of the process's  private,  dynamically  changing
                memory.
    
           Priv&Write
                This  is  all the memory used by the process marked as private and
                writable.  The significance of this  figure  is  that  it  is  the
                actual  load added by the individual process if there are multiple
                instances; pmap(1) will also report this figure.
    The first 4 fields are directly from /proc/[pid]/statm, the fifth is calculated by parsing /proc/[pid]/maps. I've highlighted the bits that I'm most concerned about, since these are "facts" inferred by me from use, experimentation, existing documentation, etc, and for which I have been unable to find direct confirmation.

    I'm also curious as to whether Data+Stack includes heap memory; it seems to me it does. If so, I could say "including stack and heap".
    Last edited by MK27; 10-07-2011 at 10:41 AM.
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  2. #2
    Registered User
    Join Date
    Nov 2010
    Location
    Long Beach, CA
    Posts
    5,566
    Quote Originally Posted by MK27 View Post
    I'm also curious as to whether Data+Stack includes heap memory; it seems to me it does. If so, I could say "including stack and heap".
    I'm by no means an expert, but I think it does include the heap, as far as your stats are concerned. A quick overview of Linux memory layout:
    Code:
    +------------+
    |   Stack    |
    |     |      |
    |     V      |
    |            |
    |Shared libs |
    |            |
    |     ^      |
    |     |      |
    |   Heap     |
    |   Data     |
    |   Text     |
    +------------+
    I can't recall if the BSS section is included in the Data portion or not, but I think it is (it's just an efficient way to store uninitialized globals, which go in the data section).

    The text and data sections are fixed at compile time, but the stack and heap and shared libs change during runtime. AFAIK, when the OS/loader allocates space for the heap in a process, it basically does so by setting the start of the heap to the first piece of memory after the data section, then extending the end of the data section, to give the allocator a place to play. I think that's why it ends up included in your stats for data+stack.
    MK27 likes this.

  3. #3
    and the hat of wrongness Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,673
    Perhaps this site can help -> LinuxMM - linux-mm.org Wiki
    MK27 likes this.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  4. #4
    Registered User Codeplug's Avatar
    Join Date
    Mar 2003
    Posts
    4,676
    VirtualSz
    >> or partially used by the process
    I'm not sure that adds much to the definition. Perhaps re-word it all to use "allocated" instead of "used"?

    ResidentSz
    >> includes ... memory shared ... by other processes spawned from the same executable. If this is the only instance running, it is an accurate measure of how much memory the process has consumed.
    Don't think that's accurate. RSS = resident in ram, or virtual pages mapped to physical pages. That may include shared libraries etc... Perhaps you meant: "if this is the only process in the entire OS". To me, it read like "if the only instance is this particular process".

    Shared_Mem
    Suggest something like: "virtual memory pages that are shared between this process and other processes: including code from shared libraries, pages still shared from a forked (but not exec'ed) process, and manually shared memory". You may want to also specify if this is rss only shared pages, or if it includes shared but not resident pages. Or if it includes shareable pages that currently aren't referenced by any other processes.

    Data+Stack
    You sure statm\data doesn't include shared data? Or that it's all rss pages? For example, the default stack size for a new thread may be 8MB, but the OS will only make resident those pages which are actually touched.

    Priv&Write
    >> The significance of this figure is that it is the actual load added by the individual process [even] if there are multiple instances
    Does adding that "even" convey your original intention?

    Have you considered hitting smaps, then falling back on maps if it isn't there?
    Exploring: Memory Usage with smaps

    gg

  5. #5
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    Quote Originally Posted by Codeplug View Post
    ResidentSz
    >> includes ... memory shared ... by other processes spawned from the same executable. If this is the only instance running, it is an accurate measure of how much memory the process has consumed.
    Don't think that's accurate. RSS = resident in ram, or virtual pages mapped to physical pages. That may include shared libraries etc...
    You're right; that is inaccurate. Much of the figure reported in "shared" is included in the RSS.

    This made me realize I should incorporate the distinction between virtual and physical addressing, since these are two different "totals" that the process contributes to. To be honest, I had thought all the stats were virtual, and that the kernel did not report anything to do with physical memory. Which made it impossible to understand "RSS" properly, lol.

    Quote Originally Posted by Codeplug View Post
    Data+Stack
    You sure statm\data doesn't include shared data? Or that it's all rss pages? For example, the default stack size for a new thread may be 8MB, but the OS will only make resident those pages which are actually touched.
    Right again; it includes shared data, and it is a measure of virtual space. I think I may actually drop the data+stack figure and add something from smaps.

    Priv&Write
    >> The significance of this figure is that it is the actual load added by the individual process [even] if there are multiple instances
    Does adding that "even" convey your original intention?
    No, what I meant here is that this is memory which cannot be shared between processes, so if you have two instances running and kill one, this is the amount of virtual space you will free up.

    The "priv" bit is maybe irrelevant; AFAICT the only memory that is not marked private has to do with hardware interfaces.

    Have you considered hitting smaps, then falling back on maps if it isn't there?
    Exploring: Memory Usage with smaps
    That's a good article.

    I'm going to keep the Priv&Write because this is the one that turned out to be most useful in the spring; the project was running under openVN, and it was the best measure WRT the total system memory usage/limit there (openVN is like a hypervisor).

    But smaps also brought to my attention PSS, which is a newer metric: pages shared with other processes divided by the number of processes using them. Obviously, this fluctuates more than anything else (for idle or stable processes). So I may do a "Proportional Resident" figure; that seems like the most accurate measure of how much real, physical memory use a process represents.

    Code:
     VirtualSz
                This is the virtual address space available to the process; ie, it
                is the maximum amount of memory that could be  consumed.   If  the
                process  does  not  use  all of this most of the time, it is not a
                very good measure of the real load incurred.
    
           ResidentSz
                Resident Set Size (RSS). This is the amount of real physical  mem‐
                ory  used  by  the  process.  However, it includes space shared by
                other processes.
    
           Shared_Mem
                This is the total of all the shared virtual mappings.
    
           Priv&Write
                This is all the virtual memory marked  as  private  and  writable.
                The  significance  of  this figure is that it is the load added by
                the individual process if it is one of multiple  instances of the
                same program;  pmap(1) will also report this.
    Thanks Codeplug
    Last edited by MK27; 10-08-2011 at 10:09 AM.
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  6. #6
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    Quote Originally Posted by MK27 View Post
    Code:
     Shared_Mem
               This is the total of all the shared virtual mappings.
    Couldn't edit because of the new one hour limit, but I believe this should be shared physical mappings. I'll do some stuff with smaps to verify that.

    I should probably remove VirtualSz too since AFAICT that is immutable anyway.
    Last edited by MK27; 10-08-2011 at 10:36 AM.
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  7. #7
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    Quote Originally Posted by MK27 View Post
    Couldn't edit because of the new one hour limit, but I believe this should be shared physical mappings.
    They are.

    I should probably remove VirtualSz too since AFAICT that is immutable anyway.
    No. What am I thinking???

    This one hour limit is really going to beef up my post count, lol.
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  8. #8
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    Quote Originally Posted by anduril462 View Post
    I'm by no means an expert, but I think it does include the heap, as far as your stats are concerned.

    The text and data sections are fixed at compile time, but the stack and heap and shared libs change during runtime. AFAIK, when the OS/loader allocates space for the heap in a process, it basically does so by setting the start of the heap to the first piece of memory after the data section, then extending the end of the data section, to give the allocator a place to play. I think that's why it ends up included in your stats for data+stack.
    For sure it is included, and thanks for the "why". This was easy to confirm:

    Code:
    #include <stdio.h>
    #include <stdlib.h>
    
    int main(int argc, const char *argv[]) {
    	printf("%d\n", getpid());
    	getchar();
    	char *x = calloc(10<<10, 1);
    	getchar();
    	return 0;
    }
    Some good stuff over at LQ (such as me redefining VirtualSz for the 3rd time), if anyone is interested in all this:

    Opinions wanted for documentation regarding process memory usage
    Last edited by MK27; 10-08-2011 at 12:52 PM.
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  9. #9
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    If anyone is interested in this, it's online now:

    Plog
    Codeplug likes this.
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. i need a c expert..
    By riel in forum C Programming
    Replies: 7
    Last Post: 01-27-2008, 06:19 AM
  2. need some expert help here!
    By magix24 in forum C Programming
    Replies: 1
    Last Post: 04-11-2006, 10:36 PM
  3. c++ expert
    By febrian81 in forum C++ Programming
    Replies: 2
    Last Post: 07-01-2002, 10:27 PM
  4. Opinions Wanted
    By Unregistered in forum C Programming
    Replies: 2
    Last Post: 03-04-2002, 08:56 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21