Thread: my code to prove working set larger than virtual bytes

  1. #1
    Registered User
    Join Date
    May 2006
    Posts
    1,579

    my code to prove working set larger than virtual bytes

    Hello everyone,


    From the definition of working set, it is a subset of virtual pages resident in physical memory -- from book Windows Internals. It means working set could not be larger than virtual memory (subset relationship).

    But the following simple code on Windows Server 2003 proves (if you monitor virtual bytes counter and working set bytes conuter from perfmon), if we do not unmap the page map file, the working set will continue to increase (and much larger than virtual bytes) until we unmap it.

    Take a breakpoint before following code section,

    Code:
    			// close mapped files to avoid leak
    			for (sectionIndex = 0; sectionIndex < size/allocSize; sectionIndex++)
    			{
    				if (map [sectionIndex])
    				{ 
    					UnmapViewOfFile(map [sectionIndex]);
    				}
    			}
    Any ideas? Does my code break the definition of working set? Why working set is much larger than virtual bytes?

    Code:
    #include <windows.h> 
    #include <stdio.h> 
    
    int main(int argc, char* argv[]) 
    { 
    	LARGE_INTEGER start,end; 
    	LARGE_INTEGER freq; 
    	QueryPerformanceCounter(&start); 
    	QueryPerformanceFrequency(&freq); 
    
    	MEMORYSTATUS memstat; 
    	void** map;
    	int sectionIndex = 0;
    	memstat.dwLength = sizeof(memstat); 
    	GlobalMemoryStatus(&memstat); 
    
    	// basic file mapping test (512 MB) 
    	long long size = 512*1024*1024; 
    
    	HANDLE mapping = 
    	CreateFileMapping(NULL,NULL,PAGE_READWRITE|SEC_COMMIT,(DWORD)(size>>32),DWORD(size),NULL); 
    	if (mapping) 
    	{ 
    		// create and destroy temporary views 
    		SYSTEM_INFO sysInfo; 
    		GetSystemInfo(&sysInfo); 
    		const int allocSize = sysInfo.dwAllocationGranularity; 
    
    		GlobalMemoryStatus(&memstat); 
    
    		void *mem = new char[allocSize]; 
    		memset(mem,0x11,allocSize); 
    
    		map = (void**) new char [sizeof(void*) * size / allocSize];
    
    		for (int i=0; i<10; i++) 
    		{ 
    
    			sectionIndex = 0;
    			for (long long offset=0; offset<=size-allocSize; offset+=allocSize) 
    			{ 
    				map [sectionIndex] = 
    				MapViewOfFile(mapping,FILE_MAP_WRITE,(DWORD)(offset>>32),(DWORD)offset,allocSize); 
    				if (map [sectionIndex]) 
    				{ 
    					memcpy(map [sectionIndex],mem,allocSize); 
    					// UnmapViewOfFile(map);
    				}
    
    				sectionIndex++;
    			} // for (long long offset=0; offset<=size-allocSize; offset+=allocSize) 
    			
    			// close mapped files to avoid leak
    			for (sectionIndex = 0; sectionIndex < size/allocSize; sectionIndex++)
    			{
    				if (map [sectionIndex])
    				{ 
    					UnmapViewOfFile(map [sectionIndex]);
    				}
    			}
    
    			GlobalMemoryStatus(&memstat); 
    
    			sectionIndex = 0;
    			for (long long offset=0; offset<=size-allocSize; offset+=allocSize) 
    			{ 
    				map [sectionIndex] = 
    				MapViewOfFile(mapping,FILE_MAP_READ,(DWORD)(offset>>32),(DWORD)offset,allocSize); 
    				if (map [sectionIndex]) 
    				{ 
    					for (int t=0; t<allocSize; t++) 
    					{ 
    						if (((char *)(map [sectionIndex]))[t]!=0x11) 
    						{ 
    							OutputDebugString("Memory read failed\n"); 
    						} 
    					} 
    				} 
    
    				UnmapViewOfFile(map [sectionIndex]);
    			} 
    
    			// close mapped files to avoid leak
    			/*
    			for (sectionIndex = 0; sectionIndex < size/allocSize; sectionIndex++)
    			{
    				if (map [sectionIndex])
    				{ 
    					UnmapViewOfFile(map [sectionIndex]);
    				}
    			}
    			*/
    
    			GlobalMemoryStatus(&memstat); 
    		} // for (int i=0; i<10; i++)
    
    		QueryPerformanceCounter(&end); 
    
    		GlobalMemoryStatus(&memstat); 
    
    		printf("Time %.3f\n", 
    		double(end.QuadPart-start.QuadPart)/double(freq.QuadPart)); 
    		CloseHandle(mapping); 
    		delete[] mem; 
    		GlobalMemoryStatus(&memstat); 
    	} //if (mapping)
    
    	return 0;
    }

    thanks in advance,
    George

  2. #2
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    > From the definition of working set, it is a subset of virtual pages resident in physical memory -- from book Windows Internals
    Oh, well I subscribe to this definition.

    My understanding of working set is just a list of all the pages you've touched in unit time (say the last second). It says nothing about whether that page is mapped to VM or is in the swap file.

    Just because you just touched it (it's in the working set) doesn't mean that it won't be very soon swapped out to the swap file (not in VM).

    Sure, if WS is bigger than VM, it might be indicative of some kind of performance problem in your program (because it may imply thrashing), but in itself, I would not regard it as being an "impossible" situation to find yourself in.

    Definition of WS from perfmon
    Quote Originally Posted by perfmon
    Working Set is the current size, in bytes, of the Working Set of this process. The Working Set is the set of memory pages touched recently by the threads in the process. If free memory in the computer is above a threshold, pages are left in the Working Set of a process even if they are not in use. When free memory falls below a threshold, pages are trimmed from Working Sets. If they are needed they will then be soft-faulted back into the Working Set before leaving main memory.
    This definition is more in keeping with the Wikipedia entry than whatever it is you're reading.

    Definition of Virtual Bytes from perfmon
    Quote Originally Posted by perfmon
    Virtual Bytes is the current size, in bytes, of the virtual address space the process is using. Use of virtual address space does not necessarily imply corresponding use of either disk or main memory pages. Virtual space is finite, and the process can limit its ability to load libraries.
    Page file bytes, care of perfmon
    Quote Originally Posted by perfmon
    Page File Bytes is the current number of bytes that this process has used in the paging file(s). Paging files are used to store pages of memory used by the process that are not contained in other files. Paging files are shared by all processes, and the lack of space in paging files can prevent other processes from allocating memory.
    Note here that "not in other files" means that for example part of your program code (which is read-only) can be swapped out (not in VM), but also not in the swap file either (because it can be recreated from the .exe)
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  3. #3
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    I too see no "problem" with the working set being larger than virtual bytes. It can happen in some situations, just like you've shown.

    Particularly since you are unmapping pages from a mapped view of a file - you previously touched those bytes, and now you are not. So they are part of the working set that you HAD, but no longer part of the virtual memory of the application (you just unmapped it, right).

    It's all statistics, and the main point is probably to look at trends, not the exact values.

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  4. #4
    Registered User
    Join Date
    May 2006
    Posts
    1,579
    Thanks Salem,


    Code:
    map [sectionIndex] = 
    				MapViewOfFile(mapping,FILE_MAP_WRITE,(DWORD)(offset>>32),(DWORD)offset,allocSize); 
    				if (map [sectionIndex]) 
    				{ 
    					memcpy(map [sectionIndex],mem,allocSize); 
    					// UnmapViewOfFile(map);
    				}
    I have studied your reply and my code again. I think may be it is cause of I reused the buffer of variable map (it is 65535 bytes in my system), and only this variable map buffer is counted as virtual address and the real memory map file itself is not mapped into process virtual space with committed pages, so the memory map file is not counted as part of the virtual address? But since file map file RAM is *touched* by current process, it is counted by working set of current process. Is that correct understanding?

    Quote Originally Posted by Salem View Post
    > From the definition of working set, it is a subset of virtual pages resident in physical memory -- from book Windows Internals
    Oh, well I subscribe to this definition.

    My understanding of working set is just a list of all the pages you've touched in unit time (say the last second). It says nothing about whether that page is mapped to VM or is in the swap file.

    Just because you just touched it (it's in the working set) doesn't mean that it won't be very soon swapped out to the swap file (not in VM).

    Sure, if WS is bigger than VM, it might be indicative of some kind of performance problem in your program (because it may imply thrashing), but in itself, I would not regard it as being an "impossible" situation to find yourself in.

    Definition of WS from perfmon

    This definition is more in keeping with the Wikipedia entry than whatever it is you're reading.

    Definition of Virtual Bytes from perfmon


    Page file bytes, care of perfmon

    Note here that "not in other files" means that for example part of your program code (which is read-only) can be swapped out (not in VM), but also not in the swap file either (because it can be recreated from the .exe)

    regards,
    George

  5. #5
    Registered User
    Join Date
    May 2006
    Posts
    1,579
    Thanks Mats,


    From my further analysis and your reply, I think only the map variable which has allocSize number of bytes in virtual memory (means have virtual memory address), since I reuse the map variable, to deal with differnet parts of the page map file, it is why I could only use *allocSize* bytes in virtual memory, but have size (512 * 1024 * 1024) bytes in working set (physical memory) -- but they do not have virtual memory address?

    I have done some further experiement, which is interesting. I resize the size of map buffer to be the same as of size variable of page map file (512 * 1024 * 1024), then I find the virtual bytes is (almost) the same as working set. :-)

    You can have a try and get hands-on experience how Windows treat virtual bytes and working set.

    Code:
    			map [sectionIndex] = 
    				MapViewOfFile(mapping,FILE_MAP_WRITE,(DWORD)(offset>>32),(DWORD)offset,allocSize);

    Quote Originally Posted by matsp View Post
    I too see no "problem" with the working set being larger than virtual bytes. It can happen in some situations, just like you've shown.

    Particularly since you are unmapping pages from a mapped view of a file - you previously touched those bytes, and now you are not. So they are part of the working set that you HAD, but no longer part of the virtual memory of the application (you just unmapped it, right).

    It's all statistics, and the main point is probably to look at trends, not the exact values.

    --
    Mats

    regards,
    George

  6. #6
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    There are probably heuristics in "mapViewOfFile" that limits the amount of pages that the file can be mapped into, but I'm not at all sure how file mapping works in Windows [I know a tiny bit more from the Linux world, but I can't say I'm an expert there either].

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  7. #7
    Registered User
    Join Date
    May 2006
    Posts
    1,579
    Thanks Mats,


    What do you mean the heuristics in "mapViewOfFile"? I have tested the 500M file mapping will result in 500M working set from working set performance counter, looks like there is either no you called limitation or the limitation is higher?

    Quote Originally Posted by matsp View Post
    There are probably heuristics in "mapViewOfFile" that limits the amount of pages that the file can be mapped into, but I'm not at all sure how file mapping works in Windows [I know a tiny bit more from the Linux world, but I can't say I'm an expert there either].

    --
    Mats

    regards,
    George

  8. #8
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    What I mean is that it's probably no strict limit [although there may be one].
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  9. #9
    Registered User
    Join Date
    May 2006
    Posts
    1,579
    Well, I agree. :-)


    BTW: how do you think my points in #5 of this thread? :-P


    Quote Originally Posted by matsp View Post
    What I mean is that it's probably no strict limit [although there may be one].

    regards,
    George

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Replies: 4
    Last Post: 01-13-2008, 02:14 AM
  2. Code Review for upcoming contest
    By Darryl in forum Contests Board
    Replies: 2
    Last Post: 02-28-2006, 02:39 PM
  3. Replies: 0
    Last Post: 02-21-2002, 06:05 PM
  4. opengl code not working
    By Unregistered in forum Windows Programming
    Replies: 4
    Last Post: 02-14-2002, 10:01 PM
  5. Exporting Object Hierarchies from a DLL
    By andy668 in forum C++ Programming
    Replies: 0
    Last Post: 10-20-2001, 01:26 PM