C Board  

Go Back   C Board > Platform Specific Boards > Linux Programming

Reply
 
LinkBack Thread Tools Display Modes
Old 07-28-2009, 01:14 PM   #1
Registered User
 
Join Date: Jul 2009
Posts: 4
Question Trackin Memory Accesses

I need to write a C program to track memory reads and writes of an application running in linux. I'm doing this caz I need to characterize the memory usage pattern for the application.

The big question is: how can I know/track through a C program that memory has been accessed (r/w) by some other application.

Any help will be appreciated. Thanks
lsolano is offline   Reply With Quote
Old 07-28-2009, 01:21 PM   #2
Senior software engineer
 
brewbuck's Avatar
 
Join Date: Mar 2007
Location: Portland, OR
Posts: 5,381
Quote:
Originally Posted by lsolano View Post
I need to write a C program to track memory reads and writes of an application running in linux. I'm doing this caz I need to characterize the memory usage pattern for the application.

The big question is: how can I know/track through a C program that memory has been accessed (r/w) by some other application.

Any help will be appreciated. Thanks
This is an extremely difficult problem. Programs like Valgrind solve this problem by simulating the execution of instructions -- you may find that the easiest path to what you want is to use, or extend, Valgrind.

I can imagine a method which could work without emulating the program or instrumenting it in any way, but it's likely to be thousands of times slower than simply using Valgrind. If you really, really want to hear this (incredibly inefficient) method, I'll expound
__________________
"Congratulations on your purchase. To begin using your quantum computer, set the power switch to both off and on simultaneously." -- raftpeople@slashdot
brewbuck is offline   Reply With Quote
Old 07-28-2009, 01:37 PM   #3
Registered User
 
Join Date: Jul 2009
Posts: 4
Thanks brewbuck, I kind of had in mind that it would be a difficult problem. Please go ahead and explain the method you are thinking of.
However, let me tell you what I'm exactly trying to do: I have an application that generates many threads; each thread does exactly the same, but over different pieces of data stored in memory. What I need to do, is to track memory to profile the pattern for memory reads and writes for that application, in such a way that I can compute required bandwidth as well as idle times in memory.

Last edited by lsolano; 07-28-2009 at 01:45 PM. Reason: giving more details
lsolano is offline   Reply With Quote
Old 07-28-2009, 01:43 PM   #4
Senior software engineer
 
brewbuck's Avatar
 
Join Date: Mar 2007
Location: Portland, OR
Posts: 5,381
Quote:
Originally Posted by lsolano View Post
Thanks brewbuck, I kind of had in mind that it would be a difficult problem. Please go ahead and explain the method you are thinking of.
1. Use ptrace() to attach to the process and wait for it to suspend.
2. Parse the /proc/XXX/maps file to get a memory map of the process
3. Find a small page with execute permissions and inject a stub function into it. This stub function will call mmap() to allocate another page of code memory, then suspend itself with a kill( SIGSTOP )
4. Twiddle the registers to cause the stub to execute, then restart the process.
5. Wait for the process to stop again.
6. Into the newly mapped memory page, write another stub which allows you to invoke mprotect(). Also, restore the data that was previously in the code page which you overwrote.
7. Call this stub repeatedly (by register fiddling), to mprotect() the entire process address space to be unreadable/unwritable/unexecutable
8. Restart the process where it left off.
9. Wait for the process to access memory, at which time it will receive a SIGSEGV
10. Examine the address which caused the fault. Log it somewhere.
11. Invoke the mprotect() stub in the process to cause the faulting page to become readable.
12. Use ptrace() to single step the process by one instruction.
13. Wait for suspension again, then invoke the mprotect() stub to set the page back to no permissions.
14. Go to step 8.

This involves multiple system calls and page table manipulations for every single memory access. It's going to be ludicrously slow, but it should work in theory. You'd be better off learning about libVEX from Valgrind and trying to extend it to do what you want. There are mailing lists where people discuss such Valgrind extensions.
__________________
"Congratulations on your purchase. To begin using your quantum computer, set the power switch to both off and on simultaneously." -- raftpeople@slashdot
brewbuck is offline   Reply With Quote
Old 07-28-2009, 01:50 PM   #5
Guest
 
Sebastiani's Avatar
 
Join Date: Aug 2001
Posts: 4,923
Quote:
Originally Posted by brewbuck View Post
1. Use ptrace() to attach to the process and wait for it to suspend.
2. Parse the /proc/XXX/maps file to get a memory map of the process
3. Find a small page with execute permissions and inject a stub function into it. This stub function will call mmap() to allocate another page of code memory, then suspend itself with a kill( SIGSTOP )
4. Twiddle the registers to cause the stub to execute, then restart the process.
5. Wait for the process to stop again.
6. Into the newly mapped memory page, write another stub which allows you to invoke mprotect(). Also, restore the data that was previously in the code page which you overwrote.
7. Call this stub repeatedly (by register fiddling), to mprotect() the entire process address space to be unreadable/unwritable/unexecutable
8. Restart the process where it left off.
9. Wait for the process to access memory, at which time it will receive a SIGSEGV
10. Examine the address which caused the fault. Log it somewhere.
11. Invoke the mprotect() stub in the process to cause the faulting page to become readable.
12. Use ptrace() to single step the process by one instruction.
13. Wait for suspension again, then invoke the mprotect() stub to set the page back to no permissions.
14. Go to step 8.

This involves multiple system calls and page table manipulations for every single memory access. It's going to be ludicrously slow, but it should work in theory. You'd be better off learning about libVEX from Valgrind and trying to extend it to do what you want. There are mailing lists where people discuss such Valgrind extensions.
Interesting solution.
Sebastiani is offline   Reply With Quote
Old 07-28-2009, 01:53 PM   #6
Registered User
 
Join Date: Nov 2008
Posts: 75
Quote:
Originally Posted by lsolano View Post
Thanks brewbuck, I kind of had in mind that it would be a difficult problem. Please go ahead and explain the method you are thinking of.
However, let me tell you what I'm exactly trying to do: I have an application that generates many threads; each thread does exactly the same, but over different pieces of data stored in memory. What I need to do, is to track memory to profile the pattern for memory reads and writes for that application, in such a way that I can compute required bandwidth as well as idle times in memory.
I may be completely misinterpreting your requirements, so I may be completely wrong about this, but couldn't you simply modify the code of the app to log those reads and writes? Like a debugging wrapper of the functions used, or something like that?
MisterIO is offline   Reply With Quote
Old 07-28-2009, 01:59 PM   #7
Senior software engineer
 
brewbuck's Avatar
 
Join Date: Mar 2007
Location: Portland, OR
Posts: 5,381
Quote:
Originally Posted by MisterIO View Post
I may be completely misinterpreting your requirements, so I may be completely wrong about this, but couldn't you simply modify the code of the app to log those reads and writes? Like a debugging wrapper of the functions used, or something like that?
That might be an option. If C++, you can create a smart pointer class which acts like a raw pointer but logs all dereferences. Using this simple method though, it isn't possible to distinguish between reads and writes. You could create a proxy object which behaves like a reference which could distinguish between reads and writes, though.
__________________
"Congratulations on your purchase. To begin using your quantum computer, set the power switch to both off and on simultaneously." -- raftpeople@slashdot
brewbuck is offline   Reply With Quote
Old 07-28-2009, 02:04 PM   #8
Registered User
 
Join Date: Jul 2009
Posts: 4
Quote:
Originally Posted by MisterIO View Post
I may be completely misinterpreting your requirements, so I may be completely wrong about this, but couldn't you simply modify the code of the app to log those reads and writes? Like a debugging wrapper of the functions used, or something like that?
That is not possible because the threads are running at the "same time", and hence the actual memory reads/writes are done when the memory is available, which is not necessarily the same time as the read/write code request. Therefore, if I modify the code to log the times for those reads and writes, those times won't be the actual times when the read/write operation took place.
lsolano is offline   Reply With Quote
Old 07-28-2009, 02:15 PM   #9
Registered User
 
Join Date: Jul 2009
Posts: 4
Quote:
Originally Posted by brewbuck View Post
1. Use ptrace() to attach to the process and wait for it to suspend.
2. Parse the /proc/XXX/maps file to get a memory map of the process
3. Find a small page with execute permissions and inject a stub function into it. This stub function will call mmap() to allocate another page of code memory, then suspend itself with a kill( SIGSTOP )
4. Twiddle the registers to cause the stub to execute, then restart the process.
5. Wait for the process to stop again.
6. Into the newly mapped memory page, write another stub which allows you to invoke mprotect(). Also, restore the data that was previously in the code page which you overwrote.
7. Call this stub repeatedly (by register fiddling), to mprotect() the entire process address space to be unreadable/unwritable/unexecutable
8. Restart the process where it left off.
9. Wait for the process to access memory, at which time it will receive a SIGSEGV
10. Examine the address which caused the fault. Log it somewhere.
11. Invoke the mprotect() stub in the process to cause the faulting page to become readable.
12. Use ptrace() to single step the process by one instruction.
13. Wait for suspension again, then invoke the mprotect() stub to set the page back to no permissions.
14. Go to step 8.

This involves multiple system calls and page table manipulations for every single memory access. It's going to be ludicrously slow, but it should work in theory. You'd be better off learning about libVEX from Valgrind and trying to extend it to do what you want. There are mailing lists where people discuss such Valgrind extensions.
Thanks
lsolano is offline   Reply With Quote
Reply

Tags
linux, memory

Thread Tools
Display Modes

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
available memory from task manager George2 Tech Board 10 01-18-2008 02:32 AM
Still confused why working set larger than virtual memory George2 Tech Board 4 01-13-2008 02:14 AM
Question regarding Memory Leak clegs C++ Programming 29 12-07-2007 01:57 AM
Memory problem with Borland C 3.1 AZ1699 C Programming 16 11-16-2007 11:22 AM
Shared Memory - shmget questions hendler C Programming 1 11-29-2005 02:15 AM


All times are GMT -6. The time now is 07:18 AM.


Powered by vBulletin® Version 3.8.1
Copyright ©2000 - 2009, Jelsoft Enterprises Ltd.
Search Engine Optimization by vBSEO 3.3.0 RC2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22