Trackin Memory Accesses

This is a discussion on Trackin Memory Accesses within the Linux Programming forums, part of the Platform Specific Boards category; I need to write a C program to track memory reads and writes of an application running in linux. I'm ...

  1. #1
    Registered User
    Join Date
    Jul 2009
    Posts
    4

    Question Trackin Memory Accesses

    I need to write a C program to track memory reads and writes of an application running in linux. I'm doing this caz I need to characterize the memory usage pattern for the application.

    The big question is: how can I know/track through a C program that memory has been accessed (r/w) by some other application.

    Any help will be appreciated. Thanks

  2. #2
    Captain Crash brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,251
    Quote Originally Posted by lsolano View Post
    I need to write a C program to track memory reads and writes of an application running in linux. I'm doing this caz I need to characterize the memory usage pattern for the application.

    The big question is: how can I know/track through a C program that memory has been accessed (r/w) by some other application.

    Any help will be appreciated. Thanks
    This is an extremely difficult problem. Programs like Valgrind solve this problem by simulating the execution of instructions -- you may find that the easiest path to what you want is to use, or extend, Valgrind.

    I can imagine a method which could work without emulating the program or instrumenting it in any way, but it's likely to be thousands of times slower than simply using Valgrind. If you really, really want to hear this (incredibly inefficient) method, I'll expound
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  3. #3
    Registered User
    Join Date
    Jul 2009
    Posts
    4
    Thanks brewbuck, I kind of had in mind that it would be a difficult problem. Please go ahead and explain the method you are thinking of.
    However, let me tell you what I'm exactly trying to do: I have an application that generates many threads; each thread does exactly the same, but over different pieces of data stored in memory. What I need to do, is to track memory to profile the pattern for memory reads and writes for that application, in such a way that I can compute required bandwidth as well as idle times in memory.
    Last edited by lsolano; 07-28-2009 at 01:45 PM. Reason: giving more details

  4. #4
    Captain Crash brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,251
    Quote Originally Posted by lsolano View Post
    Thanks brewbuck, I kind of had in mind that it would be a difficult problem. Please go ahead and explain the method you are thinking of.
    1. Use ptrace() to attach to the process and wait for it to suspend.
    2. Parse the /proc/XXX/maps file to get a memory map of the process
    3. Find a small page with execute permissions and inject a stub function into it. This stub function will call mmap() to allocate another page of code memory, then suspend itself with a kill( SIGSTOP )
    4. Twiddle the registers to cause the stub to execute, then restart the process.
    5. Wait for the process to stop again.
    6. Into the newly mapped memory page, write another stub which allows you to invoke mprotect(). Also, restore the data that was previously in the code page which you overwrote.
    7. Call this stub repeatedly (by register fiddling), to mprotect() the entire process address space to be unreadable/unwritable/unexecutable
    8. Restart the process where it left off.
    9. Wait for the process to access memory, at which time it will receive a SIGSEGV
    10. Examine the address which caused the fault. Log it somewhere.
    11. Invoke the mprotect() stub in the process to cause the faulting page to become readable.
    12. Use ptrace() to single step the process by one instruction.
    13. Wait for suspension again, then invoke the mprotect() stub to set the page back to no permissions.
    14. Go to step 8.

    This involves multiple system calls and page table manipulations for every single memory access. It's going to be ludicrously slow, but it should work in theory. You'd be better off learning about libVEX from Valgrind and trying to extend it to do what you want. There are mailing lists where people discuss such Valgrind extensions.
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  5. #5
    Guest Sebastiani's Avatar
    Join Date
    Aug 2001
    Location
    Waterloo, Texas
    Posts
    5,708
    Quote Originally Posted by brewbuck View Post
    1. Use ptrace() to attach to the process and wait for it to suspend.
    2. Parse the /proc/XXX/maps file to get a memory map of the process
    3. Find a small page with execute permissions and inject a stub function into it. This stub function will call mmap() to allocate another page of code memory, then suspend itself with a kill( SIGSTOP )
    4. Twiddle the registers to cause the stub to execute, then restart the process.
    5. Wait for the process to stop again.
    6. Into the newly mapped memory page, write another stub which allows you to invoke mprotect(). Also, restore the data that was previously in the code page which you overwrote.
    7. Call this stub repeatedly (by register fiddling), to mprotect() the entire process address space to be unreadable/unwritable/unexecutable
    8. Restart the process where it left off.
    9. Wait for the process to access memory, at which time it will receive a SIGSEGV
    10. Examine the address which caused the fault. Log it somewhere.
    11. Invoke the mprotect() stub in the process to cause the faulting page to become readable.
    12. Use ptrace() to single step the process by one instruction.
    13. Wait for suspension again, then invoke the mprotect() stub to set the page back to no permissions.
    14. Go to step 8.

    This involves multiple system calls and page table manipulations for every single memory access. It's going to be ludicrously slow, but it should work in theory. You'd be better off learning about libVEX from Valgrind and trying to extend it to do what you want. There are mailing lists where people discuss such Valgrind extensions.
    Interesting solution.
    Code:
    #include <cmath>
    #include <complex>
    bool euler_flip(bool value)
    {
        return std::pow
        (
            std::complex<float>(std::exp(1.0)), 
            std::complex<float>(0, 1) 
            * std::complex<float>(std::atan(1.0)
            *(1 << (value + 2)))
        ).real() < 0;
    }

  6. #6
    Registered User
    Join Date
    Nov 2008
    Posts
    75
    Quote Originally Posted by lsolano View Post
    Thanks brewbuck, I kind of had in mind that it would be a difficult problem. Please go ahead and explain the method you are thinking of.
    However, let me tell you what I'm exactly trying to do: I have an application that generates many threads; each thread does exactly the same, but over different pieces of data stored in memory. What I need to do, is to track memory to profile the pattern for memory reads and writes for that application, in such a way that I can compute required bandwidth as well as idle times in memory.
    I may be completely misinterpreting your requirements, so I may be completely wrong about this, but couldn't you simply modify the code of the app to log those reads and writes? Like a debugging wrapper of the functions used, or something like that?

  7. #7
    Captain Crash brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,251
    Quote Originally Posted by MisterIO View Post
    I may be completely misinterpreting your requirements, so I may be completely wrong about this, but couldn't you simply modify the code of the app to log those reads and writes? Like a debugging wrapper of the functions used, or something like that?
    That might be an option. If C++, you can create a smart pointer class which acts like a raw pointer but logs all dereferences. Using this simple method though, it isn't possible to distinguish between reads and writes. You could create a proxy object which behaves like a reference which could distinguish between reads and writes, though.
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  8. #8
    Registered User
    Join Date
    Jul 2009
    Posts
    4
    Quote Originally Posted by MisterIO View Post
    I may be completely misinterpreting your requirements, so I may be completely wrong about this, but couldn't you simply modify the code of the app to log those reads and writes? Like a debugging wrapper of the functions used, or something like that?
    That is not possible because the threads are running at the "same time", and hence the actual memory reads/writes are done when the memory is available, which is not necessarily the same time as the read/write code request. Therefore, if I modify the code to log the times for those reads and writes, those times won't be the actual times when the read/write operation took place.

  9. #9
    Registered User
    Join Date
    Jul 2009
    Posts
    4
    Quote Originally Posted by brewbuck View Post
    1. Use ptrace() to attach to the process and wait for it to suspend.
    2. Parse the /proc/XXX/maps file to get a memory map of the process
    3. Find a small page with execute permissions and inject a stub function into it. This stub function will call mmap() to allocate another page of code memory, then suspend itself with a kill( SIGSTOP )
    4. Twiddle the registers to cause the stub to execute, then restart the process.
    5. Wait for the process to stop again.
    6. Into the newly mapped memory page, write another stub which allows you to invoke mprotect(). Also, restore the data that was previously in the code page which you overwrote.
    7. Call this stub repeatedly (by register fiddling), to mprotect() the entire process address space to be unreadable/unwritable/unexecutable
    8. Restart the process where it left off.
    9. Wait for the process to access memory, at which time it will receive a SIGSEGV
    10. Examine the address which caused the fault. Log it somewhere.
    11. Invoke the mprotect() stub in the process to cause the faulting page to become readable.
    12. Use ptrace() to single step the process by one instruction.
    13. Wait for suspension again, then invoke the mprotect() stub to set the page back to no permissions.
    14. Go to step 8.

    This involves multiple system calls and page table manipulations for every single memory access. It's going to be ludicrously slow, but it should work in theory. You'd be better off learning about libVEX from Valgrind and trying to extend it to do what you want. There are mailing lists where people discuss such Valgrind extensions.
    Thanks

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. available memory from task manager
    By George2 in forum Tech Board
    Replies: 10
    Last Post: 01-18-2008, 01:32 AM
  2. Replies: 4
    Last Post: 01-13-2008, 01:14 AM
  3. Question regarding Memory Leak
    By clegs in forum C++ Programming
    Replies: 29
    Last Post: 12-07-2007, 12:57 AM
  4. Memory problem with Borland C 3.1
    By AZ1699 in forum C Programming
    Replies: 16
    Last Post: 11-16-2007, 10:22 AM
  5. Shared Memory - shmget questions
    By hendler in forum C Programming
    Replies: 1
    Last Post: 11-29-2005, 01:15 AM

Tags for this Thread


1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21