Multithreaded app logging

This is a discussion on Multithreaded app logging within the Linux Programming forums, part of the Platform Specific Boards category; Greetings to all. This is my first contribution here. I have been doing multithreaded work for years on a variety ...

  1. #1
    Registered User jeffcobb's Avatar
    Join Date
    Dec 2009
    Location
    Henderson, NV
    Posts
    875

    Post Multithreaded app logging

    Greetings to all. This is my first contribution here. I have been doing multithreaded work for years on a variety of platforms, most recently on Playstation 3/CellBE and Linux for Sony, not that it matters other than to point out its not Windows. One of the trickiest things we have come across is being able to troubleshoot logic problems in heavily multithreaded applications. For those not immediately wincing, the problem is that often the only thing you really have to work with due to platform or other issues is the dreaded printf() or writing to a log file. While both approaches work to an extent the real issue is that doing so causes the kernel to generate unwanted and artificial context switches, resulting in the app working one way with debug statements and another way without. How many times has it happened that you had some nasty bug that you finally trap with printf() or log messages, fix the bug, build in release mode and suddenly the app works completely different? This is what I refer to as the "observer effect" for lack of anything original.

    I have spent a lot of time thinking about what could be done for this problem and I think I have come up with a simple germ of an idea to help with it. The system is not by any means complete, more of a proof of concept right now but I think that it has promise. The next real-world system I have to work on I will put it to the test. So how does it work?

    In short I had the idea of a passive logging server that provided each out-of-process client a unique memory block to play in, used as a circular queue. Since each client then just pushes data (log msgs) to a slot in memory, no synchronization is needed. More, the messages themselves are tokenized such that each one uses a predictable block of memory and the write can be accomplished with a simple memcpy(). How they are tokenized is beyond the scope of this post but is spelt out more in the article on my website (link to that and demo code at bottom of post). The server itself is never instantiated directly; rather the first client instantiated creates the server and as the last client goes out of scope (in theory at the close of the application) the server is destroyed. As each successive client is created the server allocates a block of memory for that particular client to use. As part of destruction the server iterates through the allocated blocks, converting the tokenized logs into textual messages and writes them all to a disk-file, aggregating them in time-stamp order.

    I have a more long-winded article on my blog along with a link to the source code. FWIW it is GPL 3.0. You will need cmake to built the demo app which shows general message construction and how it works in a heavier multithreaded environment. Caveat: the exact message creation/tokenization is still very alpha; I think I will be able to make it more "real world" once I use it a few times in something other than a test-harness. In any event this code and article should explain the direction I am taking on this and would be interested in the thoughts of other folks used to dealing with these kinds of things...

    Cheers
    Jeff
    Microlog 1.0a
    Last edited by jeffcobb; 12-13-2009 at 01:29 PM.

  2. #2
    Registered User jeffcobb's Avatar
    Join Date
    Dec 2009
    Location
    Henderson, NV
    Posts
    875
    Geez thanks to whoever posted my entire message here as a comment on my site, did a lot of good... :-/

    Jeff

  3. #3
    Dae
    Dae is offline
    Deprecated Dae's Avatar
    Join Date
    Oct 2004
    Location
    Canada
    Posts
    1,034
    Quote Originally Posted by jeffcobb View Post
    Geez thanks to whoever posted my entire message here as a comment on my site, did a lot of good... :-/

    Jeff
    Pretty sure that's a WordPress feature. When it gets a hit, it checks the referer, see's it's mentioning your post, and adds the comment as a linkback. It assumes someone is commenting on your article (happened to be you) on another site, so it adds it as a comment. Never seen that before? Usually the snippet is smaller... but your link was isolated at the bottom so it has no context. Pretty sure it can be turned off, or deleted.
    Warning: Have doubt in anything I post.

    GCC 4.5, Boost 1.40, Code::Blocks 8.02, Ubuntu 9.10 010001000110000101100101

  4. #4
    Registered User jeffcobb's Avatar
    Join Date
    Dec 2009
    Location
    Henderson, NV
    Posts
    875
    Quote Originally Posted by Dae View Post
    Pretty sure that's a WordPress feature. When it gets a hit, it checks the referer, see's it's mentioning your post, and adds the comment as a linkback. It assumes someone is commenting on your article (happened to be you) on another site, so it adds it as a comment. Never seen that before? Usually the snippet is smaller... but your link was isolated at the bottom so it has no context. Pretty sure it can be turned off, or deleted.
    Didn't know that. The Quoting stuff I have seen but it was always just a line or two...I left it in there since I don't delete comments unless it is obviously spam or derogatory...

    Thanks for the tip......

    Jeff
    C/C++ Environment: GNU CC/Emacs
    Make system: CMake
    Debuggers: Valgrind/GDB

  5. #5
    Frequently Quite Prolix dwks's Avatar
    Join Date
    Apr 2005
    Location
    Canada
    Posts
    8,045
    I just wanted to say that I really like this project. I think it's a great idea.

    Just an idea: I think it's important for messages to have priority. In one of my more elaborate message logging systems, each message was assigned a type; some types clearly had an ordering of importance, but others did not. I was able to specify exactly which types of messages I wanted to see: masking out unimportant message types, etc. If you implemented something like this, you could streamline exactly the messages you needed, and reduce the impact of logging on system behaviour even more.
    dwk

    Seek and ye shall find. quaere et invenies.

    "Simplicity does not precede complexity, but follows it." -- Alan Perlis
    "Testing can only prove the presence of bugs, not their absence." -- Edsger Dijkstra
    "The only real mistake is the one from which we learn nothing." -- John Powell


    Other boards: DaniWeb, TPS
    Unofficial Wiki FAQ: cpwiki.sf.net

    My website: http://dwks.theprogrammingsite.com/
    Projects: codeform, xuni, atlantis, nort, etc.

  6. #6
    Registered User jeffcobb's Avatar
    Join Date
    Dec 2009
    Location
    Henderson, NV
    Posts
    875
    Quote Originally Posted by dwks View Post
    I just wanted to say that I really like this project. I think it's a great idea.

    Just an idea: I think it's important for messages to have priority. In one of my more elaborate message logging systems, each message was assigned a type; some types clearly had an ordering of importance, but others did not. I was able to specify exactly which types of messages I wanted to see: masking out unimportant message types, etc. If you implemented something like this, you could streamline exactly the messages you needed, and reduce the impact of logging on system behaviour even more.
    Good point and considering I had that in just about every other logging system I have written, I am ashamed to say I missed it here...forest, meet trees. I was so into the message digest thing and seeing how far I could go w/o locks and such disturbing the natural order of things that I missed one of the most basic things....Oh well, v 1.1 on its way ^__^

    Jeff
    C/C++ Environment: GNU CC/Emacs
    Make system: CMake
    Debuggers: Valgrind/GDB

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. non-MFC DLL with MFC app question.
    By Kempelen in forum Windows Programming
    Replies: 10
    Last Post: 08-20-2008, 07:11 AM
  2. best program to start
    By gooddevil in forum Networking/Device Communication
    Replies: 4
    Last Post: 05-28-2004, 05:56 PM
  3. Need help migrating console app to windows app
    By DelphiGuy in forum C++ Programming
    Replies: 1
    Last Post: 03-14-2004, 06:05 PM
  4. pasword app
    By GanglyLamb in forum C Programming
    Replies: 2
    Last Post: 06-07-2003, 10:28 AM
  5. How do I make my Linux app standalone?
    By Joda in forum C++ Programming
    Replies: 2
    Last Post: 11-27-2002, 03:53 AM

Tags for this Thread


1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21