Thread: Replacing System Calls in 2.6 (opinions from the kernel hackers here?)

  1. #1
    Registered User
    Join Date
    Sep 2001
    Posts
    4,912

    Replacing System Calls in 2.6 (opinions from the kernel hackers here?)

    Hey all,

    So it's been a while since I posted. In what little time I've had for personal projects I've been doing a lot of reading. I'll probably get back to actual coding pretty soon (this is still part of my very long-term project to make my own distro) - and one snag I'm gonna have to solve pretty early on is replacing some system calls. It used to be nice and easy, apparently (and I got my hopes up before finding out about this change), but the system call table is no longer exported (as of 2.6).

    I've been looking around, and it sounds like I have a few other options. If you have any opinions - please share! Despite all the reading, I still feel very new to this. Also, if I'm missing a good solution, please let me know.

    1. There's a kernel patch that will export the table anyway, but I'd really like to avoid the idea of needing a recompiled kernel in order to actually use this part of my code. Doing this part at run-time is a big priority of mine - so I hope it's possible.

    2. I've seen several algorithms for calculating the location of the call table. It's extremely hacky, and only works on x86 (not that I care about that part, really). Any opinions on this? I haven't tried it, but would that be reliable? I'm okay with hacky code as long as it's reliable! (edit: after some more searching I've found that this is NOT reliable. Some of the items used in tracking down the call table have moved, and I'm sure any future similiar algorithm would have the same vulnerability)

    3. I've heard a couple of people recommend security modules for intercepting system calls, but haven't heard much about them before. Is this worth investigating further? To me it sounds the hooks just let you know when a system call is called. It doesn't actually let you override that call entirely. Am I wrong? And if so, what do you recommend for learning about them? Online information seems to be scarce in that area.

    Thanks for any advice!

  2. #2
    {Jaxom,Imriel,Liam}'s Dad Kennedy's Avatar
    Join Date
    Aug 2006
    Location
    Alabama
    Posts
    1,065
    Quote Originally Posted by sean
    (this is still part of my very long-term project to make my own distro)
    Now, I personally have done the whole make a distro before -- cross compiled Slackware 12.0 to ia64 -- I like Slackware and wanted a 64 bit package. . . it worked (mostly).

    The question I have, though, is why do you need to replace the system table? I mean, I can show you how to do this in a kernel module, but why? What do you gain from doing this?

  3. #3
    Registered User
    Join Date
    Sep 2001
    Posts
    4,912
    Well I'm not making my own distro because I think it would be my dream every-day use system. I just had a bunch of ideas and this was a convenient way to tie them all together in a long-term project while I was in school. The specific idea I'm working on now, is to have a network stack that could change it's appearance at run-time. I don't know if anyone would find that useful - but I thought it would be a cool tool for learning about networking, and maybe even for setting up honeypots, or various scenarios on a network, etc... So for instance, if you run nmap, it tries a few tests to decide if it's talking to a Windows stack, or a Linux stack or what ever. I wanted to set up a system where you could load a module that would then function in such a way that it responded to each tests the way some other stack did. Each module would probably be pretty big of course, but I'm trying to come up with a design that would abstract as much of the process out as possible, so each module just has to implement a certain framework, and worry about it's unique parts only.

    The problem with that is not only the inherent complexity of writing a new TCP/IP stack to implement all the networking system calls in Linux, but also things like - what happens if the user switches stacks while a packet is being processed? So I'm trying to make it as modular an interchangeable as possible, but that might turn out to be ridiculous and impossible.

    It seems that since I'm going to have to make some pretty significant changes to the kernel anyway, I might just go ahead and write a patch the implements my framework in the kernel, and then the individual stack could be implemented as modules and changed out at run-time.

    But now I'm just rambling... So that's basically what I'm trying to do. The idea of changing the system call table was simply to intercept any requests to create new sockets, read and send data, etc... To me it seems even more hacky to try and intercept that and block it with LSMs.

  4. #4
    Officially An Architect brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,396
    Intercepting process system calls can be done completely from userspace using ptrace().
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  5. #5
    Registered User
    Join Date
    Sep 2001
    Posts
    4,912
    Yeah, it's gotta be system wide though. So if they inserted my modules, and then ran apache, IM, whatever, I'd need to intercept all networking calls from all processes, even ones I wasn't the parent of. That's why the idea of rewriting the call table was so appealing to me.

  6. #6
    Officially An Architect brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,396
    Quote Originally Posted by sean View Post
    Yeah, it's gotta be system wide though. So if they inserted my modules, and then ran apache, IM, whatever, I'd need to intercept all networking calls from all processes, even ones I wasn't the parent of. That's why the idea of rewriting the call table was so appealing to me.
    As long as the programs you are interested are "normal," i.e., they use the C library, then you can override system calls by hooking the C library. You could distribute a customized C library with your product, or you could dynamically hook the functions using the LD_PRELOAD technique.

    Of course, if you really want a kernel solution I don't think it's a terrible choice.
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  7. #7
    Registered User
    Join Date
    Nov 2008
    Posts
    75
    Quote Originally Posted by brewbuck View Post
    As long as the programs you are interested are "normal," i.e., they use the C library, then you can override system calls by hooking the C library. You could distribute a customized C library with your product, or you could dynamically hook the functions using the LD_PRELOAD technique.

    Of course, if you really want a kernel solution I don't think it's a terrible choice.
    Looking at what he intends to achieve, I can't see what he would gain by doing those user level modifications.
    I can't understand exactly what he wants to do(I mean, if you want to test how the windows tcp/ip stack responds to input, just do the test on a machine with windows installed. I'm saying this mostly because your project seems huge!), but it seems that he actually needs to make big changes at kernel level, of which the system calls interception seems the smallest one.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. formatted system calls
    By sm00t in forum C Programming
    Replies: 1
    Last Post: 04-29-2004, 11:56 PM
  2. Linux Media Player
    By joshdick in forum A Brief History of Cprogramming.com
    Replies: 43
    Last Post: 09-07-2003, 08:08 AM
  3. System calls
    By Mak in forum C Programming
    Replies: 1
    Last Post: 02-06-2003, 09:49 AM
  4. System Calls
    By Jperensky in forum C Programming
    Replies: 6
    Last Post: 03-12-2002, 02:41 PM
  5. System Calls && Variables
    By Okiesmokie in forum C++ Programming
    Replies: 6
    Last Post: 03-06-2002, 09:10 PM