Replacing System Calls in 2.6 (opinions from the kernel hackers here?)

Hey all,

So it's been a while since I posted. In what little time I've had for personal projects I've been doing a lot of reading. I'll probably get back to actual coding pretty soon (this is still part of my very long-term project to make my own distro) - and one snag I'm gonna have to solve pretty early on is replacing some system calls. It used to be nice and easy, apparently (and I got my hopes up before finding out about this change), but the system call table is no longer exported (as of 2.6).

I've been looking around, and it sounds like I have a few other options. If you have any opinions - please share! Despite all the reading, I still feel very new to this. Also, if I'm missing a good solution, please let me know.

1. There's a kernel patch that will export the table anyway, but I'd really like to avoid the idea of needing a recompiled kernel in order to actually use this part of my code. Doing this part at run-time is a big priority of mine - so I hope it's possible.

2. I've seen several algorithms for calculating the location of the call table. It's extremely hacky, and only works on x86 (not that I care about that part, really). Any opinions on this? I haven't tried it, but would that be reliable? I'm okay with hacky code as long as it's reliable! (edit: after some more searching I've found that this is NOT reliable. Some of the items used in tracking down the call table have moved, and I'm sure any future similiar algorithm would have the same vulnerability)

3. I've heard a couple of people recommend security modules for intercepting system calls, but haven't heard much about them before. Is this worth investigating further? To me it sounds the hooks just let you know when a system call is called. It doesn't actually let you override that call entirely. Am I wrong? And if so, what do you recommend for learning about them? Online information seems to be scarce in that area.

Thanks for any advice!

Quote:

---

Originally Posted by sean

(this is still part of my very long-term project to make my own distro)

---

Now, I personally have done the whole make a distro before -- cross compiled Slackware 12.0 to ia64 -- I like Slackware and wanted a 64 bit package. . . it worked (mostly).

The question I have, though, is why do you need to replace the system table? I mean, I can show you how to do this in a kernel module, but why? What do you gain from doing this?

Well I'm not making my own distro because I think it would be my dream every-day use system. I just had a bunch of ideas and this was a convenient way to tie them all together in a long-term project while I was in school. The specific idea I'm working on now, is to have a network stack that could change it's appearance at run-time. I don't know if anyone would find that useful - but I thought it would be a cool tool for learning about networking, and maybe even for setting up honeypots, or various scenarios on a network, etc... So for instance, if you run nmap, it tries a few tests to decide if it's talking to a Windows stack, or a Linux stack or what ever. I wanted to set up a system where you could load a module that would then function in such a way that it responded to each tests the way some other stack did. Each module would probably be pretty big of course, but I'm trying to come up with a design that would abstract as much of the process out as possible, so each module just has to implement a certain framework, and worry about it's unique parts only.

The problem with that is not only the inherent complexity of writing a new TCP/IP stack to implement all the networking system calls in Linux, but also things like - what happens if the user switches stacks while a packet is being processed? So I'm trying to make it as modular an interchangeable as possible, but that might turn out to be ridiculous and impossible.

It seems that since I'm going to have to make some pretty significant changes to the kernel anyway, I might just go ahead and write a patch the implements my framework in the kernel, and then the individual stack could be implemented as modules and changed out at run-time.

But now I'm just rambling... So that's basically what I'm trying to do. The idea of changing the system call table was simply to intercept any requests to create new sockets, read and send data, etc... To me it seems even more hacky to try and intercept that and block it with LSMs.

Intercepting process system calls can be done completely from userspace using ptrace().

Yeah, it's gotta be system wide though. So if they inserted my modules, and then ran apache, IM, whatever, I'd need to intercept all networking calls from all processes, even ones I wasn't the parent of. That's why the idea of rewriting the call table was so appealing to me.

Quote:

---

Originally Posted by sean

Yeah, it's gotta be system wide though. So if they inserted my modules, and then ran apache, IM, whatever, I'd need to intercept all networking calls from all processes, even ones I wasn't the parent of. That's why the idea of rewriting the call table was so appealing to me.

---

As long as the programs you are interested are "normal," i.e., they use the C library, then you can override system calls by hooking the C library. You could distribute a customized C library with your product, or you could dynamically hook the functions using the LD_PRELOAD technique.

Of course, if you really want a kernel solution I don't think it's a terrible choice.

Quote:

---

Originally Posted by brewbuck

As long as the programs you are interested are "normal," i.e., they use the C library, then you can override system calls by hooking the C library. You could distribute a customized C library with your product, or you could dynamically hook the functions using the LD_PRELOAD technique.

Of course, if you really want a kernel solution I don't think it's a terrible choice.

---

Looking at what he intends to achieve, I can't see what he would gain by doing those user level modifications.
I can't understand exactly what he wants to do(I mean, if you want to test how the windows tcp/ip stack responds to input, just do the test on a machine with windows installed. I'm saying this mostly because your project seems huge!), but it seems that he actually needs to make big changes at kernel level, of which the system calls interception seems the smallest one.