Enforcing Machine Code Restrictions?
I'm currently looking at creating a sandboxed environment for running code, but I don't necessarily wanna go down the whole VM route. Cross-platform usage is not a concern at this point. What I want to do is create a situation where you can compile the usual (in this case, x86) code and run it via a sort of loader which completely separates what you do from the operating system.
"You've described a VM you ignorant (insert bizarre foreign colloquialism here)!", you've no doubt said to yourself. Ah, but i want to be cleverererer than that! ;)
If we look at the Win32 environment specifically; let's say that I have a loader that effectively just reads your code from a file, shoves into an executable page in memory and starts a thread at its entry point. Your code isn't in PE format, you can't statically link to any Win32 functions.
So you're running in my loader's address space... but there's a problem. Although you can't import any Win32 functions, because your code is just x86 you can attempt to look at a location in the address space that you think contains KERNEL32.DLL (because every Windows program has this in its address space). From there, you can walk the export table and effectively link to the functions you need to break out of this dump and do some REAL stuff...
Now, Address Space Layout Randomization (which was introduced in Vista) mitigates this to some extent, but what I would like to do is pre-empt code from getting this far.
The best way I can think of doing this is by scanning the code prior to execution for particular instructions (like CALL for example) and analyzing the context to see if badness is being attempted. This would work swimmingly for an immediate operand, but would be harder to do with registers or indirect memory locations without effectively interpreting the code (which I want to avoid!).
One way I could perhaps alleviate this is to enforce (via software) separation of code and data segments, so that no instruction can read or write memory in the code segment. I suppose I could extend this to "locking down" the segment registers so that if instructions like "POP CS" are detected then the program can be stopped before execution. The only thing is, is there any difference between the CS and DS registers in most Win32 programs?
I know I'm in way over my head here, so I'll keep the question simple: can I do what I want to do without resorting to interpretation? Am I wasting my time even thinking about this?