Thread: load user-mode dll into kernel-mode "process"

You can not call user-mode code from kernel mode drivers. You need a service process that acts as the conduit from user-mode to kernel-mode and vice versa.

It is technically possible, in x86, to write code that can be run in both user-mode and kernel-mode, but there are LOTS of obstacles that get in the way if you haven't got full control over the whole OS (such as memory mapping, segment register setup, etc). For example, kernel-mode code may expect the memory to be locked into memory - there is no way in Windows to say "lock the user-mode code so that it doesn't get swapped out". For all you know, the user-mode process may not even be active any longer when you try to call it if your kernel driver is any way asynchronous.

--
Mats

Originally Posted by Yarin

I'm not surprised, device drivers are a pain to write, why not add to it?

Thanks.

Well, really there is a boundary between user and kernel mode, and that is there for a reason. Allowing kernel code to call back into user-code would make a hole big enough to run several buses side-by-side through into the kernel. What if your callback into the DLL leads to your code being allowed to modify the kernel content - then there is absolutely no protection against that sort of thing AT ALL.

--
Mats

Originally Posted by Yarin

Makes sense.
But...
Recently I made a custom PE loader, and when testing it I noticed that kernel32 imports from ntdll. So, how does a user-mode DLL make kernel-mode function calls?

Through any one of the standard mechanisms for doing that. Software interrupts, call gates, task gates, etc.

Windows uses system calls by INT 2Eh for it's system calls in the standar version of the NTDLL.DLL. There are, as brewbuck says, several ways to solve this problem in x86:
- Call gates - a descriptor in the GDT (or possibly ldt) holds an entry for where the system calls go.
- Interrupt gates - an entry in the IDT holds the entry for the system calls.
- Task gate - similar to call-gates/interrupt-gates, but CPU will save task-state. I'm not aware of any OS that uses task-gates except for very special things [stack/double fault handling in Linux and Windows uses a task-gate to get a new stack pointer so that the OS can handle running out of stack or other "catastrophic" error like that].
- SYSCALL/SYSENTER - special model specific register hold the system call entry points.

--
Mats

Originally Posted by matsp

Windows uses system calls by INT 2Eh for it's system calls in the standar version of the NTDLL.DLL. There are, as brewbuck says, several ways to solve this problem in x86:
- Call gates - a descriptor in the GDT (or possibly ldt) holds an entry for where the system calls go.
- Interrupt gates - an entry in the IDT holds the entry for the system calls.
- Task gate - similar to call-gates/interrupt-gates, but CPU will save task-state. I'm not aware of any OS that uses task-gates except for very special things [stack/double fault handling in Linux and Windows uses a task-gate to get a new stack pointer so that the OS can handle running out of stack or other "catastrophic" error like that].
- SYSCALL/SYSENTER - special model specific register hold the system call entry points.

--
Mats

One method which I haven't seen used (probably because it's inefficient) is to simply have usermode call directly to a kernel address -- the resulting fault triggers a lookup of the faulting address in a table. If the address corresponds to the entry point of a kernel API the kernel twiddles the CPL in the TSS and then restarts the call, also replacing the return value on the stack with the address of a trampoline which simply does the inverse in order to return control back to userspace.

Sounds sick, but from userspace it looks like you're simply making a call directly to kernelspace.

Originally Posted by brewbuck

One method which I haven't seen used (probably because it's inefficient) is to simply have usermode call directly to a kernel address -- the resulting fault triggers a lookup of the faulting address in a table. If the address corresponds to the entry point of a kernel API the kernel twiddles the CPL in the TSS and then restarts the call, also replacing the return value on the stack with the address of a trampoline which simply does the inverse in order to return control back to userspace.

Sounds sick, but from userspace it looks like you're simply making a call directly to kernelspace.

In theory you could do that - but it is, as you say, inefficient - not so much for the lookup part [there's probably some sort of lookup anyways, and if you do it right, you can just do some initial range (value must be between x and y) and then a shift/mask operation to get an index into a table. But the combination of fault handling being not so nice to (modern) processors:

• because it will continue executing several more instructions in the mean-time, then having to "roll back" to the faulting call instruction.
• Modifying the content of the stack-frame is no good for the "return stack optimisation" - this leads to "branch mispredict" handling, and the resulting pipeline flush and refill.

--
Mats