Jochen Kalmbach's WebLog

Great post about "How system calls work"

Today there was a great post from Brian Catlin in the microsoft.public.windbg: Re: How to get past SYSENTER?

Here is the message:

The executable is calling a system service, which is a routine in the operating system, generally exported by the Executive, and is not part of your executable. Disassembling the operating system isn’t really going to get you anything, but since we’re on the topic, here’s how it all works:

NTDLL is used to call into the operating system, which is (generally) in the address range (0×80000000-0xFFFFFFFF). The operating system addresses are not accessible in user-mode; therefore a special protected mechanism (using a CPU instruction) is used to control the transition from user-mode to kernel-mode. NTDLL loads the system service number into the EAX register, then copies the address the processor-specific kernel-mode transition code on the Kernel-User shared page (0×7FFE0000 + 0×300) into the EDX register, then calls through the EDX register.

 MOV    EAX, Service Number
 MOV    EDX, MM_SHARED_USER_DATA_VA + UsSystemCall
 CALL   EDX
 RET    n

The processor-specific kernel-mode transition code depends upon whether the CPU is Intel, AMD or Pentium2 and earlier (Win2K and earlier). INT 2E vectors through the IDT (entry number 0×2E), while SYSCALL and SYSENTER vector through model-specific registers that are initialized at system boot time.

 Win2K and earlier:
 LEA    EDX, [ESP+4]
 INT    2E  ; Ends up calling KiSystemService
 RET

 WinXP and later (Intel):
 MOV    EDX, ESP
 SYSENTER  ; Ends up calling KiFastCallEntry, which then calls KiSystemService
 RET

 AMD K6 and later
 MOV    EDX, ESP
 SYSCALL  ; Ends up calling KiSystemCall, which then calls KiSystemService
 RET

KiSystemService uses the system service number(in EAX) as an index into the system service dispatch table (actually, there are up to 4), which contains the address of the routine in the operating system to call. This prevents an application from calling any random address in the system; an application can only call those routines that are listed in the system service dispatch table.

This is probably way more than you wanted to know, but once I get on a roll, it’s hard to stop.

Brian

There is also an article by John.Gulbrandsen: How do Windows NT System Calls REALLY work?