pete > courses > Crash Course in System Security (CSCI 1005), Winter 2025 > Day 03

Day 03: System Calls

goals:

why system calls?

what system calls?

how system calls?

Notes

The kernel controls access to shared resources (eg, hardware). Beyond improving the efficiency of accessing shared resources, the kernel also enforces permissions. System calls are the method by which programs ask the kernel to do stuff on their behalf.

A simple program to call a simple system call: exit.c.

Compile it and run it:

$ gcc -o exit exit.c
$ ./exit

We can observe the system calls it invokes:

$ strace ./exit

Note that the second to last line says it’s calling the exit_group system call with the parameter 42. Success!

But in order for the kernel to mediate access to shared resources, the processor needs to be able to differentiate between instructions issued by unprivileged processes and instructions issued by the privileged kernel. To that end, x86 processors have a 2-bit register that stores the permission level with which it is currently operating. These are colloquially (and were historically) referred to as "rings": unprivileged code ran in ring 3, privileged code ran in ring 0. One of the primary purposes of system calls is to change from ring 3 to ring 0. (Other architectures use a different nomenclature but the principle is the same: there is a register within the process that reflects the current privilege setting of the processor itself.)

System calls are documented in section 2 of the manual, which you can consult like so:

$ man 2 exit

The manpage for exit shows that the actual call is really _exit.

To see how the system call is actually invoked, run our stupid-simple program in gdb, break at _exit, and then disassemble the instructions to see what _exit does:

$ gdb exit
<gdb> break _exit
<gdb> run
<gdb> disas

We can see there is a syscall instruction!

We can read about how the syscall instruction works in the Intel Architecture documentation (page 1076 of this pdf).

The documentation says the CPU invokes a system-call handler (ie, function) that the kernel has registered with the hardware using the WRMSR instruction. We can look through the kernel source code to see where this registering happens:

$ grep -rin wrmsr.*lstar .
./arch/x86/kernel/cpu/common.c:2032:    wrmsrl(MSR_LSTAR, (unsigned long)entry_SYSCALL_64);

whither entry_SYSCALL_64 ?

$ grep -rin "\Wentry_SYSCALL_64\W" .
...
...

The syscall calling convention is described in the comments starting on line 49 of ./arch/x86/entry/entry_64.S.

Side note: the syscall instruction is new with the 64-bit variant of x86. The old-school way of invoking system calls is to use the int 0x80 instruction instead, where int is short for "interrupt", which is a kind of exceptional event that the processor handles in a special way (in this case, it transitions to the privileged operating mode and executes the syscall handler).

New(ish) Tools

Exercises

Beginning strace

Add another user to your system.

When logged in as this new user, attempt to read a file owned by your primary user. Note that, depending on the file’s permissions, it may fail.

As your primary user, create a file and give it permissions such that the new user will not be able to read it.

As the new user, use strace to figure out how failure is being communicated to the program attempting to read the file for which you do not have permission.

Finally, figure out how to do all of the above using sudo.

Manually invoke a system call

Here’s an assembly program that invokes the exit system call: exit.asm.

Before you can run it, you’ll need to assemble and link it:

$ nasm -f elf64 -o exit.bin exit.asm
$ ld -o exit exit.bin

Run this program and verify that it calls the exit system call.

Modify this program to invoke a different system call. (The system call table is in the file arch/x86/entry/syscalls/syscall_64.tbl). Verify that it invokes this system call.

Modify the program again to print a message to the standard output. The most straightforward way to accomplish this is by using the write system call. The file descriptor for standard out (ie, the output stream that displays to the terminal) is 1. You’ll also need to specify data to output, which you can do like so in your assembly program:

SECTION .data
foo: db "hello there",0xa,0

Then you can refer to foo in assembly instructions within the .text section.

Modify this program to print a message to the standard output and then exit with an exit code of your choosing.

Test how the kernel handles unexpected syscall arguments

System calls are, as we’ve seen, a primary method of interacting with the kernel. One aspect of the kernel’s security we might want to test is its ability to handle unexpected system calls or system calls with unexpected arguments. (If the kernel fails to correctly handle a particular system call with particular arguments, we may have found a vulnerability.) Figuring out what system calls and system call arguments might break the kernel is a difficult task; one way we could make a start on this is to send a bunch of random(ish) as parameters and see if it the kernel crashes.

Write a program that creates a bunch of assembly programs, each of which invokes a different system call with different, random arguments. It should then assemble them, link them, attempt to run them, and summarize the results.

(This is an example of fuzz testing.)

Last modified: