pete > courses > CS 315 Fall 26 > lecture 06: memory bugs
Lecture 06: memory bugs
Goals
- explain what causes a segmentation fault
- explain what causes a memory leak
- debug memory-related bugs using gdb and valgrind
consider this program: segfault.c
if we compile (ignore the warning) and run it, it crashes:
$ gcc -Wall -pedantic -o segfault segfault.c $ ./segfault Type something: asdffdsa Segmentation fault (core dumped)
why?
one of the most common runtime errors one encounters when writing systems code is the dreaded segfault
(whose full name is "segmentation violation", for reasons that have now been lost in the depths of time, but have to do with how really early x86 processors used registers to access memory)
recall that UNIX recognizes three different operations one can perform on a file—read, write, and execute—and allows different permissions for each
UNIX recognizes the exact same set of operations in relation to memory
therefore, one can read memory (ie, perform a load, in architecture parlance)
or write memory (ie, a store)
or execute memory (ie, fetch an instruction to be executed)
if a program attempts to perform one of these operations, and the address of memory in question does not have the appropriate permission, The System terminates the program with a segfault
in short, a segfault occurs when a program tries to do something to memory that it is not allowed to do
this happens all the time in systems code, where we are playing crazy games with memory addresses (more games next week)
back to the program, what’s happening is that the variable s has never been given a value, so its value is effectively random
then fgets(3) uses the value of s as the memory address at which to store the user-inputted data
this memory address almost certainly does not have write permissions, thus segfault
(yes, it is theoretically possible that s could randomly contain a writeable memory address, but it is extremely unlikely, and would be an utter nightmare to debug)
we can cause s to point to writeable memory by declaring it like so:
char s[20];
this sets aside 20 bytes of memory in main’s stack frame
while this looks like an array (and indeed it is), the variable name of an array in C (ie, without any square brackets) is actually a pointer to the beginning of the array
so the compiler will happily allow us to use s as a parameter to fgets(3)
the code is fixed, but….
the pertinent question now is: how do we debug a segfault?
answer: gdb, the GNU dbugger
this should always be your first reaction, to the extent that, if you come to me with a segfault, I will ask what you discovered when you ran it in gdb
it’s okay if you can’t make sense of what gdb tells you, and I will help you understand it, but you have to have at least tried the tool
(gdb is a monstrously complex program; I have written a short description of some of the most important commands here)
gdb allows us to inspect and manage the run-time state of a program
this requires us to a) compile our programs with extra information that allows gdb to relate the machine code to the original source code and b) run our program under gdb’s control
for a), we add the -g option when compiling:
$ gcc -g -Wall -pedantic -o segfault segfault.c
for b), we precede the invocation of our program with gdb:
$ gdb ./segfault
now, inside gdb, we can run our program using the run command (which we can abbreviate to r):
(gdb) r Starting program: /home/pete/tmp/cs315-lab-02/segfault Type something: asdffdsa Program received signal SIGSEGV, Segmentation fault. 0x00007ffff7e55021 in _IO_getline_info () from /usr/lib/libc.so.6
(note the "asdffdsa" is text I typed when the prompt showed up)
this tells us precisely where the segfault occurred
ie, the machine-code instruction at address 0x00007ffff7e55021 attempted to perform a disallowed memory operation
this doesn’t help us much, because that code is way deep in the C library (libc.so.6)
the backtrace command (abbreviated bt) shows us the entire call stack of the program right now (and, since the program crashed, "right now" means at the moment of crash)
(gdb) bt #0 0x00007ffff7e55021 in _IO_getline_info () from /usr/lib/libc.so.6 #1 0x00007ffff7e54031 in fgets () from /usr/lib/libc.so.6 #2 0x0000555555555181 in main (argc=1, argv=0x7fffffffe728) at segfault.c:13
this tells us that main ran and, on line 13, it called fgets, which in turn called _IO_getline_info, which is the function that actually went boom
while it is not out of the realm of possibility, it is highly unlikely that such a simple use of fgets tickled a bug in the C library
it is far more likely that we’re just feeding it garbage parameters
since the parameters to fgets(3) are provided by main, and are in fact local variables of main and therefore stored in its stack frame, we have to tell gdb we want to operate on that frame
the numbers in the left-hand column of the bt output are numerical identifiers of the frames
so to switch to main’s frame:
(gdb) frame 2 #2 0x0000555555555181 in main (argc=1, argv=0x7fffffffe728) at segfault.c:13 13 fgets(s, 20, stdin);
and it even prints out line 13 for us
we can now check out the value of s:
(gdb) print s $1 = 0x0
clearly nonsensical, hence the problem
you will find gdb useful throughout this course
some people use it as their primary debugging tool, whereas others prefer inserting lots of printfs
either is a viable approach, but sometimes one is better than the other
so you really ought to have both tools in your toolbox
but this is an absurdly simple program and thus completely unrepresentative of real code
the segfault could happen 12 functions deep and we might have no idea how to get there
here’s a program that attempts to demonstrate something like that: stacktrace.c
the segfault happens in the function c, but if we had a super complex function-calling hierarchy, it might be really tough to tell how we actually go to c, or what its parameters were, or the values of its local variables
gdb can help here too
compile with -g and…
$ gdb ./stacktrace ... (gdb) run Starting program: /home/pete/tmp/315-06/stacktrace Program received signal SIGSEGV, Segmentation fault. 0x0000555555555186 in c (z=76) at stacktrace.c:38 38 *c = 19;
again, it tells us that it crashed on line 38
but now, we can ask for a backtrace, which is essentially a dissection of the current state of the stack: it tells us which series of function calls got us to this point:
(gdb) bt #0 0x0000555555555186 in c (z=76) at stacktrace.c:38 #1 0x000055555555516f in b (y=38) at stacktrace.c:29 #2 0x0000555555555150 in a (x=19) at stacktrace.c:21 #3 0x0000555555555132 in main (argc=1, argv=0x7fffffffe778) at stacktrace.c:14
this tells us that, on line 14, main called a (with parameter 19)
on line 21, a called b (with parameter 38)
on line 29, b called c (with parameter 76)
we can even print out state within the function c:
(gdb) print rv $1 = 0
we can also move to other stack frames and print out their state:
(gdb) frame 2 #2 0x0000555555555150 in a (x=19) at stacktrace.c:21 21 rv = b(x * 2); (gdb) print rv $2 = 0
for full details, refer to the guide or the official documentation linked from there
back traces (also known as "stack traces") may seem like they don’t give us much information
but they actually give a surprisingly helpful amount for figuring out what went wrong
in fact, if you’re using open source software and encounter a crash, if you email the authors, they’ll likely ask you to send them a backtrace
additionally, when you send automatic crash reports for software on your Mac or PC, it’s almost certainly sending a backtrace (along with other stuff)
next program, what’s wrong with it: memory-leak.c
on line 14, we get some memory from the heap and save its address in s
on line 15, we fill in that memory
on line 17, we get some more memory from the heap and save its address in s
note that we’ve now lost the address of the memory we got on line 14
we can’t ever free it!
in a small, silly program like this, that’s not a problem
but in a program intended to run for days, weeks, months, or even years, it could be fatal: we’d lose access to that memory until the program terminates (at which point all consumed resources are automatically reclaimed)
this is referred to as a memory leak
they are pernicious bugs to find, and gdb doesn’t help us much
fortunately, there exists another tool that does: valgrind
for the following sample run, I typed "hello", pressed enter, typed "world", pressed enter, and then hit Ctrl-D, which is the keystroke to indicate "end of file"
$ valgrind ./memory-leak ==31053== Memcheck, a memory error detector ==31053== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==31053== Using Valgrind-3.14.0.GIT and LibVEX; rerun with -h for copyright info ==31053== Command: ./memory-leak ==31053== hello you entered 6 characters world you entered 6 characters ==31053== ==31053== HEAP SUMMARY: ==31053== in use at exit: 3,072 bytes in 3 blocks ==31053== total heap usage: 5 allocs, 2 frees, 5,120 bytes allocated ==31053== ==31053== LEAK SUMMARY: ==31053== definitely lost: 3,072 bytes in 3 blocks ==31053== indirectly lost: 0 bytes in 0 blocks ==31053== possibly lost: 0 bytes in 0 blocks ==31053== still reachable: 0 bytes in 0 blocks ==31053== suppressed: 0 bytes in 0 blocks ==31053== Rerun with --leak-check=full to see details of leaked memory ==31053== ==31053== For counts of detected and suppressed errors, rerun with: -v ==31053== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
indeed, valgrind tells us that three blocks of memory, totalling 3072 bytes, were lost
and it tells us to run with the --leak-check=full parameter to get more details
so let’s do that (same input)
$ valgrind --leak-check=full ./memory-leak ==31057== Memcheck, a memory error detector ==31057== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==31057== Using Valgrind-3.14.0.GIT and LibVEX; rerun with -h for copyright info ==31057== Command: ./memory-leak ==31057== hello you entered 6 characters world you entered 6 characters ==31057== ==31057== HEAP SUMMARY: ==31057== in use at exit: 3,072 bytes in 3 blocks ==31057== total heap usage: 5 allocs, 2 frees, 5,120 bytes allocated ==31057== ==31057== 1,024 bytes in 1 blocks are definitely lost in loss record 1 of 2 ==31057== at 0x483777F: malloc (vg_replace_malloc.c:299) ==31057== by 0x109181: main (memory-leak.c:14) ==31057== ==31057== 2,048 bytes in 2 blocks are definitely lost in loss record 2 of 2 ==31057== at 0x483777F: malloc (vg_replace_malloc.c:299) ==31057== by 0x1091B1: main (memory-leak.c:17) ==31057== ==31057== LEAK SUMMARY: ==31057== definitely lost: 3,072 bytes in 3 blocks ==31057== indirectly lost: 0 bytes in 0 blocks ==31057== possibly lost: 0 bytes in 0 blocks ==31057== still reachable: 0 bytes in 0 blocks ==31057== suppressed: 0 bytes in 0 blocks ==31057== ==31057== For counts of detected and suppressed errors, rerun with: -v ==31057== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)
this time, it tells us where in the code the lost memory was originally allocated: one on line 14, and two on line 17, which is exactly what we expect
it is up to us as programmers, however, to figure out precisely how to deal with this: valgrind can’t know enough about a program to know where memory should have been freed, it can just identify when it hasn’t been
identify the problem with this program: invalid-read.c
it’s tricky: the loop is going after the end of the array, because it uses <= instead of <
interestingly, this does not cause a crash (test it for yourself)
but it’s definitely a bug
valgrind will catch this one, too:
$ valgrind ./invalid-read ==31100== Memcheck, a memory error detector ==31100== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==31100== Using Valgrind-3.14.0.GIT and LibVEX; rerun with -h for copyright info ==31100== Command: ./invalid-read ==31100== ==31100== Conditional jump or move depends on uninitialised value(s) ==31100== at 0x48C24FD: vfprintf (in /usr/lib/libc-2.28.so) ==31100== by 0x48CA449: printf (in /usr/lib/libc-2.28.so) ==31100== by 0x1091BE: main (invalid-read.c:17) ==31100== ==31100== Use of uninitialised value of size 8 ==31100== at 0x48BEA2E: _itoa_word (in /usr/lib/libc-2.28.so) ==31100== by 0x48C2269: vfprintf (in /usr/lib/libc-2.28.so) ==31100== by 0x48CA449: printf (in /usr/lib/libc-2.28.so) ==31100== by 0x1091BE: main (invalid-read.c:17) ==31100== ==31100== Conditional jump or move depends on uninitialised value(s) ==31100== at 0x48BEA39: _itoa_word (in /usr/lib/libc-2.28.so) ==31100== by 0x48C2269: vfprintf (in /usr/lib/libc-2.28.so) ==31100== by 0x48CA449: printf (in /usr/lib/libc-2.28.so) ==31100== by 0x1091BE: main (invalid-read.c:17) ==31100== ==31100== Conditional jump or move depends on uninitialised value(s) ==31100== at 0x48C3100: vfprintf (in /usr/lib/libc-2.28.so) ==31100== by 0x48CA449: printf (in /usr/lib/libc-2.28.so) ==31100== by 0x1091BE: main (invalid-read.c:17) ==31100== ==31100== Conditional jump or move depends on uninitialised value(s) ==31100== at 0x48C23BD: vfprintf (in /usr/lib/libc-2.28.so) ==31100== by 0x48CA449: printf (in /usr/lib/libc-2.28.so) ==31100== by 0x1091BE: main (invalid-read.c:17) ==31100== sum is 46 ==31100== ==31100== HEAP SUMMARY: ==31100== in use at exit: 0 bytes in 0 blocks ==31100== total heap usage: 1 allocs, 1 frees, 1,024 bytes allocated ==31100== ==31100== All heap blocks were freed -- no leaks are possible ==31100== ==31100== For counts of detected and suppressed errors, rerun with: -v ==31100== Use --track-origins=yes to see where uninitialised values come from ==31100== ERROR SUMMARY: 7 errors from 5 contexts (suppressed: 0 from 0)
this output is a bit more difficult to decipher
but the gist of "Conditional jump or move depends on uninitialised value(s)" is that some code made a decision based on memory that was never given an initial value
this makes sense: it’s accessing integer_array[5] but we never gave that element of the array a value
when debugging these kinds of errors, you’ll likely want to combine valgrind with gdb to see what’s actually going on inside your program (check out the next and step commands in the gdb guide)
last program: use-after-free.c
this bug is particularly nasty: note that we allocate memory with malloc, then we do stuff with that memory, then we free it, then we try to do more stuff with the memory we just freed
you might think it would cause a segfault, but it doesn’t
it’s still super dangerous, though, because passwords or cryptographic keys or the Coca-Cola recipe could have been stored in that memory, and another function could potentially still access it
it probably won’t surprise you to learn that valgrind can help identify these errors, too
running it produces a bunch of stanzas that look like this:
==31133== Invalid read of size 1 ==31133== at 0x483AC62: strlen (vg_replace_strmem.c:460) ==31133== by 0x48E3A55: puts (in /usr/lib/libc-2.28.so) ==31133== by 0x1091E5: main (use-after-free.c:25) ==31133== Address 0x4a38040 is 0 bytes inside a block of size 6 free'd ==31133== at 0x48389AB: free (vg_replace_malloc.c:530) ==31133== by 0x1091D9: main (use-after-free.c:24) ==31133== Block was alloc'd at ==31133== at 0x483777F: malloc (vg_replace_malloc.c:299) ==31133== by 0x109171: main (use-after-free.c:13)
this means that, on line 25, the program tried to read memory
this memory had been freed on line 24
but was originally allocated on line 13
again, it’s up to you to figure out how to fix it; valgrind just helps you discover the error
this is a classic error called use after free, for hopefully-obvious reasons
interestingly, if we run it, we get the following output:
$ ./use-after-free hello
(that second line is indeed empty)
this implies that calling free actually overwrites the freed memory with NULL bytes
which is new behavior—it didn’t behave this way even in the relatively recent past
the Powers That Be evidently decided that the performance overhead of clearing out the memory was justifiable because it rendered many use-after-free bugs relatively harmless
interesting side note
if I run the same program on a machine running FreeBSD (a different operating system and therefore a different implementation of malloc(3)), I get the following behavior:
$ ./use-after-free hello jello
so the FreeBSD implementation of malloc(3) does not do anything to clear memory that has been freed