pete > courses > CS 315 Fall 26 > lecture 08: binary organization

Lecture 08: binary organization

Goals

we’re going to start by reviewing the basic sequence of operations that takes us from a source file to an executable program

we’ll review (and add detail to) the various complexities we’ve previously discussed

including looking at some assembly language

you won’t be writing assembly in this course, nor will I expect you to deeply understand what it does, but we will walk through some today and next time

and then we’ll take it further by exploring the things we’ve sort of taken for granted in that area

here’s the Makefile for all of today’s programs

and here’s the stupid, simple motivating program: my_program.c

the simplest way to turn this into a runnable executable is to ask gcc to compile it and directly produce a program:

$ make my_program
gcc -g -pedantic -o my_program my_program.c

(what I mean by "directly produce a program" is one of those vague phrases for which you probably have an intuitive sense, but not a concrete one, and we’ll work on that today)

(also, I’m not using the -Wall flag today because I’ve cut some corners in writing these programs to make them more concise, but they do introduce warnings that just clutter up the output)

question: what if some other program desperately needs the functionality of my super duper function? how can another program take advantage of it?

the naive solution is to cut and paste the function prototype and the function definition into another source file

like so: my_other_program.c

and then compile that one

like so:

$ make my_other_program
gcc -g -pedantic -o my_other_program my_other_program.c

fairly straightforward, but what are the problems with this?

what if somebody comes up with a better way to implement my_super_duper_function?

unlikely as it seems, this is indeed possible

to incorporate a new version, someone would have to copy and paste the new prototype into the right place within my_other_program.c

and paste the new function definition into the separate right place in the same file

they’d also have to make sure all the old code was removed

all of these steps are error-prone

the more moving parts, the more difficult it is to ensure correctness

the solution, as we discussed in the context of assignment 2, is to extract all the parts of the source file that have to do with my_super_duper_function (ie, the prototype and the definition) and put them in their own file(s)

then, if another program wants to use them, we just have to compile these files unchanged: no fragile cut-and-paste steps, just use the whole file

the function prototype goes in the header file: my_super_duper_library.h

the function definition goes in the corresponding source file: my_super_duper_library.c

and then, in the program that uses it, we just need to #include the header file: my_better_program.c

then, to build the program, we need to compile both source files together:

$ make my_better_program
gcc -g -pedantic -o my_better_program my_better_program.c my_super_duper_library.c

nothing terribly exotic, at least superficially, but we will dig in a bit more in a moment

but first, a brief digression on header files and how #include works, that will include an explanation of the weird __MY_SUPER_DUPER_LIBRARY_H stuff in there…

#include is very simple: the compiler finds the named file and replaces the #include line with the entire contents of that file

that’s it

note that the header file above only contains the function prototype, which in the original program took the place of the #include line

so nothing really has changed from the original, we’ve just put the lines of text in different files

now let’s look at the three strange lines in that header file:

#ifndef __MY_SUPER_DUPER_LIBRARY_H
#define __MY_SUPER_DUPER_LIBRARY_H

...

#endif /* __MY_SUPER_DUPER_LIBRARY_H */

we’re going to defer an in-depth discussion of preprocessor directives (which is what these are)

suffice it to say for now: these prevent errors from circular dependency loops, in which a.h includes b.h and b.h includes a.h

using the double underscores at the beginning is a convention for this purpose

another related convention is found back in the .c file that #includes the header

note that it uses double quotes instead of angle brackets

this is a hint to the compiler that the header file is to be found in the current directory, rather than the directory that contains header files to be shared system-wide

okay, so we’ve got these two different methods to produce what we hope to be the exact same code

it’s worth seeing if it really is the same code

to do that, we have to look at the output file

since it’s compiled code—ie, machine code—ie, object code—the program to look at it is called objdump

recall also our discussion about what is in compiled programs: it’s both machine code and data, so we want to make sure here that we’re looking at the machine code, which is traditionally stored in the "text" section of the file

so here’s what we run:

$ objdump -j .text -d my_better_program

this produces a ton of output that scrolls by very quickly

one way to deal with that is to use another program to let us go through the output page by page—this kind of program is called a pager, and is precisely what shows you manpages

thus:

$ objdump -j .text -d my_better_program | less

the "|" symbol (pronounced "pipe") takes the output of the command on the left and feeds it as the input to the command on the right

the program less is, as I said, the same program used to view manpages, so the same keys work within it: arrows keys to scroll, forward-slash to search, q to quit

back to the output

we see a lot of inscrutable stuff

but way down, we see the assembly code for the main and my_super_duper_function functions

they start with lines like so:

0000000000001119 <main>:
...
0000000000001137 <my_super_duper_function>:

this means that main (ie, the machine code instructions that implement that function’s behavior) starts at address 1119 (hex)

and the instructions for my_super_duper_function start at address 1137 (hex)

the corresponding code is predictably short and not very interesting, but that’s not the point for this lecture

it’s worth looking at the particular instruction that performs the function call:

    1128:       e8 0a 00 00 00          callq  1137 <my_super_duper_function>

the callq instruction takes as parameter the (relative) address of the function to call, which is the latter four bytes of the five-byte instruction

objdump helpfully tells us its name in angle brackets on the right (though there is no notion of its name in the machine code itself: objdump has inferred it… somehow—we’ll see how in a few minutes)

let’s now look at the same output for the original version that didn’t use two source files:

$ objdump -j .text -d my_program | less

it looks pretty much the same (in fact, diff confirms they are, in fact, exactly identical)

this is interesting, but not too surprising given what we know of source files, compilation, and programs

in the first case, the compiler takes the single source file, translates functions therein to machine code, and stuffs them into a program file

in the second case, the compiler takes the two source files, translates the functions therein to machine code, and stuffs them into a program file

the result is the same, the only difference is where the functions came from in the first place

now, to ask the same question as before: what’s fragile or delicate or inefficient or inconvenient about this setup?

what if, again, somebody (inexplicably) comes up with a better way to implement my_super_duper_function?

we’d presumably ask the compiler to take the one source file that has changed (my_super_duper_library.c) and the one that hasn’t changed (my_better_program.c) and again compile them together to produce an executable

note that compiling my_better_program.c is wasted work: it hasn’t actually changed, so the machine code will be exactly the same

in a simple situation like this, with two source files, this isn’t a big deal

but imagine a big program with a few hundred source files, or a huge program with a few thousand

recompiling all of them just because a single source file changes is a huge waste of effort

the solution is to separate the "compile" step and the "combine" step

so compile my_better_program.c and save the result in what we will call a corresponding object file: my_better_program.o

separately, we’ll compile my_super_duper_library.c and save the result in a different object file: my_super_duper_library.o

then combine the two .o to produce the program

like so:

$ make my_better_program_done_right
gcc -g -pedantic -c -o my_better_program.o my_better_program.c
gcc -g -pedantic -c -o my_super_duper_library.o my_super_duper_library.c
gcc -o my_better_program_done_right my_better_program.o my_super_duper_library.o

(note that the final step does not use CFLAGS: this is because it’s not performing compilation, and thus they do not apply—we’ll see which flags do apply another day)

now, if somebody (again, utterly inexplicably) updates my_super_duper_library.c, all we need to do is recompile it to produce a new my_super_duper_library.o

and then combine that new .o file with the previously-compiled my_better_program.o file (whose corresponding source code has not changed) to produce the executable

we can mimic this situation by updating the modification time on my_super_duper_library.c, which will give make the impression that its contents have changed and thus needs to be recompiled:

$ touch my_super_duper_library.c
$ make my_better_program_done_right
gcc -g -pedantic -c -o my_super_duper_library.o my_super_duper_library.c
gcc -o my_better_program_done_right my_better_program.o my_super_duper_library.o

note that I used the exact same make command as before, but only one .c file was compiled

but two .o files (the new one and the old one) were combined to produce the program

again, not a huge win for a two-file program

but it is a potentially huge win for a 100-file program

let’s turn objdump on this thing

$ objdump -j .text -d my_better_program_done_right | less

again: no change! still the exact same contents as the previous two programs

which is what we expect, given the desired behavior, but it’s nice to see our expectations validated

but what about these .o files? they contain machine code, so they’re probably worth looking at with objdump

first the library:

$ objdump -j .text -d my_super_duper_library.o

the output is short enough that we don’t need less

it’s also unsurprising: it contains the machine code for the single function in the corresponding .c file

now the main program source file

$ objdump -j .text -d my_better_program.o

the output is also short and not terribly surprising

except this part:

   f:   e8 00 00 00 00          callq  14 <main+0x14>

compare this to the output from the last time we objdump’ed the whole program

the target address (the last four bytes) is empty!

this makes sense, though: that function is contained in a completely different file, and if the compiler only has my_better_program.c to work with, it can’t know the address of my_super_duper_function

the implication is important: one part of the "combine .o files to produce a program" task has to be to fill in the address

now for some vocabulary

a symbol is a name with an associated address

so main is a symbol and my_super_duper_function is a symbol, because they both refer to the addresses at which those functions start

additionally, the call to my_super_duper_function is a reference to that symbol: it uses the address as the target of its jump

furthermore, in the excerpted objdump output above, the compiler wasn’t able to figure out the address of that symbol

which makes it an undefined reference

ie, it refers to something, but we don’t know the address of it (yet)

you might ask yourself whether we can learn anything about symbols in object files

and the answer is yes, using the nm program

nm lists each symbol: its address, its status (more in a bit), and its name

(more precisely, it shows the contents of the symbol table, which is another part of the still unfolding complexity of the program file)

thus:

$ nm my_super_duper_library.o 
0000000000000000 T my_super_duper_function

this output tells us that this object file defines a single symbol

unsurprisingly named my_super_duper_function

it’s located at address zero within the file

and it is in the text (T) section (ie, executable code, which we expect)

the more interesting collection of symbols is in the main file:

$ nm my_better_program.o
0000000000000000 T main
                 U my_super_duper_function

main is in the text section, at offset zero

but the my_super_duper_function symbol is undefined (U), which we expect

next vocabulary word: linking is the task of combining these object files and resolving references between them

so before linking we’ve got two object files, one with an undefined reference

after linking, we’ve got the contents of the two object files saved in one program file, and no undefined references

final observation: the two object files we just looked at contain the machine code for exactly two functions

but the programs we looked at contain machine code for many more functions

where do they come from?

the answer is that these are the infrastructural functions that support the running of C programs

they are collectively referred to as the "C runtime"

and we can find object files somewhere on the machine with filenames containing the string "crt"

(recall the strace output showed a lot of stuff happening before the "meat" of the programs took over: those steps are performed by the C runtime—we’ll look at them in more detail over the coming weeks)

Last modified: