I've been banging my head against the keyboard for three hours over a fucking Segmentation Fault in a C program that, in theory, should be trivial. I opened GDB, did a backtrace to see where it had crashed, and there it was, laughing in my face: it had smashed the Call Stack by using the strcpy function to copy a text string much longer than the buffer could swallow.

Writing code in C or Assembly comes with these things. It gives you the keys to the Ferrari, all the power and speed, but in return, it lets you crash into a wall at 200 km/h if you don't pay attention to the road. So I decided that, since I'm wide awake, I'm going to write this down. For myself and for anyone who is tinkering with x86 Assembly and can't quite grasp how the hell the darn call stack works at the silicon level.

The mud: How the call stack works

To understand the Call Stack or the stack, you have to forget about the comforts of high-level languages like Java or PHP. Here there is no garbage collector or safety net. At the processor level (a Pentium IV HT in my case, which sounds like a fighter jet when I compile a large project), the stack is nothing more than a continuous segment of RAM that we use as if it were a stack of plates. The last plate you put on the stack is necessarily the first one you take off. It's what we call a LIFO (Last In, First Out) structure.

In the 32-bit x86 architecture, there is a peculiarity that usually confuses people a lot at first: the stack grows "downwards". That is, it starts at a high memory address (for example, 0xBFFFF000) and, as we push data, it occupies lower addresses (0xBFFFEFFC, etc.).

To manage this whole affair, we have two main registers in the CPU that are the true foremen of the construction site:

  • ESP (Extended Stack Pointer): It is the stack pointer. It always, without exception, points to the top of the stack (the last byte we pushed).
  • EBP (Extended Base Pointer): It serves as our anchor point. A constant reference within our function to be able to access local variables and the parameters passed to us, regardless of whether ESP goes up or down.

The Stack Frame ritual

When we call a function (using a call instruction in assembly), the machine does two magical things underneath: 1. It pushes (push) the address of the next instruction onto the stack. This is the sacred "return address", to know where the program must return when the function finishes. 2. It jumps to the memory address where the function code begins.

Once we land inside the function, the first thing is to set up what is called the Stack Frame. It's a ritual, almost a prayer, that you will see in absolutely any disassembled code if you throw it into OllyDbg or look at it with GDB:

mi_funcion:
    ; --- Function prologue ---
    push ebp        ; We save the previous function's EBP so we don't lose it
    mov ebp, esp    ; The current ESP becomes our new base (EBP)
    sub esp, 24     ; Make room: allocate 24 bytes for local variables

    ; --- Here goes the meat, the real code ---
    ; The parameters we receive are "above" the EBP: [ebp+8], [ebp+12]...
    ; Our local variables are "below": [ebp-4], [ebp-8]...

    ; --- Function epilogue ---
    mov esp, ebp    ; Restore ESP. We clean up locals in one fell swoop.
    pop ebp         ; Restore the original EBP of whoever called us
    ret             ; Pop the return address and jump back

The exact problem that has soured my night is that, by stuffing data wildly into a local variable (imagine I write 40 bytes into a 24-byte space), the code kept writing memory "upwards". I overwrote my own saved EBP and, infinitely worse, the return address.

When my function finished and executed the ret, the processor grabbed the garbage I had written, thought it was a valid memory address, and tried to execute code there. Segmentation fault right off the bat! The kernel of my trusty Linux detected the trickery and killed the process on the spot to avoid a major disaster.

Late-night reflections

Tinkering at this low level is tough, why kid ourselves. Sometimes, when the C compiler spits out incomprehensible errors, I wish I could dedicate myself to making forums in PHP or fighting with CSS, and forever forget about registers, memory offsets, and core dumps.

But there is something incredibly satisfying about understanding exactly what the machine does under the sheets of your C programs. Knowing how parameters are pushed with a push, how memory is read, and how the ret takes us back home makes you an infinitely better programmer. It gives you absolute control over the hardware.

Seeing how computing is advancing in this year 2005, with AMD's new 64-bit processors rearing their heads and operating systems trying to protect memory more (I know they are working on hardware-level protections, like the NX bit, so you can't execute code injected into the stack), the way to exploit these buffer overflows will change. It will be harder for hackers to slip in shellcodes trivially.

Even so, the fundamental architecture, the concept of the Stack Frame, will remain the same. It is the heart of how functions and programs interact. If you master the stack, you master the machine.

I'm going to bed before the sun comes up and the birds start singing. Tomorrow is another day.