4. Program Analysis

Program analysis is the discipline of determining what a program actually does, as opposed to what its documentation, its source code, or its author claims it does. For a security practitioner this is foundational: vulnerability research, malware analysis, exploit development, and patch verification all reduce to reading a binary's real behaviour, either by reasoning over its code without running it (static analysis) or by observing it while it executes (dynamic analysis). This chapter starts at the machine level with x86-64 assembly, because every higher-level tool in the chain is ultimately a convenience layer over the instructions the CPU will execute.

<a id="41-assembly-x86-64"></a>

4.1 Assembly x86-64

Assembly is the lowest level at which a human can still read a program, and it is the level at which compilers, linkers, and attackers all meet; this section builds the x86-64 model from the registers upward until you can read compiler output and assemble a working program by hand.

CPU Architecture, Registers, and Flags

Theory

Before asking what instructions do, it is worth asking why a CPU needs registers at all. Main memory is enormous but slow: a read from DRAM costs on the order of a hundred CPU cycles. Arithmetic takes one. If every operation had to fetch its operands from memory, the processor would spend virtually all of its time waiting. Registers are a tiny set of storage slots built directly into the CPU core, accessible in a fraction of a cycle. They exist so the CPU has somewhere to hold the handful of values it is working on right now.

The consequence is that almost all x86-64 code follows the same rhythm: load values from memory into registers, compute on registers, store results back to memory. When you read disassembly and it looks repetitive, that rhythm is what you are seeing.

x86-64 gives you sixteen general-purpose 64-bit registers. Six of them have names inherited from the 16-bit era (rax, rbx, rcx, rdx, rsi, rdi), two are structurally special (rsp, rbp), and eight are simply numbered (r8 through r15). "General-purpose" is slightly optimistic: by hardware or by convention, several carry specific duties.

Register	Conventional role	Notes for analysis
rax	Return value; accumulator	A function's result is almost always here
rbx	Callee-saved scratch	Preserved across calls
rcx	4th integer argument; shift/loop count	shl/rep implicitly use cl
rdx	3rd integer argument; high half of mul/div	Paired with rax for 128-bit results
rsi	2nd integer argument; string source	rep movsb source
rdi	1st integer argument; string destination	The first thing to inspect at a call
rsp	Stack pointer	Modified by push/pop/call/ret
rbp	Frame pointer (by convention)	Often repurposed when optimizing
r8–r9	5th and 6th integer arguments
r10–r11	Caller-saved scratch	r10 is the 4th arg in syscalls
r12–r15	Callee-saved scratch	Long-lived locals in optimized code

Two registers are not in that list but are central. rip is the instruction pointer: it holds the address of the next instruction. You cannot write to it directly with mov; it changes only through control-flow instructions (jmp, call, ret) — which is precisely why "controlling rip" is the goal of memory-corruption exploitation. rflags is a bag of single-bit status flags set as a side effect of arithmetic.

The flags answer the question: how did the last arithmetic operation turn out? This is how a CPU makes decisions. There is no "compare and branch" instruction; there is a cmp that sets flags and a separate jcc that reads them.

Flag	Name	Set when
ZF	Zero	Result was exactly zero (so cmp a,b with ZF=1 means a == b)
SF	Sign	Most significant bit of the result is 1 (negative, if signed)
CF	Carry	Unsigned overflow out of the top bit (borrow, for subtraction)
OF	Overflow	Signed overflow: the sign flipped when it mathematically should not have
PF	Parity	Low byte has an even number of set bits (rarely relevant)
AF	Adjust	BCD arithmetic carry (effectively obsolete)

The CF/OF distinction is the single most common source of confusion, and it is a real security issue, not a trivia point. CF describes overflow if you interpret the bits as unsigned; OF describes overflow if you interpret them as signed. The same add instruction sets both, and which one the compiler then tests is determined entirely by the C types in the source. A signed/unsigned mix-up in a length check is a bug class, and at the assembly level it is visible as jb (unsigned "below") where jl (signed "less") was intended, or the reverse.

Finally, every 64-bit register can be addressed at narrower widths, which is a frequent source of reverse-engineering errors:

  63                             31             15      7     0
  +------------------------------+--------------+-------+-----+
  |                          rax                              |
  +------------------------------+--------------+-------+-----+
                                 |          eax               |   32-bit
                                                |     ax      |   16-bit
                                                | ah  |  al   |   8-bit

Writing to eax zeroes the upper 32 bits of rax. Writing to ax or al leaves the upper bits untouched. That asymmetry is deliberate (it makes 32-bit code patterns cheap) and it is why mov eax, eax is a real instruction used to truncate a value to 32 bits.

Examples

  Conceptual layout of one CPU core
  ─────────────────────────────────────────────────────────────

        ┌──────────────────────── CPU CORE ────────────────────┐
        │                                                      │
        │  ┌─ Registers (≈1 cycle) ─┐   ┌─ Control ─────────┐  │
        │  │ rax rbx rcx rdx        │   │ rip  (next instr) │  │
        │  │ rsi rdi rbp rsp        │   │ rflags  ZF SF CF  │  │
        │  │ r8 ... r15             │   │         OF PF AF  │  │
        │  └────────────────────────┘   └───────────────────┘  │
        │                 │                                    │
        │        ┌────────┴────────┐                           │
        │        │       ALU       │  sets flags as a side     │
        │        └─────────────────┘  effect of every op       │
        └──────────────────│───────────────────────────────────┘
                           │  load / store
                 ┌─────────┴──────────┐
                 │  L1 → L2 → L3      │  ~4 to ~40 cycles
                 └─────────┬──────────┘
                           │
                 ┌─────────┴──────────┐
                 │   Main memory      │  ~100+ cycles
                 └────────────────────┘

; How a C comparison becomes flags plus a branch.
; Source:  if (count == 0) { ... }

    mov     eax, DWORD [rbp-4]   ; load `count` into eax (zero-extends into rax)
    test    eax, eax             ; AND eax with itself: sets ZF if eax == 0,
                                 ; cheaper than `cmp eax, 0` and encodes shorter
    jne     .Lskip               ; jump if ZF == 0, i.e. count != 0
    ; ... body of the if ...
.Lskip:

; The signed/unsigned trap, visible only in the jump chosen.
    cmp     eax, edx
    jb      .Ltoo_small          ; UNSIGNED compare: reads CF
    ; ...
    cmp     eax, edx
    jl      .Ltoo_small          ; SIGNED compare: reads SF and OF
; If `eax` holds a user-controlled length that the programmer declared as
; `int` but the check should have been unsigned, a negative value passes
; `jb`-style logic as a huge positive number -- or passes `jl` as "small".

// Observe flag behaviour from C by forcing an overflow.
#include <stdio.h>
#include <limits.h>

int main(void) {
    unsigned int u = UINT_MAX;
    unsigned int ru = u + 1;      // wraps to 0; CF was set, defined behaviour

    int s = INT_MAX;
    int rs = s + 1;               // OF was set; UNDEFINED behaviour in C

    printf("unsigned wrap: %u\n", ru);   // 0
    printf("signed  wrap: %d\n", rs);    // typically INT_MIN, but not guaranteed
    return 0;
}

Key Takeaways

• Registers exist because memory is roughly two orders of magnitude slower than arithmetic; nearly all x86-64 code is load, compute, store.
• There are sixteen general-purpose 64-bit registers, but convention assigns specific duties (arguments, return value, stack) that make disassembly readable.
• rip cannot be written directly — only control-flow instructions change it, which is why hijacking it is the objective of memory-corruption attacks.
• Arithmetic sets flags as a side effect; comparison and branching are two separate instructions joined only by rflags.
• CF is unsigned overflow and OF is signed overflow; which one the code tests reveals the source-level signedness and is a frequent bug source.
• Writing a 32-bit subregister (eax) zeroes the upper half of the 64-bit register, while writing 16- or 8-bit subregisters does not.

🧪 Practice

1. Write a three-line assembly snippet that leaves rax equal to 0 using xor, and explain which flags that instruction sets.
2. For the byte values 0xFF and 0x01, compute 0xFF + 0x01 and state the values of ZF, CF, and OF, first interpreting the bytes as unsigned and then as two's-complement signed.
3. Compile int f(int a, int b){ return a < b; } and the unsigned variant with gcc -O1 -S -masm=intel, then identify the single instruction that differs and explain why.
4. Interview-style: "A length check in C is written if (len > MAX) return -1; where len is an int populated from network data. Walk me through what the generated assembly does when the attacker sends a length whose top bit is set." Hint: focus on which conditional jump the compiler picks for a signed type, and what the subsequent use of len as a size argument does once the check passes.

Memory Model, Endianness, and Word Sizes

Theory

A process does not see RAM. It sees a virtual address space: a flat, byte-addressable array of 2^48 usable addresses (on current x86-64 implementations) that the memory-management unit maps onto physical pages. "Flat" is the key word — there is no type information, no bounds, and no structure other than what the program imposes by convention. A pointer is just an integer. This is exactly why memory-safety bugs exist: the hardware will happily read byte 1,000,000 of a 16-byte buffer, because to the hardware there is no buffer, only addresses.

Within that space, data has a size and a layout. The size question is the easier one, though x86-64 terminology is a historical trap: a "word" on x86 means 16 bits, because the architecture was 16-bit in 1978 and the name was never updated.

Term	Bits	Bytes	Assembly suffix / directive	C type (Linux LP64)
byte	8	1	db, BYTE	char
word	16	2	dw, WORD	short
doubleword	32	4	dd, DWORD	int
quadword	64	8	dq, QWORD	long, pointer
double quadword	128	16	ddq, XMMWORD	SIMD / long double storage

Linux x86-64 uses the LP64 model: long and pointers are 64-bit, int stays 32-bit. Windows x64 uses LLP64, where long remains 32-bit. Code that assumes sizeof(long) == sizeof(void*) breaks when ported, and that assumption has produced real vulnerabilities in pointer-to-integer casts.

The layout question is endianness: when a multi-byte value is stored at an address, which byte lands first? x86-64 is little-endian — the least significant byte goes at the lowest address.

The usual analogy is writing a date. Big-endian is 2026-10-01: most significant unit first, which reads naturally. Little-endian is 01-10-2026: least significant first, which looks backwards to a human but means that reading the first byte alone already gives you the low-order part, so truncating a 64-bit value to 32 or 8 bits is just reading fewer bytes from the same address — no adjustment needed.

For security work, endianness is not an abstract detail. It determines the byte order of every address you put into an exploit payload, every length field you forge, and every hex dump you read. A 64-bit address 0x00007fffffffe3a0 appears in memory as a0 e3 ff ff ff 7f 00 00. Getting this backwards is the single most common beginner mistake in exploit construction, which is why pwntools provides p64() rather than expecting you to do it manually.

A related layout rule is alignment and struct padding. The CPU prefers (and for some SIMD instructions requires) that an N-byte value sit at an address that is a multiple of N. Compilers therefore insert padding bytes inside structures. Those padding bytes are not initialized, which is a genuine information-leak vector: a kernel that copies a struct to userspace with copy_to_user leaks whatever stale stack data occupied the padding.

Examples

  The value 0x0123456789ABCDEF stored at address 0x1000
  ───────────────────────────────────────────────────────────────

  LITTLE-ENDIAN (x86-64, ARM in normal config)
  addr:  1000 1001 1002 1003 1004 1005 1006 1007
  byte:   EF   CD   AB   89   67   45   23   01
          ^^ least significant byte at the lowest address
          Reading 1 byte at 0x1000 -> 0xEF  (the low byte: free truncation)

  BIG-ENDIAN (network byte order, SPARC, s390x)
  addr:  1000 1001 1002 1003 1004 1005 1006 1007
  byte:   01   23   45   67   89   AB   CD   EF
          ^^ most significant byte first: reads like we write numbers

import struct

addr = 0x00007fffffffe3a0

# Little-endian: what actually sits in x86-64 memory, and what an
# exploit payload must contain.
print(struct.pack("<Q", addr).hex())   # a0e3ffffff7f0000

# Big-endian, a.k.a. network byte order, for protocol fields.
print(struct.pack(">Q", addr).hex())   # 00007fffffffe3a0

# A length field read with the wrong endianness is not a small error:
raw = bytes.fromhex("00000100")
print(struct.unpack("<I", raw)[0])     # 65536
print(struct.unpack(">I", raw)[0])     # 256

// Struct padding: the sizes do not add up, and the gaps are uninitialized.
#include <stdio.h>
#include <stddef.h>

struct bad {
    char  a;      // offset 0
                  // offsets 1-3: PADDING so that `b` is 4-byte aligned
    int   b;      // offset 4
    char  c;      // offset 8
                  // offsets 9-15: PADDING so the struct's size is a
                  //               multiple of its strictest alignment (8)
    void *d;      // offset 16
};                // sizeof == 24, not 1+4+1+8 == 14

int main(void) {
    printf("size=%zu  b@%zu  d@%zu\n",
           sizeof(struct bad), offsetof(struct bad, b), offsetof(struct bad, d));

    // Reordering largest-to-smallest removes most padding.
    struct good { void *d; int b; char a; char c; };  // sizeof == 16
    printf("good size=%zu\n", sizeof(struct good));
    return 0;
}

# Confirm endianness directly from a compiled constant.
printf '#include <stdio.h>\nint x = 0x41424344;\nint main(void){return 0;}\n' > e.c
gcc -c e.c -o e.o
objdump -s -j .data e.o | grep -A1 'Contents'
# On x86-64 the bytes appear as 44434241 -> "DCBA", i.e. reversed.

Key Takeaways

• A process sees a flat, untyped virtual address space; the absence of hardware-level bounds or type information is the root enabler of memory-safety bugs.
• On x86 a "word" is 16 bits for historical reasons; use byte/dword/qword precisely when reading assembly.
• Linux x86-64 is LP64 (long and pointers are 64-bit, int is 32-bit); Windows x64 is LLP64, so sizeof(long) == sizeof(void*) is not portable.
• x86-64 is little-endian: the least significant byte occupies the lowest address, so addresses appear byte-reversed in memory dumps and exploit payloads.
• Network byte order is big-endian, which is why protocol code is full of htonl/ntohl conversions.
• Compilers insert alignment padding inside structs; that padding is uninitialized and copying whole structs across a trust boundary can leak stale memory.

🧪 Practice

1. Write out the exact byte sequence, in order, that the 32-bit value 0xDEADBEEF occupies in little-endian memory, then do the same for big-endian.
2. Define a struct containing char, double, char, int in that order, predict its size with padding, verify with sizeof and offsetof, then reorder the fields to minimize it.
3. Using Python's struct, write a function that takes a 64-bit address and returns the byte string you would place in a stack-overflow payload on x86-64; test it against pwntools' p64().
4. Interview-style: "A kernel driver fills a struct, sets every named field, and copies it to userspace. Why might this still leak kernel memory?" Hint: think about the bytes that exist in the struct but have no field name, and where the memory holding that struct was most recently used.

Instruction Set Basics: mov, arithmetic, logic

Theory

x86-64 is a CISC instruction set with roughly a thousand instructions, but reverse engineering real binaries requires fluency in perhaps thirty. The reason is that compilers emit a narrow, predictable subset. Learn that subset and the rest can be looked up when it appears.

The first thing to internalize is the operand order convention, because there are two and they are reversed. In Intel syntax the destination comes first: mov rax, rbx means "rax becomes rbx", reading like an assignment statement. In AT&T syntax (GNU's default, what you see from objdump unless you ask otherwise) the source comes first and registers carry a % sigil: mov %rbx, %rax does the same thing. Mixing these up inverts the meaning of every line you read, so always confirm which syntax a tool is producing.

Aspect	Intel syntax	AT&T syntax
Direction	op dst, src	op src, dst
Registers	rax	%rax
Immediates	5, 0x10	$5, $0x10
Memory	[rbp-8]	-8(%rbp)
Size	implied, or BYTE/QWORD PTR	instruction suffix: movb, movq
Tools	NASM, Ghidra, Windows	GAS, GDB and objdump by default

This guide uses Intel syntax throughout; pass -masm=intel to GCC and set disassembly-flavor intel in GDB to match.

mov is the workhorse and, despite the name, it copies rather than moves. The one rule that surprises newcomers is that x86-64 has no memory-to-memory mov. At most one operand may be a memory reference, so copying between two memory locations always takes two instructions with a register in between. (The rep movsb string instructions are the exception, and the compiler emits them only for bulk copies.)

Size matters, and so does how narrow values are widened:

• mov eax, ecx — 32-bit copy; implicitly zero-extends into the upper 32 bits of rax.
• movzx rax, bl — explicit zero-extension of an 8-bit value: upper bits become 0. This is what an unsigned char promotion compiles to.
• movsx rax, bl — explicit sign-extension: the upper bits are filled with the value's sign bit. This is a signed char or int-to-long promotion.

The movzx/movsx distinction is a direct security signal. When a 32-bit length is sign-extended into a 64-bit size argument, a value like 0xFFFFFFFF becomes 0xFFFFFFFFFFFFFFFF rather than 4294967295. Finding movsx where a size is being computed is a reason to look harder.

Also note lea ("load effective address"), which belongs in this group even though it looks like a memory instruction. lea rax, [rbx+rcx*4+8] computes the address rbx + rcx*4 + 8 and puts it in rax without touching memory. Compilers use it constantly as a free three-operand arithmetic instruction.

Arithmetic instructions are mostly unsurprising, with three that deserve attention:

• add, sub, inc, dec, neg — all set flags.
• imul / mul — signed and unsigned multiply. The one-operand forms write a 128-bit result across rdx:rax, which is how a compiler detects multiplication overflow.
• idiv / div — division requires the dividend pre-extended into rdx:rax, hence the cqo (sign-extend rax into rdx) or xor edx, edx (zero it, for unsigned) that always precedes it. Dividing by zero raises #DE, delivered as SIGFPE.

Logic and bit manipulation rounds out the set: and, or, xor, not, plus shifts shl/shr (logical) and sar (arithmetic, sign-preserving). Two idioms appear so often they should be read as single words:

• xor eax, eax — the canonical "set to zero". It is two bytes and breaks the dependency chain, so compilers prefer it over mov eax, 0.
• test rax, rax — AND without storing the result; used purely to set ZF/SF, i.e. "is this zero or negative?"

A subtle point with real consequences: the difference between shr and sar is the difference between unsigned and signed division by a power of two. shr shifts in zeros; sar replicates the sign bit. Seeing shr applied to something the source declared signed means the value is being treated as unsigned somewhere, which is worth investigating.

Examples

; ─── Data movement ───────────────────────────────────────────────
    mov     rax, rbx            ; register to register (a copy, not a move)
    mov     rax, 0x1234         ; immediate to register
    mov     rax, [rbp-8]        ; LOAD: read 8 bytes from memory
    mov     [rbp-8], rax        ; STORE: write 8 bytes to memory
    mov     BYTE [rdi], 0       ; size must be explicit when ambiguous

;   mov     [rdi], [rsi]        ; INVALID: no memory-to-memory form
    mov     rax, [rsi]          ; so it becomes two instructions
    mov     [rdi], rax

; ─── Width and extension ────────────────────────────────────────
    mov     eax, ecx            ; zero-extends into rax implicitly
    movzx   eax, BYTE [rdi]     ; unsigned char -> int
    movsx   eax, BYTE [rdi]     ; signed char   -> int (sign-filled)
    movsxd  rax, ecx            ; int -> long. If ecx is a length read from
                                ; input, 0xFFFFFFFF becomes -1 as a 64-bit
                                ; value, i.e. SIZE_MAX when used as a size.

; ─── lea: address arithmetic that never touches memory ──────────
    lea     rax, [rbx+rcx*4+8]  ; rax = rbx + rcx*4 + 8  (no load)
    lea     rax, [rdi+rdi*2]    ; rax = rdi * 3, in one instruction

; ─── Arithmetic ─────────────────────────────────────────────────
    add     rax, rbx            ; rax += rbx, sets ZF/SF/CF/OF
    sub     rax, 1              ; rax -= 1
    imul    rax, rbx            ; signed multiply, 64-bit result
    cqo                         ; sign-extend rax into rdx:rax, required...
    idiv    rbx                 ; ...before signed divide. rax=quot, rdx=rem

; ─── Logic and the two idioms to read as single words ───────────
    xor     eax, eax            ; eax = 0  (idiomatic; 2 bytes, no dependency)
    test    rax, rax            ; set flags from rax without changing it
    je      .Lnull              ; -> "if (rax == 0)"

    and     rax, 0xFF           ; keep the low byte
    shl     rax, 3              ; rax *= 8
    sar     rax, 3              ; SIGNED   rax /= 8  (preserves sign)
    shr     rax, 3              ; UNSIGNED rax /= 8  (shifts in zeros)

// Source that makes the compiler reveal each pattern above.
long widen_signed(int n)      { return n; }          // -> movsxd
unsigned long widen_unsig(unsigned n) { return n; }  // -> mov eax, edi
long times_three(long n)      { return n * 3; }      // -> lea rax,[rdi+rdi*2]
long div_signed(long n)       { return n / 8; }      // -> sar (plus a fixup)
unsigned long div_unsig(unsigned long n) { return n / 8; }  // -> shr
int is_zero(long n)           { return n == 0; }     // -> test / sete

# Read the generated code in Intel syntax, without the noise of
# directives and debug metadata.
gcc -O1 -S -masm=intel -fno-asynchronous-unwind-tables snippets.c -o -

Key Takeaways

• Intel syntax is op dst, src; AT&T syntax is op src, dst with % on registers — confirm which your tool emits before reading anything.
• mov copies, and x86-64 permits at most one memory operand, so memory-to-memory copies always go through a register.
• movzx is zero-extension and movsx/movsxd is sign-extension; sign-extension applied to a length or size is a strong vulnerability signal.
• lea performs address arithmetic without accessing memory and is routinely used as free multi-operand arithmetic.
• idiv/div require the dividend pre-extended into rdx:rax, which is why cqo or xor edx, edx always precedes them.
• Read xor eax, eax as "eax = 0" and test rax, rax as "is rax zero?" — they are idioms, not interesting computation.
• sar is signed shift-right and shr is unsigned; the choice encodes the source-level signedness of the value.

🧪 Practice

1. Translate int x = (a + b) * 4 - 1; into x86-64 Intel-syntax assembly assuming a is in edi and b in esi, using lea at least once.
2. Write the two-instruction sequence that copies the 8 bytes at [rsi] to [rdi], then explain why one instruction is impossible.
3. Compile unsigned long f(int n){ return n; } and unsigned long g(unsigned n){ return n; } at -O1, diff the assembly, and explain what an attacker passing -1 achieves in each case.
4. Interview-style: "In a disassembly you see movsxd rax, edi immediately before a call malloc. Why does that line make you nervous?" Hint: consider the full 64-bit value produced when edi holds an attacker-controlled negative number, and what malloc does with an enormous size versus what the subsequent copy loop does.

Addressing Modes and Effective Addresses

Theory

An addressing mode is the syntax by which an instruction names the memory location it wants. x86-64 collapses all of them into one general form, and knowing that form is what lets you read array indexing, struct field access, and pointer chasing straight out of disassembly.

The complete form is:

    [ base + index * scale + displacement ]

• base — any 64-bit register (or rip).
• index — any 64-bit register except rsp.
• scale — the literal 1, 2, 4, or 8. Nothing else is encodable.
• displacement — a signed 8- or 32-bit constant.

Every component is optional. The CPU computes the sum — the effective address — then performs the access. The hardware does this in the address-generation unit essentially for free, which is why compilers lean on it so heavily.

The reason this single form is worth memorizing is that each component maps onto a specific source-level construct:

Pattern	Means	Typical source
[rax]	dereference a pointer	*p
[rbp-0x18]	local variable at a fixed frame offset	a stack local
[rax+0x10]	field at offset 16 of a struct	s->field
[rax+rcx*4]	element rcx of an int array at rax	a[i]
[rax+rcx*8+0x10]	element rcx of a long[] field at offset 16	s->arr[i]
[rip+0x2e35]	RIP-relative: a global or constant	a global variable, a string literal

The scale values 1/2/4/8 are not arbitrary: they are exactly the sizes of char, short, int, and long/pointer. So the scale tells you the element type of the array being indexed. Seeing *8 means you are looking at an array of 64-bit things — pointers, most likely, if it is followed by a dereference.

RIP-relative addressing is new in x86-64 and deserves special emphasis because it is everywhere in modern binaries. Instead of encoding an absolute address, the instruction encodes a displacement from the next instruction's address. The benefit is position independence: the same machine code works no matter where the module is loaded, which is what makes PIE (position-independent executables) and ASLR cheap. The cost for an analyst is that the displacement in the disassembly is meaningless until you add it to the instruction's own address — which is why good disassemblers annotate these with the resolved target and symbol name.

One structural detail matters for exploitation: rsp cannot be an index register. The instruction encoding has no slot for it in that position. This is why [rsp+rax] is legal (as a base) but [rax+rsp*1] is not, and why [rsp]-based addressing requires an extra encoding byte (a SIB byte).

Examples

  Effective address computation
  ───────────────────────────────────────────────────────────

    mov  rax, [rbx + rcx*8 + 0x10]
                │      │  │     │
                │      │  │     └── displacement: struct field offset,
                │      │  │         or array base offset
                │      │  └──────── scale: element SIZE -> element TYPE
                │      └─────────── index: the loop/array variable
                └────────────────── base: the object or array pointer

    effective address = rbx + (rcx * 8) + 0x10
    then: load 8 bytes from that address into rax


  RIP-relative, as it appears and as it resolves
  ───────────────────────────────────────────────────────────
    401126:  48 8d 3d d7 0e 00 00   lea rdi, [rip+0xed7]
             ^                                  ^
             address of THIS instruction        displacement
             length = 7 bytes

    next instruction address = 0x401126 + 7 = 0x40112d
    target = 0x40112d + 0xed7 = 0x402004   <- the string literal

; Each line below corresponds to one line of the C in the next block.
; rdi = struct Packet *p, esi = int i

    mov     eax, [rdi]              ; p->version       (offset 0)
    mov     eax, [rdi+4]            ; p->length        (offset 4)
    mov     rax, [rdi+8]            ; p->name          (offset 8, a pointer)
    movzx   eax, BYTE [rdi+rsi]     ; p->payload[i]    (scale 1 -> char array)
    mov     eax, [rdi+rsi*4+0x20]   ; p->counts[i]     (scale 4 -> int array)
    mov     rax, [rdi+rsi*8+0x40]   ; p->handlers[i]   (scale 8 -> pointers)
    lea     rdi, [rip+0xed7]        ; address of a string literal / global

// The structure that produces exactly the offsets above.
struct Packet {
    int    version;      // +0x00
    int    length;       // +0x04
    char  *name;         // +0x08
    char   payload[24];  // +0x10
    int    counts[8];    // +0x20  -- note: overlaps conceptually in this
                         //          illustration; see exercise 2
    void (*handlers[4])(void);  // +0x40
};

int read_version(struct Packet *p)            { return p->version; }
int read_count(struct Packet *p, int i)       { return p->counts[i]; }
void dispatch(struct Packet *p, int i)        { p->handlers[i](); }
// `dispatch` compiles to a load with scale 8 followed by an indirect call.
// If `i` is unchecked, that is a controlled jump: a prime exploitation target.

# Let objdump resolve RIP-relative targets for you.
objdump -d -M intel --no-show-raw-insn ./prog | grep 'rip+' | head
# Ghidra and objdump with symbols will append e.g. "# 402004 <message>"

Key Takeaways

• All x86-64 memory operands reduce to [base + index*scale + displacement], computed by hardware at no cost.
• The scale is restricted to 1, 2, 4, or 8, which are exactly the common element sizes — so the scale reveals the array's element type.
• A bare displacement off rbp or rsp is a local variable; a small displacement off another register is a struct field offset.
• RIP-relative addressing encodes a distance from the next instruction, enabling position-independent code; resolve it by adding the displacement to the address after the instruction.
• rsp can be a base but never an index register.
• An indexed load with scale 8 followed by an indirect call is a function-pointer table dispatch, and an unvalidated index there is directly exploitable.

🧪 Practice

1. Compute the effective address of mov rax, [rbx+rsi*4+0x18] when rbx = 0x7fff0000 and rsi = 6.
2. Given [rdi+rsi*2+0xc], state the most likely element type of the array and the byte offset of the array within the structure, and justify both.
3. A lea rdi, [rip+0x1b2] instruction is 7 bytes long and begins at 0x1149. Compute the target address, then verify your arithmetic against objdump -d output on any binary.
4. Interview-style: "You are reverse-engineering a stripped binary and see a loop containing mov rax, [rbx+rcx*8] followed by call rax. What have you found, and what do you check next?" Hint: identify the data structure implied by scale 8 plus an indirect call, then ask where rcx originates and whether anything bounds it.

Control Flow: jmp, conditional Jumps, loops

Theory

High-level languages give you if, while, for, switch, and short-circuit boolean operators. The CPU has exactly one mechanism: change rip, or do not. Every control structure you know is a pattern built from unconditional jumps, conditional jumps, and labels. Recognizing those patterns in reverse is the core skill of reading disassembly.

The essential separation, already met with flags, is that comparison and branching are different instructions:

1. cmp a, b computes a - b and discards the result, keeping only the flags. test a, b computes a AND b the same way.
2. jcc reads those flags and either sets rip to the label or falls through.

Because the flags were set by a subtraction, the condition names read in terms of the first operand: after cmp rax, rbx, jl means "jump if rax < rbx".

The conditional jumps come in signed and unsigned families, and keeping them straight is both a correctness and a security matter:

Condition	Signed mnemonic	Unsigned mnemonic	Flags tested
equal	je / jz	je / jz	ZF = 1
not equal	jne / jnz	jne / jnz	ZF = 0
less than	jl / jnge	jb / jnae	signed: SF ≠ OF; unsigned: CF = 1
less or equal	jle	jbe	signed: ZF or SF ≠ OF; unsigned: CF or ZF
greater than	jg	ja	signed: ZF=0 and SF=OF; unsigned: CF=0 and ZF=0
greater or equal	jge	jae	signed: SF = OF; unsigned: CF = 0

The mnemonic families are a reliable tell: l/g means the source type was signed; b/a ("below"/"above") means unsigned. When auditing a bounds check in disassembly, the family tells you whether a negative index can slip through.

Jumps are also either direct (the target is a constant encoded in the instruction) or indirect (jmp rax, call [rax+8] — the target comes from a register or memory). This distinction drives everything downstream: static analysis can follow direct jumps trivially, while indirect jumps require knowing a runtime value, which is why obfuscators convert direct control flow into indirect, and why control-flow-integrity defences focus exclusively on indirect transfers.

Loops compile into one of two shapes, and recognizing which you are looking at saves time:

• Test-at-top (naive, -O0): jump to the condition, evaluate, jump back to the body. Two jumps per iteration.
• Test-at-bottom / rotated (optimized): check the condition once before entering, then put the test at the end so each iteration costs one conditional jump. The -O2 form of nearly every loop.

Finally, a switch statement compiles into one of three forms depending on case density, and identifying which one tells you about the original source:

• Comparison chain for few or scattered cases — indistinguishable from else if.
• Jump table for dense cases: bounds-check the index, then jmp [table + idx*8]. The table is a readable array of code addresses in .rodata, which is a gift to an analyst reconstructing a protocol dispatcher.
• Binary search tree of comparisons for many sparse cases.

A note on cmov: compilers often replace a short if with a conditional move (cmovle rax, rbx) to avoid a branch. This has no jump at all, so a control-flow graph shows straight-line code where the source had a branch. Besides confusing beginners, this is also a constant-time programming technique, since it removes the timing and branch-prediction side channel.

Examples

  if / else
  ─────────────────────────────────────────────
      cmp   eax, edx
      jle   .Lelse          ; inverted condition jumps AWAY
      <then block>
      jmp   .Lend
  .Lelse:
      <else block>
  .Lend:

  Note: the compiler INVERTS the source condition. Source `if (a > b)`
  becomes `jle` to the else branch. Always account for this inversion.


  while loop, -O0 shape (test at top)        while loop, -O2 shape (rotated)
  ─────────────────────────────────────      ──────────────────────────────────
      jmp   .Lcond                               cmp   eax, edx      ; guard
  .Lbody:                                        jge   .Ldone        ; skip all
      <body>                                 .Lbody:
  .Lcond:                                        <body>
      cmp   eax, edx                             cmp   eax, edx
      jl    .Lbody                               jl    .Lbody
                                             .Ldone:
      2 jumps per iteration                      1 jump per iteration

; switch (op) { case 0..3: ... } compiled as a JUMP TABLE
        cmp     edi, 3
        ja      .Ldefault           ; UNSIGNED compare: one check covers both
                                    ; negative (huge when unsigned) and > 3.
                                    ; This is why compilers use `ja` here.
        mov     eax, edi
        lea     rdx, [rip+.Ltable]
        jmp     [rdx+rax*8]         ; indirect jump through the table

.Ltable:                            ; lives in .rodata; 8 bytes per entry
        dq      .Lcase0
        dq      .Lcase1
        dq      .Lcase2
        dq      .Lcase3

// Source variants worth compiling and comparing at -O0 and -O2.
int clamp(int v, int lo, int hi) {
    if (v < lo) return lo;          // -> cmp / jge, or a cmovl at -O2
    if (v > hi) return hi;
    return v;
}

int sum_to(int n) {
    int total = 0;
    for (int i = 0; i < n; i++)     // -> rotated loop at -O2
        total += i;
    return total;
}

int dispatch(int op) {
    switch (op) {                   // dense cases -> jump table
        case 0: return 10;
        case 1: return 20;
        case 2: return 30;
        case 3: return 40;
        default: return -1;
    }
}

// Short-circuit && becomes two jumps to the SAME failure label:
int both(int a, int b) { return a > 0 && b > 0; }

# See the control-flow graph rather than a linear listing.
gcc -O2 -c cf.c -o cf.o
objdump -d -M intel cf.o            # linear
r2 -q -c 'aaa; s sym.dispatch; agf' ./cf.o   # radare2 ASCII CFG

Key Takeaways

• There is one control-flow primitive: conditionally or unconditionally change rip; every if, loop, and switch is a pattern over it.
• cmp/test set flags and jcc consumes them; the compiler normally inverts the source condition so the jump skips the block.
• jl/jg indicate signed comparisons and jb/ja indicate unsigned ones — this reveals the source types and whether negative values can bypass a check.
• Direct jumps are statically resolvable; indirect jumps are not, which is why obfuscation favours them and CFI defends them.
• Optimized loops are rotated: one guard before entry, then a single conditional jump per iteration.
• A dense switch becomes a bounds check plus an indirect jump through a table of code addresses in .rodata, which is highly informative when reversing dispatchers.
• cmov implements a branch without a jump, which both hides the branch from a CFG and removes a timing side channel.

🧪 Practice

1. Hand-write Intel-syntax assembly for while (i < n) { sum += a[i]; i++; } in the rotated (test-at-bottom) form, assuming a in rdi, n in esi.
2. Compile a switch with cases 1, 2, 3, 400 and another with cases 1, 2, 3, 4, then explain from the assembly why the compiler chose different strategies.
3. Given cmp eax, 0x10 followed by jbe, state what range of eax values takes the jump, and contrast that with jle.
4. Interview-style: "An array bounds check in disassembly reads cmp edi, 0x3f / jg .Lfail. The index is used to write into a 64-element array. Is the check sound?" Hint: jg is the signed family — ask what happens when edi is negative, and what the subsequent scaled store does with that index.

The Stack: push, pop, rsp, and rbp

Theory

The stack exists to answer a problem that registers cannot: there are only sixteen of them, and functions call other functions to arbitrary depth. Each invocation needs somewhere to keep its local variables and its return address, and that storage must be created and destroyed in strict last-in-first-out order, because calls nest. A stack is the data structure that matches that lifetime exactly.

The analogy is a stack of plates, with one crucial twist: on x86-64 the stack grows downward, toward lower addresses. rsp always points at the most recently pushed item — the top of the stack, which is the lowest used address. This is purely a historical convention (it let the stack and the heap grow toward each other from opposite ends of a small address space), but it has permanent consequences:

• push decrements rsp by 8, then writes.
• pop reads, then increments rsp by 8.
• A buffer on the stack is written from low to high addresses — that is, upward, toward the saved return address. This single fact is why stack buffer overflows are exploitable at all.

Two registers describe a stack frame:

• rsp — the stack pointer. Hardware-managed by push, pop, call, and ret. It moves constantly.
• rbp — the frame pointer. A convention, not hardware: it is set once at function entry and then stays fixed, giving stable offsets to locals ([rbp-8], [rbp-0x10]) even as rsp moves during nested calls.

Because rbp is only a convention, compilers free it up when optimizing (-O2 implies -fomit-frame-pointer), addressing locals off rsp instead. That makes the code one register richer and slightly faster, and it makes manual stack-frame reconstruction harder — which is why perf and profilers often ask you to rebuild with -fno-omit-frame-pointer.

The ABI imposes two rules that are easy to forget and cause strange crashes:

1. rsp must be 16-byte aligned at the point of a call. SSE instructions such as movaps fault on misaligned access, so a one-off misalignment often manifests as a crash inside an unrelated library function.
2. There is a 128-byte red zone below rsp in the System V ABI. A leaf function (one that calls nothing) may use [rsp-8] through [rsp-128] as scratch without adjusting rsp at all, because signal handlers are required not to clobber it. This is why small leaf functions appear to have no prologue. Kernel code compiles with -mno-red-zone because interrupts do not honour it.

Security-wise, the stack is the most attacked region in the process, because it mixes three kinds of data with no separation: attacker-influenced buffers, saved register values, and return addresses. The chapter on exploitation builds directly on this layout.

Examples

  Stack growth and one frame's layout
  ─────────────────────────────────────────────────────────────────

  HIGH addresses                                   (stack base, env/argv)
      │
      │   ...caller's frame...
      │   +--------------------------+
      │   |  argument 7+ (if any)    |   args beyond 6 are passed here
      │   +--------------------------+
      │   |  RETURN ADDRESS          |  <- pushed by `call`
      │   +--------------------------+
      │   |  saved rbp               |  <- pushed by the prologue; rbp points here
      │   +--------------------------+  <= rbp
      │   |  local variables         |     addressed as [rbp-8], [rbp-0x10], ...
      │   |  char buf[64]            |     buf starts at the LOW end and is
      │   |                          |     written UPWARD toward the saved rbp
      │   +--------------------------+     and the return address
      │   |  callee-saved spills     |
      │   +--------------------------+  <= rsp   (top of stack = lowest address)
      │   |   128-byte RED ZONE      |     usable by leaf functions without
      │   +--------------------------+     adjusting rsp
      ▼
  LOW addresses                                    (grows this way)

  Overflowing `buf` writes toward HIGHER addresses, so a long enough
  write reaches saved rbp and then the RETURN ADDRESS.

; push / pop are compound operations
    push    rax             ; equivalent to:  sub rsp, 8   /  mov [rsp], rax
    pop     rax             ; equivalent to:  mov rax,[rsp] /  add rsp, 8

; Allocating locals in bulk is cheaper than repeated pushes
    sub     rsp, 0x40       ; reserve 64 bytes of locals in one instruction
    mov     QWORD [rsp+0x8], 0
    ; ...
    add     rsp, 0x40       ; release them

; Alignment maintenance, the detail that causes mysterious crashes
    push    rbx             ; rsp now ...8 (misaligned for a call)
    sub     rsp, 8          ; re-align to 16 before calling anything that
    call   some_sse_function ; might execute movaps
    add     rsp, 8
    pop     rbx

// Demonstrating the direction of growth empirically.
#include <stdio.h>

void inner(int depth) {
    int local;
    printf("depth %d: &local = %p\n", depth, (void *)&local);
    if (depth < 3) inner(depth + 1);
}

int main(void) {
    char a[8], b[8];
    // Addresses DECREASE as recursion deepens: the stack grows downward.
    inner(1);
    // Within one frame, declaration order does NOT guarantee layout order;
    // the compiler may place `a` and `b` in either relative position.
    printf("&a = %p  &b = %p\n", (void *)a, (void *)b);
    return 0;
}

# Watch rsp and the stack contents change in GDB.
gdb -q ./prog -ex 'break inner' -ex run \
    -ex 'info registers rsp rbp' \
    -ex 'x/8gx $rsp' \
    -ex 'bt'

Key Takeaways

• The stack provides per-invocation storage whose LIFO lifetime matches the nesting of function calls, which registers alone cannot supply.
• On x86-64 the stack grows toward lower addresses: push decrements rsp, pop increments it.
• Buffers on the stack are filled upward toward the saved frame pointer and return address, which is the mechanical reason stack overflows can hijack control flow.
• rsp is hardware-maintained; rbp is a software convention that optimized builds frequently discard.
• The ABI requires rsp to be 16-byte aligned at a call; violations typically surface as faults inside unrelated SSE code.
• The 128-byte red zone below rsp lets leaf functions skip prologue bookkeeping, and kernel builds must disable it.
• A stack frame deliberately interleaves attacker-reachable buffers with saved control data, which is why it needs dedicated mitigations.

🧪 Practice

1. Starting from rsp = 0x7fffffffe200, give the value of rsp after push rax, push rbx, pop rcx, and state which address rcx was loaded from.
2. Write a C function with a 32-byte local array, compile at -O0, and use objdump to determine the exact rbp-relative offset of the array and the distance in bytes from its start to the saved return address.
3. Explain why sub rsp, 0x40 is preferred over eight push instructions, considering both instruction count and alignment.
4. Interview-style: "Why does the fact that the stack grows downward while buffers are written upward matter for security?" Hint: describe what lies at higher addresses than a local buffer within and just above its frame, and which of those items the CPU will later load into rip.

Function Prologue and Epilogue

Theory

A function is not a hardware concept. The CPU only knows how to jump. "Function" is an agreement between caller and callee about who saves what, where arguments live, and how to get back — and the prologue and epilogue are the instruction sequences that implement the callee's half of that agreement.

The prologue has three jobs:

1. Save the caller's frame pointer so it can be restored later.
2. Establish this function's frame pointer so locals have stable offsets.
3. Reserve space for locals by moving rsp down.

The epilogue undoes all of it in reverse and returns. The classic unoptimized pair:

; PROLOGUE                         ; EPILOGUE
push    rbp                        mov     rsp, rbp    ; discard locals
mov     rbp, rsp                   pop     rbp         ; restore caller's frame
sub     rsp, 0x20                  ret                 ; pop return addr into rip

The leave instruction is exactly mov rsp, rbp + pop rbp in one opcode, so leave; ret is the same epilogue written shorter.

Understanding call and ret as a matched pair is essential:

• call target pushes the address of the next instruction onto the stack, then jumps to target.
• ret pops an 8-byte value off the stack into rip.

Notice what ret does not do: it does not verify that the value it pops is a legitimate return address. It takes whatever eight bytes rsp points at and starts executing there. That is not a bug in x86 — it is the design — and it is the entire basis of return-address overwrite attacks and return-oriented programming. The CPU's trust in the stack is total.

Optimized code produces prologues that look quite different, and recognizing the variations matters:

• Frame-pointer omission (-O2 default): no push rbp at all. Locals are addressed off rsp, and rbp becomes a general-purpose register.
• Leaf functions: no prologue whatsoever. They use the red zone or registers only.
• Callee-saved spills: push rbx / push r12 appear at entry because the ABI requires preserving those registers; matching pops appear at exit in reverse order.
• Stack protector (-fstack-protector-strong): the prologue loads a random canary from thread-local storage (mov rax, fs:0x28) and stores it just below the saved rbp; the epilogue compares it and calls __stack_chk_fail on mismatch. Seeing fs:0x28 is the instant tell that a function has stack-protector coverage.
• Shrink-wrapping: the compiler moves the prologue past an early-return path, so a function can have multiple prologues on different paths.

For an analyst, the prologue is also the primary function-boundary signal in a stripped binary. Disassemblers scan for push rbp; mov rbp, rsp and endbr64 (the Intel CET landing pad that begins every indirect-call target in modern builds) to guess where functions start. Obfuscators attack precisely this heuristic.

Examples

  Stack state through a call, step by step
  ───────────────────────────────────────────────────────────────

  (1) In caller, before `call foo`        rsp -> | caller local |
                                                 +--------------+

  (2) `call foo` executes                 rsp -> | RETURN ADDR  |
      -- pushes address of the                   | caller local |
         instruction after the call              +--------------+

  (3) `push rbp` (prologue)               rsp -> | saved rbp    |
                                                 | RETURN ADDR  |
                                                 | caller local |

  (4) `mov rbp, rsp`                   rsp,rbp-> | saved rbp    |  rbp anchored
                                                 | RETURN ADDR  |  here for the
                                                 | caller local |  whole call

  (5) `sub rsp, 0x20`                     rsp -> | locals (32B) |
                                                 |     ...      |
                                            rbp->| saved rbp    |
                                                 | RETURN ADDR  |

      Locals are now [rbp-0x08] .. [rbp-0x20]
      Saved rbp is   [rbp]
      Return address is [rbp+8]   <-- the exploitation target

; ─── Unoptimized (-O0): textbook frame ───────────────────────────
victim:
        push    rbp
        mov     rbp, rsp
        sub     rsp, 0x50
        mov     [rbp-0x48], rdi     ; spill the argument to a local slot
        lea     rax, [rbp-0x40]     ; address of a local buffer
        ; ...
        leave                       ; == mov rsp, rbp ; pop rbp
        ret

; ─── With -fstack-protector-strong ───────────────────────────────
guarded:
        push    rbp
        mov     rbp, rsp
        sub     rsp, 0x50
        mov     rax, QWORD fs:0x28  ; read the canary from TLS
        mov     QWORD [rbp-0x8], rax ; place it directly below saved rbp,
                                     ; so ANY linear overflow toward the
                                     ; return address must cross it first
        ; ... body ...
        mov     rax, QWORD [rbp-0x8]
        sub     rax, QWORD fs:0x28   ; unchanged?
        je      .Lok
        call    __stack_chk_fail     ; aborts the process
.Lok:
        leave
        ret

; ─── Optimized leaf (-O2): no frame at all ───────────────────────
add_two:
        endbr64                     ; CET landing pad (modern toolchains)
        lea     eax, [rdi+rsi]      ; compute and return; no stack use
        ret

# Confirm which functions in a binary carry a stack canary.
objdump -d -M intel ./prog | grep -B8 '__stack_chk_fail' | grep -E '^[0-9a-f]+ <'
# Or check the whole binary's mitigations at once:
checksec --file=./prog

Key Takeaways

• A "function" is an ABI agreement, not a hardware feature; the prologue and epilogue implement the callee's obligations.
• call pushes the return address and jumps; ret pops eight bytes into rip with no validation whatsoever.
• That unconditional trust in the stacked return address is the mechanism behind return-address overwrites and ROP.
• leave is shorthand for mov rsp, rbp; pop rbp.
• Optimized builds omit the frame pointer, give leaf functions no prologue at all, and may shrink-wrap the prologue onto selected paths.
• mov rax, fs:0x28 in a prologue is the signature of stack-protector instrumentation; the canary sits between locals and the saved return address.
• Prologue patterns and endbr64 are how disassemblers locate function boundaries in stripped binaries, so obfuscators target those patterns.

🧪 Practice

1. Write the prologue and epilogue for a function needing 48 bytes of locals that also preserves rbx and r12, keeping rsp 16-byte aligned at any internal call.
2. Compile the same function with -O0 and -O2 -fno-stack-protector, then list every difference in the prologue and explain the purpose of each.
3. Build a function with a local char buf[64] under -fstack-protector-strong and determine, from the disassembly, exactly how many bytes separate the start of buf from the canary and from the return address.
4. Interview-style: "Explain, in terms of what ret actually does, why a stack canary stops a classic overflow but does not stop a heap-based function-pointer overwrite." Hint: trace which bytes a linear stack write must pass through to reach the return address, then ask whether an indirect call through a heap pointer passes through any of them.

Calling Conventions: System V AMD64 ABI

Theory

If the prologue is the callee's half of the contract, the calling convention is the contract itself. It answers four questions that caller and callee must agree on without consulting each other: where arguments go, where the return value goes, who is responsible for preserving each register, and how the stack is left afterwards. Get it wrong by one register and the program misbehaves in ways that look like data corruption rather than a protocol error.

Linux, macOS, and the BSDs on x86-64 use the System V AMD64 ABI. Windows x64 uses a different one, and the differences are large enough that cross-platform shellcode and FFI bindings must handle them explicitly.

Integer and pointer arguments go in registers, in this order — memorize it, because it is the single most useful fact for reading disassembly:

    rdi, rsi, rdx, rcx, r8, r9     then the stack, right to left

Floating-point arguments go in xmm0–xmm7. The return value is in rax (and rdx for a 128-bit return, xmm0 for floating point).

The second half of the contract is register preservation, and it splits the sixteen registers into two classes:

Class	Registers	Meaning
Caller-saved (volatile)	rax, rcx, rdx, rsi, rdi, r8–r11, all xmm	A call may destroy these. Save them yourself if you need them afterwards.
Callee-saved (non-volatile)	rbx, rbp, r12–r15	A function must leave these exactly as it found them.

The reason for two classes is economy: if every register were callee-saved, trivial functions would pay to save registers nobody cared about; if every register were caller-saved, loops around calls would spill constantly. Splitting the set lets the compiler keep long-lived values in callee-saved registers and short-lived ones in caller-saved registers.

For analysis, this has an immediate payoff: when you see push rbx at a function's entry, you know rbx holds something the function intends to keep across its own internal calls — usually the main object or loop variable. Callee-saved registers point you at a function's important state.

Three further rules trip people up constantly:

1. Variadic functions (printf) require al to hold the number of vector registers used. Forgetting mov eax, 0 before calling printf from hand-written assembly causes a crash in the varargs prologue.
2. Structs are passed in registers if they are at most 16 bytes and have no unaligned or non-trivial members; larger ones are copied to the stack and passed by hidden pointer. A function returning a large struct receives a hidden first argument in rdi pointing at the destination, shifting every visible argument one register to the right.
3. Stack arguments are the caller's to clean up, and rsp must be 16-byte aligned at the call.

Feature	System V AMD64 (Linux/macOS)	Microsoft x64 (Windows)
Integer args	rdi, rsi, rdx, rcx, r8, r9	rcx, rdx, r8, r9
Float args	xmm0–xmm7	xmm0–xmm3 (positionally shared with integers)
Register args	6 integer + 8 float	4 total, by position
Callee-saved	rbx, rbp, r12–r15	rbx, rbp, rdi, rsi, r12–r15
Shadow space	none	caller reserves 32 bytes for the callee
Red zone	128 bytes	none
Variadics	al = vector register count	all varargs also spilled to shadow space

Examples

  Argument placement for:  int f(int a, long b, char *c, int d, int e, int g, int h)
  ──────────────────────────────────────────────────────────────────────────────

     a -> edi        (32-bit arg uses the 32-bit subregister)
     b -> rsi
     c -> rdx
     d -> ecx
     e -> r8d
     g -> r9d
     h -> [rsp]      <- 7th argument and beyond go on the stack

  Return value -> eax

  Stack at the moment of `call f`:
        rsp -> | h (8-byte slot) |     pushed by the CALLER
               | ...             |     rsp must be 16-byte aligned HERE

; Calling  write(1, msg, 13)  through libc, by hand.
        mov     edi, 1              ; arg1: fd
        lea     rsi, [rip+msg]      ; arg2: buffer
        mov     edx, 13             ; arg3: count
        call    write
        ; rax now holds the return value (bytes written, or -1)
        ; rcx, rsi, rdi, r8-r11 may have been destroyed by the call

; Calling a variadic function: `al` is MANDATORY.
        lea     rdi, [rip+fmt]      ; "value = %d\n"
        mov     esi, 42             ; the one variadic integer argument
        xor     eax, eax            ; 0 vector registers used -- omit this
        call    printf              ; and printf faults on some libc builds

; Preserving a value across a call: use a CALLEE-SAVED register.
        push    rbx                 ; we must restore it before returning
        mov     rbx, rdi            ; keep the object pointer here...
        call    helper              ; ...because rdi is caller-saved and
        mov     rax, [rbx]          ; would be clobbered by `helper`
        pop     rbx
        ret

// Struct passing: observe the hidden pointer for large returns.
struct Small { int a; int b; };            // 8 bytes  -> packed into rdi/rax
struct Big   { long a, b, c, d; };         // 32 bytes -> memory, hidden ptr

struct Small make_small(int x);            // returns in rax
struct Big   make_big(int x);              // rdi = hidden dest ptr, esi = x

// Compile with: gcc -O1 -S -masm=intel abi.c -o -
// and confirm that `make_big`'s visible argument lands in esi, not edi.

Key Takeaways

• The calling convention is the caller/callee contract covering argument location, return value, register preservation, and stack discipline.
• On System V AMD64 the integer argument order is rdi, rsi, rdx, rcx, r8, r9, then the stack right-to-left; the return value is in rax.
• Caller-saved registers (rax, rcx, rdx, rsi, rdi, r8–r11) may be destroyed by any call; callee-saved registers (rbx, rbp, r12–r15) must be restored.
• push rbx or push r12 in a prologue signals a value the function keeps alive across its own calls — often the primary object pointer.
• Variadic calls require al to hold the count of vector registers used, or they can fault.
• Large struct returns consume rdi as a hidden destination pointer, shifting all visible arguments one register right.
• Windows x64 differs substantially (four register args, shadow space, no red zone), so cross-platform low-level code must handle both.

🧪 Practice

1. List the register or stack location of every argument for void f(char *a, int b, double c, int d, double e, int g, int h, int i).
2. Write assembly that calls memcpy(dst, src, n) where dst is currently in rbx, src in r12, and n in rax, then returns dst; preserve all callee-saved registers correctly.
3. Compile a function that calls another function inside a loop, at -O2, and identify which register the compiler chose for the loop counter; explain the choice in terms of the two register classes.
4. Interview-style: "You write shellcode that works standalone but crashes when injected into a target that calls it as a function. Name three ABI obligations you may have violated." Hint: think about which registers you overwrote without saving, whether rsp was still 16-byte aligned, and whether you left the stack where the caller expects it.

System Calls and Linux ABI

Theory

Everything a program does that affects the world outside its own memory — reading a file, opening a socket, allocating pages, creating a process — must go through the kernel, because only the kernel has the privilege to touch hardware and other processes. A system call is the controlled doorway across that privilege boundary.

The need for a special mechanism follows from the hardware. User code runs in ring 3 and kernel code in ring 0; a plain call into kernel memory would fault, and even if it did not, letting user code jump to an arbitrary kernel address would eliminate the boundary entirely. So the transition is made through a single instruction that the hardware handles specially: on x86-64, syscall. It switches to ring 0 and jumps to a fixed entry point the kernel registered at boot (via the LSTAR MSR). The user process chooses which service it wants by number, not by address — the kernel's dispatch table is the only thing it can reach.

The syscall convention is deliberately similar to but distinct from the function-call convention, and confusing the two is a classic bug:

	Function call (System V)	Linux syscall
Selector	address in call	number in rax
Arg 1	rdi	rdi
Arg 2	rsi	rsi
Arg 3	rdx	rdx
Arg 4	rcx	r10
Arg 5	r8	r8
Arg 6	r9	r9
Return	rax	rax
Clobbered	caller-saved set	rcx and r11

rcx is the fourth argument for functions but is destroyed by syscall (the hardware stores the return address there, and rflags in r11). Hence r10 takes its place. Writing rcx instead of r10 in hand-written syscall code is an error that produces a confusing EINVAL or EFAULT.

Error reporting is also different from libc. The kernel returns negative errno values directly in rax: a return of -2 means ENOENT. There is no errno variable at the kernel boundary — libc's wrapper is what converts a value in [-4095, -1] into a -1 return plus errno. When reading raw syscall traces or writing shellcode, check the sign of rax.

For security work, syscalls are the natural observation and enforcement point, because they are the complete inventory of what a program can do to the outside world:

• Observation: strace and eBPF/auditd tracing record syscalls, so behavioural detection and sandbox policy are both written in syscall terms.
• Enforcement: seccomp-bpf filters syscalls by number and argument, which is how containers, browsers, and systemd services drop capability they do not need. A tight seccomp policy is one of the most effective exploit mitigations available, because post-exploitation almost always needs execve, socket, or ptrace.
• Shellcode: shellcode avoids libc precisely because invoking syscall directly needs no imports, no dynamic loader, and no symbol resolution.

One optimization worth knowing: not every "syscall" actually enters the kernel. The vDSO maps a kernel-provided page into every process so that clock_gettime and getpid-like reads can be serviced in user space. In an strace log their absence is expected, not suspicious.

Examples

  Crossing the privilege boundary
  ─────────────────────────────────────────────────────────────

   USER SPACE (ring 3)
      mov eax, 1          ; __NR_write
      mov edi, 1          ; fd = stdout
      lea rsi, [rip+msg]  ; buf
      mov edx, 13         ; count
      syscall  ────────────┐
                           │  hardware: switch to ring 0,
                           │  jump to the address in LSTAR,
                           │  save return addr in rcx, flags in r11
   ────────────────────────┼──────────────────────────────────────
   KERNEL SPACE (ring 0)   ▼
      entry_SYSCALL_64
        -> seccomp filter check (may kill or return EPERM here)
        -> bounds-check rax against the syscall table
        -> sys_write(fd, buf, count)
        -> sysret ────────┐
   ────────────────────────┼──────────────────────────────────────
   USER SPACE              ▼
      rax = bytes written, or a NEGATIVE errno (e.g. -9 = EBADF)

; A complete static program using only syscalls: no libc at all.
; Build:  nasm -f elf64 hello.asm && ld hello.o -o hello
    global _start
    section .rodata
msg:    db  "hello, syscall", 10
msglen  equ $ - msg

    section .text
_start:
    mov     eax, 1              ; __NR_write  (x86-64 numbering)
    mov     edi, 1              ; fd = 1 (stdout)
    lea     rsi, [rel msg]      ; buf
    mov     edx, msglen         ; count
    syscall                     ; rcx and r11 are now destroyed

    mov     eax, 60             ; __NR_exit
    xor     edi, edi            ; status = 0
    syscall                     ; does not return

; For contrast, the 4th-argument trap:
;   mov rcx, flags   <- WRONG for a syscall; rcx is clobbered
;   mov r10, flags   <- correct

// Bypassing libc's wrapper to see raw kernel return values.
#include <stdio.h>
#include <unistd.h>
#include <sys/syscall.h>
#include <errno.h>

int main(void) {
    // libc wrapper: returns -1 and sets errno
    int a = open("/nonexistent", 0);
    printf("libc:   ret=%d errno=%d\n", a, errno);

    // raw syscall via syscall(2), which still applies libc's errno mapping
    long b = syscall(SYS_open, "/nonexistent", 0);
    printf("syscall: ret=%ld errno=%d\n", b, errno);
    // In pure assembly with no wrapper, rax would simply contain -2 (ENOENT).
    return 0;
}

# Observe the complete external behaviour of a program.
strace -f -e trace=all -s 128 ./prog 2>&1 | head -40

# Count syscalls to build a seccomp allowlist from real behaviour.
strace -c -f ./prog

# Verify a process's active seccomp mode (2 == filter mode).
grep Seccomp /proc/self/status

Key Takeaways

• System calls are the only route from user space to kernel services, selected by number in rax rather than by address, so user code can never target arbitrary kernel code.
• The Linux syscall ABI uses rdi, rsi, rdx, r10, r8, r9 — r10 replaces rcx because syscall destroys rcx and r11.
• The kernel returns negative errno values in rax directly; the errno convention is a libc-level construct.
• Syscalls are the complete inventory of a program's external effects, which makes them the right layer for both behavioural monitoring and sandboxing.
• seccomp-bpf filtering by syscall number and arguments is a high-value mitigation because exploitation payloads almost always need execve, socket, or ptrace.
• Shellcode uses syscall directly to avoid any dependency on libc, imports, or the dynamic loader.
• vDSO-serviced calls such as clock_gettime never enter the kernel and therefore never appear in syscall traces.

🧪 Practice

1. Write a NASM program that reads up to 64 bytes from stdin and writes them back to stdout using only read, write, and exit; assemble, link statically, and run it.
2. Modify a working syscall sequence to pass the fourth argument in rcx instead of r10, run it, and explain the resulting behaviour.
3. Run strace -c on /bin/ls, then write a seccomp allowlist containing only the syscalls it actually used; identify which of them would still be needed to spawn a shell.
4. Interview-style: "An attacker achieves arbitrary code execution inside a process confined by a seccomp filter that permits only read, write, exit, and futex. What can they still accomplish?" Hint: separate what requires a new syscall from what only requires reading and writing memory and existing file descriptors.

Compilation Pipeline: Source to Object to Binary

Theory

"Compiling" is colloquially one step but is actually four, each with a distinct input, output, and failure mode. Knowing which stage you are in tells you where to look when something goes wrong, and — more importantly for security — which stage a given protection or weakness lives in. Stack canaries are inserted by the compiler; RELRO is decided by the linker; ASLR is applied by the loader. Confusing these leads to hardening flags that silently do nothing.

The four stages:

1. Preprocessing (cpp): textual. Expands #include, substitutes macros, resolves #ifdef. Output is still C, just much larger. Macro-related bugs — a MAX(a,b) that evaluates its argument twice, a missing parenthesis changing operator precedence — are born here and are invisible in the original source.
2. Compilation (cc1): C to assembly, per translation unit. This is where optimization happens, where undefined behaviour gets exploited by the optimizer, and where instrumentation (canaries, sanitizers, coverage) is injected. Crucially, the compiler sees one file at a time and knows nothing about other objects.
3. Assembly (as): assembly text to a relocatable object file. Nearly mechanical. Produces machine code with placeholders wherever an address is not yet known, plus a relocation table describing each placeholder.
4. Linking (ld): combines object files and libraries, assigns final addresses, resolves symbols, and emits an executable or shared object. This is where "undefined reference" errors come from — they are symbol-resolution failures, not syntax problems.

A fifth stage happens at run time: loading (ld.so for dynamic executables), which maps segments into memory, applies ASLR, resolves library dependencies, and performs remaining relocations.

The separation into translation units is the reason link-time optimization (LTO) exists and the reason some bugs appear only in release builds: with LTO, the optimizer finally sees across files and can exploit undefined behaviour it previously could not observe.

For security, the most important consequence of the pipeline is the distinction between where a flag applies and what verifies it:

Protection	Applied at	Flag	Verified with
Stack canary	compile	-fstack-protector-strong	objdump grep for __stack_chk_fail
Fortified libc calls	compile (preprocessor + compile)	-D_FORTIFY_SOURCE=2 -O2	symbol names like __memcpy_chk
CET / indirect-branch tracking	compile	-fcf-protection=full	endbr64 instructions; readelf -n
Non-executable stack	link	-z noexecstack	readelf -l GNU_STACK flags
RELRO	link	-z relro -z now	readelf -l GNU_RELRO; BIND_NOW
PIE / ASLR support	compile and link	-fPIE -pie	ELF type DYN vs EXEC
ASLR itself	load	kernel setting	/proc/sys/kernel/randomize_va_space

Note that _FORTIFY_SOURCE requires optimization to be enabled, because it relies on the compiler knowing object sizes — a build at -O0 silently gets nothing. This is a very common real-world hardening gap.

It is also worth keeping the artifact types straight, since tooling behaves differently on each:

• .o relocatable object — unresolved symbols, relocations pending, no entry point.
• .a static archive — just a bundle of .o files; the linker copies in only the members it needs.
• .so shared object — position-independent, loadable at run time, has its own dynamic symbol table.
• executable — ET_EXEC (fixed load address) or ET_DYN (PIE, relocatable, which is the modern default).

Examples

  The pipeline, with the artifact at each boundary
  ───────────────────────────────────────────────────────────────────

   main.c  util.c
      │       │
      │ cpp   │     #include expansion, macro substitution
      ▼       ▼
   main.i  util.i        (still C, often 50x larger)
      │       │
      │ cc1   │     OPTIMIZATION + instrumentation
      ▼       ▼          <- canaries, ASan, coverage inserted HERE
   main.s  util.s        (assembly text)
      │       │
      │ as    │
      ▼       ▼
   main.o  util.o        (machine code + RELOCATIONS, addresses unknown)
      │       │
      └───┬───┘  +  libc.so, libm.a
          │ ld        symbol resolution, address assignment
          ▼              <- RELRO, noexecstack, PIE decided HERE
      ./prog             (ET_DYN if PIE)
          │
          │ ld.so     at execution: mmap segments, APPLY ASLR,
          ▼           resolve dynamic symbols
     running process

# Stop after each stage and inspect the intermediate artifact.
gcc -E hello.c -o hello.i      # 1. preprocess  (inspect: wc -l hello.i)
gcc -S hello.i -o hello.s      # 2. compile     (inspect: cat hello.s)
gcc -c hello.s -o hello.o      # 3. assemble    (inspect: objdump -dr hello.o)
gcc hello.o -o hello           # 4. link        (inspect: readelf -d hello)

# What the linker actually invokes, with every implicit flag revealed:
gcc -v hello.o -o hello 2>&1 | grep collect2

# A fully hardened build line, with each flag's stage noted:
gcc -O2 \
    -fstack-protector-strong \     # compile: canaries
    -fstack-clash-protection \     # compile: probe large stack allocations
    -fcf-protection=full \         # compile: endbr64 / CET
    -D_FORTIFY_SOURCE=2 \          # compile: needs -O2 or it does nothing
    -fPIE -pie \                   # compile+link: PIE for full ASLR
    -Wl,-z,relro,-z,now \          # link: full RELRO
    -Wl,-z,noexecstack \           # link: NX stack
    app.c -o app

# Verify the result rather than trusting the flags.
checksec --file=./app
readelf -lW ./app | grep -E 'GNU_STACK|GNU_RELRO'
readelf -d ./app | grep -E 'BIND_NOW|FLAGS'

// A preprocessor-born bug that no amount of reading the .c file reveals.
#define SQUARE(x) x * x            // missing parentheses
#define MAX(a, b) ((a) > (b) ? (a) : (b))   // evaluates an argument twice

int main(void) {
    int n = 2;
    int wrong = SQUARE(n + 1);     // expands to: n + 1 * n + 1  ==  5, not 9
    int i = 0;
    int m = MAX(i++, 3);           // i++ evaluated twice on one path
    // Run `gcc -E` to see exactly what the compiler received.
    return wrong + m;
}

Key Takeaways

• Compilation is four stages — preprocess, compile, assemble, link — plus loading at run time, and each has distinct failure modes.
• The compiler sees one translation unit at a time; LTO is what lets the optimizer reason across files, which can change the behaviour of undefined-behaviour-dependent code.
• Hardening flags apply at specific stages: canaries and CET at compile time, RELRO and NX at link time, ASLR at load time.
• -D_FORTIFY_SOURCE=2 is silently inert without optimization enabled, a frequent real-world gap.
• PIE requires both -fPIE at compile and -pie at link; a non-PIE binary limits ASLR to libraries only.
• Always verify mitigations on the produced artifact with checksec or readelf rather than trusting the build flags.
• Preprocessor bugs are invisible in the source file; gcc -E shows what the compiler actually consumed.

🧪 Practice

1. Run each pipeline stage separately on a two-line C program and record the size of each intermediate artifact; explain why the preprocessed file is so much larger.
2. Build the same program twice, once with -D_FORTIFY_SOURCE=2 -O0 and once with -D_FORTIFY_SOURCE=2 -O2, then use nm -D to show which build actually got the fortified __*_chk functions.
3. Take a program with a deliberately out-of-bounds stack write and determine, by testing, which of -fstack-protector-strong, -z relro, and ASLR affects the outcome, explaining each result by reference to the stage at which it applies.
4. Interview-style: "A team reports they have 'enabled all hardening flags' but a scanner still flags their binary as missing NX and RELRO. Where do you look?" Hint: separate flags the compiler consumes from flags that must reach the linker, and consider how a build system might pass CFLAGS without passing LDFLAGS.

ELF File Format and Sections

Theory

An executable file is not a memory image. It is a structured archive that tells the kernel and the dynamic loader how to build a memory image. On Linux that structure is ELF (Executable and Linkable Format), and reading it is a prerequisite for almost all binary analysis, because every analysis tool's first job is to parse it.

ELF's central design idea is that the same file is described twice, for two different audiences:

• Sections (.text, .data, .rodata, …) are the linker's view: fine-grained, named, carrying relocation and symbol information. Dozens of them. They are needed to build a program.
• Segments, a.k.a. program headers, are the kernel's view: a handful of coarse, page-aligned regions each with permissions, saying "map these bytes at this address with these flags". They are needed to run a program.

Several sections are grouped into one segment because they share permissions. .text and .rodata both end up in a read-only executable or read-only segment; .data and .bss land in a writable one. This grouping is why a single R E segment can contain read-only data: data in an executable segment is usable as ROP material, which is a real consequence of the grouping.

The practical upshot: a stripped binary has lost its section-header table and symbols, and still runs perfectly, because execution needs only the program headers. The sections were for the linker, and the linker is already done.

The layout of the file:

Part	Purpose
ELF header	Magic \x7fELF, class (64-bit), endianness, type (EXEC/DYN/REL/CORE), machine, entry point, offsets to the two header tables
Program headers	The segment table: what to map where, with which permissions
Sections	The actual content
Section headers	Names, sizes, types, flags for each section

The sections you must be able to recognize:

Section	Contents	Perms	Analysis relevance
.text	machine code	R-X	the code you disassemble
.rodata	string literals, constants, jump tables, vtables	R--	strings and switch tables live here
.data	initialized writable globals	RW-	configuration, writable function pointers
.bss	zero-initialized globals; occupies no file space	RW-	large buffers; explains file-vs-memory size gaps
.plt / .plt.sec	call stubs for imported functions	R-X	the trampolines you see at every library call
.got / .got.plt	resolved addresses of imports	RW-/R--	GOT overwrite target; RELRO protects this
.dynsym / .dynstr	dynamic symbols and their names	R--	survives stripping; lists imports/exports
.symtab / .strtab	full symbol table	—	removed by strip
.rela.dyn / .rela.plt	relocation entries	R--	what the loader must patch
.init_array / .fini_array	constructor/destructor pointers	RW-/R--	code that runs before main; a persistence target
.eh_frame	unwind tables	R--	lets you recover function boundaries in stripped binaries
.note.gnu.build-id	unique build hash	R--	links a binary to its symbols/source build

Two details that matter in practice. First, .bss has a file size of zero but a memory size that can be gigabytes — it is described, not stored. An ELF that claims a 2 GB .bss is a few kilobytes on disk. Second, .init_array holds pointers that run before main, which makes it both a legitimate mechanism (C++ static constructors, __attribute__((constructor))) and a classic place to hide malicious initialization.

Finally, ET_EXEC versus ET_DYN is how you tell PIE from non-PIE: a PIE executable is reported as a shared object (DYN) with an entry point near zero, because its real load base is chosen at run time.

Examples

  One ELF file, two views of the same bytes
  ───────────────────────────────────────────────────────────────────

  FILE (section view, the LINKER's)        MEMORY (segment view, the KERNEL's)
  ┌─────────────────────────────┐
  │ ELF header                  │
  ├─────────────────────────────┤
  │ Program headers             │─────┐
  ├─────────────────────────────┤     │   0x400000  ┌──────────────┐
  │ .init  .plt  .text          │─────┼──────────►  │  LOAD   R-X  │ code
  ├─────────────────────────────┤     │             ├──────────────┤
  │ .rodata  .eh_frame          │─────┼──────────►  │  LOAD   R--  │ consts
  ├─────────────────────────────┤     │             ├──────────────┤
  │ .init_array  .got  .data    │─────┼──────────►  │  LOAD   RW-  │ globals
  ├─────────────────────────────┤     │             │  .bss (zero- │
  │ .bss  (size only, no bytes) │─ ─ ─┘             │   filled by  │
  ├─────────────────────────────┤                   │   the kernel)│
  │ .symtab .strtab  (stripped) │ not mapped        └──────────────┘
  ├─────────────────────────────┤
  │ Section headers (stripped)  │ not mapped
  └─────────────────────────────┘

  Stripping removes the bottom two groups. The program still runs,
  because running needs only the PROGRAM headers.

# The ELF header: type, entry point, machine.
readelf -h ./prog
#   Type: DYN (Position-Independent Executable)  <- PIE
#   Entry point address: 0x1060                  <- small => load base added later

# Segments, their permissions, and the section-to-segment mapping.
readelf -lW ./prog

# Sections with sizes and flags.
readelf -SW ./prog

# Demonstrate that .bss costs nothing on disk.
printf 'char big[100*1024*1024];\nint main(void){return big[0];}\n' > bss.c
gcc bss.c -o bss
ls -l bss                       # a few kilobytes
readelf -SW bss | grep '\.bss'  # ~100 MB memory size, offset shared, no file data

# Dynamic symbols survive stripping; full symbols do not.
nm -D ./prog | head             # .dynsym: imports and exports
nm ./prog | head                # .symtab: gone after `strip`
strip ./prog && nm ./prog       # "no symbols"

# Code that runs before main -- check this in any suspicious binary.
objdump -s -j .init_array ./prog

// .init_array in action: this executes before main.
#include <stdio.h>

__attribute__((constructor))
static void early(void) {
    // A pointer to this function is placed in .init_array by the compiler,
    // and the C runtime calls every entry before invoking main.
    puts("runs before main");
}

// Section placement made explicit, as malware and packers do.
__attribute__((section(".rodata")))
const char banner[] = "read-only, lands in .rodata";

int main(void) { puts(banner); return 0; }

Key Takeaways

• ELF describes the same file twice: sections for the linker (fine-grained, named) and segments for the kernel (coarse, page-aligned, permissioned).
• Execution requires only program headers, which is why stripped binaries run fine without section headers or symbols.
• Sections are grouped into segments by permission, so read-only data shares an executable segment with code and becomes available as ROP material.
• .bss stores only a size, so memory footprint can vastly exceed file size.
• .dynsym survives stripping and still reveals imports and exports; .symtab does not.
• .init_array holds function pointers invoked before main, which is both a language feature and a code-execution hiding place.
• ET_DYN with a small entry point indicates a PIE binary, whose real base address is chosen at load time.

🧪 Practice

1. Run readelf -lW on /bin/ls and map every section into its containing segment, then state each segment's permissions.
2. Create a program with a 50 MB global array and verify from ls -l and readelf -S that the file remains small; explain precisely what the kernel does at load time.
3. Strip a binary, then use nm -D, readelf -SW, and objdump -d to determine how much analysis capability you lost and how much remains.
4. Interview-style: "A binary you are analysing has no main-like symbol and appears to do nothing in .text, yet it exhibits network activity immediately on execution. Where do you look first?" Hint: ask what code the C runtime executes before main is ever reached, and which section holds the pointers to it.

Linking, Relocations, and the PLT/GOT

Theory

When the assembler processes call printf, it does not know where printf will be. It cannot: printf lives in a shared library that will be mapped at an address chosen at run time. So the assembler emits a placeholder and records a relocation — a note saying "at file offset X, patch in the address of symbol printf, computed this way". Linking and loading are the processes of consuming those notes.

Relocations are resolved in two waves:

1. Static linking (ld, at build time) resolves everything it can: references between your own object files, and references into static libraries, which are copied into the output.
2. Dynamic linking (ld.so, at load time) resolves what remains: references into shared libraries, whose load addresses are now known.

Dynamic linking exists because the alternative is untenable: statically linking libc into every binary multiplies disk and memory use, and a security fix in libc would require rebuilding every program on the system. The trade-off is a startup cost and a run-time indirection — and that indirection is where the PLT and GOT come in.

The problem they solve: .text must be read-only and shared between processes (that is most of the memory saving), so it cannot be patched with resolved addresses — patching it would make each process's copy unique. The solution is to route every external reference through a writable table:

• The GOT (Global Offset Table, .got.plt) is an array of pointers in writable memory. One entry per imported function.
• The PLT (Procedure Linkage Table, .plt) is an array of tiny code stubs in .text. One stub per imported function. Each stub does jmp [GOT entry].

Your code calls the PLT stub (a fixed, local address — no relocation needed in .text). The stub jumps to whatever the GOT entry contains. The loader writes the real address into the GOT. Read-only code, resolved addresses, one extra jump.

On top of that sits lazy binding, the default: resolving every symbol at startup is wasted work for functions never called. So the loader initially points each GOT entry back into the PLT, at a stub that pushes the symbol's index and jumps to the resolver (_dl_runtime_resolve). The resolver finds the real address, writes it into the GOT, and jumps to it. Every later call goes straight through. The first call to a function is slow; the rest are one indirection.

The security consequence is direct and historically important. With lazy binding, the GOT must be writable for the lifetime of the process. An attacker with an arbitrary write primitive can overwrite a GOT entry — say, the one for free — with the address of system, and the program's own next call to free(ptr) becomes system(ptr). This is GOT overwrite, and it was a standard exploitation technique.

The mitigation is RELRO:

Mode	Flag	Effect
None	-z norelro	GOT fully writable; GOT overwrite trivially available
Partial	-z relro (common default)	Non-PLT GOT and .init_array made read-only after relocation; .got.plt stays writable for lazy binding
Full	-z relro -z now	All symbols resolved eagerly at startup, then the entire GOT is mapped read-only; GOT overwrite eliminated

Full RELRO trades a small startup cost for the removal of an entire exploitation primitive, and it is the right default for anything security-relevant.

Two related mechanisms worth knowing: symbol interposition means the first definition found in the search order wins, which is what makes LD_PRELOAD function hooking possible (covered later under instrumentation); and DT_RPATH/DT_RUNPATH embed library search paths in the binary, which is a genuine privilege-escalation vector when a setuid binary carries a writable or relative rpath.

Examples

  First call to printf@plt with LAZY binding
  ─────────────────────────────────────────────────────────────────────

  your code            .plt (R-X)              .got.plt (RW-)
  ┌──────────────┐    ┌──────────────────┐    ┌─────────────────────┐
  │ call         │───►│ printf@plt:      │    │ [printf] ──┐        │
  │  printf@plt  │    │   jmp [GOT+n] ───┼───►│            │        │
  └──────────────┘    │   push n         │◄───┘  initially │        │
                      │   jmp .plt[0] ───┼───┐  points BACK to the  │
                      └──────────────────┘   │  next PLT instruction│
                                             ▼                      │
                                   _dl_runtime_resolve              │
                                     - find printf in libc          │
                                     - WRITE its address ───────────┘
                                     - jump to it

  SECOND and later calls
  ┌──────────────┐    ┌──────────────────┐    ┌─────────────────────┐
  │ call         │───►│ printf@plt:      │    │ [printf] = 0x7f..a0 │
  │  printf@plt  │    │   jmp [GOT+n] ───┼───►└──────────┬──────────┘
  └──────────────┘    └──────────────────┘               ▼
                                                   libc printf

  ATTACK: an arbitrary write into .got.plt replaces 0x7f..a0 with
  &system. The program's own next `printf(buf)` executes system(buf).
  FULL RELRO maps .got.plt read-only after startup, removing this.

# 1. See the unresolved relocation in an object file.
printf '#include <stdio.h>\nint main(void){puts("hi");return 0;}\n' > r.c
gcc -c r.c -o r.o
objdump -dr r.o | grep -A1 'call'
#   call  <offset>           and a relocation line:
#   R_X86_64_PLT32   puts-0x4      <- "patch this with puts' PLT address"

# 2. See the PLT stub and its GOT reference in the linked binary.
gcc r.c -o r
objdump -d -j .plt -M intel ./r
objdump -R ./r                 # dynamic relocations, GOT slot per symbol

# 3. Watch lazy binding happen, live.
gdb -q ./r \
  -ex 'break main' -ex run \
  -ex 'info function puts' \
  -ex 'x/gx &puts@got.plt'    # before: points into .plt
# step past the first puts call, then re-read the same address:
#                             # after:  points into libc

# 4. Compare RELRO modes on the same source.
gcc r.c -o r_none -Wl,-z,norelro
gcc r.c -o r_part -Wl,-z,relro
gcc r.c -o r_full -Wl,-z,relro,-z,now
for b in r_none r_part r_full; do printf '%s: ' "$b"; checksec --file=./$b 2>/dev/null | tail -1; done
readelf -d ./r_full | grep -E 'BIND_NOW|FLAGS'   # BIND_NOW present => eager

# 5. Force eager binding on an existing binary without rebuilding.
LD_BIND_NOW=1 ./r

# 6. Inspect embedded library search paths (a setuid risk).
readelf -d ./prog | grep -E 'RPATH|RUNPATH'

// A GOT overwrite in miniature: an arbitrary write turned into execution.
// Build WITHOUT full RELRO to observe it:
//   gcc -no-pie -Wl,-z,norelro got.c -o got
#include <stdio.h>
#include <stdlib.h>

extern void *_GLOBAL_OFFSET_TABLE_;

int main(void) {
    // In a real exploit the attacker does not know these addresses; they
    // are obtained from a leak. The point here is only the mechanism.
    void **got_puts = (void **)&puts;   // conceptual: the GOT slot for puts
    (void)got_puts;

    // With a writable GOT, replacing the slot for `puts` with `system`
    // means this ordinary-looking call executes a shell command instead.
    puts("/bin/sh");                    // becomes system("/bin/sh")

    // With FULL RELRO the write itself faults: the page is read-only.
    return 0;
}

Key Takeaways

• A relocation is a recorded instruction to patch an address that was unknown at assembly time; linking and loading consume them in two waves.
• Dynamic linking saves memory and makes library security updates possible, at the cost of startup work and one level of indirection.
• .text must stay read-only and shareable, so external calls route through the PLT (code stubs) and the GOT (writable pointer table).
• Lazy binding resolves each symbol on its first call by having the resolver write the real address into the GOT.
• A writable GOT plus an arbitrary-write primitive yields code execution via GOT overwrite; the program's own subsequent call does the work.
• Partial RELRO leaves .got.plt writable; only full RELRO (-z relro -z now) resolves eagerly and maps the whole GOT read-only.
• LD_BIND_NOW=1 forces eager binding at run time, and embedded RPATH/RUNPATH entries are a privilege-escalation surface in setuid binaries.

🧪 Practice

1. Compile a one-line program calling puts, then use objdump -dr on the object file to show the relocation and objdump -R on the executable to show the resulting GOT slot.
2. In GDB, print the GOT entry for a library function before and after its first call, and explain the two different values you see.
3. Build the same source with no, partial, and full RELRO; use readelf -lW and checksec to document the difference, and identify precisely which section changes permission.
4. Interview-style: "Why does full RELRO cost startup time, and when would you nevertheless choose partial RELRO?" Hint: relate eager resolution to the number of imported symbols a program has versus the number it actually calls, and consider a process whose startup latency is the primary performance metric.

Reading Compiler-Generated Assembly

Theory

Reading disassembly efficiently is a pattern-recognition skill, not a translation exercise. A beginner reads instruction by instruction and drowns; an experienced analyst skims for landmarks, recognizes idioms as single concepts, and only reads closely where the behaviour is actually in question. The difference is knowing what the compiler routinely emits, so that routine code can be skipped.

The most important preparation is realizing that optimized assembly does not correspond line-by-line to source. The compiler is free to, and will:

• Inline functions, so a call you expect is simply absent.
• Unroll loops, so one source loop becomes four copies of the body.
• Vectorize, turning a byte loop into xmm/ymm register operations that look nothing like the source.
• Reorder independent operations and hoist loop-invariant computation out.
• Replace arithmetic: division by a constant becomes a multiply by a magic reciprocal plus shifts; x % 8 becomes and; x * 9 becomes lea.
• Delete code whose only effect is undefined behaviour — including, notoriously, a memset that zeroes a buffer just before it goes out of scope, which is why explicit_bzero exists.

So the practical workflow is: read -O0 output when you want to understand the source structure; read -O2 output when you need to understand what will actually run. For vulnerability work it must be the latter, because the bug may exist only in the optimized form.

A reading strategy that works:

1. Establish the boundaries. Find the prologue, the epilogue, and every ret.
2. Identify the arguments. Note the first use of rdi, rsi, rdx, rcx, r8, r9 — those uses tell you the parameters and often their types (a 32-bit use means int; a dereference means a pointer).
3. Find the calls. The call targets outline what the function does better than any individual instruction. A strcpy call tells you more in one second than reading fifty instructions.
4. Find the loops. Any backward jump is a loop; identify the counter and the bound.
5. Only then read the arithmetic, and only in the blocks that matter.

An idiom dictionary saves enormous time. Read these as single words:

Pattern	Meaning
xor eax, eax	eax = 0
test rax, rax / je	if (p == NULL)
lea rax, [rdi+rdi*2]	multiply by 3
mov eax, edi / shr eax, 31 / add / sar	signed division by a power of two, with rounding fixup
imul rax, rcx, 0xAAAAAAAB + shr	division by a constant via magic reciprocal
movzx eax, al after setcc	a boolean result being widened
mov rax, fs:0x28	stack-protector canary load
endbr64 at a function's first instruction	CET indirect-branch landing pad
rep stosq / movdqa loops	memset/memcpy inlined
cdqe, cqo	sign-extension before use or division

Finally, use the tools that remove friction: gcc -S -masm=intel -fverbose-asm annotates each instruction with the source variable it refers to, -fno-asynchronous-unwind-tables removes .cfi noise, and objdump -S interleaves source with disassembly when debug info is present. For quick experiments, a local Compiler Explorer instance or gcc -S on a two-line function is the fastest way to learn what a given construct compiles to.

Examples

// Source for the comparison below.
int sum_bytes(const unsigned char *p, int n) {
    int total = 0;
    for (int i = 0; i < n; i++)
        total += p[i];
    return total;
}

; ─── -O0: structurally faithful to the source, verbose ────────────
sum_bytes:
        push    rbp
        mov     rbp, rsp
        mov     [rbp-0x18], rdi      ; spill p
        mov     [rbp-0x1c], esi      ; spill n
        mov     DWORD [rbp-4], 0     ; total = 0
        mov     DWORD [rbp-8], 0     ; i = 0
        jmp     .Lcond               ; test-at-top loop
.Lbody:
        mov     eax, [rbp-8]         ; i
        movsxd  rdx, eax             ; sign-extend i for addressing
        mov     rax, [rbp-0x18]      ; p
        add     rax, rdx             ; p + i
        movzx   eax, BYTE [rax]      ; p[i], ZERO-extended (unsigned char)
        add     [rbp-4], eax         ; total += p[i]
        add     DWORD [rbp-8], 1     ; i++
.Lcond:
        mov     eax, [rbp-8]
        cmp     eax, [rbp-0x1c]
        jl      .Lbody               ; SIGNED compare: i and n are int
        mov     eax, [rbp-4]
        pop     rbp
        ret

; ─── -O2: same meaning, different shape entirely ──────────────────
sum_bytes:
        endbr64
        test    esi, esi             ; guard: is n <= 0?
        jle     .Lempty              ; if so, return 0 without looping
        mov     eax, esi             ; (at higher -O levels this becomes a
        ...                          ;  vectorized paddb/psadbw loop over
        xor     eax, eax             ;  16 bytes per iteration -- the source
.Lloop:                              ;  loop is gone entirely)
        movzx   edx, BYTE [rdi]
        add     rdi, 1
        add     eax, edx
        cmp     rdi, rcx
        jne     .Lloop               ; rotated: ONE jump per iteration
        ret
.Lempty:
        xor     eax, eax
        ret

# Annotated assembly: each line tagged with the source variable.
gcc -O2 -S -masm=intel -fverbose-asm -fno-asynchronous-unwind-tables s.c -o -

# Source interleaved with disassembly (requires -g).
gcc -O2 -g -c s.c -o s.o && objdump -S -M intel s.o

# Confirm whether a call was inlined away.
gcc -O2 -c s.c -o s.o && nm s.o | grep helper   # absent => inlined

# Watch the optimizer delete a security-relevant memset.
cat > wipe.c <<'EOF'
#include <string.h>
void handle(char *pw) {
    char copy[64];
    strcpy(copy, pw);
    memset(copy, 0, sizeof copy);   // DEAD STORE: copy is never read again,
}                                   // so -O2 may remove this entirely.
EOF
gcc -O2 -S -masm=intel wipe.c -o - | grep -c 'memset\|rep stos'
# Use explicit_bzero() or memset_s() instead, which the compiler may not elide.

Key Takeaways

• Optimized assembly has no line-by-line correspondence to source; inlining, unrolling, vectorization, and strength reduction all break it.
• Read -O0 to understand source structure and -O2 to understand what will actually execute — vulnerability analysis requires the latter.
• Work top-down: boundaries, then arguments, then calls, then loops, and read arithmetic last and only where it matters.
• Call targets are the highest-information instructions in a function; a single strcpy tells you more than dozens of arithmetic lines.
• Maintain an idiom dictionary so routine patterns (xor eax,eax, test/je, magic-reciprocal division, canary loads) are read as single concepts.
• The optimizer legally deletes dead stores, including buffer-wiping memset calls, which is why explicit_bzero exists.
• -fverbose-asm, -masm=intel, and objdump -S substantially reduce the effort of reading compiler output.

🧪 Practice

1. Compile a function computing n / 10 for a 32-bit int at -O2 and explain the magic constant and shift sequence the compiler emits instead of idiv.
2. Write a loop summing an int array, compile at -O0, -O2, and -O3, and describe how the loop's shape changes at each level.
3. Reproduce the dead-store elimination of a memset that wipes a local buffer, then show that explicit_bzero survives; explain why that difference exists.
4. Interview-style: "You are told a bug reproduces only in release builds. Name three optimizer behaviours that could cause that and how you would confirm each." Hint: consider undefined behaviour that the optimizer is permitted to assume never happens, the removal of code with no observable effect, and changes in stack and register layout that alter what an overflow actually overwrites.

Writing and Assembling Minimal Programs

Theory

Reading assembly and writing it exercise different muscles, and writing a small program by hand is the fastest route to confidence. When you write assembly you must make every decision the compiler normally makes silently: which register holds what, how the stack stays aligned, where a string constant lives, and how the program terminates. Nothing can be glossed over, because nothing is supplied for you. For an analyst this is valuable in both directions — building a tiny program teaches you to recognize the same structures when you later disassemble someone else's.

The first choice is the entry model. A normal C program starts at main, but main is called by the C runtime (_start → __libc_start_main → main), which sets up argc/argv, the environment, locale, and buffered I/O. A minimal program can skip all of that by defining _start directly and talking to the kernel through syscall. The result links with no C library at all, which makes it an excellent object of study: its disassembly contains only the code you wrote, with none of the runtime boilerplate that normally dominates a small binary.

Three practical constraints appear immediately when you work at this level, and understanding them is exactly what makes real-world binary analysis tractable:

1. No libc means no convenience functions. There is no printf, no malloc. Output is a write syscall; exit is an exit syscall. This is why malware and firmware that avoid libc are harder to read — the helpful landmarks are gone.
2. Position independence. A program that does not know its own load address must reference its data rip-relatively rather than by absolute address. NASM's [rel msg] and GAS's msg(%rip) express this. Recognizing rip-relative data references is how you locate strings and tables in a PIE binary.
3. Termination is explicit. If _start simply "returns", there is nothing to return to and the process crashes with SIGSEGV. The exit syscall is mandatory, which is a small but memorable lesson about how the process model actually works.

The toolchain for hand-written assembly has two common front ends. NASM uses Intel syntax and assembles to an object file that ld then links; the GNU assembler (as/gas) uses AT&T syntax by default and is what gcc drives internally. For learning, NASM plus ld keeps the pipeline visible and matches the Intel syntax used throughout this chapter.

Finally, building minimal programs is the natural place to connect back to the ELF format from earlier: a hand-written _start program, assembled and linked, is the smallest possible real ELF file, and inspecting it with readelf shows the program headers, the single executable segment, and the entry point you chose — with no sections or symbols you did not create. It is the cleanest specimen available for practising the inspection commands.

Examples

; A complete freestanding program: write a message, then exit.
; No libc, no _start boilerplate -- only the code written here.
; Build:  nasm -f elf64 min.asm -o min.o
;         ld min.o -o min
;         ./min ; echo "exit status: $?"

        global  _start

        section .rodata
msg:    db      "minimal program", 10      ; 10 = newline
msglen  equ     $ - msg                     ; assembler computes the length

        section .text
_start:
        ; write(1, msg, msglen)
        mov     eax, 1                       ; __NR_write
        mov     edi, 1                       ; fd = stdout
        lea     rsi, [rel msg]               ; rip-relative data reference
        mov     edx, msglen
        syscall

        ; exit(0) -- mandatory; without it the process faults on "return"
        mov     eax, 60                      ; __NR_exit
        xor     edi, edi                     ; status = 0
        syscall

# Build and inspect the smallest real ELF you can produce.
nasm -f elf64 min.asm -o min.o
ld min.o -o min
./min

# The object file: your code plus any relocations, before linking.
objdump -dr -M intel min.o

# The linked binary is a clean specimen for the ELF commands from earlier.
readelf -h min            # entry point == your _start; type EXEC (no -pie here)
readelf -lW min           # one LOAD R-X segment, one LOAD R-- for .rodata
size min                  # text/data/bss breakdown, all tiny
file min                  # "statically linked", because there is no libc

# Contrast: the same "hello" through gcc pulls in the whole C runtime.
printf '#include <stdio.h>\nint main(void){puts("hi");return 0;}\n' > h.c
gcc h.c -o h_dyn
objdump -d -M intel h_dyn | grep -c '^[0-9a-f]'   # far more code than you wrote

; The same program in GNU assembler (AT&T) syntax, for comparison.
; Build:  as att.s -o att.o && ld att.o -o att
        .global _start
        .section .rodata
msg:    .ascii "minimal program\n"
        .set msglen, . - msg

        .text
_start:
        movl    $1, %eax            # __NR_write   (note: src, dst order)
        movl    $1, %edi            # fd
        leaq    msg(%rip), %rsi     # rip-relative, AT&T spelling
        movl    $msglen, %edx
        syscall

        movl    $60, %eax           # __NR_exit
        xorl    %edi, %edi
        syscall

Key Takeaways

• Writing a minimal program forces every decision the compiler usually hides — register use, stack alignment, data placement, and termination — which is the fastest way to internalize them.
• Defining _start instead of main skips the C runtime and links with no libc, producing a binary whose disassembly is only the code you wrote.
• Without libc there are no convenience functions; output and exit are raw syscalls, which mirrors why libc-free malware and firmware are harder to read.
• A program that does not know its load address must use rip-relative data references, and recognizing those is how you find strings and tables in PIE binaries.
• Termination must be an explicit exit syscall; "returning" from _start faults, which illustrates the process model concretely.
• A hand-assembled _start program is the smallest real ELF file and the cleanest specimen for practising readelf/objdump inspection.

🧪 Practice

1. Assemble and link the minimal _start program above, run it, and confirm its exit status is zero; then remove the exit syscall and explain the crash that results.
2. Use readelf -hlW on your minimal binary and on a gcc-built hello-world, and list every structural difference you observe.
3. Rewrite the program to read a line from stdin with the read syscall and echo it back with write, keeping it libc-free.
4. Interview-style: "You are handed a small statically linked ELF with a single _start, no symbols, and no libc calls. What does that combination tell you about the author's intent and the difficulty of analysis?" Hint: consider what landmarks a libc-free, symbol-free binary removes for an analyst, and why someone might choose to build one that way.

<a id="42-static-program-analysis"></a>

4.2 Static Program Analysis

Static analysis reasons about a program without executing it, reading the binary or source to recover structure and prove properties; this section covers the tools that turn raw bytes into readable code and the formal techniques that reason about all possible executions at once.

Disassembly with objdump and Ghidra

Theory

Disassembly is the inverse of assembly: turning machine-code bytes back into mnemonic instructions. It sounds mechanical, and for a single known instruction it is, but doing it across a whole binary runs into a genuinely hard problem that shapes every tool you will use.

The problem is that on x86-64, code and data are mixed in the same address space, and instructions have variable length (1 to 15 bytes). A disassembler must decide where each instruction begins, and there is no marker. If it starts one byte too early or too late, it decodes a completely different, equally valid-looking instruction stream. This is not a rare edge case — it is the central difficulty, and it is why two disassemblers can disagree about the same bytes.

Two strategies address it, with opposite trade-offs:

Strategy	How it works	Strength	Weakness
Linear sweep	Decode from the start, one instruction after another, straight through	Covers every byte; simple	Misinterprets inline data (jump tables, constants) as instructions and desynchronizes
Recursive descent	Follow control flow: decode an instruction, then follow its jumps/calls to find more code	Does not decode data as code; accurate	Misses code reachable only through indirect jumps it cannot resolve

objdump uses linear sweep. It is fast, scriptable, and ideal for quick inspection, but it will cheerfully turn a jump table in .text into nonsense instructions. Professional tools — Ghidra, IDA, Binary Ninja, radare2 — use recursive descent augmented with heuristics (function-prologue scanning, symbol hints, and increasingly, analysis of indirect jumps) to recover what pure recursion misses.

The reason to graduate from objdump to a tool like Ghidra is not just accuracy; it is interactivity and accumulated context. Ghidra builds a database of the whole program: cross-references (who calls this function, who reads this global), named variables you can rename as you understand them, type information you can apply, and a decompiler. Analysis is iterative — you rename a function to parse_header, and suddenly every call site becomes readable — and a linear text dump cannot support that loop. Ghidra is also free and open-source (released by the NSA in 2019), which is why it has become the default teaching tool.

A critical safety point: disassemblers and decompilers routinely parse untrusted, possibly malicious binaries, and parsers have bugs. Analyse malware in an isolated VM, never on your main host, and be aware that some samples specifically target analysis tools.

Examples

  Why instruction boundaries are the whole problem
  ─────────────────────────────────────────────────────────────

  The same five bytes, decoded from two different starting points:

  bytes:   EB 01 48 31 C0
           │
  start@0: EB 01          jmp  0x3        ; jump OVER the next byte
           ... then @3:   31 C0           xor  eax, eax
           (the 48 at offset 2 was a data byte, never executed)

  start@2: 48 31 C0       xor  rax, rax   ; a DIFFERENT instruction
           (decoding one byte off produces plausible but wrong code)

  A linear sweep that does not know byte 2 is skipped will desync here.
  This is exactly the trick anti-disassembly code uses deliberately.

# objdump: linear sweep. Fast, but trusts that every byte is code.
objdump -d -M intel --no-show-raw-insn ./prog | sed -n '/<main>:/,/ret/p'

# Disassemble a specific range only (useful when a region is data).
objdump -d -M intel --start-address=0x1149 --stop-address=0x1180 ./prog

# Show raw bytes alongside mnemonics, to verify boundaries by hand.
objdump -d -M intel ./prog | grep -A20 '<main>:'

# Ghidra, headless (scriptable) analysis of a binary into a project.
$GHIDRA_HOME/support/analyzeHeadless /tmp/proj MyProj \
    -import ./prog -postScript ListFunctionsScript.java -deleteProject

# radare2 / rizin: interactive recursive descent on the command line.
r2 -q -c 'aaa; afl' ./prog          # analyse all, then list functions
r2 -q -c 'aaa; s main; pdf' ./prog  # print disassembly of main as a graph

  The analyst's iterative loop in Ghidra (why interactivity matters)
  ─────────────────────────────────────────────────────────────────

   1. Auto-analysis finds functions   FUN_00101149, FUN_001011a0, ...
   2. Follow calls, read strings      "Invalid magic", "version %d"
   3. Rename as understanding grows    FUN_00101149 -> parse_header
   4. Apply a struct type to a ptr      undefined8 -> Header *
   5. Every XREF now reads clearly      all call sites show parse_header(hdr)
      -> repeat on the next function

   A flat `objdump` dump supports none of steps 3-5; the knowledge
   has nowhere to live.

Key Takeaways

• Disassembly's core difficulty is finding instruction boundaries, because x86-64 mixes code and data and uses variable-length instructions with no start markers.
• Linear sweep decodes every byte in order (fast, but misreads inline data); recursive descent follows control flow (accurate, but misses indirect-only code).
• objdump is linear sweep — excellent for quick, scriptable inspection but unreliable where data sits inside .text.
• Ghidra, IDA, Binary Ninja, and radare2 use recursive descent plus heuristics and, crucially, support an iterative workflow with cross-references, renaming, and types.
• The real value of an interactive tool is accumulated context: each rename or type application makes the rest of the program more readable.
• Always analyse untrusted binaries in an isolated VM; analysis tools parse hostile input and some malware targets them directly.

🧪 Practice

1. Disassemble the same function with objdump -d and with radare2's pdf, and identify any location where the two disagree; explain which is likely correct and why.
2. Construct a tiny assembly file that places a few data bytes in .text before a label, assemble it, and observe how objdump's linear sweep misinterprets those bytes.
3. Load a small C binary into Ghidra, rename three auto-named functions based on the strings they reference, and describe how the cross-reference view changed your understanding.
4. Interview-style: "When would you deliberately prefer objdump's linear sweep over a recursive-descent tool?" Hint: think about a situation where you need to see every byte decoded, including regions that control flow never reaches from the entry point.

Control Flow Graph Reconstruction

Theory

A control flow graph (CFG) is the single most useful abstraction in program analysis. It turns a linear list of instructions into a picture of how execution can move, and almost every higher analysis — decompilation, data-flow analysis, symbolic execution, finding unreachable code — is built on top of it.

The building block is the basic block: a maximal straight-line sequence of instructions with exactly one entry (the first instruction) and one exit (the last). Execution that enters a basic block is guaranteed to run all of it. A block ends at a branch, a call, a return, or just before any instruction that is the target of a jump. The CFG is then a directed graph whose nodes are basic blocks and whose edges are the possible transfers of control between them.

The reason basic blocks are the right unit is that within a block nothing interesting happens to control flow, so analyses can treat the whole block as one step. All the complexity lives at the edges, where execution can diverge or merge.

Reconstructing a CFG from a binary is a three-step process:

1. Find the leaders — instructions that start a basic block: the entry point, any jump/branch target, and any instruction immediately following a branch.
2. Form the blocks — each leader begins a block that runs until the next leader or a control-flow instruction.
3. Add the edges — from each block's terminating instruction to its possible successors: a conditional jump adds two edges (taken and fall-through); an unconditional jump adds one; a ret adds none (it returns to an unknown-at-this-level caller).

The hard part, again, is indirect transfers. A jmp rax or call [rax+8] has a target that depends on a runtime value the CFG does not know. Switch jump tables, virtual function calls, and function pointers all produce these. A disassembler recovers what it can (reading a jump table out of .rodata when it recognizes the pattern) and leaves the rest as gaps — which is precisely why indirect calls are both an analysis obstacle and a favourite of obfuscators.

Patterns in the CFG map directly to source constructs, and learning to read the shape is as valuable as reading the instructions:

• A block with two outgoing edges is an if or a loop condition.
• A back edge (an edge to a block that dominates it) is a loop.
• A block many edges point to is a common merge point — often a shared error handler or function epilogue.
• A diamond (split then merge) is an if/else.

The decompiler depends entirely on a correct CFG: it recovers high-level if/while/for by matching these structural patterns, a process called structuring. When a decompiler emits tangled goto statements, it usually means the CFG had a shape (often from hand-written assembly or obfuscation) that does not fit any clean high-level pattern.

Examples

  Source, basic blocks, and the resulting CFG
  ───────────────────────────────────────────────────────────────

  int f(int n) {              B0 (entry)
      int s = 0;                 s = 0; i = 0
      for (int i = 0;            cmp i, n ─────────┐
           i < n; i++) {     ┌──►B1 (loop header)  │
          if (i & 1)         │   cmp i, n          │
              s += i;        │   jge  B5 ───────────┼──┐
      }                      │                      │  │
      return s;              │   B2  test i,1       │  │
  }                          │      je  B4          │  │
                             │     / \              │  │
                             │   B3   \  (i & 1)    │  │
                             │  s+=i   \            │  │
                             │     \   /            │  │
                             │      B4  i++         │  │
                             └───────┘ jmp B1 ──────┘  │
                                        B5 (exit) ◄─────┘
                                        return s

  Edges: B0->B1, B1->B2, B1->B5, B2->B3, B2->B4, B3->B4, B4->B1 (BACK EDGE)
  The back edge B4->B1 is the loop. B2's two edges are the `if`.

# Generate a CFG visually from a binary.
r2 -q -c 'aaa; s sym.f; agf' ./prog         # ASCII CFG in the terminal
r2 -q -c 'aaa; s sym.f; agfd' ./prog > f.dot # export Graphviz DOT
dot -Tpng f.dot -o f.png                     # render it

# Ghidra shows the CFG in its "Function Graph" window; headless export:
objdump -d -M intel ./prog | awk '/<f>:/,/ret/'  # raw blocks to eyeball

# Let the COMPILER emit its own CFG for comparison with the recovered one.
gcc -O1 -fdump-tree-cfg-graph f.c -c -o /dev/null   # GCC's internal CFG
clang -S -emit-llvm f.c -o - | opt -dot-cfg         # LLVM CFG as DOT

# A toy CFG builder: find leaders, then edges, from a linear listing.
# Input is a list of (addr, mnemonic, target) tuples.
def leaders(insns):
    leads = {insns[0][0]}                     # entry is always a leader
    for i, (addr, mnem, target) in enumerate(insns):
        if mnem in ("jmp", "je", "jne", "jl", "jg"):
            if target is not None:
                leads.add(target)             # branch target is a leader
            if i + 1 < len(insns):
                leads.add(insns[i + 1][0])    # fall-through is a leader
    return sorted(leads)
# Edges then follow: each block's last insn points to target and/or
# fall-through. Indirect jumps (target is None) leave an unresolved edge.

Key Takeaways

• A control flow graph is the foundation for nearly all program analysis; its nodes are basic blocks and its edges are possible control transfers.
• A basic block is a maximal single-entry, single-exit instruction sequence, so analyses can treat it as one atomic step.
• CFG reconstruction is: find leaders, form blocks, add edges — with conditional branches adding two edges and unconditional jumps one.
• Indirect transfers (jmp rax, virtual calls, jump tables) are the hard part because their targets depend on runtime values.
• CFG shape reveals source constructs: a back edge is a loop, a diamond is if/else, and a heavily-targeted block is often a shared error path or epilogue.
• Decompilers recover high-level control structures by matching CFG patterns; goto-laden output signals a CFG that fits no clean pattern.

🧪 Practice

1. Draw the basic blocks and CFG edges for a function containing a while loop with an if inside it, labelling the back edge.
2. Use radare2's agf on a compiled function with a nested loop and verify that the number of back edges equals the number of loops.
3. Write out the leader-finding steps by hand for a short instruction listing containing one conditional and one unconditional jump.
4. Interview-style: "A decompiler produces a function full of goto statements where you expected clean loops. What are two plausible causes?" Hint: consider control-flow shapes that no structured if/while can express, and why an obfuscator or a hand assembler might deliberately create them.

Identifying Functions, Strings, and Constants

Theory

Before you can analyse what a binary does, you have to find the pieces: where functions begin and end, what text it contains, and which numeric constants it uses. In a binary with symbols this is trivial — the symbol table names everything. The interesting case, and the common one in malware and commercial software, is a stripped binary where those names are gone and you must recover the structure from the bytes themselves.

Finding functions without symbols relies on a stack of heuristics, in rough order of reliability:

1. Call targets. Every call 0x401160 tells you 0x401160 is a function entry. This is the strongest signal because it is factual, not heuristic.
2. Prologue patterns. push rbp; mov rbp, rsp and, in modern binaries, a leading endbr64 mark function starts. Reliable until the compiler omits frame pointers.
3. Unwind information. .eh_frame (present even in many stripped binaries because exceptions and stack unwinding need it) contains one entry per function with its address range. This is often the best source of function boundaries in an otherwise stripped binary, and tools like objdump --dwarf=frames read it directly.
4. Padding and alignment. Functions are often aligned to 16 bytes with nop or int3 (0xCC) padding between them, giving a visual boundary.

Finding strings is the highest-value-per-effort activity in all of reverse engineering. A single error message, URL, file path, registry key, or format string can tell you what a function does faster than reading its code. The strings utility extracts runs of printable characters, but using it well requires knowing its limits:

• By default it finds ASCII; malware frequently uses UTF-16LE (common on Windows), which needs strings -el.
• Strings may be obfuscated — XOR-encoded, stacked (built one character at a time on the stack to defeat strings), or encrypted and decoded at runtime. Absence of strings is itself a signal.
• The cross-references to a string matter more than the string: Ghidra's "show references to" on a string jumps you straight to the code that uses it.

Finding constants is a more specialized but powerful technique, because many algorithms have signature constants that identify them on sight:

Constant	Identifies
0x67452301, 0xEFCDAB89, ...	MD5 / SHA-1 initialization vectors
0x6A09E667, ...	SHA-256 initialization vector
0x9E3779B9	TEA / XXTEA delta (golden ratio)
0x5F3759DF	the fast inverse square root constant
0x63516358 ... S-box bytes	AES
0xDEADBEEF, 0xCAFEBABE	sentinels, magic numbers, debug markers
0xEDB88320	reversed CRC-32 polynomial

Recognizing a crypto constant lets you label an entire function as "this is AES key expansion" without reading a single arithmetic instruction, and tools like Ghidra's analysers, FindCrypt, and YARA rules automate exactly this.

Examples

# ── Strings: the first command to run on any unknown binary ──
strings -n 8 ./sample                 # ASCII runs of >=8 chars
strings -el ./sample                  # 16-bit little-endian (Windows wide)
strings -t x ./sample                 # with file offsets, to locate them
strings ./sample | grep -Ei 'http|/tmp|\.dll|key|passw|%s'  # triage filter

# ── Functions in a stripped binary ──
nm ./sample 2>&1                      # "no symbols" confirms it is stripped
objdump --dwarf=frames ./sample | grep -c 'FDE'   # function count from .eh_frame
r2 -q -c 'aaa; afl~?' ./sample        # radare2's recovered function count
objdump -d -M intel ./sample | grep -cE 'endbr64|push.*rbp'  # prologue tally

# ── Constants: identify algorithms by their magic numbers ──
objdump -s -j .rodata ./sample | grep -i '67452301\|6a09e667'  # MD5/SHA IVs
# In radare2, scan for known crypto material:
r2 -q -c 'aaa; /c aes' ./sample       # search for AES-like constants
# YARA with a crypto-constants ruleset does this across many files at once.

  How one string collapses an analysis
  ─────────────────────────────────────────────────────────────

  Stripped function FUN_00101340, 60 instructions, meaningless names.
  Its only string XREF:  "POST /gate.php HTTP/1.1\r\nHost: %s\r\n"

  Conclusion in 5 seconds, before reading any code:
    - This is C2 beacon construction.
    - %s is the callback host -> find where it is written to locate the
      hard-coded or config-sourced C2 address.
    - Rename FUN_00101340 -> build_beacon_request, and every caller
      becomes readable.

// Stack-string obfuscation: why `strings` sometimes finds nothing.
void connect_c2(void) {
    char host[8];
    host[0] = 'e'; host[1] = 'v'; host[2] = 'i'; host[3] = 'l';
    host[4] = '.'; host[5] = 'c'; host[6] = 'o'; host[7] = 0;
    // "evil.co" never appears as a contiguous run in the file, so
    // `strings` misses it. It is assembled at runtime, one byte per
    // `mov BYTE [rsp+k], imm`. Recognizing a cluster of such stores
    // is itself the tell. A debugger reveals the string once built.
}

Key Takeaways

• In a stripped binary you must recover functions, strings, and constants from bytes; symbols would otherwise name everything for free.
• Function discovery ranks call targets (factual) above prologue patterns above .eh_frame unwind data above alignment padding — and .eh_frame is often the best boundary source in stripped binaries.
• Extracting strings is the highest-value first step; a single URL, path, or format string can identify a function's purpose instantly.
• strings defaults to ASCII, so use -el for UTF-16LE, and treat the absence of strings (stack-built or encrypted) as a signal in itself.
• Cross-references to a string are more useful than the string alone, because they point directly at the code that uses it.
• Signature constants identify algorithms on sight — crypto IVs, the TEA delta, CRC polynomials — letting you label whole functions without reading them.

🧪 Practice

1. Run strings -n 8 and strings -el on a Windows PE sample (or any binary) and explain any strings that appear only in the wide-character pass.
2. Strip a C binary, then recover its function count three ways — .eh_frame FDE count, radare2's afl, and a prologue grep — and explain any discrepancies.
3. Compile a program that uses SHA-256, locate its initialization-vector constants in .rodata with objdump -s, and confirm they match the published values.
4. Interview-style: "A malware sample produces almost no output from strings, yet clearly performs network communication. What techniques might the author be using, and how would you recover the hidden strings?" Hint: think about how a string can exist in a running process but not as a contiguous run in the file, and which analysis mode — static or dynamic — recovers it most easily.

Decompilation and Its Limitations

Theory

A decompiler raises disassembly one more level, from assembly to C-like pseudocode. Where a disassembler answers "what instructions are here?", a decompiler answers "what did the programmer probably write?". Ghidra, IDA's Hex-Rays, and Binary Ninja all include one, and for most analysts the decompiler view is where the real work happens, because reading 200 lines of pseudo-C is far faster than 800 lines of assembly.

Understanding what a decompiler does explains exactly where it fails. It runs a pipeline: disassemble, build the CFG, lift instructions into an intermediate representation, perform data-flow analysis to track values through registers and memory, recover variables and types, structure the CFG into if/while/for, and finally print C. Each stage makes inferences, and inferences can be wrong.

The single most important thing to internalize is that decompiler output is a reconstruction, not the original source, and it is frequently incorrect in ways that matter for security. Specific, recurring limitations:

• Types are guessed. The binary has no struct definitions; the decompiler infers types from how values are used. You routinely see undefined8, int where it was size_t, or a pointer treated as an integer. A signed-vs-unsigned misinference can hide the very integer-overflow bug you are hunting.
• Variable names are invented. iVar3, local_28, uStack_40 carry no meaning; the semantic names are gone forever (unless debug info survived). Renaming them as you understand them is the core of the workflow.
• Optimizations are baked in. The decompiler shows the optimized program, so inlined functions vanish, loops appear unrolled or vectorized, and strength-reduced arithmetic (a multiply-and-shift) appears literally rather than as the division it implements.
• Structuring can fail. Irreducible control flow produces gotos and while(true) with breaks that the original never had.
• Compiler and runtime code leaks in. Stack-protector checks, PLT thunks, and C++ exception machinery appear as mysterious extra code that was never in the source.

The correct mental model is that the decompiler is a hypothesis generator, not an oracle. It gives you a readable starting point that you refine — applying types, renaming variables, fixing function signatures — and critically, you verify any security-relevant conclusion against the disassembly, because the pseudocode may have abstracted away the exact byte-level behaviour that constitutes the bug. A classic failure is a decompiler rendering a 32-bit truncation or a sign extension invisibly, hiding an exploitable size miscalculation that is plainly there in the assembly.

For C++ the challenges multiply: name mangling, this pointers passed implicitly, virtual calls through vtables, and templates that expand into many near-identical functions. Specialized analysis (vtable recovery, RTTI parsing) helps, but C++ decompilation remains materially harder than C.

Examples

  The same function at three levels of abstraction
  ─────────────────────────────────────────────────────────────

  ORIGINAL SOURCE (lost)          DECOMPILER OUTPUT (reconstructed)
  ───────────────────────         ──────────────────────────────────
  int sum(int *a, size_t n) {     int sum(int *param_1, ulong param_2) {
      int t = 0;                      int iVar1;        // was `t`
      for (size_t i = 0;              ulong uVar2;      // was `i`
           i < n; i++)                iVar1 = 0;
          t += a[i];                  for (uVar2 = 0; uVar2 < param_2;
      return t;                            uVar2 = uVar2 + 1)
  }                                       iVar1 += param_1[uVar2];
                                        return iVar1;
                                    }
  Names invented, size_t recovered as ulong, structure intact here --
  but see the dangerous case below where structure HIDES a bug.

// A decompiler can make an exploitable bug LOOK safe.
//
// What the decompiler prints (reconstructed, looks fine):
void copy_record(char *dst, char *src, int len) {
    if (len < 128)                 // looks like a proper bounds check
        memcpy(dst, src, len);
}
//
// What the DISASSEMBLY shows (the actual behaviour):
//   cmp   edi, 0x80           ; compare len as SIGNED (jl, not jb)
//   jge   .Lskip
//   movsxd rdx, edi           ; len SIGN-EXTENDED to 64-bit size_t
//   call  memcpy
//
// If len is a negative int (e.g. -1), it passes `len < 128`, then
// sign-extends to 0xFFFFFFFFFFFFFFFF as memcpy's size_t argument:
// a massive copy. The decompiler's clean `if (len < 128)` hid both the
// signedness of the compare and the sign-extension. Always confirm
// size/length logic against the assembly.

# Drive Ghidra's decompiler headless and dump pseudocode for one function.
$GHIDRA_HOME/support/analyzeHeadless /tmp/p P -import ./bin \
    -postScript DecompileFunctionScript.java target_func -deleteProject

# Cross-check: always keep the disassembly beside the pseudocode.
objdump -d -M intel ./bin | sed -n '/<copy_record>:/,/ret/p'

# In Ghidra's GUI the workflow is: decompile -> rename vars (L) ->
# retype (Ctrl+L) -> fix the function signature (F) -> re-decompile.

Key Takeaways

• A decompiler reconstructs C-like pseudocode from a binary, which is far faster to read than assembly and is where most analysis happens.
• Its output is an inference-laden reconstruction, not the original source, and can be wrong in security-relevant ways.
• Types are guessed, variable names are invented, and optimizations are baked in, so inlined functions and strength-reduced arithmetic look nothing like the source.
• Treat the decompiler as a hypothesis generator: refine it by applying types and names, and verify any security conclusion against the disassembly.
• Decompilers can make an exploitable bug look safe by hiding truncation, sign extension, or signedness of a comparison — the exact details that constitute the vulnerability.
• C++ is substantially harder owing to name mangling, implicit this, virtual dispatch, and template expansion.

🧪 Practice

1. Compile a function at -O2, decompile it in Ghidra, and list every way the pseudocode differs from your source (names, types, structure, inlined calls).
2. Write a function with a signed/unsigned length-check bug, then compare how the decompiler renders it against the disassembly; identify where the pseudocode obscures the flaw.
3. Take the undefined8/iVar output for a struct-manipulating function, apply a correct struct type in Ghidra, and describe how the pseudocode improves.
4. Interview-style: "Why should you never conclude 'this bounds check is sufficient' from decompiler output alone?" Hint: name two transformations — one about integer width, one about signedness — that the pseudocode can represent inaccurately, and state where the ground truth lives.

Symbol Stripping and Obfuscation Effects

Theory

Everything to this point assumed the analyst is working against an ordinary binary. In adversarial settings — malware, DRM, anti-cheat, licensing — the author actively fights analysis. Understanding their techniques matters for two reasons: defenders must analyse obfuscated malware, and developers must set realistic expectations about what obfuscation does and does not buy (it raises cost; it never provides a security guarantee, per Kerckhoffs).

The techniques form a ladder of increasing effort and increasing analyst pain:

Stripping is the baseline and is not really obfuscation — it is just removing information the program does not need to run. strip deletes .symtab and .strtab, erasing all local and function names. It costs nothing, breaks no functionality, and removes the single most useful analysis aid. Dynamic symbols (.dynsym) usually survive because the dynamic linker needs them, so imports remain visible. This is why malware is often statically linked and stripped: it removes even the import names.

Name mangling and dead stripping go further in release builds, but true obfuscation adds deliberate obstacles:

Technique	What it does	Effect on analysis	Countermeasure
Symbol stripping	removes names	functions become FUN_xxxx	recover via .eh_frame, strings, XREFs
String encryption	strings stored encrypted, decoded at runtime	strings finds nothing useful	dynamic analysis; find and run the decryptor
Control-flow flattening	replaces structured flow with a dispatcher loop + state variable	CFG becomes one giant switch; structure lost	symbolic execution; deobfuscation passes
Opaque predicates	always-true/false conditions that look data-dependent	inserts dead branches the analyst must disprove	constraint solving to prove the predicate constant
Instruction substitution	replaces ops with equivalent sequences (a+b -> a-(-b))	code volume balloons; patterns hidden	pattern-based simplification
Virtualization / packing	code is bytecode for a custom VM, or compressed/encrypted and unpacked at runtime	static analysis sees only the VM/unpacker	dump after unpacking; reverse the VM (very costly)
Anti-disassembly	junk bytes exploiting linear sweep desync	disassembler shows wrong instructions	recursive descent; manual boundary fixes

Control-flow flattening deserves emphasis because it is common and distinctive. It takes a function's basic blocks, numbers them, and rewrites control flow as a while(1) loop around a switch(state): each block sets state to its successor's number instead of jumping directly. The CFG collapses into a star — every block connects only to the dispatcher — destroying the structural patterns a decompiler relies on. Recognizing the dispatcher pattern is the first step; recovering the original edges generally needs symbolic execution to determine each block's next-state value.

Packing is the most common malware technique and the most important to recognize. A packed binary is a small unpacking stub plus a blob of compressed or encrypted original code; at runtime the stub reconstructs the real code in memory and jumps to it. Static analysis of the file sees only the stub. Tells include very few imports, a high-entropy section, a section that is writable and executable, and an entry point in an unusual section. The standard response is dynamic: let it unpack itself under a monitor, then dump the real code from memory (the original entry point, OEP, is where the stub hands off).

The strategic point: obfuscation is an economic control, not a cryptographic one. It increases the time and skill required, which can be decisive against casual attackers and commercially meaningful for DRM, but a determined analyst with enough time defeats any of it. Designs must never place a secret where only obfuscation protects it.

Examples

  Control-flow flattening: before and after
  ─────────────────────────────────────────────────────────────

  ORIGINAL CFG                      FLATTENED CFG
  ────────────                      ─────────────────────────────
      A                                   ┌───────────┐
      │                              ┌───►│ dispatcher│◄──┐
      B                              │    │ switch(st)│   │
     / \                             │    └─┬─┬─┬─┬─┬─┘   │
    C   D          becomes           │      │ │ │ │ │     │
     \ /                             │      A B C D E ────┘
      E                              └──────┘ (each block sets `st`
                                              to its successor's id)

  Structure (the diamond C/D under B) is gone. Every block links only
  to the dispatcher, so the decompiler cannot recover the if/else.

# Detect packing by section entropy and permissions.
readelf -SW ./sample | awk '{print $2, $8}'      # look for RWX or odd sections
# A quick entropy check (high, uniform entropy ~= compressed/encrypted):
python3 - <<'PY'
import math, sys
d = open("sample","rb").read()
from collections import Counter
c = Counter(d); n = len(d)
H = -sum((v/n)*math.log2(v/n) for v in c.values())
print(f"entropy: {H:.2f} bits/byte  ({'likely packed' if H>7.2 else 'normal'})")
PY

# Identify common packers by signature.
# (detect-it-easy / die, or:) 
strings ./sample | grep -i 'UPX\|upx!'           # UPX leaves visible markers
upx -d ./sample -o unpacked                       # UPX specifically is reversible

# Few imports is itself suspicious for a program that clearly does a lot.
objdump -T ./sample | grep -c UND                 # unusually low => packed/static

// String encryption: the decryptor is the thing to find.
// Static `strings` sees only ciphertext; the key and routine reveal all.
static char enc[] = {0x2a,0x27,0x2e,0x2e,0x2d,0x62,0x37,0x2d,0x28,0x2e,0x26};
void reveal(void) {
    char out[sizeof enc + 1];
    for (size_t i = 0; i < sizeof enc; i++)
        out[i] = enc[i] ^ 0x42;        // single-byte XOR; decodes to "hello world"
    out[sizeof enc] = 0;
    // Under a debugger, breakpoint AFTER this loop and read `out` to
    // recover the plaintext without reversing the algorithm by hand.
}

Key Takeaways

• Stripping merely removes names the program does not need at runtime; it is the cheapest, most common analysis obstacle, and .dynsym usually survives it.
• Obfuscation forms a ladder: string encryption, control-flow flattening, opaque predicates, instruction substitution, virtualization, packing, and anti-disassembly, each costing more to defeat.
• Control-flow flattening collapses the CFG into a dispatcher-plus-state-variable loop, destroying the structure decompilers rely on; recovery typically needs symbolic execution.
• Packing is the signature malware technique: a stub reconstructs the real code in memory at runtime, so static analysis of the file sees only the stub.
• Recognize packing by high-entropy sections, writable-and-executable sections, very few imports, and an unusual entry point; respond by unpacking dynamically and dumping from memory.
• Obfuscation is an economic control that raises attacker cost; it never provides a cryptographic guarantee, so no secret should rely on it alone.

🧪 Practice

1. Compile a program, measure the entropy of its .text section, then pack it with UPX and measure again; explain the change and what it signals to an analyst.
2. Strip a binary and document exactly which analysis capabilities you lost (nm, local names) and which you retained (nm -D, .eh_frame boundaries).
3. Write a function obfuscated with a single-byte XOR string decoder, confirm strings cannot read the plaintext, then recover it by setting a debugger breakpoint after the decode loop.
4. Interview-style: "A vendor says their licensing check is 'secure because the binary is heavily obfuscated.' Critique that claim." Hint: relate it to Kerckhoffs's principle and the distinction between raising attacker cost and providing a measurable security guarantee.

Taint Analysis Concepts

Theory

Most serious vulnerabilities share one shape: data that an attacker controls reaches a place where it can do damage, without being adequately checked in between. SQL injection is attacker input reaching a query; command injection is attacker input reaching a shell; a buffer overflow is an attacker-controlled length reaching a copy. Taint analysis is the technique that makes this shape explicit and checkable: it tracks which values in a program are influenced by untrusted input and warns when such a value reaches a dangerous operation.

The vocabulary is three words:

• A source is where untrusted data enters: a network read, argv, a file, an environment variable, an HTTP parameter. Values from a source are marked tainted.
• A sink is a sensitive operation where tainted data is dangerous: system(), exec(), a SQL execute(), memcpy's size argument, a format string.
• Propagation is the rule set for how taint spreads: if b = a + 1 and a is tainted, then b is tainted; if s2 = strcpy(s2, s1) and s1 is tainted, s2 becomes tainted.

The core idea is the taint lattice: every value carries a bit (or a richer label) saying "influenced by untrusted input or not". Operations combine the labels of their inputs. The analysis is a reachability question — can taint flow from any source to any sink? — and a flagged flow is a candidate vulnerability.

Two crucial refinements separate a naive analysis from a useful one:

Sanitization removes taint. A value that passes through a correct validator — an integer parse that bounds the range, an escaping function, a prepared-statement parameter binding — should be untainted afterward, because it is no longer dangerous. The hard part is that the analysis must know which functions genuinely sanitize for which sinks; escaping for HTML does not sanitize for SQL. Mismodelled sanitizers are the main source of both false negatives (trusting a bad sanitizer) and false positives (not recognizing a good one).

Explicit vs implicit flows. Explicit (data) flow is direct assignment: y = x. Implicit (control) flow leaks through decisions: if (secret == 0) y = 1; else y = 2; copies information about secret into y without ever assigning secret to it. Tracking implicit flows is necessary for information-leak analysis but causes taint explosion — follow every branch and almost everything becomes tainted — so most practical tools track explicit flows and selectively model implicit ones.

Taint analysis comes in two deployment modes with a familiar trade-off:

	Static taint	Dynamic taint
When	at analysis time, over all paths	at runtime, on the executed path
Coverage	all paths (sound-ish)	only paths actually run
False positives	many (over-approximates)	few (observes real flows)
False negatives	fewer	misses unexecuted paths
Cost	solver/analysis time	heavy runtime slowdown (often 2-30x)
Tools	CodeQL, Semgrep, Checkmarx	Triton, libdft, Valgrind-based

Taint analysis is the engine behind modern SAST (static application security testing) tools and behind DIFT (dynamic information-flow tracking) used in exploit analysis and fuzzing. Its practical ceiling is precision: too coarse and it drowns you in false positives; too narrow and it misses real flows. Tuning sources, sinks, and sanitizers for a specific codebase is where the real engineering effort goes.

Examples

  A taint flow, from source to sink
  ─────────────────────────────────────────────────────────────

  SOURCE                PROPAGATION                      SINK
  ──────                ───────────                      ────
  user = recv(sock) ──► cmd = "ping " + user ──────────► system(cmd)
   [tainted]             [tainted: concat spreads it]     [DANGER]

  With a sanitizer in the path:
  user = recv(sock) ──► if (is_valid_host(user)) ──────► system(cmd)
   [tainted]             clean = user  [UNTAINTED]        [safe, IF the
                                                           validator is
                                                           truly sufficient]

# A minimal dynamic taint tracker, to show the mechanics.
class Tainted(str):
    """A string value carrying a taint label."""
    pass

def source_recv():                       # SOURCE: untrusted input
    return Tainted("8.8.8.8; rm -rf /")   # attacker-controlled

def concat(a, b):                        # PROPAGATION rule
    result = str(a) + str(b)
    if isinstance(a, Tainted) or isinstance(b, Tainted):
        return Tainted(result)           # taint spreads through concatenation
    return result

def sanitize_host(s):                    # SANITIZER: validate, then untaint
    import re
    if re.fullmatch(r"[0-9.]{1,15}", s): # strict allowlist
        return str(s)                    # returns a plain str: UNTAINTED
    raise ValueError("invalid host")

def sink_system(cmd):                    # SINK
    if isinstance(cmd, Tainted):
        print(f"[VULN] tainted data reaches system(): {cmd!r}")
    else:
        print(f"[safe] running: {cmd!r}")

user = source_recv()
sink_system(concat("ping ", user))       # [VULN] -- flagged
clean = sanitize_host(source_recv().split(";")[0].strip())  # raises: blocked

-- Why the SINK, not the SOURCE, is where sanitization must be correct.
-- Tainted `name` reaching string concatenation = classic injection:
query = "SELECT * FROM users WHERE name = '" + name + "'";   -- SINK: unsafe

-- Parameter binding is a sanitizer the analysis should recognize: the
-- tainted value can no longer alter query STRUCTURE, only supply data.
query = "SELECT * FROM users WHERE name = ?";                -- safe
db.execute(query, [name]);   -- `name` stays data; taint here is harmless

# Real static taint analysis with CodeQL (conceptual invocation).
codeql database create db --language=cpp --command='make'
codeql database analyze db \
    --format=sarifv2 --output=results.sarif \
    cpp-security-queries.qls   # includes taint-tracking queries for
                               # tainted-size memcpy, command injection, etc.

# Semgrep expresses source->sink taint rules compactly:
semgrep --config 'p/command-injection' ./src

Key Takeaways

• Taint analysis makes explicit the shape shared by most vulnerabilities: attacker-controlled data reaching a dangerous operation unchecked.
• The model is sources (where untrusted data enters), sinks (dangerous operations), and propagation rules (how taint spreads through computation).
• Sanitization removes taint, but only if the analysis correctly models which functions sanitize for which sink — escaping for HTML does not sanitize for SQL.
• Explicit (data) flows are tracked reliably; implicit (control-flow) flows leak through decisions and cause taint explosion, so they are modelled selectively.
• Static taint covers all paths but over-approximates (false positives); dynamic taint observes real executed flows but misses unrun paths and is slow.
• Taint tracking is the engine behind SAST tools and dynamic information-flow tracking; its usefulness is governed by the precision of its sources, sinks, and sanitizers.

🧪 Practice

1. For the C statement sequence n = atoi(argv[1]); buf = malloc(n); memcpy(buf, src, n);, identify the source, the sink, and whether any sanitization occurs.
2. Extend the Python taint tracker above to propagate taint through a substring operation and through list indexing, and test it.
3. Explain, with a two-line example, why escaping a value for HTML output does not sanitize it for use in a shell command.
4. Interview-style: "A SAST tool reports a tainted-data flow into a SQL query, but the developer insists it is safe because they 'validate the input.' How do you adjudicate this?" Hint: ask what the validation actually guarantees about the value relative to the specific sink, and whether it constrains structure or only format.

Symbolic Execution Fundamentals

Theory

Testing runs a program on one concrete input and sees what happens. To be confident a program is correct or to find the one input that triggers a bug, you would have to try them all — impossible for any real input space. Symbolic execution is the idea that lets you reason about many inputs at once: instead of running the program on a concrete value like x = 5, run it on a symbol x = α representing "any value", and track the arithmetic and constraints symbolically.

The mechanism rests on two pieces of state the engine maintains as it executes:

• The symbolic store maps each variable to an expression over the input symbols. After y = x + 3, the store holds y = α + 3 rather than a number.
• The path condition is the conjunction of the branch conditions taken to reach the current point. When execution hits if (y > 10), the engine forks: one path continues with path condition α + 3 > 10, the other with α + 3 ≤ 10.

At any point, the path condition is a logical formula describing exactly the set of inputs that reach here. To get a concrete input that drives execution down a particular path — say, to a vulnerable memcpy — you hand the path condition to an SMT solver (Z3, CVC5), which either produces a satisfying assignment (a real input that triggers the path) or proves none exists. This is the superpower: symbolic execution plus a solver generates the exact input that reaches a target, rather than waiting for a fuzzer to stumble onto it.

A worked micro-example makes it concrete. For code that calls a bug on input (x+3)*2 == 24, the engine accumulates the constraint (α + 3) * 2 == 24, asks the solver, and gets α = 9 — the precise triggering input, derived, not guessed.

The fundamental limitation is path explosion. Every branch potentially doubles the number of paths, so a loop bounded by input, or a sequence of conditionals, produces exponentially many paths and the analysis cannot finish. This is the central engineering problem, and the mitigations define the field:

Problem	Mitigation
Exponential paths	concolic execution: run concretely, symbolic only along that path, and flip one branch at a time (DART/SAGE, driejcler.)
Unbounded loops	bound loop iterations; merge paths at join points
Calls into libraries/syscalls	function summaries or concretize the environment
Solver cost	caching, constraint simplification, independent-constraint slicing
Symbolic memory (pointer depends on input)	concretize addresses, or use theory-of-arrays (expensive)

Concolic execution (concrete + symbolic) is the dominant practical compromise: the program runs on a real concrete input, the engine records symbolic constraints only along that one executed path, then negates one branch condition and solves to produce a new concrete input that takes the other direction. Repeating this systematically explores new paths one at a time while keeping each solver query tractable. This is how modern tools such as angr, KLEE (source-level, on LLVM IR), S2E, and Microsoft's SAGE achieve deep coverage, and it is increasingly paired with fuzzing in hybrid fuzzing, where the solver's job is to get past the narrow checks (magic-value comparisons, checksums) that random fuzzing cannot guess.

The honest framing: symbolic execution is extraordinarily powerful for reaching deep, hard-to-trigger states and for proving a path is or is not feasible, but it does not scale to whole large programs unaided. In practice it is aimed at a specific function or target, bounded carefully, and combined with other techniques.

Examples

  Forking on a branch; the path condition selects inputs
  ─────────────────────────────────────────────────────────────

  int f(int x) {              store: x = α
      int y = x + 3;          store: y = α + 3
      if (y * 2 == 24) {              ╱            ╲
          bug();        path cond:   (α+3)*2 == 24   (α+3)*2 != 24
      }                        │                      │
      return 0;          solver: α = 9           solver: α = anything else
  }                      -> INPUT THAT HITS bug()  -> input that avoids it

  The engine did not guess 9. It derived it from the constraint.

# Symbolic reasoning directly with the Z3 solver.
from z3 import Int, Solver, sat

x = Int('x')                      # the symbolic input: "any integer"
y = x + 3                         # symbolic store update

s = Solver()
s.add((y * 2) == 24)              # the path condition to reach bug()
if s.check() == sat:
    m = s.model()
    print("input that reaches bug():", m[x])   # x = 9
else:
    print("bug() is unreachable")

# Add a second, conflicting constraint to see UNSAT (infeasible path):
s.add(x > 100)                    # now require x = 9 AND x > 100
print(s.check())                  # unsat -> that combined path cannot occur

# angr: let a symbolic engine find the input that reaches a target address.
import angr, claripy

proj = angr.Project("./crackme", auto_load_libs=False)

# A 16-byte symbolic input ("any password").
flag = claripy.BVS("flag", 16 * 8)
state = proj.factory.entry_state(stdin=flag)

simgr = proj.factory.simulation_manager(state)
simgr.explore(
    find=0x401337,        # address printing "Correct!" -- reach this
    avoid=0x401300,       # address printing "Wrong"    -- prune these paths
)

if simgr.found:
    sol = simgr.found[0]
    # Ask the solver for concrete stdin that drives execution to `find`.
    print("password:", sol.posix.dumps(0))
# angr explores symbolically, forking at branches, and the solver turns
# the surviving path condition into the exact bytes that unlock the target.

Key Takeaways

• Symbolic execution runs a program on symbols standing for "any input", tracking a symbolic store of expressions and a path condition of branch constraints.
• The path condition at any point is a formula describing exactly the inputs that reach that point; an SMT solver turns it into a concrete triggering input or proves none exists.
• Its superpower is generating the precise input that reaches a target (a bug, a protected branch) by construction, rather than by random search.
• Path explosion — branches multiplying paths exponentially — is the fundamental limit and the reason it does not scale to whole large programs unaided.
• Concolic (concrete + symbolic) execution is the dominant practical compromise: run on a real input, collect constraints along that path, then flip one branch and re-solve.
• Tools such as angr, KLEE, S2E, and SAGE combine these ideas, and hybrid fuzzing uses the solver specifically to pass the narrow checks that random fuzzing cannot guess.

🧪 Practice

1. Using Z3, find an integer x such that (x ^ 0x5A) + 7 == 100, then add a constraint making the system unsatisfiable and confirm the solver reports unsat.
2. Hand-simulate symbolic execution of a function with two nested if statements, writing the symbolic store and path condition for each of the (up to) four leaf paths.
3. Explain, with a concrete loop example, why for (i = 0; i < n; i++) with symbolic n causes path explosion, and describe two ways an engine bounds it.
4. Interview-style: "Why are symbolic execution and fuzzing often combined rather than used alone?" Hint: contrast what random mutation is good at (fast, broad, shallow coverage) with what a solver is good at (passing a single narrow if (input == 0xDEADBEEF) check that fuzzing would almost never hit).

<a id="43-dynamic-program-analysis"></a>

4.3 Dynamic Program Analysis

Dynamic analysis observes a program as it actually runs, trading the completeness of static reasoning for the ground truth of real execution; this section covers debuggers, tracers, memory-error detectors, fuzzing, and crash triage — the toolkit for watching software behave and for provoking it into misbehaving.

Debugging with GDB and Extensions

Theory

A debugger is the microscope of dynamic analysis: it lets you stop a running program at any point and inspect or modify its complete state — registers, memory, stack, source variables. Where static analysis infers what might happen, the debugger shows what is happening, which resolves every question that static reasoning left ambiguous (what does this indirect call actually target? what value did this input produce?).

The foundation underneath GDB is the kernel's ptrace facility, the same primitive covered later in this section. ptrace lets one process (the debugger) control another (the debuggee): stop it, read and write its memory and registers, and receive notification of its signals and syscalls. Every breakpoint, every print, every single-step is ultimately a ptrace operation. Understanding this explains the debugger's powers and its limits — notably that a program can detect it is being traced by trying to ptrace itself, the basis of a common anti-debugging check.

GDB's model is a small set of orthogonal capabilities:

• Stopping: breakpoints (stop at an address), watchpoints (stop when memory changes), catchpoints (stop on a syscall or signal).
• Inspecting: info registers, x (examine memory), print, backtrace.
• Modifying: write registers and memory, call functions, jump to arbitrary addresses — the debugger has total control.
• Progressing: step one instruction (si), step one source line (step), continue (continue), finish the current frame (finish).

Raw GDB is powerful but spartan, which is why extensions are essential for security work. GDB ships an embedded Python interpreter, and the community has built rich front ends on it — GEF, pwndbg, and PEDA are the three common ones. They add, for exploitation and reversing specifically: a live register/stack/code dashboard on every stop, heap-chunk inspection that understands glibc's allocator, pattern generation for finding offsets, and commands that visualize the memory layout. The difference between bare GDB and GDB with pwndbg for exploit development is the difference between a flashlight and a floodlight.

A few workflow facts matter disproportionately:

• TUI mode (gdb -tui or Ctrl-x a) shows source/assembly and registers in split panes.
• -tui with layout asm is how you watch assembly execute line by line.
• Loading symbols and source (-g builds) turns the debugger from an assembly-level tool into a source-level one; without them you work in registers and addresses.
• You can attach to an already-running process with gdb -p PID, essential for analysing daemons and for situations where you cannot control startup.

For malware analysis, the same caution as always applies: debug untrusted code only in an isolated VM, and be aware that sophisticated samples alter their behaviour when they detect a debugger.

Examples

# Build with symbols for source-level debugging.
gcc -g -O0 vuln.c -o vuln

# A complete non-interactive debugging session via -ex (scriptable).
gdb -q ./vuln \
    -ex 'break main'            \
    -ex 'run'                   \
    -ex 'next'                  \
    -ex 'print argc'            \
    -ex 'info registers rdi rsi' \
    -ex 'x/4gx $rsp'            \
    -ex 'backtrace'             \
    -ex 'continue' -ex 'quit'

# Attach to a running process (e.g. a daemon) by PID.
sudo gdb -q -p "$(pgrep -n vuln)"

# TUI: watch assembly and registers update as you step.
gdb -q ./vuln -ex 'layout asm' -ex 'layout regs' -ex 'break main' -ex run

  Typical pwndbg / GEF dashboard shown automatically on every stop
  ─────────────────────────────────────────────────────────────────
    REGISTERS
     RAX 0x0   RBX 0x0   RDI 0x7fffffffe3a0 -> "AAAA..."
     RIP 0x000055...1189 <vuln+31>
   DISASM
    ► 0x55...1189 <vuln+31>   call   strcpy@plt
      0x55...118e <vuln+36>   nop
   STACK
     00:0000  rsp  0x7fffffffe2a0 -> 0x4141414141414141 ('AAAAAAAA')
     01:0008       0x7fffffffe2a8 -> 0x4141414141414141
   BACKTRACE
     f 0  0x55...1189 vuln
     f 1  0x55...11c3 main
  ── these panes appear WITHOUT being asked, which is the whole point ──

# GDB's Python API: scripting the debugger for repeatable analysis.
# Save as trace_calls.py, load with: gdb -x trace_calls.py ./vuln
import gdb

class LogStrcpy(gdb.Breakpoint):
    def stop(self):                       # called each time the bp is hit
        # System V: 1st arg in rdi (dst), 2nd in rsi (src).
        src = gdb.parse_and_eval("$rsi").string()
        print(f"[strcpy] copying {len(src)} bytes: {src!r}")
        return False                      # False => log and keep going
LogStrcpy("strcpy")
gdb.execute("run")

Key Takeaways

• A debugger gives total read/write access to a running program's state, resolving questions static analysis cannot — such as the real target of an indirect call.
• GDB is built on the kernel's ptrace facility; every breakpoint and inspection is ultimately a ptrace operation, which is also why a program can detect tracing by tracing itself.
• Its capabilities are orthogonal: stopping (breakpoints/watchpoints/catchpoints), inspecting, modifying, and progressing (step/continue/finish).
• Extensions (GEF, pwndbg, PEDA) add exploitation-focused views — live register/stack/code dashboards, glibc heap inspection, offset pattern generation — and are effectively mandatory for security work.
• -g builds enable source-level debugging; without symbols you work at the register and address level, and gdb -p PID attaches to running processes.
• Debug untrusted binaries only in an isolated VM, since capable malware changes behaviour when it detects a debugger.

🧪 Practice

1. Compile a small program with -g, set a breakpoint on a function, run to it, and print all six integer-argument registers; confirm they match the source arguments.
2. Use a watchpoint to stop the moment a specific global variable changes, and identify the line responsible.
3. Install pwndbg or GEF, trigger a stack buffer overflow in a toy program, and use the dashboard to observe the saved return address being overwritten.
4. Interview-style: "A program behaves normally when run directly but exits immediately under GDB. What is likely happening and how would you investigate?" Hint: consider what ptrace returns when a process is already being traced, and how a program could use that result as a check.

Breakpoints, Watchpoints, and Stepping

Theory

Stopping a program at the right moment is the central skill of debugging, and the three mechanisms for doing so — breakpoints, watchpoints, and stepping — work in fundamentally different ways. Knowing how each is implemented explains when each is fast, when it is slow, and why you sometimes get a limited number of them.

Software breakpoints are the workhorse. To set one at an address, the debugger saves the byte there and replaces it with a single-byte trap instruction, int3 (opcode 0xCC). When execution reaches it, the CPU raises a breakpoint exception, the kernel delivers SIGTRAP, and the debugger regains control. To continue, the debugger restores the original byte, single-steps over it, re-inserts 0xCC, and resumes. This is why you can set essentially unlimited software breakpoints (they cost only a saved byte each) and why they only work on writable code — you cannot patch a byte into ROM or a page you lack permission to modify. It also means a program can detect a software breakpoint by checksumming its own code and noticing the 0xCC.

Hardware breakpoints use dedicated CPU debug registers (DR0–DR3, with DR7 controlling them). The CPU itself compares the program counter against the stored address and traps on a match, modifying no memory at all. Because they are register-based, they are invisible to code checksums and work on read-only or even non-present memory — but there are only four of them, a hard limit set by the hardware.

Watchpoints answer a different question: not "when does execution reach here?" but "when does this data change?". This is invaluable when a variable is being corrupted and you do not know by whom. Watchpoints come in two flavours with wildly different costs:

Type	Mechanism	Speed	Limit
Hardware watchpoint	debug registers trap on access to an address range	near-native	4 registers, limited size (1/2/4/8 bytes each)
Software watchpoint	debugger single-steps the entire program, re-checking the value after every instruction	100-1000x slowdown	unlimited, any size

GDB uses hardware watchpoints automatically when the size fits; it silently falls back to the agonizingly slow software form otherwise, which is why watching a large region can appear to hang.

Stepping granularities matter for efficiency:

• stepi / si — one machine instruction; the finest grain, for assembly-level work.
• nexti / ni — one instruction, but step over calls (run the whole called function at speed).
• step / s — one source line, descending into called functions.
• next / n — one source line, stepping over calls.
• finish — run until the current function returns, then stop; the fastest way out of a function you entered by mistake.

The practical discipline is to step over (n/ni) by default and step into (s/si) only the specific call you care about, because stepping into every library call wastes enormous time. Conditional breakpoints (break foo if count == 500) and breakpoint commands (run a script each time a breakpoint hits) are what make debugging a loop's 500th iteration tractable rather than hopeless.

Examples

  Software breakpoint mechanics (the 0xCC swap)
  ─────────────────────────────────────────────────────────────

  Original code at 0x1189:    48 89 e5   mov rbp, rsp
                              ▲
  Debugger sets a breakpoint: save 0x48, write 0xCC:
                              CC 89 e5   (int3 ; ...)
                              ▲
  CPU hits 0xCC -> #BP exception -> SIGTRAP -> debugger stops.
  To continue: restore 0x48, single-step once, re-write 0xCC, resume.

  => unlimited software breakpoints, but requires WRITABLE code, and a
     self-checksum can SEE the 0xCC. Hardware breakpoints avoid both at
     the cost of only four being available.

# Breakpoints: plain, conditional, and with an auto-run command.
gdb -q ./loop \
  -ex 'break process'                 \
  -ex 'break process if i == 500'     \
  -ex 'commands 2'  -ex 'print i'     \
  -ex 'print buf'   -ex 'continue'    \
  -ex 'end'                           \
  -ex 'run'

# Hardware vs software breakpoints explicitly.
gdb -q ./prog -ex 'hbreak main' -ex run    # hbreak = hardware breakpoint

# A watchpoint to catch who corrupts a variable.
gdb -q ./prog \
  -ex 'break main' -ex run \
  -ex 'watch global_state'      \  # stop when global_state's value changes
  -ex 'continue'                \  # GDB reports the old and new values and
  -ex 'backtrace'                  # the stack of whoever wrote it

# Read/write/access watchpoints:
#   watch  X   -> on write
#   rwatch X   -> on read
#   awatch X   -> on either

  Choosing a stepping command
  ─────────────────────────────────────────────────────────────
   You are stopped at:   call   parse_header

   si  -> enters parse_header, stops on its first instruction
   ni  -> runs all of parse_header at speed, stops after the call returns
   s   -> enters parse_header (source-line granularity)
   n   -> runs all of parse_header, stops at the next source line
   finish (after si/s) -> runs to parse_header's return, prints ret value

   Rule of thumb: n/ni by default; s/si only INTO the one call you suspect.

Key Takeaways

• Software breakpoints swap in an int3 (0xCC) byte, so they are effectively unlimited but need writable code and are visible to self-checksumming.
• Hardware breakpoints use CPU debug registers, leave memory untouched (invisible to checksums), and work on read-only memory, but only four exist.
• Watchpoints stop on data change; hardware watchpoints are near-native speed but tiny and few, while software watchpoints single-step the whole program and are hundreds of times slower.
• GDB silently falls back from hardware to software watchpoints when the watched region is too large, which can look like a hang.
• Stepping granularity ranges from one instruction (si/ni) to one source line (s/n); finish runs out of the current function.
• Step over by default and step into only the suspected call; conditional breakpoints and breakpoint commands make loop-iteration debugging feasible.

🧪 Practice

1. Set a software breakpoint, then examine the target byte in memory before and after setting it to observe the 0xCC substitution.
2. Write a program that corrupts a global variable deep in a call chain, then use a watchpoint to identify the exact instruction and backtrace responsible.
3. Set a conditional breakpoint that fires only on the 1000th iteration of a loop and confirm it does not stop on earlier iterations.
4. Interview-style: "Why might a hardware watchpoint be essential rather than merely convenient when debugging a real-time or timing-sensitive program?" Hint: compare the runtime slowdown of the software watchpoint mechanism against the hardware one, and consider what a 500x slowdown does to code that depends on timing.

Inspecting Registers, Stack, and Heap at Runtime

Theory

Once a program is stopped, the value of the debugger is entirely in how well you can read its state. Three regions carry almost all the information you need: the registers (the CPU's immediate working set), the stack (local variables, saved return addresses, call history), and the heap (dynamically allocated objects). Each has its own inspection idioms, and fluency with them is what separates productive debugging from aimless poking.

Registers are read with info registers (all) or print $rax (one). The things to look at first, driven by the ABI from 4.1: rdi/rsi/rdx/rcx/r8/r9 at a call tell you the arguments; rax after a call is the return value; rip is where you are; rsp and rbp frame the current locals. Reading rflags tells you how the last comparison turned out. The skill is not memorizing commands but knowing which register answers your current question.

Examining memory uses GDB's x command, whose syntax is a compact language worth learning once: x/NFU address, where N is a count, F a format, and U a unit size. x/16gx $rsp means "16 giant (8-byte) words in hex starting at the stack pointer" — the single most-used command in exploit work. The formats (x hex, d decimal, s string, i instruction) and units (b byte, h half, w word, g giant) compose freely.

The stack is where you reconstruct what is happening now and how you got here. backtrace (bt) walks the saved frame pointers to show the call chain; frame N / up / down move between frames so you can inspect each caller's locals; info locals and info args print a frame's variables when symbols are present. When symbols are absent, you read the stack manually with x/x $rsp and interpret the values: addresses pointing into .text are likely return addresses, addresses pointing into the stack are likely saved frame pointers, and ASCII-looking values are likely string data. This manual reading is exactly what you do when analysing a crash in a stripped binary.

The heap is the hardest region to inspect, because it has structure the debugger does not natively understand: glibc's malloc interleaves your data with allocator metadata (chunk sizes, free-list pointers) that governs how memory is managed and that attackers manipulate in heap-exploitation. Bare GDB shows you raw bytes; this is precisely where pwndbg and GEF earn their place, adding commands like heap (list all chunks), bins (show the free lists), and vis_heap_chunks (a visual map of the heap with metadata annotated). Understanding the heap layout — chunk headers, the size field with its flag bits, the free-list pointers in freed chunks — is a prerequisite for understanding use-after-free and heap-overflow bugs covered in the exploitation chapter.

A practical cross-cutting technique: info proc mappings (or reading /proc/PID/maps) shows the complete memory map — where each library, the stack, and the heap are loaded. With ASLR this changes every run, and this map is how you find the actual runtime addresses you need. It is also how you confirm which regions are executable, which matters for both exploitation and defence.

Examples

# Registers: the ABI tells you what each one means at a given point.
gdb -q ./prog -ex 'break handle_request' -ex run \
  -ex 'info registers rdi rsi rdx' \   # the first three arguments
  -ex 'print (char*)$rdi'          \   # arg1 as a C string
  -ex 'finish'                     \   # run to return...
  -ex 'print $rax'                     # ...and read the return value

# The `x` command: the format/unit mini-language.
#   x/16gx $rsp    16 giant words, hex      (stack dump, the classic)
#   x/8i  $rip     8 instructions ahead     (disassemble around PC)
#   x/s   $rdi     a C string
#   x/4wd &arr     4 words, signed decimal  (an int[4])
gdb -q ./prog -ex 'break main' -ex run -ex 'x/16gx $rsp' -ex 'x/5i $rip'

  Reading a raw stack dump (stripped binary, no symbols)
  ─────────────────────────────────────────────────────────────
   (gdb) x/8gx $rsp
   0x7fffffffe2a0:  0x0000000000000000  0x00007fffffffe3b8
                    ^ a local (zero)     ^ points INTO the stack
                                           => a saved rbp
   0x7fffffffe2b0:  0x4141414141414141  0x00005555555551c3
                    ^ "AAAAAAAA"          ^ points into .text
                      => overflow data      => a RETURN ADDRESS
   0x7fffffffe2c0:  0x00007fffffffe3a0  0x0000555555555060
                    ^ stack ptr (arg)     ^ .text (code ptr)

   Classification by where a value POINTS is how you read an unsymbolized
   stack: .text => code/return addr, stack => saved rbp/pointers,
   ASCII => data.

# The heap: bare GDB sees bytes; pwndbg/GEF see structure.
# (with pwndbg loaded)
gdb -q ./heapprog -ex 'break main' -ex run \
  -ex 'heap'              \   # list every chunk with size and status
  -ex 'bins'              \   # the free lists (tcache, fastbins, ...)
  -ex 'vis_heap_chunks'       # annotated visual map

# Raw view for comparison: a glibc chunk header precedes user data.
#   [ prev_size(8) ][ size(8) | flags ][ user data ... ]
#   The low 3 bits of `size` are flags (PREV_INUSE etc.), which is why
#   a 0x20-byte chunk shows a size field of 0x21.

# The full runtime memory map -- where everything actually landed.
gdb -q ./prog -ex 'break main' -ex run -ex 'info proc mappings'
cat /proc/$(pgrep -n prog)/maps        # same information outside GDB

Key Takeaways

• Three regions hold almost all runtime information: registers (CPU working set), stack (locals, saved return addresses, call history), and heap (dynamic objects).
• Which register answers your question follows from the ABI: argument registers at a call, rax after it, rip/rsp/rbp for location and frame.
• The x/NFU command is a compact format language; x/16gx $rsp (sixteen 8-byte hex words from the stack) is the most-used inspection in exploit work.
• backtrace and frame navigation reconstruct the call chain; without symbols you classify stack values by where they point (.text = return address, stack = saved rbp, ASCII = data).
• The heap carries allocator metadata the debugger does not natively understand, which is why pwndbg/GEF commands like heap, bins, and vis_heap_chunks are essential for heap-bug analysis.
• info proc mappings / /proc/PID/maps reveals the actual ASLR'd runtime layout and which regions are executable.

🧪 Practice

1. Stop a program at a function entry and, using only info registers and x, print its first three arguments and the string one of them points to.
2. Trigger a stack overflow in a toy program and, from a raw x/16gx $rsp dump with no symbols, identify the saved frame pointer and the return address by where they point.
3. Allocate several heap buffers, free one, and use pwndbg's heap and bins to observe the freed chunk moving into a free list; note the metadata in its header.
4. Interview-style: "Looking at a raw stack dump of a stripped binary, how do you distinguish a saved return address from a saved frame pointer from attacker data?" Hint: think about which memory region each kind of value points into, and what range the stack, the code, and injected ASCII occupy.

Syscall Tracing with strace and ltrace

Theory

Sometimes you do not need to see how a program works internally, only what it does to the outside world: which files it opens, which network connections it makes, which other programs it launches. For that, tracing the boundary is far more efficient than stepping through code. strace traces system calls (the kernel boundary from 4.1) and ltrace traces library calls (the libc boundary). Together they give a high-level behavioural transcript without any source or symbols.

The power of this approach rests on a fact established earlier: syscalls are the complete inventory of a program's external effects. A program cannot read a file, touch the network, or spawn a process without a syscall, so an strace log is a guaranteed-complete record of everything the program did outside its own memory. For malware triage, misconfiguration debugging ("why can't it find its config file?"), and understanding an unknown binary, this completeness is exactly what you want.

strace is built on ptrace with syscall-stop: the kernel stops the tracee on entry to and exit from each syscall, and strace reads the arguments and return value, decoding them into human-readable form (turning flag bitmasks into names, printing string buffers, resolving file descriptors). The most useful options are about reducing noise:

Option	Effect
-f	follow forked children (essential; many programs do their work in children)
-e trace=open,openat,read	filter to specific syscalls
-e trace=network / file / process	filter by category
-p PID	attach to a running process
-c	summary mode: count and time each syscall type
-s 256	print up to 256 bytes of string arguments (default is short)
-y	annotate file descriptors with the paths/sockets they refer to
-k	print a stack trace for each syscall (find who called it)

ltrace works one level up, intercepting calls into shared libraries via the PLT (from 4.1). It shows you strcpy("buf", "attacker input"), malloc(64), strcmp(input, "secret") — which is enormously revealing, because it exposes the arguments to the functions the program chose to use. For a crackme or a licence check, an ltrace revealing a strcmp against a hardcoded string can end the analysis instantly. Its limitation is that it only sees calls that actually go through the PLT: statically linked functions, inlined functions, and direct syscalls are invisible to it.

The two tools are complementary, and their blind spots define when to use each:

• Use strace for I/O, network, process, and file behaviour — the external effects. It cannot miss a syscall.
• Use ltrace to see the arguments and return values of library functions, including string comparisons and memory operations. It misses anything not routed through the dynamic PLT.
• Neither shows internal computation; for that you need a debugger.

The performance cost is real — each traced call incurs context switches — so tracing slows a program substantially and can perturb timing-sensitive behaviour. For high-volume production tracing, eBPF-based tools (bpftrace, strace --seccomp-bpf) achieve far lower overhead and are the modern choice, though plain strace remains the fastest way to answer "what is this program doing?" interactively.

Examples

# "Why can't it find its config?" -- the canonical strace use.
strace -f -e trace=openat -s 200 ./app 2>&1 | grep -i 'config\|ENOENT'
# ENOENT lines reveal every path the program TRIED and failed to open.

# A complete behavioural transcript, following children.
strace -f -s 256 -y -o trace.log ./suspicious_binary
#   -f  follow forks        -y annotate fds with paths/sockets
#   -s  show long strings    -o write to a file (keeps stdout clean)

# Network activity only (fast malware triage).
strace -f -e trace=network ./sample
# connect(3, {AF_INET, sin_port=htons(443), sin_addr="203.0.113.5"}, 16)
#   ^ the C2 address, in plaintext, without touching the code.

# Summary mode: where does the program spend its syscalls?
strace -c -f ./app
#   % time  calls  syscall      -> a profile of external activity

# ltrace: see the ARGUMENTS to library calls.
ltrace -f -s 128 ./crackme
# strcmp("hunter2", "s3cr3t_p4ss") = -1   <- the expected password, revealed
# strcpy(0x7fff..., "AAAA...")            <- an unbounded copy, a bug smell

# Attach to something already running.
sudo strace -f -p "$(pgrep -n daemon)" -e trace=file

# Modern low-overhead alternative with eBPF:
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'

  Where each tool can and cannot see
  ─────────────────────────────────────────────────────────────

    your program
        │  calls strcmp()  ───────────────►  ltrace sees this
        │  (via PLT into libc)
        ▼
    libc strcmp  ── pure computation ──►  NEITHER tool sees the internals
        │
        │  calls write()   ───────────────►  strace sees this
        ▼
    ─────────── kernel boundary (syscall) ───────────
        │
     kernel services the write

   A STATICALLY linked strcmp bypasses the PLT -> ltrace misses it,
   but its eventual write() syscall still appears in strace.

Key Takeaways

• strace traces the kernel boundary (syscalls) and ltrace the library boundary (PLT calls); together they produce a behavioural transcript with no source or symbols needed.
• Because syscalls are the complete inventory of external effects, an strace log cannot miss any file, network, or process activity a program performs.
• strace is ptrace-based with syscall-stops and decodes arguments into readable form; its key options reduce noise (-f, -e trace=, -c, -s, -y).
• ltrace reveals the arguments and return values of library calls — including revealing strcmp against hardcoded secrets — but only for calls routed through the dynamic PLT.
• Neither tool shows internal computation; use a debugger for that, and remember ltrace misses statically linked, inlined, or direct-syscall code.
• Tracing adds real overhead and perturbs timing; eBPF tools like bpftrace are the low-overhead choice for production or high-volume tracing.

🧪 Practice

1. Run strace -f -e trace=openat on any command-line tool and identify every configuration or library file it searches for, including the ones that returned ENOENT.
2. Use ltrace on a simple password-checking program that compares against a hardcoded string, and recover the expected password from the trace alone.
3. Compare strace -c output for /bin/true and /bin/ls, and explain which syscall categories account for the difference.
4. Interview-style: "A binary makes a network connection, but ltrace shows no connect() call. How is that possible, and how would you confirm the connection?" Hint: consider which calls ltrace can and cannot observe, and which tool is guaranteed to see a connection regardless of how the program makes it.

Memory Error Detection with Valgrind

Theory

C and C++ give the programmer manual control over memory and, in exchange, no safety net: a read past the end of a buffer, a use of freed memory, or a forgotten free produces no error at the language level. The hardware sees only addresses. Bugs of this kind are both the leading cause of security vulnerabilities and notoriously hard to find, because they often do no visible damage at the moment they occur — the corruption surfaces much later, somewhere unrelated. Valgrind exists to catch these the instant they happen.

The mechanism is what makes Valgrind both powerful and slow. It is not a library you link or a compiler option; it is a dynamic binary instrumentation framework that runs your program on a synthetic CPU. Valgrind disassembles the program's machine code into an intermediate representation, inserts its own checking instructions around every memory access, and re-executes the result. Its default tool, Memcheck, uses this to maintain two pieces of shadow state for every byte the program touches:

• Addressability ("A bits"): is this byte currently legal to access at all? Bytes outside any allocation, or in freed memory, are unaddressable.
• Validity ("V bits"): for addressable bytes, has this byte been initialized? Reading uninitialized memory is tracked bit by bit.

With this shadow memory, Memcheck catches, at the exact instruction responsible:

• Reads/writes outside allocated blocks (heap buffer overflows and underflows).
• Use of memory after free (use-after-free) and freeing the same block twice (double-free).
• Use of uninitialized values — and, crucially, only complains when the uninitialized value actually affects observable behaviour (a branch, a syscall, an output), which avoids a flood of harmless noise.
• Memory leaks: blocks still allocated but no longer reachable at exit.
• Mismatched allocation/deallocation (new[] freed with delete, etc.).

The cost is a 10–50x slowdown, because every memory access is instrumented and every allocation shadowed. That is acceptable for a test run or a targeted investigation but not for production or for fuzzing at scale.

This slowdown is exactly why Valgrind is increasingly complemented — and for many workflows replaced — by compiler sanitizers (the next topic), which instrument at compile time for far lower overhead. The division of labour in practice: sanitizers for fast, continuous, CI- and fuzzing-scale checking during development; Valgrind when you need to check a binary you cannot recompile, when you need its particularly thorough uninitialized-value tracking, or when you want its other tools — cachegrind (cache profiling), callgrind (call-graph profiling), helgrind and DRD (data-race detection), and massif (heap profiling).

A key practical point: Valgrind requires no recompilation and no source, so it works on third-party and release binaries, though it is dramatically more useful with debug symbols (-g), which let it report the exact source line of each error rather than just an address. Also note that Memcheck's heap focus means it is weaker on stack and global buffer overflows; AddressSanitizer covers those better, which is one reason the two coexist.

Examples

// Four classic bugs, each caught by Memcheck at the responsible line.
#include <stdlib.h>
#include <string.h>

int main(void) {
    char *p = malloc(16);

    // (1) Heap overflow: writing 1 byte past the 16-byte block.
    p[16] = 'X';                 // Memcheck: "Invalid write of size 1"

    // (2) Uninitialized read that affects a branch.
    int x;
    if (x == 42)                 // Memcheck: "Conditional jump depends on
        p[0] = 'a';              //            uninitialised value(s)"

    // (3) Use-after-free.
    free(p);
    p[0] = 'b';                  // Memcheck: "Invalid write ... after free"

    // (4) Double free.
    free(p);                     // Memcheck: "Invalid free() ... already freed"

    // (5) Leak: this block is never freed and is lost at exit.
    char *leak = malloc(100);
    (void)leak;
    return 0;                    // Memcheck leak summary: "100 bytes ... lost"
}

# Build WITH symbols so Valgrind reports source lines, then run Memcheck.
gcc -g -O0 bugs.c -o bugs
valgrind --leak-check=full --track-origins=yes --error-exitcode=1 ./bugs
#   --leak-check=full    enumerate each leaked block with its allocation stack
#   --track-origins=yes  trace an uninitialized value back to WHERE it was born
#                        (invaluable, but adds extra slowdown)
#   --error-exitcode=1   make CI fail on any detected error

  A Memcheck report, annotated
  ─────────────────────────────────────────────────────────────
  Invalid write of size 1
     at 0x109165: main (bugs.c:9)               <- the exact faulting line
   Address 0x4a4d050 is 0 bytes after a block of size 16 alloc'd
     at 0x484...: malloc (vg_replace_malloc.c)
     by 0x109155: main (bugs.c:6)               <- where the block was allocated

  Reading: a 1-byte write landed immediately AFTER a 16-byte heap block,
  i.e. p[16] on a malloc(16). The two stacks pin both the error site and
  the allocation, which together usually identify the fix immediately.

Key Takeaways

• C/C++ provide no runtime memory safety, and memory errors are both the top source of vulnerabilities and hard to find because damage often surfaces far from its cause.
• Valgrind runs the program on a synthetic CPU via dynamic binary instrumentation, needing no recompilation or source — though -g symbols make its reports far more useful.
• Memcheck maintains shadow addressability ("A") and validity ("V") bits per byte, catching overflows, use-after-free, double-free, uninitialized reads, and leaks at the exact faulting instruction.
• It reports an uninitialized value only when it actually affects a branch, syscall, or output, which suppresses harmless noise.
• The 10–50x slowdown makes it suitable for targeted investigation and test runs but not for production or large-scale fuzzing.
• Compiler sanitizers complement it with much lower overhead for CI and fuzzing; Valgrind remains the choice for un-recompilable binaries and offers extra tools (callgrind, helgrind, massif).

🧪 Practice

1. Write a program with a one-byte heap overflow and confirm Valgrind reports the exact source line and the allocation site; then fix it and confirm a clean run.
2. Introduce an uninitialized-variable read that controls a branch, run Memcheck with --track-origins=yes, and explain what the origin trace adds over the basic report.
3. Create a program that leaks memory only on an error path, and use --leak-check=full to find it; explain why the leak is invisible on the success path.
4. Interview-style: "A team runs Valgrind clean in CI but still ships a stack buffer overflow. How is that consistent?" Hint: recall which memory region Memcheck instruments most thoroughly and which it covers weakly, and which tool from the next topic would have caught the stack bug.

Sanitizers: ASan, MSan, UBSan

Theory

Sanitizers answer the same question as Valgrind — "is this program violating memory or language rules?" — but with a different strategy that changes the economics entirely. Instead of instrumenting a finished binary at runtime, the compiler inserts the checks during compilation. The program checks itself as it runs. Because the compiler knows the types, sizes, and structure of everything (information lost in a bare binary), the instrumentation is far more precise and far cheaper: a typical 2x slowdown versus Valgrind's 10–50x. That difference is what makes sanitizers usable in continuous integration and, critically, in fuzzing, where billions of executions make Valgrind's overhead prohibitive.

They are a compile-time feature: you rebuild with a flag (-fsanitize=...), and the resulting binary carries the checks. This means you need source (or at least to be able to recompile), which is the main trade-off against Valgrind.

The family divides by what each one checks, and they are not fully combinable:

AddressSanitizer (ASan) — -fsanitize=address — catches spatial and temporal memory errors: heap, stack, and global buffer overflows (covering the stack and global cases Valgrind handles weakly), use-after-free, use-after-return, and double-free. Its mechanism is shadow memory plus redzones: it surrounds every allocation with poisoned "redzone" bytes and maps a shadow byte for every 8 bytes of application memory recording which are poisoned. Any access to a poisoned byte traps immediately with a detailed report. It also provides a quarantine for freed memory so the addresses are not immediately reused, which is what lets it detect use-after-free reliably.

MemorySanitizer (MSan) — -fsanitize=memory — catches exactly one thing ASan does not: reads of uninitialized memory. It tracks the initialization state of every bit and reports when uninitialized data influences a branch or a syscall. It is powerful but demanding: to avoid false positives it needs all code, including libraries, to be instrumented, which is why it is less commonly deployed than ASan.

UndefinedBehaviorSanitizer (UBSan) — -fsanitize=undefined — catches C/C++ undefined behaviour that is not strictly a memory error: signed integer overflow, shifts past the type width, null-pointer dereference and misaligned access, dividing INT_MIN by -1, and array-index-out-of-bounds where the bound is known. This matters enormously for security because the optimizer is permitted to assume UB never happens and will delete checks or miscompile code that relies on it (the signed-overflow and sign-extension traps from 4.1 live here). UBSan is cheap and composable, and many of its sub-checks can be enabled individually.

The compatibility rules are worth memorizing to avoid wasted builds:

	Combines with	Notes
ASan + UBSan	yes	the standard development combination
ASan + MSan	no	both claim the shadow-memory mechanism; mutually exclusive
MSan + UBSan	yes	but MSan's all-code requirement still applies
ThreadSanitizer (TSan)	with UBSan only	-fsanitize=thread, for data races; not with ASan/MSan

The standard practice: build with -fsanitize=address,undefined -fno-omit-frame-pointer -g for day-to-day development and CI, run MSan separately in its own build when chasing uninitialized-memory bugs, and TSan separately for concurrency. In fuzzing, ASan+UBSan on the fuzz target is close to mandatory, because without it a memory corruption that does not crash outright goes undetected and the fuzzer never records it as a finding.

Examples

// Bugs that each sanitizer is designed to catch.
#include <stdlib.h>
#include <limits.h>

int stack_overflow(void) {
    int a[4];
    return a[5];                 // ASan: "stack-buffer-overflow"
}

int use_after_return(void) {
    int local = 42;
    static int *p;
    p = &local;                  // dangling after return...
    return *p;                   // ASan (with detect_stack_use_after_return)
}

int signed_overflow(int n) {
    return n + 1;                // UBSan: "signed integer overflow" when n==INT_MAX
}

int bad_shift(int n) {
    return 1 << n;               // UBSan: "shift exponent too large" for n>=32
}

int main(void) {
    char *p = malloc(8);
    free(p);
    char c = p[0];               // ASan: "heap-use-after-free"
    (void)c; (void)signed_overflow(INT_MAX); (void)bad_shift(40);
    return stack_overflow();
}

# The standard development build: ASan + UBSan together.
gcc -fsanitize=address,undefined -fno-omit-frame-pointer -g bugs.c -o bugs
./bugs
#   ==12345==ERROR: AddressSanitizer: heap-use-after-free on address 0x...
#     READ of size 1 at 0x... thread T0
#       #0 main bugs.c:20            <- faulting line
#     freed by thread T0 here:       <- where free() happened
#     previously allocated here:     <- where malloc() happened

# MemorySanitizer MUST be built separately (Clang; mutually exclusive w/ ASan).
clang -fsanitize=memory -fsanitize-memory-track-origins -g unic.c -o unic
./unic   # reports "use-of-uninitialized-value" with an origin trace

# Make UBSan abort (and print) rather than continue past the UB.
gcc -fsanitize=undefined -fno-sanitize-recover=all -g bugs.c -o bugs_ub

# The fuzzing-grade target: sanitizer + coverage instrumentation.
clang -fsanitize=address,undefined,fuzzer -g target.c -o fuzzer

  Choosing the tool
  ─────────────────────────────────────────────────────────────
   Need to catch...                     Use
   ─────────────────────────────        ───────────────────────────
   heap/stack/global overflow, UAF      ASan
   uninitialized memory reads           MSan   (separate build)
   signed overflow, bad shift, UB       UBSan  (compose with ASan)
   data races                           TSan   (separate build)
   a binary you CANNOT recompile        Valgrind (no source needed)

   Day-to-day: -fsanitize=address,undefined   (ASan + UBSan)
   ASan and MSan can NEVER be combined in one build.

Key Takeaways

• Sanitizers instrument at compile time, so the program checks itself; knowing the types and sizes makes the checks more precise and far cheaper (~2x) than Valgrind's runtime instrumentation.
• That low overhead is what makes them viable in CI and essential in fuzzing, where Valgrind's cost is prohibitive — but they require source or the ability to recompile.
• ASan catches spatial and temporal memory errors including the stack and global overflows Valgrind handles weakly, using redzones, shadow memory, and a free quarantine.
• MSan catches uninitialized-memory reads specifically and needs all code instrumented to avoid false positives, so it runs in its own build.
• UBSan catches undefined behaviour that is not a memory error — signed overflow, oversized shifts, null/misaligned access — which matters because the optimizer may delete code that relies on UB.
• ASan and MSan are mutually exclusive; the standard combination is ASan+UBSan, with MSan and TSan run as separate builds.

🧪 Practice

1. Build a program with a stack buffer overflow under -fsanitize=address and confirm ASan reports it with the faulting line, then confirm plain Valgrind Memcheck does not.
2. Write code with a signed integer overflow, build with -fsanitize=undefined -fno-sanitize-recover=all, and observe the diagnostic; then show at -O2 how relying on that overflow produces surprising results without the sanitizer.
3. Attempt to compile a program with both -fsanitize=address and -fsanitize=memory, observe the error, and explain the underlying mechanism conflict.
4. Interview-style: "Why is building a fuzz target with AddressSanitizer close to mandatory?" Hint: consider what happens to a heap overflow that corrupts memory but does not immediately crash, and whether a fuzzer would otherwise ever record it as a finding.

Instrumentation and Binary Hooking

Theory

Instrumentation is the practice of inserting extra code into a program to observe or alter its behaviour, and hooking is the specific technique of intercepting a function call so your code runs instead of, or alongside, the original. These are the tools you reach for when you want to change what a running program does without having its source — to log every call to malloc, to make a licence check always succeed, to trace a cryptographic key as it flows through a function, or to sandbox an application by interposing on the functions it uses to touch the world.

The techniques span a spectrum from "requires nothing but an environment variable" to "rewrites machine code in memory", and choosing the right one is most of the skill.

LD_PRELOAD is the simplest and most elegant, and it exploits symbol interposition from the dynamic-linking discussion in 4.1: the dynamic linker resolves each symbol to the first definition it finds, and a library named in LD_PRELOAD is searched before all others. So a preloaded library that defines malloc captures every call the program makes to malloc. Inside your replacement you typically call dlsym(RTLD_NEXT, "malloc") to find and invoke the real one, wrapping rather than replacing it. This needs no source, no recompilation, and no special privileges — but it only works for calls that go through the dynamic symbol table, so statically linked binaries and direct syscalls are immune. It is also, for exactly the same reason, a known persistence and credential-theft vector, which is why setuid binaries ignore LD_PRELOAD.

Binary rewriting / inline hooking operates at the instruction level. The classic technique overwrites the first few bytes of the target function with a jmp to your hook; the hook does its work and then executes the overwritten instructions (a "trampoline") before jumping back. This works even on statically linked functions and on code that bypasses the PLT, which is why it is the mechanism behind both EDR products (hooking Windows API functions to watch for malicious behaviour) and the malware and game cheats that evade them. Frameworks like Frida, Microsoft Detours, and PLT/GOT hooking automate it.

Dynamic binary instrumentation (DBI) frameworks — Intel Pin, DynamoRIO, Frida, QBDI, and Valgrind's core — are the heavyweight option: they take control of the entire execution and let you insert analysis code at arbitrary granularity (every instruction, every basic block, every memory access). This is how you build a custom tracer, a taint-tracking engine, or a coverage collector. The cost is substantial slowdown and complexity, but the visibility is total.

Technique	Granularity	Needs source?	Sees static/syscalls?	Typical use
LD_PRELOAD	exported library functions	no	no	wrap malloc, fake a function, interpose I/O
PLT/GOT hooking	imported functions	no	no	redirect specific imports
Inline hooking	any function with a known address	no	yes	EDR, cheats, Frida Interceptor
DBI (Pin/DynamoRIO/Frida)	instruction / block / access	no	yes	coverage, taint, custom analysis
Compiler instrumentation	anything the compiler sees	yes	yes	sanitizers, coverage-guided fuzzing

Frida deserves a special mention because it has become the default for interactive instrumentation: it injects a JavaScript engine into the target process and lets you write hooks in JavaScript against a live program — attach to a running app, intercept a function, read and rewrite its arguments and return value, all without restarting. For mobile and application security assessment it is close to ubiquitous.

The dual-use nature is unavoidable and worth stating plainly: the same interposition that lets a defender's EDR watch for malicious API calls lets an attacker's rootkit hide files, and the same LD_PRELOAD that wraps malloc for debugging can steal credentials. These are techniques, and their ethics live entirely in authorization and intent.

Examples

// LD_PRELOAD hook: wrap malloc to log every allocation, then call the real one.
// Build:  gcc -shared -fPIC hook.c -o hook.so -ldl
// Use:    LD_PRELOAD=./hook.so ./target
#define _GNU_SOURCE
#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>

void *malloc(size_t size) {
    // Find the REAL malloc, the next one in the library search order.
    static void *(*real_malloc)(size_t) = NULL;
    if (!real_malloc)
        real_malloc = dlsym(RTLD_NEXT, "malloc");

    void *p = real_malloc(size);
    // Use write/fprintf carefully to avoid recursing into malloc.
    fprintf(stderr, "[hook] malloc(%zu) = %p\n", size, p);
    return p;                    // transparent wrapper: behaviour unchanged
}

// LD_PRELOAD to defeat a trivial licence check WITHOUT touching the binary.
// The target calls time(NULL) to see if a trial has expired; freeze it.
#define _GNU_SOURCE
#include <time.h>
time_t time(time_t *t) {
    time_t fixed = 1704067200;   // always "January 2024"
    if (t) *t = fixed;
    return fixed;                // the expiry comparison now always passes
}
// Demonstrates interposition's power and why setuid binaries must ignore it.

// Frida: hook a function in a LIVE process and rewrite its result.
// Run:  frida -l hook.js -f ./target   (or -p PID to attach)
const addr = Module.getExportByName(null, 'check_password');
Interceptor.attach(addr, {
    onEnter(args) {
        // args[0] is the first argument (System V: rdi).
        this.input = args[0].readUtf8String();
        console.log(`[frida] check_password("${this.input}")`);
    },
    onLeave(retval) {
        console.log(`[frida]   -> ${retval}`);
        retval.replace(1);       // force "password correct" regardless
    }
});

# PLT/GOT hooking and DBI, at a glance.
LD_PRELOAD=./hook.so ltrace -f ./target     # combine interposition with tracing

# Intel Pin: run a pintool that counts executed instructions (coverage basis).
pin -t inscount.so -- ./target

# Confirm a setuid binary ignores LD_PRELOAD (a security boundary).
LD_PRELOAD=./hook.so /usr/bin/passwd        # hook does NOT load; by design

Key Takeaways

• Instrumentation inserts observation or control code into a program; hooking specifically intercepts a function so your code runs in its place or alongside it.
• LD_PRELOAD exploits symbol interposition — the first definition wins — to wrap exported library functions with no source or recompilation, but it cannot touch statically linked code or direct syscalls, and setuid binaries ignore it.
• Inline/binary hooking overwrites a function's opening bytes with a jump to a trampoline, working even on static functions; it underlies both EDR products and the malware that evades them.
• Dynamic binary instrumentation frameworks (Pin, DynamoRIO, Frida, QBDI) give total, arbitrary-granularity visibility at the cost of heavy slowdown.
• Frida injects a scripting engine into a live process, letting you intercept and rewrite arguments and return values interactively, which has made it the default for app and mobile assessment.
• These techniques are strictly dual-use; the identical mechanism serves defence and offence, so their legitimacy rests entirely on authorization and intent.

🧪 Practice

1. Write an LD_PRELOAD library that logs every open() call with its path, test it on a command-line tool, and explain why it misses a statically linked copy of the same tool.
2. Use dlsym(RTLD_NEXT, ...) to wrap free() so it counts total frees, and confirm the program's behaviour is otherwise unchanged.
3. Use Frida to attach to a simple program and log the arguments and return value of a chosen function without restarting it.
4. Interview-style: "Why do setuid binaries ignore LD_PRELOAD, and what class of attack would be possible if they did not?" Hint: consider what an attacker-supplied library running inside a process that executes with elevated privileges could do, and why the loader must drop that input at a privilege boundary.

Runtime Tracing with ptrace

Theory

ptrace ("process trace") is the Linux system call that underlies nearly every dynamic-analysis tool discussed in this section — GDB, strace, ltrace all sit on top of it. Understanding it directly demystifies how debugging actually works and lets you build custom tracers when off-the-shelf tools do not fit. The name of the concept is one process observing and controlling another.

The model is a tracer/tracee relationship. A tracer attaches to a tracee (either by PTRACE_ATTACH to a running process or by the tracee calling PTRACE_TRACEME and then execve, which is how a debugger launches a program under its control). Once attached, the kernel stops the tracee and hands control to the tracer at defined events, and the tracer drives execution forward with ptrace requests. The essential operations:

Request	Effect
PTRACE_TRACEME	tracee declares itself traceable by its parent
PTRACE_ATTACH / PTRACE_SEIZE	tracer attaches to an existing process
PTRACE_PEEKTEXT/PEEKDATA	read a word from the tracee's memory
PTRACE_POKETEXT/POKEDATA	write a word into the tracee's memory
PTRACE_GETREGS / SETREGS	read/write the tracee's registers
PTRACE_CONT	resume until the next signal
PTRACE_SINGLESTEP	execute exactly one instruction, then stop
PTRACE_SYSCALL	resume until the next syscall entry or exit (this is how strace works)
PTRACE_SETOPTIONS	configure behaviour (e.g. trace children)

With these you can reconstruct every debugger feature: a breakpoint is POKETEXT an int3, wait for the SIGTRAP, GETREGS/PEEKDATA to inspect, then restore and continue; a syscall tracer is a loop of PTRACE_SYSCALL and GETREGS. The tracer learns of each stop through waitpid, exactly as it would for any child process, and reads the stop reason from the returned status.

The security relevance of ptrace runs in two directions:

As an analysis tool, it is the foundation for building exactly the tracer you need — a syscall filter logger, a targeted memory watcher, an automated unpacker that single-steps until the entropy of a region drops (signalling decompression finished) and then dumps it.

As an attack surface and a defence, ptrace is powerful enough to read and write another process's memory, so the kernel guards it. The Yama LSM (/proc/sys/kernel/yama/ptrace_scope) restricts who may trace whom: a value of 1 (the common default) allows tracing only direct children unless you have CAP_SYS_PTRACE, blocking the classic attack of attaching to another user's process to steal its secrets. This is also why ptrace-based injection is a monitored technique in EDR. And because a process can call ptrace(PTRACE_TRACEME) on itself — and a process can only be traced once — a program can detect a debugger by trying to trace itself: if the call fails, a debugger is already attached. This is the single most common anti-debugging check, covered next.

The practical caution is that raw ptrace is fiddly: it operates a word at a time, requires careful waitpid status decoding, and differs in subtle ways across architectures. For most tasks a higher-level tool or a library wrapper is the right choice; writing raw ptrace is for when you need behaviour those tools do not offer.

Examples

// A minimal syscall tracer: the essence of strace in ~30 lines.
// Build: gcc tracer.c -o tracer ; Run: ./tracer /bin/echo hi
#include <stdio.h>
#include <sys/ptrace.h>
#include <sys/wait.h>
#include <sys/user.h>
#include <unistd.h>

int main(int argc, char *argv[]) {
    pid_t child = fork();
    if (child == 0) {
        // CHILD: declare traceable, then exec the target.
        ptrace(PTRACE_TRACEME, 0, NULL, NULL);
        execvp(argv[1], &argv[1]);        // stops on exec; parent takes over
        _exit(1);
    }

    // PARENT (tracer): loop over syscall-entry stops.
    int status;
    waitpid(child, &status, 0);           // initial stop after exec
    while (1) {
        // Run until the next syscall ENTRY, then stop.
        ptrace(PTRACE_SYSCALL, child, NULL, NULL);
        waitpid(child, &status, 0);
        if (WIFEXITED(status)) break;     // tracee finished

        struct user_regs_struct regs;
        ptrace(PTRACE_GETREGS, child, NULL, &regs);
        // orig_rax holds the syscall NUMBER at entry (see ABI, 4.1).
        printf("syscall %llu (rdi=0x%llx rsi=0x%llx)\n",
               regs.orig_rax, regs.rdi, regs.rsi);

        // Run to syscall EXIT so the next PTRACE_SYSCALL lands on the
        // following entry rather than this call's exit.
        ptrace(PTRACE_SYSCALL, child, NULL, NULL);
        waitpid(child, &status, 0);
        if (WIFEXITED(status)) break;
    }
    return 0;
}

// The anti-debugging self-trace check, and why it works.
#include <sys/ptrace.h>
#include <stdio.h>
int main(void) {
    // A process can be traced by at most ONE tracer. If a debugger is
    // already attached, this TRACEME fails (returns -1).
    if (ptrace(PTRACE_TRACEME, 0, NULL, NULL) == -1) {
        printf("debugger detected -- exiting\n");
        return 1;
    }
    printf("running normally\n");
    return 0;
}
// Bypass: have the debugger intercept ptrace and force a 0 return,
// or patch the comparison -- covered in the next topic.

# The Yama policy that governs who may ptrace whom.
cat /proc/sys/kernel/yama/ptrace_scope
#   0 = classic (any process of the same uid)
#   1 = restricted: only direct children, unless CAP_SYS_PTRACE (common default)
#   2 = admin-only    3 = ptrace disabled entirely

# Temporarily relax it for a debugging session (needs root):
sudo sysctl kernel.yama.ptrace_scope=0

Key Takeaways

• ptrace is the single kernel primitive beneath GDB, strace, and ltrace: one process stopping, inspecting, and controlling another.
• A tracer attaches via PTRACE_ATTACH or the tracee's own PTRACE_TRACEME+execve, then drives execution with requests like CONT, SINGLESTEP, and SYSCALL, learning of each stop through waitpid.
• Every debugger feature reduces to these calls: a breakpoint is poking an int3 and catching the SIGTRAP; a syscall tracer is a PTRACE_SYSCALL loop reading registers with GETREGS.
• The Yama ptrace_scope setting restricts who may trace whom, blocking the attach-and-steal-secrets attack; the common default permits tracing only direct children without CAP_SYS_PTRACE.
• Because a process can be traced only once, a program can detect a debugger by calling PTRACE_TRACEME on itself and checking for failure — the most common anti-debug check.
• Raw ptrace is low-level and fiddly (word-at-a-time, careful status decoding), so reach for it only when higher-level tools cannot do the job.

🧪 Practice

1. Compile and run the minimal syscall tracer above on /bin/echo and correlate its output with strace -f on the same command.
2. Extend the tracer to decode and print the syscall name for a handful of common numbers, using the x86-64 syscall table.
3. Observe your system's ptrace_scope value, then demonstrate that attaching GDB to a process you do not own is blocked at the default setting and explain which attack that prevents.
4. Interview-style: "Explain how you would implement a software breakpoint using only ptrace requests." Hint: describe writing a trap byte with POKETEXT, catching the resulting signal with waitpid, inspecting with GETREGS, and the dance of restoring the original byte to continue.

Fuzzing Fundamentals and Coverage Feedback

Theory

Fuzzing is the brute-force success story of modern security: throw a flood of malformed and unexpected inputs at a program and watch for crashes, each of which is a candidate bug. It sounds unsophisticated, and the earliest fuzzers were — feeding random bytes to a program and seeing what broke. What transformed fuzzing from a curiosity into the single most productive bug-finding technique in the industry was one idea: coverage feedback.

The problem with purely random fuzzing is that real programs are full of narrow gates. Code like if (input[0]==0x7f && input[1]=='E' && ...) guards everything behind it, and the probability that random bytes satisfy such a check is astronomically small. A blind fuzzer spends essentially all its time bouncing off the first validation and never reaches the interesting code. This is the same wall that motivated symbolic execution in 4.2.

Coverage-guided fuzzing breaks through it with an evolutionary loop:

1. Instrument the target so that each execution reports which code it covered (which basic blocks or edges ran).
2. Keep a corpus of inputs. Pick one, mutate it (flip bits, splice, insert known-interesting values).
3. Run the mutated input. If it covered any new edge not seen before, it reached somewhere new — so add it to the corpus as a stepping stone. Otherwise discard it.
4. Repeat, billions of times.

The effect is that the fuzzer learns the input structure by reward. An input that happens to get the first magic byte right covers a new edge, so it is saved; later mutations build on it to get the second byte right, and so on. The fuzzer climbs through the validation gates incrementally, without ever being told what the format is. This is why coverage-guided fuzzers like AFL++, libFuzzer, and honggfuzz find bugs that random fuzzing never could.

Fuzzers divide along two axes:

• Mutation-based vs generation-based. Mutation-based (AFL++) starts from seed inputs and mutates them — easy to set up, format-agnostic. Generation-based (grammar fuzzers, Peach) builds inputs from a specification of the format — more work, but reaches deep structural validity that mutation struggles with.
• Dumb vs coverage-guided. The distinction above; coverage guidance is the dividing line between old and modern fuzzing.

Two practices make fuzzing dramatically more effective and are worth stating as rules:

Pair the fuzzer with sanitizers. A fuzzer detects bugs by observing crashes, but a memory corruption often does not crash immediately — a one-byte heap overflow may silently corrupt and continue. AddressSanitizer (from the sanitizers topic) turns that silent corruption into an immediate, loud crash the fuzzer records. Fuzzing without ASan misses most memory bugs; this is the single most important fuzzing configuration fact.

Write a good harness and seed corpus. Most fuzzing targets a harness — a small function (LLVMFuzzerTestOneInput) that takes a byte buffer and feeds it to the code under test. A good harness exercises a real parsing entry point, is fast (fuzzers run thousands of executions per second), and is deterministic. A seed corpus of valid sample inputs gives the fuzzer a head start past the outer validation.

The modern frontier combines fuzzing with symbolic execution (hybrid fuzzing, e.g. Driller): the fuzzer does the fast, broad exploration, and when it gets stuck at a hard check it cannot mutate past, the symbolic engine solves for an input that gets through, which is handed back to the fuzzer as a new seed. And large-scale continuous fuzzing — Google's OSS-Fuzz runs fuzzers against hundreds of open-source projects around the clock — has found tens of thousands of real vulnerabilities, making it a cornerstone of modern software assurance.

Examples

// A libFuzzer harness: the standard shape of a modern fuzz target.
// Build: clang -g -fsanitize=address,fuzzer harness.c parser.c -o fuzz
// Run:   ./fuzz corpus/        (it fuzzes forever, saving crashes)
#include <stddef.h>
#include <stdint.h>

// The function under test: a parser with a deliberate bug.
int parse_packet(const uint8_t *data, size_t len);

// libFuzzer calls this repeatedly with mutated `data`.
int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
    parse_packet(data, size);     // ASan turns any corruption into a crash
    return 0;                     // non-zero is reserved; always return 0
}

// The parser being fuzzed, with the classic length-trust bug.
#include <string.h>
#include <stdint.h>
#include <stddef.h>
int parse_packet(const uint8_t *data, size_t len) {
    if (len < 2) return -1;
    uint8_t claimed = data[0];    // attacker-controlled "payload length"
    char buf[64];
    // BUG: `claimed` is trusted and never checked against `len` or 64.
    // A coverage-guided fuzzer quickly learns to make data[0] large,
    // gets past the `len < 2` gate (new coverage), then overflows buf.
    memcpy(buf, data + 1, claimed);   // ASan: stack-buffer-overflow
    return buf[0];
}

# AFL++: instrument at compile time, then fuzz with a seed corpus.
afl-clang-fast -fsanitize=address -g parser_main.c -o target_afl
mkdir -p seeds && printf '\x01A' > seeds/valid     # one valid-ish seed
afl-fuzz -i seeds -o findings -- ./target_afl @@   # @@ = input file path
#   The AFL++ TUI shows: execs/sec, edges covered, unique crashes found.

# libFuzzer run with useful limits.
./fuzz corpus/ -max_len=4096 -timeout=5 -rss_limit_mb=2048
#   Crashing inputs are written as crash-<hash> for later triage.

# Minimize a crashing input to the smallest reproducer.
afl-tmin -i findings/crashes/id:000000* -o minimized -- ./target_afl @@

  Why coverage feedback beats random fuzzing
  ─────────────────────────────────────────────────────────────

   Target:  if (in[0]==0x7f && in[1]=='E' && in[2]=='L' && in[3]=='F') { ... }

   BLIND fuzzer:  P(hitting all four bytes at random) = (1/256)^4
                  ~ 1 in 4 billion per attempt. Never gets inside.

   COVERAGE-GUIDED fuzzer:
     mutation sets in[0]=0x7f  -> covers the 2nd comparison (NEW edge)
       -> SAVE this input as a seed
     mutate the saved seed, set in[1]='E' -> covers 3rd compare (NEW)
       -> SAVE
     ... four incremental steps instead of one astronomical leap.
   Each solved byte is rewarded and built upon: the gate is climbed.

Key Takeaways

• Fuzzing feeds floods of malformed inputs to a program and treats each crash as a candidate bug; coverage feedback is what turned it from a curiosity into the industry's top bug finder.
• Blind random fuzzing cannot pass narrow validation gates (magic bytes, checksums); the probability of satisfying them by chance is negligible.
• Coverage-guided fuzzing runs an evolutionary loop — mutate, run, keep inputs that reach new edges — so the fuzzer incrementally learns the input structure by reward.
• Always pair a fuzzer with AddressSanitizer: memory corruption frequently does not crash on its own, and ASan turns silent corruption into a recorded finding — the most important fuzzing configuration fact.
• A good harness exercises a real, fast, deterministic parsing entry point, and a seed corpus of valid inputs gives the fuzzer a head start past outer validation.
• Hybrid fuzzing adds a symbolic engine to solve the hard checks the fuzzer gets stuck on, and continuous services like OSS-Fuzz have found tens of thousands of real vulnerabilities.

🧪 Practice

1. Write a libFuzzer harness for a small parser with a buffer overflow, build it with -fsanitize=address,fuzzer, and confirm the fuzzer finds the bug within seconds.
2. Run the same target with and without AddressSanitizer and explain any difference in whether the overflow is detected.
3. Give a fuzzer a target guarded by a four-byte magic value; measure (by reading the coverage counter) how the corpus grows as it solves each byte.
4. Interview-style: "Your fuzzer has run for days at millions of executions per second but coverage has plateaued well short of a function you know is reachable. What is likely blocking it and what would you do?" Hint: think about what kind of input check random mutation almost never satisfies — a checksum or a specific 32-bit constant — and which 4.2 technique is designed to solve exactly that.

Crash Triage and Root Cause Analysis

Theory

A fuzzer that runs for a day can produce thousands of crashing inputs. The overwhelming majority are duplicates of a handful of underlying bugs, and some are not security-relevant at all. Crash triage is the process of turning a pile of crashes into a short, deduplicated, prioritized list of distinct bugs; root cause analysis is then determining, for each one, why it crashes and whether it is exploitable. This is where dynamic analysis pays off, and it is a distinct skill from finding the crash in the first place.

The first job is deduplication, because triaging the same bug a thousand times is wasted effort. Crashes are grouped by a signature, most commonly a hash of the crashing call stack (the top N frames). Tools like AFL++'s crash exploration mode, casr, and the !exploitable-style classifiers automate this. A subtlety: the same bug can crash at different stacks (a corrupted heap crashes later, somewhere unrelated), and different bugs can share a stack, so stack-hash dedup is a heuristic, not ground truth — the real test is whether the same fix resolves them.

The second job is minimization. A fuzzer's crashing input is usually bloated with irrelevant bytes. Minimizing it to the smallest input that still triggers the crash (AFL++'s afl-tmin, libFuzzer's -minimize_crash) makes the subsequent analysis vastly easier, because you can see exactly which bytes matter.

The third and central job is root cause analysis: reproduce under a debugger or sanitizer and work backward from the crash to the defect. The crucial insight, repeated from the Valgrind discussion, is that the crash site is usually not the bug site. A heap overflow corrupts metadata but crashes later when the allocator touches it; a use-after-free crashes when the freed memory is reused, not when it was freed. The analysis is a backward hunt from symptom to cause. The tools that make this tractable:

• AddressSanitizer reports the allocation and free stacks alongside the access, which for heap bugs often names the cause directly.
• Reverse debugging (rr, GDB's record/reverse-continue) is transformative here: it records an execution deterministically and lets you run backward from the crash to find where the bad value originated — turning "how did this pointer get corrupted?" from hours of guessing into a watchpoint and a reverse-continue.
• Core dumps (ulimit -c unlimited, then gdb prog core) capture the state at crash time for post-mortem analysis when you cannot reproduce interactively.

The final job is exploitability assessment — classifying how dangerous a crash is, to prioritize fixing. The rough hierarchy, from most to least severe:

Crash type	Typical exploitability
Controlled write to a controlled address	usually exploitable (write-what-where)
rip control (crash at an attacker-influenced address)	usually exploitable
Heap or stack overflow with attacker-controlled data	often exploitable
Use-after-free	often exploitable
Read of a controlled address (info leak)	moderate; aids other bugs
Null-pointer dereference	usually just a denial of service
Division by zero, assertion failure	usually not a security issue

Classifiers like the GDB exploitable plugin automate a first pass, but they are heuristic; the signal they give (for example, that the faulting instruction is a write through a register holding attacker bytes) guides where to spend deeper analysis. The output of good triage is a bug report that a developer can act on: a minimized reproducer, the root-cause location, the bug class, and an exploitability judgement.

Examples

# ── Deduplicate a pile of crashes by crashing-stack signature ──
for c in findings/crashes/id:*; do
    # Get a short stack hash from the sanitizer/debugger for each crash.
    ASAN_OPTIONS=abort_on_error=0 ./target_asan "$c" 2>&1 \
      | grep -m3 '#' | md5sum | cut -c1-8
done | sort | uniq -c | sort -rn
#   The counts show how many "unique" crashes collapse into few real bugs.

# ── Minimize a crashing input to its essential bytes ──
afl-tmin -i findings/crashes/id:000000,sig:11 -o min.bin -- ./target_afl @@
./fuzz -minimize_crash=1 -runs=100000 crash-abcd1234   # libFuzzer equivalent

# ── Post-mortem from a core dump ──
ulimit -c unlimited
./target_asan min.bin                         # crashes, drops a core
gdb -q ./target_asan core -ex 'bt' -ex 'info registers rip' -ex quit

# ── First-pass exploitability classification ──
gdb -q ./target -ex "run min.bin" -ex "exploitable" -ex quit
#   -> e.g. "Exploitable: write to controlled address" with a rationale.

# Reverse debugging with rr: the highest-leverage root-cause tool.
rr record ./target min.bin          # record the crashing run deterministically
rr replay                           # replay it in GDB
# Inside the rr/GDB session:
#   (rr) continue                   # run forward to the crash
#   (rr) watch -l corrupted_ptr     # watch the location that holds the bad value
#   (rr) reverse-continue           # run BACKWARD to the last write to it
#   -> stops at the exact instruction that corrupted the pointer, which is
#      the ROOT CAUSE, often many functions away from the crash site.

  Crash site vs bug site: the fundamental triage insight
  ─────────────────────────────────────────────────────────────
   time ──►
   [t0] overflow writes past buf, corrupting an adjacent heap chunk's
        size field                               <-- THE BUG (root cause)
   [t1] ...program continues normally for a while...
   [t2] free() reads the corrupted size, walks a bad pointer -> SIGSEGV
                                                  <-- THE CRASH (symptom)

   Forward debugging starts at t2 and must reason backward.
   ASan reports the allocation/free stacks; rr lets you reverse-continue
   from t2 to t0 directly. Without them, t0 is found by painful guessing.

Key Takeaways

• Triage turns thousands of fuzzer crashes into a short, deduplicated, prioritized list of distinct bugs; root cause analysis then explains and rates each one.
• Deduplication by crashing-stack hash is a useful heuristic but not ground truth — the same bug can crash at different stacks and vice versa, so the real test is whether one fix resolves them.
• Minimizing a crashing input to its essential bytes makes every later step far easier by revealing exactly which bytes matter.
• The crash site is usually not the bug site: overflows and use-after-frees surface later and elsewhere, so analysis is a backward hunt from symptom to cause.
• AddressSanitizer's allocation/free stacks and reverse debuggers like rr (record once, reverse-continue to the originating write) make that backward hunt tractable.
• Exploitability assessment ranks crashes — controlled write-what-where and rip control at the top, null dereferences and assertion failures near the bottom — to prioritize fixing; automated classifiers give a heuristic first pass.

🧪 Practice

1. Collect several crashing inputs for one underlying bug, compute a stack-hash signature for each, and confirm they deduplicate to a single group.
2. Take a bloated crashing input and minimize it with afl-tmin or libFuzzer's -minimize_crash, then identify which specific bytes trigger the bug.
3. Record a crashing run with rr, set a watchpoint on the corrupted value, and use reverse-continue to find the instruction that caused the corruption; compare the root-cause location with the crash site.
4. Interview-style: "A fuzzer reports a crash that is a null-pointer dereference and another that is a heap write through an attacker-influenced pointer. You can only fix one today. Which, and why?" Hint: map each to the exploitability hierarchy — what can an attacker achieve with a controlled write versus a null dereference — and weigh denial of service against potential code execution.

Anti-Debugging Techniques and Bypasses

Theory

Programs that do not want to be analysed — malware above all, but also DRM, anti-cheat, and some commercial software — actively resist the dynamic-analysis tools covered in this section. Anti-debugging is the collection of techniques a program uses to detect, hinder, or evade a debugger or tracer. For a defender analysing malware, recognizing and neutralizing these is a routine and necessary skill; for a developer, it is a reminder that these measures raise an analyst's cost but, per Kerckhoffs, never provide a guarantee.

The techniques fall into a few families, each exploiting a specific mechanism from earlier in this chapter:

Debugger-presence detection asks "am I being traced?" in various ways:

• Self-ptrace: call ptrace(PTRACE_TRACEME) on itself; since a process can be traced only once, failure means a debugger is already attached (from the ptrace topic). The most common check on Linux.
• /proc/self/status: read the TracerPid field — non-zero means a tracer is attached.
• IsDebuggerPresent / PEB flag on Windows: a flag in the process environment block set by the OS when a debugger is present.
• Parent-process check: if the parent is gdb or strace rather than a shell, be suspicious.

Timing checks exploit the fact that single-stepping and breakpoints make execution thousands of times slower. The program measures the wall-clock time (rdtsc, clock_gettime) across a short region; if it took far longer than it should, a human is stepping through it. This is effective against interactive debugging and hard to fully defeat.

Breakpoint and patch detection exploits how software breakpoints work (from the breakpoints topic): the program checksums its own code and notices the 0xCC byte a software breakpoint inserts, or scans for 0xCC directly. It may also detect inline hooks by checking whether its functions begin with an unexpected jmp.

Anti-attach and debugger disruption: install a SIGTRAP handler so that breakpoint traps are swallowed by the program instead of reaching the debugger; spawn a child that ptraces the parent so no debugger can attach (the slot is taken); or use exception-based control flow that confuses debuggers.

Anti-instrumentation and environment detection: look for the artifacts of analysis environments — VM indicators (specific hardware strings, timing of certain instructions), known analysis-tool process names, sandbox-specific file paths — and behave benignly when detected, a technique called sandbox evasion that is rampant in modern malware (it does nothing malicious while watched, then activates on a real target).

The analyst's response is systematic, and each bypass targets the mechanism directly:

Anti-debug technique	Bypass
Self-ptrace / TracerPid	Hook ptrace to return 0 (via LD_PRELOAD); patch the check; set a GDB catchpoint and alter the return value
Timing (rdtsc)	Patch out the check; hook the time source to return consistent values; use record/replay so timing is irrelevant
Code checksum / 0xCC scan	Use hardware breakpoints (leave no 0xCC); patch the comparison
SIGTRAP handler steal	Configure the debugger to not pass the signal to the program
VM/sandbox evasion	Harden the analysis VM to hide artifacts; patch the environment checks

Two general-purpose techniques dominate in practice. Patching — locating the detection check in the disassembly and overwriting the conditional jump so the "debugger detected" branch is never taken — is the universal solution once you find the check; it works against any detection because it removes the check itself. Interposition — using LD_PRELOAD to replace ptrace or the time source with a lying version — defeats whole classes of check without touching the binary. Tools like ScyllaHide and GDB plugins automate the common ones. The deeper lesson, again, is economic: anti-debugging increases the time and skill an analyst must invest, which can be worthwhile for slowing mass analysis, but a determined analyst defeats all of it, so it must never be the sole protection for anything that matters.

Examples

// A program bundling several common Linux anti-debug checks.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ptrace.h>
#include <x86intrin.h>   // __rdtsc

static int being_traced_ptrace(void) {
    // A debugger already holds the single trace slot -> this fails.
    return ptrace(PTRACE_TRACEME, 0, NULL, NULL) == -1;
}

static int being_traced_status(void) {
    FILE *f = fopen("/proc/self/status", "r");
    char line[256]; int pid = 0;
    while (f && fgets(line, sizeof line, f))
        if (sscanf(line, "TracerPid:\t%d", &pid) == 1) break;
    if (f) fclose(f);
    return pid != 0;                 // non-zero TracerPid => traced
}

static int being_stepped_timing(void) {
    unsigned long long t0 = __rdtsc();
    volatile int x = 0;
    for (int i = 0; i < 1000; i++) x += i;   // trivial work
    unsigned long long dt = __rdtsc() - t0;
    return dt > 100000ULL;           // far too slow => single-stepping
}

int main(void) {
    if (being_traced_ptrace() || being_traced_status() || being_stepped_timing()) {
        printf("analysis environment detected; behaving benignly\n");
        return 0;                    // real malware would hide its payload here
    }
    printf("payload would run now\n");
    return 0;
}

# Bypass 1: LD_PRELOAD a lying ptrace (defeats the self-ptrace check).
cat > fakeptrace.c <<'EOF'
#define _GNU_SOURCE
#include <sys/ptrace.h>
long ptrace(int req, ...) { return 0; }   // always "success", never -1
EOF
gcc -shared -fPIC fakeptrace.c -o fakeptrace.so
LD_PRELOAD=./fakeptrace.so gdb -q ./target   # the TRACEME check now passes

# Bypass 2: in GDB, catch the ptrace syscall and force its return to 0.
gdb -q ./target \
  -ex 'catch syscall ptrace' -ex run \
  -ex 'set $rax = 0' -ex 'continue'   # overwrite the return value each hit

# Bypass 3: patch the check out entirely (the universal method).
objdump -d -M intel ./target | grep -A3 being_traced_ptrace
#   find the `test eax,eax ; jne <detected>` and flip jne->jmp-over, or
#   NOP the call, with a hex editor / radare2:
r2 -w -q -c 's sym.main; /c jne; wa nop @ <addr>' ./target

  Patching: the universal bypass, conceptually
  ─────────────────────────────────────────────────────────────
   BEFORE (the check fires):          AFTER (patched):
     call  being_traced                 call  being_traced
     test  eax, eax                      test  eax, eax
     jne   .Ldetected   ──────────►      nop ; nop   (jne removed)
     <run payload>                       <run payload>   ; always reached

   Removing the conditional jump makes the "detected" branch unreachable,
   defeating the check regardless of how cleverly it decided it was traced.

Key Takeaways

• Anti-debugging is the set of techniques a program uses to detect, hinder, or evade debuggers and tracers; recognizing and neutralizing them is a routine malware-analysis skill.
• Detection families map to earlier mechanisms: self-ptrace and TracerPid (the trace-once rule), timing checks via rdtsc (stepping is thousands of times slower), and code checksums that spot the 0xCC of a software breakpoint.
• Disruption and evasion include stealing SIGTRAP, pre-ptraceing to occupy the trace slot, and sandbox evasion that behaves benignly while it detects analysis then activates on a real target.
• Bypasses target the mechanism: hardware breakpoints leave no 0xCC, LD_PRELOAD supplies a lying ptrace or time source, and record/replay makes timing checks irrelevant.
• Patching the detection check — overwriting the conditional jump so the "detected" branch is never taken — is the universal solution because it removes the check itself.
• Anti-debugging is an economic control that raises analyst cost; per Kerckhoffs it is never a guarantee and must not be the sole protection for anything important.

🧪 Practice

1. Compile the multi-check program above, confirm each check fires under GDB, then defeat the self-ptrace check with an LD_PRELOAD stub and verify the payload branch is reached.
2. Use a GDB syscall catchpoint on ptrace to force its return value and bypass the detection without any preloaded library.
3. Locate the TracerPid check in the disassembly and patch the conditional jump so the check is neutralized; verify with objdump that the branch is gone.
4. Interview-style: "A malware sample runs its payload when executed directly but does nothing under any debugger or in your sandbox VM. Describe your approach to analysing it anyway." Hint: consider combining check-neutralizing bypasses (patching, interposition) with an analysis method that does not rely on an attached debugger at all, such as recording execution or hardening the VM to hide its artifacts.

---

<a id="5-program-exploitation"></a>