5. Program Exploitation
Program exploitation is the study of how memory-safety defects in software become attacker-controlled behaviour, and — just as importantly — of why each generation of defenses was built and where it holds. This chapter explains the mechanics of the classic bug classes, the techniques that turn them into control of a program, the mitigations that raise the cost of each technique, and the secure-coding practices that prevent the bugs in the first place. The orientation throughout is defensive: you cannot write code that resists these attacks, triage a crash, or evaluate a hardening decision without understanding the mechanism it addresses.
<a id="51-memory-corruption-foundations"></a>
5.1 Memory Corruption Foundations
Every exploit in this chapter begins with a program writing or reading memory it should not, so this section establishes the memory model the attacker works against and the specific defect classes that break it, building directly on the process layout and C semantics from the program-analysis chapter.
Process Memory Layout: Text, Data, Heap, Stack
Theory
To understand how corrupting memory leads to control, you first have to know what lives where. When the kernel loads a program it builds a virtual address space — the flat, byte-addressable array from the program-analysis chapter — and populates it with regions that differ in two properties that matter enormously for security: what they contain and what permissions they carry. An exploit is, at bottom, a scheme to get attacker-influenced data into a region and then have the CPU treat it as something more privileged than data.
The regions, from low addresses to high:
| Region | Contents | Permissions | Grows | Security relevance |
|---|---|---|---|---|
| Text | machine code, read-only constants | r-x | fixed | the code; writable-text would allow trivial patching |
Data (.data) | initialized globals | rw- | fixed | writable function pointers, config |
| BSS | zero-initialized globals | rw- | fixed | large buffers |
| Heap | malloc/new allocations | rw- | upward | overflow into allocator metadata, UAF |
| (gap / mmap region) | shared libraries, large malloc, mmap'd files | varies | — | libc is here; ret2libc targets it |
| Stack | call frames: locals, saved rbp, return addresses | rw- | downward | return-address overwrite; the classic target |
Two design facts drive most of what follows. First, the stack and heap grow toward each other from opposite ends of the address space — a layout chosen so each can expand without a fixed partition, and the reason a buffer written past its end runs into adjacent data. Second, the W^X principle ("write XOR execute"): a well-configured page is either writable or executable, never both. This is the architectural basis of NX/DEP (covered in 5.4), and understanding it explains why early exploits that simply wrote code to the stack and jumped to it no longer work, which in turn is why ROP exists.
The security lens on this layout is simple to state: the attacker wants to move control flow to code of their choosing. The regions tell you both the targets (anything writable that the CPU later treats as an address — return addresses on the stack, function pointers in .data, GOT entries, heap metadata) and the constraints (code regions are read-only, so you cannot just overwrite a function's instructions). Every later technique is a way to navigate this geography.
Examples
A typical x86-64 process address space (not to scale)
───────────────────────────────────────────────────────────────
HIGH 0x7fff... ┌───────────────────────┐
│ argv, envp, auxv │
├───────────────────────┤
│ STACK rw- │ grows DOWN ↓
│ return addrs, locals │ (toward the heap)
│ ........... │
│ │
│ (unused gap) │
│ │
│ mmap / shared libs │ libc.so, ld.so r-x / rw-
│ (ASLR-randomized) │
│ │
│ ........... │
│ HEAP rw- │ grows UP ↑
├───────────────────────┤ (toward the stack)
│ BSS rw- │ zero-init globals
├───────────────────────┤
│ .data rw- │ init globals, writable ptrs
├───────────────────────┤
│ .text r-x │ code: READ-ONLY, EXECUTABLE
LOW 0x400000 └───────────────────────┘
W^X: no region is both writable AND executable. Hence an attacker
cannot write shellcode to the stack and run it (NX), and must reuse
existing executable code -> ret2libc / ROP (section 5.2).
# See the real, live layout of a running process.
cat /proc/self/maps | awk '{print $1, $2, $6}'
# 563...000-563...000 r-xp /bin/cat <- text, r-x
# 563...000-563...000 rw-p /bin/cat <- data, rw-
# 7f2...000-7f2...000 r-xp libc.so.6 <- library code
# 7fff...000-7fff...000 rw-p [stack] <- the stack, rw-, NO x
# ... [heap] <- the heap, rw-, NO x
# Confirm no region is writable AND executable (W^X) on a modern binary:
grep -E 'rwxp' /proc/self/maps || echo "no W+X regions (W^X holds)"
// Where does each object live? Print addresses to see the regions.
#include <stdio.h>
#include <stdlib.h>
int global_init = 1; // .data
int global_zero; // BSS
const char *msg = "in text"; // the string literal lives in .rodata/text
void func(void) {} // .text
int main(void) {
int local = 0; // stack
void *heap = malloc(16); // heap
printf("text (func) %p\n", (void*)func);
printf(".data (global) %p\n", (void*)&global_init);
printf("bss (zero) %p\n", (void*)&global_zero);
printf("heap (malloc) %p\n", heap);
printf("stack (local) %p\n", (void*)&local);
// Observe: text lowest, then data/bss, then heap, with stack highest.
free(heap);
return 0;
}
Key Takeaways
- A process address space is a set of regions differing in contents and permissions; exploitation is about getting attacker data into a region and having the CPU treat it as a more privileged thing (usually an address).
- Stack and heap grow toward each other from opposite ends, which is why an overflow runs into adjacent data structures.
- W^X ("write XOR execute") means no page is both writable and executable, which is the basis of NX/DEP and the reason code-reuse attacks (ret2libc, ROP) replaced code injection.
- The attacker's targets are writable values the CPU later uses as addresses: return addresses, function pointers, GOT entries, and heap metadata.
/proc/PID/mapsshows the real runtime layout, including ASLR-randomized library and stack bases.
🧪 Practice
- Compile and run the address-printing program above and order the regions by address; confirm the stack is highest and text lowest.
- Examine
/proc/self/mapsforcatand identify the permissions of the stack, the heap, and libc's code segment; confirm none arerwx.- Explain, using the grow-toward-each-other layout, why a very deep recursion and a very large heap allocation can in principle collide, and what the guard page does about it.
- Interview-style: "Why did the universal shift from code injection to code reuse happen, and which single architectural feature forced it?" Hint: name the page-permission invariant that makes a writable stack non-executable, and state what the attacker must therefore reuse instead of inject.
Unsafe C Functions and Memory Safety Gaps
Theory
C was designed for a world where the programmer is trusted and performance is paramount, so its standard library is full of functions that operate on buffers without knowing how big those buffers are. The function is handed a pointer and told to copy until it hits a terminator or a count the caller supplies; if the data is longer than the destination, the function writes past the end, because nothing in the language or the function stops it. This is not a bug in the functions — it is their specification. The bug is always at the call site, where a size assumption turns out to be wrong.
The root cause is that in C an array decays to a bare pointer, and a pointer carries no length. char buf[64] knows its size at compile time, but the moment you pass buf to strcpy, that knowledge is gone — the callee sees only an address. So the entire burden of bounds is on the programmer's discipline, and the historically unsafe functions are those that make that discipline easy to get wrong.
The canonical offenders, and why each is dangerous:
| Unsafe function | Problem | Safer replacement | Caveat on the replacement |
|---|---|---|---|
gets(buf) | reads a line with no bound at all | fgets(buf, n, stdin) | always bounded; gets was removed from C11 |
strcpy(d, s) | copies until s's NUL, ignoring d's size | strlcpy / snprintf | strncpy is not a safe drop-in (see below) |
strcat(d, s) | appends ignoring remaining space | strlcat / snprintf | track remaining space, not total |
sprintf(d, fmt, …) | formats with no output bound | snprintf(d, n, …) | check the return value for truncation |
scanf("%s", buf) | %s is unbounded | scanf("%63s", buf) | width must be buffer size minus one |
strncpy(d, s, n) | bounded, but does not NUL-terminate if s is ≥ n long | snprintf or manual NUL | a non-terminated string is its own bug |
The strncpy trap deserves emphasis because it is a classic "I used the safe one" mistake: strncpy writes at most n bytes, but if the source is n bytes or longer it copies exactly n bytes and leaves the destination without a terminating NUL. The next strlen or printf then reads past the end of the buffer — the bug moved, it did not disappear. Safety is a property of the whole call, not of choosing a function whose name contains an n.
The modern replacements split into two philosophies. The BSD strlcpy/strlcat family always NUL-terminates and returns the length that would have been written, so truncation is detectable. The snprintf approach formats into a sized buffer and is the most portable choice. On systems with Annex K, the _s functions (strcpy_s, etc.) add runtime constraint checks, though their adoption is uneven. The deeper point, which the secure-coding section develops, is that the function is the smallest part of the fix — the real fix is propagating buffer sizes alongside pointers and checking them.
Examples
// The same copy, written three ways: dangerous, subtly broken, and safe.
#include <stdio.h>
#include <string.h>
void dangerous(const char *input) {
char buf[16];
strcpy(buf, input); // if strlen(input) >= 16 -> overflow.
// No bound is consulted. Classic CWE-120.
}
void subtly_broken(const char *input) {
char buf[16];
strncpy(buf, input, sizeof buf); // bounded write, BUT:
// if input is >= 16 chars, buf has NO terminating NUL.
printf("%s\n", buf); // reads past buf -> out-of-bounds read.
}
void safe(const char *input) {
char buf[16];
// snprintf always NUL-terminates and never writes past the size.
int n = snprintf(buf, sizeof buf, "%s", input);
if (n < 0 || (size_t)n >= sizeof buf)
fprintf(stderr, "warning: input truncated\n"); // truncation detected
// buf is now a valid, bounded, NUL-terminated string regardless of input.
}
// Why a pointer loses the size: the information is simply gone.
#include <stdio.h>
void callee(char *p) {
// p is just an address. There is NO portable way to ask "how big is p?"
// sizeof(p) == 8 (the pointer size), NOT the buffer size.
printf("sizeof(p) = %zu\n", sizeof p); // 8, always
}
int main(void) {
char buf[64];
printf("sizeof(buf) = %zu\n", sizeof buf); // 64, only HERE at compile time
callee(buf); // the 64 does not travel with buf
return 0;
}
# The compiler and linker will warn about the worst offenders -- listen.
gcc -O2 -Wall -Wextra unsafe.c -o unsafe
# warning: 'gets' is deprecated / dangerous ...
# Fortified libc catches some overflows at runtime (see 5.4):
gcc -O2 -D_FORTIFY_SOURCE=2 unsafe.c -o unsafe # adds __*_chk bounds checks
Key Takeaways
- C's unsafe string/buffer functions operate without knowing the destination size; the defect is always the caller's wrong size assumption, not the function itself.
- An array decays to a bare pointer that carries no length, so
sizeofon a parameter gives the pointer size, not the buffer size — the size must be passed and checked explicitly. gets,strcpy,strcat,sprintf, and unboundedscanf("%s")are the classic offenders; preferfgets,snprintf, and thestrlcpy/strlcatfamily.strncpyis bounded but does not NUL-terminate when the source fills the buffer, turning an overflow into an out-of-bounds read — using ann-named function is not automatically safe.- The real fix is propagating and checking buffer sizes; choosing a safer function is only the smallest part of it.
🧪 Practice
- Rewrite a function that uses
strcpyandstrcatto build a path, making it safe withsnprintfand detecting truncation via the return value.- Demonstrate the
strncpynon-termination bug: copy a 20-character string into a 10-byte buffer withstrncpy, then show thatstrlen/printfreads past the buffer.- Compile a program containing
getswith-Walland document every warning; then replace it with a boundedfgetsand show the warning disappears.- Interview-style: "A junior developer says they fixed all the overflows by replacing every
strcpywithstrncpy. What do you check before agreeing?" Hint: consider whatstrncpydoes to the terminator when the source is exactly as long as, or longer than, the destination, and what the next read of that buffer does.
Undefined Behavior as a Security Problem
Theory
Most programmers think of undefined behaviour (UB) as "the program might crash." For security, that framing is dangerously incomplete. UB is a contract: the C and C++ standards define behaviour only for programs that obey certain rules, and for programs that break them the standard imposes no requirements at all. The compiler is therefore free to assume UB never happens — and modern optimizers aggressively exploit that assumption, sometimes deleting the very safety checks a programmer wrote. UB is not just a latent crash; it is a license the programmer unintentionally grants the compiler to transform the code in surprising, security-relevant ways.
The reason this matters more every year is that compilers keep getting better at optimization, and "better" means "more willing to assume UB is absent." Code that worked for a decade can break when the compiler is upgraded, because a new optimization finally takes advantage of a UB that was always there. This is why UB is a security problem and not merely a correctness nuisance.
The security-relevant forms of UB and what the optimizer does with each:
| Undefined behaviour | The optimizer's assumption | Security consequence |
|---|---|---|
| Signed integer overflow | "signed arithmetic never overflows" | if (x + 1 < x) overflow checks are deleted as "always false" |
| Null-pointer dereference | "a dereferenced pointer is non-null" | a null check after a dereference is removed, re-enabling the bug |
| Out-of-bounds access | "indices stay in bounds" | bounds checks the compiler deems redundant are dropped |
Use-after-free / dangling | "freed memory is not used" | reordering around the free can widen the window |
| Reading uninitialized memory | "the value is whatever is convenient" | can leak stale data; checks on it are unstable |
| Strict-aliasing violation | "pointers of different types do not alias" | stores can be reordered or elided, corrupting state |
The most infamous example is the deleted overflow check. A programmer writes if (ptr + len < ptr) return ERROR; intending to catch a pointer-arithmetic overflow. But signed overflow (and out-of-bounds pointer arithmetic) is UB, so the compiler reasons "ptr + len can never be less than ptr, because that would require overflow, which cannot happen" — and removes the check entirely. The guard the programmer wrote is simply gone from the binary, and the overflow it was meant to stop now proceeds unchecked. This exact pattern has produced real CVEs, including in the Linux kernel.
The defensive response has three parts. First, know the rules well enough to avoid writing UB — especially signed overflow, which is why security-critical size arithmetic uses unsigned types or checked-arithmetic builtins (developed in 5.5). Second, enable the sanitizers from the program-analysis chapter: UBSan catches these at runtime during testing, turning silent optimization into a loud diagnostic. Third, do not rely on post-hoc checks that depend on UB having already occurred — a check that can only be true if UB happened will be optimized away. The correct idiom tests operands before the dangerous operation, never the result after.
Examples
// The deleted overflow check: the guard vanishes at -O2.
#include <stdio.h>
#include <string.h>
int append_unsafe(char *buf, size_t bufsize, const char *data, int len) {
// INTENDED guard: catch a length that would overrun the buffer.
// BUG: `used + len` can overflow `int` (UB), so the compiler may assume
// the sum never wraps and conclude this comparison is "always false",
// deleting the check. Build at -O2 and inspect: the branch can disappear.
int used = (int)strlen(buf);
if (used + len < 0) // trying to detect overflow AFTER it occurs
return -1; // <- optimizer may remove this line
if (used + len >= (int)bufsize)
return -1;
memcpy(buf + used, data, len);
return 0;
}
// CORRECT: test BEFORE the operation, with types that cannot invoke UB.
int append_safe(char *buf, size_t bufsize, const char *data, size_t len) {
size_t used = strlen(buf);
if (len > bufsize - used - 1) // unsigned, operands checked beforehand
return -1; // (and bufsize - used - 1 is valid since
memcpy(buf + used, data, len); // used < bufsize is an invariant here)
buf[used + len] = '\0';
return 0;
}
// The null-check-after-deref removal.
#include <stdio.h>
int deref(int *p) {
int v = *p; // the compiler now ASSUMES p != NULL from here on
if (p == NULL) // ...so this check is "provably false" and is DELETED
return -1; // -> the intended guard is gone at -O2
return v;
}
// The fix is to check p for null BEFORE dereferencing it, not after.
# Catch all of these at test time with UBSan (from the program-analysis chapter).
gcc -fsanitize=undefined -fno-sanitize-recover=all -g ub.c -o ub
./ub
# ub.c:9: runtime error: signed integer overflow: 2147483647 + 1 ...
# Prove the optimizer removed the check: compare the generated code.
gcc -O0 -S -masm=intel ub.c -o ub_O0.s
gcc -O2 -S -masm=intel ub.c -o ub_O2.s
diff <(grep -c 'cmp\|test' ub_O0.s) <(grep -c 'cmp\|test' ub_O2.s) # fewer at -O2
# Make signed overflow defined (a mitigation, at some cost) if you cannot avoid it:
gcc -fwrapv ub.c -o ub # signed overflow now wraps predictably, checks survive
Key Takeaways
- Undefined behaviour is a contract: for programs that break the rules the standard requires nothing, and optimizers assume UB never happens — which can delete the programmer's own safety checks.
- The danger grows with compiler versions, because better optimization means more aggressive exploitation of the no-UB assumption; long-working code can break on a toolchain upgrade.
- The classic case is a post-hoc overflow check (
if (a + b < a)) that the compiler removes because signed overflow is UB, re-enabling the overflow it was meant to stop. - A null check placed after a dereference is likewise deleted, because the dereference let the compiler assume non-null.
- Defend by avoiding UB (unsigned or checked arithmetic for sizes), enabling UBSan in testing, and always testing operands before the dangerous operation rather than the result after;
-fwrapvmakes signed overflow defined when avoidance is impractical.
🧪 Practice
- Compile the
append_unsafe/derefexamples at-O0and-O2with-Sand confirm from the assembly that a check present at-O0is absent at-O2.- Trigger the signed-overflow diagnostic with
-fsanitize=undefinedand then show that-fwrapvchanges the program's behaviour by making the overflow defined.- Rewrite a size calculation that uses
intto usesize_twith a before-the-operation bound check, and explain why the new version cannot invoke the same UB.- Interview-style: "A code reviewer sees
if (ptr + n < ptr) return -1;guarding a copy and approves it as a correct overflow check. Why might that guard not exist in the shipped binary?" Hint: recall what the standard says about pointer arithmetic that goes out of bounds, and what the optimizer is therefore entitled to assume about the comparison's result.
Buffer Overflow Concept and Impact
Theory
A buffer overflow is the archetypal memory-corruption bug: a program writes more data into a fixed-size buffer than the buffer can hold, and the excess spills into whatever happens to sit next to it in memory. The reason this is catastrophic rather than merely wrong is the geography from the layout topic — on the stack, what sits next to a local buffer, at higher addresses, is the saved frame pointer and then the return address the CPU will load into rip when the function returns. Overflow far enough and you are no longer corrupting data; you are choosing where the program goes next.
Build the intuition from the write direction. A stack buffer is filled from its start (low address) upward (toward higher addresses). The saved return address is above the buffer. So a linear write that is too long marches up through the buffer, over the saved frame pointer, and into the return address. When the function executes ret, the CPU pops that now-attacker-controlled value into rip and jumps there. The ret instruction, as established in the program-analysis chapter, performs no validation — it trusts the stack absolutely.
The impact scales with how much control the attacker has, and it is useful to think of exploitation as a ladder of increasingly powerful primitives:
| Primitive | What the attacker controls | Typical outcome |
|---|---|---|
| Crash / DoS | enough to corrupt critical data | the program dies (still a vulnerability) |
| Control-flow hijack | the saved return address | execution jumps to an attacker-chosen address |
| Arbitrary code execution | the return address and injected or reused code | the attacker runs their own logic |
| Information disclosure | an over-read (e.g. Heartbleed) | secrets (keys, canaries, addresses) leak out |
It is worth stressing that overflow includes over-reads, not just over-writes. Heartbleed — one of the most damaging vulnerabilities ever — was a read overflow: a length field was trusted, and the server returned far more memory than it should have, leaking private keys. The same root cause (a trusted length exceeding a real buffer) produces disclosure instead of corruption.
The classic stack-smashing scenario, which the next section develops into full exploitation, is: a function copies untrusted input into a local buffer with an unbounded or mis-bounded copy; the input is longer than the buffer; the copy overwrites the saved return address; and on return, control transfers to an address the attacker placed in the overflowing bytes. Every mitigation in 5.4 — canaries, NX, ASLR, CFI — is a direct response to some step of this scenario, which is why understanding the scenario is the prerequisite for understanding the defenses.
Examples
Stack layout during an overflow of char buf[8]
───────────────────────────────────────────────────────────────
BEFORE copy: AFTER copying 24 'A' bytes:
higher addr higher addr
┌───────────────────┐ ┌───────────────────┐
│ return address │ 0x...5589 │ 0x4141414141414141│ <- OVERWRITTEN
├───────────────────┤ ├───────────────────┤ = new rip on ret
│ saved rbp │ │ 0x4141414141414141│ <- OVERWRITTEN
├───────────────────┤ ├───────────────────┤
│ buf[8] (garbage) │ │ "AAAAAAAA" │
└───────────────────┘ └───────────────────┘
lower addr lower addr
write grows UPWARD ───────────────────────────────────────►
On `ret`, the CPU pops 0x4141414141414141 into rip and jumps there.
The attacker chose that value by controlling the overflowing bytes.
// A deliberately vulnerable function (compile with mitigations OFF to study).
// gcc -g -O0 -fno-stack-protector -z execstack -no-pie vuln.c -o vuln
#include <stdio.h>
#include <string.h>
void vulnerable(const char *input) {
char buf[8]; // 8-byte buffer...
strcpy(buf, input); // ...copied into with NO bound.
// If `input` is longer than 8 bytes (plus alignment/saved rbp), the
// copy reaches and overwrites the saved return address above buf.
printf("buf = %s\n", buf);
}
int main(int argc, char **argv) {
if (argc > 1) vulnerable(argv[1]);
return 0; // with a long argv[1], the return from vulnerable() goes astray
}
# Observe the crash and the overwritten rip under a debugger (study only).
gcc -g -O0 -fno-stack-protector -no-pie vuln.c -o vuln
gdb -q ./vuln \
-ex 'run AAAAAAAAAAAAAAAAAAAAAAAA' \
-ex 'info registers rip' -ex 'quit'
# Program received signal SIGSEGV
# rip = 0x4141414141414141 <- 'A's (0x41) landed in the instruction pointer
# A read overflow leaks adjacent memory (the Heartbleed shape, in miniature):
# a trusted length N larger than the real buffer causes N bytes to be
# returned, exposing whatever follows the buffer -- keys, tokens, pointers.
Key Takeaways
- A buffer overflow writes past a buffer's end into adjacent memory; on the stack, that adjacent memory includes the saved return address, turning data corruption into control-flow control.
- Stack buffers fill upward toward the saved return address, and
retloads that address intoripwithout any validation, which is the mechanical core of stack smashing. - Exploitation is a ladder of primitives: crash, control-flow hijack, arbitrary code execution, and information disclosure.
- Overflows include over-reads, not only over-writes; Heartbleed was a read overflow driven by a trusted length, leaking private keys.
- Every mitigation in this chapter targets a specific step of the classic stack-smash scenario, so the scenario is the key to understanding the defenses.
🧪 Practice
- Compile the vulnerable program with mitigations disabled and, under GDB, determine the exact number of input bytes needed before
ripis overwritten; explain how buffer size, alignment, and the saved frame pointer contribute to that offset.- Draw the stack frame of
vulnerableand annotate which input byte ranges land in the buffer, the savedrbp, and the return address.- Write a bounded version of
vulnerableusingsnprintfand show under GDB that the same long input no longer reaches the return address.- Interview-style: "Explain how a bug that only reads too much memory, never writing, can be as severe as one that executes code." Hint: think about what sits in server memory next to a buffer — session keys, other users' data, pointers that defeat ASLR — and what an attacker does once they can read it.
Heap Overflows and Allocator Metadata
Theory
The stack is not the only place buffers live. Long-lived and variable-sized data goes on the heap, via malloc/free (or new/delete). A heap overflow is the same root cause — writing past a buffer — but the consequences are different and often subtler, because the heap's layout is dynamic and, crucially, the allocator stores its own bookkeeping data inline, right next to your allocations. Corrupting that bookkeeping is what turns a heap overflow into control.
To see why, you need the allocator's basic design. malloc does not hand out bare memory; it manages a pool divided into chunks. Each chunk has a small header — in glibc's allocator, a prev_size and a size field (with the low bits of size repurposed as flags) — immediately before the region it returns to you. Freed chunks are threaded onto free lists ("bins") using pointers that the allocator stores inside the now-unused user area of the freed chunk. So the heap interleaves attacker-writable data with allocator-critical metadata and pointers.
Two adjacent glibc heap chunks
─────────────────────────────────────────────
chunk A chunk B
┌──────────┬──────────┐ ┌──────────┬──────────┐
│prev_size │ size|flags│ │prev_size │ size|flags│
├──────────┴──────────┤ ├──────────┴──────────┤
│ user data (A) │─ ─ ─►│ user data (B) │
│ returned by malloc │ ^ │ │
└─────────────────────┘ │ └─────────────────────┘
overflow of A's user data writes INTO
chunk B's header (size/flags) and data
An overflow of chunk A's user data runs directly into chunk B's header. Historically, attackers exploited this by forging the size field and the free-list pointers so that a later free or malloc performed an attacker-chosen write — the "unlink" technique wrote an arbitrary value to an arbitrary address by abusing the doubly-linked-list pointer updates. Modern glibc hardens against the classic variants with integrity checks (size sanity, unlink consistency checks, pointer alignment checks, and, recently, safe-linking, which XORs free-list pointers with a per-heap secret). The result is an ongoing arms race: each class of corruption prompts a new check, and researchers find new metadata to abuse ("House of" techniques are named families of these).
Heap exploitation is harder than stack exploitation for two reasons, and both inform the defense. First, the attacker often does not control adjacent layout directly — they must perform heap grooming (also called heap feng shui): making a sequence of allocations and frees that arranges the target object next to the overflowable buffer. Second, the useful targets are application objects, not just metadata: a C++ object with a vtable pointer, a function pointer in a struct, or a length field controlling a later copy. Overwriting a vtable pointer so a virtual call jumps to attacker-chosen code is one of the most common modern heap-exploitation primitives, because it sidesteps allocator hardening entirely.
For defenders, the practical signals are: heap overflows corrupt data or crash in the allocator (inside malloc/free) far from the overflowing code, making them a prime example of the "crash site is not the bug site" principle from crash triage; AddressSanitizer with its redzones around every chunk is the most effective detector; and hardened allocators (glibc's checks, hardened_malloc, Windows' LFH mitigations) raise the cost of metadata attacks specifically.
Examples
// A heap overflow that corrupts an adjacent object's function pointer.
// (Study with: gcc -g -O0 -fsanitize=address heap.c -o heap)
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
struct worker {
char name[16];
void (*action)(void); // a function pointer -- the juicy target
};
void normal_action(void) { puts("normal action"); }
void danger_action(void) { puts("attacker-selected action ran"); }
int main(void) {
char *input_buf = malloc(16); // chunk A: an overflowable buffer
struct worker *w = malloc(sizeof *w); // chunk B: likely ADJACENT to A
w->action = normal_action;
// Overflow input_buf: enough bytes to reach chunk B's `action` pointer.
// In a real attack the overwrite value is the address of danger_action.
memset(input_buf, 'A', 64); // ASan: "heap-buffer-overflow" here
w->action(); // if action was overwritten, control is hijacked
free(w); free(input_buf);
return 0;
}
# ASan pinpoints a heap overflow at the write, with both stacks.
gcc -g -O0 -fsanitize=address heap.c -o heap
./heap
# ==ERROR== heap-buffer-overflow ... WRITE of size 64
# #0 main heap.c:23 <- the overflowing memset
# located 0 bytes after a 16-byte region allocated here:
# #0 malloc ... #1 main heap.c:16 <- the under-sized chunk A
# Without ASan, the symptom often appears LATER, inside the allocator:
gcc -g -O0 heap.c -o heap_noasan
./heap_noasan # may crash in free()/malloc() internals -- far from line 23
# Inspect live heap metadata (glibc) with pwndbg/GEF (from program-analysis):
gdb -q ./heap_noasan -ex 'break main' -ex run -ex 'heap' -ex 'bins'
Why heap bugs are hard: grooming arranges the victim
─────────────────────────────────────────────────────────────
1. Spray: allocate many objects to make the heap predictable.
2. Free a hole where the victim object will land.
3. Allocate the overflowable buffer so it precedes the victim.
4. Allocate/position the victim (e.g. a vtable-bearing object).
5. Trigger the overflow -> precise overwrite of the victim's pointer.
Each step is probabilistic; grooming makes the layout deterministic.
Key Takeaways
- Heap overflows share the root cause of stack overflows but corrupt a dynamic layout, and the allocator stores metadata (chunk headers, free-list pointers) inline next to user data.
- Classic heap exploits forged chunk metadata so that
free/mallocperformed an attacker-chosen write (the "unlink" primitive); modern allocators add integrity checks and safe-linking, driving an ongoing arms race. - The most durable heap targets are application objects — C++ vtable pointers, function pointers in structs, length fields — because overwriting them bypasses allocator hardening.
- Exploitation usually requires heap grooming (feng shui) to place the victim object adjacent to the overflowable buffer, which makes heap bugs harder than stack bugs.
- Heap overflows typically crash inside the allocator far from the bug; AddressSanitizer's per-chunk redzones are the most effective detector, and hardened allocators specifically raise the cost of metadata attacks.
🧪 Practice
- Build the heap example with AddressSanitizer and confirm it reports the overflow at the
memsetwith both the write and allocation stacks; then build without ASan and observe where it crashes instead.- Use pwndbg/GEF's
heapandbinscommands to inspect chunk headers before and after freeing an allocation, and identify thesizefield and its flag bits.- Explain, with a drawing, why overwriting a C++ object's vtable pointer leads to control-flow hijack on the next virtual method call.
- Interview-style: "A fuzzer finds a crash deep inside
malloc, but the code under test does not callmallocnear the crashing line. How do you reason about the root cause?" Hint: recall the crash-site-versus-bug-site principle and what a heap overflow does to metadata that only the next allocator operation reads.
Use-After-Free and Double Free
Theory
The overflows so far are spatial errors — accessing outside a buffer's bounds. Use-after-free and double-free are temporal errors — accessing memory outside its valid lifetime. The memory is the right size and you are within its bounds; the problem is when you touch it. These bugs dominate modern exploitation because the spatial defenses are now strong, and because they arise naturally from the hardest problem in manual memory management: knowing when an object is truly dead.
A use-after-free (UAF) happens when a pointer is used after the memory it points to has been freed. The pointer still holds the old address — it has become a dangling pointer — but that memory now belongs to the allocator and may have been handed out to a different allocation. Reading through the dangling pointer returns whatever the new owner wrote; writing through it corrupts the new owner's data. The analogy is a hotel room key that still opens a door after you have checked out: the lock works, but someone else lives there now, and either you see their belongings or you disturb them.
The exploitation power of UAF comes from type confusion through reallocation. The attacker's recipe is: free an object, then quickly allocate a different object of the same size so the allocator reuses the freed chunk, so that the dangling pointer now overlays the attacker's new object. If the freed object had a vtable or function pointer, the attacker now controls it through the contents of the replacement allocation — and the program calls it. This "free, reallocate with controlled data, trigger use" pattern is the backbone of countless browser and kernel exploits.
A double-free frees the same pointer twice. On its own that corrupts the allocator's free lists (the chunk appears on a bin twice), and historically that could be leveraged into returning the same chunk from two independent malloc calls — giving two "owners" of one region, which is a powerful primitive for engineering UAF-style type confusion. Modern allocators detect the simplest double-frees (glibc's tcache and fastbin checks reject an immediate re-free), but subtler sequences still matter.
The root cause for all of these is ownership ambiguity: when it is unclear which code is responsible for freeing an object and when, the same object gets freed twice, or freed while another reference still exists. The defenses therefore cluster around making lifetimes explicit:
- Null the pointer after freeing (
free(p); p = NULL;) so a later use or re-free is a null-deref (a crash) rather than a UAF (exploitable). This is cheap and catches many cases. - Clear ownership discipline: one owner frees; everyone else holds non-owning references with a known-shorter lifetime.
- Reference counting / smart pointers (C++
shared_ptr/unique_ptr) make ownership a type-level property. - Memory-safe languages (Rust's borrow checker, garbage-collected runtimes) eliminate the class by construction — the subject of a later topic.
- Detection: AddressSanitizer's quarantine delays chunk reuse and traps dangling accesses; the kernel and allocators add use-after-free mitigations (e.g. heap layout randomization, delayed free).
Examples
// Use-after-free leading to type confusion via reallocation.
// Study with: gcc -g -O0 -fsanitize=address uaf.c -o uaf
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
struct session { void (*on_close)(void); char user[24]; };
void legit_close(void) { puts("closing session"); }
int main(void) {
struct session *s = malloc(sizeof *s);
s->on_close = legit_close;
free(s); // s is now DANGLING -- memory returned to allocator
// (A real bug: some other code path keeps using `s` after this free.)
// Attacker-controlled allocation of the SAME size reuses the chunk:
char *reuse = malloc(sizeof *s);
memset(reuse, 0x42, sizeof *s); // fills the old session's memory
s->on_close(); // UAF: calls through a pointer now full of 0x42
// ASan: "heap-use-after-free" READ at this line
free(reuse);
return 0;
}
// The cheap, high-value habit: null after free.
void good_pattern(void) {
char *p = malloc(32);
/* ... use p ... */
free(p);
p = NULL; // a later *p or free(p) now faults predictably,
// converting a silent UAF/double-free into an
// immediate, non-exploitable null dereference.
}
// Double free, and why modern glibc rejects the obvious case:
void double_free(void) {
char *p = malloc(16);
free(p);
free(p); // glibc: "double free or corruption (fasttop)"
// -- the tcache/fastbin check detects immediate re-free
}
# ASan catches both temporal bugs with allocation + free + use stacks.
gcc -g -O0 -fsanitize=address uaf.c -o uaf
./uaf
# ==ERROR== heap-use-after-free READ ...
# #0 main uaf.c:19 <- the use
# freed by thread T0 here:
# #0 free ... #1 main uaf.c:12 <- WHERE it was freed
# previously allocated here:
# #0 malloc ... #1 main uaf.c:9 <- WHERE it was allocated
Key Takeaways
- Use-after-free and double-free are temporal errors: the access is in-bounds but outside the memory's valid lifetime, unlike spatial overflows.
- A freed pointer becomes dangling; the memory may be reallocated to a different object, so reading leaks the new owner's data and writing corrupts it.
- UAF is exploited by freeing an object, reallocating a same-sized object with attacker-controlled contents, and triggering a use — classic type confusion that hijacks vtables or function pointers.
- Double-free corrupts allocator free lists and historically yielded two owners of one chunk; modern allocators detect the simplest cases.
- The root cause is ownership ambiguity; defenses make lifetimes explicit — null-after-free, single-owner discipline, smart pointers, memory-safe languages — and ASan's quarantine is the primary detector.
🧪 Practice
- Build the UAF example with AddressSanitizer and identify, from the report, the allocation site, the free site, and the use site; explain how those three locations together reveal the bug.
- Add
p = NULL;after eachfreein a small program with a double-free and show that the symptom changes from heap corruption to a null-pointer dereference.- Rewrite a C snippet that shares a raw pointer between two owners using C++
std::shared_ptr, and explain how reference counting removes the double-free risk.- Interview-style: "Why have use-after-free bugs become more prominent in exploitation than stack overflows over the last decade?" Hint: consider which bug class the mature mitigations (canaries, NX, ASLR, FORTIFY) target most directly, and which class those mitigations barely touch.
Integer Overflow and Type Confusion
Theory
Not every memory-corruption bug starts as a memory operation. Many start as arithmetic — a calculation that produces a value the programmer did not anticipate, which is then used as a size, an index, or a length. Integer bugs are insidious precisely because the dangerous operation (the overflow) looks completely innocent and the corruption happens later, somewhere that looks correct.
Integers in C are fixed-width, so arithmetic is modular: exceeding the maximum wraps around. For a 32-bit unsigned value, 0xFFFFFFFF + 1 == 0. The security problem is a calculation like malloc(count * size): if count and size are attacker-influenced, their product can wrap to a small number, so malloc returns a tiny buffer — and then a loop writes count elements into it, overflowing massively. The allocation "succeeded," every individual element write looks in-bounds relative to the intended count, yet the buffer is far too small. This exact pattern (allocation-size overflow) is one of the most common integer-bug-to-heap-overflow pipelines.
The forms worth distinguishing:
| Form | Example | Consequence |
|---|---|---|
| Unsigned wraparound | len = a - b where b > a | huge positive length; over-read/over-write |
| Signed overflow | INT_MAX + 1 | UB (per the UB topic); checks deleted |
| Multiplication overflow | n * size wraps small | under-sized allocation, then overflow |
| Signed/unsigned confusion | signed len compared < then used as size_t | negative passes the check, becomes enormous |
| Truncation | 64-bit length stored into 32-bit field | high bits lost; size mismatch |
The signed/unsigned confusion case ties back to the assembly chapter's jl versus jb distinction. A length declared int is checked if (len < MAX) with a signed comparison; an attacker supplies a negative value, which is "less than MAX," so it passes. The value is then passed to memcpy, whose size parameter is size_t (unsigned), and the negative number sign-extends into an astronomically large size. The check and the use disagreed about the type, and the attacker exploited the gap.
Type confusion is the object-level analogue: the program treats a block of memory as one type when it is actually another. This happens through bad casts, through unions used incorrectly, through deserialization that trusts a type tag, and — as seen above — through use-after-free reallocation. The danger is that different types have different layouts: if the program thinks offset 8 is an integer but the object actually has a function pointer there, writing the "integer" sets a pointer the program will later call. Type confusion is especially prevalent in language runtimes and browsers (JavaScript engines), where a mismatch between the engine's assumed type and the actual object is a direct route to control.
The defenses are about checked arithmetic and type safety: validate operands before arithmetic (the before-not-after rule again), use compiler builtins like __builtin_mul_overflow that report wraparound, prefer size_t for sizes and avoid mixing signedness, enable UBSan and -ftrapv/-fsanitize=integer, and in C++ prefer static_cast/dynamic_cast discipline over C-style casts. The secure-coding section develops safe integer arithmetic in detail.
Examples
// Allocation-size multiplication overflow: the allocation "succeeds" tiny.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
void *make_array_unsafe(size_t count, size_t size) {
// BUG: count * size can wrap. With count = 0x100000001, size = 16 on a
// 32-bit size_t this wraps to 16 -> a 16-byte buffer for "billions" of
// elements. Even on 64-bit, large attacker values can wrap.
void *p = malloc(count * size); // under-sized on overflow
return p; // caller then writes `count` elements
}
void *make_array_safe(size_t count, size_t size) {
size_t bytes;
// Checked multiply: returns true on overflow, so we refuse safely.
if (__builtin_mul_overflow(count, size, &bytes)) {
fprintf(stderr, "allocation size overflow\n");
return NULL; // refuse rather than under-allocate
}
return malloc(bytes); // bytes is correct or we already bailed
// (calloc(count, size) does exactly this overflow check internally.)
}
// Signed/unsigned confusion: a negative length passes a signed check.
#include <string.h>
#include <stdio.h>
void copy_record(char *dst, const char *src, int len) { // len is SIGNED
if (len > 1024) return; // signed compare: a NEGATIVE len passes
// len is now implicitly converted to size_t for memcpy: a huge value.
memcpy(dst, src, len); // len = -1 -> 0xFFFFFFFFFFFFFFFF bytes
}
// Fix: use size_t for len, or check `if (len < 0 || len > 1024) return;`
# Catch integer UB and wraparound at test time.
gcc -fsanitize=undefined,integer -g int.c -o int # clang: -fsanitize=integer
./int
# runtime error: unsigned integer overflow: 4294967295 * 16 ...
# calloc performs the multiply-overflow check for you; prefer it for arrays:
# p = calloc(count, size); // returns NULL on overflow instead of wrapping
Key Takeaways
- Integer bugs begin as arithmetic whose result wraps, truncates, or changes sign, and only become memory corruption when the bad value is used as a size, index, or length later.
- The canonical pipeline is multiplication overflow in an allocation size:
malloc(count * size)wraps small, the allocation succeeds tiny, and a subsequent loop overflows it. - Signed/unsigned confusion lets a negative length pass a signed
<check and then sign-extend into an enormoussize_tat amemcpy, connecting directly to thejl-versus-jbdistinction from assembly. - Type confusion treats memory as the wrong type; because layouts differ, a write intended for a data field can land on a function or vtable pointer, which is central to browser and runtime exploitation.
- Defend with checked arithmetic (
__builtin_mul_overflow,calloc),size_tfor sizes, consistent signedness, UBSan/integer sanitizers, and disciplined casts.
🧪 Practice
- Write a function that allocates
count * sizebytes unsafely, demonstrate wraparound with values that overflowsize_t, then fix it with__builtin_mul_overfloworcalloc.- Reproduce the signed/unsigned confusion by passing a negative
intlength to a function that forwards it tomemcpy, observe the result under ASan, and fix the type.- Explain, with a struct/union example, how reinterpreting one type as another can turn a benign integer write into a function-pointer overwrite.
- Interview-style: "A length field is read from the network as a 32-bit value, checked against a maximum, and passed to a copy. Name two integer pitfalls that could still make this exploitable despite the check." Hint: consider the signedness of the variable holding the length and what happens when a 64-bit size parameter receives a sign-extended negative, and separately what truncation from a wider field could hide.
Off-by-One and Boundary Errors
Theory
The off-by-one error is the smallest possible overflow — a single byte written one position too far — and its prominence in security is a lesson that no overflow is too small to matter. These bugs come from the countless places where a boundary is computed, and the near-universal source is confusion between a count and an index, or forgetting that C strings need an extra byte for the terminating NUL.
The classic C off-by-one is the NUL terminator. A buffer char buf[N] holds an N-character C string only if you count the terminator — it actually holds N-1 characters plus the NUL. Code that writes buf[strlen(src)] = '\0' after copying N characters, or that allocates strlen(s) bytes instead of strlen(s) + 1, writes exactly one byte past the end. The loop-boundary analogue is for (i = 0; i <= n; i++) where < n was meant: the array is touched at index n, one past its last valid index n-1.
Why does a single byte lead to compromise? Because of what that byte lands on. Two cases are famous:
- The null-byte overflow / "off-by-one NUL". If the single overflowing byte is a
\0and it lands on the least-significant byte of an adjacent pointer or a heap chunk's size field, it can zero out that byte — shrinking a size, or truncating a pointer to point somewhere useful. On a little-endian machine the LSB is at the lowest address, which is exactly where a forward overflow writes first. Glibc heap exploitation has a whole technique ("poison null byte") built on a single off-by-one NUL corrupting a chunk size. - The saved-register or frame-pointer byte. On the stack, one byte past a buffer can land on the least-significant byte of the saved frame pointer. Overwriting it moves the frame pointer by up to 255 bytes, which on function return can shift the stack so that a subsequent
leave; retuses attacker-influenced data — the "off-by-one to saved RBP" technique.
The broader category is boundary errors: any mistake at the edge of a range. Fence-post errors (counting the posts versus the gaps in a fence — hence the name), inclusive-versus-exclusive bound mismatches, and unsigned-underflow at zero (for (i = n; i >= 0; i--) with unsigned i never terminates and wraps). These are so common that they merit their own review discipline: whenever code computes or compares a boundary, check the edge cases explicitly — empty input, exactly-full buffer, maximum index.
The defenses are small habits with large payoff: write sizeof(buf) and sizeof(buf) - 1 deliberately and consciously; prefer < length and treat <= as a smell; allocate strlen(s) + 1; use functions that take the full buffer size and terminate for you (snprintf); and let ASan — whose redzones catch a one-byte overflow just as reliably as a hundred-byte one — run on your tests, since off-by-ones are nearly invisible to the eye but loud to the sanitizer.
Examples
// Three off-by-one classics.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
char *dup_broken(const char *s) {
char *p = malloc(strlen(s)); // BUG: forgot +1 for the NUL terminator
strcpy(p, s); // writes strlen(s)+1 bytes -> 1 past end
return p; // ASan: heap-buffer-overflow (1 byte)
}
void loop_broken(int *a, int n) {
for (int i = 0; i <= n; i++) // BUG: <= touches a[n], one past the end
a[i] = 0; // valid indices are 0..n-1
}
void nul_off_by_one(const char *src) {
char buf[8];
size_t len = strlen(src); // suppose len == 8 exactly
memcpy(buf, src, len); // fills buf[0..7] -- buf is now FULL
buf[len] = '\0'; // BUG: buf[8] is one past the end
// That single '\0' can zero the low byte of an adjacent value.
}
// The same three, corrected.
char *dup_ok(const char *s) {
char *p = malloc(strlen(s) + 1); // +1 for the terminator
if (p) strcpy(p, s);
return p;
}
void loop_ok(int *a, int n) {
for (int i = 0; i < n; i++) // strict < : indices 0..n-1
a[i] = 0;
}
void nul_ok(const char *src) {
char buf[8];
snprintf(buf, sizeof buf, "%s", src); // writes at most 7 chars + NUL,
// always within bounds, always
// terminated -- no edge arithmetic.
}
Why one byte matters: an off-by-one NUL on a little-endian size field
─────────────────────────────────────────────────────────────
adjacent heap chunk size field (little-endian):
before: 21 00 00 00 00 00 00 00 (size 0x21)
a single '\0' written one byte too far lands on the LSB:
after: 00 00 00 00 00 00 00 00 (size now 0x00) <- corrupted
The allocator's later view of chunk boundaries is now wrong, which
is the seed of the glibc "poison null byte" technique.
# ASan catches single-byte overflows exactly like large ones.
gcc -g -O0 -fsanitize=address offbyone.c -o offbyone
./offbyone
# ==ERROR== heap-buffer-overflow WRITE of size 1
# located 0 bytes after a region of size N ... <- the one byte, pinpointed
Key Takeaways
- An off-by-one is a one-byte overflow, and its prominence teaches that no overflow is too small to be exploitable — what matters is what the byte lands on.
- The dominant sources are count-versus-index confusion, inclusive-versus-exclusive bounds (
<=versus<), and forgetting the+1for a C string's NUL terminator. - A single off-by-one NUL on a little-endian size field or pointer LSB can shrink a size or truncate a pointer, which glibc's "poison null byte" technique weaponizes; on the stack it can corrupt the saved frame pointer's low byte.
- Boundary errors generalize to fence-post mistakes and unsigned underflow at zero, so edge cases (empty, exactly-full, maximum index) deserve explicit review.
- Defend with deliberate
sizeof/sizeof - 1usage, strict<bounds,strlen + 1allocations, size-taking terminating functions likesnprintf, and ASan, which catches one-byte overflows as reliably as large ones.
🧪 Practice
- Fix the three broken functions above and verify with AddressSanitizer that the one-byte overflow in each is gone.
- Write a loop with an unsigned loop counter counting down to zero, observe that it never terminates, and explain the wraparound at the boundary.
- Allocate a buffer of exactly
strlen(s)bytes, copysinto it, and show with ASan where and why the single-byte overflow occurs; then correct the allocation size.- Interview-style: "Explain how writing a single
\0one byte past a heap buffer could help an attacker, given it only zeroes one byte." Hint: think about which byte of a multi-byte little-endian size or pointer value sits at the lowest address, and what shrinking or truncating that value does to the allocator's or program's later view of memory.
<a id="52-stack-overflow-exploitation"></a>
5.2 Stack Overflow Exploitation
This section turns the stack buffer overflow from a crash into controlled execution, developing the techniques in the order they were historically defeated and rebuilt: from overwriting a return address, through code injection, to the code-reuse methods (ret2libc, ROP) that modern defenses forced into existence. The framing is defensive and conceptual — understanding these mechanisms is what lets you reason about why each mitigation in 5.4 exists.
Overwriting Saved Return Addresses
Theory
The previous section established that an overflow can reach the saved return address. This topic is about doing it precisely: understanding the exact stack layout well enough to overwrite the return address with a specific value and nothing important by accident. Precision is the whole game — a sloppy overwrite crashes, a precise one redirects.
Recall the frame layout from the program-analysis chapter. When a function runs, the stack (from lower to higher addresses) holds: the function's local buffers, then the saved rbp (if a frame pointer is used), then the return address placed there by the caller's call. An overflow of a local buffer writes upward through this structure. To overwrite the return address, the attacker must send exactly enough bytes to fill the buffer and any space between it and the saved rbp, overwrite the 8-byte saved rbp, and then place the desired 8 bytes where the return address lives.
That "exactly enough" quantity is the offset, and finding it is the first task of any stack exploit. Because the compiler may add alignment padding and may place the buffer at an unexpected distance from the saved registers, the offset is rarely just the buffer's declared size. Two standard methods find it:
- The cyclic pattern (De Bruijn sequence). Send a long, non-repeating pattern like
Aa0Aa1Aa2...where every 4- or 8-byte window is unique. When the program crashes, the value inrip(or on the stack where the return address was) is a slice of the pattern; its position in the pattern is the offset. Tools (pwntools'cyclic/cyclic_find,pattern_create/pattern_offset) automate this and it is the standard approach. - Manual counting under a debugger, incrementing the input length until
ripis first affected.
Little-endianness matters here, exactly as in the program-analysis chapter: the address you want in rip must be written least-significant-byte first. A target address 0x401156 is placed as the bytes 56 11 40 00 00 00 00 00. Getting this byte order wrong is the most common beginner mistake, which is why exploit libraries provide a p64() helper to pack an integer into the correct 8-byte little-endian form.
The security value of understanding this precisely is twofold. It explains why a stack canary (a random value placed between the buffer and the saved return address) is so effective: a linear overflow cannot reach the return address without first overwriting the canary, and the function checks the canary before returning. And it explains why exploits are brittle — they depend on exact offsets that change with compiler, optimization level, and sometimes platform, which is relevant when assessing whether a given crash is realistically exploitable.
Examples
The overwrite target, measured
───────────────────────────────────────────────────────────────
higher addresses
┌────────────────────┐
│ return address │ <- bytes [offset .. offset+8) = GOAL
├────────────────────┤
│ saved rbp │ <- bytes [offset-8 .. offset) = filler
├────────────────────┤
│ (alignment pad) │ <- compiler-dependent; part of the offset
├────────────────────┤
│ char buf[64] │ <- bytes [0 .. 64) = filler
└────────────────────┘
lower addresses
payload = filler (offset bytes) || p64(target_address)
The ONLY precise way to learn `offset` is to measure it.
# Finding the offset with a cyclic pattern (conceptual, pwntools).
from pwn import cyclic, cyclic_find, p64
# 1. Send a unique pattern long enough to reach past the return address.
pattern = cyclic(200) # Aa0Aa1Aa2... no 8-byte window repeats
# (feed `pattern` to the target; it crashes; read the faulting value)
# 2. Suppose the debugger shows rsp/rip contained these 8 bytes at crash:
crashed_with = 0x6161616161616166 # example value observed in the debugger
offset = cyclic_find(crashed_with) # position of that slice in the pattern
print("offset to return address:", offset)
# 3. Build a precise payload: filler up to the return address, then target.
TARGET = 0x401156 # address to redirect to (study target)
payload = b"A" * offset + p64(TARGET) # p64 => little-endian 8-byte packing
# p64(0x401156) == b"\x56\x11\x40\x00\x00\x00\x00\x00" (LSB first)
# Measuring the offset under GDB with a cyclic pattern (study harness).
gdb -q ./vuln \
-ex 'run $(python3 -c "import sys; sys.stdout.buffer.write(b\"Aa0Aa1Aa2Aa3Aa4Aa5Aa6Aa7Aa8Aa9\")")' \
-ex 'info registers rip rsp' -ex 'quit'
# Inspect which 8 bytes of the pattern landed where the return address was;
# the pattern's uniqueness makes the position -- the offset -- unambiguous.
Key Takeaways
- Overwriting a return address precisely requires knowing the exact offset from the buffer's start to the saved return address, which is rarely just the buffer's declared size because of alignment and layout.
- The standard way to find the offset is a cyclic (De Bruijn) pattern: a unique-window sequence whose crashed slice reveals its own position, automated by
cyclic/pattern_offsettools. - The target address must be written least-significant-byte-first (little-endian);
p64()-style packing exists precisely to avoid byte-order mistakes. - Understanding the precise layout explains why a stack canary between the buffer and the return address is so effective — a linear overflow must cross it first.
- Stack exploits are brittle because offsets depend on compiler, optimization, and platform, which matters when judging whether a crash is realistically exploitable.
🧪 Practice
- Using a cyclic pattern under GDB, determine the exact offset to the return address for a vulnerable program compiled at
-O0and again at-O2, and explain any difference.- Hand-pack the address
0x00000000004011b6into its 8-byte little-endian byte sequence, then verify againstp64().- Explain precisely why a stack canary defeats the overwrite described here, referring to the order in which a linear write reaches the canary versus the return address.
- Interview-style: "Why is the offset to the return address often not equal to the declared size of the overflowed buffer?" Hint: consider stack alignment requirements, the saved frame pointer, and any padding the compiler inserts between the buffer and the saved registers.
Controlling the Instruction Pointer
Theory
Overwriting the return address gives you one thing: the value the CPU loads into rip at the next ret. "Controlling the instruction pointer" is the pivotal moment of an exploit — the instant the program stops doing what its author intended and starts doing what the attacker's data directs. But a controlled rip is not yet a working exploit; it is a primitive, and the attacker's real problem is what address to put there. Everything after this point in the chapter is about answering that question under increasingly hostile defenses.
The central difficulty is that the useful destinations are not at predictable addresses, for two reasons that correspond to two mitigations:
- You cannot run data you wrote. Thanks to NX/DEP (W^X from the layout topic), the buffer you control — on the stack or heap — is not executable. So
rippointing at your own bytes just faults. This rules out the simplest attack (inject code, jump to it) on any modern target and forces code reuse. - You do not know where anything is. Thanks to ASLR, the addresses of libraries, the stack, and (for PIE) the program itself are randomized each run. So even code that exists at a usable location is at an address the attacker must discover, usually via an information leak (the subject of the format-string section).
This reframes rip control as a search for reachable, useful, known-address code. The historical progression of answers is exactly the structure of the rest of this section:
| Era | What rip points at | Defeated by |
|---|---|---|
| Code injection | attacker's shellcode on the stack | NX/DEP |
| ret2libc | a whole libc function (e.g. system) | ASLR (partially), limited to existing functions |
| ROP | a chain of tiny existing instruction sequences | CFI, shadow stacks (partially) |
There are also destinations beyond the return address that give rip control, and recognizing them is important for both offense understanding and defense: indirect calls and jumps through attacker-corrupted function pointers (a struct field, a C++ vtable pointer, a GOT entry). These do not require a stack overflow at all — a heap UAF that corrupts a vtable pointer yields rip control at the next virtual call. This is why Control-Flow Integrity (5.4) protects all indirect transfers, not just returns.
A practical note on how you confirm rip control during analysis: the unmistakable signal is a crash where rip holds a value you placed — all 0x41 bytes if you sent As, or a recognizable slice of a cyclic pattern. Seeing your own bytes in rip is the proof that you have turned a memory-safety bug into a control-flow primitive, and it is the dividing line between "this is a crash" and "this is potentially exploitable" in triage.
Examples
rip control is a PRIMITIVE, not the whole exploit
───────────────────────────────────────────────────────────────
corrupt saved return addr ─┐
corrupt a vtable pointer ─┼──► rip = attacker-chosen value
corrupt a GOT entry ─┤ │
corrupt a function ptr ─┘ │ but WHERE should it point?
▼
┌───────────────── NX forbids ──────────────────┐
│ your own injected bytes (stack/heap) -> FAULT │
└────────────────────────────────────────────────┘
│ so reuse existing code:
▼
┌──────── but ASLR hides addresses ─────────┐
│ need a LEAK to learn where libc/PIE live │
└────────────────────────────────────────────┘
=> the rest of this section answers "where should rip point".
# The triage signal: your bytes appear in rip (study harness).
gdb -q ./vuln -ex 'run AAAAAAAAAAAAAAAAAAAAAAAABBBBBBBB' \
-ex 'info registers rip' -ex 'quit'
# rip = 0x4242424242424242 <- the 'B' bytes: rip is fully controlled.
# Contrast with a non-control crash (e.g. a null deref) where rip holds a
# legitimate code address -- that is a bug, but not (yet) a control primitive.
// rip control WITHOUT a stack overflow: a corrupted function pointer.
#include <stdio.h>
#include <stdlib.h>
struct handler { void (*fn)(const char *); char buf[32]; };
void dispatch(struct handler *h, const char *msg) {
// If an earlier bug overwrote h->fn with an attacker value, THIS call
// transfers rip there -- no return-address overwrite needed. CFI exists
// precisely to validate indirect calls like this one.
h->fn(msg);
}
Key Takeaways
- Controlling the instruction pointer is a primitive — a controlled
ripvalue — not a finished exploit; the hard part is choosing an address that is both useful and reachable under the defenses. - NX/DEP means attacker-written buffers are not executable, so
ripcannot usefully point at injected code, forcing code-reuse techniques. - ASLR means useful code is at randomized addresses, so the attacker generally needs an information leak to learn where it is.
- Return addresses are only one route to
ripcontrol; corrupted function pointers, C++ vtable pointers, and GOT entries give the same primitive through indirect calls, which is why CFI protects all indirect transfers. - In triage,
ripholding attacker-supplied bytes is the signal that a memory-safety bug has become a control-flow primitive — the line between a mere crash and a potential exploit.
🧪 Practice
- Craft an input that places recognizable bytes (e.g. all
Bs) intoripunder GDB and confirm the register holds exactly those bytes.- Explain, for a target with NX enabled, why pointing
ripat the overflowing buffer's address results in a fault rather than code execution.- Describe a scenario in which
ripis controlled without any stack overflow, naming the corrupted structure and the instruction that transfers control.- Interview-style: "You have reliable control of
ripon a modern 64-bit Linux target with NX and ASLR enabled. Why is your exploit still far from finished, and what do you need next?" Hint: enumerate the two things you still lack — executable attacker code is forbidden, and the addresses of reusable code are randomized — and name the capability that resolves the second.
Shellcode Construction and Constraints
Theory
"Shellcode" is the traditional name for a small payload of machine code that an exploit arranges to run — historically to launch a shell, hence the name. In the modern defended landscape shellcode is often not directly runnable (NX forbids executing attacker buffers), but understanding its constraints remains essential for two defensive reasons: it explains what intrusion-detection and anti-exploitation systems look for, and it clarifies why the field moved to code reuse. This topic treats shellcode conceptually — as a set of constraints a payload must satisfy — rather than as a recipe to produce weaponized bytes.
The reason shellcode is written in raw machine code rather than compiled C is the environment it runs in, which is radically more hostile than a normal program's. Three constraints dominate, each flowing from how the payload is delivered:
- No assumptions about the environment. The payload executes inside a victim process it did not set up. It cannot assume a particular library is loaded at a known address, cannot rely on the C runtime, and often cannot know its own address. This is why classic shellcode talks directly to the kernel via the
syscallinstruction (from the program-analysis chapter) rather than calling libc, and why it is written to be position-independent — referencing its own data relatively, because it does not know where it was placed. - Forbidden bytes. The payload usually travels through functions that interpret certain bytes specially. A string-copy stops at a null byte (
0x00), so shellcode delivered viastrcpymust contain none — which forces avoiding instructions that encode a zero (for example, choosingxor eax, eaxto zero a register instead ofmov eax, 0, whose encoding contains zero bytes). Other channels forbid newlines, whitespace, or non-alphanumeric bytes. The set of allowed bytes is the "bad-char" constraint, and satisfying it is much of the craft. - Size limits. The payload must fit in the space the bug provides, which may be only a few dozen bytes, forcing extreme compactness.
Because self-contained payloads are hard to deliver within these constraints, exploits commonly use a staged approach conceptually: a tiny first stage, which fits the constraints, does nothing but receive or locate a larger second stage and transfer control to it. This separates the "fits in the hole and avoids bad chars" problem from the "does the actual work" problem.
From the defender's standpoint, these constraints are exactly what signatures and heuristics key on. A run of bytes that is valid position-independent code containing a syscall and no null bytes, appearing in a network buffer or a file, is anomalous — which is why intrusion-detection systems historically scanned for shellcode patterns and NOP sleds (next topic), and why modern defenses instead make the payload unrunnable (NX) so its byte pattern no longer matters. It also explains why W^X/NX is such a high-value mitigation: it does not try to recognize shellcode (an arms race the defender loses), it removes the ability to execute attacker-controlled bytes at all.
Examples
Why shellcode is machine code, not C: the constraints
───────────────────────────────────────────────────────────────
Constraint Cause Consequence
───────────────── ─────────────────────── ──────────────────
position-independent payload's address unknown rip-relative refs only
no NUL bytes (0x00) travels through strcpy avoid zero-encoding ops
small (tens of bytes) bug provides little space hand-tuned, staged
no libc / loader runs in a foreign process direct `syscall` usage
W^X makes it unrunnable NX on stack/heap pages => code reuse instead
; Illustrative: why a "bad char" forces an instruction choice.
; GOAL: set rax to 0 (a step in many payloads).
mov eax, 0 ; encodes as B8 00 00 00 00 -> contains NUL bytes!
; an strcpy-delivered payload is TRUNCATED here.
xor eax, eax ; encodes as 31 C0 -> no NUL, 2 bytes.
; this is why shellcode zeroes registers via xor.
; The same pressure shapes every instruction choice: the allowed byte set,
; not readability, dictates the encoding. This is the craft of the constraint,
; shown conceptually rather than as a deliverable payload.
# Defensive use: scan a buffer for shellcode-like properties (a detector).
def looks_like_shellcode(buf: bytes) -> bool:
# Heuristics an IDS might use (illustrative, not exhaustive):
has_no_nulls = b"\x00" not in buf[:64] # payloads often avoid NULs
has_syscall = b"\x0f\x05" in buf # the x86-64 `syscall` opcode
long_run = max((len(r) for r in buf.split(b"\x00")), default=0) > 40
# A long null-free run of valid code containing a syscall is suspicious.
return has_no_nulls and has_syscall and long_run
# The lesson: detection is a heuristic arms race, which is why NX -- making
# such bytes NON-EXECUTABLE regardless of pattern -- is the stronger defense.
Key Takeaways
- Shellcode is a small machine-code payload an exploit runs; understanding its constraints is defensively valuable even though NX usually makes it directly unrunnable today.
- It is written in raw, position-independent machine code using direct
syscalls because it runs in a foreign process with no reliable libc, loader, or known addresses. - Delivery channels forbid certain bytes — most commonly the null byte, since string copies stop at it — forcing instruction choices like
xor eax, eaxovermov eax, 0to avoid zero-encoded bytes. - Tight size limits and bad-char constraints motivate staged payloads: a tiny constrained first stage that fetches or locates a larger second stage.
- These very constraints are what signature-based detection keys on; NX is the stronger defense because it removes executability entirely rather than trying to recognize payloads in an arms race.
🧪 Practice
- Explain, with the encodings, why
xor eax, eaxis preferred overmov eax, 0when a payload must avoid null bytes, and find one other instruction pair with the same trade-off.- List the constraints a payload must satisfy if it is delivered through a function that stops at null bytes and also strips newlines, and describe how a staged approach separates concerns.
- Extend the
looks_like_shellcodeheuristic to also flag a long run of a single repeated byte (a NOP-sled indicator) and discuss its false-positive risk.- Interview-style: "Why did defenders largely abandon signature-based shellcode detection in favor of NX/DEP?" Hint: compare the economics of recognizing an unbounded space of functionally equivalent byte patterns against the economics of making any attacker-controlled page non-executable regardless of its contents.
NOP Sleds and Payload Placement
Theory
Even once an attacker has a payload and (in the pre-NX era, or on targets without NX) a writable-executable region to put it in, a practical problem remains: you have to land rip on the payload's first instruction, and you often do not know the payload's exact address. ASLR, stack-layout variation, and environment differences (the size of environment variables shifts the stack) all make the precise address uncertain. The NOP sled is the classic engineering answer to this uncertainty.
A NOP sled (or NOP slide) is a long run of no-operation instructions placed immediately before the payload. nop (0x90 on x86) does nothing and advances to the next instruction. If rip lands anywhere within the sled, execution simply slides down through the NOPs — doing nothing — until it reaches the payload and runs it. The sled converts a requirement to hit one exact address into a requirement to hit anywhere in a wide range, dramatically improving reliability. The analogy is an aircraft-carrier landing: instead of needing to touch down on a single point, you need only to catch anywhere on a long deck and slide to a stop.
guess an address somewhere in here ───┐
▼
[ 90 90 90 90 90 90 90 90 90 90 90 90 | payload... ]
└────────── NOP sled ─────────────┘ └─ real code
land anywhere in the sled -> slide right -> execute the payload.
A wider sled tolerates a worse address guess.
This connects directly to payload placement — the broader question of where in the process's memory the payload lives and how rip is steered to it. Historically the payload went in the overflowed stack buffer itself, with the return address pointing back into the sled. When the buffer was too small, attackers placed the payload in an environment variable or another input the program stored at a more predictable location, and pointed rip there. The general skill is identifying any attacker-controlled region whose address is known or guessable, and that is (on an NX-free target) executable.
For defenders, NOP sleds are a vivid illustration of why the modern mitigations are shaped as they are:
- NX/DEP defeats the entire approach: if no attacker-controlled page is executable, it does not matter how wide the sled is or how reliably you land on it — the first NOP faults. This is why NOP-sled-plus-shellcode is a historical technique on hardened systems and why the field moved to code reuse (ret2libc, ROP), which needs no executable attacker memory at all.
- ASLR attacks the address guess: by randomizing the base of the stack and libraries, ASLR widens the uncertainty the sled was invented to paper over. A sled helps against small, static layout variation but cannot cover the full entropy of modern ASLR without an enormous (and often impractical) spray.
- NOP sleds also made exploits detectable: a long run of
0x90bytes in a network stream is a classic IDS signature, which in turn produced "NOP-equivalent" sleds using varied do-nothing instructions to evade signatures — another arms race that NX ended by making the question moot.
Examples
Why the sled improves reliability (pre-NX / NX-off targets)
───────────────────────────────────────────────────────────────
Without a sled: the return address must equal the payload's EXACT
start. If your guess is off by even one byte -> crash.
return addr ──► must hit this one byte ✗ brittle
With a 2 KB sled: any guess landing in the 2 KB sled works.
return addr ──► hit anywhere here ──slide──► payload ✓ robust
ASLR shifts the whole region each run, so the GUESS itself is the
problem a sled cannot solve beyond its own width.
# The STRUCTURE of a sled-based payload (conceptual, not a working exploit).
from pwn import asm, p64
NOP = b"\x90"
sled = NOP * 2048 # wide landing zone
payload = b"...machine code..." # the actual payload bytes
# The return address is an ADDRESS somewhere inside the sled region. On an
# ASLR+NX target this does not work at all; it is shown to illustrate the
# placement logic the mitigations were designed to break.
guess_into_sled = 0x7fffffffe000 # a guessed stack address in the sled
buf = b"A" * OFFSET + p64(guess_into_sled) + sled + payload
# filler return addr landing zone code
# Defensive signature: a long 0x90 run is a classic NOP-sled indicator.
# (Illustrative detection, not an exploit.)
python3 - <<'PY'
data = open("captured_input.bin","rb").read()
longest = max((len(r) for r in data.split(b"\x90")), default=0) # wrong metric
# Better: find the longest CONSECUTIVE 0x90 run.
run = best = 0
for b in data:
run = run + 1 if b == 0x90 else 0
best = max(best, run)
print("longest NOP run:", best, "(>100 is suspicious)")
PY
# The real fix is NX: with non-executable input pages, the sled is inert.
Key Takeaways
- A NOP sled is a run of no-op instructions before a payload; landing
ripanywhere in it slides execution down to the payload, converting an exact-address requirement into a wide-range one. - It exists to tolerate address uncertainty from ASLR, stack-layout variation, and environment differences — a reliability engineering trick, not a vulnerability in itself.
- Payload placement is the general problem of putting the payload in a known-or-guessable, executable region and steering
ripto it; environment variables and other stored inputs were common placements. - NX/DEP defeats NOP-sled-plus-shellcode entirely by making attacker pages non-executable, which is the central reason the field moved to code reuse.
- Long
0x90runs are a classic IDS signature, spawning NOP-equivalent evasion; NX made that arms race moot by removing executability rather than detecting patterns.
🧪 Practice
- Explain quantitatively how doubling the width of a NOP sled affects the probability of a successful land under a fixed amount of address uncertainty.
- Write a detector that reports the longest consecutive run of
0x90bytes in a file and discuss why a varied "NOP-equivalent" sled would evade it.- Describe why a NOP sled provides no benefit on a target with NX enabled, tracing what happens when
riplands on the first0x90of a non-executable page.- Interview-style: "An attacker faces 16 bits of stack-base entropy from ASLR. Why can a NOP sled not simply be made wide enough to cover it in a typical stack overflow?" Hint: relate the number of distinct possible base addresses to the amount of contiguous memory the bug lets the attacker control, and consider whether the buffer can even hold a sled that large.
Local vs. Remote Exploitation
Theory
The same memory-corruption bug can be attacked in two very different settings, and the distinction shapes the attacker's goals, the information available, and the difficulty — so it is a useful lens for both offense understanding and defensive prioritization. Local exploitation targets a program running on a machine the attacker already has some access to; remote exploitation targets a program across a network, with no prior foothold.
The defining differences:
| Dimension | Local exploitation | Remote exploitation |
|---|---|---|
| Attacker's starting point | has an account / code execution on the box | no access; only the network interface |
| Usual goal | privilege escalation (user -> root) | initial access (outsider -> code execution) |
| Target programs | setuid binaries, services, kernel, drivers | network daemons, servers, parsers |
| Information available | can read the binary, /proc, run debuggers, retry freely | often blind; limited retries; crashes may be logged |
| ASLR/leak situation | can often read addresses from the local system | usually needs a remote info-leak in the same bug |
| Feedback | rich (local debugging) | sparse (connection resets, timing, error strings) |
Local exploitation is typically about privilege escalation. The attacker can already run code as an unprivileged user and wants more — exploiting a setuid-root binary, a privileged service, or a kernel bug to gain root. The decisive advantage is information: the attacker can read the target binary, inspect /proc, attach a debugger, and retry as often as they like. ASLR is far weaker locally because addresses often leak through readable files or interfaces, and offsets can be measured directly.
Remote exploitation is typically about initial access — getting any code execution on a machine the attacker cannot otherwise touch. It is much harder because the attacker is often blind: they cannot see the target's memory, cannot attach a debugger, and may get only a single indication of success or failure (the connection stays open, or it resets). Crucially, defeating ASLR remotely usually requires an information leak within the same vulnerability or a chained one, because there is no local file to read addresses from. This is why the format-string section's leak primitives and the general "leak then overwrite" pattern are so central to remote exploitation. A further complication is crash sensitivity: a failed attempt may crash the service (alerting defenders, or triggering a restart that re-randomizes ASLR), so remote exploits favor techniques that fail silently or that can be made reliable in one shot.
A special sub-case worth naming is the forking server: a daemon that fork()s a child per connection. Because fork copies the parent's address space without re-randomizing, every child has the same ASLR layout as the parent. An attacker can therefore use many connections to brute-force or leak the layout one byte at a time ("ASLR byte-by-byte" via crash/no-crash oracles), since a crash in one child does not re-randomize the others. This makes forking servers notably easier remote targets than programs that re-exec per connection, and it is a concrete reason the defensive guidance for high-value network services includes re-executing rather than merely forking.
For defenders, the local/remote distinction drives prioritization: a remotely reachable, unauthenticated memory-corruption bug in a network-facing parser is the highest-severity class (it grants initial access from nothing), whereas a local-only bug requires the attacker to already be present. This is exactly the reasoning behind CVSS's attack-vector metric and behind patching network-facing services first.
Examples
The information gap that makes remote exploitation hard
───────────────────────────────────────────────────────────────
LOCAL REMOTE
───── ──────
read /proc/PID/maps ───► know addrs only a socket ───► blind
attach gdb ───► full state no debugger ───► infer from behavior
retry infinitely ───► measure crash = maybe logged / restart
read the binary file ───► offsets may not even have the binary
=> remotely, you usually must LEAK addresses using the bug itself,
then use them before the process (or its ASLR) changes.
Why a forking server weakens ASLR
───────────────────────────────────────────────────────────────
parent (randomized once at exec)
│ fork() ── child 1 same layout as parent
│ fork() ── child 2 same layout as parent
│ fork() ── child 3 same layout ...
A crash in child 2 does NOT re-randomize child 3. The attacker can
probe one guess per connection against an UNCHANGING layout:
a crash/no-crash oracle leaks the base byte by byte.
Mitigation: re-exec (not just fork) per connection so each child
gets a fresh randomization, restoring ASLR's full entropy.
# Local privilege-escalation surface: find setuid binaries to audit.
find / -perm -4000 -type f 2>/dev/null # setuid-root programs: local targets
# Each is a candidate where a memory bug could escalate to the file owner (root).
# Remote surface: what is listening, and on which interface (from earlier chapters).
ss -tlnp 2>/dev/null # network-facing services = remote targets
# A parser in any of these, reachable unauthenticated, is top-severity.
Key Takeaways
- The same bug differs sharply by setting: local exploitation usually aims at privilege escalation with rich information and unlimited retries, while remote exploitation aims at initial access, often blind and with sparse feedback.
- ASLR is far weaker locally because addresses leak through readable files and interfaces; remotely, defeating ASLR usually requires an information leak in the same or a chained vulnerability.
- Remote exploits are crash-sensitive: a failed attempt may crash the service, alert defenders, or restart with fresh randomization, so reliability and silent failure are prized.
- Forking servers weaken ASLR because
forkcopies the parent's layout without re-randomizing, enabling byte-by-byte leaking across many connections; re-execing per connection restores full entropy. - Remotely reachable, unauthenticated memory-corruption bugs are the highest-severity class and drive patch prioritization, which is the logic behind CVSS's attack-vector metric.
🧪 Practice
- List the concrete information advantages a local attacker has over a remote one, and for each, explain how it reduces the difficulty of defeating ASLR.
- Explain why a forking server lets an attacker brute-force an address that a per-connection re-
execing server would protect, referring to when re-randomization occurs.- Enumerate your machine's setuid binaries and its listening network services, and classify each as a local or remote exploitation surface.
- Interview-style: "Two memory-corruption bugs are reported: one in a setuid local utility, one in an unauthenticated network parser. Both are equally easy to trigger. Which do you patch first and why?" Hint: consider what prior access each attack requires and which one turns an outsider into code execution on the machine.
Return-to-libc Attacks
Theory
Return-to-libc (ret2libc) is the first and simplest code-reuse technique, invented specifically to defeat NX/DEP. The insight is elegant: if you cannot execute your own injected code because no attacker page is executable, then do not inject code — reuse code that is already present and already executable. The process has already mapped the C library, full of useful, executable functions. Instead of pointing the overwritten return address at your buffer, point it at an existing libc function such as system.
To see why this works, recall exactly what ret does and how arguments are passed. After an overflow, the attacker controls the stack where the return address sits. If they place the address of system there, then on ret the CPU jumps into system as though it had been called. But a called function expects its arguments and a return address laid out according to the calling convention. So the attacker crafts the overflow to build, on the stack, a fake call frame: the address of system, followed by a return address for system to return to, with the argument placed per the ABI.
Here the architecture matters. On 32-bit x86 the calling convention passes arguments on the stack, so ret2libc is especially natural: the attacker lays out [&system][return addr][&"/bin/sh"] and system finds its argument exactly where it expects. On x86-64, arguments go in registers (rdi first, per the System V ABI), and the stack does not directly supply them — so a pure "return straight into system" does not place the argument. The attacker needs a way to load rdi first, which is a pop rdi; ret sequence — and that single observation is the seed of Return-Oriented Programming (the next topic). In other words, ret2libc on 64-bit naturally generalizes into ROP.
The classic ret2libc target is system("/bin/sh"), which requires finding the address of system and of a "/bin/sh" string (conveniently, one exists inside libc itself). This is also where ASLR bites: the address of system is randomized, so the attacker needs to know libc's base — again motivating the information leak. A refinement, ret2plt, targets the program's own PLT stubs (from the program-analysis chapter) for imported functions, which on a non-PIE binary are at fixed addresses, sidestepping libc ASLR for functions the program already imports.
For defenders, ret2libc is the technique that proves a crucial point: NX alone is insufficient. It stops code injection but not code reuse. This is precisely why the mitigation stack kept growing — ASLR was strengthened to hide the addresses ret2libc needs, and later CFI was added to stop the control-flow transfer into the middle of a function. Understanding ret2libc is understanding why no single mitigation is enough, which is the theme of the layered-defense topic.
Examples
ret2libc on 32-bit x86: a fake call frame on the stack
───────────────────────────────────────────────────────────────
After the overflow, the stack (low -> high) is crafted as:
[ filler .............. ] <- fills buf up to the return address slot
[ &system ] <- overwritten return addr: `ret` jumps here
[ &exit (or junk) ] <- return address FOR system (where it returns)
[ &"/bin/sh" ] <- system's argument (32-bit: args on the stack)
On `ret`: rip = &system; system reads its arg from the stack -> shell.
No injected code executed -> NX is bypassed entirely.
Why x86-64 needs a gadget (the bridge to ROP)
───────────────────────────────────────────────────────────────
System V AMD64: the 1st argument must be in RDI, not on the stack.
So you cannot just "return into system" and have it find its arg.
[ &(pop rdi ; ret) ] <- a tiny reusable snippet: load rdi, then ret
[ &"/bin/sh" ] <- popped into rdi by the gadget
[ &system ] <- then this runs, with rdi already set
That "pop rdi ; ret" borrowed from existing code IS a ROP gadget.
ret2libc on 64-bit is thus the simplest ROP chain.
# Locate the pieces ret2libc needs (study/analysis, not a weaponized exploit).
# Address of system and of a "/bin/sh" string inside the loaded libc:
objdump -T /lib/x86_64-linux-gnu/libc.so.6 | grep ' system$'
strings -a -t x /lib/x86_64-linux-gnu/libc.so.6 | grep '/bin/sh'
# These addresses are offsets within libc; ASLR randomizes libc's base, so a
# real attack also needs a LEAK of that base -- the reason mitigations layer.
# checksec shows whether NX is on (making code injection fail, forcing ret2libc):
checksec --file=/bin/ls
Key Takeaways
- Return-to-libc defeats NX by reusing existing executable code instead of injecting new code: the overwritten return address points at a real libc function like
system. - The attacker builds a fake call frame on the stack so the reused function receives its arguments and a return address per the calling convention.
- On 32-bit x86, arguments live on the stack, making ret2libc direct; on x86-64 they live in registers, so a
pop rdi; retgadget is needed first — which is the conceptual seed of ROP. - The technique still needs the address of
system(and a"/bin/sh"string), which ASLR randomizes, so a real attack also requires an information leak; ret2plt targets fixed PLT stubs on non-PIE binaries to sidestep libc ASLR. - ret2libc proves that NX alone is insufficient — it stops injection but not reuse — which is why the mitigation stack grew to include stronger ASLR and CFI.
🧪 Practice
- Draw the crafted 32-bit stack layout for a ret2libc call to a one-argument function, labeling the overwritten return address, the fake return address, and the argument.
- Explain why the identical layout does not supply the argument on x86-64, and what minimal gadget resolves it.
- Use
objdump -Tandstringsto locate the libc offsets ofsystemand a"/bin/sh"string, and explain why those offsets are not enough without knowing libc's runtime base.- Interview-style: "A system has NX enabled but a non-PIE binary and no ASLR on the main executable. How does that combination make a code-reuse attack easier, even without leaking libc?" Hint: consider which of the program's own fixed-address structures (PLT stubs for imported functions) an attacker could return into without needing any randomized library address.
Return-Oriented Programming (ROP) Chains
Theory
Return-Oriented Programming generalizes ret2libc into a Turing-complete technique, and it is the dominant code-reuse method on modern systems. The motivating problem: ret2libc lets you call whole existing functions, but what if the function you need is not present, or you need to perform computation (set a register, do arithmetic, make a syscall with specific arguments) that no single function provides? ROP answers this by reusing not whole functions but tiny fragments of existing code, stitched together into an arbitrary program.
The key building block is the gadget: a short sequence of instructions that ends in a ret. For example, pop rdi; ret is a gadget that loads the next stack value into rdi and then returns. Because ret pops the next address off the stack and jumps to it, a stack filled with gadget addresses becomes a program: each ret hands control to the next gadget, and the stack itself is the instruction stream. The attacker does not supply code — they supply a sequence of addresses of code that already exists, interleaved with the data those gadgets consume.
The analogy that makes this click is a ransom note assembled from magazine clippings. You cannot write new letters, but by cutting individual words and letters out of existing magazines and pasting them in order, you can compose any message. ROP gadgets are the clippings; the chain is the pasted-up note. The "magazines" are the executable code already in the process — libc, the program itself — which NX cannot help against, because this code is supposed to be executable.
How a chain runs, step by step:
stack (grows down; ret consumes upward) executable code (existing)
┌───────────────────┐ ┌──────────────────────┐
│ &gadget1 pop rdi;ret ──────────────────────► pop rdi ; ret │
│ value_for_rdi │ ◄─ popped into rdi └──────────────────────┘
│ &gadget2 pop rsi;ret ──────────────────────► pop rsi ; ret
│ value_for_rsi │ ◄─ popped into rsi
│ &target_function ─────────────────────────► e.g. a libc function,
│ ... │ now called with rdi,rsi set
└───────────────────┘
Each gadget's trailing `ret` advances to the next stack entry.
Why ROP is so powerful and so hard to stop:
- It is Turing-complete in practice. Real binaries, especially with libc linked, contain enough gadgets (register loads, memory reads/writes, arithmetic, syscalls, conditional control via stack pivots) to compute anything. The usual practical goal is more modest — set up the registers for a single
execvesyscall, or callmprotectto make a region executable and then jump to injected code (defeating NX by turning it off for one page). - It needs no executable attacker memory, so NX is irrelevant to it by construction.
- It reuses legitimate code, so it has no distinctive byte signature to detect — the "code" is just a list of addresses.
ROP is precisely what motivated the next generation of defenses, and understanding ROP is understanding why they take the shape they do. ASLR attacks ROP by randomizing where the gadgets are (so the attacker must leak an address first). CFI and shadow stacks attack the mechanism itself: a shadow stack keeps a protected copy of return addresses and detects when a ret goes somewhere the real call did not intend, which breaks the chain because every gadget relies on ret going to an attacker-chosen place. Intel CET's shadow stack is the hardware realization of this. A ROP variant, JOP (Jump-Oriented Programming), uses indirect jmp/call gadgets instead of ret to evade shadow-stack defenses, which is why CFI protects forward edges (indirect calls/jumps) as well as backward edges (returns).
Examples
A ROP chain to set up registers for a syscall (conceptual structure)
───────────────────────────────────────────────────────────────
GOAL (illustrative): call execve("/bin/sh", NULL, NULL)
System V syscall ABI: rax=59, rdi=&"/bin/sh", rsi=0, rdx=0, then syscall.
stack contents (addresses of EXISTING gadgets + data):
&(pop rdi ; ret) ; gadget: load rdi
&"/bin/sh" ; data popped into rdi
&(pop rsi ; ret) ; gadget: load rsi
0 ; data popped into rsi
&(pop rdx ; ret) ; gadget: load rdx
0 ; data popped into rdx
&(pop rax ; ret) ; gadget: load syscall number
59 ; __NR_execve
&(syscall ; ret) ; gadget: perform the syscall
Every address is to code ALREADY in the binary/libc. NX cannot help.
ASLR means these addresses must first be learned via a leak.
# Discovering gadgets and assembling a chain (analysis tooling, conceptual).
# ROPgadget / ropper enumerate the gadgets present in a binary:
# ROPgadget --binary ./target | grep ': pop rdi ; ret'
# ropper --file ./target --search 'pop rdi; ret'
from pwn import ELF, ROP, p64
elf = ELF("./target") # parse the binary's executable segments
rop = ROP(elf) # pwntools finds gadgets automatically
rop.raw(rop.find_gadget(["pop rdi", "ret"]).address)
rop.raw(next(elf.search(b"/bin/sh"))) # address of an existing string
rop.raw(elf.symbols.get("system", 0)) # reuse an existing function
chain = rop.chain() # the assembled list of addresses + data
# chain is just bytes of ADDRESSES: no executable payload, so NX is moot.
print(rop.dump()) # human-readable view of the chain
# Count the raw material: how many gadgets does a binary offer? (analysis)
ROPgadget --binary /lib/x86_64-linux-gnu/libc.so.6 | wc -l
# typically tens of thousands -- ample for a Turing-complete chain.
# Shadow-stack / CET status of a binary (the mitigation that breaks ret-chains):
readelf -n ./target | grep -i 'SHSTK\|IBT' # CET shadow stack / indirect-branch
Key Takeaways
- ROP generalizes ret2libc by chaining tiny existing instruction fragments ("gadgets," each ending in
ret) into an arbitrary program whose instruction stream is the attacker-controlled stack. - Each gadget's trailing
retjumps to the next attacker-chosen address, so a stack of gadget addresses plus data executes like a program made of magazine clippings. - ROP needs no executable attacker memory and reuses legitimate code, so NX is irrelevant to it and it has no byte signature; real binaries with libc contain enough gadgets to be Turing-complete.
- A common practical goal is minimal — set up registers for one syscall, or call
mprotectto make a page executable — rather than a full computation in gadgets. - ROP is what motivated ASLR to be strengthened (hide the gadgets) and CFI/shadow stacks to be introduced (detect returns going to unintended targets); JOP uses indirect jumps to evade shadow stacks, which is why CFI protects forward edges too.
🧪 Practice
- Use
ROPgadgetorropperon a local binary to findpop rdi; retand two other register-loading gadgets, and explain what each would contribute to a chain.- Write out, as a stack diagram, the sequence of gadget addresses and data values needed to load three registers and then perform a syscall, labeling each entry.
- Explain precisely why NX provides no protection against a ROP chain, referring to where the gadgets live and whether any attacker-written page is executed.
- Interview-style: "A hardware shadow stack (Intel CET) is enabled on a target. Why does it break classic ROP, and what attack variant tries to get around it?" Hint: consider what a shadow stack checks at every
retand why that invalidates gadget chaining, then recall which kind of indirect branch a jump-oriented variant uses instead ofret.
Gadget Discovery and Chain Building
Theory
If ROP is assembling a ransom note from clippings, gadget discovery is the act of reading through the magazines to catalog which clippings exist, and chain building is arranging them into the desired message. This topic is the practical craft behind ROP, and understanding it clarifies both why ROP is so reliably achievable and why certain defensive measures (reducing gadget availability) are pursued.
Gadget discovery starts from a surprising fact about x86-64: because instructions are variable-length and can be decoded starting at any byte (the misalignment property from the disassembly topic), the set of usable gadgets is far larger than the instructions the compiler actually emitted. A tool scans the executable segments for every ret byte (0xC3) and then looks backward from each one, decoding the preceding bytes as instructions. Critically, it also finds unintended gadgets: sequences that appear when you start decoding in the middle of a legitimate instruction. A mov instruction's immediate operand might contain a 0xC3 byte that, decoded from a different offset, becomes a ret preceded by useful bytes. This is why a modest binary yields thousands of gadgets, and why simply "not writing gadget-like code" does not work — the gadgets are an emergent property of the byte stream.
The tools of the trade are ROPgadget, ropper, and pwntools' built-in ROP class. They categorize gadgets by function: register loads (pop reg; ret), memory writes (mov [reg], reg2; ret), arithmetic (add, xor), stack manipulation (pop rsp, pivots), and syscall/call gadgets. An analyst building a chain searches this catalog for the specific operations the chain requires.
Chain building is then a constraint-satisfaction exercise: you have a goal state (specific register and memory values, then a transfer like syscall or a function call) and a set of available gadgets, and you must order them so that each produces an input the next one consumes, without any gadget's side effects clobbering a value you already set up. The practical complications are what make it a craft:
- Side effects. A
pop rdi; retis clean, but many real gadgets do extra things —pop rdi; pop rbp; retconsumes two stack slots, so the chain must account for the extra value. Gadgets that modify flags or other registers can wreck a carefully built state. - Bad bytes. The same bad-char constraint from shellcode applies: if the chain travels through a string copy, no gadget address may contain a null byte, which disqualifies some otherwise-perfect gadgets and forces alternatives.
- Missing gadgets. Sometimes the exact operation you want has no gadget; you synthesize it from several (e.g. no
mov rax, rbxgadget, butpush rbx; pop raxexists), the way you would compute a value you cannot load directly.
For defenders, gadget discovery explains the rationale behind several real mitigations. Reducing the gadget supply is a goal of techniques like -fcf-protection (which inserts endbr64 landing pads and makes many indirect targets invalid), compiler passes that avoid emitting certain byte patterns, and binary rewriting that removes unintended rets. More fundamentally, it explains why shadow stacks are considered the strong defense: rather than trying to reduce the enormous, emergent gadget supply (a losing battle), they invalidate the mechanism — the ability of ret to go to an arbitrary gadget — regardless of how many gadgets exist.
Examples
Unintended gadgets: decoding from the "wrong" offset
───────────────────────────────────────────────────────────────
Legitimate instruction emitted by the compiler:
48 8b 04 25 c3 00 00 00 mov rax, [0xc3] (a normal load)
▲
Start decoding ONE byte later, at the 0xc3:
c3 ret <- an UNINTENDED gadget
And decoding a few bytes earlier can yield "<something> ; ret".
The compiler never wrote a `ret` here, yet one exists in the bytes.
This is why gadget counts are huge and why avoidance is futile.
# Catalog the gadgets in a binary by category (analysis tooling).
ROPgadget --binary ./target --only 'pop|ret' # register-load gadgets
ropper --file ./target --search 'mov [%], %' # memory-write gadgets
ropper --file ./target --search 'syscall' # syscall gadgets
# Count unintended vs intended is not directly separable, but the sheer
# total shows the raw material available:
ROPgadget --binary ./target | tail -1 # "Unique gadgets found: N"
# Chain building with side effects and bad-char awareness (conceptual).
from pwn import ELF, ROP
elf = ELF("./target")
rop = ROP(elf)
# pwntools solves ordering and extra-pop side effects automatically when
# you express the GOAL rather than hand-placing gadgets:
rop.call("mprotect", [0x7ffff7a00000, 0x1000, 7]) # make a page rwx (goal)
chain = rop.chain()
# Verify the chain contains no forbidden bytes for a string-copy delivery:
assert b"\x00" not in chain, "null byte in chain; pick alternate gadgets"
# A gadget like `pop rdi; pop rbp; ret` forces an extra stack slot: the tool
# inserts filler for rbp so the next gadget still lands correctly.
print(rop.dump())
Key Takeaways
- Gadget discovery scans executable bytes for
ret(and other transfer) bytes and decodes backward, finding far more gadgets than the compiler emitted because x86-64 instructions can be decoded from any offset. - Unintended gadgets — sequences that appear when decoding starts mid-instruction — are why even small binaries yield thousands of gadgets and why "avoiding gadget-like code" cannot work.
- Tools (ROPgadget, ropper, pwntools) catalog gadgets by category: register loads, memory writes, arithmetic, stack pivots, and syscalls.
- Chain building is constraint satisfaction: order gadgets so each feeds the next while accounting for side effects (extra pops, clobbered registers/flags) and bad-byte constraints, synthesizing missing operations from combinations.
- The emergent, enormous gadget supply is why reducing gadgets is hard and why shadow stacks — which invalidate the
ret-to-anywhere mechanism rather than counting gadgets — are the stronger defense.
🧪 Practice
- Run a gadget-discovery tool on a local binary and categorize ten gadgets you find into register-load, memory-write, and control-transfer groups.
- Find an example of an unintended gadget by locating a
retbyte inside a larger instruction's encoding, and explain how decoding offset changes the meaning.- Given a goal of setting
rdi,rsi, andraxand the only register-load gadget available beingpop rdi; pop rbp; ret, describe how you would still set all three registers and what filler the chain needs.- Interview-style: "Why is reducing the number of ROP gadgets in a binary a weaker defense than deploying a shadow stack?" Hint: contrast attacking a large, emergent supply you can only partially shrink against invalidating the single mechanism every gadget chain depends on at
rettime.
Stack Pivoting and Limited Overflows
Theory
ROP chains can be long, but real bugs are often small — an overflow might give you only enough space to overwrite the return address and a few bytes beyond, nowhere near enough to hold a full chain. Stack pivoting is the technique that bridges this gap, and it is one of the most important concepts for understanding how attackers turn a constrained primitive into a powerful one.
The core idea: the stack pointer rsp determines where the ROP chain is read from. If the attacker can point rsp at a region they fully control — a large buffer elsewhere in memory, typically on the heap or in a static buffer — then the length limit of the original overflow no longer matters, because the chain lives in that larger region. The small overflow is used only to execute a single pivot gadget that relocates rsp, and from there the real chain runs from the roomy controlled buffer. The analogy: the overflow is a short on-ramp, and the pivot merges you onto the highway (the big buffer) where the real journey happens.
Pivot gadgets are sequences that load rsp from an attacker-influenced source:
| Pivot gadget | Effect |
|---|---|
pop rsp; ret | load rsp directly from the stack (next value) |
xchg rax, rsp; ret | swap rsp with a register the attacker controls |
leave; ret (mov rsp, rbp; pop rbp) | set rsp from rbp, which an earlier overwrite may control |
add rsp, N; ret | shift the stack by a fixed amount (skip over a region) |
The leave; ret pivot connects back to the off-by-one topic: if an attacker controls the saved rbp (even just its low bytes), then a function's leave; ret epilogue sets rsp from that controlled rbp, pivoting the stack. This is how a tiny frame-pointer corruption becomes full control, and it is why the saved frame pointer is a security-relevant value, not just bookkeeping.
Limited overflows more broadly are the art of doing a lot with a little, and stack pivoting is the headline example. Related constrained-primitive techniques include:
- Partial overwrites. Overwriting only the low byte(s) of a return address or pointer, which (because of little-endianness) changes where it points by a bounded amount without needing to know the full randomized address — a way to redirect control under ASLR without a full leak, at the cost of limited range.
- One-gadget / "magic gadget." A single address in libc that spawns a shell if certain register/memory constraints happen to hold, letting a one-slot overwrite suffice when the stars align.
- Chaining a small overflow into a bigger primitive. Using the limited write to set up a read, then the read to leak, then the leak to enable a larger write — the "primitive laddering" that characterizes real exploit development.
For defenders, stack pivoting underscores why the size of an overflow is a poor proxy for its severity: a two-pointer overwrite that enables a pivot can be just as dangerous as a kilobyte overflow. It also motivates defenses that check stack-pointer sanity — some CFI schemes and stack-integrity checks detect an rsp that suddenly points into the heap, and shadow stacks make the pivoted return addresses fail validation regardless of where rsp points.
Examples
Stack pivot: escaping a tiny overflow into a big controlled buffer
───────────────────────────────────────────────────────────────
The bug only lets you overwrite the return addr + 8 bytes. Not enough
for a chain. But you earlier stored a LARGE buffer on the heap (0x4040).
small overflow sets up: heap buffer at 0x4040 (you control):
[ &(pop rsp ; ret) ] <- ret here [ &gadget1 ]
[ 0x4040 ] <- popped to rsp [ data ]
[ &gadget2 ]
After `pop rsp`, rsp = 0x4040. The next `ret` reads from the HEAP buffer,
and the full ROP chain executes from there. The tiny overflow is just
the on-ramp; the pivot merges onto the large chain.
leave;ret pivot driven by a controlled saved rbp (ties to off-by-one)
───────────────────────────────────────────────────────────────
function epilogue: leave ; == mov rsp, rbp ; pop rbp
ret
If an overflow set saved rbp = &controlled_buffer - 8, then:
leave -> rsp = controlled_buffer - 8 ; pop rbp (consumes 8)
ret -> reads the next value from controlled_buffer -> chain begins
A corruption of only the saved frame pointer thus yields a pivot.
# Find pivot gadgets in a binary (analysis).
ROPgadget --binary ./target | grep -E ': (pop rsp|xchg .*, rsp|leave) ; ret'
ropper --file ./target --search 'xchg %, rsp; ret'
ropper --file ./target --search 'add rsp, %; ret'
# A one-gadget scan for libc (analysis tool; constraints must hold):
one_gadget /lib/x86_64-linux-gnu/libc.so.6 # prints addresses + their constraints
Key Takeaways
- Stack pivoting relocates
rspto a large attacker-controlled region so a small overflow — too small to hold a chain — can still execute a full ROP chain that lives elsewhere. - The small overflow runs only a pivot gadget (
pop rsp; ret,xchg rax, rsp; ret,leave; ret,add rsp, N; ret); the real chain is read from the roomy buffer after the pivot. - The
leave; retpivot ties to the off-by-one topic: controlling the savedrbp(even its low bytes) lets a function's epilogue setrspfrom attacker data, so frame-pointer corruption is security-relevant. - Related limited-overflow techniques include partial-pointer overwrites (redirect within a bounded range without a full leak), one-gadgets (a single constraint-dependent address), and laddering small primitives into larger ones.
- Overflow size is a poor proxy for severity; defenses that sanity-check
rspand shadow stacks that validate return addresses regardless ofrspare the relevant countermeasures.
🧪 Practice
- Find a
pop rsp; retorleave; retpivot gadget in a local binary and explain, as a stack diagram, how it would redirect the chain into a heap buffer.- Explain how controlling only the saved frame pointer's low byte can lead to a
leave; retpivot, connecting it to the off-by-one NUL technique.- Describe a partial-pointer-overwrite scenario in which an attacker redirects a return address within the same page without knowing the full ASLR-randomized address.
- Interview-style: "A bug allows overwriting only 16 bytes past a buffer — the saved frame pointer and the return address, nothing more. Why might this be fully exploitable despite the tiny size?" Hint: think about what those 16 bytes let you set (
rbpand the first gadget), and how a pivot could move execution to a much larger buffer you control elsewhere.
Building Exploits Against Real Binaries
Theory
The preceding topics are individual techniques; this one is about the process of combining them against a real, defended target. Real exploitation is not a single trick but a methodical campaign of discovering primitives and laddering them into control, under mitigations that each must be defeated in turn. Understanding this process is what lets a defender reason realistically about exploitability — the difference between "this crashes" and "this is a weaponizable bug" is usually a chain of several steps, each of which can fail.
The general methodology, roughly in order:
- Reconnaissance. Determine what you are up against. Run
checksecto enumerate mitigations (canary, NX, PIE, RELRO), identify the libc version, and study the binary with the static- and dynamic-analysis tools from the program-analysis chapter. Mitigations present dictate which techniques are even applicable. - Find and characterize the bug. Identify the vulnerability and the primitive it yields — a stack overflow (control of return address), a heap overflow (control of adjacent object), a format string (read/write), a UAF (type confusion). Measure offsets with cyclic patterns.
- Defeat the information barriers. If ASLR and PIE are on, you need a leak before you can use any address. Often the first use of the bug is purely to leak — a format-string read, an over-read, an uninitialized-memory disclosure — establishing libc and/or binary base addresses.
- Build the control primitive. With addresses known, construct the payload: a ROP chain (discovered gadgets), a ret2libc call, a GOT overwrite, or a vtable hijack, as the mitigations allow.
- Achieve the goal. Typically either a syscall sequence (via ROP) or a call to a function like
system/execve, or, to sidestep NX for injected code, amprotect/VirtualProtectcall that makes a controlled region executable. - Make it reliable. A one-shot remote exploit must work without a debugger and ideally without crashing on failure. Reliability engineering — accounting for layout variance, choosing stable offsets, handling partial failures — is often the hardest part and is what separates a proof-of-concept from a dependable exploit.
The crucial conceptual point is primitive laddering: exploits are built by converting weak primitives into stronger ones. A relative write becomes an arbitrary write; an arbitrary write plus a leak becomes control of a function pointer; control of a function pointer becomes code execution. Each rung uses the capability gained on the previous one. Thinking in these terms — "what primitive do I have, and what can it be upgraded into?" — is both how attackers build exploits and how defenders assess them.
This is also where the professional and ethical boundary is sharpest. Exploit development against real targets is legitimate only within explicit authorization: CTF competitions (deliberately vulnerable binaries), your own test systems, sanctioned penetration-testing engagements with written scope, and coordinated vulnerability research with responsible disclosure. The same skills applied to systems you do not own, without authorization, are criminal under computer-misuse law in essentially every jurisdiction. The reason to learn this process is to build and defend software and to assess risk accurately — not to attack others — and the standard practice environment is intentionally vulnerable CTF-style binaries where no real system is at risk.
Examples
The exploitation campaign as a dependency graph
───────────────────────────────────────────────────────────────
checksec ──► know mitigations ──┐
▼
find bug ──► measure offset ──► need addresses?
│ yes (ASLR/PIE on)
▼
LEAK primitive (fmt string / over-read)
│ -> libc base, PIE base
▼
CONTROL primitive (ROP / ret2libc / GOT)
│
▼
GOAL (syscall / system / mprotect+code)
│
▼
RELIABILITY (one-shot, no-crash, layout-stable)
A break at ANY stage stops the exploit -- which is why layered
mitigations (5.4) are effective even when each is individually bypassable.
# Step 1 recon on a (study/CTF) binary: what must be defeated?
checksec --file=./challenge
# RELRO: Full | Canary: Yes | NX: Yes | PIE: Yes
# => need: leak for PIE+libc, canary leak/bypass, ROP (NX), no GOT overwrite
file ./challenge ; strings -a ./challenge | grep -i 'libc\|2\.3' # libc hints
# The SKELETON of an exploit script against a deliberately vulnerable CTF
# binary, showing the laddering structure (not a weaponized exploit).
from pwn import process, ELF, ROP, p64, u64
elf = ELF("./challenge")
io = process("./challenge") # local, intentionally vulnerable target
# 1. LEAK stage: use the bug to disclose an address, compute libc base.
leak = u64(io.recv(8)) # (bug-specific leak, details omitted)
libc_base = leak - KNOWN_OFFSET # anchor the randomized library
log_info = libc_base # now every libc address is known
# 2. CONTROL stage: build a ROP chain using the now-known addresses.
rop = ROP(elf)
rop.call("some_function", [arg1]) # addresses resolved from the leak
payload = b"A"*OFFSET + rop.chain() # offset measured earlier via cyclic()
# 3. GOAL + deliver. Each stage depended on the previous one's result.
io.sendline(payload)
io.interactive() # on a CTF target you now control it
Key Takeaways
- Real exploitation is a methodical campaign — recon, find the bug, defeat information barriers, build a control primitive, achieve the goal, make it reliable — not a single trick.
checksecand version identification come first because the mitigations present determine which techniques are even applicable.- Under ASLR/PIE, the bug's first use is often purely to leak an address, since no absolute address can be used until the layout is known.
- Exploits are built by primitive laddering: relative write to arbitrary write to pointer control to code execution, each rung using the last rung's capability — the key mental model for both building and assessing exploits.
- Reliability (one-shot, no-crash, layout-stable) is frequently the hardest stage and is what distinguishes a proof-of-concept from a dependable exploit.
- Exploit development is legitimate only under explicit authorization — CTFs, owned test systems, scoped pentests, coordinated disclosure — and the standard practice environment is intentionally vulnerable binaries.
🧪 Practice
- Run
checksecon three local binaries and, for each, list which techniques from this section are ruled in or out by the mitigations present.- For a deliberately vulnerable CTF-style binary (one you are authorized to use), write out the dependency graph of primitives your exploit would need, marking which stage requires a leak.
- Explain the concept of primitive laddering with a concrete three-rung example, naming the capability gained at each rung.
- Interview-style: "Why can a set of individually bypassable mitigations still make a target effectively unexploitable in practice?" Hint: think of the exploit as a chain of dependent stages and consider what the probability of success becomes when each stage must be defeated and any single failure aborts the whole attempt.
<a id="53-format-string-vulnerabilities"></a>
5.3 Format String Vulnerabilities
Format string bugs are a distinct and instructive vulnerability class in which the attacker controls the format string passed to a printf-family function, turning a logging call into an arbitrary memory read and write; this section builds from how printf actually works up to how a single careless % becomes a full exploitation primitive and how it defeats ASLR.
printf Family and Format Specifier Semantics
Theory
To understand the vulnerability you must first understand precisely what printf does, because the bug is entirely a consequence of its normal, documented behaviour being driven by attacker-controlled input. printf is not a simple print function — it is a tiny interpreter. It reads the format string character by character, copying ordinary characters to the output, and whenever it encounters a % it treats what follows as a directive instructing it to fetch an argument and format it.
The critical realization is that the format string decides how many arguments to consume and how to interpret each one. printf("%d %s", a, b) tells printf: "fetch one integer argument and print it as a decimal, then fetch one pointer argument and print the string it points to." printf has no independent knowledge of how many arguments were actually passed — it trusts the format string completely and fetches exactly as many as the directives request, from wherever the calling convention says arguments live. This blind trust is the entire vulnerability: if the attacker controls the format string, they control how many arguments printf fetches and how it interprets them, even though the programmer passed no corresponding arguments at all.
The common specifiers, with their security relevance:
| Specifier | Meaning | Security relevance |
|---|---|---|
%d / %u / %x | print an integer (decimal/unsigned/hex) | leaks an argument slot's value |
%s | print the string at a pointer argument | dereferences an argument; can crash or leak memory |
%p | print an argument as a pointer value | the easiest way to dump raw slots |
%n | write the number of bytes output so far to a pointer argument | the write primitive — this is the dangerous one |
%N$x | positional: print the Nth argument | "direct parameter access" (a later topic) |
width/precision (%100x) | pad output to a given width | inflate the byte count for %n writes |
The decisive distinction is that most specifiers read an argument, but %n writes through one. %n takes a pointer argument and stores into it the number of characters printed so far. It was intended for column-alignment calculations, but in the hands of an attacker controlling the format string, it is a primitive to write a chosen value to a chosen address — which is why %n is at the center of format-string exploitation and why modern libc and compilers restrict or forbid it.
The security framing to carry forward: a format string is code, in the sense that it is an instruction stream an interpreter executes. Passing attacker data as the format string is therefore exactly as dangerous as passing attacker data as code — the same category of error as eval(user_input) in a scripting language. This reframing is the key to both recognizing and preventing the bug.
Examples
// The safe call vs the vulnerable call -- the ONLY difference is one argument.
#include <stdio.h>
void log_message(const char *user_input) {
// SAFE: user_input is DATA, printed literally via a constant format.
printf("%s", user_input); // no matter what % chars it contains
// VULNERABLE: user_input IS the format string.
printf(user_input); // if it contains %x/%s/%n, printf acts on them
// e.g. user_input = "%x %x %x" makes printf fetch and print 3 "arguments"
// that were never passed -- reading whatever is in those slots.
}
printf as an interpreter driven by the format string
───────────────────────────────────────────────────────────────
printf("%d %s", 42, ptr)
read 'format': %d -> fetch arg slot #1 -> print 42
' ' -> copy a space
%s -> fetch arg slot #2 (a pointer) -> print *ptr
printf(attacker) where attacker = "%p %p %p"
read 'format': %p -> fetch arg slot #1 -> print its raw value
%p -> fetch arg slot #2 -> print its raw value
%p -> fetch arg slot #3 -> print its raw value
NO arguments were passed, yet printf fetches three. It trusts the
format string's directives absolutely.
# The compiler warns loudly about a non-constant format string -- heed it.
gcc -Wall -Wformat -Wformat-security fmt.c -o fmt
# warning: format not a string literal and no format arguments [-Wformat-security]
# This single warning class catches essentially all format-string bugs at
# compile time, which is why -Wformat-security is a standard hardening flag.
Key Takeaways
printfis an interpreter: it reads the format string and, for each%directive, fetches and formats an argument, trusting the format string about how many arguments exist and what types they are.- The vulnerability is that an attacker-controlled format string controls how many arguments
printffetches and how it interprets them, even though no matching arguments were passed. - Most specifiers read an argument;
%nuniquely writes through a pointer argument the count of bytes output so far, making it the write primitive at the heart of the attack. %N$positional specifiers and width fields are the tools that make reads and writes precise and controllable.- A format string is effectively code; passing attacker data as the format string is the same category of error as evaluating attacker data as code, and
-Wformat-securitycatches it at compile time.
🧪 Practice
- Explain the exact difference in behaviour between
printf("%s", s)andprintf(s)whenscontains the characters%x %x, and why only one is dangerous.- Compile a program containing
printf(user_input)with-Wformat-securityand document the warning; then fix it and confirm the warning disappears.- For the format string
"%d-%s", state how many argumentsprintfwill fetch and what type each must be for the call to be well-defined.- Interview-style: "Why is it accurate to describe a format string vulnerability as a code-injection bug rather than merely a memory bug?" Hint: consider that
printfexecutes the directives in the format string like an instruction stream, and compare passing attacker input there to passing it to a scripting language'seval.
Variadic Functions and Argument Retrieval
Theory
To understand how a format string reads and writes memory, you need to know how variadic functions — functions taking a variable number of arguments, like printf(const char *fmt, ...) — actually retrieve those arguments at the machine level. The vulnerability is a direct consequence of the fact that a variadic function cannot tell how many arguments it was given, or where the real arguments end.
A normal function knows its parameters. A variadic function knows only its fixed parameters (here, fmt) and then walks forward through the argument-passing locations to find the rest, guided entirely by the format string. On x86-64 System V (from the program-analysis chapter), the first six integer/pointer arguments are passed in registers (rdi, rsi, rdx, rcx, r8, r9) and the rest on the stack. For printf, rdi holds the format string pointer, so the variadic arguments begin in rsi and proceed rdx, rcx, r8, r9, then spill onto the stack.
This is implemented via the va_list mechanism (va_start, va_arg, va_end). Conceptually, va_arg maintains a cursor that advances through the argument locations each time a directive requests an argument. The function has no bounds on this cursor — it keeps advancing as long as the format string keeps asking. So when the attacker supplies "%x %x %x %x %x %x %x %x", printf reads the five remaining argument registers and then starts reading stack memory, slot by slot, interpreting whatever is there as the "arguments" the format string requested. Those stack slots contain saved registers, local variables, saved return addresses, and the stack canary — the live contents of the current call stack.
Where printf's variadic cursor walks (x86-64 System V)
───────────────────────────────────────────────────────────────
rdi = fmt (fixed) <- not a variadic arg
rsi = va-arg #1 ─┐
rdx = va-arg #2 │ the first 5 "%" directives read these registers
rcx = va-arg #3 │ (which hold whatever was there -- maybe nothing
r8 = va-arg #4 │ meaningful was passed)
r9 = va-arg #5 ─┘
[rsp+0] = va-arg #6 ─┐
[rsp+8] = va-arg #7 │ further directives read STACK SLOTS: locals,
[rsp+16] = va-arg #8 │ saved registers, canary, return addresses...
... ─┘
The cursor has no end. Each extra "%x" advances it one slot further
into the live stack, dumping memory the attacker never should see.
The security implications follow immediately:
- Reading past the real arguments reads live stack memory. This is the mechanism of the information-disclosure topics that follow — dumping stack slots exposes the canary, saved frame pointers, and pointers into libc and the binary that break ASLR.
- The argument count mismatch is undetectable at runtime. Because the function cannot know the true count, there is no runtime check that could notice the format string asking for arguments that do not exist. This is why the defense must be static (constant format strings, compiler warnings) rather than a runtime guard inside
printf. - The ABI determines what the first several reads return. On x86-64 the first five directives read registers, so an attacker typically sees register garbage before the reads reach the controllable stack region — a detail that matters for the "direct parameter access" offset calculations in a later topic.
Examples
// A variadic function walks its arguments with no idea where they end.
#include <stdarg.h>
#include <stdio.h>
int my_sum(int count, ...) {
va_list ap;
va_start(ap, count); // cursor starts after the fixed `count`
int total = 0;
for (int i = 0; i < count; i++) // WE bound it with `count`...
total += va_arg(ap, int); // ...but printf is bounded only by the
va_end(ap); // format string, which the ATTACKER controls
return total;
// If `count` lied (said 10 when 3 were passed), va_arg would read
// 7 slots of whatever follows on the stack -- exactly printf's problem.
}
# Watch printf read register then stack slots as "arguments" (study harness).
cat > leak.c <<'EOF'
#include <stdio.h>
int main(void) {
char local_marker[8] = "ABCDEFG"; // a recognizable stack value
char buf[64];
fgets(buf, sizeof buf, stdin);
printf(buf); // VULNERABLE: attacker controls fmt
return 0;
}
EOF
gcc -g -O0 -fno-stack-protector leak.c -o leak
echo '%p %p %p %p %p %p %p %p %p %p' | ./leak
# prints a row of raw slot values: the first few are registers, then stack
# slots -- and "0x4746454443424241" (="ABCDEFG") appears once the cursor
# reaches local_marker. The attacker is reading live stack memory.
Key Takeaways
- A variadic function knows only its fixed parameters and walks forward through argument locations for the rest, guided solely by the format string — it cannot know the true argument count.
- On x86-64 System V,
printf's variadic arguments start inrsiand proceed throughrdx, rcx, r8, r9, then spill to the stack, so extra directives eventually read live stack memory. - Reading past the real arguments dumps stack slots — locals, saved registers, the canary, and return addresses — which is the engine of format-string information disclosure.
- The argument-count mismatch is undetectable at runtime because the function cannot know the real count, so the defense must be static (constant formats,
-Wformat-security), not a runtime check inprintf. - The first several directives read registers before reaching the controllable stack region, a detail that matters for computing argument offsets in exploitation.
🧪 Practice
- Using the
leak.charness, feed increasing numbers of%pspecifiers and identify at which position the recognizable stack marker appears; explain why the first few positions show register contents.- Write a variadic function that is told a count larger than the number of arguments actually passed, and observe (carefully, under a sanitizer) that it reads adjacent stack data — relating this to
printf's behaviour.- Explain why no runtime check inside
printfcan detect a format string that requests more arguments than were passed.- Interview-style: "On x86-64, why does an attacker often see several values of register garbage before their
%preads reach useful stack contents?" Hint: recall which registers carry the first arguments in the System V ABI and that the variadic cursor starts in registers before spilling to the stack.
Arbitrary Memory Read with %x and %s
Theory
The first exploitation capability a format string grants is reading memory, and it comes in two strengths. The weaker, %x/%p, dumps the values in argument slots — useful for leaking whatever happens to be on the stack. The stronger, %s, treats an argument slot as a pointer and prints the string at that address — which, if the attacker can control the pointer, becomes an arbitrary read of any address in the process.
Start with the stack dump. As the previous topic showed, a sequence of %p specifiers walks printf's cursor through registers and then stack slots, printing each as a raw value. This alone is a serious information leak: the stack contains the canary, saved frame pointers, and return addresses and pointers that reveal the base of libc and the PIE binary. An attacker sends many %ps, reads back the row of values, and identifies which slot holds which kind of pointer (a later topic covers using these to defeat ASLR). This is a stack read — it can only see what is already on the stack.
The leap to arbitrary read comes from %s. When printf processes %s, it takes the corresponding argument slot, interprets it as a char *, and prints bytes from that address until a NUL. If the attacker can arrange for a slot to contain an address of their choosing, then %s reads and returns the memory at that address. The standard way to control a slot's contents is that the format string itself lives on the stack (it was read into a stack buffer): the attacker places the target address as raw bytes inside their own input, then uses positional specifiers (next topic) to make %s consume that part of their input as its pointer argument. In effect, the attacker's input supplies both the instruction (%s) and the operand (the target address), and printf dereferences it.
%p dump (stack read) vs %s via controlled pointer (arbitrary read)
───────────────────────────────────────────────────────────────
%p %p %p ... -> prints VALUES of slots: leaks canary, libc ptrs
(limited to what is already on the stack)
[target_addr bytes in input] ... %7$s
-> %s takes slot #7 = target_addr (from input),
dereferences it, prints memory AT target_addr
(reads ANY readable address -> arbitrary read)
Two practical constraints shape the arbitrary read, and both are defensively informative:
%sstops at a NUL and faults on an unmapped address. Reading binary data with embedded zeros is truncated, and pointing%sat an unmapped address crashes the process — which, remotely, may be a detectable failure. Attackers often prefer%p-style reads to first locate readable regions before using%s.- The read is the gateway to defeating ASLR. On its own an arbitrary read is "only" disclosure, but disclosure of a single libc pointer collapses ASLR for the whole library, which is exactly what a remote code-execution chain needs. This is why a format-string bug is rated so severely even when "just" a read: it typically provides the leak that every other technique in this chapter depends on.
Examples
# Stack read: dump slots to find interesting pointers (study harness).
echo 'AAAA %p %p %p %p %p %p %p %p' | ./leak
# AAAA 0x7ffff7... 0x0 0x55555555... 0x4141414141414141 ...
# ^ libc/stack ptrs (defeat ASLR) ^ our "AAAA" input appears,
# proving the format string buffer itself is reachable on the stack.
# Turning a controlled slot into an arbitrary read (conceptual, CTF target).
from pwn import p64
TARGET = 0x404000 # address we want to read (e.g. a GOT entry)
# Place TARGET in the input so a positional %s consumes it as its pointer.
# Suppose slot 7 is where our input's first 8 bytes land (found by %p dump):
payload = p64(TARGET) + b"%7$s" # %7$s derefs slot 7 = TARGET -> prints *TARGET
# Sending this returns the bytes stored at 0x404000, up to the first NUL.
# This reads ARBITRARY process memory using only a logging call.
// The code that enables all of the above is still just this:
#include <stdio.h>
int main(void){ char b[128]; fgets(b,sizeof b,stdin); printf(b); return 0; }
// ^ attacker controls fmt
// Fix: printf("%s", b); -- the target address in the input is then printed
// literally, never dereferenced.
Key Takeaways
- Format strings grant two read strengths:
%x/%pdump the raw values in argument slots (a stack read), while%sdereferences a slot as a pointer and prints the memory there. - A row of
%pspecifiers leaks the canary, saved frame pointers, and libc/PIE pointers present on the stack — a serious disclosure by itself. %sbecomes an arbitrary read when the attacker controls a slot's contents, which they do by placing the target address in their own input (which lives on the stack) and selecting it with a positional specifier.%sstops at NUL bytes and faults on unmapped addresses, so attackers often locate readable regions with%pbefore dereferencing with%s.- Even a read-only format-string bug is severe because leaking one libc pointer collapses ASLR for the whole library, supplying the leak that the rest of exploitation depends on.
🧪 Practice
- Using a format-string study harness, dump enough stack slots with
%pto locate one pointer into libc and one into the stack; explain how you recognize each by its value range.- Explain the mechanism by which placing an address in your own input and using a positional
%sturns a stack read into an arbitrary read of that address.- Describe two reasons an attacker might prefer
%preads over%swhen probing an unknown remote target.- Interview-style: "Why is a format-string bug that only allows reading, never writing, still often sufficient to enable remote code execution elsewhere in an exploit chain?" Hint: consider what a single leaked libc or PIE pointer does to ASLR, and which later techniques become usable once the randomized base addresses are known.
Direct Parameter Access
Theory
The reads and writes described so far have a practical annoyance: to reach the tenth argument slot you would write ten specifiers (%x%x%x%x%x%x%x%x%x%x), consuming and printing nine slots you do not care about just to reach the one you want. Direct parameter access is the format-string feature that removes this clumsiness, and it is what makes format-string exploitation precise and reliable rather than a scattershot dump.
The feature is the positional specifier %N$, where N is an argument index. %7$x means "fetch and format argument number 7 directly," skipping straight to it without touching slots 1 through 6. It is a standard, documented part of the format syntax (originally for internationalization, where translated strings reorder arguments). For an attacker it is the mechanism for targeting a specific slot: combined with the previous topic's technique, %7$s dereferences exactly slot 7 as a pointer, and %7$n (next topic) writes through exactly slot 7.
Why this matters so much: exploitation requires the attacker to control which slot the dangerous specifier operates on, because the slot they can fill with a chosen address (the one holding their own input) is at a specific, fixed offset. Direct parameter access lets them name that offset directly. The workflow is:
- Find the offset. Send a recognizable marker followed by a dump, e.g.
AAAAAAAA %p %p %p ...or betterAAAAAAAA %1$p %2$p %3$p ..., and see at which position the marker (0x4141414141414141) appears. That position is the offset at which the attacker's input lands in the argument sequence. pwntools'fmtstrhelpers and the common%p.%p.%pprobe automate this. - Use the offset. If the marker appeared at position 6, then
%6$sdereferences the attacker-supplied address, and%6$nwrites to it. Every subsequent read or write uses that known offset.
Finding and using the offset with direct parameter access
───────────────────────────────────────────────────────────────
Probe: "AAAAAAAA" + "%1$p %2$p %3$p %4$p %5$p %6$p"
Output: ... 0x4141414141414141 appears at %6$p
=> the input buffer is argument slot 6.
Now target precisely:
read : <addr bytes> + "%6$s" -> *addr printed
write: <addr bytes> + "%6$n" -> value written to addr
No wasted specifiers; the operation lands on exactly slot 6.
The security lesson is that direct parameter access converts format-string bugs from "interesting crash" to "reliable, surgical read/write primitive." It is also why defenses cannot rely on the awkwardness of the attack: there is no "too deep to reach" slot, because %N$ reaches any depth directly. The only robust defenses are preventing attacker control of the format string in the first place (constant formats, -Wformat-security) and, as a mitigation, libc's %n restrictions and FORTIFY checks covered in 5.4.
Examples
# Step 1: find the offset where your input lands in the argument sequence.
python3 -c 'print("AAAAAAAA" + "".join(f"%{i}$p " for i in range(1,11)))' | ./leak
# output shows 0x4141414141414141 at, say, the 6th printed value ->
# the input buffer is argument slot 6. (Alignment may shift this; adjust.)
# Step 2: use the discovered offset for surgical reads (conceptual, CTF).
from pwn import p64
OFFSET = 6 # found above
GOT_PUTS = 0x404018 # address to read (a GOT entry, e.g.)
# Arbitrary read of GOT_PUTS using direct parameter access:
read_payload = p64(GOT_PUTS) + f"%{OFFSET}$s".encode()
# -> %6$s derefs slot 6 (our p64 address) and prints the bytes there.
# pwntools can also build these automatically once the offset is known:
# from pwn import fmtstr_payload
# fmtstr_payload(OFFSET, {target_addr: value}) # builds a %n write chain
Why the offset can shift by alignment
───────────────────────────────────────────────────────────────
If the input buffer is not 8-byte aligned relative to the argument
frame, the chosen address bytes may straddle two slots. The fix is to
pad the input so the address lands cleanly on a single slot boundary,
e.g. prefix a few bytes so p64(addr) begins exactly at slot OFFSET.
Key Takeaways
- Direct parameter access (
%N$) fetches argument number N directly, eliminating the need to consume intervening slots and making format-string exploitation precise. - It lets the attacker target the specific slot that holds their own input (and thus a chosen address), which is the key to turning the read/write primitives into arbitrary ones.
- The standard workflow is a two-step: probe with a marker plus positional specifiers to find the offset where the input lands, then use
%OFFSET$s/%OFFSET$nto operate on it. - Because
%N$reaches any depth, no slot is "too deep" to attack, so defenses cannot rely on the difficulty of reaching a slot. - Alignment can shift the effective offset, requiring input padding so a chosen address lands cleanly on a single slot boundary.
🧪 Practice
- Using a study harness, determine the exact argument offset at which your input lands, using positional
%N$pprobes, and explain how alignment could change the answer.- Construct (for a CTF target you are authorized to use) a direct-parameter-access read payload that dereferences a chosen address at the discovered offset.
- Explain why
%N$makes the "too many specifiers to reach the slot" obstacle irrelevant, and what that implies for defense.- Interview-style: "How would you reliably determine the argument offset at which attacker input appears in a format-string bug, and why is that offset the linchpin of the exploit?" Hint: think about sending a unique marker with positional specifiers and matching the marker's bytes in the output, then recall which operations (read, write) must land on exactly that slot.
Arbitrary Write with %n
Theory
Reading memory is powerful, but the step that makes format strings a route to full control is the write primitive, and it comes from a single specifier: %n. Where every other directive consumes an argument and produces output, %n does the reverse — it consumes a pointer argument and writes to it the number of characters printf has output so far. It produces no visible output; its entire effect is the store. Understanding %n is understanding how a logging bug becomes an arbitrary memory write.
The mechanism combines two things already established. First, from direct parameter access, the attacker can make %n operate on a slot whose contents they control — specifically, a target address placed in their own input. So %6$n writes to the address the attacker put in slot 6: that is write-where fully under attacker control. Second, the value written is the output-character count, and the attacker controls that too, because printf's width field lets them pad the output to any length. Printing %100x outputs 100 characters (a number padded to width 100), so a preceding %100x%6$n makes the count 100+ at the moment %n fires, writing roughly 100 to the target. That is write-what: by choosing the width, the attacker chooses the value.
Putting both together: [target address in input] %<width>x %<offset>$n writes a chosen value to a chosen address. This is a complete write-what-where primitive built entirely from a careless printf.
%n as a write-what-where primitive
───────────────────────────────────────────────────────────────
input = p64(TARGET) + "%<N>x" + "%6$n"
│ │ │
│ │ └ write the running count to the
│ │ address in slot 6 = TARGET
│ └ pad output to N chars -> the count becomes ~N
└ TARGET address, placed where slot 6 will read it
result: *TARGET = N (both the value N and the address TARGET chosen
by the attacker)
Two practical refinements, both defensively relevant:
- Writing a full 64-bit value needs byte-by-byte writes. You cannot print 2^47 characters to write a large address. The standard technique writes the target value one or two bytes at a time using the length modifiers
%hhn(write a single byte) and%hn(write two bytes), each aimed at consecutive addresses (TARGET,TARGET+1, …), with the width adjusted so the cumulative count equals each byte's desired value. pwntools'fmtstr_payloadcomputes these staggered widths automatically. This matters because it means a%nexploit is a short burst of writes, not one giant one. - The targets are the same high-value pointers as every other technique. A
%nwrite typically overwrites a GOT entry (redirecting a library call tosystem, as in the program-analysis chapter's GOT-overwrite discussion), a saved return address, or a function pointer. Combined with a prior leak (defeating ASLR) and the GOT being writable (partial RELRO),%nturns a format string into code execution.
This is precisely why %n has been progressively locked down, and those mitigations are the defensive payoff of understanding it: modern glibc refuses %n when the format string is in writable memory (the _FORTIFY_SOURCE and %n-in-writable-segment checks), some libraries disable %n entirely, and Windows' printf dropped %n support by default long ago. Full RELRO (5.4) removes the GOT as a target. And the root-cause fix — never let attacker data be the format string — eliminates the primitive outright.
Examples
Byte-by-byte write to place a full address (why %hhn is used)
───────────────────────────────────────────────────────────────
GOAL: write 0x00007ffff7a52290 to TARGET (8 bytes).
Printing 2^47 chars is impossible, so write one byte at a time:
addr TARGET+0 := 0x90 via %hhn after padding count to 0x90
addr TARGET+1 := 0x22 via %hhn after padding count to 0x122 (mod 256)
addr TARGET+2 := 0xa5 ...
...
Each %hhn writes the LOW byte of the running count. Widths are chosen so
the cumulative count's low byte equals each target byte in turn.
(pwntools fmtstr_payload computes the staggered widths automatically.)
# Building a %n write with pwntools (conceptual, authorized CTF target).
from pwn import fmtstr_payload
OFFSET = 6 # argument offset of our input (found earlier)
GOT_EXIT = 0x404040 # a writable GOT entry to redirect
WIN_FUNC = 0x4011d6 # address to write there (from a leak)
# fmtstr_payload emits the %<width>c ... %hhn sequence with correct widths:
payload = fmtstr_payload(OFFSET, {GOT_EXIT: WIN_FUNC}, write_size='byte')
# Sending `payload` makes printf write WIN_FUNC into GOT_EXIT; the program's
# next call through that GOT slot transfers control to WIN_FUNC.
# This is the format-string analogue of the GOT overwrite from chapter 4.
# Mitigations that specifically target %n (verify on your toolchain).
# 1. FORTIFY refuses %n in a writable-segment format string at runtime:
gcc -O2 -D_FORTIFY_SOURCE=2 fmt.c -o fmt_fortified
# *** %n in writable segment detected *** (abort at runtime)
# 2. The compile-time warning that prevents the bug entirely:
gcc -Wformat -Wformat-security fmt.c
# 3. Full RELRO removes the GOT as a %n target:
gcc -Wl,-z,relro,-z,now fmt.c -o fmt_relro
Key Takeaways
%nwrites the count of characters output so far to a pointer argument, producing no output; it is the write primitive that turns a format-string bug into memory corruption.- Combined with direct parameter access (choose the address) and the width field (choose the value via the output count),
%nis a complete write-what-where primitive. - Large values are written byte-by-byte with
%hhn/%hnat consecutive addresses, with staggered widths computed so each byte comes out right — so a%nexploit is a short burst of small writes. - The usual targets match every other technique: GOT entries, saved return addresses, and function pointers, which with a prior ASLR-defeating leak yield code execution.
%nis heavily mitigated — glibc's writable-segment refusal, FORTIFY, Windows disabling it by default, and full RELRO removing the GOT target — but the root fix is never letting attacker data be the format string.
🧪 Practice
- Explain how the width field (e.g.
%100x) controls the value that a following%nwrites, and why%hhnis needed to write a full 64-bit address.- For an authorized CTF target, use
fmtstr_payloadto construct a payload that writes a chosen value into a chosen GOT entry, and explain each component of the generated string.- Compile a vulnerable program with
-D_FORTIFY_SOURCE=2and demonstrate that a%nin a writable-segment format string is refused at runtime; explain what FORTIFY checked.- Interview-style: "Describe how a format-string
%nwrite and the GOT-overwrite technique from binary analysis combine to achieve code execution, and name two mitigations that each independently break the combination." Hint: connect write-what-where to redirecting a library call, then recall which linker option makes the GOT read-only and which libc check refuses%nin writable memory.
Leaking Stack Contents and Canaries
Theory
This topic focuses on one specific, high-value use of the format-string read primitive: extracting the stack canary and other stack secrets. It deserves its own treatment because the canary is the mitigation standing between a stack overflow and a return-address overwrite, and a format-string leak is the cleanest way to defeat it — a concrete illustration of how two vulnerabilities combine to bypass a defense that neither could beat alone.
Recall the canary from the program-analysis chapter: the compiler places a random value (loaded from fs:0x28) between a function's local buffers and its saved return address, and checks it in the epilogue; if a linear overflow corrupted it, the check fails and the program aborts before returning. The canary's security rests entirely on the attacker not knowing its value — if they know it, they can include the correct canary in their overflow payload, so the check passes and the overwrite of the return address proceeds undetected.
A format-string read is a direct way to learn the canary, because the canary is on the stack, sitting in one of the argument slots printf's cursor walks through. The attacker dumps slots with %p (or targets one with %N$p) and identifies the canary by its distinctive signature:
- It is a large 64-bit value with high entropy (looks random), unlike pointers which cluster in recognizable ranges.
- Its least-significant byte is
0x00on glibc x86-64. This is deliberate: the null low byte stops string functions from reading or copying the canary, but it also makes the canary instantly recognizable in a slot dump — a random-looking 8-byte value ending in00.
Recognizing the canary in a %p stack dump
───────────────────────────────────────────────────────────────
%p dump slots:
0x7ffff7... <- cluster near 0x7fff... : a stack pointer
0x5555555... <- cluster near 0x5555... : a PIE code pointer
0x7f3a9c1e4b2d__00 <- high-entropy, ends in 00 : THE CANARY
0x0000000000000001 <- small integer : a local variable
The trailing 00 and randomness make the canary unmistakable.
Once leaked, the canary enables the overflow the canary was meant to stop. The combined attack is: use the format string to read and record the canary, then (in the same session, before the process re-randomizes) send a stack-overflow payload that overwrites the buffer, writes the correct canary value back into the canary's position, and then overwrites the saved return address. The epilogue check sees the expected canary and passes; control is hijacked anyway. This is the canonical "leak-then-smash" that every discussion of canaries notes as their key limitation.
The same dump simultaneously yields the other ingredients a full exploit needs: a PIE code pointer (recognizable by its address range) reveals the binary's randomized base, and a libc pointer reveals libc's base — both defeating ASLR, the subject of the next topic. So a single format-string leak often hands the attacker everything at once: the canary, the PIE base, and the libc base. For defenders, this is the clearest argument for why mitigations must be layered and why preventing information disclosure is as important as preventing writes — a leak-only bug quietly dismantles canaries and ASLR together.
Examples
# Leak the canary by dumping stack slots (study harness with canary ON).
gcc -g -O0 -fstack-protector-all leak.c -o leak_canary
python3 -c 'print("%p "*20)' | ./leak_canary
# Scan the output for the high-entropy value ending in 00 -> the canary.
# Also note the 0x5555... (PIE base) and 0x7ffff7... (libc) pointers.
# Leak-then-smash structure (conceptual, authorized CTF target only).
from pwn import process, u64, p64
io = process("./leak_canary")
# 1. LEAK: read the canary via a targeted positional specifier.
io.sendline(b"%15$p") # offset 15 found to hold the canary
canary = int(io.recvline().strip(), 16)
assert canary & 0xff == 0 # sanity: glibc canary ends in 0x00
# 2. SMASH: overflow, but restore the correct canary so the check passes.
payload = b"A" * BUF_TO_CANARY # filler up to the canary slot
payload += p64(canary) # the CORRECT canary -> epilogue passes
payload += b"B" * 8 # saved rbp
payload += p64(RET_TARGET) # overwritten return address
io.sendline(payload)
# The canary check succeeds (value matches), yet rip is still hijacked.
Key Takeaways
- A stack canary's security depends entirely on the attacker not knowing its value; learning it lets the attacker replay the correct canary in an overflow so the epilogue check passes.
- A format-string read leaks the canary directly because it sits on the stack among the slots
printf's cursor walks; it is recognizable as a high-entropy 64-bit value ending in0x00. - The canary's null low byte, intended to stop string functions from copying it, also makes it unmistakable in a slot dump.
- The combined "leak-then-smash" attack reads the canary, then overflows while writing the correct canary back and overwriting the return address — hijacking control despite the canary.
- The same dump typically also yields a PIE code pointer and a libc pointer (defeating ASLR), so one leak can dismantle the canary and ASLR together — the strongest argument for layered defenses and for treating disclosure as seriously as writes.
🧪 Practice
- Compile a program with
-fstack-protector-all, leak the stack with%pspecifiers, and identify the canary by its entropy and trailing0x00; distinguish it from the PIE and libc pointers in the same dump.- Explain precisely why knowing the canary value lets a linear stack overflow succeed despite the stack-protector check.
- Describe why the combined attack must leak and smash within the same process lifetime (for a non-forking server) rather than across restarts.
- Interview-style: "A stack canary is enabled, yet an attacker with a format-string bug still overwrites the return address successfully. Explain how, and state what property of the canary made it leakable." Hint: consider where the canary lives relative to
printf's argument cursor, how the attacker recognizes it in a dump, and what they place in the canary's position during the subsequent overflow.
Defeating ASLR via Information Leaks
Theory
Address Space Layout Randomization is the mitigation that makes every address-dependent technique in this chapter uncertain: by randomizing the base addresses of libraries, the stack, the heap, and (for PIE) the executable itself at each run, it denies the attacker the absolute addresses that ret2libc, ROP, and %n writes all require. This topic is about the universal counter: an information leak that discloses even one runtime address, from which the attacker reconstructs the layout. Understanding this is understanding why "ASLR is enabled" is necessary but not sufficient, and why eliminating information disclosure is a first-class security goal.
The key insight that makes leaks so effective is that ASLR randomizes a base, not the internal structure. Everything within a single loaded module keeps its relative offsets — the distance from libc's base to system, or to a "/bin/sh" string, or to a useful gadget, is fixed and known (it depends only on the libc build, which the attacker can identify). So leaking a single pointer into libc lets the attacker compute libc's base (leaked address minus the known offset of whatever it points to), and from that base every other libc address follows by adding known offsets. One leak collapses the randomization of an entire module.
One leak defeats a whole module's ASLR
───────────────────────────────────────────────────────────────
libc loaded at random base B (unknown to attacker).
Known from the libc build (fixed offsets):
offset(system) = 0x50d60
offset(/bin/sh) = 0x1d8678
offset(a gadget) = 0x2a3e5
Leak ANY libc pointer, e.g. a return address pointing at
__libc_start_main+128 = B + 0x29d90 + 128.
Then: B = leaked_value - (0x29d90 + 128)
And now system = B + 0x50d60, etc. -- the entire library is de-randomized.
Leaks come from many sources, and recognizing them is defensively useful because each is a bug class to prevent:
| Leak source | How it discloses an address |
|---|---|
Format string (%p/%s) | dumps stack/pointer values directly (this section) |
| Uninitialized memory | a struct's padding or an unset field returned to the attacker contains a stale pointer |
| Over-read (Heartbleed-style) | a trusted length returns adjacent memory containing pointers |
| Error messages / debug output | a crash log or verbose error prints an address |
| Timing / oracle side channels | repeated probing narrows the base statistically |
The "leaked pointer type" determines which base you recover. A stack pointer reveals the stack location (useful for stack pivots or placing data). A libc pointer (a return address into libc, a function pointer, or a GOT entry's resolved value) reveals libc's base — the most valuable, since libc has the gadgets and system. A PIE code pointer (a return address into the program, a code pointer in a vtable) reveals the executable's base, needed to use the program's own gadgets and symbols when PIE is enabled.
The defensive conclusions are sharp. First, ASLR and leak-prevention are a package: ASLR without disclosure control is routinely bypassed, which is why secure coding (5.5) treats uninitialized-memory and over-read bugs as seriously as writes. Second, entropy matters: 32-bit ASLR had too few bits and was brute-forceable (especially on forking servers), while 64-bit ASLR provides enough entropy that brute force is impractical — making the single-leak path the realistic one. Third, mitigations that re-randomize (re-exec per connection) or that make leaks less useful (position-independent everything, so there is no fixed-address module to anchor to) directly attack this step.
Examples
# Reconstructing libc's base from one leaked pointer (conceptual, CTF).
# Offsets are specific to the exact libc build; identify it first (e.g. with
# a libc-database lookup from a couple of leaked values).
LEAKED = 0x7f3a9c1e9d90 # a leaked return address into libc
OFF_IN_LIBC = 0x29d90 + 128 # known offset of what it points at
libc_base = LEAKED - OFF_IN_LIBC # the module base is now known
system = libc_base + 0x50d60 # every libc address follows by adding
binsh = libc_base + 0x1d8678 # its fixed offset to the recovered base
pop_rdi = libc_base + 0x2a3e5 # a gadget inside libc
# The attacker now has concrete addresses to build a ROP chain or ret2libc,
# despite ASLR -- because only the base was random, not the offsets.
# Observe that ASLR randomizes the BASE but preserves internal offsets.
for i in 1 2 3; do
setarch "$(uname -m)" -R cat /proc/self/maps | grep 'libc' | head -1 # ASLR off: same
done
for i in 1 2 3; do
cat /proc/self/maps | grep -m1 'libc' # ASLR on: base differs each run...
done
# ...but within any single run, system is always base + the same offset.
# Check and set the system ASLR level (the knob the mitigation exposes):
cat /proc/sys/kernel/randomize_va_space # 0=off, 1=partial, 2=full (default)
Key Takeaways
- ASLR randomizes module base addresses, denying attackers the absolute addresses that ret2libc, ROP, and
%nwrites require. - Because only the base is randomized while internal offsets stay fixed, leaking a single pointer into a module lets the attacker compute that module's base and then every address within it.
- A libc leak is the most valuable (it yields
system,"/bin/sh", and gadgets); stack and PIE leaks yield the stack and executable bases respectively. - Leaks arise from format strings, uninitialized memory, over-reads, verbose errors, and side channels — so ASLR and disclosure-prevention are a package, and leak bugs are as serious as write bugs.
- 64-bit ASLR has enough entropy to make brute force impractical, so the realistic bypass is a single information leak; re-randomizing per connection and eliminating fixed-address modules attack that path.
🧪 Practice
- Given a leaked libc return address and the known offset it points at, compute libc's base and then the address of
system, showing the arithmetic.- Observe
/proc/self/mapsacross several runs with ASLR on and confirm the libc base changes while the offset of a chosen symbol within libc stays constant.- For each leak source in the table, name a secure-coding practice from later in this chapter that would prevent it.
- Interview-style: "Explain why a single leaked pointer is usually enough to defeat ASLR for an entire shared library." Hint: consider what ASLR actually randomizes versus what stays fixed within a loaded module, and how the attacker uses known build-specific offsets to turn one address into all of them.
Combining Leaks with Overflow Primitives
Theory
The final format-string topic is synthesis: how the read and write primitives, the leaks, and the overflow techniques from the previous section combine into a complete exploit against a fully mitigated target. This is the capstone that ties the chapter together, and its lesson is the central strategic truth of exploitation: modern exploits are multi-stage pipelines, and each mitigation forces an additional, independent stage. No single primitive suffices; capability is assembled.
The canonical pipeline against a target with all standard mitigations (canary, NX, PIE, full ASLR) runs in stages, each resolving one defense:
- Leak to defeat the canary. Use the format-string read to disclose the stack canary (the leaking-canaries topic), so a later overflow can replay it.
- Leak to defeat ASLR. From the same or an additional read, disclose a libc pointer and a PIE pointer, establishing both base addresses (the defeating-ASLR topic).
- Build the control primitive for NX. Because NX forbids injected code, construct a ROP chain or ret2libc call using the now-known addresses (the code-reuse topics).
- Deliver via the overflow. Use the stack overflow — including the correct canary and the gadget/return addresses computed from the leaks — to overwrite the saved return address and trigger the chain.
The structure is a dependency chain, and recognizing it is the key defensive insight. Stage 3 cannot happen without the addresses from stage 2; stage 4 cannot succeed without the canary from stage 1. Each mitigation (canary, ASLR, NX) is individually bypassable, but each bypass requires its own capability, and the capabilities must be acquired in order. This is why layered mitigations work even though each is defeatable: the attacker must defeat all of them, and the probability of chaining every stage reliably — especially remotely, blind, and without crashing — is far lower than defeating any one. A missing leak primitive (no way to read memory) can make the whole pipeline impossible even when a powerful write primitive exists, because stage 3 needs addresses that only a read can supply.
The full pipeline as a dependency chain
───────────────────────────────────────────────────────────────
[fmt-string READ] ──► canary value ───────────────┐
│ │
└──► libc pointer ──► libc base ──► gadgets ─┤
└──► PIE pointer ──► prog base ──► symbols ┤
▼
build ROP chain (defeats NX)
│
[stack OVERFLOW] ◄── replay canary + chain ────────┘
│
▼
control flow hijacked -> goal
Remove ANY input (e.g. no read primitive) and the chain breaks.
Two forms of combination are worth distinguishing, because they reflect how real bugs present:
- One bug providing multiple primitives. A format string alone gives both read (leak) and write (
%n), so an attacker can sometimes complete the whole pipeline with a single vulnerability — leak the canary and bases with%p, then overwrite a GOT entry with%n. This is why format strings are rated so severely: one bug is both the leak and the weapon. - Chaining separate bugs. More often, real exploits chain distinct vulnerabilities: an information-disclosure bug supplies the leak, and a separate memory-corruption bug supplies the write. The infamous browser and kernel exploit chains are long sequences of such bugs, each handing the next the capability it needs. The defensive corollary is that a "mere" info-leak bug, dismissed as low severity in isolation, is often the enabling first link that makes a separate corruption bug exploitable.
The overarching takeaway for a defender or secure developer is to think in terms of the whole chain: assess a bug not only by what it does alone but by what stage it could supply in a pipeline, prevent information disclosure as rigorously as memory corruption, and rely on defense-in-depth precisely because it forces the attacker to win every stage rather than just one.
Examples
# A single format-string bug completing the whole pipeline (conceptual, CTF).
from pwn import process, u64, p64, fmtstr_payload
io = process("./target") # authorized, deliberately vulnerable binary
# Stage 1+2: ONE read pass leaks canary, libc base, and PIE base.
io.sendline(b"%p " * 30)
slots = [int(x, 16) for x in io.recvline().split() if x != b"(nil)"]
canary = next(v for v in slots if v & 0xff == 0 and v > 0xffff) # ends in 00
libc_base = find_libc_ptr(slots) - LIBC_LEAK_OFFSET # (helper)
pie_base = find_pie_ptr(slots) - PIE_LEAK_OFFSET
# Stage 3: build a ret2libc call now that addresses are known (defeats NX).
system = libc_base + SYSTEM_OFF
binsh = libc_base + BINSH_OFF
pop_rdi= pie_base + POP_RDI_OFF
# Stage 4: the overflow replays the canary, then the ROP chain.
payload = b"A"*BUF + p64(canary) + b"B"*8 # filler + canary + saved rbp
payload += p64(pop_rdi) + p64(binsh) + p64(system)# ret2libc, addresses from leaks
io.sendline(payload)
io.interactive()
# Every value sent was DERIVED from the leak; without stage 1+2, stage 4 fails.
Why a "low-severity" info leak is often the linchpin
───────────────────────────────────────────────────────────────
Bug A: information disclosure (leaks a libc pointer) -- rated "medium"
Bug B: stack overflow, NX+ASLR+canary on -- "not exploitable
alone" (no leak)
A + B together: A supplies the addresses B needs -> full RCE.
Triage that scores A and B independently misses the chain. Assess bugs
by the STAGE they could supply, not only by their standalone impact.
Key Takeaways
- Modern exploits are multi-stage pipelines in which each mitigation forces an additional, independent stage: leak the canary, leak the bases (defeat ASLR), build a code-reuse chain (defeat NX), then deliver via the overflow.
- The stages form a dependency chain — the ROP chain needs the leaked addresses, the overflow needs the leaked canary — which is exactly why layered mitigations are effective even though each is individually bypassable.
- A format string is severe because one bug supplies both the read (leak) and the write (
%n), potentially completing the whole pipeline alone. - Real exploits more often chain separate vulnerabilities, so a "low-severity" information-disclosure bug is frequently the enabling first link that makes a separate corruption bug exploitable.
- Defensively, assess each bug by the stage it could supply in a chain, prevent disclosure as rigorously as corruption, and rely on defense-in-depth to force the attacker to win every stage.
🧪 Practice
- Draw the dependency chain for a full exploit against a canary+NX+PIE+ASLR target, and for each stage state which primitive supplies it and which mitigation it defeats.
- Explain how a single format-string bug can serve as both the leak and the write stage, and why that makes it more dangerous than a bug offering only one primitive.
- Given an information-disclosure bug rated "medium" in isolation and a separate overflow that is "not exploitable" alone under ASLR, explain how combining them changes the severity of both.
- Interview-style: "Argue, using the dependency-chain model, why defense-in-depth remains worthwhile even though a skilled attacker can bypass each individual mitigation." Hint: consider that the attacker must defeat every stage in sequence and that any missing primitive breaks the chain, then reason about the compounded reliability cost of chaining all stages remotely and blind.
<a id="54-exploit-mitigations"></a>
5.4 Exploit Mitigations
Having seen how each attack works, this section examines the defenses that were built to stop them, in the order they were developed and defeated; the recurring theme is that no single mitigation is sufficient, and that their value comes from being layered so an attacker must defeat all of them.
Stack Canaries and Their Bypasses
Theory
The stack canary (also called a stack cookie or stack guard) is the oldest widely deployed anti-exploitation mitigation, and it targets exactly one thing: the sequential stack buffer overflow that overwrites a saved return address. The name is a reference to the canary in a coal mine — a sacrificial sentinel placed where danger arrives first, whose death warns of the threat before it reaches the miners.
The mechanism, from the program-analysis chapter, is a random value placed by the compiler between the local buffers and the saved return address. Because a classic stack overflow writes sequentially from low to high addresses, it must pass through the canary's position to reach the return address — it cannot skip over it. The function's epilogue reloads the canary and compares it to the known-good value (stored in thread-local storage at fs:0x28); if they differ, the overflow corrupted the canary on its way to the return address, and the program calls __stack_chk_fail, which aborts. The attacker's overflow is detected before the corrupted return address is ever used.
The design details that make it effective:
- Randomness per process. The canary is generated at process start from a secure source, so it is unpredictable and differs between runs.
- The null byte. glibc's canary has a
0x00least-significant byte, so string functions (which stop at NUL) cannot easily copy a leaked canary, and an overflow using a string function cannot write a canary containing arbitrary bytes without including the null. - Selective application.
-fstack-protectorguards only functions with character arrays;-fstack-protector-strong(the common default) guards more;-fstack-protector-allguards everything at some cost.
But the canary protects only against one attack shape, and its bypasses are instructive because each reveals a limitation that another mitigation must cover:
| Bypass | Why the canary does not stop it |
|---|---|
| Information leak | if the attacker reads the canary (format string, over-read), they replay it in the overflow so the check passes (the leaking-canaries topic) |
| Non-sequential write | an arbitrary-write primitive (e.g. %n, a controlled index) overwrites the return address directly without touching the canary |
| Overwriting past the check | corrupting data used before the epilogue (a local function pointer, a loop bound) achieves control without ever reaching return |
| Heap / UAF bugs | the canary only guards the stack; heap overflows and use-after-free are entirely outside its scope |
| Brute force (forking servers) | on a fork-based server the canary does not re-randomize, so it can be guessed byte-by-byte via crash/no-crash oracles |
The defensive lesson is precise: the canary is cheap and effective against the specific, historically dominant bug (sequential stack overflow), but it is narrow. It does nothing against non-stack bugs, non-sequential writes, or an attacker who can read memory — which is exactly why it is one layer among several, and why its presence (checksec showing "Canary: Yes") is necessary but far from a guarantee.
Examples
// What the compiler inserts (conceptual, matching the program-analysis chapter).
void guarded(const char *input) {
unsigned long __canary = *(unsigned long *)(FS_BASE + 0x28); // load canary
char buf[64];
strcpy(buf, input); // a sequential overflow must cross __canary
// to reach the return address above buf
if (*(unsigned long *)(FS_BASE + 0x28) != __canary) // epilogue check
__stack_chk_fail(); // abort: the overflow was detected
// ... normal return only if the canary is intact ...
}
# Observe the canary and its bypass boundary.
gcc -fstack-protector-strong -g vuln.c -o vuln_canary
checksec --file=./vuln_canary | grep Canary # Canary: Yes
# A sequential overflow is caught:
./vuln_canary "$(python3 -c 'print("A"*100)')"
# *** stack smashing detected ***: terminated <- __stack_chk_fail fired
# But the canary does NOT help against a heap overflow or an arbitrary write:
# those never cross the stack canary, so the same binary offers no protection
# for bugs of that shape. Confirm which functions even have a canary:
objdump -d vuln_canary | grep -l 'fs:0x28' >/dev/null && echo "canary present in some funcs"
Key Takeaways
- The stack canary is a per-process random value placed between local buffers and the saved return address; a sequential overflow must corrupt it to reach the return address, and the epilogue check aborts when it is corrupted.
- Its effectiveness rests on randomness, a null low byte that resists string-function copying, and selective application via the
-fstack-protector*flags. - It protects only against the sequential stack overflow and is bypassed by information leaks (replay the canary), non-sequential/arbitrary writes (skip it entirely), pre-return corruption, and any non-stack bug.
- On forking servers the canary does not re-randomize, enabling byte-by-byte brute force.
- The canary is cheap and effective within its narrow scope but is necessarily one layer among several; its presence is necessary, not sufficient.
🧪 Practice
- Compile a vulnerable program with and without
-fstack-protector-strong, trigger a sequential overflow in each, and contrast the outcomes; identify the__stack_chk_failmessage.- Explain why a
%n-based arbitrary write can overwrite a return address without ever tripping the canary.- Describe a scenario where overwriting a local variable (not the return address) achieves control before the epilogue canary check runs.
- Interview-style: "A binary shows 'Canary: Yes' in checksec, yet it is exploited via a stack overflow. Name two distinct ways that is possible." Hint: consider an attacker who can first read memory, and separately an attacker whose write is not sequential from the buffer toward the return address.
Non-Executable Memory (DEP/NX)
Theory
Non-executable memory — called NX (No-eXecute) on the hardware bit, DEP (Data Execution Prevention) on Windows, and W^X as the principle — is the mitigation that killed classic code injection. It targets the original exploit shape directly: write shellcode into a buffer, jump to it, run it. NX makes the "run it" step impossible by ensuring the buffer's memory page is not executable.
The mechanism is a single permission bit per memory page, enforced by the CPU's memory-management unit. Historically x86 page tables lacked a dedicated execute-permission bit (a page was executable if it was readable), which is precisely why early exploits could run data. The NX bit (added to x86-64 page tables, and retrofitted to 32-bit via PAE) lets the OS mark a page non-executable; any attempt to fetch an instruction from it raises a fault and the process is killed. The OS applies this to mark the stack, the heap, and other data regions non-executable while keeping code regions (.text, libraries) executable — enforcing the W^X invariant: a page is writable or executable, but never both. Attacker-controlled data pages are writable, therefore not executable.
The analogy: NX separates the "recipe" from the "ingredients." Before NX, the CPU would happily cook (execute) anything in the pantry, so an attacker who slipped a malicious recipe among the ingredients got it cooked. NX enforces that only pages in the designated cookbook (code regions) are ever read as recipes; data pages are ingredients and are never cooked, no matter what is written there.
NX is extremely effective at what it does and essentially free (it is a hardware feature), which is why it is universally enabled. But — and this is the point that drives the rest of the chapter — NX stops code injection, not control-flow hijacking. The attacker can still overwrite a return address or function pointer; they simply cannot point it at their own injected bytes. So NX did not end exploitation; it redirected it toward code reuse — ret2libc and ROP — which execute only existing, legitimately executable code and therefore never violate NX. The entire code-reuse section exists because NX closed the injection door and attackers walked through the reuse window.
A few practical notes with defensive relevance:
- NX can be checked and must be present:
checksecreports "NX: enabled"; its absence (e.g. a binary linked with-z execstack, or an old JIT that marks pages RWX) is a serious weakness that re-enables injection. - JITs and NX. Just-in-time compilers (JavaScript engines, the JVM) legitimately need to write code and then execute it. They do this by writing to a writable page and then re-mapping it executable (
mprotect/VirtualProtecttoggling W then X, never both at once), preserving W^X. A JIT that keeps pages RWX is a classic weakness attackers target. - The
mprotectbypass. A ROP chain can callmprotectto make a data page executable and then jump to injected shellcode there — "turning off" NX for one page. This is why NX must be paired with ASLR (so the attacker cannot easily build the ROP chain) and CFI (so the control transfer is caught).
Examples
W^X: the same page cannot be both writable and executable
───────────────────────────────────────────────────────────────
region perms can hold code? can attacker write?
───────── ───── ────────────── ───────────────────
.text r-x yes (executed) no (read-only)
stack rw- NO (NX fault) yes
heap rw- NO (NX fault) yes
JIT page rw- then r-x (toggled, never rwx) by the JIT only
An attacker's shellcode on the stack/heap sits on an rw- page; fetching
an instruction from it faults -> SIGSEGV. Injection is dead.
# Verify NX and detect the dangerous RWX / execstack cases.
checksec --file=/bin/ls | grep NX # NX: enabled
readelf -lW ./prog | grep GNU_STACK # RW should appear, NOT RWE
grep 'rwxp' /proc/self/maps && echo "WARNING: W+X page present" || echo "W^X holds"
# A binary built to allow an executable stack (a self-inflicted weakness):
gcc -z execstack inj.c -o inj_bad # GNU_STACK becomes RWE
checksec --file=./inj_bad | grep NX # NX: disabled
# On this binary, classic code injection to the stack works again.
// The legitimate JIT pattern that preserves W^X (conceptual).
#include <sys/mman.h>
#include <string.h>
void *jit_emit(const unsigned char *code, size_t n) {
// 1. Allocate WRITABLE (not executable) memory and write the code.
void *p = mmap(NULL, n, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
memcpy(p, code, n);
// 2. Flip to EXECUTABLE (dropping write) -- never W and X simultaneously.
mprotect(p, n, PROT_READ | PROT_EXEC); // W^X preserved across the toggle
return p; // now safe to call
}
Key Takeaways
- NX/DEP marks data pages (stack, heap) non-executable via a per-page CPU permission bit, enforcing W^X: a page is writable or executable, never both.
- It directly defeats classic code injection — attacker-written bytes live on writable, non-executable pages, so fetching an instruction from them faults.
- NX stops code injection but not control-flow hijacking; the attacker can still redirect a return address or pointer, just not to injected code, which is what pushed exploitation toward ret2libc and ROP.
- JITs legitimately need W-then-X and preserve the invariant by toggling permissions (never RWX); a JIT or binary that keeps pages RWX re-enables injection.
- A ROP chain can call
mprotectto make a page executable, so NX must be layered with ASLR and CFI; verify NX withchecksecand treatexecstack/RWX pages as serious weaknesses.
🧪 Practice
- Use
checksecandreadelf -lWto confirm NX on a normal binary, then build one with-z execstackand show theGNU_STACKpermission change.- Explain, tracing the CPU's instruction fetch, why jumping to shellcode on the stack faults when NX is enabled.
- Describe how a JIT compiler can execute dynamically generated code without ever violating W^X.
- Interview-style: "NX is enabled on a target, yet an exploit still runs attacker-supplied shellcode. Explain the most likely technique." Hint: think about a ROP chain whose only job is to call a function that changes a data page's permissions, and what the chain does immediately afterward.
Address Space Layout Randomization
Theory
Address Space Layout Randomization (ASLR) is the mitigation that attacks the addresses every code-reuse technique depends on. After NX forced attackers from injection to reuse (ret2libc, ROP), those techniques still needed to know where the reusable code was — the address of system, of gadgets, of "/bin/sh". ASLR's premise is simple and powerful: if the attacker cannot predict those addresses, they cannot build the payload. It randomizes the base addresses of the process's memory regions at load time, so the layout differs on every run.
What gets randomized:
- Shared libraries (libc and others) — their load base is randomized, so
system's address changes each run. This is the most important, since libc holds the gadgets. - The stack — its base is randomized, so stack addresses (needed for pivots or to locate injected data) are unpredictable.
- The heap / mmap region — randomized, so heap addresses vary.
- The executable itself — only if it is compiled as a PIE (Position-Independent Executable); a non-PIE binary loads at a fixed address, leaving its own code and gadgets predictable even with ASLR on. This is why PIE is a separate, essential companion mitigation (next topic).
The analogy: ret2libc and ROP are like a burglar who has a floor plan of the building and knows exactly which door leads to the vault. ASLR reshuffles the building's layout every night, so yesterday's floor plan is useless. The burglar must first obtain a current floor plan — which is the information leak.
ASLR's strength is measured in entropy — the number of random bits in each base address, i.e. how many possible positions there are. This is the crux of its effectiveness and its limits:
- 32-bit ASLR was weak. With only ~8–16 bits of entropy for libc, an attacker could brute-force the base in seconds to hours, especially against a forking server where failed guesses do not re-randomize. 32-bit ASLR was routinely defeated.
- 64-bit ASLR is strong. With ~28–30 bits of entropy or more, brute force is impractical (billions of attempts, each risking a crash). On 64-bit, the realistic bypass is not brute force but a single information leak (the defeating-ASLR topic): one leaked pointer collapses a module's randomization because offsets within the module are fixed.
The two dominant bypass categories, both already developed in this chapter, define the defensive priorities:
| Bypass | Counter |
|---|---|
| Information leak | eliminate disclosure bugs (uninitialized memory, over-reads, format strings); this is the main path on 64-bit |
| Brute force | adequate entropy (64-bit) plus re-randomization per connection (re-exec, not just fork) |
| Non-randomized region | compile as PIE so the executable is randomized too; avoid fixed-address mappings |
The defensive conclusions are that ASLR and NX are a matched pair (NX forces reuse, ASLR hides what to reuse), that ASLR without PIE leaves the executable exploitable, and — most importantly — that ASLR's real-world strength depends entirely on the absence of information leaks, which is why preventing disclosure is a first-class secure-coding goal.
Examples
# Watch ASLR randomize bases across runs (and the knob that controls it).
cat /proc/sys/kernel/randomize_va_space # 2 = full ASLR (Linux default)
for i in 1 2 3; do cat /proc/self/maps | grep -m1 'libc'; done
# base differs each run: 7f2a..., 7f8c..., 7f05... (libc is relocated)
# Disable ASLR for debugging/study ONLY (never in production):
setarch "$(uname -m)" -R ./prog # -R = no randomization for this run
echo 0 | sudo tee /proc/sys/kernel/randomize_va_space # system-wide off (study)
# Measure entropy roughly: how many bits of the libc base vary?
for i in $(seq 20); do cat /proc/self/maps | grep -m1 libc | cut -d- -f1; done | sort -u | wc -l
Why 64-bit ASLR shifts the attacker from brute force to leaks
───────────────────────────────────────────────────────────────
32-bit: libc base has ~8-16 bits entropy
-> at most ~65k possibilities -> brute-forceable (esp. forking)
64-bit: libc base has ~28-30 bits entropy
-> hundreds of millions of possibilities, each wrong guess may
CRASH the process -> brute force impractical
-> attacker needs ONE leaked pointer instead (offsets are fixed)
Hence: on modern targets, "defeat ASLR" means "find an info leak".
Key Takeaways
- ASLR randomizes the base addresses of libraries, stack, heap, and (only if PIE) the executable at load time, denying attackers the fixed addresses that ret2libc and ROP require.
- Its strength is measured in entropy: 32-bit ASLR had too few bits and was brute-forceable, while 64-bit ASLR has enough entropy that brute force is impractical.
- On 64-bit the realistic bypass is a single information leak, because only module bases are randomized while internal offsets stay fixed — one leaked pointer de-randomizes a whole module.
- A non-PIE executable loads at a fixed address even with ASLR on, leaving its own code and gadgets predictable, which is why PIE is an essential companion.
- ASLR and NX are a matched pair, and ASLR's real-world strength depends on the absence of information-disclosure bugs, making leak prevention a first-class goal.
🧪 Practice
- Observe
/proc/self/mapsacross several runs to confirm the libc, stack, and heap bases are randomized, and usesetarch -Rto show the layout becoming fixed.- Explain quantitatively why 32-bit ASLR was brute-forceable while 64-bit ASLR generally is not, in terms of entropy and the cost of a wrong guess.
- Compile a program as non-PIE and show, using
/proc/self/maps, that its own code segment is not randomized even with system ASLR enabled.- Interview-style: "On a modern 64-bit system with full ASLR, why do attackers rarely attempt to brute-force addresses, and what do they do instead?" Hint: compare the number of possible base addresses and the consequence of a wrong guess against the effect of leaking a single pointer when intra-module offsets are fixed.
Position Independent Executables and RELRO
Theory
This topic covers two mitigations that close specific gaps the previous ones leave open: PIE, which extends ASLR to the executable itself, and RELRO, which protects the GOT from the overwrite attacks seen in the program-analysis and format-string chapters. They are grouped because both are linker-level decisions that complete the ASLR/NX picture.
PIE (Position-Independent Executable) addresses a hole in ASLR. ASLR randomizes libraries and the stack, but historically the main executable loaded at a fixed address (0x400000 on x86-64). That meant the program's own code — including any gadgets in it and its PLT stubs — was at predictable addresses even with ASLR fully enabled, so an attacker could build a ROP chain from the executable's own code without needing any leak. PIE fixes this by compiling the executable as position-independent (like a shared library), so its load base is randomized too. With PIE, nothing is at a predictable address, and the attacker needs a leak to use even the program's own gadgets. The cost is a small performance overhead from position-independent addressing (RIP-relative, from the program-analysis chapter) and an extra randomized base to defeat — which is the point.
The telltale sign, from the ELF topic, is the binary's type: a PIE reports as ET_DYN (a shared object) with a tiny entry point, because its real base is chosen at load time; a non-PIE is ET_EXEC with a fixed entry point.
RELRO (RELocation Read-Only) addresses the GOT-overwrite primitive. Recall from the program-analysis chapter that the GOT (Global Offset Table) holds the resolved addresses of imported library functions, and that with lazy binding it must be writable — making it a prime target for an arbitrary write (a %n, a heap overflow) that redirects a library call to system. RELRO hardens this:
| RELRO mode | Flag | Effect | GOT overwrite? |
|---|---|---|---|
| No RELRO | -z norelro | GOT fully writable | trivially available |
| Partial RELRO | -z relro (common default) | non-PLT GOT and .init_array made read-only after startup; .got.plt stays writable for lazy binding | still available on .got.plt |
| Full RELRO | -z relro -z now | all symbols resolved eagerly at startup, then the entire GOT is mapped read-only | eliminated |
Full RELRO resolves every imported symbol at program start (eager binding) and then marks the whole GOT read-only, so a later write to a GOT entry faults. The cost is slightly slower startup (all symbols resolved up front, even unused ones), which is why partial RELRO — leaving .got.plt writable for lazy binding — is a common default and full RELRO is the recommended setting for security-sensitive programs.
Together, PIE and full RELRO remove two specific attacker conveniences: PIE denies fixed-address gadgets in the executable, and full RELRO denies the GOT as a write target. Neither is sufficient alone, and both illustrate the chapter's theme — each closes one door, forcing the attacker to find another (a leak for PIE, a different write target for RELRO).
Examples
# PIE: the ELF type tells you whether the executable is randomized.
gcc vuln.c -o vuln_pie # PIE is the default on modern toolchains
gcc -no-pie vuln.c -o vuln_nopie
readelf -h vuln_pie | grep Type # Type: DYN (Position-Independent Executable)
readelf -h vuln_nopie | grep Type # Type: EXEC (fixed load address)
for b in vuln_pie vuln_nopie; do # confirm randomization behaviour:
echo "$b:"; for i in 1 2; do setarch "$(uname -m)" cat /proc/self/maps >/dev/null; done
done
checksec --file=./vuln_pie | grep PIE # PIE: enabled
# RELRO: compare the three modes and their GOT protection.
gcc -Wl,-z,norelro vuln.c -o v_no
gcc -Wl,-z,relro vuln.c -o v_partial
gcc -Wl,-z,relro,-z,now vuln.c -o v_full
for b in v_no v_partial v_full; do printf '%-10s ' "$b"; checksec --file=./$b 2>/dev/null | grep -o 'RELRO:[^|]*'; done
readelf -d v_full | grep -E 'BIND_NOW|FLAGS' # BIND_NOW present => eager => full
What full RELRO changes about the GOT
───────────────────────────────────────────────────────────────
Partial RELRO: Full RELRO:
.got r-- (read-only) .got r--
.got.plt rw- (WRITABLE!) ◄──── .got.plt r-- (read-only after init)
^ an arbitrary write here redirects ^ a write here now FAULTS;
a library call -> system the GOT-overwrite primitive is gone
Cost: full RELRO resolves all imports at startup (slower launch).
Key Takeaways
- PIE extends ASLR to the main executable by compiling it position-independent, so its load base is randomized and its own code/gadgets are no longer at fixed addresses; without PIE, ASLR leaves the executable predictable.
- A PIE reports as
ET_DYNwith a small entry point; a non-PIE isET_EXECwith a fixed entry point. - RELRO protects the GOT: partial RELRO (common default) leaves
.got.pltwritable for lazy binding, so GOT overwrite remains possible, while full RELRO (-z relro -z now) resolves all symbols eagerly and maps the entire GOT read-only, eliminating the primitive. - Full RELRO's cost is slightly slower startup from eager symbol resolution, which is the recommended trade for security-sensitive programs.
- PIE and full RELRO each close one specific door — fixed-address gadgets and the GOT write target — reinforcing that mitigations are layered, not individually sufficient.
🧪 Practice
- Build a program as PIE and non-PIE, confirm the ELF
Typedifference withreadelf -h, and explain why a non-PIE binary offers an attacker fixed-address gadgets despite ASLR.- Build the same source with no, partial, and full RELRO, and use
checksecandreadelf -dto document the GOT protection and the presence ofBIND_NOW.- Explain precisely why partial RELRO still permits a GOT-overwrite attack while full RELRO does not, naming the section whose permissions differ.
- Interview-style: "A latency-sensitive service uses partial RELRO to avoid eager-binding startup cost. What attack remains available, and what alternative would you propose to keep both startup speed and GOT protection?" Hint: identify which GOT section stays writable under partial RELRO and what a
%nor heap write could do to it, then consider whether the startup cost of full RELRO is actually significant for a long-running service.
Control Flow Integrity
Theory
Control Flow Integrity (CFI) is the mitigation aimed squarely at code reuse — ret2libc, ROP, and JOP. The previous mitigations attack the preconditions of those attacks (NX removes injected code, ASLR hides addresses), but CFI attacks the mechanism itself: it enforces that the program's control flow follows only paths the original code intended, so an attacker who redirects a return or an indirect call to a gadget is detected because that transfer was never a legitimate edge in the program's control-flow graph.
The motivating insight is that code-reuse attacks all do the same abnormal thing: they transfer control to a location that the normal program would never jump to from that point. A ret in a legitimate program returns to the instruction after the call that invoked the function; a ROP chain makes ret jump to the middle of some unrelated function (a gadget). An indirect call through a function pointer normally targets one of a small set of real functions; a hijacked pointer targets a gadget or an arbitrary address. CFI's job is to recognize these illegitimate transfers and stop the program.
Control transfers divide into two kinds, and CFI protects each differently — a distinction that matters because they correspond to ROP versus JOP:
- Backward edges are returns (
ret), which should go back to the caller. The strong protection here is a shadow stack: a separate, protected region that keeps its own copy of every return address pushed at eachcall. At eachret, the hardware compares the return address on the normal stack against the shadow copy; a mismatch (which any ROP chain produces, since it put gadget addresses on the stack) aborts the program. Intel CET (Control-flow Enforcement Technology) implements this in hardware, making it fast and robust. - Forward edges are indirect calls and jumps (
call rax,jmp [rax]), which should target a valid function. The protection here is indirect-branch validation: the compiler assigns a label or type signature to valid targets, and every indirect transfer checks that its destination carries the right label. Intel CET's IBT (Indirect Branch Tracking) requires every valid indirect-call target to begin with anendbr64instruction (the landing pad seen in the program-analysis chapter); an indirect call landing anywhere without anendbr64faults. Compiler CFI schemes (Clang's-fsanitize=cfi, Microsoft's Control Flow Guard) refine this with type-based checks so a function pointer can only call functions of the matching signature.
CFI protects both edge directions (defeating ROP and JOP)
───────────────────────────────────────────────────────────────
BACKWARD edge (ret): shadow stack
call pushes retaddr onto BOTH the normal and the shadow stack
ret compares them -> ROP's forged stack retaddr mismatches -> ABORT
FORWARD edge (indirect call/jmp): IBT / type checks
every valid target starts with `endbr64`
an indirect call to a gadget (no endbr64) -> ABORT
type-based CFI additionally requires a matching function signature
CFI is strong but not absolute, and its limits are instructive:
- Coarse-grained CFI (any
endbr64is a valid target) still permits "CFI-aware" ROP/JOP that uses only gadgets starting at valid landing pads — a smaller but non-empty gadget set. Fine-grained, type-based CFI shrinks this further but is harder to deploy without breaking legitimate indirect calls. - Data-only attacks need no illegitimate control transfer at all. If an attacker corrupts a data value — a
isAdminflag, a length used in a later copy, a file path — the program's control flow stays legitimate while its behaviour is subverted. CFI, which checks control flow, cannot see this. Data-only (and the general "data-oriented programming") attacks are the frontier precisely because they sidestep CFI. - Deployment friction. Shadow stacks require hardware (recent CPUs) and OS support; type-based CFI requires whole-program knowledge (hard with dynamic linking and function-pointer-heavy code). Adoption has been gradual.
The defensive takeaway is that CFI meaningfully raises the cost of code reuse — especially a hardware shadow stack, which breaks classic ROP outright — and is the most significant anti-exploitation advance since ASLR, but it is not the end of exploitation: it pushes attackers toward CFI-aware reuse and data-only attacks, which is why it too is one layer rather than a complete solution.
Examples
# Enable and verify CFI-related protections.
gcc -fcf-protection=full -O2 app.c -o app_cet # endbr64 (IBT) + shadow-stack hints
readelf -n ./app_cet | grep -iE 'SHSTK|IBT' # CET properties present?
objdump -d ./app_cet | grep -c endbr64 # one landing pad per indirect target
# Clang's type-based forward-edge CFI (needs LTO + visibility).
clang -flto -fvisibility=hidden -fsanitize=cfi -O2 app.c -o app_cfi
# An indirect call through a mismatched-type function pointer traps at runtime.
# Microsoft Control Flow Guard (Windows), for comparison:
# cl /guard:cf app.c # adds forward-edge indirect-call validation
// Why a shadow stack breaks ROP, and what a data-only attack evades.
#include <stdio.h>
#include <string.h>
void process(const char *in) {
int is_admin = 0; // a DATA value, not a control target
char buf[16];
strcpy(buf, in); // overflow can corrupt is_admin (adjacent)
if (is_admin) // control flow stays LEGITIMATE either way
puts("granted"); // CFI cannot detect that is_admin was
else // flipped by a data overwrite
puts("denied");
// - A ROP attempt to redirect `process`'s RETURN is caught by a shadow
// stack (the forged return address mismatches the shadow copy).
// - But flipping `is_admin` is a DATA-ONLY attack: no bad control
// transfer occurs, so CFI/shadow stack see nothing wrong.
}
Key Takeaways
- CFI attacks the mechanism of code reuse directly, enforcing that control transfers follow only edges the original program intended, so redirecting a return or indirect call to a gadget is detected.
- Backward edges (returns) are protected by a shadow stack — a protected copy of return addresses checked at each
ret— which Intel CET implements in hardware and which breaks classic ROP outright. - Forward edges (indirect calls/jumps) are protected by indirect-branch validation: CET's IBT requires targets to begin with
endbr64, and type-based CFI further requires a matching function signature. - Coarse-grained CFI still permits CFI-aware reuse using landing-pad gadgets, and fine-grained CFI is harder to deploy; data-only attacks bypass CFI entirely because they corrupt data without any illegitimate control transfer.
- CFI, especially a hardware shadow stack, is the most significant anti-reuse advance since ASLR but remains one layer, pushing attackers toward CFI-aware and data-oriented techniques.
🧪 Practice
- Compile a program with
-fcf-protection=full, count theendbr64instructions, and explain what each one marks and what an indirect call to a non-endbr64address would do.- Explain, step by step, how a shadow stack detects a ROP chain's forged return address, and why this protection is independent of ASLR.
- Construct a small example of a data-only attack (e.g. flipping an
is_adminflag via an overflow) and explain why CFI cannot detect it.- Interview-style: "A target has a hardware shadow stack and IBT enabled. Which exploitation techniques are now substantially harder, and which emerging class remains viable?" Hint: map the shadow stack to backward-edge (ret) attacks and IBT to forward-edge (indirect call) attacks, then recall the class of attack that subverts behaviour without any illegitimate control transfer.
Fortify Source and Compiler Hardening Flags
Theory
The mitigations so far are runtime defenses against exploitation. This topic covers compile-time hardening — flags that make the compiler either insert runtime checks or refuse dangerous constructs outright, catching bugs before they can be exploited. The reason these matter is leverage: a one-line change to a build configuration can add bounds checks to thousands of library calls across an entire codebase, at little or no performance cost, which is among the highest return-on-effort security measures available.
_FORTIFY_SOURCE is the centerpiece. It is a macro that, combined with optimization, makes glibc replace certain unsafe function calls with "checked" variants (__memcpy_chk, __strcpy_chk, __sprintf_chk, and others) when the compiler can determine the destination buffer's size at compile time. The checked variant knows the real buffer size and aborts (__chk_fail) if the operation would overflow it. Crucially, this turns an exploitable overflow into a controlled abort, and it does so for the exact functions (strcpy, memcpy, sprintf, gets) that dominate the overflow bug class.
The key operational facts, which are also the common pitfalls:
- It requires optimization.
_FORTIFY_SOURCErelies on the compiler's ability to compute object sizes (via__builtin_object_size), which only works with optimization enabled. A build at-O0silently gets no fortification — a frequent real-world gap where teams believe they are hardened but are not. - It is best-effort. When the destination size cannot be determined at compile time (e.g. a heap buffer of dynamic size passed through several functions), fortification cannot apply and the call falls back to the unchecked version. So it catches many overflows, not all.
- Levels.
=1checks only cases that should not change conforming behaviour;=2is stricter (e.g. it also forbids%nin writable-memory format strings, tying to the format-string section);=3(newer) adds dynamic object-size checks for more coverage.
Beyond _FORTIFY_SOURCE, a standard hardening flag set combines the compile-time and link-time mitigations from across this section into one build configuration. The table consolidates them with the stage each applies at (from the program-analysis compilation-pipeline topic):
| Flag | Stage | Protection |
|---|---|---|
-D_FORTIFY_SOURCE=2 (with -O2) | compile | checked libc calls; %n restriction |
-fstack-protector-strong | compile | stack canaries |
-fstack-clash-protection | compile | probes large stack allocations (stack-clash attacks) |
-fcf-protection=full | compile | CET IBT / shadow-stack support |
-fPIE + -pie | compile+link | PIE (full ASLR of the executable) |
-Wl,-z,relro,-z,now | link | full RELRO |
-Wl,-z,noexecstack | link | NX stack |
-Wall -Wextra -Wformat-security -Wconversion | compile | warnings that catch bug sources (format strings, implicit conversions) |
The warnings deserve emphasis because they prevent bugs rather than merely mitigating them: -Wformat-security catches format-string vulnerabilities at compile time, -Wconversion flags the implicit signed/unsigned and truncation conversions behind integer bugs, and -Werror turns the most important of these into hard build failures. Treating warnings as the first line of defense is a theme the secure-coding section develops.
The overarching lesson is that hardening flags are cheap, broad, and verifiable — but only if actually applied and verified. The recurring failure mode is a flag that does not take effect (_FORTIFY_SOURCE without -O2, LDFLAGS not passed to the linker), which is why the discipline is always to verify the produced binary with checksec/readelf rather than trusting the build invocation.
Examples
# A complete hardened build line, annotated by stage.
gcc -O2 \
-D_FORTIFY_SOURCE=2 \ # compile: checked libc calls (NEEDS -O2)
-fstack-protector-strong \ # compile: canaries
-fstack-clash-protection \ # compile: probe big stack allocations
-fcf-protection=full \ # compile: CET IBT / shadow stack
-fPIE -pie \ # compile+link: PIE -> full ASLR
-Wl,-z,relro,-z,now \ # link: full RELRO
-Wl,-z,noexecstack \ # link: NX stack
-Wall -Wextra -Wformat-security -Wconversion -Werror \ # bug-source warnings
app.c -o app
# ALWAYS verify the produced binary rather than trusting the flags:
checksec --file=./app
# RELRO: Full | Canary: Yes | NX: enabled | PIE: PIE | FORTIFY: Yes
# The classic silent failure: FORTIFY does NOTHING without optimization.
gcc -D_FORTIFY_SOURCE=2 -O0 app.c -o app_broken
nm -D app_broken | grep _chk || echo "no checked functions: FORTIFY inert at -O0"
// What FORTIFY turns an overflow into: an abort instead of corruption.
#include <string.h>
void copy(const char *src) {
char buf[8];
strcpy(buf, src); // with -D_FORTIFY_SOURCE=2 -O2 and a known buf size,
// this becomes __strcpy_chk(buf, src, 8); if src is
// longer than 8, it calls __chk_fail() and aborts:
// *** buffer overflow detected ***: terminated
// The overflow is contained; no return address is hit.
}
Key Takeaways
- Compile-time hardening flags add runtime checks or forbid dangerous constructs across an entire codebase cheaply, making them among the highest-leverage security measures.
_FORTIFY_SOURCEreplaces unsafe libc calls with checked variants that know the destination size and abort on overflow, but it requires optimization (silently inert at-O0) and is best-effort (only when the size is known at compile time).- Level
=2also restricts%nin writable-memory format strings, connecting to the format-string mitigations. - A standard hardening set combines canaries, stack-clash protection, CET, PIE, full RELRO, NX, and bug-catching warnings, each applying at a specific compile or link stage.
- Warnings (
-Wformat-security,-Wconversion,-Werror) prevent bug sources rather than mitigating them, and are a first line of defense. - Flags must be verified on the produced binary with
checksec/readelf, because the common failure is a flag that silently did not take effect.
🧪 Practice
- Build a program with a
strcpyoverflow using-D_FORTIFY_SOURCE=2 -O2and confirm it aborts with a buffer-overflow message; then rebuild at-O0and show the protection is gone.- Use
checksecandnm -D | grep _chkto verify which mitigations and checked functions a given binary actually has, and reconcile that with its build flags.- Introduce a format-string bug and an implicit signed/unsigned conversion, then show that
-Wformat-securityand-Wconversioncatch them at compile time.- Interview-style: "A team sets
-D_FORTIFY_SOURCE=2in theirCFLAGSbut a scanner reports no fortification in the shipped binary. Give two independent reasons this can happen." Hint: consider the optimization level the build actually uses, and whether the destination buffer sizes in their code are knowable at compile time.
Sandboxing and seccomp Filtering
Theory
Every mitigation so far tries to prevent an attacker from gaining control. Sandboxing accepts that control might be gained and focuses on containing the damage: even if an attacker achieves code execution inside a process, a sandbox restricts what that code can do to the rest of the system. This is a shift from prevention to confinement, and it reflects a mature security philosophy — assume breach, and limit blast radius. It is the application of least privilege (from the foundations chapter) to a running process.
The key realization, from the program-analysis syscall topic, is that a process can only affect the outside world through system calls. Reading a file, opening a socket, spawning a program, reading another process's memory — all require syscalls. Therefore, restricting which syscalls a process may make directly restricts what a compromised process can do. This is the principle behind Linux seccomp (secure computing mode).
seccomp-bpf lets a process install a BPF filter program that the kernel consults on every syscall. The filter inspects the syscall number and arguments and returns a verdict: allow, deny with an error, kill the process, or trap. The powerful move is to install an allowlist: permit exactly the syscalls the program legitimately needs and kill it on anything else. Then, when an exploit achieves code execution and tries to call execve to spawn a shell — or socket to phone home, or ptrace to attack another process — the syscall is not on the allowlist and the kernel terminates the process. The attacker has code execution but no way to use it, because the actions that matter are forbidden.
This is why seccomp is one of the most effective post-exploitation mitigations: exploitation almost always needs a small set of powerful syscalls (execve, socket/connect, open/openat on sensitive paths, ptrace), and a tight allowlist removes exactly those. A document parser does not need execve; a sandboxed renderer does not need to open arbitrary files. Denying the syscalls they never legitimately use costs nothing and defangs most exploits.
Sandboxing is broader than seccomp, and real systems layer several mechanisms:
| Mechanism | What it confines |
|---|---|
| seccomp-bpf | which syscalls (and argument values) a process may make |
| Namespaces | what a process can see: its own PID, mount, network, user views (the basis of containers) |
| cgroups | how many resources (CPU, memory, I/O) it may consume |
| Capabilities | fine-grained slices of root privilege, dropped individually |
| MAC (SELinux/AppArmor) | mandatory policy on files, ports, and operations, independent of user permissions |
| chroot / pivot_root | the filesystem subtree a process can reach |
Browsers are the canonical example of defense-in-depth sandboxing: each renderer process runs with a strict seccomp allowlist, in restrictive namespaces, with almost no capabilities, so that a full compromise of the renderer (which parses hostile web content) yields an attacker who still cannot read the user's files or escape to the system without also finding a separate sandbox-escape bug. This is the "assume the renderer will be hacked, make it not matter" design.
The defensive lessons: seccomp and sandboxing do not prevent the bug or even the code execution, but they convert "attacker runs code in the process" into "attacker runs code that can do almost nothing" — raising the attacker's cost from one bug to a chain (the exploit plus a sandbox escape). The main engineering challenge is building a correct, minimal policy: too tight breaks the program, too loose defeats the purpose, and the right allowlist is derived from observing the program's real syscall usage (strace -c from the program-analysis chapter).
Examples
// A minimal seccomp allowlist: permit only what the program needs, kill the rest.
// Build: gcc seccomp.c -o seccomp -lseccomp
#include <seccomp.h>
#include <unistd.h>
int main(void) {
// Default action for anything NOT explicitly allowed: kill the process.
scmp_filter_ctx ctx = seccomp_init(SCMP_ACT_KILL);
// Allow exactly the syscalls this program legitimately uses:
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(read), 0);
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 0);
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(exit_group), 0);
seccomp_load(ctx); // from here on, the filter is enforced
// Legitimate work: read/write are allowed.
char buf[64];
ssize_t n = read(0, buf, sizeof buf);
write(1, buf, n);
// If a later exploit tries execve("/bin/sh", ...) to get a shell, the
// kernel sees a syscall NOT on the allowlist and KILLS the process:
// execve(...); -> SIGSYS, process terminated. Code exec is useless.
return 0;
}
# Derive the allowlist from the program's REAL syscall usage (from chapter 4).
strace -c -f ./myprogram 2>&1 | awk 'NR>2 {print $NF}' # the syscalls it uses
# Build the seccomp policy from exactly that set; nothing more.
# Confirm a process is in seccomp filter mode (2 = filter):
grep Seccomp /proc/self/status
# Run something under a ready-made sandbox profile (systemd example):
systemd-run --user -p SystemCallFilter='@system-service' -p NoNewPrivileges=yes ./myprogram
# @system-service is a curated allowlist; combine with namespaces for more.
Key Takeaways
- Sandboxing shifts from preventing compromise to containing it: even with code execution, a sandboxed process can do little to the rest of the system — the application of least privilege to a running process.
- Because a process affects the world only through syscalls, restricting syscalls directly restricts what a compromised process can do; seccomp-bpf filters each syscall by number and arguments.
- A tight seccomp allowlist is a high-value post-exploitation mitigation because exploitation almost always needs
execve,socket,open, orptrace, and denying those defangs most exploits. - Real sandboxes layer seccomp with namespaces (what a process sees), cgroups (resources), capabilities (privilege slices), MAC (SELinux/AppArmor), and chroot; browsers are the canonical defense-in-depth example.
- Sandboxing converts "attacker runs code" into "attacker runs code that can do almost nothing," forcing the attacker to find an additional sandbox-escape bug; the main challenge is building a correct minimal policy, derived from observed syscall usage.
🧪 Practice
- Write a seccomp allowlist for a program that only reads stdin and writes stdout, then confirm that an added
execvecall causes the kernel to kill the process.- Use
strace -cto enumerate a small program's real syscalls and construct a minimal allowlist from exactly that set; test that the program still works.- Explain how namespaces, seccomp, and dropped capabilities each contribute differently to a browser renderer's sandbox, and what a full compromise of the renderer still cannot do.
- Interview-style: "An attacker achieves reliable code execution inside a PDF parser that runs under a strict seccomp allowlist permitting only
read,write,mmap, andexit. Why is this a much weaker position than it sounds, and what would the attacker need next?" Hint: enumerate what the attacker cannot do withoutexecve,socket, oropen, and name the additional class of bug required to escape the confinement.
Layered Mitigation Effectiveness
Theory
This topic synthesizes the whole section into its central thesis: no single mitigation is sufficient, and security comes from layering them so that an attacker must defeat every layer, in order, to succeed. This is defense-in-depth applied to exploitation, and understanding why it works is the most important takeaway of the chapter — more important than any individual mitigation's details.
The reasoning starts from an honest admission: every mitigation in this section is individually bypassable, and this chapter has shown exactly how. Canaries fall to information leaks. NX falls to code reuse. ASLR falls to a single leaked pointer. RELRO is sidestepped by writing to targets other than the GOT. CFI is evaded by CFI-aware reuse or data-only attacks. Seccomp is escaped by a second bug. If you evaluate each mitigation alone, each looks defeatable — and attackers and vendors sometimes argue about individual bypasses as if they settled the matter. They do not, because real targets deploy the mitigations together.
The power of layering comes from the dependency chain from the combining-primitives topic. A modern exploit is not one trick but a sequence of stages, each defeating one mitigation and each requiring a specific capability that the previous stage provided:
Why layers multiply the attacker's cost
───────────────────────────────────────────────────────────────
To exploit a canary+NX+ASLR+RELRO+CFI+seccomp target, the attacker must:
1. find an info-leak -> defeat ASLR (get addresses) ┐
2. leak the canary -> defeat the stack canary │ each
3. build a code-reuse chain -> defeat NX │ stage
4. use a non-GOT write target -> defeat RELRO │ needs
5. use CFI-aware gadgets or data-only -> defeat CFI │ the
6. find a sandbox escape -> defeat seccomp ┘ prior
Miss ANY capability and the chain breaks. Each layer is a separate
bug or primitive the attacker must independently obtain.
Two multiplicative effects make this far stronger than the sum of its parts:
- Capability requirements compound. Each layer demands a distinct primitive (a leak, a write, a specific gadget type, a second bug). A single memory-corruption bug rarely provides all of them, so the attacker must find and chain multiple bugs — which is exponentially harder than finding one. Mitigations do not just block; they raise the number of distinct vulnerabilities an attack requires.
- Reliability degrades across stages. Remotely and blind, each stage has some probability of failure (a wrong offset, a crash, a re-randomization). The overall success probability is the product of per-stage probabilities, so adding a layer that forces another uncertain stage can drop reliable exploitation from "routine" to "impractical," even if that layer is theoretically bypassable.
This reframes how to evaluate both mitigations and bugs. A mitigation's worth is not "can it be bypassed in isolation?" (almost always yes) but "does it force the attacker to acquire an additional, independent capability?" (if yes, it is pulling its weight). And a bug's severity is not its standalone impact but which stage it could supply in a chain — which is why a humble information leak is treated seriously, since it supplies the stage that unlocks all the others.
The practical guidance that follows: enable every applicable layer (the hardening-flags build set plus runtime ASLR, CET, and sandboxing), because their value is cumulative and their individual bypassability is not an argument against any of them; prioritize eliminating information disclosure, since the leak stage is the most common linchpin; and assume breach at the innermost layer by sandboxing, so that even a fully successful exploit of the process is contained. Defense-in-depth is not redundancy for its own sake — it is the deliberate construction of a chain of independent obstacles, each cheap to deploy and each forcing the attacker to solve a separate, hard problem.
Examples
Individually bypassable, collectively formidable
───────────────────────────────────────────────────────────────
Mitigation Bypassed alone by... But requires the attacker to...
────────── ───────────────────── ────────────────────────────────
Canary an info leak ...first HAVE a read primitive
NX ROP/ret2libc ...first KNOW addresses (ASLR!)
ASLR one leaked pointer ...first HAVE a leak
Full RELRO non-GOT write target ...find another writable target
CFI/shadow data-only / CFI-aware reuse ...constrain to a tiny gadget set
seccomp a sandbox-escape bug ...find a SECOND vulnerability
No layer stands alone; together they demand multiple independent bugs.
# The full picture for one binary: are the layers actually present?
checksec --file=./app
# RELRO: Full | Canary: Yes | NX: enabled | PIE: PIE | FORTIFY: Yes
readelf -n ./app | grep -iE 'SHSTK|IBT' # CET layers
cat /proc/sys/kernel/randomize_va_space # ASLR level (2 = full)
grep Seccomp /proc/<pid>/status # sandbox layer at runtime
# A target missing any single row is one capability cheaper to exploit.
Key Takeaways
- The central thesis of exploit mitigation: no single defense is sufficient, and security comes from layering them so the attacker must defeat every layer in sequence.
- Every mitigation is individually bypassable (this chapter showed how), so evaluating any one alone is misleading; real targets deploy them together.
- Layering works because the stages form a dependency chain where each defeat requires a specific capability the previous stage supplied, so the attacker must obtain multiple independent primitives — often multiple distinct bugs.
- The effect is multiplicative: capability requirements compound (exponentially more bugs needed) and reliability degrades across stages (the product of per-stage success probabilities), so a theoretically bypassable layer can still make reliable exploitation impractical.
- Evaluate a mitigation by whether it forces an additional independent capability, and a bug by which stage it could supply; prioritize eliminating information disclosure and assume-breach via sandboxing.
🧪 Practice
- For a target with canary, NX, full ASLR, full RELRO, CET, and seccomp, write out the ordered list of capabilities an attacker must acquire and which mitigation each defeats.
- Explain, using per-stage success probabilities, why adding one more uncertain stage can disproportionately reduce an exploit's overall reliability.
- Using
checksec,readelf -n, and the ASLR sysctl, audit a local binary and its runtime for which mitigation layers are present and which are missing.- Interview-style: "A colleague argues that since every mitigation 'has a known bypass,' spending effort enabling all of them is pointless. Rebut this using the dependency-chain model." Hint: distinguish bypassable-in-isolation from bypassable-in-combination, and reason about how many distinct bugs and primitives the attacker must chain when all layers are present versus when one is removed.
<a id="55-secure-coding"></a>
5.5 Secure Coding
The most effective response to every vulnerability in this chapter is to never introduce it; this section turns the attack knowledge into defensive practice, covering the coding habits, language choices, standards, and review disciplines that prevent memory-safety and related bugs at the source.
Safe String and Buffer Handling in C
Theory
String and buffer handling is where most C memory-safety bugs originate, so getting it right is the single highest-impact secure-coding skill. The root problem, from the unsafe-functions topic, is that C strings are bare pointers with no attached length and a fragile NUL-terminator convention, and the standard functions do not enforce bounds. Safe handling is therefore a discipline of always knowing the size of every buffer and never writing past it — a discipline, because the language will not enforce it for you.
The foundational mental shift is to treat a buffer as a pointer plus a length, always traveling together. Every function that writes to a buffer must receive its size, and every write must be checked against that size. This is the invariant that unsafe code violates (passing only the pointer) and that safe code maintains. Modern C codebases often formalize this with a small struct bundling pointer and length, precisely so the size cannot be lost.
The concrete rules, building on the unsafe-functions and off-by-one topics:
- Always pass and respect buffer sizes. Use
snprintfoversprintf,fgetsovergets, boundedscanf(%63s), and the size-taking memory functions with a correct size argument. - Guarantee NUL-termination. After any bounded copy, ensure a terminator —
snprintfdoes it for you;strncpydoes not, so it needs a manualbuf[n-1] = '\0'. - Account for the terminator in sizes. Allocate
strlen(s) + 1; a buffer of size N holds an N-1 character string. - Prefer functions that report truncation.
snprintf's return value tells you if output was truncated;strlcpy/strlcatreturn the length that would have been written. Detecting truncation lets you fail safely rather than silently produce a wrong (and possibly dangerous) result. - Check lengths before combining. Before
strcat-style appends, verify the remaining space, computing against remaining capacity, not total.
The platform landscape of "safer" functions is worth knowing because the choice has trade-offs:
| Family | Guarantees | Caveat |
|---|---|---|
snprintf | bounded, always NUL-terminates, reports truncation | most portable; the default recommendation |
strlcpy/strlcat (BSD) | bounded, always NUL-terminates, reports needed length | not in standard glibc (link libbsd) |
strcpy_s/memcpy_s (Annex K) | bounded, runtime constraint handler | optional, unevenly implemented; controversial |
strncpy/strncat | bounded write only | does NOT guarantee termination — not a safe default |
The highest-leverage practice, though, is to let the compiler and sanitizers help: build with -D_FORTIFY_SOURCE=2 -O2 (checked libc calls), -Wall -Wextra, and run tests under AddressSanitizer. These catch the overflows that slip past manual discipline, and they are the reason the modern advice is "safe functions and FORTIFY and ASan," not any one alone. The deeper message, which the memory-safe-languages topic completes, is that string handling in C is error-prone by design, and the most reliable fix for a new project is often to not write the parsing in unsafe C at all.
Examples
// Buffer-as-pointer-plus-length: the invariant that prevents overflow.
#include <stdio.h>
#include <string.h>
#include <stdbool.h>
// A tiny bounded-buffer helper: size travels WITH the pointer.
typedef struct { char *data; size_t cap; size_t len; } sbuf;
bool sbuf_append(sbuf *b, const char *s) {
size_t slen = strlen(s);
if (slen > b->cap - b->len - 1) // check BEFORE writing (unsigned-safe
return false; // because len < cap is an invariant)
memcpy(b->data + b->len, s, slen); // write exactly slen bytes
b->len += slen;
b->data[b->len] = '\0'; // always re-terminate
return true; // caller learns if it did not fit
}
// The same path-building task: unsafe, then safe with truncation detection.
#include <stdio.h>
#include <string.h>
// UNSAFE: no bounds, silent overflow on long inputs.
void build_path_bad(char *out, const char *dir, const char *file) {
strcpy(out, dir); // overflow if dir is long
strcat(out, "/"); // more overflow
strcat(out, file); // and more
}
// SAFE: bounded, terminated, and truncation is DETECTED and refused.
int build_path_ok(char *out, size_t outsz, const char *dir, const char *file) {
int n = snprintf(out, outsz, "%s/%s", dir, file); // bounded + terminated
if (n < 0 || (size_t)n >= outsz) // truncation check
return -1; // fail safely, do not
return 0; // use a truncated path
}
# The defense-in-depth build for string-heavy C code.
gcc -O2 -D_FORTIFY_SOURCE=2 -Wall -Wextra -fsanitize=address string.c -o string
./string # ASan catches any overflow the manual discipline missed
Key Takeaways
- Most C memory-safety bugs start in string/buffer handling, because C strings are bare pointers with no length and the standard functions do not enforce bounds.
- The core invariant is that a buffer is a pointer plus a length that always travel together; every write is checked against the size, often formalized with a small bundling struct.
- Prefer
snprintf,fgets, and boundedscanf; always guarantee NUL-termination (whichstrncpydoes not), account for the terminator in sizes, and prefer functions that report truncation so you can fail safely. snprintfis the portable default;strlcpy/strlcatare good but non-standard; Annex K_sfunctions are uneven;strncpyis not a safe default.- Layer compiler help on top of discipline:
-D_FORTIFY_SOURCE=2 -O2,-Wall -Wextra, and AddressSanitizer catch what manual care misses, and for new projects, not writing parsing in unsafe C at all is the strongest fix.
🧪 Practice
- Rewrite a function that builds a string with
strcpy/strcatto usesnprintfwith truncation detection, and test it with an input that would overflow the old version.- Implement the
sbufbounded-buffer helper withappendand aconcat, and verify under AddressSanitizer that it never overflows regardless of input length.- Demonstrate the
strncpynon-termination pitfall and fix it two ways: a manual terminator and a switch tosnprintf.- Interview-style: "Why is detecting truncation (not just preventing overflow) important for security, and which functions make it possible?" Hint: think about what happens when a security-relevant value — a path, a hostname, an allowlist entry — is silently cut short, and how
snprintf's return value orstrlcpy's length result lets you refuse that outcome.
Bounds Checking and Defensive Programming
Theory
Bounds checking is the practice of verifying, before every memory access driven by an index or length, that the access stays within the valid range. Defensive programming is the broader mindset it belongs to: writing code that behaves safely even when its inputs, callers, and environment violate expectations. Both exist because the alternative — assuming inputs are well-formed and callers are correct — is exactly the assumption every exploit in this chapter abuses.
The "why" is that memory-safety bugs are, at root, missing or incorrect checks. An overflow is a missing length check; an out-of-bounds index is a missing range check; an integer bug is a missing overflow check. Defensive programming internalizes that every value crossing a trust boundary is potentially hostile and must be validated before use, and that the cost of a check is trivial compared to the cost of a vulnerability.
The core bounds-checking rules:
- Validate indices against both ends.
if (i < 0 || i >= n) return error;— the signed lower-bound check matters because a negative index (from the integer-bug topic) otherwise reads before the buffer. - Validate lengths against remaining capacity, computed with types that cannot wrap (prefer
size_t, check before arithmetic). - Check at the boundary, then trust internally. Validate untrusted input once, at the point it enters your trusted code, and track that it is now validated — rather than re-checking scattershot or, worse, not at all.
- Fail safely. A failed check should produce a safe, defined outcome (an error return, a rejected request), never undefined behaviour or a "best effort" that proceeds with a bad value.
Defensive programming adds supporting techniques, each with a security purpose:
| Technique | Purpose |
|---|---|
Precondition checks / assert | document and enforce invariants; catch violations early in testing |
| Parameter validation | reject null pointers, out-of-range sizes, invalid enums at function entry |
| Fail-closed defaults | the default branch of a switch, the else of a validation, denies rather than permits |
| Minimizing trust scope | the smaller the trusted region, the fewer places a check can be missed |
| Avoiding silent coercion | explicit casts and conversions so signedness/truncation are intentional |
A crucial nuance is the distinction between assert and runtime validation, because confusing them is itself a vulnerability. assert is for programmer invariants that should be impossible to violate if the code is correct — it is a development-time check, and in release builds assert is typically compiled out (NDEBUG). Therefore assert must never be used to validate untrusted input: if you write assert(len <= max) to check network data, the check vanishes in the release build and the overflow proceeds. Untrusted input requires a real runtime check (if (len > max) return error;) that is always present. Use assert to catch your own bugs; use explicit validation to defend against hostile input.
The practical balance is that defensive programming should be targeted, not paranoid noise. The highest-value checks are at trust boundaries (input parsing, API entry points, privilege transitions); internal code that only ever receives already-validated data from trusted callers need not re-validate everything, as long as the boundary is well-defined and enforced. Over-checking everywhere can obscure the logic and create a false sense that every check is load-bearing; the discipline is to know where the trust boundary is and defend it rigorously.
Examples
// Bounds checking at a trust boundary, with safe failure.
#include <stddef.h>
#include <errno.h>
// Reads element `i` from an array of length `n`. Index is untrusted.
int array_get(const int *a, size_t n, long i, int *out) {
if (a == NULL || out == NULL) return -1; // parameter validation
if (i < 0 || (size_t)i >= n) { // BOTH ends: reject negatives
errno = ERANGE; // and >= n (out of bounds)
return -1; // fail safely, no access
}
*out = a[i]; // now provably in range
return 0;
}
// The assert-vs-validation trap: one of these disappears in release builds.
#include <assert.h>
#include <string.h>
void handle_packet_WRONG(const char *data, size_t len) {
char buf[256];
assert(len <= sizeof buf); // BUG: compiled out when NDEBUG is set,
memcpy(buf, data, len); // so the overflow proceeds in RELEASE.
}
void handle_packet_RIGHT(const char *data, size_t len) {
char buf[256];
if (len > sizeof buf) // a REAL check: always present, even in
return; // release; untrusted input demands this.
memcpy(buf, data, len); // provably safe
// Use assert() only for invariants that your OWN correct code guarantees,
// never for validating data that crosses a trust boundary.
}
# Keep asserts honest: run tests with them ENABLED, ship with real checks.
gcc -O2 -DNDEBUG app.c -o app_release # asserts are GONE here -- verify your
# security checks are NOT asserts.
gcc -O0 app.c -o app_debug # asserts active: catch invariant breaks
# A grep that flags the dangerous pattern in review:
grep -n 'assert(.*len\|assert(.*size\|assert(.*count' *.c # suspicious: input?
Key Takeaways
- Memory-safety bugs are at root missing or incorrect checks, so bounds checking — verifying every index/length against its valid range before access — directly prevents them.
- Defensive programming treats every value crossing a trust boundary as potentially hostile and validates it before use, failing safely when a check fails.
- Validate indices against both ends (a negative index is as dangerous as an oversized one), validate lengths with non-wrapping types checked before arithmetic, and check at the boundary then trust internally.
assertis for programmer invariants and is compiled out in release builds, so it must never validate untrusted input — that requires an always-present runtime check.- Target checks at trust boundaries rather than scattering them everywhere; know where the boundary is and defend it rigorously, using fail-closed defaults.
🧪 Practice
- Write an
array_getthat safely rejects negative and out-of-range indices, and demonstrate that a negative index is caught by the lower-bound check.- Take a function that uses
assertto validate a length from untrusted input, compile it with-DNDEBUG, and show that the overflow now occurs; then fix it with a real check.- Identify the trust boundary in a small program that parses input and passes it to internal functions, and place validation precisely there rather than everywhere.
- Interview-style: "A reviewer finds
assert(index < array_len);guarding an access whereindexcomes from a request parameter. Why is this a security bug, and what is the fix?" Hint: consider what happens toassertwhen the code is compiled withNDEBUGfor release, and what therefore guards the access in production.
Input Validation and Canonicalization
Theory
Input validation is the practice of checking that data conforms to expectations before acting on it, and canonicalization is the essential companion: reducing data to a single, standard form before validating it, so that validation cannot be fooled by alternative encodings. Together they are the first line of defense not just for memory safety but for nearly every vulnerability class — injection, path traversal, authentication bypass — which is why they recur throughout the security curriculum. This topic focuses on the principles that apply broadly.
The foundational principle is allowlisting over blocklisting (from the foundations chapter). A blocklist tries to enumerate bad inputs and reject them, which fails because the space of bad inputs is unbounded and attackers invent new ones; an allowlist specifies exactly what is acceptable and rejects everything else, which is safe by construction — including inputs nobody has thought of yet. "Accept [a-z0-9_]{1,32}" is robust; "reject inputs containing ../" is a game of whack-a-mole. Validate for what you expect (type, length, range, format, set membership), and reject anything that does not match.
The reason canonicalization must come first is that the same logical value can have many textual representations, and an attacker chooses the representation that slips past the check but is dangerous once decoded. This is the heart of countless bypasses:
| Canonicalization issue | Example | Why validation alone fails |
|---|---|---|
| Path traversal | ../, ..%2f, ....//, symlinks | the string differs but resolves to the same forbidden path |
| URL/percent encoding | %2e%2e%2f decodes to ../ | the check sees %2e, the filesystem sees . |
| Unicode normalization | file vs file, overlong UTF-8 | different bytes, same or confusable meaning |
| Case / trailing chars | ADMIN, admin , admin. | case-insensitive or trailing-dot filesystems |
| Double encoding | %252e -> %2e -> . | one decode pass is not enough |
The rule is: canonicalize to one definitive form, then validate that form, then use that same form. For a file path, resolve it to an absolute, symlink-free real path (realpath) and then check it is within the allowed directory — never validate the raw user string and then open a different, un-canonicalized one. Validating one form and using another is the classic mistake, and it is closely related to the TOCTOU problem in a later topic (the path you checked and the path you used must be the same path).
The broader input-validation discipline:
- Validate at the trust boundary, as early as possible, before the data influences any decision or operation.
- Validate type, length, range, and format — a value should be the right kind of thing, the right size, within bounds, and matching an expected pattern.
- Reject, do not sanitize-and-continue, when possible. Stripping bad characters and proceeding is error-prone (what if stripping creates a new bad value, like
....//becoming../after removing./?); outright rejection of malformed input is safer. - Validate on the server / trusted side. Client-side validation is a usability feature, not a security control, because the client is under attacker control.
For memory safety specifically, input validation is where length and count fields from untrusted sources get bounded before they reach allocations and copies — the check that would have stopped most of this chapter's overflows. A length read from the network, validated against a sane maximum at the boundary, never becomes the oversized memcpy size.
Examples
// Allowlist validation: accept only what is expected, reject all else.
#include <ctype.h>
#include <string.h>
#include <stdbool.h>
bool valid_username(const char *s) {
size_t n = strlen(s);
if (n < 1 || n > 32) return false; // length bounds first
for (size_t i = 0; i < n; i++) {
unsigned char c = (unsigned char)s[i];
if (!(islower(c) || isdigit(c) || c == '_'))
return false; // allowlist: [a-z0-9_] only
}
return true; // anything else is rejected
}
// Canonicalize THEN validate THEN use the SAME canonical path.
#include <stdlib.h>
#include <string.h>
#include <limits.h>
#include <stdio.h>
FILE *open_in_dir(const char *base, const char *user_path) {
char resolved[PATH_MAX];
// 1. CANONICALIZE: resolve .. and symlinks to a real absolute path.
if (!realpath(user_path, resolved)) // (file must exist for realpath)
return NULL;
// 2. VALIDATE: the canonical path must be inside the allowed base.
size_t blen = strlen(base);
if (strncmp(resolved, base, blen) != 0 || resolved[blen] != '/')
return NULL; // outside base -> reject
// 3. USE the SAME canonical path we validated -- never the raw input.
return fopen(resolved, "r");
// Validating `user_path` but opening it raw would allow ../ traversal;
// validating `resolved` but opening `user_path` would be a TOCTOU gap.
}
# Double-decoding defeats single-pass validation (illustrative).
from urllib.parse import unquote
raw = "%252e%252e%252f" # attacker input
once = unquote(raw) # "%2e%2e%2f" -- still looks safe to a check
twice = unquote(once) # "../" -- the real, dangerous value
# Lesson: decode fully to the canonical form BEFORE validating, and validate
# the final form, not an intermediate one.
print(once, "->", twice)
Key Takeaways
- Input validation checks that data matches expectations before use; canonicalization reduces it to one standard form first, so validation cannot be fooled by alternative encodings.
- Allowlisting (accept exactly what is expected) is safe by construction; blocklisting (enumerate the bad) is an unwinnable arms race.
- Canonicalize, then validate the canonical form, then use that same form — validating one representation and using another is the classic traversal/injection bypass and is related to TOCTOU.
- Many bypasses exploit multiple representations of one value: path traversal variants, percent and double encoding, Unicode normalization, case and trailing characters.
- Validate type, length, range, and format at the trust boundary on the trusted side; prefer rejection over sanitize-and-continue; for memory safety, bound untrusted length and count fields here before they reach allocations and copies.
🧪 Practice
- Write an allowlist validator for an identifier and show it rejects inputs that a
../-blocklist would miss; explain why the allowlist is robust against unanticipated inputs.- Implement a safe "open a file within a directory" function using
realpathplus a prefix check, and demonstrate it blocks a../traversal attempt.- Show how double URL-encoding defeats a single-pass decode-and-check, and fix it by canonicalizing fully before validating.
- Interview-style: "Explain why validating a file path and then opening it can still be exploitable even if the validation logic is correct." Hint: consider whether the exact bytes you validated are the exact path you open (encoding, symlinks), and whether the filesystem state could change between the check and the open.
Safe Integer Arithmetic
Theory
Integer bugs (from the integer-overflow topic) are a leading cause of memory corruption because a wrong arithmetic result silently becomes a wrong size, index, or length. Safe integer arithmetic is the discipline of computing those values in ways that cannot silently wrap, truncate, or change sign — and of detecting when an operation would. Since so many overflows begin as malloc(a * b) or len = x - y, this discipline prevents a whole category of vulnerabilities at the arithmetic, before any memory is touched.
The guiding principles follow directly from how integers misbehave:
- Check before, not after. As the undefined-behaviour topic showed, a post-hoc overflow check (
if (a + b < a)) is both UB for signed types and often optimized away. The safe pattern checks whether the operation would overflow before performing it, using the operands and the type's known limits. For multiplication,if (a != 0 && b > SIZE_MAX / a) overflow;detects wrap without performing the overflowing multiply. - Prefer unsigned for sizes, but respect its wrap.
size_tis the correct type for sizes and counts, and its wrapping is well-defined (unlike signed), but well-defined wrapping is still a bug if unintended —a - bwithb > ayields a huge value. Checkb <= abefore subtracting. - Avoid mixing signedness. Signed/unsigned comparisons and conversions are where negative values become enormous (the
memcpysign-extension case). Keep size arithmetic uniformly unsigned, and convert deliberately with explicit range checks. - Beware truncation. Storing a 64-bit length into a 32-bit field drops the high bits; validate that a value fits its destination type before narrowing.
The best practice today is to use checked-arithmetic primitives rather than hand-rolled checks, because the compiler builtins are correct, efficient (they use the hardware overflow flag directly), and clear:
| Tool | Use |
|---|---|
__builtin_add_overflow(a, b, &r) etc. (GCC/Clang) | returns true on overflow; the standard C approach |
ckd_add/ckd_mul/ckd_sub (C23 <stdckdint.h>) | the standardized checked-arithmetic macros |
calloc(count, size) | allocates count*size with the multiply-overflow check built in |
reallocarray(ptr, count, size) | realloc with the same overflow check |
-fsanitize=undefined,integer | catches overflow/truncation at test time |
-ftrapv / -fwrapv | trap on, or define, signed overflow |
The single most valuable habit is using calloc/reallocarray for array allocations instead of malloc(count * size), because they perform exactly the multiplication-overflow check that the integer-bug topic identified as the most common pipeline — and they do it correctly, every time, with no extra code. Where a general checked operation is needed, __builtin_*_overflow (or C23's ckd_*) is the idiom. Hand-written checks are acceptable but must follow the before-not-after rule and use the right types, and they should be reserved for cases the builtins do not cover.
Examples
// Checked arithmetic with compiler builtins: correct and wrap-proof.
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
void *alloc_array(size_t count, size_t size) {
size_t bytes;
if (__builtin_mul_overflow(count, size, &bytes)) { // true if it WOULD wrap
return NULL; // refuse, do not under-
} // allocate
return malloc(bytes);
// Simpler still: `return calloc(count, size);` does this check internally.
}
size_t safe_sub(size_t a, size_t b) {
if (b > a) { /* underflow */ return 0; } // check BEFORE subtracting
return a - b; // well-defined and intended
}
// The before-not-after multiplication check, done by hand (when no builtin).
#include <stdint.h>
#include <stdbool.h>
bool mul_would_overflow(size_t a, size_t b) {
// Checks the operands against the type limit WITHOUT performing the
// overflowing multiply, so no UB and nothing for the optimizer to delete.
return a != 0 && b > SIZE_MAX / a;
}
# Prefer the allocation helpers that check for you.
# malloc(count * size) -- WRONG: product can wrap to a tiny size
# calloc(count, size) -- RIGHT: returns NULL on overflow
# reallocarray(p, count, size) -- RIGHT: realloc with the overflow check
# Catch integer bugs at test time.
gcc -fsanitize=undefined,integer -g int.c -o int # clang: -fsanitize=integer
./int # runtime error: unsigned integer overflow / implicit truncation ...
Key Takeaways
- Integer bugs become memory corruption when a wrapped, truncated, or sign-changed value is used as a size, index, or length, so safe arithmetic prevents a whole category of overflows at the source.
- Check whether an operation would overflow before performing it, using operands and type limits — never a post-hoc check, which is UB for signed types and may be optimized away.
- Use
size_tfor sizes (its wrap is defined but still a bug if unintended, so check subtraction), avoid mixing signedness, and validate that values fit before narrowing. - Prefer checked-arithmetic primitives —
__builtin_*_overflow, C23ckd_*, and especiallycalloc/reallocarrayfor array allocations — over hand-rolled checks. - Enable
-fsanitize=undefined,integerin testing to catch overflow and truncation that slip through.
🧪 Practice
- Replace a
malloc(count * size)withcalloc(or__builtin_mul_overflow+malloc) and demonstrate with wrapping values that the safe version refuses instead of under-allocating.- Write a
safe_subforsize_tthat checks for underflow before subtracting, and show that the naive version produces a huge value whenb > a.- Implement
mul_would_overflowwithout performing the multiply, and explain why the before-the-operation form is immune to the optimizer removing it.- Interview-style: "Why is
calloc(n, size)safer thanmalloc(n * size), and in what situation does the difference prevent an exploit?" Hint: recall what happens ton * sizewhen the product exceedsSIZE_MAX, whatmallocthen returns, and how a subsequent loop writingnelements behaves versuscallocreturning NULL.
Error Handling and Fail-Closed Design
Theory
Error handling is where a surprising number of vulnerabilities live, because errors are the paths least tested and least thought about — and an attacker's job is often precisely to force the program down an error path the developer assumed would rarely run. Secure error handling rests on one principle, fail closed (also called fail secure): when something goes wrong, the system should default to the safe, restrictive state — deny access, reject the request, abort the operation — never to the permissive one. The opposite, failing open, turns an error into a security bypass.
The "why" is that errors change control flow in ways attackers exploit. If an authorization check throws an exception and the catch block logs and continues, the check was effectively skipped — the attacker induced an error to bypass it. If a function that allocates memory does not check for NULL and proceeds to write, the allocation failure becomes a null-dereference or worse. If a parse error leaves a buffer partially initialized and the code uses it anyway, the error produced uninitialized-memory use (a leak source from the ASLR topic). In each case the bug is not the error itself but the unsafe handling of it.
The core practices:
- Check every error return. In C, functions signal failure through return values (
mallocreturnsNULL,readreturns-1,fopenreturnsNULL), and ignoring them is a leading bug source. Every call that can fail must have its result checked before its output is used. - Fail closed by default. The default case of a decision — the
else, thedefault:of aswitch, the catch-all of a validator — must deny or reject. An unknown state is an unsafe state. This mirrors the allowlist principle: anything not explicitly permitted is denied. - Do not use partial results after an error. If an operation fails midway, treat its outputs as invalid — do not read the half-filled buffer, do not trust the partially parsed structure.
- Clean up safely on every path. Freeing, closing, and unlocking must happen on error paths too, and exactly once — error paths are where double-frees and leaks (and the use-after-free from an early free) commonly hide.
- Do not leak information in error messages. Error output to untrusted users should be generic; detailed errors (addresses, stack traces, internal paths) are an information-disclosure source (recall the ASLR leak table) and belong in server-side logs, not responses.
A recurring concrete pattern is the fail-open authorization bug, which is worth recognizing because it is both common and severe. Code that grants access unless it can prove it should be denied — rather than denying unless it can prove it should be granted — fails open: any error in the proving (a timeout, an exception, an unexpected value) results in access. The fail-closed version inverts this: start denied, grant only on an explicit, successful positive check, and let every error path leave the state denied.
The defensive lesson connects to the whole chapter: many exploits work by driving the program into a rare state — an allocation failure under memory pressure, a malformed input that half-parses, an unexpected enum value — where the error handling is weakest. Testing error paths (fault injection, allocation-failure simulation, fuzzing that reaches error branches) and designing them to fail closed is therefore not a nicety but a core security practice.
Examples
// Checking every fallible call, and failing closed with safe cleanup.
#include <stdio.h>
#include <stdlib.h>
int load_config(const char *path, char **out) {
*out = NULL; // define the output up front
FILE *f = fopen(path, "r");
if (!f) return -1; // check: fopen can fail
char *buf = malloc(4096);
if (!buf) { fclose(f); return -1; } // check: malloc can fail; clean up
size_t n = fread(buf, 1, 4096, f);
if (ferror(f)) { // check: read can fail
free(buf); fclose(f); // clean up on the error path too
return -1; // do NOT return a partial buffer
}
buf[n < 4096 ? n : 4095] = '\0';
fclose(f);
*out = buf; // only expose output on success
return 0;
}
// Fail-open vs fail-closed authorization: the same check, opposite defaults.
#include <stdbool.h>
// FAIL-OPEN (dangerous): grants unless it can prove denial. Any error -> access.
bool authorized_bad(const char *user, const char *resource) {
bool denied = false;
if (policy_denies(user, resource)) // if this throws/errors, `denied`
denied = true; // stays false -> access GRANTED
return !denied;
}
// FAIL-CLOSED (safe): denies unless it can prove a positive grant.
bool authorized_good(const char *user, const char *resource) {
bool granted = false;
if (policy_explicitly_allows(user, resource)) // must SUCCEED and return
granted = true; // true to grant
return granted; // any error path -> DENIED
}
# Surface unchecked error returns and untested error paths.
gcc -O2 -Wall -Wextra -Wunused-result app.c -o app # warns on ignored results
# Static analysers flag missing NULL checks and unchecked returns:
clang --analyze app.c # reports "Value stored to ... is never checked"
cppcheck --enable=all app.c # flags resource leaks on error paths
# Fuzzing reaches error branches that unit tests rarely do (chapter 4).
Key Takeaways
- Error paths are the least-tested code and a frequent vulnerability home; secure error handling means failing closed — defaulting to the safe, restrictive state on any error, never the permissive one.
- Errors alter control flow in exploitable ways: a skipped check via an exception, an unchecked
NULLallocation, or a used-after-error partial buffer each turn an error into a memory or authorization bug. - Check every fallible call's result before using its output, make default/
else/catchbranches deny, discard partial results after an error, and clean up exactly once on every path. - The fail-open authorization bug — granting unless denial is proven — is common and severe; invert it to grant only on an explicit successful check so every error path leaves the state denied.
- Keep error messages to untrusted users generic to avoid information disclosure, and test error paths deliberately (fault injection, allocation-failure simulation, fuzzing), since exploits target rare states where handling is weakest.
🧪 Practice
- Rewrite a function that ignores
malloc,fopen, andreadreturn values so that each is checked and the error paths free/close correctly and return a safe failure.- Convert a fail-open authorization function to fail-closed, and explain how an induced error changes the outcome in each version.
- Compile a program with
-Wunused-resultand a static analyzer, and fix every unchecked error return and error-path resource leak they report.- Interview-style: "Describe how an attacker could turn a memory-allocation failure into a security vulnerability, and what coding practice prevents it." Hint: consider a program under memory pressure whose
mallocreturnsNULLbut that writes through the pointer anyway, and what checking the return and failing closed would do instead.
Avoiding Race Conditions and TOCTOU
Theory
A race condition is a bug where the correctness of an operation depends on the timing or interleaving of events that are not properly synchronized, so that a different-than-expected interleaving produces a wrong — and often exploitable — result. The security-critical subclass is TOCTOU (Time-Of-Check to Time-Of-Use): a program checks a condition, then acts on it, and an attacker changes the relevant state in the window between the check and the use, so the action operates on a different reality than the one that was validated. It is the concurrency analogue of the input-validation rule "validate and use the same thing" — here the thing can change underneath you.
The classic analogy is a bouncer who checks an ID at the door and then, a moment later, stamps the hand of whoever is now standing there — if people can swap places in that moment, the stamp lands on someone who was never checked. The check was valid when made; it was stale by the time it was used.
The canonical filesystem TOCTOU is the check-then-open pattern. A privileged program verifies that a file is safe to access (not a symlink, owned by the right user, within an allowed directory) and then opens it by path. Between the access()/stat() check and the open(), an attacker replaces the file (or a directory component) with a symlink to a sensitive target. The check validated the original file; the open follows the attacker's symlink to /etc/shadow. The gap is small but, with techniques to widen it (filesystem contention, careful timing), reliably exploitable.
A filesystem TOCTOU race
───────────────────────────────────────────────────────────────
privileged program attacker (racing)
───────────────────── ──────────────────────
stat("/tmp/x") -> safe, regular
rm /tmp/x ; ln -s /etc/shadow /tmp/x
open("/tmp/x") -> opens /etc/shadow via the symlink ✗
The path "/tmp/x" named a safe file at CHECK time and a sensitive
target at USE time. The name was re-pointed in the gap.
The general defenses attack the gap itself:
- Operate on handles, not names. The root cause is re-resolving a name (path, key, ID) twice. Open the resource once to get a stable handle (a file descriptor), then check and use that handle — a descriptor refers to the actual object, immune to later renaming. Use
fstat(fd)notstat(path),openatwithO_NOFOLLOWto refuse symlinks, and the*atfamily to pin directory components. - Make check-and-use atomic. Use operations that check and act in one indivisible step:
open(O_CREAT|O_EXCL)creates-only-if-absent atomically (no check-then-create gap),mkstempcreates a temp file atomically, andflock/lockfserialize access. - Synchronize shared state. For in-process races (shared memory, globals, signal handlers), use locks, atomics, or lock-free discipline so no other thread can change the state between check and use. Signal handlers are a notorious source: a handler interrupting a non-reentrant function can corrupt shared state (async-signal-safety).
- Drop privileges and reduce the window. Minimizing the time and privilege during which a race matters reduces exploitability, though it is a mitigation, not a fix.
The defensive lesson generalizes beyond files: any pattern of "look up X, verify X, then act on X by looking it up again" is a potential TOCTOU, whether X is a file, a database row, a user's permission, or a memory object. The robust shape is "acquire a stable reference to X once, then verify and use that reference," which eliminates the re-resolution the attacker exploits.
Examples
// TOCTOU: check-by-path then open-by-path -- the exploitable pattern.
#include <unistd.h>
#include <fcntl.h>
#include <sys/stat.h>
int read_safe_file_WRONG(const char *path) {
struct stat st;
if (stat(path, &st) != 0 || !S_ISREG(st.st_mode)) // CHECK (time of check)
return -1; // "it's a regular file"
// <-- attacker swaps `path` for a symlink to /etc/shadow here -->
int fd = open(path, O_RDONLY); // USE (time of use):
return fd; // may open the symlink
}
// Fixed: open ONCE to a handle, then check the HANDLE. No re-resolution.
int read_safe_file_RIGHT(const char *path) {
int fd = open(path, O_RDONLY | O_NOFOLLOW); // refuse a symlink atomically
if (fd < 0) return -1;
struct stat st;
if (fstat(fd, &st) != 0 || !S_ISREG(st.st_mode)) { // check the OPEN object
close(fd); // (the fd, not a name)
return -1;
}
return fd; // the fd refers to the exact object we validated; immune to
// any later rename/symlink swap of `path`.
}
// Atomic create-if-absent removes the check-then-create gap.
#include <fcntl.h>
int create_exclusive(const char *path) {
// O_CREAT|O_EXCL fails if the file already exists -- one indivisible step,
// so there is no window between "does it exist?" and "create it".
return open(path, O_CREAT | O_EXCL | O_WRONLY, 0600);
}
# Spot the dangerous check-then-use pattern in review.
grep -nE 'access\(|stat\(' *.c # a path check often followed by open(path)
# Prefer the handle-based and atomic alternatives:
# stat(path)+open(path) -> open(path,O_NOFOLLOW)+fstat(fd)
# if(!exists) create -> open(O_CREAT|O_EXCL) / mkstemp
Key Takeaways
- A race condition depends on event timing/interleaving; TOCTOU is the security subclass where state changes between a check and the use that relied on it, so the action operates on different reality than was validated.
- The classic case is check-by-path then open-by-path: an attacker swaps the path for a symlink in the gap, so the open follows it to a sensitive target.
- The root cause is re-resolving a name twice; the primary fix is to operate on a stable handle — open once, then
fstat/use the descriptor — which is immune to later renaming. - Make check-and-use atomic (
open(O_CREAT|O_EXCL),mkstemp, locks) and synchronize in-process shared state with locks/atomics, minding async-signal-safety in handlers. - The pattern generalizes: "look up, verify, then act by looking up again" is a TOCTOU for any resource; "acquire a stable reference once, then verify and use it" is the robust shape.
🧪 Practice
- Rewrite a check-then-open function to open once with
O_NOFOLLOWand validate the file descriptor withfstat, and explain why the fd-based version is immune to a symlink swap.- Replace a "check if a file exists, then create it" sequence with an atomic
open(O_CREAT|O_EXCL)and explain the race the atomic form eliminates.- Describe an in-process race involving a signal handler modifying a shared variable, and a synchronization approach that makes it safe.
- Interview-style: "A setuid program uses
access()to confirm the invoking user may read a file, thenopen()s it to read on their behalf. Why is this a classic vulnerability, and how do you fix it?" Hint: consider what the attacker can change about the path between theaccesscheck and theopen, and why operating on a file descriptor obtained once — or usingopenat/O_NOFOLLOW— removes the window.
Secure Use of Temporary Files and Privileges
Theory
Two operational concerns — temporary files and privilege management — recur in system programming and are each a well-trodden source of local privilege-escalation bugs. They are grouped here because both are about handling shared, dangerous resources carefully, and both reward the same principle: acquire the minimum, hold it for the shortest time, and use atomic, race-free operations.
Temporary files are dangerous because they usually live in a world-writable shared directory (/tmp, /var/tmp) where any local user can create, read, and manipulate files. A program that creates a temp file naively — guessing a name and opening it — is vulnerable to exactly the TOCTOU and symlink problems from the previous topic:
- Predictable names let an attacker pre-create the file (or a symlink with that name) before the program does, so the program's write lands on an attacker-chosen target or an attacker-readable file.
- Non-atomic creation (check-if-exists, then create) races with the attacker.
- The sticky bit on
/tmp(which prevents deleting others' files) mitigates some but not all of this, so programs must still defend themselves.
The correct practice is to use the library functions designed for this: mkstemp creates a temp file with a randomized, unique name and opens it atomically with O_CREAT|O_EXCL, returning a file descriptor — no name-guessing, no race, no symlink-following. mkdtemp does the same for directories. Never use the historically unsafe tmpnam, tempnam, or mktemp (which only generate a name, leaving a race before you open it). Better still, where supported, create unnamed temp files (O_TMPFILE) that have no directory entry at all and so cannot be targeted by name.
Privilege management concerns programs that run with elevated privileges (setuid-root binaries, services started as root) and must use that privilege for a narrow purpose. The governing principle is least privilege over time: hold elevated privilege only for the specific operation that needs it, and drop it permanently as soon as possible, so that a bug in the large remainder of the program executes unprivileged. A setuid-root program that does all its work as root turns any memory-corruption bug into root compromise; the same program that drops to the invoking user's identity after the one privileged step (say, binding a low port) confines a later bug to that user.
The critical subtleties, which are themselves classic bugs:
| Concern | Pitfall | Correct practice |
|---|---|---|
| Dropping privilege | dropping only the effective UID (reversible) | drop real, effective, and saved UIDs with setresuid, so it cannot be regained |
| Order of drops | dropping UID before GID (then lacking privilege to drop GID) | drop supplementary groups and GID before UID |
| Verifying the drop | assuming the drop succeeded | check the return value and re-read the IDs to confirm |
| Scope of privilege | running the whole program as root | privilege-separate: a small privileged helper, a large unprivileged worker |
| Environment | trusting PATH, LD_PRELOAD, etc. in a setuid context | sanitize the environment; the loader ignores LD_PRELOAD for setuid, but PATH and others still matter |
Privilege separation is the architectural form of this principle: split the program into a small, audited, privileged component and a larger unprivileged component that does the risky work (parsing, network I/O), communicating over a narrow interface. OpenSSH is the canonical example — the privileged monitor handles only authentication-critical operations while an unprivileged child processes the untrusted network protocol, so a bug in the protocol parser does not yield root. This connects directly to the sandboxing topic: least privilege and confinement are the "assume breach" defenses that limit what a successful exploit achieves.
Examples
// Safe temp file: atomic, unpredictable, race-free.
#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>
int make_temp(void) {
char tmpl[] = "/tmp/myprog.XXXXXX"; // XXXXXX is replaced with random chars
int fd = mkstemp(tmpl); // creates O_CREAT|O_EXCL atomically,
if (fd < 0) return -1; // returns an fd -- no name race, no
unlink(tmpl); // symlink follow. Optionally unlink
return fd; // now so it vanishes on close.
// NEVER: tmpnam()/mktemp() then open() -- that reintroduces the TOCTOU gap.
}
// Dropping privilege permanently and in the correct order.
#include <unistd.h>
#include <sys/types.h>
#include <grp.h>
#include <stdio.h>
int drop_privileges(uid_t uid, gid_t gid) {
// 1. Supplementary groups and GID FIRST (while still privileged enough).
if (setgroups(1, &gid) != 0) return -1;
if (setresgid(gid, gid, gid) != 0) return -1; // real, effective, saved
// 2. Then the UID -- all three, so privilege cannot be regained.
if (setresuid(uid, uid, uid) != 0) return -1;
// 3. VERIFY the drop actually took effect (do not assume success).
if (setresuid(0, 0, 0) == 0) // if we can regain root, the drop
return -1; // FAILED -- refuse to continue
return 0; // now the rest of the program is
// unprivileged; a later bug is contained
}
# Audit the local privilege surface these practices protect.
find / -perm -4000 -type f 2>/dev/null # setuid-root binaries: each should
# drop privilege ASAP and privilege-separate
ls -ld /tmp # confirm the sticky bit (drwxrwxrwt)
# Prefer O_TMPFILE where available (no directory entry at all):
# fd = open("/tmp", O_TMPFILE | O_RDWR, 0600); // nameless, untargetable
Key Takeaways
- Temp files and privilege management both reward the same discipline: acquire the minimum resource, hold it briefly, and use atomic, race-free operations.
- Temp files in shared world-writable directories are vulnerable to predictable-name and non-atomic-creation attacks; use
mkstemp/mkdtemp(atomic, random, returns an fd) orO_TMPFILE, nevertmpnam/mktemp. - Least privilege over time means holding elevated privilege only for the operation that needs it and dropping it permanently as soon as possible, so a later bug runs unprivileged.
- Dropping privilege correctly requires dropping real/effective/saved IDs (
setresuid/setresgid), dropping groups and GID before UID, and verifying the drop actually succeeded. - Privilege separation splits a program into a small privileged component and a large unprivileged one (as in OpenSSH), confining bugs in the risky code and connecting least privilege to the sandboxing/assume-breach model.
🧪 Practice
- Replace an unsafe temp-file creation (
tmpnam+open) withmkstemp, and explain which race and which symlink attack the change eliminates.- Write a
drop_privilegesfunction that drops GID before UID, uses the real/effective/saved variants, and verifies it cannot regain root; explain each step's purpose.- Enumerate your system's setuid-root binaries and, for one, reason about what a memory-corruption bug in it would grant and how privilege dropping would contain it.
- Interview-style: "A setuid-root network tool does all its work as root. Describe how you would restructure it to limit the impact of a parsing bug, and name the two techniques involved." Hint: think about dropping privilege immediately after the one operation that needs root, and splitting the untrusted-input handling into a separate unprivileged process communicating over a narrow channel.
Memory-Safe Languages: Rust and Managed Runtimes
Theory
Every bug in this chapter — buffer overflow, use-after-free, double-free, integer-driven corruption, format-string, TOCTOU-on-memory — is possible because C and C++ let the programmer manage memory manually and make mistakes the language does not catch. Memory-safe languages eliminate most of these classes by construction: the language itself guarantees that memory is accessed only within bounds and only during its valid lifetime, so the programmer cannot write the bug in the first place. This is the most decisive secure-coding measure available, and it reframes the whole chapter: the techniques above make unsafe code safer, but a memory-safe language makes the bug class impossible (within safe code).
There are two broad approaches, with different trade-offs:
Managed runtimes with garbage collection (Java, C#, Go, Python, JavaScript, and most other modern languages) achieve safety at runtime. They prevent spatial errors with bounds-checked array access (an out-of-range index throws an exception instead of corrupting memory) and temporal errors with garbage collection (memory is freed only when no reference remains, so use-after-free and double-free cannot occur — there is no manual free). The cost is runtime overhead (GC pauses, bounds-check instructions, indirection) and less control over memory layout and timing, which is why they are less used for the lowest-level systems code. But for the vast majority of software, the safety is well worth the overhead, and the entire family of memory-corruption exploits simply does not apply.
Rust achieves safety at compile time with no garbage collector and no runtime overhead, which is what makes it suitable for systems programming where C and C++ previously had no safe alternative. Its mechanism is the ownership and borrowing model enforced by the borrow checker:
- Every value has a single owner; when the owner goes out of scope, the value is freed exactly once — no double-free, no leak, no use-after-free, because the compiler tracks lifetimes.
- References are borrows, and the compiler enforces that you may have either many immutable borrows or exactly one mutable borrow at a time — never both — which prevents data races and iterator-invalidation bugs at compile time.
- Array and slice accesses are bounds-checked, so spatial overflows become panics (a safe, controlled abort), not corruption.
- Integer overflow panics in debug builds and wraps in a defined way in release (configurable), removing the UB.
The result is that safe Rust cannot exhibit buffer overflows, use-after-free, double-free, or data races — the compiler rejects the program. The escape hatch is the unsafe keyword, which lets a programmer perform operations the borrow checker cannot verify (raw pointer dereferences, FFI to C). This is a deliberate, auditable boundary: unsafe code is explicitly marked, so review and scrutiny concentrate on the small unsafe regions rather than the whole codebase. The security model is "safe by default, unsafe where necessary and visible."
The industry significance is large and worth stating plainly: studies from Microsoft and Google consistently attribute roughly 70% of serious security vulnerabilities to memory-safety bugs, and major projects (the Linux kernel, Android, Windows components, Chromium) are adopting Rust specifically to eliminate that majority at the source. Government and industry guidance now actively recommends memory-safe languages for new development. The pragmatic reality is that existing C/C++ will not be rewritten wholesale, so the secure-coding techniques earlier in this section remain essential — but for new code, choosing a memory-safe language is the single highest-leverage decision, because it removes the bug classes rather than mitigating them.
Examples
// Rust: the borrow checker rejects the bugs at COMPILE time.
fn main() {
// Use-after-free is impossible: `v` is freed when its owner scope ends,
// and the compiler forbids using a moved/dropped value.
let v = vec![1, 2, 3];
let r = &v[0]; // an immutable borrow of v
// v.push(4); // COMPILE ERROR: cannot mutate v while `r` borrows it
println!("{}", r); // borrow rules prevent iterator/UAF invalidation
// Bounds are checked: an out-of-range index panics safely, not corrupts.
let a = [10, 20, 30];
let i = 5;
// println!("{}", a[i]); // PANIC: index out of bounds -- a safe abort,
// never an out-of-bounds read of adjacent memory.
let _ = (a, i);
}
// Managed runtime (Java): bounds checks and GC make the bug classes vanish.
public class Safe {
public static void main(String[] args) {
int[] a = new int[3];
try {
int x = a[5]; // throws ArrayIndexOutOfBoundsException
} catch (ArrayIndexOutOfBoundsException e) {
// A controlled, catchable error -- NOT silent memory corruption.
}
// No manual free(): the GC reclaims objects only when unreachable,
// so use-after-free and double-free cannot be expressed at all.
}
}
// The `unsafe` escape hatch: explicitly marked, so audits focus here.
fn read_raw(p: *const u8) -> u8 {
unsafe { // the programmer asserts this deref is valid;
*p // the compiler does NOT verify it -- review this.
}
}
// Security model: safe-by-default, with every unverifiable operation
// confined to visible `unsafe` blocks that reviewers scrutinize closely.
Key Takeaways
- Memory-safe languages eliminate most of this chapter's bug classes by construction, making them the most decisive secure-coding measure: the techniques above make unsafe code safer, but a safe language makes the bug impossible in safe code.
- Managed runtimes (Java, Go, C#, Python, JS) achieve safety at runtime via bounds-checked indexing (out-of-range throws, not corrupts) and garbage collection (no manual free, so no UAF/double-free), at the cost of runtime overhead and less layout control.
- Rust achieves safety at compile time with no GC via ownership/borrowing: single-owner lifetimes prevent double-free/UAF, the aliasing rule (many readers xor one writer) prevents data races, and slices are bounds-checked.
- Safe Rust cannot exhibit buffer overflows, use-after-free, double-free, or data races; the
unsafekeyword is an explicit, auditable boundary for operations the compiler cannot verify. - About 70% of serious vulnerabilities are memory-safety bugs, so for new code choosing a memory-safe language is the highest-leverage decision; existing C/C++ still needs the earlier techniques since it will not be rewritten wholesale.
🧪 Practice
- Write a Rust snippet that the borrow checker rejects (e.g. mutating a vector while a reference to its element is live) and explain which C bug class the rejection corresponds to.
- Demonstrate in a managed language that an out-of-bounds array access produces a catchable exception rather than memory corruption, and relate this to the overflow topic.
- Explain what the
unsafekeyword in Rust permits, and why confining unverifiable operations to marked blocks improves auditability compared to C where every pointer operation is implicitly unsafe.- Interview-style: "If memory-safe languages eliminate ~70% of vulnerabilities, why are the C/C++ secure-coding practices in this section still essential?" Hint: consider the enormous existing base of C/C++ in operating systems and libraries, the cost of rewriting it, and what
unsafeblocks and FFI boundaries still require even in Rust projects.
Secure Coding Standards: CERT C and MISRA
Theory
Individual secure-coding habits are valuable, but a team needs a shared, enforceable, and auditable definition of what safe code looks like — otherwise each developer applies their own partial understanding and the gaps vary from file to file. Secure coding standards codify the accumulated knowledge of how a language's features are misused into concrete, checkable rules, so that safety becomes a property the whole codebase is measured against rather than a matter of individual diligence. They are the written, institutional form of the practices in this section.
The two most important for C and C++ serve related but distinct goals:
CERT C (and CERT C++), from Carnegie Mellon's Software Engineering Institute, is a security-focused standard. Its rules target exactly the vulnerability classes in this chapter — each rule names a dangerous construct, explains the risk, gives non-compliant and compliant examples, and assesses severity and likelihood. Rules are organized by topic (strings, integers, memory management, formatted I/O) and split into rules (violations are defects that must be fixed) and recommendations (good practice). Examples map directly onto what you have learned: "STR31-C: Guarantee that storage for strings has sufficient space for character data and the null terminator" (the off-by-one and overflow topics), "INT32-C: Ensure that operations on signed integers do not result in overflow" (the integer-bug topic), "FIO30-C: Exclude user input from format strings" (the format-string section), "MEM30-C: Do not access freed memory" (use-after-free). CERT C is, in effect, this chapter rendered as an enforceable checklist.
MISRA C (and MISRA C++), from the Motor Industry Software Reliability Association, is a safety-focused standard originally for automotive and now broadly used in safety-critical embedded systems (aerospace, medical, industrial). Its emphasis is on reliability and predictability: it restricts C to a safer subset by forbidding constructs whose behaviour is undefined, unspecified, implementation-defined, or merely error-prone — even some that are not security bugs per se but that make code hard to reason about. MISRA is stricter and more restrictive than CERT C (it bans things like dynamic memory allocation in the hardest safety levels, most uses of goto, and implicit conversions), reflecting its domain where a failure can cost lives and where certification requires demonstrable conformance.
| Aspect | CERT C | MISRA C |
|---|---|---|
| Primary goal | security (prevent vulnerabilities) | safety/reliability (predictable behaviour) |
| Origin / domain | SEI; general secure software | automotive; safety-critical embedded |
| Style | risk-assessed rules + recommendations | a restricted, safer language subset |
| Strictness | targets dangerous constructs | broader; forbids many error-prone-but-legal constructs |
| Conformance | guidance, often selectively adopted | often mandated, with documented deviations for certification |
The practical value, and the reason to know these exist, is threefold. First, they are enforceable by tooling: static analyzers (the next topic) ship with CERT and MISRA rule packs, so conformance is checked automatically in CI rather than by manual reading — turning a standard into a gate. Second, they provide a common vocabulary: "this violates INT32-C" is a precise, non-arguable review comment, replacing vague "this looks risky." Third, they encode institutional memory: rules exist because the construct caused real failures, so following them inherits decades of hard-won lessons without each team having to rediscover them.
The honest caveat is that standards are a means, not an end. Blind rule-following produces box-checking and deviation-paperwork without understanding; a developer who knows why INT32-C exists (the integer-overflow-to-heap-overflow pipeline) applies it sensibly and recognizes the cases the rule cannot express. The standards are most effective layered with tooling that enforces them, review that understands them, and the language-level safety of the previous topic where a memory-safe language can sidestep whole rule categories.
Examples
// CERT C rules mapped to this chapter's bug classes.
#include <string.h>
#include <stdio.h>
// Violates STR31-C (insufficient space for data + NUL) -> the off-by-one topic.
char bad_str[8];
void v_str(const char *in){ strcpy(bad_str, in); } // may overflow
// Violates INT32-C (signed overflow) -> the integer-bug topic.
int v_int(int a, int b){ return a + b; } // may overflow (UB)
// Violates FIO30-C (user input in a format string) -> the format-string section.
void v_fmt(const char *u){ printf(u); } // format-string bug
// Violates MEM30-C (access freed memory) -> the use-after-free topic.
#include <stdlib.h>
void v_uaf(void){ char *p = malloc(8); free(p); p[0] = 1; } // UAF
// Compliant versions follow this section's practices:
void ok_str(const char *in){ snprintf(bad_str, sizeof bad_str, "%s", in); }
int ok_int(int a, int b){ int r; return __builtin_add_overflow(a,b,&r)?-1:r; }
void ok_fmt(const char *u){ printf("%s", u); }
# Enforce standards automatically in CI with analyzer rule packs.
# MISRA checking (e.g. cppcheck's MISRA addon):
cppcheck --addon=misra --enable=all src/ # reports e.g. "misra-c2012-21.3"
# CERT checking (clang-tidy ships cert-* checks):
clang-tidy -checks='cert-*' src/*.c # reports e.g. "cert-err33-c"
# A violation becomes a precise, enforceable gate rather than a review opinion.
Where standards sit in the defense stack
───────────────────────────────────────────────────────────────
language choice (memory-safe?) ─► removes whole rule categories
coding standard (CERT/MISRA) ─► defines "safe" as checkable rules
static analysis (next topic) ─► enforces the rules automatically
code review ─► applies judgment the rules cannot encode
Each layer catches what the others miss; standards are the shared,
enforceable definition the tooling and reviewers work from.
Key Takeaways
- Secure coding standards turn accumulated knowledge of language misuse into concrete, checkable, auditable rules, making safety a codebase-wide property rather than individual diligence.
- CERT C is security-focused: risk-assessed rules targeting exactly this chapter's bug classes (STR31-C strings, INT32-C integer overflow, FIO30-C format strings, MEM30-C use-after-free) — effectively this chapter as an enforceable checklist.
- MISRA C is safety-focused for critical embedded systems: it restricts C to a safer subset, forbidding undefined/unspecified/error-prone constructs, and is stricter than CERT, often mandated with documented deviations for certification.
- Their value is tooling-enforceability (analyzer rule packs in CI), a common precise vocabulary for review, and encoded institutional memory of real failures.
- Standards are a means, not an end: they work best layered with language-level safety, static-analysis enforcement, and review that understands why each rule exists.
🧪 Practice
- For three functions containing a string overflow, a signed-overflow, and a format-string bug, name the CERT C rule each violates and rewrite them to comply.
- Run
cppcheck --addon=misraorclang-tidy -checks='cert-*'on a small C file with deliberate bugs and map each reported rule ID to the bug it caught.- Explain the difference in goals and strictness between CERT C and MISRA C, and give an example of a construct MISRA forbids that CERT would not necessarily flag.
- Interview-style: "A team adopts MISRA C and reports 100% conformance, yet a security review still finds exploitable bugs. How is that possible, and what does it say about the role of standards?" Hint: consider that a reliability-focused subset does not target every security issue, that deviations may be documented-and-accepted, and that conformance to rules is not the same as understanding the threats the rules only partially cover.
Compiler Warnings and Static Analysis in Practice
Theory
The cheapest, fastest, and most consistently available bug-finding tools are the ones already built into the compiler and the static analyzers that extend it. They read the code without running it (static analysis, from the program-analysis chapter) and flag constructs that are likely wrong — and because they run on every build, they catch bugs at the moment of writing, when they are cheapest to fix. Treating these tools as a primary defense, rather than noise to be suppressed, is one of the highest-return secure-coding practices.
The layered toolset, from cheapest to most thorough:
Compiler warnings are the first layer and cost nothing extra. Modern compilers detect a remarkable range of security-relevant mistakes if you enable the warnings: -Wall -Wextra turn on the broad set, and specific flags target this chapter's bug classes — -Wformat-security (format-string bugs), -Wconversion and -Wsign-conversion (implicit integer conversions behind signed/unsigned and truncation bugs), -Warray-bounds (some overflows), -Wshadow, -Wuninitialized. The decisive practice is -Werror: promoting warnings to errors so they must be fixed, not ignored. A warning that is allowed to persist is quickly buried under others and becomes invisible; -Werror keeps the signal clean. The common objection — "too many warnings in legacy code" — is addressed by enabling -Werror for specific high-value warnings or on new code, not by disabling warnings wholesale.
Static analyzers are the next layer, performing deeper interprocedural analysis than the compiler's local checks. They track values across functions, model memory and taint (from the program-analysis chapter), and find bugs that require reasoning about paths: null-dereferences on a specific branch, use-after-free across function calls, leaks on error paths, tainted data reaching a sink. The common tools:
| Tool | Niche |
|---|---|
Clang Static Analyzer / scan-build | path-sensitive C/C++ analysis; null-deref, leaks, UAF |
clang-tidy | lint + modernization + CERT checks; fast, CI-friendly |
cppcheck | standalone, no build integration needed; MISRA addon |
| Coverity / CodeQL / Semgrep | deep, enterprise/interprocedural; custom taint queries |
GCC -fanalyzer | built-in static analysis for C (double-free, leaks, UAF) |
Sanitizers (ASan/UBSan/MSan from the program-analysis chapter) are the complementary dynamic layer, catching at runtime (during tests and fuzzing) the bugs that static analysis cannot prove statically. The mature practice combines all three: compiler warnings as -Werror gates on every build, static analysis in CI on every pull request, and sanitizer-instrumented builds running the test suite and fuzzers.
Two practical realities shape effective use. First, false positives are the enemy of adoption: a tool that cries wolf gets ignored, so the discipline is to tune the analyzer (suppress known-safe patterns deliberately and specifically, not globally), fix or annotate real issues, and keep the signal trustworthy. Second, these tools are necessary but not sufficient: static analysis is undecidable in general, so it misses bugs (false negatives) and cannot replace review or testing — it is one layer that catches a large, cheap-to-fix fraction, freeing human review to focus on the design-level issues tools cannot see. The goal is a clean build and a clean analyzer run as the baseline, below which no code is merged, so that human attention is spent on what automation cannot catch.
Examples
# Layer 1: compiler warnings as hard gates.
gcc -O2 -Wall -Wextra \
-Wformat-security -Wconversion -Wsign-conversion \
-Warray-bounds -Wshadow -Wuninitialized \
-Werror \ # warnings MUST be fixed, not ignored
app.c -o app
# Layer 2: static analysis (run in CI on every change).
gcc -fanalyzer -c app.c # GCC's built-in analyzer: UAF, leaks, double-free
scan-build gcc -c app.c # Clang analyzer: path-sensitive bugs
clang-tidy -checks='cert-*,clang-analyzer-*' app.c
cppcheck --enable=all --error-exitcode=1 app.c # fail CI on findings
# CodeQL / Semgrep for custom taint queries (tainted length -> memcpy, etc.):
semgrep --config 'p/c' app.c
# Layer 3: dynamic analysis on the test suite (from the program-analysis chapter).
gcc -fsanitize=address,undefined -g app.c -o app_san && ./app_san < tests
// Each line triggers a specific warning/analysis finding -- all cheap to fix.
#include <stdio.h>
void demo(int len, const char *u) {
char buf[8];
int i = len; // -Wconversion if a narrowing conversion occurs
printf(u); // -Wformat-security: non-literal format
char *p = 0; // -fanalyzer: p is NULL...
*p = 1; // ...and dereferenced -> null-deref reported
buf[i] = 0; // -Warray-bounds may flag if i is known OOB
}
// A clean build with the full warning set and a clean analyzer run are the
// MINIMUM bar; they catch these before the code is ever committed.
Key Takeaways
- Compiler warnings and static analyzers read code without running it and flag likely bugs on every build, catching them when they are cheapest to fix — a primary defense, not noise to suppress.
- Enable broad and targeted warnings (
-Wall -Wextra -Wformat-security -Wconversion -Wsign-conversion -Warray-bounds) and promote them to errors with-Werrorso they must be fixed, keeping the signal clean. - Static analyzers (Clang analyzer,
clang-tidy,cppcheck, GCC-fanalyzer, Coverity/CodeQL/Semgrep) perform deeper interprocedural, path-sensitive analysis, finding null-derefs, UAF, leaks, and tainted-data flows that local warnings miss. - Combine the three layers: warnings as build gates, static analysis in CI per pull request, and sanitizer-instrumented builds running tests and fuzzers.
- Manage false positives deliberately to preserve trust and adoption, and remember these tools are necessary but not sufficient — static analysis is undecidable, so it misses bugs and complements rather than replaces review and testing.
🧪 Practice
- Take a C file with a format-string bug, an implicit narrowing conversion, and a null-dereference, and show which compiler warning or
-fanalyzerfinding catches each.- Configure a build with a strong warning set plus
-Werrorand fix every resulting error; explain why-Werrorchanges developer behaviour compared to warnings alone.- Run two different static analyzers on the same buggy file and compare what each finds and misses, including any false positives, and describe how you would handle the false positives.
- Interview-style: "A team says 'our static analyzer run is clean, so the code is secure.' Critique this claim." Hint: consider the theoretical limits of static analysis (undecidability, false negatives), the bug classes that only show up at runtime, and the design-level issues no tool can see, then describe what clean static analysis actually guarantees versus what it does not.
Code Review Checklists for Memory Safety
Theory
The final secure-coding practice is human review guided by a checklist — a structured list of the specific, recurring mistakes a reviewer should look for. The reason a checklist matters is that unaided human review is inconsistent: a reviewer catches what they happen to think of, which varies with fatigue, familiarity, and attention. A checklist converts review from ad hoc into systematic, ensuring that the known-dangerous patterns — the exact ones this chapter has catalogued — are checked every time, by everyone. It is the same principle (encode hard-won knowledge into a repeatable process) that made checklists transform aviation and surgery.
The complementary point is that review catches what tools cannot. Compilers and static analyzers excel at mechanical, local, pattern-matchable bugs; they are weak at semantic and design issues — a missing authorization check, a trust boundary in the wrong place, a length that is validated against the wrong maximum, a comment that reveals the code does not do what the function name claims. Human review with domain understanding is where those are caught. So review and tooling are layered: run the tools first (so humans do not waste attention on what automation catches), then review focuses on what remains.
A memory-safety review checklist, synthesizing this chapter, organized by the questions to ask at each dangerous construct:
- Buffers and copies. Is every buffer write bounded by the destination size? Is the size passed alongside every buffer pointer? Are
strcpy/strcat/sprintf/getsabsent (replaced by bounded, terminating forms)? Is NUL-termination guaranteed after every copy? Is the terminator counted in every allocation (+1)? - Indices and bounds. Is every index checked against both ends (negative and over-range)? Are loop bounds
<not<=where appropriate? Any<=, any unsigned countdown-to-zero — examined? - Integer arithmetic. Could any size/length/count calculation overflow, wrap, or truncate? Are allocations
calloc/reallocarrayor checked with__builtin_*_overflow? Any signed/unsigned mixing in a size context? - Memory lifetime. Does every allocation have exactly one owner and one
free? Any pointer used afterfree? Is each pointer set toNULLafter freeing? Any path that frees twice or leaks on error? - Untrusted input. Is every value from a trust boundary validated (type, length, range, format) against an allowlist before use? Is it canonicalized before validation? Are length/count fields from input bounded before reaching allocations and copies?
- Format strings. Is any attacker-influenced value ever the format argument (
printf(user)), rather than a constant with the data as a separate argument? - Error paths. Is every fallible call's result checked? Do all paths fail closed and clean up exactly once? Are partial results discarded after an error?
- Concurrency / TOCTOU. Is any resource re-resolved by name between check and use (open a handle once instead)? Is shared state synchronized? Are check-and-act operations atomic where needed?
- Privilege and secrets. Is privilege dropped as early as possible and verified? Are temp files created atomically? Are sensitive buffers wiped with a non-elidable zeroing (
explicit_bzero)?
The practical guidance for using a checklist well: it is a prompt for attention, not a box-ticking ritual — the reviewer still has to understand the code. It should be living (updated when a new bug class or a repeated mistake appears), proportionate (deeper scrutiny for code at trust boundaries and in unsafe/privileged regions), and paired with the tooling and standards from the previous topics so that review spends human judgment where it is uniquely valuable: the semantic and design questions that no analyzer can answer. The combination — memory-safe languages where possible, standards defining safe code, static analysis and warnings enforcing them, and checklist-guided review catching the rest — is the full, layered secure-coding practice this section has built toward.
Examples
A memory-safety code-review checklist (condensed, per construct)
───────────────────────────────────────────────────────────────
[ ] BUFFERS every write bounded? size travels with pointer?
no strcpy/strcat/sprintf/gets? NUL-terminated? +1 for NUL?
[ ] INDICES both ends checked (negative AND >= len)? < not <=?
[ ] INTEGERS size calc can't overflow/truncate? calloc/checked mul?
no signed/unsigned mixing in sizes?
[ ] LIFETIME one owner, one free? no use-after-free? NULL after free?
no double-free or leak on error paths?
[ ] INPUT validated (type/len/range/format, allowlist) at boundary?
canonicalized BEFORE validation? length fields bounded?
[ ] FORMAT no attacker value as a format string?
[ ] ERRORS every return checked? fail closed? cleanup once? no partials?
[ ] TOCTOU handle not re-resolved name? shared state synchronized?
[ ] PRIVILEGE dropped early & verified? atomic temp files? secrets wiped?
Run tools FIRST; use this for the semantic/design bugs tools miss.
// A snippet and the checklist questions it should trigger in review.
int handle(int sock) {
char buf[128];
int len = recv_len(sock); // INPUT: is `len` validated? INTEGERS:
char *data = malloc(len); // signed? bounded? -> calloc/check
recv(sock, data, len); // BUFFERS: is `len` <= malloc size?
char path[64];
sprintf(path, "/data/%s", data); // BUFFERS: sprintf unbounded! FORMAT ok
FILE *f = fopen(path, "r"); // ERRORS: fopen result unchecked.
fread(buf, 1, len, f); // BUFFERS: len may exceed buf (128)!
return 0; // LIFETIME: `data` leaked; `f` leaked.
}
// A checklist walk flags: unvalidated/possibly-negative len, unchecked malloc,
// unbounded sprintf, unchecked fopen, len-vs-buf overflow, and two leaks --
// a half-dozen issues a structured review catches that a glance might not.
Key Takeaways
- A code-review checklist converts human review from inconsistent and ad hoc into systematic, ensuring the known-dangerous patterns from this chapter are examined every time by everyone.
- Review catches what tools cannot: semantic and design bugs (misplaced trust boundaries, wrong validation targets, missing authorization) that compilers and analyzers miss.
- The checklist walks each dangerous construct with specific questions: bounded buffers and NUL-termination, both-ends index checks, overflow-safe integer arithmetic, single-owner lifetimes, allowlist input validation after canonicalization, no attacker-controlled format strings, fail-closed error handling, TOCTOU-safe resource use, and early verified privilege dropping.
- Use the checklist as a prompt for understanding, not box-ticking; keep it living, apply deeper scrutiny at trust boundaries and in privileged/unsafe code, and run tools first so review focuses on what they cannot catch.
- The complete secure-coding practice is layered: memory-safe languages where possible, standards defining safe code, warnings and static analysis enforcing them, and checklist-guided human review catching the semantic remainder.
🧪 Practice
- Apply the checklist to the
handlefunction above and enumerate every issue it surfaces, writing the specific fix for each.- Build a short review checklist tailored to a codebase you know, adding any recurring mistake you have seen that the generic list omits.
- Explain, with one example each, a bug class that static analysis catches better than review and a bug class that review catches better than static analysis.
- Interview-style: "Why keep a code-review checklist when you already run compiler warnings, static analysis, and sanitizers in CI?" Hint: consider the categories of bug that are semantic or design-level — a validation against the wrong limit, a trust boundary in the wrong place, a missing authorization check — and whether any automated tool can recognize that the code does the wrong thing even when it is memory-safe.
<a id="6-system-and-network-security"></a>