11. Systems Programming
Everything so far stayed inside a single process using only the standard library. This chapter crosses the boundary into the operating system: starting and reaping processes, reading and writing through raw file descriptors, talking over the network, and running several threads at once. These interfaces are POSIX rather than ISO C, they report errors through a different convention than you may expect, and they are where a program stops being a calculation and starts being a participant in a running system.
<a id="111-processes-and-the-os-interface"></a>
11.1 Processes and the OS Interface
A process is a running program plus everything the kernel tracks on its behalf: memory, open files, a user identity, and a parent. This section covers how a process receives its inputs, how it asks the kernel for services, how it creates and replaces other processes, and how it learns that something happened to it.
Command-Line Arguments and Environment
Theory
A program needs inputs before it can do anything, and the operating system supplies two sets of them at startup. Both arrive without any parsing, allocation, or system call on your part — they are simply there when main begins.
Command-line arguments arrive through main's parameters:
int main(int argc, char *argv[])
argc is the count of arguments, and argv is an array of that many string pointers plus a terminating NULL. The essential detail that trips people up: argv[0] is the program's own name, not the first real argument. So a program invoked as ./tool -v file.txt receives argc == 3, with the actual arguments at argv[1] and argv[2]. Loops must start at 1.
Two guarantees are worth knowing. argv[argc] is always NULL, so you can iterate either by counting or by walking until NULL. And the strings are writable — modifying them is legal, which is how programs like ps show altered command lines, though it is rarely a good idea.
The shell, not your program, does the splitting. ./tool "two words" delivers one argument containing a space; ./tool *.txt delivers however many files the shell's glob matched, because the shell expanded it before your program ever ran. Understanding that boundary explains most "why did my program see something different from what I typed" confusion.
Environment variables are the second channel: a set of NAME=value strings inherited from the parent process. Where arguments are explicit and per-invocation, the environment is ambient and inherited — the right place for configuration that applies to every invocation (PATH, HOME, LANG, TZ) rather than to this particular one.
Access it with getenv("NAME"), which returns a pointer to the value or NULL if unset. That pointer is owned by the environment, not by you: do not free it, do not modify it, and copy it if you need it after a later setenv call, which may reallocate the whole block. POSIX adds setenv, unsetenv, and the environ global for walking the entire set.
There is a third, non-standard form of main taking char *envp[]. It works on POSIX systems but getenv and environ are the portable path.
For anything beyond two or three flags, parse arguments with getopt rather than by hand. It handles the conventions users expect — clustered short flags (-abc), an option's argument attached or separate (-ofile or -o file), and -- terminating the options — and it reports unknown options for you. Rolling this by hand gets subtly wrong in ways users notice.
Examples
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
/* POSIX declares 'environ' but no header does; this is the standard idiom. */
extern char **environ;
int main(int argc, char *argv[])
{
/* argv[0] is the PROGRAM NAME -- real arguments start at index 1. */
printf("invoked as: %s\n", argv[0]);
printf("argc = %d\n", argc);
for (int i = 1; i < argc; i++) /* note: i = 1, not 0 */
printf(" argv[%d] = \"%s\"\n", i, argv[i]);
/* argv[argc] is guaranteed NULL, so walking works too. */
printf("argv[argc] is %s\n", argv[argc] == NULL ? "NULL" : "not NULL");
/* --- environment: ambient configuration, inherited from the parent --- */
const char *home = getenv("HOME"); /* NULL if unset */
const char *path = getenv("PATH");
printf("HOME = %s\n", home ? home : "(unset)");
printf("PATH = %.40s...\n", path ? path : "(unset)");
/* The returned pointer belongs to the environment: never free it, and
copy it if you need it to survive a later setenv. */
if (home) {
char *copy = strdup(home);
if (copy) { printf("safe copy: %s\n", copy); free(copy); }
}
/* Count the whole environment by walking to the NULL terminator. */
size_t n = 0;
for (char **e = environ; *e; e++) n++;
printf("%zu environment variables; first three:\n", n);
for (size_t i = 0; i < 3 && environ[i]; i++)
printf(" %.60s\n", environ[i]);
/* Read a tunable with a sensible default -- the common config pattern. */
const char *level = getenv("APP_LOG_LEVEL");
int verbosity = level ? atoi(level) : 1;
printf("verbosity = %d\n", verbosity);
setenv("APP_CHILD_MARKER", "1", 1); /* 1 = overwrite if present */
printf("marker now: %s\n", getenv("APP_CHILD_MARKER"));
return 0;
}
Proper option parsing with getopt:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h> /* getopt, optarg, optind, opterr */
int main(int argc, char *argv[])
{
int verbose = 0;
const char *outfile = NULL;
int count = 1;
int opt;
/* The optstring: a letter alone is a flag, a letter followed by ':'
takes an argument, and a leading ':' makes a missing argument
report as ':' rather than '?'. */
while ((opt = getopt(argc, argv, ":vo:n:h")) != -1) {
switch (opt) {
case 'v': verbose++; break; /* -vvv increments thrice */
case 'o': outfile = optarg; break; /* optarg holds the value */
case 'n': count = atoi(optarg); break;
case 'h':
printf("usage: %s [-v] [-o FILE] [-n N] [files...]\n", argv[0]);
return EXIT_SUCCESS;
case ':': /* option missing its argument */
fprintf(stderr, "%s: option -%c requires an argument\n", argv[0], optopt);
return EXIT_FAILURE;
case '?': /* unrecognized option */
default:
fprintf(stderr, "%s: unknown option -%c\n", argv[0], optopt);
return EXIT_FAILURE;
}
}
printf("verbose=%d outfile=%s count=%d\n",
verbose, outfile ? outfile : "(stdout)", count);
/* getopt leaves 'optind' at the first NON-option argument. */
printf("%d operand(s):\n", argc - optind);
for (int i = optind; i < argc; i++)
printf(" %s\n", argv[i]);
return EXIT_SUCCESS;
}
WHAT THE SHELL DOES BEFORE YOUR PROGRAM RUNS
you type: ./tool -v "two words" *.txt
the shell: splits on whitespace, honours quoting, expands globs,
substitutes variables, then calls execve with the result
your main receives:
argc = 5
argv[0] = "./tool" <- the program name
argv[1] = "-v"
argv[2] = "two words" <- quoting preserved the space
argv[3] = "a.txt" <- the glob was expanded by the SHELL
argv[4] = "b.txt"
argv[5] = NULL <- always
ARGUMENTS vs ENVIRONMENT
arguments explicit, per-invocation, ordered, visible in 'ps'
environment ambient, inherited by every child, unordered, hidden
Key Takeaways
argv[0]is the program's own name, so real arguments start at index 1, andargv[argc]is alwaysNULL.- The shell performs splitting, quote handling, and glob expansion before your program starts — your program sees only the result.
- Environment variables are inherited ambient configuration;
getenvreturns a pointer you must not free or modify, and which a latersetenvmay invalidate. - Use arguments for per-invocation inputs and the environment for settings that should apply to every invocation and be inherited by children.
- Parse anything beyond a couple of flags with
getopt, which handles clustering, attached arguments, and--, and leavesoptindat the first operand.
🧪 Practice
- Write a program that prints each argument with its index and length, then run it with quoted strings, globs, and no arguments at all.
- Extend the
getoptexample with a-d DIRoption and a mandatory operand, reporting a usage error when the operand is missing.- Write a program that reads a configuration value from an argument if given, otherwise from an environment variable, otherwise from a built-in default, and prints which source won.
- Interview-style: "When should a setting be a command-line flag rather than an environment variable?" Hint: think about which one a child process inherits automatically, and which one shows up in
ps.
System Calls vs Library Calls
Theory
Your program cannot read a file. It cannot allocate a page of memory, open a socket, or create a process. Those actions require privileges the CPU denies to ordinary code, because a program that could do them directly could also read another process's memory or scribble on the disk.
The CPU therefore runs in (at least) two modes. User mode is where your code runs, with restricted instructions and access only to its own address space. Kernel mode is where the operating system runs, with full access to hardware. A system call is the controlled doorway between them: a special instruction that transfers control to a fixed entry point in the kernel, which validates your request, performs it, and returns.
The cost of that doorway is what makes the distinction practical rather than academic. A system call must switch privilege level, save and restore register state, validate every pointer you passed, and often flush CPU state that speculation depends on. It costs on the order of hundreds of nanoseconds to a few microseconds — perhaps a hundred times an ordinary function call.
A library call, by contrast, is an ordinary function in a library linked into your process. strlen never leaves user mode. And this is where the standard library earns its keep: fprintf is a library call that buffers your data and issues a write system call only when the buffer fills. Writing a million short lines through fprintf might make 250 system calls; writing them through raw write makes a million.
That gives the practical rule: prefer buffered library I/O for ordinary work, and drop to raw system calls when you need control the library does not offer — exact byte counts, non-blocking behavior, or a file descriptor to hand to select.
Three signatures of a system call are worth recognizing in the wild:
They report errors differently from most library functions. POSIX system calls return -1 on failure and set errno, without exception. Some can also return a short count — write may write fewer bytes than you asked without that being an error at all.
They can be interrupted. A blocking system call interrupted by a signal returns -1 with errno == EINTR, having done nothing. This is not a failure to report to the user; it is a request to retry.
They are the boundary the system can observe. strace shows every system call a process makes, with arguments and results — which makes it the single most useful tool for understanding what a program actually does, as opposed to what its source suggests.
Note that the C standard library's own functions are usually thin wrappers: fopen calls open, malloc calls mmap or brk (but only occasionally, satisfying most requests from its own pool). The wrapper is where buffering, retrying, and portability live.
Examples
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h> /* write, close -- SYSTEM CALLS */
#include <fcntl.h> /* open -- SYSTEM CALL */
#include <errno.h>
#include <time.h>
#define LINES 20000
/* Every fputs may be a library call only: data accumulates in the stdio
buffer, and one write() is issued per bufferful (typically 4 KB). */
static double with_buffered_io(const char *path)
{
FILE *f = fopen(path, "w");
if (!f) { perror(path); return -1; }
struct timespec t0, t1;
clock_gettime(CLOCK_MONOTONIC, &t0);
for (int i = 0; i < LINES; i++)
fprintf(f, "line %d\n", i); /* buffered: few syscalls */
fclose(f); /* flushes the tail */
clock_gettime(CLOCK_MONOTONIC, &t1);
return (t1.tv_sec - t0.tv_sec) + (t1.tv_nsec - t0.tv_nsec) / 1e9;
}
/* Raw write(): one SYSTEM CALL per line, each crossing into the kernel. */
static double with_raw_syscalls(const char *path)
{
int fd = open(path, O_WRONLY | O_CREAT | O_TRUNC, 0644);
if (fd < 0) { perror(path); return -1; }
struct timespec t0, t1;
clock_gettime(CLOCK_MONOTONIC, &t0);
for (int i = 0; i < LINES; i++) {
char buf[32];
int n = snprintf(buf, sizeof buf, "line %d\n", i);
if (write(fd, buf, (size_t)n) < 0) { perror("write"); break; }
}
close(fd);
clock_gettime(CLOCK_MONOTONIC, &t1);
return (t1.tv_sec - t0.tv_sec) + (t1.tv_nsec - t0.tv_nsec) / 1e9;
}
int main(void)
{
printf("buffered (fprintf): %.4f s\n", with_buffered_io("/tmp/buffered.txt"));
printf("raw (write): %.4f s\n", with_raw_syscalls("/tmp/raw.txt"));
remove("/tmp/buffered.txt");
remove("/tmp/raw.txt");
return 0;
}
The two error conventions, side by side:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <string.h>
int main(void)
{
/* LIBRARY call: returns NULL on failure, and also sets errno. */
FILE *f = fopen("/no/such/file", "r");
if (f == NULL)
printf("fopen -> NULL, errno=%d (%s)\n", errno, strerror(errno));
/* SYSTEM call: returns -1 on failure and sets errno. ALWAYS -1,
never NULL, and a non-negative return is a valid descriptor. */
int fd = open("/no/such/file", O_RDONLY);
if (fd < 0)
printf("open -> -1, errno=%d (%s)\n", errno, strerror(errno));
/* A short write is NOT an error -- the loop that handles it correctly
is covered later, but note that the return value is a COUNT. */
fd = open("/tmp/demo.txt", O_WRONLY | O_CREAT | O_TRUNC, 0644);
if (fd >= 0) {
const char msg[] = "hello\n";
ssize_t n = write(fd, msg, sizeof msg - 1);
printf("write -> %zd bytes requested %zu\n", n, sizeof msg - 1);
close(fd);
remove("/tmp/demo.txt");
}
/* EINTR: a blocking call interrupted by a signal. The correct
response is to RETRY, not to report a failure. */
printf("EINTR is %d -- retry, do not abort\n", EINTR);
return 0;
}
# strace shows the syscall boundary directly. Count them for each version:
$ strace -c -e trace=write ./buffered_version
% time seconds usecs/call calls errors syscall
100.00 0.000512 2 196 write
$ strace -c -e trace=write ./raw_version
% time seconds usecs/call calls errors syscall
100.00 0.041233 2 20000 write
# ^^^^^ 100x more crossings for the same data
USER MODE / KERNEL MODE
+-----------------------------------------------------+
| YOUR PROCESS (user mode) |
| |
| strlen() memcpy() qsort() <- pure library, |
| never leaves |
| fprintf() ---> [ stdio buffer ] |
| | |
+------------------------|----------------------------+
| write() <- SYSTEM CALL, only when
| the buffer fills
=========================|============================ privilege boundary
v
+-----------------------------------------------------+
| KERNEL (kernel mode) |
| validate arguments, copy data, schedule I/O |
+-----------------------------------------------------+
Crossing costs ~100x an ordinary call. Buffering exists to cross rarely.
| Library call | System call | |
|---|---|---|
| Runs in | user mode | kernel mode |
| Cost | nanoseconds | hundreds of ns to microseconds |
| Examples | strlen, qsort, malloc (usually) | read, write, open, fork |
| Error convention | varies (NULL, EOF, 0) | always -1 plus errno |
| Interruptible by a signal | no | yes — EINTR |
Visible to strace | no | yes |
| Buffered | often | never |
Key Takeaways
- A system call crosses a privilege boundary into the kernel and costs roughly a hundred times an ordinary function call; a library call stays in user mode.
- Buffered stdio exists to turn many library calls into few system calls — this is why
fprintfin a loop vastly outperforms rawwritein a loop. - POSIX system calls signal failure with
-1pluserrno, uniformly; library functions use varied conventions, so read each one's documentation. - A system call interrupted by a signal fails with
EINTRand must be retried, not reported as an error. stracelists every system call a process makes, making it the fastest way to see what a program actually asks of the kernel.
🧪 Practice
- Run both versions of the I/O benchmark under
strace -cand record the syscall counts and total time for each.- Use
setvbufto give the buffered version a 1-byte buffer, then re-measure and explain the result.- Write a program that reads a file with
fgetcand another that reads it withreadone byte at a time, and compare the syscall counts.- Interview-style: "Why is
writein a loop so much slower thanfprintfin a loop, when both end up writing the same bytes?" Hint: count how many times each one crosses into the kernel.
fork, exec, and wait
Theory
Unix creates new processes in a way that surprises everyone at first: there is no "run this program" call. Instead there are two separate operations, and understanding why they are separate is understanding Unix process management.
fork() duplicates the calling process. The child gets a copy of the parent's memory, its open file descriptors, its working directory, and its environment. Both processes then continue from the same point — the return from fork — as two independent processes.
The famous consequence is that fork returns twice, with different values, which is how each copy knows which one it is:
| Return value | Meaning | Runs in |
|---|---|---|
0 | you are the child | child |
> 0 | the child's PID | parent |
-1 | fork failed; no child exists | parent only |
Copying an entire address space sounds ruinously expensive. It is not, because of copy-on-write: the kernel maps the same physical pages into both processes marked read-only, and only makes a private copy of a page when one of them writes to it. A fork is therefore cheap regardless of how much memory the parent holds.
exec() replaces the current process image with a different program. The PID stays the same, open file descriptors stay open (unless marked close-on-exec), but the code, data, heap, and stack are discarded and replaced. A successful exec never returns — there is nothing to return to. Reaching the line after exec means it failed.
Splitting creation from replacement is what makes shell features possible. Between fork and exec the child is still running your code and can redirect its own file descriptors, change its working directory, drop privileges, or set up a pipe — and then exec the target program, which inherits all of it without knowing anything about it. A combined "spawn" call would have to anticipate every such adjustment as a parameter.
The exec family varies along two axes, encoded in the suffix letters:
| Function | Arguments | Path search |
|---|---|---|
execl | list, ending in (char *)NULL | no — full path required |
execv | vector (char *[]) | no |
execlp / execvp | list / vector | yes — searches PATH |
execle / execve | plus an explicit environment | no |
wait() and waitpid() let the parent collect a finished child's exit status. This is not optional bookkeeping. When a child exits, the kernel keeps a small record of it until the parent collects it; an uncollected finished child is a zombie, visible in ps as <defunct>, holding a process-table slot forever. A long-running parent that forks without waiting eventually exhausts the process table.
The mirror case: if the parent exits first, the child is orphaned and re-parented to init (PID 1), which reaps it automatically. Orphans are harmless; zombies are not.
wait blocks until any child exits. waitpid can target a specific child and, with WNOHANG, can poll without blocking. The status it fills in is a packed integer that must be decoded with macros — WIFEXITED/WEXITSTATUS for a normal exit, WIFSIGNALED/WTERMSIG for death by signal. Reading the raw integer is meaningless.
Examples
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h> /* fork, exec*, getpid, getppid */
#include <sys/wait.h> /* wait, waitpid, W* macros */
#include <errno.h>
#include <string.h>
int main(void)
{
printf("parent: pid=%d\n", (int)getpid());
fflush(stdout); /* flush BEFORE forking, or the child inherits a
copy of the buffer and the output appears twice */
pid_t pid = fork(); /* this call returns TWICE, in two processes */
if (pid < 0) {
perror("fork");
return EXIT_FAILURE;
}
if (pid == 0) {
/* ------------------- CHILD ------------------- */
printf("child: pid=%d ppid=%d\n", (int)getpid(), (int)getppid());
/* This is the window that makes fork/exec powerful: we are still
running our own code and can adjust the environment the new
program will inherit. */
setenv("CHILD_MARKER", "set-by-parent-code", 1);
/* execvp searches PATH and replaces this process image entirely.
On success it NEVER RETURNS. */
char *args[] = { "echo", "hello from the exec'd program", NULL };
execvp(args[0], args);
/* Reaching here means exec FAILED. Use _exit, not exit: the stdio
buffers belong to the parent's copy and must not be flushed twice. */
fprintf(stderr, "execvp: %s\n", strerror(errno));
_exit(127); /* 127 is the shell's "not found" code */
}
/* ------------------- PARENT ------------------- */
printf("parent: forked child %d, waiting\n", (int)pid);
int status;
pid_t done = waitpid(pid, &status, 0); /* blocks until the child ends */
if (done < 0) { perror("waitpid"); return EXIT_FAILURE; }
/* The status integer is PACKED: decode it with macros, never read it raw. */
if (WIFEXITED(status))
printf("child exited normally with code %d\n", WEXITSTATUS(status));
else if (WIFSIGNALED(status))
printf("child killed by signal %d (%s)\n",
WTERMSIG(status), strsignal(WTERMSIG(status)));
return EXIT_SUCCESS;
}
Copy-on-write, zombies, and reaping several children:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/wait.h>
static int global_counter = 100;
int main(void)
{
int local = 5;
/* --- memory is COPIED, not shared: the child's writes are private --- */
pid_t pid = fork();
if (pid < 0) { perror("fork"); return 1; }
if (pid == 0) {
global_counter += 1000; /* copy-on-write triggers HERE */
local += 1000;
printf("child: global=%d local=%d\n", global_counter, local);
_exit(0); /* 1100 / 1005 */
}
wait(NULL); /* reap it, discarding the status */
printf("parent: global=%d local=%d (unchanged)\n", global_counter, local);
/* --- forking several children and reaping ALL of them --- */
const int N = 3;
for (int i = 0; i < N; i++) {
pid_t c = fork();
if (c < 0) { perror("fork"); break; }
if (c == 0) {
/* Each child exits with a distinct status so we can tell them apart. */
_exit(i + 1);
}
}
/* Loop until wait reports ECHILD ("no children left"), which is the
only reliable way to reap every child regardless of finish order. */
int status;
pid_t child;
while ((child = wait(&status)) > 0) {
if (WIFEXITED(status))
printf("reaped %d -> exit %d\n", (int)child, WEXITSTATUS(status));
}
/* Skipping this loop would leave three ZOMBIES: finished processes whose
exit status nobody collected, each holding a process-table slot. */
printf("all children reaped\n");
return 0;
}
fork() SPLITS, exec() REPLACES
BEFORE FORK AFTER FORK AFTER EXEC IN CHILD
+-------------+ +-------------+ +-------------+
| parent | | parent | | parent |
| pid 100 | | pid 100 | | pid 100 |
| code: mine | fork() | fork -> 101 | | waiting... |
+-------------+ -----> +-------------+ +-------------+
+-------------+ +-------------+
| CHILD | execvp() | CHILD |
| pid 101 | -------> | pid 101 |
| fork -> 0 | | code: /bin/echo
| same code | | fds INHERITED
| copy-on-write | memory REPLACED
+-------------+ +-------------+
The gap between fork and exec is where the child redirects its own
file descriptors, changes directory, or drops privileges -- which is
exactly how a shell implements ls > out.txt
ZOMBIE: child exited, parent never waited -> record kept forever
ORPHAN: parent exited first -> child re-parented to init, reaped for you
| Call | Creates a process | Returns | On success |
|---|---|---|---|
fork | yes, a duplicate | twice: 0 in child, PID in parent | both continue |
exec* | no — replaces this one | never | old image is gone |
wait / waitpid | no | the reaped child's PID | status decoded via W* macros |
system | yes (forks a shell) | the shell's status | convenient, but unsafe with untrusted input |
posix_spawn | yes | 0 or an error code | fork+exec combined, for constrained platforms |
Key Takeaways
forkduplicates the process and returns twice —0to the child, the child's PID to the parent — with memory copied lazily through copy-on-write.execreplaces the current process image and never returns on success; any code after it runs only if it failed.- Splitting the two is what allows the child to redirect descriptors, change directory, or drop privileges before the new program starts.
- Always reap children with
waitorwaitpid, or they become zombies that occupy process-table slots permanently; orphans, by contrast, are adopted and reaped byinit. - Decode the wait status with
WIFEXITED/WEXITSTATUSandWIFSIGNALED/WTERMSIG, and call_exitrather thanexitin a child that must not flush the parent's inherited stdio buffers.
🧪 Practice
- Write a program that forks three children, each sleeping a different duration, and prints them in completion order as they are reaped.
- Deliberately create a zombie by forking without waiting, then observe it with
ps -el | grep defunctbefore the parent exits.- Implement a minimal shell loop that reads a line, splits it on spaces, forks,
execvps the result, and reports the exit status.- Interview-style: "Why does Unix separate
forkfromexecinstead of providing a single spawn call?" Hint: think about what a shell must do between deciding to run a program and actually running it.
Exit Status and Signals
Theory
A process ends in one of two ways, and the distinction matters because they carry different information and are reported differently.
Normal termination happens when the program returns from main or calls exit. It supplies an exit status: a small integer conventionally meaning 0 for success and non-zero for failure. On POSIX only the low 8 bits survive, so exit(256) is seen as 0 and exit(-1) as 255. Shells expose the status as $?, and it is what makes cmd1 && cmd2 work.
Abnormal termination happens when a signal kills the process. There is no exit status — instead the wait status records which signal. This is why a program killed by SIGKILL and a program that returned 9 are entirely different events, even though both might loosely be called "exit 9".
A signal is an asynchronous notification delivered to a process: the software equivalent of a tap on the shoulder. Some come from the kernel reporting a fault the process caused (SIGSEGV for an invalid memory access, SIGFPE for integer division by zero), some from the user (SIGINT from Control-C), some from other processes (kill), and some from the process's own state changing (SIGCHLD when a child exits).
Every signal has a default action, and knowing the categories saves a lot of guessing:
| Signal | Number | Default action | Typical source | Catchable |
|---|---|---|---|---|
SIGINT | 2 | terminate | Control-C | yes |
SIGQUIT | 3 | terminate + core | Control-\ | yes |
SIGKILL | 9 | terminate | kill -9 | no |
SIGSEGV | 11 | terminate + core | invalid memory access | yes |
SIGPIPE | 13 | terminate | writing to a closed pipe | yes |
SIGTERM | 15 | terminate | kill (polite default) | yes |
SIGCHLD | 17 | ignore | a child exited | yes |
SIGSTOP | 19 | stop | Control-Z (via SIGTSTP) | no |
Two entries are special on purpose: SIGKILL and SIGSTOP cannot be caught, blocked, or ignored. That is a deliberate guarantee — the system must always retain a way to stop a runaway process. It also means "trap SIGKILL to clean up" is impossible, and any cleanup-on-exit design must accept that it can be skipped.
SIGPIPE deserves special mention because it surprises network and pipeline programmers. Writing to a pipe or socket whose reader has closed raises SIGPIPE, whose default action terminates your process — so a server can vanish silently when a client disconnects. The standard fix is to ignore SIGPIPE and handle the EPIPE error from write instead.
When a shell reports a signal death it usually encodes it as 128 + signal number, so a process killed by SIGKILL (9) shows $? as 137. That is a shell convention, not a kernel one; a program using waitpid should decode with WIFSIGNALED and WTERMSIG.
Examples
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/wait.h>
#include <signal.h>
#include <string.h>
/* Fork a child, have it end in the requested way, and decode the result. */
static void run_child(const char *what)
{
pid_t pid = fork();
if (pid < 0) { perror("fork"); return; }
if (pid == 0) {
if (strcmp(what, "success") == 0) _exit(EXIT_SUCCESS);
else if (strcmp(what, "failure") == 0) _exit(3);
else if (strcmp(what, "big") == 0) _exit(300); /* truncated to 8 bits */
else if (strcmp(what, "abort") == 0) abort(); /* SIGABRT */
else if (strcmp(what, "segv") == 0) {
raise(SIGSEGV); /* simulate a fault */
}
else if (strcmp(what, "kill") == 0) {
raise(SIGKILL); /* uncatchable */
}
_exit(0);
}
int status;
if (waitpid(pid, &status, 0) < 0) { perror("waitpid"); return; }
printf("%-8s -> ", what);
if (WIFEXITED(status)) {
/* NORMAL termination: an exit status, truncated to 8 bits. */
printf("exited, status %d\n", WEXITSTATUS(status));
} else if (WIFSIGNALED(status)) {
/* ABNORMAL termination: a SIGNAL, not a status. */
int sig = WTERMSIG(status);
printf("killed by signal %d (%s)%s\n", sig, strsignal(sig),
WCOREDUMP(status) ? ", core dumped" : "");
}
}
int main(void)
{
const char *cases[] = { "success", "failure", "big", "abort", "segv", "kill" };
for (size_t i = 0; i < sizeof cases / sizeof *cases; i++)
run_child(cases[i]);
/* 'big' shows the truncation: exit(300) becomes 300 & 0xFF == 44. */
printf("\n300 & 0xFF = %d -- only the low 8 bits survive\n", 300 & 0xFF);
return 0;
}
Sending signals, and the SIGPIPE trap:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <signal.h>
#include <sys/wait.h>
#include <errno.h>
#include <string.h>
int main(void)
{
/* --- sending a signal to another process --- */
pid_t pid = fork();
if (pid < 0) { perror("fork"); return 1; }
if (pid == 0) {
pause(); /* wait for any signal */
_exit(0);
}
usleep(100000); /* give the child time to reach pause() */
kill(pid, SIGTERM); /* the POLITE termination request */
int status;
waitpid(pid, &status, 0);
if (WIFSIGNALED(status))
printf("child terminated by %s\n", strsignal(WTERMSIG(status)));
/* --- kill(pid, 0): does this process exist and may I signal it? --- */
if (kill(pid, 0) < 0 && errno == ESRCH)
printf("pid %d no longer exists\n", (int)pid);
/* --- SIGPIPE: the trap that silently kills servers --- */
/* Default action for SIGPIPE is TERMINATE. Ignoring it converts the
event into a normal EPIPE error from write(), which we can handle. */
signal(SIGPIPE, SIG_IGN);
int fds[2];
if (pipe(fds) == 0) {
close(fds[0]); /* close the READ end */
ssize_t n = write(fds[1], "data", 4); /* nobody is reading */
if (n < 0 && errno == EPIPE)
printf("write failed with EPIPE -- process survived because "
"SIGPIPE was ignored\n");
close(fds[1]);
}
/* Uncatchable signals: setting a handler for SIGKILL simply fails. */
if (signal(SIGKILL, SIG_IGN) == SIG_ERR)
printf("cannot catch SIGKILL: %s\n", strerror(errno));
return 0;
}
# The shell reports a signal death as 128 + signal number.
$ ./crash_program ; echo "status=$?"
Segmentation fault (core dumped)
status=139 # 128 + 11 (SIGSEGV)
$ sleep 100 & kill -9 %1 ; wait ; echo "status=$?"
status=137 # 128 + 9 (SIGKILL)
$ (exit 3) ; echo "status=$?"
status=3 # a normal exit status
# List every signal your system defines:
$ kill -l
Key Takeaways
- Normal termination carries an exit status truncated to its low 8 bits; abnormal termination carries a signal number instead, and the two are different events.
- Decode a wait status with
WIFEXITED/WEXITSTATUSandWIFSIGNALED/WTERMSIG— the raw integer is packed and meaningless on its own. SIGKILLandSIGSTOPcannot be caught, blocked, or ignored, so no cleanup handler can be guaranteed to run.SIGPIPEterminates the process by default when writing to a closed pipe or socket; ignore it and handleEPIPEfromwriteinstead.- Shells report a signal death as
128 + signum, andkill(pid, 0)tests for a process's existence without sending anything.
🧪 Practice
- Write a program that exits with a status taken from
argv[1], and confirm from the shell that values above 255 wrap.- Write a parent that forks a child which loops forever, sends it
SIGTERMafter one second, and reports exactly how it died.- Build a two-process pipeline where the reader exits early, and demonstrate the writer dying from
SIGPIPEand then surviving once it is ignored.- Interview-style: "Why can't a process catch
SIGKILL?" Hint: consider what an administrator needs to be able to guarantee about any process on the system.
signal and sigaction Basics
Theory
Catching a signal means installing a handler: a function the kernel calls, asynchronously, when the signal arrives. Your normal control flow is suspended wherever it happened to be, the handler runs, and then execution resumes. That "wherever it happened to be" is the whole difficulty.
There are two APIs, and the choice between them is not a matter of taste.
signal() is in ISO C, is two lines to use, and has historically inconsistent semantics. On old System V systems the handler reset itself to the default after firing once, creating a race; on BSD it stayed installed and restarted interrupted system calls. Modern glibc follows BSD, but the standard permits either. Use it only for SIG_IGN and SIG_DFL, where the ambiguity does not matter.
sigaction() is POSIX, specifies everything explicitly, and is what production code uses. It lets you state which other signals to block during the handler, whether to restart interrupted system calls, and whether you want extended information about the signal's cause.
The rules for what a handler may do are far stricter than most people assume, because a handler can interrupt your program between any two machine instructions — including in the middle of malloc updating its free lists, or printf updating the stdio buffer. If the handler then calls the same function, it operates on half-updated state.
A function safe to call in this situation is async-signal-safe. POSIX publishes the list, and it is short. write, _exit, kill, and signal are on it. printf, malloc, free, and almost everything else are not. A handler that calls printf works nearly always and deadlocks occasionally, which is the worst possible failure mode.
Similarly, the only variable type a handler may safely share with the main program is volatile sig_atomic_t. volatile stops the compiler from caching it in a register across the handler's invisible interruption; sig_atomic_t guarantees reads and writes happen in one indivisible step.
Those two constraints produce the standard design, which is what nearly all real programs do:
The handler sets a flag and returns. The main loop checks the flag and does the real work.
Everything unsafe — logging, freeing, closing files, printing a summary — happens in the main loop, where all functions are legal again.
Two more mechanics matter. A handler receives the signal number as its only argument, so one function can serve several signals. And signals do not queue: if the same signal arrives three times while blocked, it is delivered once. Never use signal counts to count events.
Finally, installing a handler affects blocking system calls. Without SA_RESTART, an interrupted read returns -1 with EINTR and you must retry; with SA_RESTART, the kernel restarts it for you. Choose deliberately — a program that wants Control-C to interrupt a long read needs EINTR, while one that merely wants to note the signal wants SA_RESTART.
Examples
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <signal.h>
#include <string.h>
#include <errno.h>
/* The ONLY safe way to share state with a handler:
volatile -> the compiler must reload it, not cache it in a register
sig_atomic_t -> reads and writes are indivisible */
static volatile sig_atomic_t got_sigint = 0;
static volatile sig_atomic_t got_sigterm = 0;
/* A handler may interrupt the program between ANY two instructions, so it
may only call async-signal-safe functions. printf, malloc, and free are
NOT safe. Setting a flag and returning is the safe universal pattern. */
static void flag_handler(int signum)
{
switch (signum) {
case SIGINT: got_sigint = 1; break;
case SIGTERM: got_sigterm = 1; break;
default: break;
}
/* write() IS async-signal-safe, unlike printf, so a minimal note is legal. */
const char msg[] = "[signal received]\n";
ssize_t ignored = write(STDERR_FILENO, msg, sizeof msg - 1);
(void)ignored; /* silence the unused-result warning */
}
static int install(int signum, void (*fn)(int), int restart)
{
struct sigaction sa;
memset(&sa, 0, sizeof sa); /* zero EVERY field first */
sa.sa_handler = fn;
sigemptyset(&sa.sa_mask); /* which signals to block in the handler */
sigaddset(&sa.sa_mask, SIGINT); /* block these two so the handler */
sigaddset(&sa.sa_mask, SIGTERM); /* cannot interrupt itself */
/* SA_RESTART: restart interrupted system calls automatically.
Without it, a blocking read() returns -1 with errno == EINTR. */
sa.sa_flags = restart ? SA_RESTART : 0;
return sigaction(signum, &sa, NULL); /* POSIX: explicit, unambiguous */
}
int main(void)
{
if (install(SIGINT, flag_handler, 1) < 0 ||
install(SIGTERM, flag_handler, 1) < 0) {
perror("sigaction");
return EXIT_FAILURE;
}
/* Ignore SIGPIPE outright -- signal() is fine for SIG_IGN/SIG_DFL. */
signal(SIGPIPE, SIG_IGN);
printf("pid %d running; press Ctrl-C or send SIGTERM\n", (int)getpid());
/* THE STANDARD SHUTDOWN PATTERN: the handler only sets a flag; all the
real work -- which is not async-signal-safe -- happens out here. */
for (int tick = 0; tick < 3; tick++) {
if (got_sigint) {
printf("SIGINT noticed; cleaning up safely in the main loop\n");
got_sigint = 0; /* signals do NOT queue: reset it */
}
if (got_sigterm) {
printf("SIGTERM noticed; shutting down\n");
break;
}
usleep(100000);
}
printf("clean exit\n");
return EXIT_SUCCESS;
}
Blocking signals around a critical section, and sigaction's extended form:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <signal.h>
#include <string.h>
static volatile sig_atomic_t deliveries = 0;
/* SA_SIGINFO gives a three-argument handler with details about the cause. */
static void info_handler(int signum, siginfo_t *info, void *ucontext)
{
(void)ucontext;
deliveries++;
/* info->si_pid is the SENDING process; si_code says why it was sent.
Only note them here -- inspecting them fully belongs in the main loop. */
(void)signum; (void)info;
}
int main(void)
{
struct sigaction sa;
memset(&sa, 0, sizeof sa);
sa.sa_sigaction = info_handler; /* note: sa_sigaction, not sa_handler */
sa.sa_flags = SA_SIGINFO | SA_RESTART;
sigemptyset(&sa.sa_mask);
if (sigaction(SIGUSR1, &sa, NULL) < 0) { perror("sigaction"); return 1; }
/* --- blocking signals around a critical section --- */
sigset_t block, saved;
sigemptyset(&block);
sigaddset(&block, SIGUSR1);
/* Defer SIGUSR1 while we touch state a handler must not see half-updated. */
if (sigprocmask(SIG_BLOCK, &block, &saved) < 0) { perror("sigprocmask"); return 1; }
raise(SIGUSR1); /* delivered LATER, not now */
raise(SIGUSR1); /* signals do NOT queue: still one */
printf("inside critical section, deliveries=%d\n", (int)deliveries);
/* Restore the previous mask: the pending signal is delivered here. */
sigprocmask(SIG_SETMASK, &saved, NULL);
printf("after unblocking, deliveries=%d\n", (int)deliveries);
printf("two raises produced %d delivery -- signals are not counted\n",
(int)deliveries);
return 0;
}
WHY A HANDLER MUST NOT CALL printf OR malloc
main: printf("hello")
|
+--> stdio locks the buffer
+--> starts writing into it <-- SIGNAL ARRIVES HERE
|
handler: v
printf("caught!") --> tries to lock the SAME buffer
which this thread already holds
-> deadlock, or corrupted output
Same story for malloc: the handler enters a free list that is
half-updated, and corrupts the heap.
THE SAFE PATTERN
handler: flag = 1; return; (async-signal-safe only)
main loop: if (flag) { do the real work here }
signal() | sigaction() | |
|---|---|---|
| Standard | ISO C | POSIX |
| Semantics across systems | ambiguous | fully specified |
| Handler stays installed | not guaranteed | yes |
| Block other signals during handler | no control | sa_mask |
| Restart interrupted syscalls | unspecified | SA_RESTART, explicit |
| Extended signal information | no | SA_SIGINFO |
| Use for | SIG_IGN and SIG_DFL only | everything else |
Key Takeaways
- Prefer
sigactionoversignal:signal's semantics vary between systems, whilesigactionspecifies handler persistence, masking, and restart behavior explicitly. - A handler may interrupt the program between any two instructions, so it may call only async-signal-safe functions —
writeand_exitare safe,printf,malloc, andfreeare not. - Share state with a handler only through
volatile sig_atomic_t, and use the flag-and-return pattern so all unsafe work happens in the main loop. - Signals do not queue: several deliveries of the same signal while it is blocked collapse into one, so never count events with signal counts.
SA_RESTARTdecides whether an interrupted blocking call is restarted or fails withEINTR— choose according to whether the signal should interrupt long operations.
🧪 Practice
- Install a
SIGINThandler that sets a flag, and write a main loop that prints a summary and exits cleanly on the second Control-C.- Write a handler that calls
printfin a tight loop while the main program also prints, run it under load, and look for interleaved or corrupted output.- Install a
SIGALRMhandler with and withoutSA_RESTART, callalarm(1)before a blockingreadfrom the terminal, and compare the two behaviors.- Interview-style: "Why is calling
mallocinside a signal handler unsafe?" Hint: consider what happens if the signal arrives while the main program is already insidemalloc.
<a id="112-files-and-io-at-the-system-level"></a>
11.2 Files and I/O at the System Level
Beneath FILE * and its buffer sits a smaller, sharper interface: integer file descriptors and unbuffered system calls that do exactly what you ask, once. This section covers that layer — what a descriptor really is, the four calls that drive it, what happens when I/O would block, how to wait on many sources at once, and how to skip the copying entirely with mmap.
File Descriptors
Theory
A file descriptor is a small non-negative integer that names an open file — or a pipe, socket, terminal, or device. It is deliberately not a pointer or a handle object: it is an index into a per-process table the kernel maintains on your behalf.
That indirection is the whole design. The number 3 means nothing by itself; it means "entry 3 of this process's open-file table", and that entry points to a kernel structure holding the current offset, the access mode, and a reference to the underlying file. Two processes can both hold descriptor 3 referring to completely different files.
Three descriptors are open before your program starts, by convention rather than magic:
| Descriptor | Constant | Purpose |
|---|---|---|
| 0 | STDIN_FILENO | standard input |
| 1 | STDOUT_FILENO | standard output |
| 2 | STDERR_FILENO | standard error |
New descriptors are always assigned the lowest available number. That rule sounds like trivia and is actually the mechanism behind shell redirection: close descriptor 1, then open a file, and the file becomes descriptor 1 because 1 was the lowest free slot. Everything that writes to stdout now writes to the file, without knowing anything changed.
The modern way to do that deliberately is dup2(oldfd, newfd), which makes newfd a copy of oldfd, closing newfd first if it was open. It is atomic, unlike close-then-open, so a signal cannot arrive at the wrong moment.
The crucial subtlety is what "a copy" means. Descriptors created by dup/dup2 or inherited across fork share the same open-file description, which means they share the file offset. Reading through one advances the other. A separate open of the same file creates an independent description with its own offset. Confusing the two produces file corruption that looks random.
TWO PROCESSES, TWO KINDS OF SHARING
process A kernel open-file table inode / file
fd table
[0] --------+
[1] ----+ | +-------------------+
[2] +-> | | offset: 1024 | +-----------+
[3] --------+------> | mode: O_RDWR |--------> | data.txt |
+-------------------+ +-----------+
process B ^
fd table +-------------------+ |
[3] ---------------> | offset: 0 |---------------+
| mode: O_RDONLY |
+-------------------+
A's fd 3 and B's fd 3 have SEPARATE offsets (separate open() calls).
A's fd 0 and fd 3 above share ONE offset (dup2) -- reading through
either advances both.
Descriptors are a finite resource. Each process has a limit (ulimit -n, often 1024 or a few thousand), and a descriptor leak — opening without closing in a loop — eventually causes open to fail with EMFILE. Because the failure appears far from the leak, this is a genuinely annoying bug; ls -l /proc/self/fd and lsof are how you find it.
Two flags matter for real programs. FD_CLOEXEC marks a descriptor to be closed automatically on exec, which prevents a child program from inheriting descriptors it has no business holding — a real security concern when the child is less trusted. Open with O_CLOEXEC to set it atomically rather than with a separate fcntl, which would leave a window in a threaded program.
Finally, the relationship with FILE *: fileno(fp) extracts the descriptor from a stream, and fdopen(fd, mode) wraps a descriptor in a stream. Mixing buffered and raw access to the same file is a reliable way to corrupt your output, so pick one layer per descriptor.
Examples
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h> /* read, write, close, dup, dup2, STDOUT_FILENO */
#include <fcntl.h> /* open, O_* flags */
#include <string.h>
#include <errno.h>
int main(void)
{
/* The three standard descriptors are open before main runs. */
printf("stdin=%d stdout=%d stderr=%d\n",
STDIN_FILENO, STDOUT_FILENO, STDERR_FILENO);
/* A new descriptor gets the LOWEST FREE number -- usually 3. */
int fd = open("/tmp/fd_demo.txt", O_RDWR | O_CREAT | O_TRUNC, 0644);
if (fd < 0) { perror("open"); return EXIT_FAILURE; }
printf("opened /tmp/fd_demo.txt as fd %d\n", fd);
write(fd, "hello descriptors\n", 18);
/* --- dup2: what shell redirection actually does ------------------- */
int saved_stdout = dup(STDOUT_FILENO); /* keep a way back */
if (saved_stdout < 0) { perror("dup"); return EXIT_FAILURE; }
if (dup2(fd, STDOUT_FILENO) < 0) { /* fd 1 now refers to the file */
perror("dup2"); return EXIT_FAILURE;
}
/* printf still writes to "stdout" -- it has no idea it is now a file. */
printf("this line lands in the FILE, not the terminal\n");
fflush(stdout); /* flush before restoring */
dup2(saved_stdout, STDOUT_FILENO); /* put the terminal back */
close(saved_stdout);
printf("stdout restored to the terminal\n");
/* --- shared vs independent offsets -------------------------------- */
int dup_fd = dup(fd); /* SHARES the offset with fd */
int new_fd = open("/tmp/fd_demo.txt", O_RDONLY); /* INDEPENDENT offset */
lseek(fd, 0, SEEK_SET);
char a[6] = {0}, b[6] = {0};
read(fd, a, 5); /* fd offset: 0 -> 5 */
read(dup_fd, b, 5); /* SHARED: continues from 5 */
printf("fd read \"%s\", dup_fd read \"%s\" (shared offset)\n", a, b);
char c[6] = {0};
read(new_fd, c, 5); /* independent: starts at 0 again */
printf("new_fd read \"%s\" (independent offset)\n", c);
close(dup_fd); close(new_fd); close(fd);
/* --- O_CLOEXEC: do not leak descriptors into exec'd children ------ */
int secret = open("/tmp/fd_demo.txt", O_RDONLY | O_CLOEXEC);
if (secret >= 0) {
int flags = fcntl(secret, F_GETFD);
printf("FD_CLOEXEC set: %s\n", (flags & FD_CLOEXEC) ? "yes" : "no");
close(secret);
}
remove("/tmp/fd_demo.txt");
return EXIT_SUCCESS;
}
Descriptor exhaustion, and finding a leak:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <string.h>
int main(void)
{
int count = 0;
/* Leak descriptors deliberately: open without ever closing. */
for (;;) {
int fd = open("/dev/null", O_RDONLY);
if (fd < 0) {
/* EMFILE = this PROCESS hit its limit (ulimit -n)
ENFILE = the whole SYSTEM ran out */
printf("open failed after %d descriptors: %s\n",
count, strerror(errno));
break;
}
count++;
if (count > 100000) { printf("stopping early\n"); break; }
}
/* The failure appears wherever the next open() happens, which is
usually far from the code that leaked. Inspect the real list with:
ls -l /proc/self/fd (Linux)
lsof -p <pid> (portable-ish) */
printf("check 'ls -l /proc/%d/fd' to see what is open\n", (int)getpid());
return 0;
}
File descriptor (int) | Stream (FILE *) | |
|---|---|---|
| Layer | kernel, system call | C standard library |
| Buffered | no | yes |
| Portability | POSIX | ISO C, everywhere |
Works with select/poll | yes | no (extract with fileno) |
| Handles sockets and pipes | yes | via fdopen |
| Error reporting | -1 plus errno | NULL, EOF, ferror |
| Convert | fdopen(fd, mode) | fileno(fp) |
Key Takeaways
- A file descriptor is an index into a per-process table, not a pointer; the same number in two processes usually refers to different files.
- New descriptors always take the lowest free number, which is the mechanism behind shell redirection and the reason
dup2works as it does. - Descriptors from
dup/dup2or inherited acrossforkshare one file offset; separateopencalls on the same file have independent offsets. - Descriptors are finite — leaking them causes
opento fail withEMFILEfar from the leaking code; inspect/proc/self/fdor uselsofto find it. - Use
O_CLOEXECso descriptors are not inherited by exec'd children, and avoid mixing raw descriptor I/O with buffered stream I/O on the same file.
🧪 Practice
- Write a program that redirects its own
stdoutto a file withdup2, writes several lines, then restores the terminal and confirms both destinations received the right output.- Demonstrate the shared-offset behavior:
dupa descriptor, read through both, and show that the offset advanced once per byte, not twice.- Write a loop that opens files until it fails, print the limit you reached, and compare it against
ulimit -n.- Interview-style: "How does the shell implement
command > file.txt?" Hint: think about which descriptor number the child must end up with, and what the lowest-free-number rule gives you.
open, read, write, close
Theory
These four system calls are the entire raw I/O interface. Everything else — FILE *, sockets, pipes — is built on them or resembles them.
open(path, flags, mode) returns a descriptor or -1. The flags argument combines one required access mode with any number of optional flags:
| Flag | Meaning |
|---|---|
O_RDONLY / O_WRONLY / O_RDWR | access mode — exactly one is required |
O_CREAT | create if missing (then mode matters) |
O_EXCL | with O_CREAT, fail if it already exists |
O_TRUNC | truncate to zero length |
O_APPEND | every write goes to the end, atomically |
O_NONBLOCK | do not block (see the next topic) |
O_CLOEXEC | close automatically on exec |
The third argument is the permission mode, used only when creating. It is modified by the process's umask, so requesting 0666 typically produces 0644. Passing 0 by accident creates a file nobody can open, including you.
O_CREAT | O_EXCL together are the atomic "create it, but only if it does not exist" primitive — the correct way to create a lock file, because checking with stat and then creating leaves a window where another process can win the race.
read(fd, buf, count) returns the number of bytes actually read, 0 at end of file, or -1 on error. The critical fact, which the next topic explores in depth: a short read is normal, not an error. Asking for 4096 bytes from a pipe, socket, or terminal commonly returns far fewer. Code that assumes read fills the buffer is broken on every input source except a regular file.
write(fd, buf, count) returns the number of bytes written, which can also be less than requested. For regular files it usually writes everything; for pipes and sockets it may not.
close(fd) releases the descriptor. It can fail — a delayed write error may surface here — so a program that must know its data was stored should check it. And note that close does not guarantee the data reached the disk; it only reached the kernel's cache. fsync(fd) is what forces it to stable storage, at considerable cost.
Two properties of the descriptor layer distinguish it from stdio:
No buffering. Every read and write is a system call. This gives exact control — you know precisely what crossed the boundary and when — and it is why a byte-at-a-time loop through read is disastrously slow.
Exact semantics. O_APPEND guarantees that the seek-to-end and the write are one atomic operation, which is what makes concurrent appends to a log file safe. Buffered stdio cannot promise that.
Because short transfers are normal, the essential idiom is the full-write loop: keep writing until everything is transferred, retrying on EINTR. Getting this right once, in a helper function, and using it everywhere is the difference between code that works and code that works until the load increases.
Examples
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <string.h>
/* --- the two helpers every program using raw I/O needs ---------------- */
/* Write ALL n bytes, handling short writes and EINTR.
Returns 0 on success, -1 on a real error. */
static int write_all(int fd, const void *buf, size_t n)
{
const char *p = buf;
while (n > 0) {
ssize_t written = write(fd, p, n);
if (written < 0) {
if (errno == EINTR) continue; /* interrupted: retry, not fail */
return -1; /* a genuine error */
}
p += written; /* advance by what ACTUALLY went */
n -= (size_t)written; /* which may be less than asked */
}
return 0;
}
/* Read up to n bytes, stopping at EOF. Returns the count, or -1 on error. */
static ssize_t read_all(int fd, void *buf, size_t n)
{
char *p = buf;
size_t total = 0;
while (total < n) {
ssize_t got = read(fd, p + total, n - total);
if (got < 0) {
if (errno == EINTR) continue;
return -1;
}
if (got == 0) break; /* 0 means END OF FILE */
total += (size_t)got;
}
return (ssize_t)total;
}
int main(void)
{
/* --- create: O_CREAT|O_EXCL is the atomic "only if absent" ---------- */
const char *path = "/tmp/raw_io_demo.txt";
remove(path);
/* The mode 0644 is masked by the process umask, so the result is
typically 0644 & ~umask -- often 0644 exactly. */
int fd = open(path, O_WRONLY | O_CREAT | O_EXCL, 0644);
if (fd < 0) { perror("open O_EXCL"); return EXIT_FAILURE; }
const char *msg = "line one\nline two\nline three\n";
if (write_all(fd, msg, strlen(msg)) < 0) { perror("write"); close(fd); return 1; }
/* close() can report a DELAYED write error -- check it when the data
matters. Note it does NOT mean the bytes reached the disk. */
if (close(fd) < 0) { perror("close"); return EXIT_FAILURE; }
/* A second O_EXCL create must fail: that is the point of the flag. */
int again = open(path, O_WRONLY | O_CREAT | O_EXCL, 0644);
if (again < 0 && errno == EEXIST)
printf("O_EXCL correctly refused to clobber the existing file\n");
else close(again);
/* --- read it back in chunks --------------------------------------- */
fd = open(path, O_RDONLY);
if (fd < 0) { perror("open"); return EXIT_FAILURE; }
char buf[16];
ssize_t n;
int chunk = 0;
/* Note the loop condition: > 0 continues, 0 is EOF, < 0 is an error. */
while ((n = read(fd, buf, sizeof buf)) > 0)
printf("chunk %d: %zd bytes \"%.*s\"\n", ++chunk, n, (int)n, buf);
if (n < 0) perror("read");
close(fd);
/* read_all instead: fill the buffer completely, stopping only at EOF. */
fd = open(path, O_RDONLY);
if (fd >= 0) {
char whole[64];
ssize_t got = read_all(fd, whole, sizeof whole - 1);
if (got >= 0) { whole[got] = '\0'; printf("read_all got %zd bytes\n", got); }
close(fd);
}
/* --- O_APPEND: atomic seek-to-end + write, safe with many writers -- */
fd = open(path, O_WRONLY | O_APPEND);
if (fd >= 0) {
write_all(fd, "appended\n", 9); /* always lands at the end,
regardless of any lseek */
close(fd);
}
/* --- fsync: force the data to STABLE STORAGE ---------------------- */
fd = open(path, O_WRONLY | O_APPEND);
if (fd >= 0) {
write_all(fd, "durable\n", 8);
if (fsync(fd) < 0) perror("fsync"); /* expensive, but survives a
power loss; close() does not */
close(fd);
}
remove(path);
return EXIT_SUCCESS;
}
A correct file copy — the canonical use of these four calls:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <string.h>
#define BUFSZ 65536 /* large enough that syscall overhead is amortized */
static int write_all(int fd, const void *buf, size_t n)
{
const char *p = buf;
while (n > 0) {
ssize_t w = write(fd, p, n);
if (w < 0) { if (errno == EINTR) continue; return -1; }
p += w; n -= (size_t)w;
}
return 0;
}
static int copy_file(const char *src, const char *dst)
{
int in = open(src, O_RDONLY);
if (in < 0) return -1;
/* 0666 is masked by umask; O_TRUNC discards any existing content. */
int out = open(dst, O_WRONLY | O_CREAT | O_TRUNC, 0666);
if (out < 0) { int e = errno; close(in); errno = e; return -1; }
char *buf = malloc(BUFSZ);
int rc = -1;
ssize_t n;
if (!buf) goto done;
while ((n = read(in, buf, BUFSZ)) > 0) /* 0 ends the loop at EOF */
if (write_all(out, buf, (size_t)n) < 0) goto done;
if (n < 0) goto done; /* the read itself failed */
rc = 0;
done:
free(buf);
close(in);
/* close() on the OUTPUT can surface a deferred write error, so its
result decides success when everything else went well. */
if (close(out) < 0) rc = -1;
return rc;
}
int main(void)
{
FILE *f = fopen("/tmp/copy_src.txt", "w");
if (f) { fputs("some content to copy\n", f); fclose(f); }
if (copy_file("/tmp/copy_src.txt", "/tmp/copy_dst.txt") < 0)
perror("copy_file");
else
printf("copied successfully\n");
remove("/tmp/copy_src.txt");
remove("/tmp/copy_dst.txt");
return 0;
}
Key Takeaways
openreturns the lowest free descriptor or-1; the permission argument applies only withO_CREATand is filtered by the process umask.O_CREAT | O_EXCLis the atomic create-if-absent primitive — checking withstatfirst leaves a race window.- A short
readorwriteis normal, not an error, especially on pipes, sockets, and terminals;readreturning0means end of file. - Wrap raw I/O in
write_all/read_allhelpers that loop on partial transfers and retry onEINTR, and use them everywhere. closecan report a deferred write error and must be checked when data matters, but onlyfsyncforces bytes to stable storage.
🧪 Practice
- Implement
catusing onlyopen,read,write, andclose, handling short reads andEINTRcorrectly.- Compare copy throughput with buffer sizes of 1, 512, 4096, and 65536 bytes on a 100 MB file, and explain the curve.
- Use
O_CREAT | O_EXCLto build a lock file, run two instances concurrently, and confirm exactly one acquires it.- Interview-style: "Your
writereturns 300 when you asked it to write 1000 bytes. Is that an error?" Hint: consider what the kernel does when a pipe's buffer is nearly full.
Blocking vs Non-Blocking I/O
Theory
When you call read on a socket with no data available, what should happen? There are two defensible answers, and POSIX lets you choose.
Blocking is the default. The kernel suspends your process until data arrives, then returns it. The code reads naturally — read gives you data, full stop — and while you are blocked the process consumes no CPU. This is the right model for a program doing one thing at a time.
The problem appears when you must handle several sources. Blocked on a read from client A, you cannot notice that client B sent something, that a timer expired, or that the user pressed a key. The process is parked.
Non-blocking mode changes the contract: if the operation cannot proceed immediately, it returns -1 immediately with errno set to EAGAIN (or EWOULDBLOCK, the same value on Linux). Nothing was read, nothing failed — the answer is simply "not right now, try later".
That single change is what makes event-driven programs possible. Combined with select or poll, it lets one thread service thousands of connections: ask which descriptors are ready, service exactly those, never block on any of them.
Enable it either at open time with O_NONBLOCK, or afterwards with fcntl. The fcntl form must read the existing flags and add to them — overwriting the flag word instead of OR-ing into it is a common bug that silently clears the access mode.
int flags = fcntl(fd, F_GETFL, 0); /* fetch first */
fcntl(fd, F_SETFL, flags | O_NONBLOCK); /* then add */
Three points are essential to using this correctly.
EAGAIN is not an error. Treating it as one — logging it, closing the connection — turns a normal condition into a bug. It means "retry when ready", and the whole point of select/poll is to know when that is, rather than spinning.
Non-blocking mode is a property of the open-file description, not the descriptor. Setting it affects every descriptor sharing that description, including ones inherited across fork — which has surprised many programmers when a child unexpectedly saw EAGAIN.
Regular files are always "ready". O_NONBLOCK has essentially no effect on them: a read from a local file blocks on disk I/O regardless. Non-blocking mode is meaningful for pipes, sockets, terminals, and other things that can genuinely have nothing available. For files, the answer is asynchronous I/O or a thread pool.
Busy-waiting — calling read in a tight loop until it succeeds — technically works and is almost always wrong: it burns an entire CPU core to avoid waiting. Non-blocking I/O should be paired with a readiness mechanism, which is the next topic.
Examples
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <string.h>
/* Add O_NONBLOCK without clobbering the existing flags. */
static int set_nonblocking(int fd)
{
int flags = fcntl(fd, F_GETFL, 0); /* READ the current flags */
if (flags < 0) return -1;
/* OR the new flag in. Writing fcntl(fd, F_SETFL, O_NONBLOCK) would
discard O_APPEND, the access mode, and everything else. */
return fcntl(fd, F_SETFL, flags | O_NONBLOCK);
}
int main(void)
{
int fds[2];
if (pipe(fds) < 0) { perror("pipe"); return EXIT_FAILURE; }
int rd = fds[0], wr = fds[1];
/* --- blocking behavior (the default) ------------------------------ */
printf("pipe is empty; a blocking read here would hang forever.\n");
/* --- switch the read end to non-blocking -------------------------- */
if (set_nonblocking(rd) < 0) { perror("fcntl"); return EXIT_FAILURE; }
char buf[64];
ssize_t n = read(rd, buf, sizeof buf);
if (n < 0 && (errno == EAGAIN || errno == EWOULDBLOCK)) {
/* This is the NORMAL, expected outcome -- not an error to report. */
printf("read -> EAGAIN: nothing available, try again later\n");
} else {
printf("unexpected: n=%zd errno=%d\n", n, errno);
}
/* Now put data in, and the same call succeeds immediately. */
write(wr, "now there is data", 17);
n = read(rd, buf, sizeof buf);
if (n > 0) printf("read -> %zd bytes: \"%.*s\"\n", n, (int)n, buf);
/* --- a non-blocking WRITE can also transfer only part of the data -- */
set_nonblocking(wr);
size_t total = 0;
char big[4096];
memset(big, 'x', sizeof big);
/* Fill the pipe until the kernel buffer is full. */
for (;;) {
ssize_t w = write(wr, big, sizeof big);
if (w < 0) {
if (errno == EAGAIN) {
printf("pipe buffer full after %zu bytes -> EAGAIN\n", total);
break; /* again: normal, not an error */
}
if (errno == EINTR) continue;
perror("write");
break;
}
total += (size_t)w; /* w may be a PARTIAL write */
if (total > 1u << 20) break; /* safety stop */
}
close(rd); close(wr);
return EXIT_SUCCESS;
}
The busy-wait trap, and why readiness notification exists:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <time.h>
#include <sys/select.h>
int main(void)
{
int fds[2];
if (pipe(fds) < 0) return 1;
int flags = fcntl(fds[0], F_GETFL, 0);
fcntl(fds[0], F_SETFL, flags | O_NONBLOCK);
/* A child writes after a short delay, so the parent must wait somehow. */
pid_t pid = fork();
if (pid == 0) {
close(fds[0]);
usleep(200000); /* 200 ms */
write(fds[1], "ready", 5);
close(fds[1]);
_exit(0);
}
close(fds[1]);
/* --- WRONG: busy-waiting burns a full CPU core doing nothing ------ */
long spins = 0;
char buf[16];
struct timespec t0, t1;
clock_gettime(CLOCK_MONOTONIC, &t0);
for (;;) {
ssize_t n = read(fds[0], buf, sizeof buf);
if (n > 0) break;
if (n < 0 && errno != EAGAIN) break;
spins++; /* millions of useless syscalls */
}
clock_gettime(CLOCK_MONOTONIC, &t1);
printf("busy-wait: %ld failed read() calls in %.3f s of 100%% CPU\n",
spins, (t1.tv_sec - t0.tv_sec) + (t1.tv_nsec - t0.tv_nsec) / 1e9);
printf("The fix is to ask the kernel to TELL you when the descriptor is\n"
"ready -- select, poll, or epoll -- and sleep until then.\n");
close(fds[0]);
return 0;
}
| Blocking | Non-blocking | |
|---|---|---|
| No data available | sleeps until there is | returns -1, errno == EAGAIN |
| CPU while waiting | none — the process is suspended | none if paired with select/poll |
| Code shape | simple, sequential | event loop with state per connection |
| Handling many sources | needs a thread or process each | one thread can serve thousands |
| Partial transfers | still possible | very likely |
| Meaningful for regular files | n/a | no — files are always "ready" |
Key Takeaways
- Blocking mode suspends the process until the operation can proceed; non-blocking mode returns immediately with
EAGAINwhen it cannot. EAGAIN/EWOULDBLOCKmeans "not right now", not "something went wrong" — treating it as an error is a common and damaging mistake.- Set the flag with
fcntl(F_GETFL)thenF_SETFLwith the flags OR-ed in; overwriting the flag word discards the access mode and other settings. - Non-blocking is a property of the open-file description, so it is shared by duplicated and inherited descriptors.
O_NONBLOCKis meaningless for regular files, and non-blocking I/O without a readiness mechanism degenerates into busy-waiting that burns a whole core.
🧪 Practice
- Create a pipe, set the read end non-blocking, and demonstrate
EAGAINbefore data is written and success afterwards.- Fill a pipe's buffer with non-blocking writes until you get
EAGAIN, and report the kernel pipe capacity you discovered.- Set
stdinnon-blocking and write a loop that prints a tick every 100 ms while still reading input when it appears, usingusleepbetween polls. Then explain whyselectis better.- Interview-style: "Your non-blocking server uses 100% CPU with no traffic. What is wrong?" Hint: the loop never sleeps because nothing tells it when to wake.
I/O Multiplexing with select and poll
Theory
Non-blocking I/O tells you a descriptor is not ready. It does not tell you when it becomes ready, and polling in a loop wastes a core. I/O multiplexing closes that gap: hand the kernel a set of descriptors, and it puts your process to sleep until at least one of them can be used without blocking.
This is what lets a single-threaded server handle ten thousand connections. The alternative — a thread or process per connection — costs a stack and a scheduler entry each, and collapses under load. Multiplexing costs one descriptor per connection plus a little state.
There are two portable interfaces, and one important non-portable one.
select is the oldest and most widely available:
int select(int nfds, fd_set *readfds, fd_set *writefds,
fd_set *exceptfds, struct timeval *timeout);
You build up to three fd_set bitmaps, pass the highest descriptor plus one as nfds, and select returns with the sets modified in place to contain only the ready descriptors. Its limitations are real:
FD_SETSIZEcaps the descriptor number, typically at 1024. A descriptor numbered 1024 or higher cannot be represented, and using one is undefined behavior — often silent stack corruption.- The sets are destroyed on each call, so you must rebuild them every iteration.
- The timeout struct may be modified, so it must be reset too.
poll fixes all three:
int poll(struct pollfd *fds, nfds_t nfds, int timeout);
You pass an array of struct pollfd, each with a descriptor, an events mask of what you care about, and a revents field the kernel fills in. There is no descriptor-number limit, the input is not destroyed (only revents changes), and the timeout is a plain millisecond integer. Prefer poll for new code.
Both share a scalability limit: they are O(n) in the number of descriptors watched, because the kernel must examine every one on every call and you must scan the results. At a few hundred descriptors this is irrelevant; at tens of thousands it dominates. That is what epoll (Linux), kqueue (BSD and macOS), and IOCP (Windows) exist to solve — they keep the interest set in the kernel across calls and report only what changed, giving O(ready) instead of O(watched). They are not portable, which is why portable code uses poll and high-performance code uses a library that wraps whichever is native.
Two practical notes. Readiness is a hint, not a promise of quantity: poll saying a socket is readable means at least one byte is available, not that your whole buffer will fill — so always pair multiplexing with non-blocking descriptors and short-transfer handling. And a listening socket becomes "readable" when a connection is pending, which is how a server folds accept into the same loop as everything else.
The POLLHUP and POLLERR bits in revents are set by the kernel even if you did not request them, and ignoring them is how event loops end up spinning on a closed connection forever.
Examples
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <poll.h>
#include <fcntl.h>
#include <errno.h>
#include <string.h>
#include <sys/wait.h> /* wait */
#define MAX_FDS 16
static int set_nonblocking(int fd)
{
int f = fcntl(fd, F_GETFL, 0);
return f < 0 ? -1 : fcntl(fd, F_SETFL, f | O_NONBLOCK);
}
int main(void)
{
/* Two pipes stand in for two independent clients. */
int a[2], b[2];
if (pipe(a) < 0 || pipe(b) < 0) { perror("pipe"); return EXIT_FAILURE; }
set_nonblocking(a[0]);
set_nonblocking(b[0]);
/* Children write at different times, so neither source can be waited
on first without risking a delay on the other. */
pid_t p1 = fork();
if (p1 == 0) {
close(a[0]); close(b[0]); close(b[1]);
usleep(150000); write(a[1], "from pipe A", 11);
close(a[1]); _exit(0);
}
pid_t p2 = fork();
if (p2 == 0) {
close(a[0]); close(a[1]); close(b[0]);
usleep(50000); write(b[1], "from pipe B", 11);
close(b[1]); _exit(0);
}
close(a[1]); close(b[1]);
/* Build the watch list once: poll does NOT destroy it. */
struct pollfd fds[MAX_FDS];
memset(fds, 0, sizeof fds);
fds[0].fd = a[0]; fds[0].events = POLLIN; /* POLLIN: readable */
fds[1].fd = b[0]; fds[1].events = POLLIN;
nfds_t nfds = 2;
int open_count = 2;
while (open_count > 0) {
/* Sleep until something is ready, or 2000 ms elapse. */
int ready = poll(fds, nfds, 2000);
if (ready < 0) {
if (errno == EINTR) continue; /* interrupted: just retry */
perror("poll"); break;
}
if (ready == 0) { printf("timeout: nothing became ready\n"); break; }
for (nfds_t i = 0; i < nfds; i++) {
if (fds[i].fd < 0) continue; /* already closed */
/* POLLHUP and POLLERR are set by the KERNEL whether or not you
asked for them. Ignoring them makes the loop spin forever. */
if (fds[i].revents & (POLLHUP | POLLERR)) {
printf("fd %d hung up\n", fds[i].fd);
close(fds[i].fd);
fds[i].fd = -1; /* -1 makes poll skip it */
open_count--;
continue;
}
if (fds[i].revents & POLLIN) {
char buf[128];
/* Readable means AT LEAST ONE byte -- not a full buffer. */
ssize_t n = read(fds[i].fd, buf, sizeof buf);
if (n > 0) {
printf("fd %d: \"%.*s\"\n", fds[i].fd, (int)n, buf);
} else if (n == 0) { /* EOF: the writer closed */
printf("fd %d: EOF\n", fds[i].fd);
close(fds[i].fd);
fds[i].fd = -1;
open_count--;
} else if (errno != EAGAIN && errno != EINTR) {
perror("read");
}
}
}
}
while (wait(NULL) > 0) ; /* reap both children */
printf("all sources drained\n");
return EXIT_SUCCESS;
}
The same logic with select, showing what makes it more awkward:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/select.h>
#include <sys/wait.h> /* wait */
#include <errno.h>
#include <string.h>
int main(void)
{
int fds[2];
if (pipe(fds) < 0) return 1;
pid_t pid = fork();
if (pid == 0) {
close(fds[0]);
usleep(100000);
write(fds[1], "hello via select", 16);
close(fds[1]);
_exit(0);
}
close(fds[1]);
int rd = fds[0];
/* select IMPOSES a hard limit: a descriptor >= FD_SETSIZE cannot be
represented, and using one is undefined behavior. */
printf("FD_SETSIZE = %d (descriptors >= this cannot be watched)\n", FD_SETSIZE);
if (rd >= FD_SETSIZE) { fprintf(stderr, "fd too large for select\n"); return 1; }
for (;;) {
/* The set and the timeout are MODIFIED by select, so both must be
rebuilt on EVERY iteration -- a classic source of bugs. */
fd_set readfds;
FD_ZERO(&readfds);
FD_SET(rd, &readfds);
struct timeval tv = { .tv_sec = 2, .tv_usec = 0 };
/* nfds is the HIGHEST descriptor PLUS ONE, not a count. */
int ready = select(rd + 1, &readfds, NULL, NULL, &tv);
if (ready < 0) { if (errno == EINTR) continue; perror("select"); break; }
if (ready == 0) { printf("timeout\n"); break; }
if (FD_ISSET(rd, &readfds)) { /* the set now holds only ready fds */
char buf[64];
ssize_t n = read(rd, buf, sizeof buf);
if (n > 0) printf("read: \"%.*s\"\n", (int)n, buf);
else { printf("EOF\n"); break; }
}
}
close(rd);
wait(NULL);
return 0;
}
ONE THREAD, MANY CONNECTIONS
THREAD PER CONNECTION MULTIPLEXED EVENT LOOP
client A --> [thread 1: read] client A --+
client B --> [thread 2: read] client B --+--> poll() sleeps
client C --> [thread 3: read] client C --+ |
... ... | one wakes it
10,000 clients = 10,000 stacks v
(~8 MB each = 80 GB) service ONLY the ready ones
10,000 clients = 1 stack + 10,000 fds
COMPLEXITY PER CALL
select / poll : O(n) -- the kernel scans every watched descriptor
epoll / kqueue: O(ready) -- interest set kept in the kernel between calls
select | poll | epoll / kqueue | |
|---|---|---|---|
| Portability | everywhere | POSIX, near-universal | Linux / BSD only |
| Descriptor limit | FD_SETSIZE (~1024) | none | none |
| Input destroyed per call | yes — rebuild every time | no | no (kernel-resident) |
| Complexity per call | O(watched) | O(watched) | O(ready) |
| Timeout resolution | microseconds | milliseconds | milliseconds |
| Use for | legacy and maximum portability | new portable code | high-connection servers |
Key Takeaways
- Multiplexing lets one thread wait on many descriptors at once, sleeping until at least one is ready — the foundation of servers that scale past a thread per connection.
- Prefer
polltoselect:selectcaps descriptor numbers atFD_SETSIZE, destroys its sets on every call, and may modify its timeout. - Both are O(n) in the number of watched descriptors;
epollandkqueueare O(ready) but non-portable, so portable code usespolland scales with a wrapper library. - Readiness means at least one byte, not a full buffer, so always combine multiplexing with non-blocking descriptors and partial-transfer handling.
- Always check
POLLHUPandPOLLERR— the kernel sets them regardless of youreventsmask, and ignoring them makes an event loop spin on a dead connection.
🧪 Practice
- Write a program that watches both
stdinand a pipe withpoll, printing which source produced data and exiting when both reach EOF.- Add a 500 ms timeout to the loop and print a heartbeat each time
pollreturns zero, without disturbing the data handling.- Convert the
pollexample toselectand list every additional step you had to take per iteration.- Interview-style: "Why does
epollscale better thanpollfor 10,000 connections?" Hint: consider what each call must transfer between user space and the kernel, and how often.
Memory-Mapped Files with mmap
Theory
Reading a file with read copies data twice: the kernel pulls it from disk into its page cache, then copies it into your buffer. For large files this copying is pure overhead, and the buffer is a second copy of data the kernel already holds.
mmap removes the copy entirely by mapping the file's pages directly into your process's address space. After the call, the file is an array of bytes at some address. Reading it is a pointer dereference. Writing it is an assignment. There is no read, no write, and no buffer of your own.
void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset);
The mechanism is the virtual memory system rather than the I/O system. The kernel sets up page-table entries that point at the file's pages but marks them not present. When you first touch an address, the CPU raises a page fault, the kernel loads that page from disk, and your instruction resumes. Data is therefore loaded lazily, on demand, one page at a time — you can map a 10 GB file on a machine with 1 GB of RAM and read scattered parts of it without ever loading the rest.
Two flag choices define the behavior:
prot | Access allowed |
|---|---|
PROT_READ | read |
PROT_WRITE | write |
PROT_EXEC | execute |
PROT_NONE | none — useful for guard pages |
flags | Effect |
|---|---|
MAP_SHARED | writes go to the file and are visible to other mappers |
MAP_PRIVATE | copy-on-write; writes are private and never reach the file |
MAP_ANONYMOUS | not backed by a file — this is how malloc gets large blocks |
MAP_FIXED | map at exactly this address (dangerous; can replace existing mappings) |
MAP_SHARED is also the simplest form of shared memory between processes: two processes mapping the same file see each other's writes immediately, with no system call per access. MAP_ANONYMOUS | MAP_SHARED gives shared memory with no file at all, which is how related processes commonly share a region.
The advantages are substantial for the right workload: no copying, lazy loading, natural random access, and page sharing between processes mapping the same file. The costs are equally real and less obvious:
mmapreturnsMAP_FAILED, which is(void *)-1, notNULL. Checking forNULLsilently succeeds on failure.- Page faults are not free. For small files or purely sequential access,
readinto a buffer is usually faster, because the fault-per-page overhead exceeds the copy it avoids. - Errors become signals. A read error on a mapped page — or reading past the end of a file that was truncated after you mapped it — arrives as
SIGBUS, not as an error return. There is noerrnoto check. - The mapping size is rounded up to whole pages, so bytes between the file's end and the end of the last page read as zero and are discarded on write.
- Durability still requires
msync. Writes go to the page cache; onlymsync(ormunmapplusfsync) pushes them toward the disk.
Always munmap when done, and note that the file descriptor may be closed immediately after mmap — the mapping keeps its own reference.
Examples
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/mman.h> /* mmap, munmap, msync */
#include <sys/stat.h> /* fstat */
#include <string.h>
#include <errno.h>
int main(void)
{
const char *path = "/tmp/mmap_demo.txt";
/* Create a file with known content. */
FILE *f = fopen(path, "w");
if (!f) { perror(path); return EXIT_FAILURE; }
for (int i = 0; i < 100; i++) fprintf(f, "line %03d padding text here\n", i);
fclose(f);
/* --- map it read-only --------------------------------------------- */
int fd = open(path, O_RDONLY);
if (fd < 0) { perror("open"); return EXIT_FAILURE; }
struct stat st;
if (fstat(fd, &st) < 0) { perror("fstat"); close(fd); return EXIT_FAILURE; }
/* mmap fails with MAP_FAILED == (void*)-1, NOT NULL. */
char *data = mmap(NULL, (size_t)st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
if (data == MAP_FAILED) { perror("mmap"); close(fd); return EXIT_FAILURE; }
/* The mapping holds its own reference: the fd may be closed now. */
close(fd);
printf("mapped %lld bytes at %p\n", (long long)st.st_size, (void *)data);
/* The file is now just an array. No read(), no buffer, no copy.
Each first touch of a page triggers a page fault that loads it. */
printf("first 26 bytes: %.26s\n", data);
/* Random access costs nothing extra -- no lseek, no syscall at all. */
printf("byte 500 onward: %.26s\n", data + 500);
/* Counting newlines is a plain memory scan. */
size_t lines = 0;
for (off_t i = 0; i < st.st_size; i++)
if (data[i] == '\n') lines++;
printf("%zu lines counted by scanning memory directly\n", lines);
munmap(data, (size_t)st.st_size);
/* --- map it read-write and modify the file through memory --------- */
fd = open(path, O_RDWR);
if (fd < 0) { perror("open rw"); return EXIT_FAILURE; }
/* MAP_SHARED: writes reach the file and other mappers.
MAP_PRIVATE would give copy-on-write, invisible to the file. */
char *rw = mmap(NULL, (size_t)st.st_size, PROT_READ | PROT_WRITE,
MAP_SHARED, fd, 0);
if (rw == MAP_FAILED) { perror("mmap rw"); close(fd); return EXIT_FAILURE; }
close(fd);
memcpy(rw, "MODIFIED", 8); /* an assignment IS a file write */
/* Writes land in the page cache; msync pushes them toward the disk. */
if (msync(rw, (size_t)st.st_size, MS_SYNC) < 0) perror("msync");
munmap(rw, (size_t)st.st_size);
/* Verify through ordinary I/O that the file really changed. */
f = fopen(path, "r");
if (f) { char buf[32] = {0}; fgets(buf, sizeof buf, f); printf("file now: %s", buf); fclose(f); }
remove(path);
return EXIT_SUCCESS;
}
Anonymous shared memory between parent and child:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/wait.h>
#include <string.h>
typedef struct { int counter; char message[64]; } Shared;
int main(void)
{
/* MAP_ANONYMOUS: no file at all -- just memory.
MAP_SHARED: the mapping SURVIVES fork as shared, not copy-on-write. */
Shared *sh = mmap(NULL, sizeof *sh, PROT_READ | PROT_WRITE,
MAP_SHARED | MAP_ANONYMOUS, -1, 0);
if (sh == MAP_FAILED) { perror("mmap"); return EXIT_FAILURE; }
sh->counter = 0;
strcpy(sh->message, "written by parent");
pid_t pid = fork();
if (pid < 0) { perror("fork"); munmap(sh, sizeof *sh); return EXIT_FAILURE; }
if (pid == 0) {
/* Ordinary memory writes -- but the PARENT sees them, because the
pages are shared rather than copied. */
sh->counter = 42;
strcpy(sh->message, "written by child");
_exit(0);
}
wait(NULL);
printf("parent sees counter=%d message=\"%s\"\n", sh->counter, sh->message);
/* With MAP_PRIVATE instead, the parent would still see 0 and its own
message: the child's writes would have been copy-on-write copies. */
munmap(sh, sizeof *sh);
return EXIT_SUCCESS;
}
read() COPIES TWICE mmap() MAPS ONCE
[ disk ] [ disk ]
| |
v DMA v DMA
+-------------+ +-------------+
| page cache | | page cache |
+-------------+ +-------------+
| ^
v copy #2 | page table entry -- NO COPY
+-------------+ +-------------+
| your buffer | | your VA | data[i] is a dereference;
+-------------+ +-------------+ a fault loads the page
WHEN mmap WINS WHEN read WINS
large files small files
random access purely sequential
repeated access one pass
sharing between processes streaming (pipes, sockets)
no need for a buffer errors as errno, not SIGBUS
Key Takeaways
mmapmaps a file into the address space so reads and writes become ordinary memory access, eliminating the copy thatreadandwriteperform.- Pages load lazily on page faults, so a file far larger than RAM can be mapped and accessed sparsely.
MAP_SHAREDmakes writes visible in the file and to other mappers;MAP_PRIVATEgives copy-on-write that never reaches the file;MAP_ANONYMOUSgives memory with no file behind it.mmapfails withMAP_FAILED((void *)-1), notNULL, and I/O errors on mapped pages arrive asSIGBUSrather than as an error return.- Use
msyncfor durability,munmapwhen done, and prefer plainreadfor small files or purely sequential access where fault overhead outweighs the saved copy.
🧪 Practice
- Map a text file read-only and count the occurrences of a character by scanning memory, then compare the timing against a
read-based version.- Map a file
MAP_SHAREDfrom two processes simultaneously and demonstrate that a write in one is immediately visible in the other.- Map a file, truncate it to zero from another process, then touch the mapping and observe the
SIGBUS.- Interview-style: "When would you use
readinstead ofmmapfor a large file?" Hint: think about access pattern, and about how each one reports a disk error.
<a id="113-networking"></a>
11.3 Networking
A socket is a file descriptor that happens to reach another machine. That framing is most of what makes the Berkeley sockets API learnable — but the network adds failure modes files do not have: partial transfers, byte-order differences, and a peer that can vanish mid-conversation. This section covers the API's shape, both transport protocols, and the two details that break more networking code than anything else.
Sockets API Overview
Theory
A socket is an endpoint for communication, represented — like everything else in Unix — as a file descriptor. Once connected, you can read and write it exactly as you would a file. What differs is the setup, because unlike a file, a network endpoint needs an address family, a protocol, and a negotiation with a peer.
socket(domain, type, protocol) creates one:
domain | Address family |
|---|---|
AF_INET | IPv4 |
AF_INET6 | IPv6 |
AF_UNIX | local, filesystem-named (fastest for same-machine IPC) |
type | Semantics |
|---|---|
SOCK_STREAM | reliable, ordered byte stream (TCP) |
SOCK_DGRAM | unreliable, unordered messages (UDP) |
SOCK_RAW | direct protocol access (needs privileges) |
The protocol argument is almost always 0, meaning "the default for this domain and type".
The two roles follow different call sequences, and memorizing the shape is most of learning the API:
SERVER CLIENT
socket() create the endpoint socket()
| |
bind() claim an address |
| |
listen() mark it as accepting |
| |
accept() <-------- connection ------- connect()
| (accept returns a NEW |
| fd for THIS client) |
read/write <------ data both ways ---> read/write
| |
close() close()
The step that surprises newcomers is accept. It returns a new descriptor for the accepted connection; the original listening socket stays open and keeps accepting. A server therefore holds one listening descriptor plus one per active client — which is precisely why the multiplexing of the previous section matters.
Addresses are the API's ugliest part, for a historical reason. bind, connect, and accept all take a generic struct sockaddr *, but you actually fill in a family-specific struct — struct sockaddr_in for IPv4, sockaddr_in6 for IPv6, sockaddr_un for Unix domain — and cast. This is C's pre-void * approach to polymorphism, and the cast is required rather than optional.
Modern code avoids hand-filling those structs by using getaddrinfo, which turns a hostname and service name into a linked list of ready-to-use address structures. It handles DNS, IPv4 and IPv6 transparently, and service-name lookup ("http" to port 80). Code written with getaddrinfo and a loop over its results is IP-version agnostic for free; code that hardcodes sockaddr_in is IPv4-only forever.
Two options are worth setting on nearly every server. SO_REUSEADDR allows binding to a port still in the TIME_WAIT state from a previous run — without it, restarting a server fails with "Address already in use" for a minute or two. SO_KEEPALIVE enables periodic probes so a connection to a crashed peer is eventually detected rather than hanging forever.
Ports below 1024 are privileged and require root to bind.
Examples
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h> /* inet_pton, inet_ntop */
#include <netdb.h> /* getaddrinfo */
#include <errno.h>
int main(void)
{
/* --- creating sockets of each kind --------------------------------- */
int tcp = socket(AF_INET, SOCK_STREAM, 0); /* IPv4 TCP */
int udp = socket(AF_INET, SOCK_DGRAM, 0); /* IPv4 UDP */
if (tcp < 0 || udp < 0) { perror("socket"); return EXIT_FAILURE; }
printf("tcp fd=%d udp fd=%d (ordinary file descriptors)\n", tcp, udp);
/* --- filling an address by hand (IPv4-only; instructive, not ideal) -- */
struct sockaddr_in addr;
memset(&addr, 0, sizeof addr); /* zero EVERY field first */
addr.sin_family = AF_INET;
addr.sin_port = htons(8080); /* htons: host -> NETWORK byte order */
/* inet_pton parses a textual address; it returns 1 on success,
0 for a malformed string, -1 for a bad family. */
if (inet_pton(AF_INET, "127.0.0.1", &addr.sin_addr) != 1) {
fprintf(stderr, "inet_pton failed\n");
close(tcp); close(udp); return EXIT_FAILURE;
}
/* And back to text, for logging. */
char text[INET_ADDRSTRLEN];
inet_ntop(AF_INET, &addr.sin_addr, text, sizeof text);
printf("address: %s:%d\n", text, ntohs(addr.sin_port));
/* --- SO_REUSEADDR: without it, a restart fails while the old socket
lingers in TIME_WAIT ------------------------------------------ */
int yes = 1;
if (setsockopt(tcp, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof yes) < 0)
perror("setsockopt SO_REUSEADDR");
else
printf("SO_REUSEADDR set: rebinding after restart will succeed\n");
close(tcp); close(udp);
/* --- getaddrinfo: the modern, IP-version-agnostic way -------------- */
struct addrinfo hints, *result, *rp;
memset(&hints, 0, sizeof hints);
hints.ai_family = AF_UNSPEC; /* AF_UNSPEC = IPv4 OR IPv6 */
hints.ai_socktype = SOCK_STREAM;
hints.ai_flags = AI_PASSIVE; /* suitable for bind() */
/* Note the error convention: getaddrinfo returns a CODE, not -1/errno,
and gai_strerror translates it. */
int rc = getaddrinfo(NULL, "8080", &hints, &result);
if (rc != 0) {
fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(rc));
return EXIT_FAILURE;
}
printf("candidate addresses for port 8080:\n");
for (rp = result; rp != NULL; rp = rp->ai_next) {
char host[NI_MAXHOST];
if (getnameinfo(rp->ai_addr, rp->ai_addrlen, host, sizeof host,
NULL, 0, NI_NUMERICHOST) == 0)
printf(" %-6s %s\n",
rp->ai_family == AF_INET ? "IPv4" :
rp->ai_family == AF_INET6 ? "IPv6" : "other", host);
}
freeaddrinfo(result); /* the list is heap-allocated */
return EXIT_SUCCESS;
}
| Call | Role | Purpose |
|---|---|---|
socket | both | create the endpoint |
bind | server | claim a local address and port |
listen | server | mark it as accepting, set the backlog |
accept | server | take one pending connection, returning a NEW fd |
connect | client | initiate a connection to a peer |
send / recv | both | like write/read, plus socket-specific flags |
sendto / recvfrom | both | datagram forms carrying the peer address |
setsockopt | both | tune behavior (SO_REUSEADDR, timeouts, buffers) |
shutdown | both | close one direction while keeping the other |
close | both | release the descriptor |
Key Takeaways
- A socket is a file descriptor, so
read,write,poll, andcloseall work on it; only the setup differs from a file. - Servers follow
socket→bind→listen→accept, andacceptreturns a new descriptor per connection while the listener keeps accepting. - Address structs are family-specific but passed as
struct sockaddr *, so the cast is mandatory;getaddrinfobuilds them for you and makes code work over IPv4 and IPv6 alike. getaddrinforeports errors as a return code translated bygai_strerror, not througherrno, and its result list must be released withfreeaddrinfo.- Set
SO_REUSEADDRon servers so a restart can rebind a port still inTIME_WAIT, and remember that ports below 1024 require privileges.
🧪 Practice
- Write a program that resolves a hostname with
getaddrinfoand prints every address returned, labelled IPv4 or IPv6.- Create a TCP socket, bind it to port 0, and use
getsocknameto discover which ephemeral port the kernel assigned.- Compare
AF_UNIXandAF_INETsockets on the same machine by timing a million small round trips through each.- Interview-style: "Why does
acceptreturn a new file descriptor instead of reusing the listening one?" Hint: consider what the server must still be able to do while a client is connected.
TCP Client and Server
Theory
TCP gives you a reliable, ordered byte stream. The kernel handles retransmission, reordering, duplicate removal, and flow control; what arrives is exactly what was sent, in order, or the connection fails. That reliability is why TCP carries HTTP, SSH, and essentially everything where correctness matters more than latency.
The phrase byte stream is the part that must be taken literally, because it causes more bugs than anything else in network programming. TCP has no concept of messages. If the sender calls write three times with 10 bytes each, the receiver may get all 30 in one read, or 7 then 23, or 30 separate single-byte reads. The boundaries are not preserved because they were never transmitted — TCP moves bytes, not records.
Therefore every TCP protocol must define its own framing. The three standard approaches:
| Framing | How | Used by |
|---|---|---|
| Delimiter | read until a marker (\n, \r\n\r\n) | HTTP headers, SMTP, line protocols |
| Length prefix | send a fixed-size length, then that many bytes | most binary protocols |
| Fixed size | every message is exactly N bytes | simple telemetry |
The server sequence has one call worth explaining. listen(fd, backlog) does not wait for anything; it converts the socket into one that accepts connections and sets the queue depth for connections completed but not yet accepted. A small backlog under load causes clients to be refused.
accept then blocks until a connection is pending and returns a new descriptor for it. The classic structure is a loop: accept, handle, close, repeat. Handling one client at a time is fine for a demo and useless in production, so real servers either fork per client, use a thread pool, or — best — use the multiplexing from the previous section.
Two behaviors regularly surprise people:
read returning 0 means the peer closed the connection. It is an orderly shutdown, not an error. A read loop must treat 0 as "connection finished", distinct from -1.
A dead peer is not detected immediately. If the remote machine loses power, your socket stays open and your read blocks indefinitely, because nothing was sent to indicate closure. SO_KEEPALIVE or an application-level heartbeat is the only way to notice.
shutdown(fd, SHUT_WR) closes only the sending direction, sending a FIN while still allowing reads. That is how a client says "I have sent everything, now tell me your answer" — with close you would lose the ability to read the reply.
Finally, connect on a blocking socket can take a long time on an unreachable host (up to minutes). Non-blocking connect plus poll with a timeout is the standard remedy.
Examples
A complete, correct echo server with length-prefixed framing:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <errno.h>
#include <signal.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#define PORT 18080
#define BACKLOG 16
/* --- the two helpers every TCP program needs ------------------------- */
static int send_all(int fd, const void *buf, size_t n)
{
const char *p = buf;
while (n > 0) {
ssize_t w = send(fd, p, n, 0);
if (w < 0) { if (errno == EINTR) continue; return -1; }
p += w; n -= (size_t)w; /* a short send is NORMAL */
}
return 0;
}
static int recv_all(int fd, void *buf, size_t n)
{
char *p = buf;
while (n > 0) {
ssize_t r = recv(fd, p, n, 0);
if (r < 0) { if (errno == EINTR) continue; return -1; }
if (r == 0) return 1; /* peer closed: orderly EOF */
p += r; n -= (size_t)r;
}
return 0;
}
/* Framing: a 4-byte big-endian length, then that many bytes. Without
framing, TCP's byte stream gives no way to tell messages apart. */
static int send_msg(int fd, const char *msg, uint32_t len)
{
uint32_t net_len = htonl(len); /* network byte order */
if (send_all(fd, &net_len, sizeof net_len) < 0) return -1;
return send_all(fd, msg, len);
}
static int recv_msg(int fd, char *buf, uint32_t cap, uint32_t *out_len)
{
uint32_t net_len;
int rc = recv_all(fd, &net_len, sizeof net_len);
if (rc != 0) return rc; /* 1 = closed, -1 = error */
uint32_t len = ntohl(net_len);
if (len >= cap) return -1; /* refuse an oversized frame */
rc = recv_all(fd, buf, len);
if (rc != 0) return rc;
buf[len] = '\0';
*out_len = len;
return 0;
}
static void serve_client(int cfd, const struct sockaddr_in *peer)
{
char text[INET_ADDRSTRLEN];
inet_ntop(AF_INET, &peer->sin_addr, text, sizeof text);
printf("[server] client %s:%d connected\n", text, ntohs(peer->sin_port));
char buf[1024];
uint32_t len;
int rc;
/* rc == 1 means the client closed cleanly; that ends the loop. */
while ((rc = recv_msg(cfd, buf, sizeof buf, &len)) == 0) {
printf("[server] received %u bytes: \"%s\"\n", len, buf);
if (send_msg(cfd, buf, len) < 0) { perror("send"); break; }
}
if (rc < 0) perror("recv");
printf("[server] client disconnected\n");
close(cfd);
}
int main(void)
{
/* A client that vanishes mid-write would otherwise kill us outright. */
signal(SIGPIPE, SIG_IGN);
int lfd = socket(AF_INET, SOCK_STREAM, 0);
if (lfd < 0) { perror("socket"); return EXIT_FAILURE; }
/* Without SO_REUSEADDR, restarting fails with EADDRINUSE while the
old socket sits in TIME_WAIT. */
int yes = 1;
setsockopt(lfd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof yes);
struct sockaddr_in addr;
memset(&addr, 0, sizeof addr);
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = htonl(INADDR_ANY); /* every local interface */
addr.sin_port = htons(PORT);
if (bind(lfd, (struct sockaddr *)&addr, sizeof addr) < 0) {
perror("bind"); close(lfd); return EXIT_FAILURE;
}
/* listen() does NOT wait -- it marks the socket as accepting and sets
the queue depth for completed-but-unaccepted connections. */
if (listen(lfd, BACKLOG) < 0) {
perror("listen"); close(lfd); return EXIT_FAILURE;
}
printf("[server] listening on port %d\n", PORT);
for (int served = 0; served < 1; served++) {
struct sockaddr_in peer;
socklen_t peerlen = sizeof peer;
/* accept returns a NEW descriptor; lfd keeps listening. */
int cfd = accept(lfd, (struct sockaddr *)&peer, &peerlen);
if (cfd < 0) {
if (errno == EINTR) continue;
perror("accept"); break;
}
serve_client(cfd, &peer);
}
close(lfd);
return EXIT_SUCCESS;
}
The matching client, using getaddrinfo so it works over IPv4 or IPv6:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <errno.h>
#include <netdb.h>
#include <sys/socket.h>
#include <arpa/inet.h>
static int send_all(int fd, const void *buf, size_t n)
{
const char *p = buf;
while (n > 0) {
ssize_t w = send(fd, p, n, 0);
if (w < 0) { if (errno == EINTR) continue; return -1; }
p += w; n -= (size_t)w;
}
return 0;
}
static int recv_all(int fd, void *buf, size_t n)
{
char *p = buf;
while (n > 0) {
ssize_t r = recv(fd, p, n, 0);
if (r < 0) { if (errno == EINTR) continue; return -1; }
if (r == 0) return 1;
p += r; n -= (size_t)r;
}
return 0;
}
/* Try every address getaddrinfo returned until one connects. This loop
is what makes a client work over both IPv4 and IPv6 without changes. */
static int connect_to(const char *host, const char *service)
{
struct addrinfo hints, *result, *rp;
memset(&hints, 0, sizeof hints);
hints.ai_family = AF_UNSPEC; /* IPv4 or IPv6, whichever works */
hints.ai_socktype = SOCK_STREAM;
int rc = getaddrinfo(host, service, &hints, &result);
if (rc != 0) { fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(rc)); return -1; }
int fd = -1;
for (rp = result; rp; rp = rp->ai_next) {
fd = socket(rp->ai_family, rp->ai_socktype, rp->ai_protocol);
if (fd < 0) continue;
if (connect(fd, rp->ai_addr, rp->ai_addrlen) == 0) break; /* success */
close(fd);
fd = -1;
}
freeaddrinfo(result);
return fd;
}
int main(int argc, char **argv)
{
const char *host = argc > 1 ? argv[1] : "127.0.0.1";
const char *port = argc > 2 ? argv[2] : "18080";
int fd = connect_to(host, port);
if (fd < 0) { fprintf(stderr, "could not connect to %s:%s\n", host, port); return 1; }
printf("[client] connected to %s:%s\n", host, port);
const char *messages[] = { "hello", "framed messages", "goodbye" };
for (size_t i = 0; i < 3; i++) {
uint32_t len = (uint32_t)strlen(messages[i]);
uint32_t net = htonl(len);
if (send_all(fd, &net, sizeof net) < 0 ||
send_all(fd, messages[i], len) < 0) { perror("send"); break; }
/* Read the echoed reply, using the same framing. */
if (recv_all(fd, &net, sizeof net) != 0) break;
uint32_t rlen = ntohl(net);
char buf[1024];
if (rlen >= sizeof buf || recv_all(fd, buf, rlen) != 0) break;
buf[rlen] = '\0';
printf("[client] echo: \"%s\"\n", buf);
}
/* Half-close: send FIN but keep reading. With close() we could not
receive anything the server still had to send. */
shutdown(fd, SHUT_WR);
close(fd);
return 0;
}
WHY TCP NEEDS FRAMING
sender: send("AAA") send("BBBB") send("CC")
the wire: TCP is a BYTE STREAM -- the boundaries are not transmitted
receiver may see ANY of these:
read -> "AAABBBBCC" one read, everything coalesced
read -> "AAAB" "BBBCC" split at an arbitrary point
read -> "A" "A" "A" "B" ... one byte at a time
ALL THREE ARE CORRECT TCP BEHAVIOR.
LENGTH-PREFIXED FRAMING FIXES IT
[ 00 00 00 03 ][ A A A ][ 00 00 00 04 ][ B B B B ]
4-byte length payload 4-byte length payload
Read exactly 4 bytes, decode n, then read exactly n bytes. Now the
receiver can reconstruct the messages regardless of how TCP split them.
Key Takeaways
- TCP is a byte stream with no message boundaries — a single
writemay arrive as severalreads and several writes may coalesce into one. - Every TCP protocol must define its own framing: a delimiter, a length prefix, or a fixed size.
listendoes not block; it marks the socket as accepting and sets the backlog, whileacceptreturns a new descriptor per connection.recvreturning0means the peer closed cleanly and is not an error; a peer that crashes is not detected at all without keepalives or heartbeats.- Ignore
SIGPIPEand setSO_REUSEADDRon servers, and useshutdown(fd, SHUT_WR)to signal "done sending" while still reading the reply.
🧪 Practice
- Run the server and client, then modify the client to send one 5000-byte message and confirm the framing still reassembles it correctly.
- Change the server to fork a child per connection so it can serve several clients at once, reaping children with a
SIGCHLDhandler.- Replace length-prefix framing with newline-delimited framing, and handle the case where a single
recvcontains one and a half lines.- Interview-style: "A client sends three 10-byte messages and the server's first
readreturns 30 bytes. Is the server broken?" Hint: ask what guarantee TCP actually makes about boundaries.
UDP Datagrams
Theory
TCP's reliability costs something: a three-way handshake before any data moves, acknowledgements, retransmission timers, and head-of-line blocking where one lost packet stalls everything behind it. UDP removes all of it. You send a datagram; it may arrive, may not, and may arrive after one you sent later.
That sounds strictly worse until you consider what the guarantees cost. For a live voice call, a packet retransmitted 200 ms late is useless — you needed it then, and playing it now would be worse than the gap. For DNS, a single request and reply is faster with one round trip than with a handshake plus a request plus a teardown. For a game position update sent 60 times a second, a lost update is superseded by the next one before a retransmission could arrive.
UDP's properties, stated precisely:
| Property | TCP | UDP |
|---|---|---|
| Connection setup | 3-way handshake | none |
| Delivery | guaranteed or the connection fails | best effort |
| Ordering | guaranteed | none |
| Duplicates | removed | possible |
| Boundaries | none — a byte stream | preserved — one send, one receive |
| Flow control | yes | none |
| Congestion control | yes | none (you must implement it) |
| Header overhead | 20 bytes | 8 bytes |
| Broadcast/multicast | no | yes |
The row that matters most in code is message boundaries. UDP is datagram-oriented: one sendto produces exactly one recvfrom, with the same byte count. All the framing work TCP requires simply disappears. The flip side is that a datagram is atomic — if your buffer is smaller than the arriving datagram, the excess is silently discarded, not saved for the next read.
The API differs in that there is no connection, so every send and receive carries an address:
sendto(fd, buf, len, flags, dest_addr, addrlen)— the destination travels with the data.recvfrom(fd, buf, len, flags, src_addr, addrlen)— fills in who sent it, which is how a server knows where to reply.
There is no listen and no accept. A UDP server just binds and starts receiving. One socket serves every client, which is why a UDP server is often simpler than a TCP one.
You can call connect on a UDP socket. It does not send anything; it merely records a default peer, letting you use send/recv and causing the kernel to filter out datagrams from other sources. It also lets you receive ICMP errors such as "port unreachable", which an unconnected UDP socket cannot.
On size: keep datagrams under roughly 1472 bytes on typical Ethernet to avoid IP fragmentation, since a fragmented datagram is lost entirely if any fragment is lost. 508 bytes is the conservative figure guaranteed not to fragment anywhere.
Applications needing some reliability over UDP add it themselves — sequence numbers, acknowledgements, and retransmission — which is exactly what QUIC does, and why "just use TCP" is usually the right answer unless you have measured a reason not to.
Examples
A UDP server and client in one program, using fork so both can run:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <errno.h>
#include <sys/socket.h>
#include <sys/wait.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <sys/time.h>
#define PORT 19090
/* ---------------- server: bind and receive; no listen, no accept -------- */
static void udp_server(void)
{
int fd = socket(AF_INET, SOCK_DGRAM, 0); /* SOCK_DGRAM, not STREAM */
if (fd < 0) { perror("socket"); _exit(1); }
int yes = 1;
setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof yes);
struct sockaddr_in addr;
memset(&addr, 0, sizeof addr);
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = htonl(INADDR_ANY);
addr.sin_port = htons(PORT);
if (bind(fd, (struct sockaddr *)&addr, sizeof addr) < 0) {
perror("bind"); close(fd); _exit(1);
}
printf("[server] listening on UDP %d (no listen/accept needed)\n", PORT);
/* Don't hang forever if the client dies. */
struct timeval tv = { .tv_sec = 3, .tv_usec = 0 };
setsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof tv);
for (int i = 0; i < 3; i++) {
char buf[1500];
struct sockaddr_in peer;
socklen_t peerlen = sizeof peer;
/* recvfrom fills in WHO sent it -- there is no connection to ask. */
ssize_t n = recvfrom(fd, buf, sizeof buf - 1, 0,
(struct sockaddr *)&peer, &peerlen);
if (n < 0) {
if (errno == EAGAIN || errno == EWOULDBLOCK) {
printf("[server] timed out waiting\n"); break;
}
perror("recvfrom"); break;
}
buf[n] = '\0';
char text[INET_ADDRSTRLEN];
inet_ntop(AF_INET, &peer.sin_addr, text, sizeof text);
/* ONE sendto produced exactly ONE recvfrom, with the same length.
No framing was needed: boundaries are preserved. */
printf("[server] datagram of %zd bytes from %s:%d: \"%s\"\n",
n, text, ntohs(peer.sin_port), buf);
/* Reply to the address recvfrom gave us. */
sendto(fd, buf, (size_t)n, 0, (struct sockaddr *)&peer, peerlen);
}
close(fd);
_exit(0);
}
/* ---------------- client: no connect required -------------------------- */
static void udp_client(void)
{
int fd = socket(AF_INET, SOCK_DGRAM, 0);
if (fd < 0) { perror("socket"); return; }
struct sockaddr_in server;
memset(&server, 0, sizeof server);
server.sin_family = AF_INET;
server.sin_port = htons(PORT);
inet_pton(AF_INET, "127.0.0.1", &server.sin_addr);
struct timeval tv = { .tv_sec = 2, .tv_usec = 0 };
setsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof tv);
const char *messages[] = { "first", "second datagram", "third" };
for (int i = 0; i < 3; i++) {
/* No handshake, no connection: the address travels with the data. */
if (sendto(fd, messages[i], strlen(messages[i]), 0,
(struct sockaddr *)&server, sizeof server) < 0) {
perror("sendto"); break;
}
char buf[1500];
struct sockaddr_in from;
socklen_t fromlen = sizeof from;
ssize_t n = recvfrom(fd, buf, sizeof buf - 1, 0,
(struct sockaddr *)&from, &fromlen);
if (n < 0) {
/* A lost datagram is NORMAL for UDP -- there is no retransmission. */
printf("[client] no reply for \"%s\" (lost or timed out)\n", messages[i]);
continue;
}
buf[n] = '\0';
printf("[client] echo of %zd bytes: \"%s\"\n", n, buf);
}
close(fd);
}
int main(void)
{
pid_t pid = fork();
if (pid < 0) { perror("fork"); return EXIT_FAILURE; }
if (pid == 0) udp_server();
usleep(200000); /* let the server bind before we send */
udp_client();
waitpid(pid, NULL, 0);
return EXIT_SUCCESS;
}
The truncation trap, which has no TCP equivalent:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
int main(void)
{
int a = socket(AF_INET, SOCK_DGRAM, 0);
int b = socket(AF_INET, SOCK_DGRAM, 0);
if (a < 0 || b < 0) { perror("socket"); return 1; }
struct sockaddr_in addr;
memset(&addr, 0, sizeof addr);
addr.sin_family = AF_INET;
addr.sin_port = 0; /* let the kernel pick */
inet_pton(AF_INET, "127.0.0.1", &addr.sin_addr);
if (bind(b, (struct sockaddr *)&addr, sizeof addr) < 0) { perror("bind"); return 1; }
socklen_t len = sizeof addr;
getsockname(b, (struct sockaddr *)&addr, &len); /* which port did we get? */
char big[1000];
memset(big, 'X', sizeof big);
sendto(a, big, sizeof big, 0, (struct sockaddr *)&addr, len);
/* Receive into a buffer far too small for the datagram. */
char small[100];
ssize_t n = recvfrom(b, small, sizeof small, 0, NULL, NULL);
printf("sent 1000 bytes, received %zd\n", n);
printf("The other 900 bytes are DISCARDED, not queued for the next read.\n");
printf("A datagram is atomic: TCP would have given them to you later.\n");
close(a); close(b);
return 0;
}
Key Takeaways
- UDP trades reliability, ordering, and flow control for lower latency and no connection setup — useful when late data is worthless.
- Datagram boundaries are preserved: one
sendtoyields exactly onerecvfromof the same size, so UDP needs none of TCP's framing work. - A datagram larger than your receive buffer is truncated and the remainder is discarded permanently, unlike a TCP stream that keeps the rest.
- There is no
listenoraccept; a server binds and serves every client through one socket, using the addressrecvfromsupplies to reply. - Keep datagrams under about 1472 bytes to avoid IP fragmentation, and implement your own sequencing and retransmission if you need any reliability.
🧪 Practice
- Write a UDP time server that replies with the current time to any datagram it receives, and a client that queries it.
- Demonstrate truncation by sending a 2000-byte datagram into a 500-byte buffer, and confirm the remainder is gone.
- Add sequence numbers and a retransmission timer to the client so it re-sends any datagram not acknowledged within 500 ms.
- Interview-style: "Why does DNS use UDP rather than TCP?" Hint: compare the number of round trips needed for one small question and one small answer.
Byte Order Conversion
Theory
A 32-bit integer occupies four bytes, but machines disagree about the order to store them in. Big-endian puts the most significant byte first; little-endian puts the least significant byte first. Both are self-consistent, and neither is wrong.
The value 0x12345678 in memory
BIG-ENDIAN (network order, SPARC, older PowerPC)
address: 100 101 102 103
byte: 12 34 56 78 most significant FIRST
LITTLE-ENDIAN (x86, x86-64, most ARM)
address: 100 101 102 103
byte: 78 56 34 12 least significant FIRST
Within one machine this is invisible — writing 0x12345678 and reading it back gives 0x12345678 either way. It becomes visible the moment bytes cross a boundary: a network, a file, or a shared memory region between machines of different architectures. Send the four bytes from a little-endian machine, interpret them on a big-endian one, and 0x12345678 becomes 0x78563412.
The internet protocols solved this by decree: network byte order is big-endian. Every multi-byte field in an IP, TCP, or UDP header is big-endian, regardless of the hosts involved. Your program must convert on the way out and on the way in.
Four macros do this, named for the direction and the width:
| Macro | Converts | Use for |
|---|---|---|
htons | host to network, short (16-bit) | port numbers |
htonl | host to network, long (32-bit) | IPv4 addresses, 32-bit fields |
ntohs | network to host, short | ports read from the wire |
ntohl | network to host, long | 32-bit fields read from the wire |
Read them as "h to n s" — the naming is systematic once you see it. On a big-endian machine they compile to nothing; on a little-endian one they become a byte-swap instruction. Either way, using them is always correct, which is why you apply them unconditionally rather than testing the platform.
The rules that keep you out of trouble:
- Convert every multi-byte value crossing the wire, including your own protocol's fields — not just the ones in
sockaddr_in. - Never convert single bytes. A
charoruint8_thas no order to get wrong. - Convert exactly once in each direction. Applying
htonstwice on a little-endian machine swaps back to the original, and this bug survives testing on a single architecture because both ends make the same mistake. - There are no standard 64-bit macros. POSIX does not define
htonll; usehtobe64/be64tohon Linux, or send two 32-bit halves.
Floating-point numbers have no standard network representation at all. Sending a raw double assumes both machines use IEEE-754 with the same byte order. Portable protocols send floats as text, as a scaled integer, or in an explicitly specified binary format.
The deeper lesson generalizes beyond networks: any binary format shared between machines must specify its byte order, which is why file formats state theirs and why writing a raw struct to disk is not portable.
Examples
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <string.h>
#include <arpa/inet.h> /* htons, htonl, ntohs, ntohl */
/* Detect this machine's byte order by inspecting the bytes of a known value. */
static const char *endianness(void)
{
uint32_t probe = 0x12345678;
unsigned char first = *(unsigned char *)&probe; /* the LOWEST address */
return first == 0x12 ? "big-endian" : "little-endian";
}
static void dump(const char *label, const void *p, size_t n)
{
const unsigned char *b = p;
printf("%-22s", label);
for (size_t i = 0; i < n; i++) printf("%02X ", b[i]);
putchar('\n');
}
int main(void)
{
printf("this machine is %s\n\n", endianness());
uint32_t host_value = 0x12345678;
uint32_t net_value = htonl(host_value); /* to NETWORK order */
dump("host 0x12345678:", &host_value, sizeof host_value);
dump("network order:", &net_value, sizeof net_value);
printf("as an integer, net_value reads as 0x%08X on this machine\n\n", net_value);
/* Ports are 16-bit: use the 's' variants. */
uint16_t port = 8080;
uint16_t net_port = htons(port);
printf("port %u -> network 0x%04X -> back to %u\n\n",
port, net_port, ntohs(net_port));
/* --- the double-conversion bug ------------------------------------ */
uint32_t once = htonl(host_value);
uint32_t twice = htonl(once); /* WRONG: swapped back */
printf("htonl applied once: 0x%08X\n", once);
printf("htonl applied twice: 0x%08X <- back to the original: a BUG\n",
twice);
printf("This survives testing when both ends make the same mistake.\n\n");
/* --- a protocol header: convert EVERY multi-byte field ------------ */
struct Header {
uint8_t version; /* single byte: NEVER convert */
uint8_t type; /* single byte: NEVER convert */
uint16_t flags; /* convert */
uint32_t length; /* convert */
uint32_t checksum; /* convert */
};
struct Header out = { .version = 1, .type = 7,
.flags = 0x00FF, .length = 4096, .checksum = 0xDEADBEEF };
/* Serialize into a byte buffer field by field. Writing the struct
directly would also bake in the compiler's PADDING. */
unsigned char wire[12];
size_t off = 0;
wire[off++] = out.version; /* 1 byte: as is */
wire[off++] = out.type;
uint16_t f = htons(out.flags); memcpy(wire + off, &f, 2); off += 2;
uint32_t l = htonl(out.length); memcpy(wire + off, &l, 4); off += 4;
uint32_t c = htonl(out.checksum); memcpy(wire + off, &c, 4); off += 4;
dump("serialized header:", wire, off);
/* Deserialize: the exact mirror, with ntoh* in place of hton*. */
struct Header in;
off = 0;
in.version = wire[off++];
in.type = wire[off++];
memcpy(&f, wire + off, 2); in.flags = ntohs(f); off += 2;
memcpy(&l, wire + off, 4); in.length = ntohl(l); off += 4;
memcpy(&c, wire + off, 4); in.checksum = ntohl(c); off += 4;
printf("round trip: version=%u type=%u flags=0x%04X length=%u checksum=0x%08X\n",
in.version, in.type, in.flags, in.length, in.checksum);
/* --- 64-bit: no standard macro exists ------------------------------ */
uint64_t big = 0x0123456789ABCDEFULL;
uint32_t hi = htonl((uint32_t)(big >> 32)); /* send as two halves */
uint32_t lo = htonl((uint32_t)(big & 0xFFFFFFFFu));
uint64_t back = ((uint64_t)ntohl(hi) << 32) | ntohl(lo);
printf("\n64-bit round trip: 0x%016llX -> 0x%016llX\n",
(unsigned long long)big, (unsigned long long)back);
return 0;
}
Key Takeaways
- Network byte order is big-endian by definition, so every multi-byte field crossing the wire must be converted regardless of the hosts involved.
- Use
htons/htonlwhen sending andntohs/ntohlwhen receiving; they compile to nothing on big-endian machines, so applying them is always correct. - Never convert single bytes, and convert exactly once per direction — a double conversion cancels out and hides itself when both ends share the bug.
- There are no standard 64-bit conversion macros; use platform functions such as
htobe64, or send two 32-bit halves. - Serialize protocol structs field by field rather than writing them wholesale, which would also transmit the compiler's padding; floating-point has no standard wire format at all.
🧪 Practice
- Write a function that detects the machine's endianness at runtime and print the raw bytes of
0xAABBCCDDbefore and afterhtonl.- Serialize a struct containing a
uint8_t, auint16_t, and auint32_tinto a byte array and back, verifying the round trip is exact.- Implement
my_htonllandmy_ntohllfor 64-bit values and test them against known byte patterns.- Interview-style: "Two programs exchange binary structs and work perfectly in testing, then fail between a server and an embedded device. What is the likely cause?" Hint: consider what both test machines had in common that the pair does not.
Handling Partial Reads and Writes
Theory
This is the single most common source of bugs in network code, and it comes from a reasonable but wrong assumption: that asking to send 1000 bytes sends 1000 bytes.
send and write return the number of bytes actually transferred, which may be less than requested. The kernel has a finite socket buffer; if it can only fit 300 bytes right now, it takes 300, returns 300, and expects you to deal with the rest. The same applies to recv and read: they return what is available now, not what you asked for.
This is not an error condition and there is no flag to disable it. It is how the interface works.
Why it survives testing is the insidious part. On loopback with small messages, buffers are almost never full, so send transfers everything every time. The bug appears in production — under load, on a slow link, with a busy receiver, or with larger messages — as truncated data or a protocol that mysteriously desynchronizes.
The cure is to never call send, recv, read, or write directly on a socket. Write two helpers once and use them everywhere:
send_allloops until every byte is transferred, advancing the pointer and decrementing the count by the amount actually moved.recv_allloops until the requested count is received, treating a0return as the peer closing.
Both must also handle EINTR, which means a signal interrupted the call and it should be retried, not reported as a failure.
The asymmetry between the two is worth understanding. For writing, you always want everything sent, so a loop is unconditionally correct. For reading, "read exactly N bytes" is only meaningful when you know N in advance — which is precisely what a length prefix gives you. With delimiter framing you instead read into a buffer, scan for the delimiter, and keep any leftover bytes for the next message. That leftover handling is mandatory: a single recv routinely contains the end of one message and the beginning of the next.
A subtle trap: recv_all returning 0 bytes at the start of a message is an orderly close, but a 0 return partway through a message means the peer closed mid-message — a protocol error, not a clean shutdown. Distinguish them, or a truncated message gets processed as if it were complete.
Finally, MSG_WAITALL on Linux asks recv to wait for the full count, but it is not universally available, still returns early on a signal or a closed connection, and does not remove the need for the loop. Write the loop.
Examples
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <errno.h>
#include <stdint.h>
#include <sys/socket.h>
#include <arpa/inet.h> /* htonl, ntohl */
/* =====================================================================
The two functions that belong in every networked C program.
===================================================================== */
/* Send every byte. Returns 0 on success, -1 on error.
A short send is NORMAL: the kernel buffer had limited room. */
int send_all(int fd, const void *buf, size_t len)
{
const char *p = buf;
size_t remaining = len;
while (remaining > 0) {
ssize_t sent = send(fd, p, remaining, 0);
if (sent < 0) {
if (errno == EINTR) continue; /* signal: retry */
return -1; /* real error */
}
/* Advance by what ACTUALLY went, not by what we requested. */
p += sent;
remaining -= (size_t)sent;
}
return 0;
}
/* Receive exactly len bytes.
Returns 0 = complete
1 = peer closed cleanly BEFORE any byte of this read
-1 = error, or the peer closed MID-message (a protocol error) */
int recv_all(int fd, void *buf, size_t len)
{
char *p = buf;
size_t remaining = len;
while (remaining > 0) {
ssize_t got = recv(fd, p, remaining, 0);
if (got < 0) {
if (errno == EINTR) continue;
return -1;
}
if (got == 0) {
/* A clean close at a message boundary is fine; halfway
through a message it means the peer died mid-send. */
return (remaining == len) ? 1 : -1;
}
p += got;
remaining -= (size_t)got;
}
return 0;
}
/* --- length-prefixed messages, built on the two helpers --------------- */
int send_message(int fd, const void *payload, uint32_t len)
{
uint32_t net_len = htonl(len);
if (send_all(fd, &net_len, sizeof net_len) < 0) return -1;
return send_all(fd, payload, len);
}
int recv_message(int fd, void *buf, uint32_t cap, uint32_t *out_len)
{
uint32_t net_len;
/* Step 1: read exactly the 4-byte length. */
int rc = recv_all(fd, &net_len, sizeof net_len);
if (rc != 0) return rc;
uint32_t len = ntohl(net_len);
/* Step 2: sanity-check it. A hostile or buggy peer can claim 4 GB,
and allocating on an untrusted length is a denial-of-service hole. */
if (len > cap) {
fprintf(stderr, "declared length %u exceeds capacity %u\n", len, cap);
return -1;
}
/* Step 3: read exactly that many bytes. */
rc = recv_all(fd, buf, len);
if (rc != 0) return -1; /* a close here is MID-message */
*out_len = len;
return 0;
}
Delimiter framing, where leftover bytes must be carried across reads:
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <errno.h>
#include <sys/socket.h>
/* A line reader that keeps whatever it over-read for the next call.
Without this buffer, the tail of a recv that contained one and a half
lines would be silently lost. */
typedef struct {
int fd;
char buf[4096];
size_t len; /* valid bytes currently held */
} LineReader;
static void lr_init(LineReader *lr, int fd) { lr->fd = fd; lr->len = 0; }
/* Returns the line length on success, 0 at EOF, -1 on error. */
static ssize_t lr_next(LineReader *lr, char *out, size_t cap)
{
for (;;) {
/* Is a complete line already buffered from a previous read? */
char *nl = memchr(lr->buf, '\n', lr->len);
if (nl) {
size_t line_len = (size_t)(nl - lr->buf);
if (line_len >= cap) return -1; /* caller's buffer too small */
memcpy(out, lr->buf, line_len);
out[line_len] = '\0';
/* KEEP the bytes after the newline: they are the start of the
next line and must survive into the following call. */
size_t consumed = line_len + 1;
memmove(lr->buf, lr->buf + consumed, lr->len - consumed);
lr->len -= consumed;
return (ssize_t)line_len;
}
if (lr->len == sizeof lr->buf) return -1; /* line too long */
/* No complete line yet: read more, appending to what we hold. */
ssize_t n = recv(lr->fd, lr->buf + lr->len, sizeof lr->buf - lr->len, 0);
if (n < 0) { if (errno == EINTR) continue; return -1; }
if (n == 0) return 0; /* EOF */
lr->len += (size_t)n;
}
}
WHY THE LOOP IS MANDATORY
send(fd, buf, 1000) with only 300 bytes of kernel buffer free
returns 300 <- NOT an error, NOT a failure
700 bytes remain unsent
WITHOUT A LOOP:
the caller assumes 1000 went, moves on -> the peer receives a
truncated message and the protocol desynchronizes permanently
WITH send_all:
send(p, 1000) -> 300 p += 300, remaining = 700
send(p+300, 700) -> 700 p += 700, remaining = 0
done, all 1000 bytes sent
WHY IT PASSES TESTING
loopback + small messages = buffers never fill = always complete.
The bug waits for load, a slow link, or a bigger message.
| Return value | send / write | recv / read |
|---|---|---|
> 0 but less than requested | partial — loop for the rest | partial — loop if you need an exact count |
== requested | complete | complete |
0 | nothing sent (retry) | peer closed / end of file |
-1 with EINTR | interrupted — retry | interrupted — retry |
-1 with EAGAIN | would block — wait for writability | would block — wait for readability |
-1 other | real error | real error |
Key Takeaways
send,recv,read, andwritemay transfer fewer bytes than requested; this is normal behavior, not an error, and cannot be turned off.- The bug hides in testing because loopback with small messages rarely fills a buffer — it appears under load, on slow links, or with larger payloads.
- Write
send_allandrecv_allonce, advancing by the amount actually transferred and retrying onEINTR, and never call the raw functions on a socket again. - A
recvreturning0at a message boundary is an orderly close; returning0partway through a message is a protocol error and must be distinguished. - With delimiter framing, always carry over the bytes read past the delimiter — a single
recvroutinely spans a message boundary — and validate any length prefix before trusting it.
🧪 Practice
- Write
send_allandrecv_allfrom scratch and test them by sending a 1 MB message through a socket with a deliberately smallSO_SNDBUF.- Instrument
send_allto count iterations, then send messages of 100 bytes and 10 MB and compare the loop counts.- Implement the
LineReaderand prove the leftover handling works by having the peer send"a\nb\nc"in a single write.- Interview-style: "Your file transfer works on localhost but corrupts data over the internet. What would you check first?" Hint: ask what loopback almost never does that a real network does constantly.
<a id="114-concurrency"></a>
11.4 Concurrency
Threads let one process do several things at once by sharing its address space — which is exactly what makes them fast and exactly what makes them dangerous. This section covers creating and joining threads, the primitives that make sharing safe, the two classic failure modes, C11's standard alternative to POSIX threads, and what the compiler and CPU are allowed to reorder behind your back.
POSIX Threads Creation and Joining
Theory
fork gives concurrency with isolation: the child has its own copy of everything, so nothing can be corrupted by a race. That isolation is also the limitation — sharing data between processes needs pipes, shared memory, or files, and each is slower and clumsier than a variable.
A thread is a second flow of control inside the same process. All threads share the heap, globals, file descriptors, and the code itself. Only the stack and the registers are private. Creating a thread is far cheaper than a process, and threads communicate simply by touching the same memory.
Process (fork) | Thread (pthread_create) | |
|---|---|---|
| Address space | private copy | shared |
| Globals and heap | private after copy-on-write | shared |
| File descriptors | copied (independent afterwards) | shared |
| Stack | private | private |
| Creation cost | higher | lower |
| Crash containment | one process dies | the whole process dies |
| Communication | pipes, shared memory, sockets | ordinary variables |
| Data races possible | no | yes — this is the whole difficulty |
The last two rows are the trade. Threads make sharing free, so they also make accidental sharing free, and a crash in any thread takes down every thread.
The API is small:
int pthread_create(pthread_t *thread, const pthread_attr_t *attr,
void *(*start_routine)(void *), void *arg);
int pthread_join(pthread_t thread, void **retval);
int pthread_detach(pthread_t thread);
The thread function has a fixed signature — takes void *, returns void * — which is C's usual way of passing anything through a generic interface. To pass several values, pass a pointer to a struct.
Three details cause most beginner bugs.
Pthreads functions return an error code directly; they do not set errno. if (pthread_create(...) != 0) is the check, and the return value is what you pass to strerror. This differs from every other POSIX call in this chapter.
Each thread needs its own argument storage. The classic mistake is passing &i from a loop: by the time the thread reads it, i has changed, so several threads see the same value or a value that no longer exists. Give each thread its own struct, or pass a pointer to a distinct array element.
Every thread must be either joined or detached. pthread_join blocks until the thread finishes and collects its return value — the thread equivalent of waitpid. A thread that is neither joined nor detached leaks its stack and bookkeeping when it exits, which is the thread version of a zombie. pthread_detach says "clean up automatically, I will not collect a result".
A thread returns a value by returning from its function or calling pthread_exit. Never return a pointer to a local variable — the stack is gone once the thread ends. Return heap memory or a pointer to something the caller owns.
Note that main returning terminates the entire process, including still-running threads. Join them first, or the program ends unpredictably.
Compile with -pthread, which sets the right preprocessor defines and links the library.
Examples
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <pthread.h>
#include <unistd.h>
#define NTHREADS 4
/* Each thread gets its OWN argument struct. Passing &i from the loop
would let the value change before the thread reads it. */
typedef struct {
int id;
int start, end;
long result; /* the thread writes its answer here */
} Work;
static void *worker(void *arg)
{
Work *w = arg; /* the fixed signature means a cast on entry */
long sum = 0;
for (int i = w->start; i < w->end; i++) sum += i;
w->result = sum; /* each thread writes to its OWN struct: no race */
printf("thread %d summed [%d, %d) = %ld\n", w->id, w->start, w->end, sum);
/* Returning a pointer to a LOCAL would be a dangling pointer -- this
thread's stack disappears the moment it returns. Point at the
caller-owned struct instead. */
return &w->result;
}
int main(void)
{
pthread_t threads[NTHREADS];
Work work[NTHREADS]; /* lives in main's frame: outlives the threads
only because we join before returning */
const int total = 1000000;
const int chunk = total / NTHREADS;
for (int i = 0; i < NTHREADS; i++) {
work[i].id = i;
work[i].start = i * chunk;
work[i].end = (i == NTHREADS - 1) ? total : (i + 1) * chunk;
work[i].result = 0;
/* pthreads returns an ERROR CODE; it does NOT set errno. */
int rc = pthread_create(&threads[i], NULL, worker, &work[i]);
if (rc != 0) {
fprintf(stderr, "pthread_create: %s\n", strerror(rc));
return EXIT_FAILURE;
}
}
/* Join every thread: this both waits for it and releases its resources.
A thread that is neither joined nor detached leaks, like a zombie. */
long grand_total = 0;
for (int i = 0; i < NTHREADS; i++) {
void *retval;
int rc = pthread_join(threads[i], &retval);
if (rc != 0) { fprintf(stderr, "pthread_join: %s\n", strerror(rc)); continue; }
grand_total += *(long *)retval; /* points into work[i] */
}
printf("total = %ld (expected %ld)\n",
grand_total, (long)total * (total - 1) / 2);
return EXIT_SUCCESS;
}
The argument-lifetime bug, and detached threads:
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <unistd.h>
static void *print_id(void *arg)
{
int id = *(int *)arg; /* dereferences whatever it points at NOW */
usleep(1000);
printf(" got id %d\n", id);
return NULL;
}
static void *detached_worker(void *arg)
{
(void)arg;
usleep(50000);
printf(" detached thread finished; its resources free themselves\n");
return NULL;
}
int main(void)
{
pthread_t t[3];
/* --- WRONG: every thread receives the SAME address -------------- */
printf("passing &i (all threads share one variable):\n");
for (int i = 0; i < 3; i++)
pthread_create(&t[i], NULL, print_id, &i); /* i changes underneath */
for (int i = 0; i < 3; i++) pthread_join(t[i], NULL);
/* Typically prints 3 3 3, or garbage once the loop variable dies. */
/* --- RIGHT: each thread gets a distinct object ------------------- */
printf("passing &ids[i] (each thread has its own):\n");
int ids[3] = { 10, 20, 30 };
for (int i = 0; i < 3; i++)
pthread_create(&t[i], NULL, print_id, &ids[i]);
for (int i = 0; i < 3; i++) pthread_join(t[i], NULL);
/* --- detached: fire and forget, never joined -------------------- */
pthread_t d;
if (pthread_create(&d, NULL, detached_worker, NULL) == 0) {
pthread_detach(d); /* resources are reclaimed automatically */
usleep(150000); /* main must not return before it runs:
returning from main kills every thread */
}
printf("main returning (this terminates ALL threads)\n");
return 0;
}
# -pthread sets the required defines AND links the library. Use it for
# both compiling and linking; plain -lpthread is not equivalent.
$ gcc -pthread -Wall program.c -o program
# ThreadSanitizer is the single most valuable tool for threaded C:
$ gcc -fsanitize=thread -g -pthread program.c -o program
Key Takeaways
- Threads share the heap, globals, and file descriptors while keeping private stacks, making communication free and data races possible.
- Pthreads functions return an error code directly and do not set
errno— check the return value and pass it tostrerror. - Give every thread its own argument storage; passing the address of a loop variable lets the value change before the thread reads it.
- Join or detach every thread, and never return a pointer to a thread's local variable, whose stack disappears when it ends.
- Returning from
mainterminates every thread, so join them first; compile and link with-pthread.
🧪 Practice
- Write a program that starts four threads, each printing its index, and demonstrate the difference between passing
&iand passing&ids[i].- Parallelize summing a 10-million-element array across N threads and compare the timing against a single-threaded loop for N = 1, 2, 4, and 8.
- Create a detached thread and confirm with
valgrindthat no resources leak when it is never joined.- Interview-style: "When would you use
forkrather than threads?" Hint: think about what happens to the other workers when one of them segfaults.
Mutexes and Condition Variables
Theory
Shared memory is a thread's advantage and its hazard. The moment two threads write the same variable, or one writes while another reads, you have a data race — and the result is not merely an unpredictable value, it is undefined behavior.
The reason is that operations that look atomic in C are not. counter++ compiles to roughly three steps:
load counter -> register
add 1
store register -> counter
Two threads can interleave those steps so that both load 5, both compute 6, and both store 6. One increment vanished. With a million increments per thread this happens constantly, and the total is reliably wrong.
A mutex (mutual exclusion lock) fixes this by ensuring that only one thread at a time executes the protected region. pthread_mutex_lock blocks until the lock is free, then takes it; pthread_mutex_unlock releases it. The region between them is a critical section.
Three disciplines make mutexes work:
- Protect the data, not the code. Decide which mutex guards which variables and document it. Every access to that data — reads included — must hold that mutex.
- Keep critical sections short. A lock held during I/O serializes every thread behind it and destroys the benefit of threading.
- Always unlock on every path. An early
returnor agotothat skips the unlock deadlocks the program permanently.
A mutex answers "may I touch this now?" It does not answer "has something I am waiting for happened yet?". Consider a consumer waiting for a queue to become non-empty: it must not hold the lock while waiting, or the producer can never take the lock to add an item. Repeatedly locking, checking, and unlocking works but burns CPU.
A condition variable solves exactly this. pthread_cond_wait(&cond, &mutex) performs three steps atomically: release the mutex, sleep until signalled, and re-acquire the mutex before returning. The atomicity is essential — a gap between releasing and sleeping would let a signal arrive unheard, and the thread would sleep forever.
Two rules about condition variables are non-negotiable:
Always wait in a loop, never in an if. pthread_cond_wait may return without any signal at all — a spurious wakeup, permitted by POSIX because it makes implementations faster on some platforms. And with several waiters, another thread may consume the item before you re-acquire the lock. The predicate must be re-checked after every wake:
while (!condition_is_true) /* while, NOT if */
pthread_cond_wait(&cond, &mutex);
Signal while holding the mutex, or immediately after. pthread_cond_signal wakes one waiter; pthread_cond_broadcast wakes all. Use signal when any one waiter can handle the event, and broadcast when the state change may satisfy several waiters or different predicates.
Together, a mutex plus a condition variable plus a predicate is the standard toolkit, and a bounded producer-consumer queue is its canonical use.
Examples
The race, and its fix:
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#define THREADS 4
#define PER_THREAD 200000
static long unsafe_counter = 0;
static long safe_counter = 0;
static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER; /* static init */
static void *increment(void *arg)
{
(void)arg;
for (int i = 0; i < PER_THREAD; i++) {
/* RACE: load, add, store is three steps. Two threads interleaving
them lose increments. This is undefined behavior, not merely
an unpredictable number. */
unsafe_counter++;
/* SAFE: only one thread executes the critical section at a time. */
pthread_mutex_lock(&lock);
safe_counter++; /* keep this region SHORT */
pthread_mutex_unlock(&lock);
}
return NULL;
}
int main(void)
{
pthread_t t[THREADS];
for (int i = 0; i < THREADS; i++) pthread_create(&t[i], NULL, increment, NULL);
for (int i = 0; i < THREADS; i++) pthread_join(t[i], NULL);
long expected = (long)THREADS * PER_THREAD;
printf("expected: %ld\n", expected);
printf("unsafe: %ld (%ld increments lost)\n",
unsafe_counter, expected - unsafe_counter);
printf("safe: %ld\n", safe_counter);
pthread_mutex_destroy(&lock);
return 0;
}
A bounded producer-consumer queue — the canonical condition-variable structure:
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <unistd.h>
#define CAPACITY 4
#define ITEMS 12
typedef struct {
int buf[CAPACITY];
size_t head, tail, count;
pthread_mutex_t lock;
pthread_cond_t not_empty; /* consumers wait here when count == 0 */
pthread_cond_t not_full; /* producers wait here when count == CAPACITY */
int closed; /* no more items will ever arrive */
} Queue;
static void queue_init(Queue *q)
{
q->head = q->tail = q->count = 0;
q->closed = 0;
pthread_mutex_init(&q->lock, NULL);
pthread_cond_init(&q->not_empty, NULL);
pthread_cond_init(&q->not_full, NULL);
}
static void queue_destroy(Queue *q)
{
pthread_mutex_destroy(&q->lock);
pthread_cond_destroy(&q->not_empty);
pthread_cond_destroy(&q->not_full);
}
static void queue_push(Queue *q, int value)
{
pthread_mutex_lock(&q->lock);
/* WHILE, not if: cond_wait can return spuriously, and another producer
may have refilled the queue before we re-acquired the lock. */
while (q->count == CAPACITY)
pthread_cond_wait(&q->not_full, &q->lock); /* atomically:
unlock, sleep, relock */
q->buf[q->tail] = value;
q->tail = (q->tail + 1) % CAPACITY;
q->count++;
/* Signal while still holding the lock: correct, and simplest to reason about. */
pthread_cond_signal(&q->not_empty);
pthread_mutex_unlock(&q->lock);
}
/* Returns 1 if an item was taken, 0 if the queue is closed and drained. */
static int queue_pop(Queue *q, int *out)
{
pthread_mutex_lock(&q->lock);
while (q->count == 0 && !q->closed)
pthread_cond_wait(&q->not_empty, &q->lock);
if (q->count == 0) { /* closed AND empty: nothing more */
pthread_mutex_unlock(&q->lock);
return 0;
}
*out = q->buf[q->head];
q->head = (q->head + 1) % CAPACITY;
q->count--;
pthread_cond_signal(&q->not_full); /* a producer may now proceed */
pthread_mutex_unlock(&q->lock);
return 1;
}
static void queue_close(Queue *q)
{
pthread_mutex_lock(&q->lock);
q->closed = 1;
/* BROADCAST, not signal: every waiting consumer must learn that no
more items are coming, not just one of them. */
pthread_cond_broadcast(&q->not_empty);
pthread_mutex_unlock(&q->lock);
}
static Queue queue;
static void *producer(void *arg)
{
(void)arg;
for (int i = 1; i <= ITEMS; i++) {
queue_push(&queue, i);
printf("produced %d\n", i);
}
queue_close(&queue); /* tell consumers no more are coming */
return NULL;
}
static void *consumer(void *arg)
{
long id = (long)arg;
int value;
while (queue_pop(&queue, &value)) {
printf(" consumer %ld got %d\n", id, value);
usleep(20000); /* simulate work OUTSIDE the lock */
}
printf(" consumer %ld exiting\n", id);
return NULL;
}
int main(void)
{
queue_init(&queue);
pthread_t prod, cons[3];
pthread_create(&prod, NULL, producer, NULL);
for (long i = 0; i < 3; i++) pthread_create(&cons[i], NULL, consumer, (void *)i);
pthread_join(prod, NULL);
for (int i = 0; i < 3; i++) pthread_join(cons[i], NULL);
queue_destroy(&queue);
printf("all items produced and consumed\n");
return 0;
}
WHY cond_wait MUST BE ATOMIC
IF IT WERE "unlock; then sleep" AS TWO STEPS:
consumer producer
----------------------------- -----------------------------
lock()
count == 0, so must wait
unlock() <-- lock is free HERE
lock()
push item
signal() <-- NOBODY IS WAITING YET
unlock()
sleep() <-- sleeps forever: the signal is GONE
pthread_cond_wait performs unlock + sleep as ONE indivisible step, so
the signal cannot slip through the gap. This is the entire reason the
mutex is a parameter to cond_wait.
AND WHY THE WAIT MUST BE A LOOP:
- spurious wakeups are permitted by POSIX
- another waiter may consume the item before you re-acquire the lock
- a broadcast wakes everyone, but only one can have the item
| Primitive | Answers | Blocking |
|---|---|---|
| Mutex | "may I touch this data now?" | until the lock is free |
| Condition variable | "has the state I need occurred?" | until signalled (plus spurious wakeups) |
pthread_cond_signal | wake one waiter | — |
pthread_cond_broadcast | wake every waiter | — |
| Read-write lock | many readers or one writer | until compatible |
| Semaphore | "are there resources available?" | until the count is positive |
Key Takeaways
counter++is three machine steps, so unsynchronized concurrent access loses updates and is undefined behavior, not just an unpredictable number.- A mutex protects data, not code: document which mutex guards which variables, hold it for every access including reads, and keep critical sections short.
pthread_cond_waitatomically unlocks, sleeps, and re-locks — the atomicity is what prevents a signal from being lost in the gap.- Always wait in a
whileloop re-checking the predicate, because spurious wakeups are permitted and another thread may consume the event first. - Use
signalwhen any one waiter suffices andbroadcastwhen the state change may satisfy several waiters, such as closing a queue.
🧪 Practice
- Run the racing counter with 4 threads and 200,000 increments each, note how many increments are lost, then confirm the mutex version is always exact.
- Replace the
whileinqueue_popwith anifand run under load until you observe a consumer proceeding with an empty queue.- Add a
queue_pop_timeoutusingpthread_cond_timedwaitthat gives up after 500 ms.- Interview-style: "Why does
pthread_cond_waittake a mutex as a parameter?" Hint: trace what a producer could do in the gap between unlocking and sleeping.
Race Conditions and Deadlock
Theory
Concurrency has two signature failure modes. One produces wrong answers; the other produces no answer at all.
A race condition exists when the result depends on the relative timing of threads. The subtype that matters most is the data race: two threads access the same memory location, at least one writes, and nothing orders the two accesses. C11 and POSIX both declare this undefined behavior — not "you get one of the two values", but genuinely undefined, because the compiler optimized on the assumption it could not happen.
Races are hard to find precisely because they are timing-dependent. A race that needs a specific interleaving may appear once in ten million runs, never on your laptop and reliably in production. Adding a printf to debug it changes the timing and makes it disappear. This is why ThreadSanitizer matters so much: it detects races that did not actually occur in that run, by tracking the happens-before relationships between accesses.
Not every race is a data race. Check-then-act is a race even when each individual step is protected:
if (!map_contains(m, key)) /* thread B can insert HERE */
map_insert(m, key, value); /* now we overwrite or duplicate */
The fix is to make the whole compound operation atomic — one lock spanning both steps — not to protect each step separately.
Deadlock is the opposite failure: threads block forever, each waiting for a resource another holds. Four conditions must all hold, and breaking any one prevents it:
- Mutual exclusion — a resource is held exclusively.
- Hold and wait — a thread holds one resource while requesting another.
- No preemption — resources are released only voluntarily.
- Circular wait — a cycle of threads each waiting on the next.
The practical cure is to break the fourth: impose a global lock ordering. If every thread that needs both A and B always takes A first, no cycle can form. Order by any consistent rule — address, ID, or a documented hierarchy.
DEADLOCK: a cycle forms
thread 1 thread 2
lock(A) ok lock(B) ok
lock(B) blocked -------> lock(A) blocked
^ |
+----------------------------+
both wait forever
FIXED: a global ordering makes a cycle impossible
thread 1 thread 2
lock(A) lock(A) blocked until 1 releases
lock(B) ...
work; unlock both then proceeds normally
Other alternatives: pthread_mutex_trylock with a back-off (release everything and retry), or simply using one coarser lock, which trades throughput for guaranteed safety.
Two related hazards are worth naming. Livelock is when threads keep responding to each other and make no progress — two people stepping aside in a corridor forever. Starvation is when a thread is perpetually denied a resource because others keep winning the race for it.
Self-deadlock deserves mention: locking a non-recursive mutex twice in the same thread blocks forever. It usually happens when a locked function calls another function that locks the same mutex. PTHREAD_MUTEX_RECURSIVE permits it, but the need for it usually signals confused ownership.
Examples
A deadlock, demonstrated and then fixed by ordering:
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <unistd.h>
#include <errno.h>
typedef struct {
int id;
long balance;
pthread_mutex_t lock;
} Account;
static Account acct_a = { 1, 1000, PTHREAD_MUTEX_INITIALIZER };
static Account acct_b = { 2, 1000, PTHREAD_MUTEX_INITIALIZER };
/* --- DEADLOCK-PRONE: locks in the order the arguments happen to be in --- */
static void transfer_unsafe(Account *from, Account *to, long amount)
{
pthread_mutex_lock(&from->lock);
usleep(1000); /* widen the window so it reproduces */
pthread_mutex_lock(&to->lock); /* the other thread holds THIS one */
from->balance -= amount;
to->balance += amount;
pthread_mutex_unlock(&to->lock);
pthread_mutex_unlock(&from->lock);
}
/* --- SAFE: a GLOBAL ORDERING makes a cycle impossible ------------------
Every thread takes the lower-id account's lock first, whatever the
direction of the transfer. No cycle can form, so no deadlock. */
static void transfer_safe(Account *from, Account *to, long amount)
{
Account *first = (from->id < to->id) ? from : to;
Account *second = (from->id < to->id) ? to : from;
pthread_mutex_lock(&first->lock); /* always the lower id first */
pthread_mutex_lock(&second->lock);
from->balance -= amount;
to->balance += amount;
pthread_mutex_unlock(&second->lock);
pthread_mutex_unlock(&first->lock);
}
/* --- ALTERNATIVE: trylock with back-off, when ordering is impractical --- */
static int transfer_trylock(Account *from, Account *to, long amount)
{
for (int attempt = 0; attempt < 100; attempt++) {
pthread_mutex_lock(&from->lock);
if (pthread_mutex_trylock(&to->lock) == 0) { /* got both */
from->balance -= amount;
to->balance += amount;
pthread_mutex_unlock(&to->lock);
pthread_mutex_unlock(&from->lock);
return 0;
}
/* Could not get the second lock: RELEASE THE FIRST and retry,
rather than holding it and waiting (which is "hold and wait"). */
pthread_mutex_unlock(&from->lock);
usleep(100 + attempt * 10); /* back off before retrying */
}
return -1;
}
static void *safe_a_to_b(void *arg)
{
(void)arg;
for (int i = 0; i < 200; i++) transfer_safe(&acct_a, &acct_b, 1);
return NULL;
}
static void *safe_b_to_a(void *arg)
{
(void)arg;
for (int i = 0; i < 200; i++) transfer_safe(&acct_b, &acct_a, 1);
return NULL;
}
int main(void)
{
printf("The unsafe version would deadlock: thread 1 holds A and wants B\n"
"while thread 2 holds B and wants A. Running the SAFE version.\n");
(void)transfer_unsafe; /* shown above, deliberately unused */
pthread_t t1, t2;
pthread_create(&t1, NULL, safe_a_to_b, NULL);
pthread_create(&t2, NULL, safe_b_to_a, NULL);
pthread_join(t1, NULL);
pthread_join(t2, NULL);
printf("A = %ld, B = %ld, total = %ld (conserved)\n",
acct_a.balance, acct_b.balance, acct_a.balance + acct_b.balance);
if (transfer_trylock(&acct_a, &acct_b, 50) == 0)
printf("trylock transfer succeeded: A = %ld, B = %ld\n",
acct_a.balance, acct_b.balance);
return 0;
}
Check-then-act, where locking each step is not enough:
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#define SLOTS 100
static int registry[SLOTS];
static int used = 0;
static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
/* Each step is individually protected -- and it is still a race, because
the state can change between the two locked regions. */
static int reserve_broken(int value)
{
pthread_mutex_lock(&lock);
int full = (used == SLOTS);
pthread_mutex_unlock(&lock); /* <-- another thread runs HERE */
if (full) return -1;
pthread_mutex_lock(&lock);
registry[used++] = value; /* 'used' may have changed:
this can overflow the array */
pthread_mutex_unlock(&lock);
return 0;
}
/* Correct: the CHECK and the ACT are inside ONE critical section, so no
other thread can change the state between them. */
static int reserve_correct(int value)
{
pthread_mutex_lock(&lock);
int rc = -1;
if (used < SLOTS) { /* check */
registry[used++] = value; /* and act, atomically together */
rc = 0;
}
pthread_mutex_unlock(&lock);
return rc;
}
static void *filler(void *arg)
{
long id = (long)arg;
for (int i = 0; i < 60; i++) reserve_correct((int)(id * 1000 + i));
return NULL;
}
int main(void)
{
(void)reserve_broken; /* shown for contrast */
pthread_t t[4];
for (long i = 0; i < 4; i++) pthread_create(&t[i], NULL, filler, (void *)i);
for (int i = 0; i < 4; i++) pthread_join(t[i], NULL);
printf("used = %d (capacity %d) -- never exceeded\n", used, SLOTS);
printf("240 reservations attempted, %d accepted, rest correctly refused\n", used);
return 0;
}
# ThreadSanitizer finds races that did NOT happen in this particular run,
# by tracking the ordering relationships between memory accesses.
$ gcc -fsanitize=thread -g -pthread race.c -o race && ./race
WARNING: ThreadSanitizer: data race (pid=12345)
Write of size 8 at 0x0000006010a0 by thread T2:
#0 increment race.c:15
Previous write of size 8 at 0x0000006010a0 by thread T1:
#0 increment race.c:15
# Helgrind (part of Valgrind) also detects lock-ordering problems:
$ valgrind --tool=helgrind ./deadlock_prone
| Failure | Symptom | Cause | Cure |
|---|---|---|---|
| Data race | wrong or corrupted values | unsynchronized concurrent access | mutex, or atomics |
| Check-then-act | rare invariant violation | state changes between locked steps | one lock spanning both |
| Deadlock | total freeze | circular wait on locks | global lock ordering, or trylock |
| Livelock | busy but no progress | threads endlessly reacting | randomized back-off |
| Starvation | one thread never proceeds | unfair scheduling or lock handoff | fair locks, priorities |
Key Takeaways
- A data race — two concurrent accesses, at least one a write, with no ordering — is undefined behavior, not merely an unpredictable value.
- Check-then-act is a race even when each step is individually locked; the compound operation must be inside a single critical section.
- Deadlock needs all four of mutual exclusion, hold-and-wait, no preemption, and circular wait — breaking the cycle with a global lock ordering is the standard fix.
pthread_mutex_trylockwith release-and-back-off is the alternative when a consistent ordering is impractical.- Races are timing-dependent and often invisible in testing, so use ThreadSanitizer and Helgrind rather than relying on runs that happened to pass.
🧪 Practice
- Write the two-account transfer with the unsafe ordering, run it with two threads transferring in opposite directions, and observe the deadlock. Then apply the ordering fix.
- Build a check-then-act race on a shared counter and detect it with
-fsanitize=thread.- Implement a bank transfer using
trylockwith exponential back-off and count how many retries occur under contention.- Interview-style: "Two threads deadlock intermittently in production but never in testing. How would you find the cause?" Hint: name a tool that detects the possibility of a lock cycle rather than waiting for one.
C11 Threads and Atomics
Theory
Until 2011, C had no threading in the language itself. Threads were a POSIX feature on Unix and a completely different API on Windows, and the C standard did not even define what concurrent access to a variable meant. C11 changed both.
<threads.h> provides a standard threading API — thrd_create, mtx_lock, cnd_wait — that mirrors pthreads closely. Its adoption has been poor: glibc only gained it in 2018, and MSVC still lacks it. In practice pthreads remains the portable choice on Unix, and C11 threads are worth recognizing more than adopting. Both are optional for an implementation, guarded by __STDC_NO_THREADS__.
<stdatomic.h> is the part that genuinely changed how C is written, because it introduced something the language previously could not express: an operation on shared data that cannot be interrupted or torn.
Recall that counter++ is load, add, store. An atomic type makes it a single indivisible operation that no other thread can observe halfway through:
_Atomic int counter = 0;
counter++; /* now atomic: no lock, no lost updates */
The gain over a mutex is real. An atomic increment compiles to one instruction with a lock prefix on x86 — no system call, no blocking, no context switch. For a shared counter or flag, it is several times faster than locking and unlocking.
The limits are equally important:
- Atomics protect one variable, not an invariant across several. Making
headandcountboth atomic does not make "push an item" atomic. Multi-variable invariants need a mutex. - Compound operations still need care.
if (atomic_load(&x) == 0) atomic_store(&x, 1);is a check-then-act race. The atomic version isatomic_compare_exchange, which does both in one step. - Only lock-free types are truly cheap.
atomic_is_lock_freetells you whether the compiler implements a type with real instructions or with a hidden lock; large structs get the lock.
The essential operations:
| Operation | Meaning |
|---|---|
atomic_load / atomic_store | read / write indivisibly |
atomic_fetch_add / _sub | read-modify-write, returning the old value |
atomic_exchange | store and return the previous value |
atomic_compare_exchange_strong | if the value equals expected, replace it — the primitive behind all lock-free algorithms |
atomic_flag_test_and_set | the one type guaranteed lock-free everywhere |
Compare-and-swap deserves emphasis: it is the building block of every lock-free data structure. The idiom is to read the current value, compute the new one, and swap only if nothing changed in the meantime — retrying if it did.
A crucial clarification: volatile is not _Atomic and never was. volatile prevents the compiler from caching or eliding an access, which is what memory-mapped hardware and signal handlers need. It provides no atomicity and no ordering between threads. Using volatile for thread synchronization is a longstanding, still-common bug.
Examples
#include <stdio.h>
#include <stdlib.h>
#include <stdatomic.h>
#include <stdbool.h>
#include <pthread.h>
#define THREADS 4
#define PER_THREAD 250000
static _Atomic long atomic_counter = 0;
static long plain_counter = 0;
static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
static long mutex_counter = 0;
static void *worker(void *arg)
{
(void)arg;
for (int i = 0; i < PER_THREAD; i++) {
plain_counter++; /* RACE: undefined behavior */
atomic_counter++; /* atomic: one indivisible op */
pthread_mutex_lock(&lock); /* correct, but heavier */
mutex_counter++;
pthread_mutex_unlock(&lock);
}
return NULL;
}
int main(void)
{
/* Is this type implemented with real instructions, or a hidden lock? */
printf("atomic long is lock-free: %s\n",
atomic_is_lock_free(&atomic_counter) ? "yes" : "no");
pthread_t t[THREADS];
for (int i = 0; i < THREADS; i++) pthread_create(&t[i], NULL, worker, NULL);
for (int i = 0; i < THREADS; i++) pthread_join(t[i], NULL);
long expected = (long)THREADS * PER_THREAD;
printf("expected: %ld\n", expected);
printf("plain: %ld (lost %ld)\n", plain_counter, expected - plain_counter);
printf("atomic: %ld\n", atomic_load(&atomic_counter));
printf("mutex: %ld\n", mutex_counter);
/* --- the explicit operations --------------------------------------- */
_Atomic int v = 10;
int old = atomic_fetch_add(&v, 5); /* returns the OLD value */
printf("\nfetch_add: old=%d new=%d\n", old, atomic_load(&v));
old = atomic_exchange(&v, 99); /* store, return the previous */
printf("exchange: old=%d new=%d\n", old, atomic_load(&v));
/* --- compare-and-swap: the primitive behind lock-free algorithms ---- */
int expected_val = 99;
bool swapped = atomic_compare_exchange_strong(&v, &expected_val, 123);
printf("CAS(99 -> 123): %s, v = %d\n", swapped ? "succeeded" : "failed",
atomic_load(&v));
expected_val = 99; /* v is 123 now, so this fails */
swapped = atomic_compare_exchange_strong(&v, &expected_val, 456);
/* On failure, expected_val is UPDATED to the actual current value --
which is what makes the retry loop below work. */
printf("CAS(99 -> 456): %s, expected updated to %d\n",
swapped ? "succeeded" : "failed", expected_val);
pthread_mutex_destroy(&lock);
return 0;
}
A lock-free maximum, showing the compare-and-swap retry loop:
#include <stdio.h>
#include <stdlib.h>
#include <stdatomic.h>
#include <stdbool.h>
#include <pthread.h>
static _Atomic long high_water = 0;
/* Atomically raise high_water to 'candidate' if it is larger. There is no
single "atomic max" instruction, so this is built from compare-and-swap:
read, decide, swap-if-unchanged, and retry if another thread won. */
static void record_max(long candidate)
{
long current = atomic_load(&high_water);
while (candidate > current) {
/* If high_water is still 'current', set it to 'candidate'.
If not, compare_exchange writes the NEW actual value into
'current' and returns false, so the loop re-evaluates. */
if (atomic_compare_exchange_weak(&high_water, ¤t, candidate))
return; /* we won the race */
/* 'current' now holds the up-to-date value -- loop and try again. */
}
}
static void *producer(void *arg)
{
long seed = (long)arg;
for (long i = 0; i < 100000; i++)
record_max((seed * 7919 + i * 31) % 1000003);
return NULL;
}
int main(void)
{
pthread_t t[4];
for (long i = 1; i <= 4; i++) pthread_create(&t[i-1], NULL, producer, (void *)i);
for (int i = 0; i < 4; i++) pthread_join(t[i], NULL);
printf("lock-free maximum found: %ld\n", atomic_load(&high_water));
/* atomic_flag is the ONLY type guaranteed lock-free on every platform.
It is the primitive from which a spinlock is built. */
atomic_flag spin = ATOMIC_FLAG_INIT;
/* Acquire: spin until we are the one that flipped it from clear to set. */
while (atomic_flag_test_and_set(&spin)) { /* spin */ }
printf("spinlock acquired\n");
atomic_flag_clear(&spin); /* release */
printf("spinlock released\n");
return 0;
}
C11 threads, for recognition:
#include <stdio.h>
#include <stdlib.h>
/* Implementations may omit threads entirely; this is the required guard. */
#ifdef __STDC_NO_THREADS__
int main(void)
{
printf("this implementation does not provide <threads.h>\n");
printf("use POSIX pthreads instead -- far more widely available\n");
return 0;
}
#else
#include <threads.h>
static int worker(void *arg) /* note: returns int, not void* */
{
int id = *(int *)arg;
printf("C11 thread %d running\n", id);
return id * 10; /* an ordinary int result */
}
int main(void)
{
thrd_t t[3];
int ids[3] = { 1, 2, 3 };
for (int i = 0; i < 3; i++)
/* thrd_success, not 0, is the success value -- another difference. */
if (thrd_create(&t[i], worker, &ids[i]) != thrd_success) {
fprintf(stderr, "thrd_create failed\n");
return EXIT_FAILURE;
}
for (int i = 0; i < 3; i++) {
int result;
thrd_join(t[i], &result); /* the result is an int, not void** */
printf("thread %d returned %d\n", ids[i], result);
}
return 0;
}
#endif
| Plain variable | volatile | _Atomic | Mutex | |
|---|---|---|---|---|
| Safe for concurrent access | no | no | yes | yes |
| Prevents compiler caching | no | yes | yes | yes |
| Indivisible read-modify-write | no | no | yes | yes |
| Orders other memory accesses | no | no | yes (per memory order) | yes |
| Protects a multi-variable invariant | no | no | no | yes |
| Cost | free | free | one instruction | syscall when contended |
| Correct use | single-threaded | hardware registers, signal flags | counters, flags, lock-free code | anything with an invariant |
Key Takeaways
- C11 added
<threads.h>and<stdatomic.h>; the threads API is poorly adopted, but atomics genuinely changed what C can express. - An atomic operation is indivisible and needs no lock, making it several times cheaper than a mutex for a single counter or flag.
- Atomics protect one variable — an invariant spanning several variables still requires a mutex, and compound check-then-act still needs compare-and-swap.
atomic_compare_exchangeis the primitive behind every lock-free algorithm, and it updates your expected value on failure so a retry loop converges.volatileis not a synchronization primitive: it stops compiler caching but provides no atomicity and no inter-thread ordering.
🧪 Practice
- Benchmark 10 million increments of a shared counter using a plain variable, an atomic, and a mutex, and explain the ranking and the plain version's wrong answer.
- Implement a lock-free stack push using
atomic_compare_exchange_weakin a retry loop.- Use
atomic_flagto build a spinlock, then measure how it compares with a mutex under low and high contention.- Interview-style: "Is
volatile int counterenough to makecounter++thread-safe?" Hint: separate "the compiler must re-read it" from "no other thread can interleave".
Memory Ordering Basics
Theory
Here is the fact that makes concurrent programming genuinely hard: neither the compiler nor the CPU executes your program in the order you wrote it.
Both reorder aggressively, and both are permitted to. The compiler hoists loads out of loops and reorders independent stores. The CPU executes instructions out of order, buffers stores, and speculates past branches. Every one of these transformations preserves the behavior of a single-threaded program — that is the only guarantee either one makes.
Another thread watching your memory can therefore see your writes happen in a different order than your source lists them. This is not a bug in the hardware; it is the contract.
The classic demonstration:
/* Thread A Thread B */
data = 42; while (!ready) { }
ready = 1; printf("%d\n", data); /* may print 0 */
Thread A writes data then ready. Thread B waits for ready and reads data. Intuitively data must be 42 — but nothing orders those two writes, so B can observe ready == 1 while data is still 0. On x86 this is rare; on ARM and POWER, with weaker memory models, it happens routinely.
Memory ordering is how you tell the compiler and CPU which reorderings are forbidden. C11 provides these levels, from weakest to strongest:
| Order | Guarantee | Typical use |
|---|---|---|
memory_order_relaxed | atomicity only, no ordering | statistics counters |
memory_order_acquire | no later access moves before this load | acquiring a lock, reading a flag |
memory_order_release | no earlier access moves after this store | releasing a lock, setting a flag |
memory_order_acq_rel | both, for read-modify-write | compare-and-swap in a lock |
memory_order_seq_cst | one total order all threads agree on | the default; use unless profiling says otherwise |
The workhorse is the acquire/release pair, and the intuition is a one-way barrier:
- A release store is a publish: everything written before it is guaranteed visible to anyone who acquires.
- An acquire load is a subscribe: everything the releasing thread wrote before its release becomes visible to you.
Together they create a happens-before relationship between the two threads. Correcting the example above is exactly this:
atomic_store_explicit(&ready, 1, memory_order_release); /* publish */
while (!atomic_load_explicit(&ready, memory_order_acquire)) {} /* subscribe */
Now data = 42 cannot be reordered after the release, and the read of data cannot be hoisted before the acquire.
memory_order_seq_cst is the default because it is the easiest to reason about: all sequentially consistent operations appear in one global order every thread agrees on. It is also the most expensive, requiring full barriers on some architectures. Start with the default. Weaker orderings buy a few percent and cost enormous reasoning effort; they belong in hot paths of libraries written by people who benchmark them.
Two practical notes. relaxed is genuinely useful for counters where you only need the final total and no ordering with other data. And x86 is a strong memory model that gives acquire/release semantics for free on ordinary loads and stores, which is exactly why code that is subtly wrong on ordering can pass every test on a laptop and fail on an ARM server.
Examples
#include <stdio.h>
#include <stdlib.h>
#include <stdatomic.h>
#include <pthread.h>
#include <unistd.h>
/* --- the message-passing pattern, done correctly --------------------- */
static int payload = 0; /* ordinary, non-atomic data */
static _Atomic int ready = 0; /* the flag that publishes it */
static void *producer(void *arg)
{
(void)arg;
/* These plain writes must become visible BEFORE the flag. */
payload = 42;
/* RELEASE: a publish barrier. Nothing written above may be reordered
below this store. Any thread that ACQUIRES this flag sees payload. */
atomic_store_explicit(&ready, 1, memory_order_release);
return NULL;
}
static void *consumer(void *arg)
{
(void)arg;
/* ACQUIRE: a subscribe barrier. Nothing below may be hoisted above it,
and everything the producer wrote before its release is now visible. */
while (atomic_load_explicit(&ready, memory_order_acquire) == 0)
; /* spin until published */
/* Guaranteed 42. Without the acquire/release pair this could read 0,
and on ARM or POWER it actually would. */
printf("consumer sees payload = %d\n", payload);
return NULL;
}
int main(void)
{
pthread_t p, c;
pthread_create(&c, NULL, consumer, NULL);
usleep(1000);
pthread_create(&p, NULL, producer, NULL);
pthread_join(p, NULL);
pthread_join(c, NULL);
/* --- relaxed: atomicity WITHOUT ordering --------------------------- */
_Atomic long hits = 0;
/* Correct for a pure statistics counter: we need every increment to
count, but we do not care when other threads observe it, and it
orders nothing else. This is the cheapest atomic operation. */
atomic_fetch_add_explicit(&hits, 1, memory_order_relaxed);
printf("relaxed counter = %ld\n",
atomic_load_explicit(&hits, memory_order_relaxed));
/* --- seq_cst: the default, and the one to start with -------------- */
_Atomic int x = 0;
atomic_store(&x, 7); /* implicitly seq_cst */
printf("seq_cst load = %d\n", atomic_load(&x));
/* An explicit standalone barrier, rarely needed directly. */
atomic_thread_fence(memory_order_seq_cst);
return 0;
}
A spinlock built from acquire/release — the pattern in miniature:
#include <stdio.h>
#include <stdatomic.h>
#include <stdbool.h>
#include <pthread.h>
typedef struct { _Atomic bool held; } SpinLock;
static void spin_lock(SpinLock *l)
{
for (;;) {
bool expected = false;
/* ACQUIRE on success: everything the previous holder wrote before
its release becomes visible to us once we take the lock. */
if (atomic_compare_exchange_weak_explicit(
&l->held, &expected, true,
memory_order_acquire, /* ordering on success */
memory_order_relaxed)) /* ordering on failure: none needed */
return;
expected = false; /* CAS overwrote it; reset */
}
}
static void spin_unlock(SpinLock *l)
{
/* RELEASE: everything we wrote inside the critical section is
guaranteed visible to the next thread that acquires this lock. */
atomic_store_explicit(&l->held, false, memory_order_release);
}
static SpinLock lock = { false };
static long shared_total = 0; /* plain: the lock orders it */
static void *worker(void *arg)
{
(void)arg;
for (int i = 0; i < 100000; i++) {
spin_lock(&lock);
shared_total++; /* protected by acquire/release */
spin_unlock(&lock);
}
return NULL;
}
int main(void)
{
pthread_t t[4];
for (int i = 0; i < 4; i++) pthread_create(&t[i], NULL, worker, NULL);
for (int i = 0; i < 4; i++) pthread_join(t[i], NULL);
printf("total = %ld (expected 400000)\n", shared_total);
return 0;
}
WHAT REORDERING LOOKS LIKE
YOU WROTE THE CPU MAY EXECUTE
----------------------- -----------------------
data = 42; ready = 1; <-- store buffer drained first
ready = 1; data = 42;
Both orders are legal for a SINGLE thread: nothing in that thread can
tell the difference. Another thread absolutely can.
ACQUIRE / RELEASE ARE ONE-WAY BARRIERS
thread A thread B
data = 42; | | read ready (ACQUIRE)
| cannot sink below | ------------------------
store ready v (RELEASE) v cannot rise above
--------------------- read data -> 42
Everything before the release is visible to everything after the
acquire. That relationship is called HAPPENS-BEFORE.
x86 gives acquire/release almost for free on ordinary loads and stores.
ARM and POWER do NOT -- which is why ordering bugs pass on a laptop
and fail on a server.
Key Takeaways
- Compilers and CPUs reorder memory operations freely, preserving only single-threaded behavior — another thread can observe your writes in a different order.
- Acquire and release are one-way barriers forming a publish/subscribe pair: a release store makes prior writes visible to any thread that acquires the same variable.
memory_order_seq_cstis the default and the easiest to reason about; use it unless profiling proves a weaker ordering is needed.memory_order_relaxedgives atomicity with no ordering, which is correct for pure statistics counters and wrong for anything that publishes data.- x86's strong memory model hides many ordering bugs, so code that passes there can still fail on ARM or POWER — reason from the standard, not from your laptop.
🧪 Practice
- Write the flag-and-payload example with plain non-atomic variables and run it in a loop; then add acquire/release and explain what changed.
- Benchmark
memory_order_relaxedagainstmemory_order_seq_cstfor 10 million atomic increments and report the difference.- Implement the spinlock with
memory_order_relaxedeverywhere and explain, referring to happens-before, why the protected data is no longer safe.- Interview-style: "Why might correct-looking threaded code work on x86 but fail on ARM?" Hint: compare how much reordering each architecture permits without explicit barriers.
Thread-Local Storage
Theory
Threads share globals, which is usually what you want and sometimes exactly what you do not. Consider a global errno: if all threads shared one, a failed call in one thread would overwrite the error another thread was about to read. The same applies to a per-thread scratch buffer, a random-number generator state, or a cached connection.
Thread-local storage gives each thread its own private instance of a variable that is nonetheless declared and used like a global. Same name, same scope, different storage per thread. errno is in fact implemented exactly this way on every modern system — which is why it works correctly in threaded programs despite looking like a global.
C11 provides the _Thread_local storage-class specifier, with thread_local as a friendlier spelling from <threads.h>:
_Thread_local int counter = 0; /* one 'counter' per thread */
Each thread gets its own copy, initialized when the thread starts. There is no synchronization needed, because there is no sharing — which makes thread-local variables both fast and immune to data races.
The initializer must be a constant expression, since it is applied at thread creation without running arbitrary code. For anything needing dynamic setup, use the POSIX API — pthread_key_create, pthread_setspecific, pthread_getspecific — which is clumsier but adds a capability _Thread_local lacks: a destructor called automatically when each thread exits. That is the correct way to manage per-thread heap allocations, because otherwise every thread leaks its buffer.
Thread-local storage is the standard cure for the family of non-reentrant library functions that return a pointer to a static buffer: strtok, asctime, localtime, inet_ntoa. In a threaded program, two threads calling localtime corrupt each other's result. The fixes, in order of preference, are the _r variants that write into a caller-supplied buffer (localtime_r, strtok_r), then thread-local storage, then a mutex.
Two caveats are worth knowing. Thread-local access is slightly slower than a plain global — typically an extra indirection through a per-thread base register — so it is not free in a hot loop. And a _Thread_local variable in a dynamically loaded library can add setup cost per thread.
The design lesson generalizes: when you find yourself protecting a global with a mutex purely because each thread wants its own value, the mutex is the wrong tool. Give each thread its own copy and the contention disappears entirely.
Examples
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <pthread.h>
#include <unistd.h>
#define NTHREADS 3
/* SHARED: one instance for the whole process. Every thread sees the same
variable, so unsynchronized updates race. */
static int shared_counter = 0;
/* THREAD-LOCAL: one instance PER THREAD. Same name, private storage.
The initializer must be a constant expression. */
static _Thread_local int local_counter = 0;
static _Thread_local char scratch[64];
static void *worker(void *arg)
{
long id = (long)arg;
for (int i = 0; i < 1000; i++) {
shared_counter++; /* RACE: all threads hit the same memory */
local_counter++; /* safe: each thread has its own -- no lock */
}
/* A private scratch buffer needs no synchronization at all. */
snprintf(scratch, sizeof scratch, "thread %ld private buffer", id);
printf("thread %ld: local_counter=%d scratch=\"%s\"\n",
id, local_counter, scratch);
return NULL;
}
int main(void)
{
pthread_t t[NTHREADS];
for (long i = 0; i < NTHREADS; i++) pthread_create(&t[i], NULL, worker, (void *)i);
for (int i = 0; i < NTHREADS; i++) pthread_join(t[i], NULL);
printf("\nshared_counter = %d (expected %d, but races lost some)\n",
shared_counter, NTHREADS * 1000);
printf("main's local_counter = %d (main has its OWN copy, never touched)\n",
local_counter);
return 0;
}
Dynamic per-thread data with a destructor, which _Thread_local cannot provide:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <pthread.h>
static pthread_key_t buffer_key;
static pthread_once_t key_once = PTHREAD_ONCE_INIT;
/* Called automatically when a thread that set this key EXITS. This is the
capability _Thread_local lacks, and the reason to use the POSIX API. */
static void free_buffer(void *buf)
{
printf(" destructor freeing buffer at %p\n", buf);
free(buf);
}
/* pthread_once guarantees this runs exactly once across all threads,
however many race to call it. */
static void make_key(void)
{
if (pthread_key_create(&buffer_key, free_buffer) != 0) {
fprintf(stderr, "pthread_key_create failed\n");
exit(EXIT_FAILURE);
}
}
/* Lazily allocate a per-thread buffer on first use. */
static char *get_buffer(void)
{
pthread_once(&key_once, make_key);
char *buf = pthread_getspecific(buffer_key);
if (buf == NULL) { /* first call in THIS thread */
buf = malloc(128);
if (!buf) return NULL;
pthread_setspecific(buffer_key, buf); /* the destructor frees it */
}
return buf;
}
static void *worker(void *arg)
{
long id = (long)arg;
char *buf = get_buffer();
if (!buf) return NULL;
snprintf(buf, 128, "data owned solely by thread %ld", id);
printf("thread %ld: buffer=%p \"%s\"\n", id, (void *)buf, buf);
/* A second call returns the SAME buffer for this thread. */
printf("thread %ld: second call returns %s pointer\n",
id, get_buffer() == buf ? "the same" : "a DIFFERENT");
return NULL; /* destructor runs here */
}
int main(void)
{
pthread_t t[3];
for (long i = 0; i < 3; i++) pthread_create(&t[i], NULL, worker, (void *)i);
for (int i = 0; i < 3; i++) pthread_join(t[i], NULL);
printf("all per-thread buffers freed automatically\n");
pthread_key_delete(buffer_key);
return 0;
}
The non-reentrant function problem, and the three fixes:
#include <stdio.h>
#include <string.h>
#include <time.h>
#include <pthread.h>
/* strtok keeps hidden static state, so two threads tokenizing at once
corrupt each other. strtok_r takes the state as a parameter instead. */
static void *tokenize(void *arg)
{
char input[64];
char *saveptr; /* THIS thread's parser state */
snprintf(input, sizeof input, "%s", (const char *)arg);
/* strtok_r: reentrant, because the state is ours, not global. */
for (char *tok = strtok_r(input, ",", &saveptr); tok;
tok = strtok_r(NULL, ",", &saveptr))
printf(" token: %s\n", tok);
/* localtime returns a pointer to a SHARED static struct tm.
localtime_r writes into a buffer we supply. */
time_t now = time(NULL);
struct tm tm_buf;
char text[64];
localtime_r(&now, &tm_buf); /* the _r form: no shared buffer */
strftime(text, sizeof text, "%H:%M:%S", &tm_buf);
printf(" time: %s\n", text);
return NULL;
}
int main(void)
{
pthread_t a, b;
pthread_create(&a, NULL, tokenize, (void *)"one,two,three");
pthread_create(&b, NULL, tokenize, (void *)"alpha,beta,gamma");
pthread_join(a, NULL);
pthread_join(b, NULL);
printf("both threads tokenized independently -- no interference\n");
return 0;
}
| Approach | Per-thread | Destructor on exit | Dynamic init | Cost |
|---|---|---|---|---|
| Plain global | no — shared | n/a | yes | free, needs locking |
| Global + mutex | no — shared | n/a | yes | lock per access |
_Thread_local | yes | no | no — constant only | one indirection |
pthread_key_* | yes | yes | yes | function call |
Caller-supplied buffer (_r) | yes, by construction | n/a | yes | free |
Key Takeaways
- Thread-local storage gives each thread a private instance of a variable that is written like a global, eliminating both the race and the need for a lock.
_Thread_localis the C11 specifier, but its initializer must be a constant expression and it provides no cleanup when a thread exits.pthread_key_createwith a destructor is the way to manage per-thread heap allocations, paired withpthread_onceto create the key exactly once.- Functions returning pointers to static buffers —
strtok,localtime,inet_ntoa— are unsafe in threaded programs; prefer their_rvariants. - If you are locking a global only because each thread wants its own value, thread-local storage removes the contention entirely.
🧪 Practice
- Declare a
_Thread_localcounter and a shared counter, increment both from four threads, and compare the final values.- Use
pthread_key_createwith a destructor to give each thread a lazily allocated buffer, and confirm with valgrind that nothing leaks.- Write two threads that both call
strtokon separate strings and observe the corruption, then fix it withstrtok_r.- Interview-style: "How can
errnobe a global and still work correctly with threads?" Hint: consider what the macro actually expands to on a modern system.