9. Network Programming

Every protocol in the previous chapters is, from a programmer's point of view, bytes moving through a file descriptor. This chapter is about the interface that makes that possible — the socket API, which has survived essentially unchanged since 1983 — and about the two decisions that determine whether a network program works at scale: how it handles many connections at once, and how it structures the bytes it sends. The recurring lesson is that the network will violate every convenience assumption you bring from local programming: writes are partial, reads arrive split, connections fail silently, and nothing is atomic.

<a id="91-the-socket-api"></a>

9.1 The Socket API

The socket API is the boundary between your program and the network stack: this subchapter covers its data structures, its call sequences, and the details that most often produce bugs.

Socket Types: Stream, Datagram, Raw

Theory

A socket is an endpoint for communication — the handle your program holds onto a conversation. On Unix it is deliberately made to look like a file descriptor, so that read, write, close, and select all work on it. That design decision is why network code composes with ordinary I/O code at all.

When you create a socket you must state what kind of communication you want, and the choice determines which transport protocol from Chapter 7 you get:

SOCK_STREAM gives you TCP: a reliable, ordered, connection-oriented byte stream. The critical word is stream. There are no message boundaries. If you send 100 bytes and then 50 bytes, the receiver may read 150 bytes at once, or 30 then 120, or 1 byte at a time. The stream is a pipe of bytes, and any structure within it is entirely your responsibility — this is the framing problem covered in 9.3, and it is the single most common source of bugs in new network code.

SOCK_DGRAM gives you UDP: unreliable, unordered, connectionless datagrams. Here message boundaries are preserved — one sendto produces exactly one recvfrom of exactly that size, or nothing at all. You trade reliability for the boundary guarantee and for lower latency.

SOCK_RAW bypasses the transport layer entirely and lets you construct IP packets, or even Ethernet frames, yourself. This is how ping sends ICMP and how traceroute manipulates TTLs. It requires elevated privileges, because a process that can forge arbitrary packets can impersonate any host.

The comparison that matters:

	SOCK_STREAM (TCP)	SOCK_DGRAM (UDP)	SOCK_RAW
Reliability	Guaranteed	None	None
Ordering	Guaranteed	None	None
Message boundaries	None — a byte stream	Preserved	Preserved
Connection	Required	Optional	None
Max single send	Unlimited (stream)	~64 KB, practically ~1472 B	MTU
Privileges	Normal	Normal	Root / CAP_NET_RAW
Typical use	HTTP, SSH, databases	DNS, VoIP, games	ping, traceroute, sniffers

The analogy that holds up: TCP is a telephone call — you establish it, speak continuously, and the words arrive in order; but the listener cannot tell where your sentences ended unless you signal it. UDP is postcards — each arrives whole or not at all, possibly out of order, possibly never, and you learn nothing about which.

One detail worth internalising early: the "maximum UDP datagram" is 65,507 bytes, but sending anything above the path MTU (typically 1500 bytes, leaving ~1472 for payload) forces IP fragmentation, and losing any one fragment discards the entire datagram. Practical UDP protocols therefore keep messages small.

Examples

The same 150 bytes, sent as two writes

 SOCK_STREAM (TCP) -- boundaries are DESTROYED
   sender:   write(100 bytes)   write(50 bytes)
   wire:     [................ 150 bytes of stream ................]
   receiver: read() may return ANY of:
                150                  <- both at once (coalesced)
                100, 50              <- coincidentally aligned
                30, 120              <- split differently
                1, 1, 1, ... x150    <- pathological but legal
   -> YOU must impose framing. The kernel will not.

 SOCK_DGRAM (UDP) -- boundaries are PRESERVED
   sender:   sendto(100 bytes)  sendto(50 bytes)
   wire:     [100-byte datagram] [50-byte datagram]
   receiver: recvfrom() returns EXACTLY 100, then EXACTLY 50
             (or one of them never arrives at all, or they swap order)


 WHERE EACH SOCKET TYPE ENTERS THE STACK

   your program
        |  SOCK_STREAM        SOCK_DGRAM         SOCK_RAW
        v       |                  |                 |
   +---------+  |                  |                 |
   |   TCP   |<-+                  |                 |
   +---------+                     |                 |
   +---------+                     |                 |
   |   UDP   |<--------------------+                 |
   +---------+                                       |
   +---------+                                       |
   |    IP   |<--------------------------------------+
   +---------+
   +---------+
   | Ethernet|  <- SOCK_RAW with AF_PACKET gets here (Linux)
   +---------+

#include <sys/socket.h>
#include <netinet/in.h>
#include <stdio.h>

int main(void) {
    /* socket(domain, type, protocol)
     * domain   = address family: AF_INET (IPv4), AF_INET6, AF_UNIX
     * type     = semantics: SOCK_STREAM, SOCK_DGRAM, SOCK_RAW
     * protocol = 0 lets the kernel pick the default for (domain, type) */

    int tcp = socket(AF_INET, SOCK_STREAM, 0);   /* -> IPPROTO_TCP */
    int udp = socket(AF_INET, SOCK_DGRAM,  0);   /* -> IPPROTO_UDP */

    /* Raw sockets need an explicit protocol AND elevated privilege. */
    int raw = socket(AF_INET, SOCK_RAW, IPPROTO_ICMP);

    printf("tcp fd=%d  udp fd=%d  raw fd=%d\n", tcp, udp, raw);
    /* tcp fd=3  udp fd=4  raw fd=-1      <- raw fails as an ordinary user
     * A socket IS a file descriptor: numbered in the same table as open files. */
    return 0;
}

import socket

# The same three types, with Python's names for them.
tcp = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
udp = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

print(f"tcp fileno={tcp.fileno()}  proto={tcp.proto}")
print(f"udp fileno={udp.fileno()}  proto={udp.proto}")
# tcp fileno=3  proto=0
# udp fileno=4  proto=0

# Demonstrate the boundary difference concretely, over loopback.
import threading

def stream_demo():
    srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    srv.bind(("127.0.0.1", 0)); srv.listen(1)
    port = srv.getsockname()[1]

    def client():
        c = socket.create_connection(("127.0.0.1", port))
        c.sendall(b"A" * 100)        # two separate writes...
        c.sendall(b"B" * 50)
        c.close()

    threading.Thread(target=client, daemon=True).start()
    conn, _ = srv.accept()
    chunks = []
    while (data := conn.recv(4096)):
        chunks.append(len(data))
    return chunks

def dgram_demo():
    srv = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    srv.bind(("127.0.0.1", 0))
    port = srv.getsockname()[1]
    c = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    c.sendto(b"A" * 100, ("127.0.0.1", port))   # two separate datagrams
    c.sendto(b"B" * 50,  ("127.0.0.1", port))
    sizes = [len(srv.recv(4096)), len(srv.recv(4096))]
    return sizes

print(f"\nTCP  reads: {stream_demo()}")
print(f"UDP  reads: {dgram_demo()}")
# TCP  reads: [150]          <- the two writes were COALESCED into one read
# UDP  reads: [100, 50]      <- boundaries preserved exactly
#   The TCP result is not deterministic: it depends on timing and Nagle.
#   That non-determinism is precisely why you cannot rely on it.

# The practical UDP size limit is the path MTU, not the protocol maximum.
def udp_limits(mtu=1500):
    return {"protocol maximum": 65507,
            "without IP fragmentation (IPv4)": mtu - 20 - 8,   # IP hdr + UDP hdr
            "safe across the internet": 508}                   # minimum guaranteed
for k, v in udp_limits().items():
    print(f"  {k:<34} {v:>6} bytes")
#   protocol maximum                    65507 bytes
#   without IP fragmentation (IPv4)      1472 bytes
#   safe across the internet              508 bytes
#   Above the MTU, losing ONE fragment discards the WHOLE datagram.

# Every socket is a file descriptor -- see them in the process table:
$ ls -l /proc/$(pgrep -n nginx)/fd | grep socket | head -3
# lrwx------ 1 root root 64 Sep 23 09:14 6 -> socket:[418223]
# lrwx------ 1 root root 64 Sep 23 09:14 7 -> socket:[418224]

# Distinguish stream from datagram sockets at the OS level:
$ ss -tlnp | head -3          # -t = TCP (stream)
# State  Recv-Q Send-Q Local Address:Port  Peer Address:Port
# LISTEN 0      511          0.0.0.0:443        0.0.0.0:*
$ ss -ulnp | head -3          # -u = UDP (datagram)
# State  Recv-Q Send-Q Local Address:Port  Peer Address:Port
# UNCONN 0      0            0.0.0.0:53         0.0.0.0:*
#        ^^^^^^ UNCONN: datagram sockets have no connection state

# Raw sockets require privilege -- this is why ping is setuid or has caps:
$ getcap $(which ping)
# /usr/bin/ping cap_net_raw=ep
$ python3 -c "import socket; socket.socket(socket.AF_INET, socket.SOCK_RAW, 1)"
# PermissionError: [Errno 1] Operation not permitted

# Watch TCP coalescing happen (two writes, one segment):
$ sudo tcpdump -i lo -n 'port 9999' -c 2
# IP 127.0.0.1.44122 > 127.0.0.1.9999: Flags [P.], length 150
#                                                      ^^^ 100+50 merged

Key takeaways

• A socket is a file descriptor, which is why network code composes with ordinary read/write/close/select code.
• SOCK_STREAM (TCP) is a byte stream with no message boundaries — the single most common source of bugs in new network code.
• SOCK_DGRAM (UDP) preserves message boundaries: one sendto yields exactly one recvfrom, or nothing.
• SOCK_RAW bypasses the transport layer, requires root or CAP_NET_RAW, and is how ping and traceroute work.
• UDP's protocol maximum is ~64 KB, but staying under the path MTU (~1472 bytes) avoids fragmentation, where one lost fragment discards the whole datagram.
• TCP coalescing is non-deterministic — code that happens to work in testing because reads align will fail in production.

🧪 Practice

1. A sender issues write(60) then write(40) on a TCP socket. List three different sequences of read() return values the receiver might legitimately observe.
2. State which socket type you would choose for a DNS resolver, a file upload, and a traceroute implementation, with one reason each.
3. Explain why sending a 4000-byte UDP datagram across the internet is riskier than sending three 1400-byte ones.
4. Interview-style: Your protocol works perfectly on loopback and corrupts messages over a real network. (Hint: what did loopback's timing hide about how your reads lined up with the sender's writes?)

Address Families and Structures

Theory

Before a socket can connect or bind, it needs an address — and this is where the API shows its age most visibly. The socket API was designed to be protocol-independent in 1983, before anyone knew which protocols would matter, so it uses a form of manual polymorphism in C that modern programmers find startling.

An address family identifies the kind of address:

Family	Address form	Use
AF_INET	32-bit IPv4 + 16-bit port	IPv4
AF_INET6	128-bit IPv6 + port + flow + scope	IPv6
AF_UNIX	Filesystem path	Local IPC on one machine
AF_PACKET	Link-layer	Raw frames (Linux)

Each family has its own structure: sockaddr_in for IPv4, sockaddr_in6 for IPv6, sockaddr_un for Unix sockets. But the API functions (bind, connect, accept) accept a single type, struct sockaddr *. So every call casts a specific structure to the generic one and passes its length separately — which is how C expressed polymorphism before anyone had a better idea.

The important consequence: struct sockaddr is never used directly. It exists only as a cast target. And because sockaddr_in6 is larger than sockaddr, code that needs to hold an address of unknown family uses struct sockaddr_storage, which is defined to be large enough and correctly aligned for any family.

Modern code should avoid filling these structures by hand. getaddrinfo() is the correct interface, and it solves several problems at once:

• It performs DNS resolution, returning addresses for a hostname.
• It is protocol-independent: ask for a name and it returns IPv4 and IPv6 results, so the same code handles both.
• It handles service names: "https" resolves to port 443 without a hardcoded table.
• It returns a linked list of candidates to try in order, which is what makes graceful IPv6-then-IPv4 fallback possible.

The older functions — gethostbyname(), inet_addr() — are IPv4-only, not thread-safe, and deprecated. Using them is the most reliable way to write code that cannot speak IPv6.

The two conversion helpers worth knowing are inet_pton() (presentation to network: "192.0.2.1" to binary) and inet_ntop() (network to presentation, the reverse). They replaced inet_addr() and inet_ntoa(), which could not express IPv6 and had ambiguous error returns.

Examples

The cast-based polymorphism the API is built on

   struct sockaddr_in            struct sockaddr_in6        struct sockaddr_un
   +------------------+          +--------------------+     +-----------------+
   | sin_family  (2)  |          | sin6_family   (2)  |     | sun_family (2)  |
   | sin_port    (2)  |          | sin6_port     (2)  |     | sun_path  (108) |
   | sin_addr    (4)  |          | sin6_flowinfo (4)  |     |                 |
   | sin_zero    (8)  |          | sin6_addr    (16)  |     |                 |
   +------------------+          | sin6_scope_id (4)  |     +-----------------+
        16 bytes                 +--------------------+          110 bytes
                                       28 bytes
            \                          |                        /
             \                         |                       /
              +--- cast to (struct sockaddr *) + pass length --+
                                       |
                                       v
                        bind() / connect() / accept()

   The FIRST FIELD of every one is the family, so the kernel reads that
   to learn how to interpret the rest. That is the entire trick.

   struct sockaddr_storage  <- big enough and aligned for ANY of them;
                               use this when the family is not known yet.


 getaddrinfo() returns a LIST to try in order

   getaddrinfo("example.com", "https", &hints, &res)
        |
        v
   [AF_INET6  2606:2800:220::1946  :443] --> try first
        |  fails (no IPv6 route)
        v
   [AF_INET   93.184.216.34        :443] --> then this
        |  succeeds
        v
   connected
   ^ this is why hardcoding AF_INET is how programs become IPv4-only

#include <sys/socket.h>
#include <netdb.h>
#include <arpa/inet.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>

int main(void) {
    struct addrinfo hints, *res, *p;

    memset(&hints, 0, sizeof hints);        /* ALWAYS zero it first */
    hints.ai_family   = AF_UNSPEC;          /* AF_UNSPEC = IPv4 OR IPv6.
                                             * Hardcoding AF_INET here is how
                                             * programs become IPv6-incapable. */
    hints.ai_socktype = SOCK_STREAM;

    /* "https" is resolved to 443 from /etc/services -- no hardcoded port. */
    int err = getaddrinfo("example.com", "https", &hints, &res);
    if (err) { fprintf(stderr, "%s\n", gai_strerror(err)); return 1; }

    /* Walk the candidate list; try each until one connects. */
    for (p = res; p != NULL; p = p->ai_next) {
        char host[INET6_ADDRSTRLEN];
        void *addr;

        if (p->ai_family == AF_INET) {
            /* THE CAST: generic sockaddr back to the family-specific struct */
            struct sockaddr_in *v4 = (struct sockaddr_in *)p->ai_addr;
            addr = &v4->sin_addr;
        } else {
            struct sockaddr_in6 *v6 = (struct sockaddr_in6 *)p->ai_addr;
            addr = &v6->sin6_addr;
        }

        /* inet_ntop: binary -> text. Works for BOTH families, unlike inet_ntoa. */
        inet_ntop(p->ai_family, addr, host, sizeof host);
        printf("%-6s %s\n", p->ai_family == AF_INET ? "IPv4" : "IPv6", host);

        int fd = socket(p->ai_family, p->ai_socktype, p->ai_protocol);
        if (fd < 0) continue;
        if (connect(fd, p->ai_addr, p->ai_addrlen) == 0) {
            printf("  connected using this one\n");
            close(fd);
            break;
        }
        close(fd);                          /* this candidate failed; try next */
    }

    freeaddrinfo(res);                      /* getaddrinfo ALLOCATES: free it */
    return 0;
}
/* IPv6   2606:2800:220:1:248:1893:25c8:1946
 * IPv4   93.184.216.34
 *   connected using this one
 *   Note: the same source code handles both families with no #ifdef. */

import socket, struct

# getaddrinfo in Python returns the same candidate list.
results = socket.getaddrinfo("example.com", "https",
                             family=socket.AF_UNSPEC,      # both families
                             type=socket.SOCK_STREAM)
for family, socktype, proto, canonname, sockaddr in results:
    fam = "IPv6" if family == socket.AF_INET6 else "IPv4"
    print(f"  {fam}  {sockaddr[0]}:{sockaddr[1]}")
#   IPv6  2606:2800:220:1:248:1893:25c8:1946:443
#   IPv4  93.184.216.34:443

# Text <-> binary conversion, for both families.
for fam, text in ((socket.AF_INET, "192.0.2.1"),
                  (socket.AF_INET6, "2001:db8::1")):
    packed = socket.inet_pton(fam, text)          # presentation -> network
    back   = socket.inet_ntop(fam, packed)        # network -> presentation
    print(f"\n{text:<16} -> {packed.hex():<34} ({len(packed)} bytes) -> {back}")
# 192.0.2.1        -> c0000201                           (4 bytes) -> 192.0.2.1
# 2001:db8::1      -> 20010db8000000000000000000000001   (16 bytes) -> 2001:db8::1

# Build a sockaddr_in by hand, to see what the C struct actually contains.
def pack_sockaddr_in(ip, port):
    """struct sockaddr_in { u16 family; u16 port; u32 addr; u8 zero[8]; }"""
    return struct.pack("!HH4s8s",
                       socket.AF_INET,                     # sin_family
                       port,                               # sin_port (BIG-endian)
                       socket.inet_pton(socket.AF_INET, ip),
                       b"\x00" * 8)                        # sin_zero padding

raw = pack_sockaddr_in("192.0.2.1", 8080)
print(f"\nsockaddr_in: {raw.hex()}  ({len(raw)} bytes)")
# sockaddr_in: 00021f90c00002010000000000000000  (16 bytes)
#              ^^^^ family=2 (AF_INET)
#                  ^^^^ port=0x1f90=8080, already big-endian
#                      ^^^^^^^^ address
#   sockaddr_in6 is 28 bytes -- which is why sockaddr_storage exists.

# Why AF_UNSPEC matters: count what you would have missed.
v4_only = socket.getaddrinfo("example.com", 443, family=socket.AF_INET)
both    = socket.getaddrinfo("example.com", 443, family=socket.AF_UNSPEC)
print(f"\nAF_INET   -> {len(v4_only)} candidate(s)")
print(f"AF_UNSPEC -> {len(both)} candidate(s)")
# AF_INET   -> 1 candidate(s)
# AF_UNSPEC -> 2 candidate(s)
#   On an IPv6-only network, the AF_INET program simply cannot connect.

# What getaddrinfo actually consults, and in what order:
$ cat /etc/nsswitch.conf | grep hosts
# hosts: files dns myhostname
#        ^^^^^ /etc/hosts is checked BEFORE DNS

# Service name -> port, the table getaddrinfo uses:
$ grep -E '^(https|ssh|domain)\s' /etc/services
# ssh     22/tcp
# domain  53/tcp
# https   443/tcp

# Resolve the way getaddrinfo does, showing both families:
$ getent ahosts example.com
# 2606:2800:220:1:248:1893:25c8:1946 STREAM example.com
# 93.184.216.34                      STREAM
#   Order here reflects RFC 6724 address selection: IPv6 first when usable.

# Unix domain sockets use a filesystem path, not an address:
$ ss -xl | head -3
# Netid State  Local Address:Port
# u_str LISTEN /run/systemd/private 12043
#              ^^^^^^^^^^^^^^^^^^^^ the "address" is a path

# Verify a program is not accidentally IPv4-only:
$ ss -tlnp | grep myserver
# LISTEN 0 511 0.0.0.0:8080 0.0.0.0:*
#             ^^^^^^^ IPv4 only -- an IPv6 client cannot reach it
# LISTEN 0 511    [::]:8080    [::]:*
#             ^^^^^^ dual-stack: accepts both (with IPV6_V6ONLY off)

Key takeaways

• Address families (AF_INET, AF_INET6, AF_UNIX, AF_PACKET) each have their own structure, unified by casting to struct sockaddr * and passing a length.
• struct sockaddr is never used directly — it is a cast target; use sockaddr_storage when the family is not known in advance.
• getaddrinfo() is the correct modern interface: it resolves names, handles service names, and returns a candidate list covering both IP versions.
• Passing AF_UNSPEC is what makes a program dual-stack; hardcoding AF_INET is the standard way programs become IPv6-incapable.
• inet_pton/inet_ntop handle both families and replace the IPv4-only inet_addr/inet_ntoa.
• getaddrinfo allocates — every success must be paired with freeaddrinfo.

🧪 Practice

1. Explain why bind() takes both a struct sockaddr * and a separate length argument.
2. Write the getaddrinfo hints structure for a dual-stack TCP server that should bind to all local interfaces.
3. State what sockaddr_storage is for and why sockaddr cannot serve the same purpose.
4. Interview-style: A service works for all users until the company enables IPv6, then a subset of clients cannot connect at all. (Hint: look for a hardcoded address family and a call that predates getaddrinfo.)

Server Calls: socket, bind, listen, accept

Theory

A server is the passive party from 8.1: it must be findable, which means occupying a known address before any client arrives. Four calls establish that, and each has a distinct job that is worth separating clearly because they are often blurred together.

socket() creates the endpoint. At this point it has no address — it is a handle to nothing in particular.

bind() assigns a local address and port. This is what claims the well-known port. The address argument chooses which interfaces to accept traffic on, and the choice matters:

• INADDR_ANY (0.0.0.0, or :: for IPv6) — all interfaces. The usual choice for a public service.
• A specific address — only that interface. Used to restrict a service to, say, a management network.
• 127.0.0.1 — loopback only, unreachable from other machines. The correct default for a database or admin interface.

listen() converts the socket from an ordinary one into a passive listening socket and creates the accept queue. Its backlog argument bounds how many completed-but-not-yet-accepted connections the kernel will hold. On Linux there are actually two queues: the SYN queue for half-open handshakes and the accept queue for completed ones; backlog sizes the latter. When it fills, new connections are dropped or refused, which appears to clients as timeouts under load.

accept() removes one completed connection from the queue and returns a new socket for it. This is the call that most often confuses newcomers, and 7.1's demultiplexing discussion explains why it must work this way: the listening socket is identified by a 2-tuple (local address, port), while each connection is a distinct 4-tuple. The listening socket is a factory, not a connection; it stays open and keeps producing new sockets.

Two operational details cause the majority of real problems.

SO_REUSEADDR. After a connection closes, the local port sits in TIME_WAIT for up to two minutes (Chapter 7). Without this option, restarting a server fails with "Address already in use" until that expires. Setting it is standard practice for any server. SO_REUSEPORT is different and newer: it lets multiple processes bind the same port, with the kernel load-balancing among them — the basis of modern multi-process servers without a shared listening socket.

Privileged ports. Binding below 1024 requires root or CAP_NET_BIND_SERVICE. The usual pattern is to bind as root then drop privileges, or to grant the capability to the binary so it never runs as root at all.

Examples

The four calls, and what each one establishes

  socket()   ->  fd exists, no address
                 +----------+
                 |  fd 3    |  (unbound)
                 +----------+
      |
  bind()     ->  fd owns a LOCAL address:port
                 +----------------------+
                 | fd 3  0.0.0.0:8080   |
                 +----------------------+
      |
  listen()   ->  fd becomes PASSIVE; queues are created
                 +----------------------+
                 | fd 3  LISTEN         |
                 |  SYN queue    [ ][ ] |  half-open handshakes
                 |  accept queue [c][c] |  <- backlog sizes THIS one
                 +----------------------+
      |
  accept()   ->  removes one; returns a NEW fd for that connection
                 +----------------------+     +---------------------------+
                 | fd 3  LISTEN (stays) |     | fd 4  ESTABLISHED         |
                 |  accept queue [c]    |     | 10.0.0.9:51234 <-> :8080  |
                 +----------------------+     +---------------------------+

  The listening socket is a FACTORY. It is never itself a connection.


 WHAT bind() ADDRESS CHOICE MEANS

   0.0.0.0:8080     reachable on every interface        <- public service
   10.0.5.2:8080    reachable only via that interface   <- management net
   127.0.0.1:8080   reachable only from this machine    <- admin/database


 THE BACKLOG, and what happens when it fills

   clients ---> [SYN queue] ---> [accept queue] ---> accept() ---> your code
                 (handshake      (completed,          |
                  in progress)    waiting)            |
                                      ^               |
                                      |               |
                              backlog bounds this     slow accept() loop
                                                      = queue fills
                                                      = connections DROPPED
                                                      = client sees a timeout

#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>

int main(void) {
    /* 1. CREATE -- no address yet. */
    int listen_fd = socket(AF_INET, SOCK_STREAM, 0);
    if (listen_fd < 0) { perror("socket"); return 1; }

    /* SO_REUSEADDR: allow rebinding while old connections are in TIME_WAIT.
     * Without this, restarting the server fails for up to 2 minutes. */
    int yes = 1;
    setsockopt(listen_fd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof yes);

    struct sockaddr_in addr;
    memset(&addr, 0, sizeof addr);
    addr.sin_family      = AF_INET;
    addr.sin_port        = htons(8080);        /* host -> network byte order */
    addr.sin_addr.s_addr = htonl(INADDR_ANY);  /* 0.0.0.0 = every interface */

    /* 2. BIND -- claim the local address and port. */
    if (bind(listen_fd, (struct sockaddr *)&addr, sizeof addr) < 0) {
        perror("bind");                         /* EADDRINUSE, EACCES (<1024) */
        return 1;
    }

    /* 3. LISTEN -- become passive; create the accept queue.
     * backlog=128 bounds COMPLETED connections awaiting accept(). */
    if (listen(listen_fd, 128) < 0) { perror("listen"); return 1; }
    printf("listening on 0.0.0.0:8080\n");

    for (;;) {
        struct sockaddr_storage peer;           /* big enough for ANY family */
        socklen_t peer_len = sizeof peer;       /* value-result: in=capacity,
                                                 * out=actual size written */

        /* 4. ACCEPT -- returns a NEW fd; listen_fd stays open for the next one. */
        int conn_fd = accept(listen_fd, (struct sockaddr *)&peer, &peer_len);
        if (conn_fd < 0) { perror("accept"); continue; }

        char host[INET6_ADDRSTRLEN];
        struct sockaddr_in *p = (struct sockaddr_in *)&peer;
        inet_ntop(AF_INET, &p->sin_addr, host, sizeof host);
        printf("accepted fd=%d from %s:%d\n", conn_fd, host, ntohs(p->sin_port));

        write(conn_fd, "hello\n", 6);
        close(conn_fd);                         /* close the CONNECTION only */
    }
    /* listening on 0.0.0.0:8080
     * accepted fd=4 from 127.0.0.1:51234
     * accepted fd=4 from 127.0.0.1:51235     <- fd reused after close
     *   listen_fd (3) is never closed: it keeps producing connections. */
}

import socket

srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# Without SO_REUSEADDR, a restart fails while old connections sit in TIME_WAIT.
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

srv.bind(("0.0.0.0", 8080))      # claim the address
srv.listen(128)                   # become passive; backlog = accept queue size
print(f"listening fd={srv.fileno()} on {srv.getsockname()}")
# listening fd=3 on ('0.0.0.0', 8080)

conn, peer = srv.accept()         # NEW socket for this connection
print(f"listening socket still fd={srv.fileno()}")
print(f"connection socket    fd={conn.fileno()} peer={peer}")
# listening socket still fd=3
# connection socket    fd=4 peer=('127.0.0.1', 51234)
#   Two DIFFERENT sockets. The listener is a factory, not a connection.

# What the bind address actually controls:
bind_choices = {
    "0.0.0.0":    "every interface -- public service",
    "10.0.5.2":   "one interface only -- e.g. a management network",
    "127.0.0.1":  "loopback only -- unreachable from other machines",
}
for addr, meaning in bind_choices.items():
    print(f"  bind('{addr}') -> {meaning}")

# Backlog sizing: what happens when accept() cannot keep up.
def queue_behaviour(arrival_rate, accept_rate, backlog, seconds=10):
    queued, dropped = 0, 0
    for _ in range(seconds):
        queued += arrival_rate - accept_rate
        if queued > backlog:
            dropped += queued - backlog
            queued = backlog              # excess is DROPPED, not buffered
        queued = max(queued, 0)
    return queued, dropped

print(f"\n{'arrivals/s':>11}{'accepts/s':>11}{'backlog':>9}{'queued':>8}{'DROPPED':>9}")
for arr, acc in ((100, 200), (500, 200), (2000, 200)):
    q, d = queue_behaviour(arr, acc, 128)
    print(f"{arr:>11}{acc:>11}{128:>9}{q:>8}{d:>9}")
#  arrivals/s  accepts/s  backlog  queued  DROPPED
#         100        200      128       0        0
#         500        200      128     128     2872
#        2000        200      128     128   17872
#   Dropped connections appear to the CLIENT as a connection timeout,
#   while the server's logs show nothing at all.

# SO_REUSEPORT: several processes on the SAME port, kernel-balanced.
import os
def spawn_workers(n, port):
    for i in range(n):
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)  # NOT REUSEADDR
        s.bind(("0.0.0.0", port))
        s.listen(128)
        # each worker has its OWN listening socket on the same port;
        # the kernel distributes incoming connections among them.
    return f"{n} independent listeners on :{port}"
print(f"\n{spawn_workers.__name__}: kernel load-balances across workers")
#   This avoids the thundering herd discussed in 9.2.

# Observe the transition from unbound to listening:
$ ss -tlnp | grep 8080
# LISTEN 0 128 0.0.0.0:8080 0.0.0.0:*  users:(("server",pid=4821,fd=3))
#        ^ ^^^ Send-Q column shows the BACKLOG for a listening socket
#        ^ Recv-Q shows connections CURRENTLY waiting to be accepted

# A server that is failing to keep up -- Recv-Q approaches Send-Q:
$ ss -tln | grep 8080
# LISTEN 128 128 0.0.0.0:8080 0.0.0.0:*
#        ^^^ ^^^ queue is FULL: connections are being dropped

# Count dropped connections due to a full accept queue:
$ nstat -az TcpExtListenOverflows TcpExtListenDrops
# TcpExtListenOverflows  1847  0.0
# TcpExtListenDrops      1847  0.0
#   Non-zero and rising = increase backlog, or accept faster.

# The kernel caps your backlog at this value regardless of what you request:
$ sysctl net.core.somaxconn
# net.core.somaxconn = 4096

# Reproduce "Address already in use" without SO_REUSEADDR:
$ ss -tan state time-wait | grep ':8080' | wc -l
# 214       <- these hold the port until they expire

# Privileged ports: grant the capability instead of running as root:
$ sudo setcap 'cap_net_bind_service=+ep' ./myserver
$ getcap ./myserver
# ./myserver cap_net_bind_service=ep
$ ./myserver --port 80      # binds :80 as an unprivileged user

# Verify SO_REUSEPORT distributes across processes:
$ ss -tlnp | grep ':8080'
# LISTEN 0 128 0.0.0.0:8080 0.0.0.0:* users:(("worker",pid=101,fd=3))
# LISTEN 0 128 0.0.0.0:8080 0.0.0.0:* users:(("worker",pid=102,fd=3))
#   Four separate listeners on one port -- impossible without SO_REUSEPORT.

Key takeaways

• The server sequence is socket (create) → bind (claim address) → listen (become passive) → accept (take one connection).
• accept() returns a NEW socket; the listening socket is a factory that stays open — this follows directly from TCP's 4-tuple demultiplexing.
• The bind address selects interfaces: 0.0.0.0 for public, a specific address for one network, 127.0.0.1 to be unreachable remotely.
• backlog sizes the accept queue; when it overflows, connections are dropped silently and clients see timeouts while server logs show nothing.
• SO_REUSEADDR avoids "Address already in use" caused by TIME_WAIT and is standard for servers; SO_REUSEPORT lets multiple processes share a port with kernel load balancing.
• Ports below 1024 require privilege — grant CAP_NET_BIND_SERVICE rather than running as root.

🧪 Practice

1. Explain why accept() returns a new file descriptor rather than reusing the listening one, referring to how TCP demultiplexes.
2. A server binds 127.0.0.1:5432 and remote clients cannot connect, though the process is running and listening. Explain and fix.
3. Describe what a client observes when the accept queue is full, and name the counter you would check on the server.
4. Interview-style: Restarting your server reliably fails for about a minute with "Address already in use". (Hint: the port is not held by your process — check which TCP state the old connections are still in, and which socket option exists for exactly this.)

Client Calls: socket, connect

Theory

The client side is shorter, which reflects 8.1's asymmetry: the client needs no stable address and does not wait for anyone. Two calls suffice.

socket() creates the endpoint, exactly as on the server.

connect() does three things at once, and separating them explains most client-side behaviour:

1. It performs an implicit bind if the socket is not already bound, assigning an ephemeral port from the range 32768–60999 on Linux. This is why clients do not call bind() — the kernel picks a source port automatically.
2. It initiates the TCP three-way handshake and blocks until it completes or fails.
3. It records the peer address, so subsequent send/recv need no destination argument.

The failure modes of connect() are worth distinguishing precisely, because they point at different problems:

Error	Meaning	Usual cause
ECONNREFUSED	RST received	Nothing listening on that port
ETIMEDOUT	No response at all	Firewall dropping, or host down
EHOSTUNREACH	ICMP unreachable	No route to the host
ENETUNREACH	No route to the network	Routing table has no entry
EINPROGRESS	Non-blocking, still working	Expected for non-blocking sockets

The distinction between refused and timed out is the most diagnostically useful thing in this table. Refused means the packet reached a machine that actively said no — the network path works. Timed out means nothing came back at all, which almost always means a firewall is dropping rather than rejecting.

connect() on a UDP socket does something different and often surprising: since UDP is connectionless, no packets are sent. It merely records a default peer, which has three effects: you can use send/recv instead of sendto/recvfrom, the kernel filters incoming datagrams to that peer only, and you begin receiving ICMP errors (such as port unreachable) that an unconnected UDP socket silently discards. That last effect is genuinely useful and frequently overlooked.

For clients that must be responsive, the blocking behaviour of connect() is a problem — a default TCP connect timeout can exceed two minutes. The solutions are a non-blocking socket plus select/poll with a timeout, or Happy Eyeballs (RFC 8305), which races IPv6 and IPv4 connections in parallel and uses whichever completes first. Happy Eyeballs is why modern browsers do not stall on broken IPv6 paths.

Examples

What connect() does, in one picture

   socket()            fd exists, unbound, no peer
      |
   connect(peer)
      |
      +--(a) IMPLICIT BIND: kernel picks a source port
      |       10.0.0.9:_____  ->  10.0.0.9:51234
      |                             ^^^^^ ephemeral, from 32768-60999
      |
      +--(b) THREE-WAY HANDSHAKE
      |       ---- SYN --------->
      |       <--- SYN/ACK ------
      |       ---- ACK --------->
      |
      +--(c) RECORD THE PEER
              now send()/recv() need no address argument
      |
      v
   connected: 10.0.0.9:51234 <-> 93.184.216.34:443


 DIAGNOSING connect() FAILURES

   ECONNREFUSED         ETIMEDOUT
   ---- SYN --->        ---- SYN --->   (silence)
   <--- RST ----        ---- SYN --->   (silence, retry)
                        ---- SYN --->   (silence, retry)
   The host EXISTS      Nothing answers at all
   and said NO.         -> firewall DROPPING, or host down
   Path works;          -> the path itself is suspect
   service is down.

   This distinction tells you whether to debug the SERVICE or the NETWORK.


 HAPPY EYEBALLS (RFC 8305): do not let broken IPv6 stall the user

   t=0     start IPv6 connect  ----X (no route, will time out in 75 s)
   t=50ms  start IPv4 connect  ----> SYN/ACK at t=90ms
   t=90ms  IPv4 wins; cancel the IPv6 attempt
   user waited 90 ms instead of 75 seconds

#include <sys/socket.h>
#include <netdb.h>
#include <fcntl.h>
#include <errno.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/select.h>

/* Connect with a timeout -- blocking connect() can take over two minutes. */
int connect_timeout(const char *host, const char *service, int timeout_ms) {
    struct addrinfo hints, *res, *p;
    memset(&hints, 0, sizeof hints);
    hints.ai_family   = AF_UNSPEC;            /* both IP versions */
    hints.ai_socktype = SOCK_STREAM;
    if (getaddrinfo(host, service, &hints, &res) != 0) return -1;

    for (p = res; p; p = p->ai_next) {
        int fd = socket(p->ai_family, p->ai_socktype, p->ai_protocol);
        if (fd < 0) continue;

        /* Make it non-blocking so connect() returns immediately. */
        int flags = fcntl(fd, F_GETFL, 0);
        fcntl(fd, F_SETFL, flags | O_NONBLOCK);

        int rc = connect(fd, p->ai_addr, p->ai_addrlen);
        if (rc == 0) { freeaddrinfo(res); return fd; }     /* instant (loopback) */

        if (errno != EINPROGRESS) {            /* a REAL error, not "in progress" */
            close(fd);
            continue;
        }

        /* Wait for writability, which signals handshake completion. */
        fd_set wset;
        FD_ZERO(&wset); FD_SET(fd, &wset);
        struct timeval tv = { timeout_ms / 1000, (timeout_ms % 1000) * 1000 };

        if (select(fd + 1, NULL, &wset, NULL, &tv) > 0) {
            /* Writable does NOT mean success -- check SO_ERROR for the verdict. */
            int err = 0; socklen_t len = sizeof err;
            getsockopt(fd, SOL_SOCKET, SO_ERROR, &err, &len);
            if (err == 0) {
                fcntl(fd, F_SETFL, flags);     /* restore blocking mode */
                freeaddrinfo(res);
                return fd;
            }
            fprintf(stderr, "connect: %s\n", strerror(err));
        } else {
            fprintf(stderr, "connect: timed out after %d ms\n", timeout_ms);
        }
        close(fd);
    }
    freeaddrinfo(res);
    return -1;
}

int main(void) {
    int fd = connect_timeout("example.com", "443", 3000);
    printf(fd >= 0 ? "connected fd=%d\n" : "failed\n", fd);
    return 0;
}
/* connected fd=3
 *   Note the two-step check: select() says "something happened",
 *   SO_ERROR says WHAT happened. Skipping the second step is a classic bug. */

import socket, errno

# The minimal client: two calls.
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(3.0)                                  # ALWAYS set a timeout
s.connect(("example.com", 443))
print(f"local  {s.getsockname()}")                 # the implicit bind's result
print(f"remote {s.getpeername()}")
# local  ('10.0.0.9', 51234)      <- ephemeral port the kernel chose
# remote ('93.184.216.34', 443)
s.close()

# Failure modes, and what each one tells you.
def diagnose(host, port, timeout=2.0):
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.settimeout(timeout)
    try:
        s.connect((host, port))
        return "connected -- service is up"
    except ConnectionRefusedError:
        return "ECONNREFUSED -- host reachable, NOTHING LISTENING on that port"
    except socket.timeout:
        return "ETIMEDOUT -- no response at all; firewall DROPPING or host down"
    except OSError as e:
        if e.errno == errno.EHOSTUNREACH: return "EHOSTUNREACH -- no route to host"
        if e.errno == errno.ENETUNREACH:  return "ENETUNREACH -- no route to network"
        return f"errno {e.errno}: {e}"
    finally:
        s.close()

for host, port in (("127.0.0.1", 9), ("example.com", 443)):
    print(f"\n{host}:{port} -> {diagnose(host, port)}")
# 127.0.0.1:9 -> ECONNREFUSED -- host reachable, NOTHING LISTENING on that port
# example.com:443 -> connected -- service is up
#   REFUSED means the path works and the service is down.
#   TIMED OUT means you cannot conclude anything about the service.

# connect() on UDP: no packets, but three real effects.
u = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
u.connect(("127.0.0.1", 9999))         # nothing is transmitted
print(f"\nUDP 'connected' to {u.getpeername()}")
u.send(b"probe")                       # send(), not sendto()
try:
    u.settimeout(0.5)
    u.recv(1024)
except ConnectionRefusedError:
    print("received ICMP port unreachable as ECONNREFUSED")
except socket.timeout:
    print("timed out")
# UDP 'connected' to ('127.0.0.1', 9999)
# received ICMP port unreachable as ECONNREFUSED
#   An UNCONNECTED UDP socket discards that ICMP silently. Connecting it
#   is how you learn the far end is not listening.

# Happy Eyeballs: race the families instead of serialising them.
import concurrent.futures, time

def attempt(sockaddr, family, timeout=3.0):
    s = socket.socket(family, socket.SOCK_STREAM)
    s.settimeout(timeout)
    t0 = time.perf_counter()
    s.connect(sockaddr[:2])
    return s, (time.perf_counter() - t0) * 1000

def happy_eyeballs(host, port, delay_ms=50):
    cands = socket.getaddrinfo(host, port, socket.AF_UNSPEC, socket.SOCK_STREAM)
    cands.sort(key=lambda c: c[0] != socket.AF_INET6)     # IPv6 first, per RFC
    with concurrent.futures.ThreadPoolExecutor(max_workers=len(cands)) as ex:
        futures = []
        for i, (fam, _, _, _, sa) in enumerate(cands):
            time.sleep(0 if i == 0 else delay_ms / 1000)  # stagger, do not flood
            futures.append(ex.submit(attempt, sa, fam))
        for fut in concurrent.futures.as_completed(futures, timeout=5):
            try:
                sock, ms = fut.result()
                return sock, ms                            # FIRST to succeed wins
            except OSError:
                continue
    raise OSError("all candidates failed")

sock, ms = happy_eyeballs("example.com", 443)
print(f"\nconnected in {ms:.0f} ms via {sock.getpeername()[0]}")
# connected in 91 ms via 93.184.216.34
#   A broken IPv6 path costs ~50 ms here instead of a 75-second timeout.

# The ephemeral port range connect() draws from:
$ sysctl net.ipv4.ip_local_port_range
# net.ipv4.ip_local_port_range = 32768 60999
#   ~28,000 ports -> that many simultaneous connections to ONE destination.

# Distinguish refused from dropped, from the command line:
$ nc -zv -w2 127.0.0.1 9
# nc: connect to 127.0.0.1 port 9 (tcp) failed: Connection refused
$ nc -zv -w2 10.255.255.1 80
# nc: connect to 10.255.255.1 port 80 (tcp) timed out
#   Refused = something answered. Timed out = nothing did.

# Watch the difference on the wire:
$ sudo tcpdump -i lo -n 'port 9' -c 2
# IP 127.0.0.1.51240 > 127.0.0.1.9: Flags [S]
# IP 127.0.0.1.9 > 127.0.0.1.51240: Flags [R.]     <- RST = refused
#   A DROPPED connection shows only repeated SYNs with no reply.

# How long a blocking connect() actually takes before giving up:
$ sysctl net.ipv4.tcp_syn_retries
# net.ipv4.tcp_syn_retries = 6
#   6 retries with exponential backoff = ~127 seconds. Always set a timeout.

# See the implicit bind's chosen source port:
$ curl -s https://example.com/ >/dev/null & sleep 0.2; ss -tnp | grep curl
# ESTAB 0 0 10.0.0.9:51234 93.184.216.34:443 users:(("curl",pid=5012,fd=5))
#                   ^^^^^ assigned by connect(), not by the program

# Confirm Happy Eyeballs behaviour in curl:
$ curl -v --happy-eyeballs-timeout-ms 200 https://example.com/ 2>&1 | grep -i trying
#   Trying 2606:2800:220:1:248:1893:25c8:1946:443...
#   Trying 93.184.216.34:443...
#   ^ both started; the first to complete is used

Key takeaways

• A client needs only socket() and connect(); no bind() is required because connect() performs an implicit bind to an ephemeral port.
• connect() also runs the three-way handshake and records the peer, so later send/recv need no address.
• ECONNREFUSED means the host answered with RST (path works, service down); ETIMEDOUT means nothing answered (firewall dropping or host down) — the most useful diagnostic distinction in network debugging.
• connect() on UDP sends nothing but sets a default peer, filters incoming datagrams, and enables ICMP error delivery that unconnected sockets discard silently.
• A blocking connect() can take over two minutes by default; use a non-blocking socket with select/poll, and check SO_ERROR — writability alone does not mean success.
• Happy Eyeballs races IPv6 and IPv4 in parallel so a broken path costs milliseconds instead of a full timeout.

🧪 Practice

1. Explain why client programs rarely call bind() and what the kernel does instead.
2. Your monitoring shows ECONNREFUSED from one region and ETIMEDOUT from another for the same service. State what each tells you and where you would look first.
3. Write the sequence of calls for a non-blocking connect with a 2-second timeout, naming the option that reports the final result.
4. Interview-style: Why does calling connect() on a UDP socket change what errors you receive, even though no packets are exchanged by the call itself? (Hint: think about what the kernel can now associate an incoming ICMP message with.)

send, recv, and Partial Transfers

Theory

This is the topic that separates working network code from code that works in testing. The rule is simple and constantly violated: send() and recv() are not guaranteed to transfer the number of bytes you asked for.

The reason is that both operate on kernel buffers, not on the network directly:

• send() copies data into the socket send buffer and returns how many bytes it copied. If the buffer has room for only 4,000 of your 10,000 bytes, it copies 4,000 and returns 4,000. The connection is fine; it simply cannot take more right now. Actual transmission happens later, asynchronously, governed by TCP's flow and congestion control from Chapter 7.
• recv() returns whatever is currently in the receive buffer, up to the size you requested. If the sender wrote 1,000 bytes and only 300 have arrived, you get 300.

So every real program needs two loops: send until everything is written, and read until the message is complete (which requires framing — see 9.3).

The return values carry three distinct meanings, and conflating them is a classic bug:

recv() returns	Meaning
> 0	That many bytes are available
0	Orderly shutdown — the peer called close() or shutdown(). EOF.
-1	Error; check errno

recv() returning 0 is not an error — it is the end of the stream, the network equivalent of EOF on a file. Code that treats it as an error loops forever; code that treats an error as EOF loses data silently.

Two errno values need special handling:

• EINTR — the call was interrupted by a signal. Retry it; nothing went wrong.
• EAGAIN/EWOULDBLOCK — a non-blocking socket had nothing to do right now. Not an error; wait for readiness (9.2) and try again.

Two behaviours surprise people:

Nagle's algorithm coalesces small writes to avoid flooding the network with tiny packets, by holding data until an ACK arrives or a full segment accumulates. For bulk transfer this is good. For request-response traffic it adds up to ~40 ms of latency, and interacts badly with delayed ACKs. TCP_NODELAY disables it, and is correct for latency-sensitive protocols.

A successful send() guarantees nothing about delivery. It means the bytes are in your kernel's buffer. They may never reach the peer — the connection may fail a moment later. Only an application-level acknowledgement proves receipt, which is why protocols have their own acks even on top of TCP.

Examples

send() and recv() operate on BUFFERS, not on the wire

  your program                kernel                       network
  +-------------+   send()   +----------------+
  | 10,000 B    |----------->| send buffer    |---- TCP segments ---->
  +-------------+ returns    | (e.g. 4 KB     |     (asynchronous,
                   4096      |  free)         |      rate-controlled)
                             +----------------+
   ^ NOT an error. The buffer was full. Call send() again with the rest.

  +-------------+   recv()   +----------------+
  |   300 B     |<-----------| recv buffer    |<--- TCP segments -----
  +-------------+ returns    | (300 B arrived |
                   300       |  so far)       |
                             +----------------+
   ^ You asked for 1000. You got what had ARRIVED.


 THE TWO LOOPS EVERY PROGRAM NEEDS

  SEND ALL                          RECV EXACTLY N
  total = 0                         got = 0
  while total < len:                while got < n:
      n = send(buf[total:])             k = recv(n - got)
      if n < 0: handle EINTR/EAGAIN     if k == 0: PEER CLOSED -> error
      total += n                        if k < 0: handle EINTR/EAGAIN
                                        got += k


 recv() RETURN VALUES -- three different things

     > 0  ---> data
       0  ---> ORDERLY SHUTDOWN (EOF). NOT an error.
      -1  ---> error; errno says which
                 EINTR      -> retry, nothing is wrong
                 EAGAIN     -> non-blocking, nothing ready, wait
                 ECONNRESET -> peer sent RST


 NAGLE + DELAYED ACK: the classic 40 ms stall

   client: write(small)  -> Nagle holds it (waiting for an ACK)
   server: delayed ACK   -> waits up to 40 ms before acking
   ...both sides waiting...
   -> 40 ms added to every request. TCP_NODELAY removes it.

#include <sys/socket.h>
#include <errno.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>

/* send_all: loop until every byte is written. This is NOT optional. */
ssize_t send_all(int fd, const void *buf, size_t len) {
    const char *p = buf;
    size_t sent = 0;
    while (sent < len) {
        ssize_t n = send(fd, p + sent, len - sent, MSG_NOSIGNAL);
        /* MSG_NOSIGNAL: return EPIPE instead of killing the process with
         * SIGPIPE when the peer has closed. Almost always what you want. */
        if (n < 0) {
            if (errno == EINTR) continue;        /* signal: just retry */
            return -1;                           /* a real error */
        }
        sent += (size_t)n;                       /* n may be < requested! */
    }
    return (ssize_t)sent;
}

/* recv_exactly: loop until n bytes arrive, or the peer closes. */
ssize_t recv_exactly(int fd, void *buf, size_t len) {
    char *p = buf;
    size_t got = 0;
    while (got < len) {
        ssize_t n = recv(fd, p + got, len - got, 0);
        if (n == 0) return 0;                    /* ORDERLY SHUTDOWN, not error */
        if (n < 0) {
            if (errno == EINTR) continue;
            return -1;
        }
        got += (size_t)n;
    }
    return (ssize_t)got;
}

int main(void) {
    /* A demonstration of why the loop matters: request a large write and
     * print what a SINGLE send() would actually have transferred. */
    char big[1024 * 1024];
    memset(big, 'x', sizeof big);

    int fd = 1;                                  /* stdout, for illustration */
    printf("requested: %zu bytes\n", sizeof big);
    /* On a real socket with a 4 KB send buffer:
     *   single send() -> returns 4096, NOT 1048576
     *   send_all()    -> returns 1048576 after ~256 iterations */
    return 0;
}

import socket, errno, threading, time

# Demonstrate a partial send with a deliberately small send buffer.
def partial_send_demo():
    srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    srv.bind(("127.0.0.1", 0)); srv.listen(1)
    port = srv.getsockname()[1]

    c = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    c.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 4096)   # tiny send buffer
    c.connect(("127.0.0.1", port))
    conn, _ = srv.accept()
    conn.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 4096)

    c.setblocking(False)
    payload = b"x" * (1024 * 1024)        # 1 MB
    try:
        n = c.send(payload)               # ONE send() call
    except BlockingIOError:
        n = 0
    print(f"requested {len(payload):,} bytes, single send() wrote {n:,}")
    # requested 1,048,576 bytes, single send() wrote 4,608
    #   Code that assumes send() writes everything just lost 99.6% of the message.
    c.close(); conn.close(); srv.close()

partial_send_demo()

# The two loops, written correctly.
def send_all(sock, data):
    """Loop until every byte is accepted by the kernel."""
    total = 0
    while total < len(data):
        try:
            n = sock.send(data[total:])
        except InterruptedError:                  # EINTR: retry
            continue
        except BlockingIOError:                   # EAGAIN: wait for writability
            continue                              # (9.2 shows how to wait properly)
        if n == 0:
            raise ConnectionError("socket connection broken")
        total += n
    return total

def recv_exactly(sock, n):
    """Loop until exactly n bytes arrive. Returns None on orderly shutdown."""
    buf = bytearray()
    while len(buf) < n:
        try:
            chunk = sock.recv(n - len(buf))
        except InterruptedError:
            continue
        if not chunk:                             # b"" == recv returned 0 == EOF
            return None                           # peer closed mid-message
        buf.extend(chunk)
    return bytes(buf)

# What recv() == 0 means, versus an error.
def classify(result, exc=None):
    if exc:                       return f"ERROR: {exc}"
    if result == b"":             return "ORDERLY SHUTDOWN (EOF) -- peer closed cleanly"
    return f"{len(result)} bytes of data"

print()
print(" ", classify(b"hello"))
print(" ", classify(b""))
print(" ", classify(None, exc="ECONNRESET -- peer sent RST"))
#   5 bytes of data
#   ORDERLY SHUTDOWN (EOF) -- peer closed cleanly
#   ERROR: ECONNRESET -- peer sent RST
#   Treating EOF as an error loops forever; treating an error as EOF loses data.

# Nagle's algorithm: the latency it adds to request-response traffic.
def request_latency(rtt_ms, nagle, delayed_ack_ms=40):
    return rtt_ms + (delayed_ack_ms if nagle else 0)

print(f"\n{'RTT':>6}{'Nagle ON':>12}{'TCP_NODELAY':>14}")
for rtt in (1, 10, 50):
    print(f"{rtt:>4}ms{request_latency(rtt, True):>10}ms{request_latency(rtt, False):>12}ms")
#    RTT    Nagle ON   TCP_NODELAY
#    1ms        41ms           1ms
#   10ms        50ms          10ms
#   50ms        90ms          50ms
#   On a fast LAN, Nagle multiplies request latency by 40x.

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)     # disable Nagle
print(f"\nTCP_NODELAY = {s.getsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY)}")
# TCP_NODELAY = 1
#   Correct for RPC, databases, games. WRONG for bulk transfer, where
#   coalescing small writes is exactly what you want.

# A successful send() proves nothing about delivery.
print("\nsend() returned 10000. What is guaranteed?")
print("  GUARANTEED : 10000 bytes are in the local kernel send buffer")
print("  NOT GUARANTEED: they were transmitted")
print("  NOT GUARANTEED: they arrived")
print("  NOT GUARANTEED: the application read them")
print("  -> only an APPLICATION-LEVEL ack proves receipt")

# Inspect and tune socket buffer sizes:
$ sysctl net.ipv4.tcp_wmem net.ipv4.tcp_rmem
# net.ipv4.tcp_wmem = 4096 16384 4194304     # min default max
# net.ipv4.tcp_rmem = 4096 131072 6291456

# Watch buffers fill in real time -- Send-Q is unacknowledged data:
$ ss -tin dst 93.184.216.34
# ESTAB 0 34560 10.0.0.9:51234 93.184.216.34:443
#         ^^^^^ 34 KB sitting in the send buffer, not yet acknowledged
#   cubic wscale:7,7 rto:204 rtt:21.3/2.1 cwnd:10 send 5.5Mbps
#                                              ^^^^^^ congestion window limits
#                                                     how fast send() drains

# A receiver that is not reading fast enough:
$ ss -tn | awk '$2 > 0 {print}' | head -3
# ESTAB 262144 0 10.0.0.9:8080 10.0.0.5:44122
#       ^^^^^^ Recv-Q full: the application is not calling recv() fast enough
#              -> the window closes -> the sender stalls

# Confirm Nagle's effect on a request-response protocol:
$ sudo tcpdump -i any -n 'port 6379' -c 4 -ttt
#  00:00:00.000000 IP client > redis: Flags [P.], length 14
#  00:00:00.039812 IP redis > client: Flags [.], ack 15
#                  ^^^^^^^^ ~40 ms delayed ACK
#  Redis sets TCP_NODELAY precisely to avoid this.

# Check whether a running process disabled Nagle:
$ ss -tio dst :6379 | grep -o 'nodelay'
# nodelay

# SIGPIPE: writing to a closed connection kills the process by default:
$ python3 -c "
import socket,signal
print(signal.getsignal(signal.SIGPIPE))"
# Handlers.SIG_DFL     <- default action is TERMINATE
#   In C, use send(..., MSG_NOSIGNAL) or ignore SIGPIPE explicitly.

Key takeaways

• send() and recv() transfer a partial amount — they operate on kernel buffers, so every program needs a send-all loop and a read-until-complete loop.
• recv() returning 0 means orderly shutdown (EOF), not an error; conflating it with -1 causes infinite loops or silent data loss.
• EINTR means retry; EAGAIN/EWOULDBLOCK on a non-blocking socket means nothing is ready, not that anything failed.
• Nagle's algorithm adds up to ~40 ms to request-response traffic by interacting with delayed ACKs; TCP_NODELAY disables it and is right for RPC, wrong for bulk transfer.
• MSG_NOSIGNAL (or ignoring SIGPIPE) prevents a closed peer from terminating your process.
• A successful send() only means the bytes are in your kernel buffer — delivery is proven only by an application-level acknowledgement.

🧪 Practice

1. Write send_all in pseudocode, handling short writes, EINTR, and a return value of 0.
2. Explain the three distinct meanings of recv() returning -1, 0, and a positive number, and what your code must do for each.
3. A request-response protocol shows a consistent 40 ms latency on a 1 ms LAN. Diagnose it and give the fix.
4. Interview-style: Your logging shows every message was sent successfully, but the peer says several never arrived, and there were no errors. (Hint: state precisely what a successful send() return value proves — and what it does not.)

Byte Order and Endianness Conversion

Theory

Different CPU architectures store multi-byte integers in different orders. Little-endian machines (x86, most ARM) store the least significant byte first; big-endian machines (older SPARC, PowerPC, and many network devices) store the most significant byte first. The 32-bit value 0x12345678 is 78 56 34 12 in memory on one and 12 34 56 78 on the other.

If two machines with different orderings exchange raw integers, the values are silently corrupted — not rejected, not flagged, just wrong. 1 becomes 16777216. This is one of the few remaining places where a program can be completely correct on your machine and completely broken on someone else's.

The solution, established early and now universal, is to define a network byte order — arbitrarily chosen as big-endian — and require every protocol to use it on the wire. Each host converts between its own ordering and the network's. On a big-endian machine the conversions are no-ops; on a little-endian machine they swap bytes. Either way the code is identical, which is the point.

The four classic macros:

Macro	Converts	Typical use
htons	Host to network, short (16-bit)	Port numbers
htonl	Host to network, long (32-bit)	IPv4 addresses, lengths
ntohs	Network to host, short	Reading a port
ntohl	Network to host, long	Reading a length prefix

Three things people get wrong:

Not everything needs conversion. Single bytes have no order. Byte arrays and text strings are already sequences of bytes and must not be swapped — running htonl over a string corrupts it. Only multi-byte numeric fields need it. Note that struct in_addr already holds its address in network order, which is why inet_pton output is used directly without conversion.

Floating-point is not covered. There is no htonf. IEEE 754 has its own representation questions, so protocols either transmit floats as a fixed-point integer, as text, or by explicitly documenting an encoding.

64-bit needs htonll, which is not standard. Linux offers htobe64/be64toh in <endian.h>; portable code defines its own or uses a serialisation library.

In higher-level languages the same discipline appears as an explicit format specifier: Python's struct module uses ! for network order, and forgetting it produces exactly the same class of bug. The general rule for any protocol work: state the byte order explicitly at every encode and decode site, rather than relying on the platform default.

Examples

The same 32-bit value, two machines

   value: 0x12345678  (decimal 305,419,896)

   BIG-ENDIAN memory (network order)    LITTLE-ENDIAN memory (x86)
   addr:  +0  +1  +2  +3                addr:  +0  +1  +2  +3
          12  34  56  78                       78  56  34  12
          ^^ most significant first            ^^ least significant first


 WHAT HAPPENS WITHOUT CONVERSION

   little-endian sender writes the integer 1:
        memory: 01 00 00 00
        wire:   01 00 00 00
   big-endian receiver reads those 4 bytes as an integer:
        value:  0x01000000 = 16,777,216

   No error. No warning. The number is simply WRONG by a factor of 16 million.


 THE CONVERSION DISCIPLINE

                  host order                    network order
   sending:   [ 78 56 34 12 ]  --htonl-->  [ 12 34 56 78 ]  --> wire
   receiving:     <--ntohl--                [ 12 34 56 78 ]  <-- wire

   On a BIG-ENDIAN host both macros are no-ops.
   The SOURCE CODE is identical either way. That is the whole design.


 WHAT NOT TO CONVERT

   port number   (uint16)  -> htons()     YES
   length prefix (uint32)  -> htonl()     YES
   single byte   (uint8)   -> nothing     no order exists
   char name[32]           -> nothing     already a byte sequence
   float/double            -> NO MACRO    encode explicitly (fixed-point/text)

#include <arpa/inet.h>
#include <endian.h>
#include <stdio.h>
#include <string.h>
#include <stdint.h>

/* Detect this machine's ordering at runtime. */
static const char *endianness(void) {
    uint32_t x = 1;
    return (*(uint8_t *)&x == 1) ? "little-endian" : "big-endian";
}

static void dump(const char *label, const void *p, size_t n) {
    printf("%-26s", label);
    for (size_t i = 0; i < n; i++) printf("%02x ", ((const uint8_t *)p)[i]);
    printf("\n");
}

int main(void) {
    printf("this machine is %s\n\n", endianness());

    uint32_t host_value = 0x12345678;
    uint32_t net_value  = htonl(host_value);      /* host -> network */

    dump("host order (in memory):", &host_value, 4);
    dump("network order (wire):",   &net_value,  4);

    uint16_t port = 8080;
    uint16_t net_port = htons(port);              /* ALWAYS for port numbers */
    printf("\nport %u -> network order 0x%04x\n", port, net_port);

    /* A protocol header: every multi-byte NUMERIC field is converted. */
    struct __attribute__((packed)) msg_header {
        uint8_t  version;      /* 1 byte  -> NO conversion needed */
        uint8_t  type;         /* 1 byte  -> NO conversion needed */
        uint16_t flags;        /* 2 bytes -> htons */
        uint32_t length;       /* 4 bytes -> htonl */
        char     tag[4];       /* bytes   -> NO conversion; it is text */
    };

    struct msg_header h = { .version = 1, .type = 3 };
    h.flags  = htons(0x00FF);
    h.length = htonl(1024);
    memcpy(h.tag, "MSG1", 4);                     /* NOT swapped: it is a string */

    dump("\nencoded header:", &h, sizeof h);
    printf("decoded: version=%u type=%u flags=0x%04x length=%u tag=%.4s\n",
           h.version, h.type, ntohs(h.flags), ntohl(h.length), h.tag);

    /* 64-bit: htonll does not exist. Linux provides these in <endian.h>. */
    uint64_t big = 0x0123456789ABCDEFULL;
    uint64_t net64 = htobe64(big);
    dump("\n64-bit network order:", &net64, 8);
    printf("round trip ok: %d\n", be64toh(net64) == big);
    return 0;
}
/* this machine is little-endian
 *
 * host order (in memory):   78 56 34 12
 * network order (wire):     12 34 56 78
 *
 * port 8080 -> network order 0x901f
 *
 * encoded header:           01 03 00 ff 00 00 04 00 4d 53 47 31
 *                           ^^ ^^ ^^^^^ ^^^^^^^^^^^ ^^^^^^^^^^^
 *                           v  t  flags  length      "MSG1" unswapped
 * decoded: version=1 type=3 flags=0x00ff length=1024 tag=MSG1
 *
 * 64-bit network order:     01 23 45 67 89 ab cd ef
 * round trip ok: 1 */

import struct, socket, sys

print(f"this machine is {sys.byteorder}-endian\n")

value = 0x12345678
print(f"native  order: {struct.pack('=I', value).hex()}")   # = platform default
print(f"little  order: {struct.pack('<I', value).hex()}")   # < explicit little
print(f"big/net order: {struct.pack('!I', value).hex()}")   # ! network (big)
# native  order: 78563412
# little  order: 78563412
# big/net order: 12345678
#   ALWAYS use '!' for protocol work. '=' is a portability bug waiting to happen.

# The corruption, demonstrated.
sent = struct.pack('<I', 1)                 # little-endian sender, no conversion
received = struct.unpack('>I', sent)[0]     # big-endian receiver
print(f"\nsender meant 1, receiver read {received:,}")
# sender meant 1, receiver read 16,777,216
#   No exception. No warning. Just a wrong number.

# Encode a protocol header, converting only what needs it.
def encode_header(version, msg_type, flags, length, tag):
    return struct.pack("!BBHI4s",      # ! = network order for ALL numeric fields
                       version,         # B: 1 byte, order is meaningless
                       msg_type,        # B: 1 byte
                       flags,           # H: 2 bytes, converted
                       length,          # I: 4 bytes, converted
                       tag)             # 4s: raw bytes, NOT converted

def decode_header(raw):
    v, t, f, ln, tag = struct.unpack("!BBHI4s", raw)
    return {"version": v, "type": t, "flags": f, "length": ln, "tag": tag.decode()}

hdr = encode_header(1, 3, 0x00FF, 1024, b"MSG1")
print(f"\nencoded: {hdr.hex()}  ({len(hdr)} bytes)")
print(f"decoded: {decode_header(hdr)}")
# encoded: 010300ff000004004d534731  (12 bytes)
# decoded: {'version': 1, 'type': 3, 'flags': 255, 'length': 1024, 'tag': 'MSG1'}

# The explicit conversion helpers, for parity with C.
print(f"\nhtons(8080) = 0x{socket.htons(8080):04x}")
print(f"ntohs(0x901f) = {socket.ntohs(0x901f)}")
print(f"htonl(1024) = 0x{socket.htonl(1024):08x}")
# htons(8080) = 0x901f
# ntohs(0x901f) = 8080
# htonl(1024) = 0x00040000

# What must NOT be converted.
cases = [
    ("uint16 port",     True,  "htons -- multi-byte number"),
    ("uint32 length",   True,  "htonl -- multi-byte number"),
    ("uint8 version",   False, "single byte has no order"),
    ("char tag[4]",     False, "byte sequence; swapping CORRUPTS it"),
    ("float",           False, "no htonf exists -- encode explicitly"),
    ("struct in_addr",  False, "already stored in network order"),
]
print()
for field, convert, why in cases:
    mark = "CONVERT " if convert else "leave as-is"
    print(f"  {field:<16} {mark:<12} {why}")

# There is no htonf. Encode floats deliberately.
def encode_float_fixed(value, scale=1000):
    """Fixed-point: transmit an integer, document the scale."""
    return struct.pack("!i", int(round(value * scale)))

def decode_float_fixed(raw, scale=1000):
    return struct.unpack("!i", raw)[0] / scale

pi = 3.14159
enc = encode_float_fixed(pi)
print(f"\n3.14159 as fixed-point: {enc.hex()} -> {decode_float_fixed(enc)}")
# 3.14159 as fixed-point: 00000c47 -> 3.141
#   Lossy but unambiguous. IEEE 754 over the wire needs an explicit contract.

# Check this machine's byte order:
$ lscpu | grep -i 'byte order'
# Byte Order:  Little Endian

$ python3 -c "import sys; print(sys.byteorder)"
# little

# See network byte order on the wire -- a port in a TCP header:
$ sudo tcpdump -i any -n -X 'port 8080' -c 1 2>/dev/null | head -4
# 0x0000:  4500 003c 1c46 4000 4006 ...
# 0x0010:  ...  c7a3 1f90
#                    ^^^^ 0x1f90 = 8080, stored BIG-ENDIAN on the wire
#                         regardless of either machine's native order.

# Verify a length prefix is encoded big-endian:
$ printf '\x00\x00\x04\x00' | xxd
# 00000000: 0000 0400
$ python3 -c "
import struct
raw = b'\x00\x00\x04\x00'
print('as big-endian   :', struct.unpack('>I', raw)[0])
print('as little-endian:', struct.unpack('<I', raw)[0])"
# as big-endian   : 1024
# as little-endian: 262144
#   Same four bytes, two answers. The protocol spec must say which.

# Find suspicious native-order packing in a codebase:
$ grep -rn "struct.pack('=" --include=*.py . | head -3
# ./proto.py:42: struct.pack('=IH', length, port)
#   '=' means platform order: correct today, broken on a big-endian peer.
$ grep -rn "struct.pack('!" --include=*.py . | wc -l
# 28        <- these are the correct ones

Key takeaways

• CPUs differ in endianness; exchanging raw integers between architectures corrupts values silently, with no error.
• Network byte order is big-endian by convention, and every protocol must use it on the wire.
• htons/htonl convert host to network; ntohs/ntohl convert back — on a big-endian machine they are no-ops, so the same source code works everywhere.
• Only multi-byte numeric fields need conversion: single bytes have no order, and byte arrays and strings must never be swapped.
• There is no htonf: floating point needs an explicit encoding (fixed-point, text, or a documented format), and 64-bit needs htobe64/be64toh.
• In higher-level languages, always state byte order explicitly (! in Python's struct); relying on the platform default is a latent portability bug.

🧪 Practice

1. The 16-bit value 0x1F90 is written to the wire. Show the byte sequence and explain what a little-endian machine reads if it skips conversion.
2. For a header containing a uint8 version, a uint16 port, a uint64 timestamp, and a 16-byte name, state which fields need conversion and which do not.
3. Explain why struct in_addr is passed to bind() without calling htonl on it.
4. Interview-style: A protocol works perfectly between two x86 servers and returns nonsense values when an ARM64 big-endian device joins. (Hint: the bug is not in the new device — find every place the code serialises an integer without stating an order.)

Graceful Shutdown and Socket Options

Theory

Closing a connection correctly is harder than opening one, because TCP connections are bidirectional and independently closable. A naive close() can discard data the peer has not yet read, and the failure is silent.

The distinction that matters:

• close() releases the file descriptor and, if this was the last reference, tears down both directions at once. Any unread data in the receive buffer causes the kernel to send an RST, which aborts the connection and may discard data already in flight.
• shutdown(fd, how) closes one direction while leaving the descriptor open:
  • SHUT_WR — "I have nothing more to send." Sends a FIN, so the peer's recv() returns 0. You can still read.
  • SHUT_RD — stop reading.
  • SHUT_RDWR — both, but the descriptor still exists.

This enables the half-close pattern, which is the correct way to end a request-response exchange where the response size is unknown: the client sends its request, calls shutdown(SHUT_WR) to signal "request complete", then reads until recv() returns 0. The server sees EOF, knows the request is finished, sends its response, and closes. HTTP's Connection: close and the classic cat file | nc host port both rely on this.

The four-way close is why TIME_WAIT exists (Chapter 7): each side sends its own FIN and acknowledges the other's. The side that closes first enters TIME_WAIT for twice the maximum segment lifetime — typically 60 seconds on Linux — so that delayed duplicate segments cannot be misinterpreted by a later connection reusing the same 4-tuple. A busy server that closes connections accumulates TIME_WAIT entries, which is why servers generally prefer to let clients close first.

The socket options worth knowing, grouped by what they fix:

Option	Level	Effect
SO_REUSEADDR	SOL_SOCKET	Rebind while old connections are in TIME_WAIT
SO_REUSEPORT	SOL_SOCKET	Multiple processes bind one port; kernel balances
SO_KEEPALIVE	SOL_SOCKET	Probe idle connections to detect dead peers
SO_RCVTIMEO / SO_SNDTIMEO	SOL_SOCKET	Timeouts on blocking calls
SO_LINGER	SOL_SOCKET	Control close() behaviour; l_linger=0 forces RST
SO_RCVBUF / SO_SNDBUF	SOL_SOCKET	Buffer sizes (bandwidth-delay product)
TCP_NODELAY	IPPROTO_TCP	Disable Nagle
TCP_USER_TIMEOUT	IPPROTO_TCP	Abort if data is unacknowledged for N ms
IPV6_V6ONLY	IPPROTO_IPV6	Off = one socket serves IPv4 and IPv6

SO_KEEPALIVE deserves emphasis because of what it does not do by default: the kernel's defaults wait two hours before the first probe. A connection to a machine that was unplugged will appear healthy for two hours unless you tune TCP_KEEPIDLE, TCP_KEEPINTVL, and TCP_KEEPCNT, or use TCP_USER_TIMEOUT, which bounds how long unacknowledged data may remain outstanding. Many applications implement their own heartbeat instead, precisely because it is portable and tunable.

Examples

close() vs shutdown()

  close(fd)                          shutdown(fd, SHUT_WR)
  +---------------------------+      +---------------------------+
  | fd is GONE                |      | fd still valid            |
  | both directions torn down |      | write side closed -> FIN  |
  | unread data -> RST        |      | read side still open      |
  +---------------------------+      +---------------------------+
   you cannot read the response       you CAN still read the response


 THE HALF-CLOSE PATTERN (how to end a request of unknown length)

   client                                server
     |---- request bytes ------------->|
     |---- request bytes ------------->|
     |---- shutdown(SHUT_WR) -> FIN -->|  recv() returns 0
     |                                 |  "request complete, now I respond"
     |<--- response bytes -------------|
     |<--- response bytes -------------|
     |<--- close() -> FIN -------------|  client's recv() returns 0
     |---- close() ------------------->|

   Without the half-close, neither side knows when the other finished.


 THE FOUR-WAY CLOSE AND TIME_WAIT

   A                                B
   |---- FIN ------------------->|   A: FIN_WAIT_1
   |<--- ACK --------------------|   A: FIN_WAIT_2, B: CLOSE_WAIT
   |                             |   (B may still send data here!)
   |<--- FIN --------------------|   B: LAST_ACK
   |---- ACK ------------------->|   A: TIME_WAIT (2 x MSL, ~60 s)
                                     B: CLOSED

   Whoever closes FIRST pays TIME_WAIT.
   -> a server that closes first accumulates thousands of them
   -> prefer to let the CLIENT close


 SO_KEEPALIVE DEFAULTS ARE NOT WHAT YOU WANT

   tcp_keepalive_time  = 7200 s   <- TWO HOURS before the first probe
   tcp_keepalive_intvl =   75 s
   tcp_keepalive_probes =   9     -> ~2h 11m to declare a peer dead

   For a service, set TCP_KEEPIDLE=60, or use TCP_USER_TIMEOUT.

#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>

void configure_server_socket(int fd) {
    int yes = 1;

    /* Rebind while old connections linger in TIME_WAIT. */
    setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof yes);

    /* Detect dead peers. The DEFAULTS wait two hours -- tune them. */
    setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, &yes, sizeof yes);
    int idle = 60, intvl = 10, cnt = 3;      /* dead in ~90 s, not ~2 hours */
    setsockopt(fd, IPPROTO_TCP, TCP_KEEPIDLE,  &idle,  sizeof idle);
    setsockopt(fd, IPPROTO_TCP, TCP_KEEPINTVL, &intvl, sizeof intvl);
    setsockopt(fd, IPPROTO_TCP, TCP_KEEPCNT,   &cnt,   sizeof cnt);

    /* Bound how long unacknowledged data may stay outstanding (Linux). */
    unsigned int user_timeout = 30000;       /* milliseconds */
    setsockopt(fd, IPPROTO_TCP, TCP_USER_TIMEOUT, &user_timeout,
               sizeof user_timeout);

    /* Latency-sensitive protocol: disable Nagle. */
    setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &yes, sizeof yes);

    /* Timeouts on blocking calls, so a stalled peer cannot hang a thread. */
    struct timeval tv = { .tv_sec = 30, .tv_usec = 0 };
    setsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof tv);
    setsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, &tv, sizeof tv);
}

/* The half-close pattern, client side. */
void send_request_and_read_all(int fd, const char *req, size_t len) {
    write(fd, req, len);

    /* "I am done sending." Sends FIN; the peer's recv() returns 0.
     * We can STILL READ -- which close() would have prevented. */
    shutdown(fd, SHUT_WR);

    char buf[4096];
    ssize_t n;
    size_t total = 0;
    while ((n = recv(fd, buf, sizeof buf, 0)) > 0) total += (size_t)n;
    /* n == 0 -> the server finished its response and closed. */
    printf("response: %zu bytes\n", total);

    close(fd);                                /* now release the descriptor */
}

/* SO_LINGER with l_linger = 0 forces an RST instead of a graceful FIN.
 * Use ONLY to shed connections under attack -- it DISCARDS unsent data. */
void abort_connection(int fd) {
    struct linger lg = { .l_onoff = 1, .l_linger = 0 };
    setsockopt(fd, SOL_SOCKET, SO_LINGER, &lg, sizeof lg);
    close(fd);                                /* sends RST, no TIME_WAIT */
}

import socket, struct, threading, time

# The half-close pattern, end to end.
def half_close_demo():
    srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    srv.bind(("127.0.0.1", 0)); srv.listen(1)
    port = srv.getsockname()[1]

    def server():
        conn, _ = srv.accept()
        request = b""
        while (chunk := conn.recv(4096)):     # read until the client's FIN
            request += chunk
        # recv() returned 0: the client called shutdown(SHUT_WR).
        print(f"  server: request complete ({len(request)} bytes), responding")
        conn.sendall(b"HTTP/1.1 200 OK\r\n\r\n" + b"z" * 500)
        conn.close()

    threading.Thread(target=server, daemon=True).start()

    c = socket.create_connection(("127.0.0.1", port))
    c.sendall(b"GET / HTTP/1.1\r\nHost: x\r\n\r\n")
    c.shutdown(socket.SHUT_WR)                # FIN: "request finished"
    response = b""
    while (chunk := c.recv(4096)):            # still readable after shutdown
        response += chunk
    print(f"  client: response {len(response)} bytes")
    c.close()
    srv.close()

half_close_demo()
#   server: request complete (28 bytes), responding
#   client: response 519 bytes
#   A close() instead of shutdown() would have made the response unreadable.

# Why servers prefer the client to close first.
def time_wait_cost(conns_per_sec, timewait_s=60):
    return conns_per_sec * timewait_s

print(f"\n{'conns/sec':>11}{'TIME_WAIT entries held':>26}")
for rate in (100, 1_000, 10_000):
    print(f"{rate:>11,}{time_wait_cost(rate):>26,}")
#   conns/sec    TIME_WAIT entries held
#         100                     6,000
#       1,000                    60,000
#      10,000                   600,000
#   Each consumes a 4-tuple and kernel memory. Whoever closes FIRST pays.

# Keepalive defaults are far too slow to be useful for a service.
def keepalive_detection(idle, intvl, cnt):
    return idle + intvl * cnt

print(f"\n{'configuration':<32}{'time to detect a dead peer':>28}")
print(f"{'kernel defaults (7200/75/9)':<32}"
      f"{keepalive_detection(7200,75,9)/60:>24,.0f} min")
print(f"{'tuned (60/10/3)':<32}"
      f"{keepalive_detection(60,10,3)/60:>24,.1f} min")
# configuration                     time to detect a dead peer
# kernel defaults (7200/75/9)                        131 min
# tuned (60/10/3)                                    1.5 min
#   A pulled cable looks like a healthy connection for two hours by default.

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 60)
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 10)
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, 3)

# SO_LINGER: the three distinct close behaviours.
behaviours = [
    ("default (l_onoff=0)",        "close() returns at once; kernel delivers "
                                   "remaining data in the background"),
    ("l_onoff=1, l_linger=N",      "close() BLOCKS up to N s waiting for the "
                                   "peer to acknowledge"),
    ("l_onoff=1, l_linger=0",      "close() sends RST immediately; UNSENT DATA "
                                   "IS DISCARDED, no TIME_WAIT"),
]
print()
for name, effect in behaviours:
    print(f"  {name:<26} {effect}")

# What each state in the close sequence means when you see it in `ss`.
states = {
    "FIN_WAIT_1": "we sent FIN, waiting for its ACK",
    "FIN_WAIT_2": "our FIN acked; waiting for THEIR FIN (peer may still send)",
    "CLOSE_WAIT": "THEY sent FIN; our app has not called close() -- A BUG if many",
    "LAST_ACK":   "we sent our FIN after theirs; awaiting the final ACK",
    "TIME_WAIT":  "we closed first; holding the 4-tuple for 2xMSL",
}
print()
for state, meaning in states.items():
    print(f"  {state:<12} {meaning}")
#   Many CLOSE_WAIT sockets is the classic signature of an application
#   that is leaking file descriptors by never calling close().

# Count connections by state -- the first thing to check on a busy server:
$ ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn
#   8412 ESTAB
#   1204 TIME_WAIT
#    317 CLOSE_WAIT       <- suspicious: the app is not calling close()
#      1 LISTEN

# CLOSE_WAIT that does not drain is a file-descriptor leak:
$ ss -tan state close-wait | wc -l
# 318
$ ls /proc/$(pgrep -n myserver)/fd | wc -l
# 1021
$ cat /proc/$(pgrep -n myserver)/limits | grep 'open files'
# Max open files  1024  1024  files
#   About to hit the limit: accept() will start returning EMFILE.

# TIME_WAIT tuning -- prefer SO_REUSEADDR over disabling safety mechanisms:
$ sysctl net.ipv4.tcp_fin_timeout net.ipv4.tcp_tw_reuse
# net.ipv4.tcp_fin_timeout = 60
# net.ipv4.tcp_tw_reuse = 2        # 2 = enabled for outbound connections only

# Kernel keepalive defaults -- note the first value is in SECONDS:
$ sysctl net.ipv4.tcp_keepalive_time net.ipv4.tcp_keepalive_intvl \
         net.ipv4.tcp_keepalive_probes
# net.ipv4.tcp_keepalive_time = 7200      <- 2 hours
# net.ipv4.tcp_keepalive_intvl = 75
# net.ipv4.tcp_keepalive_probes = 9

# Verify a process actually enabled keepalive:
$ ss -tion dst 93.184.216.34
# ESTAB 0 0 10.0.0.9:51234 93.184.216.34:443
#   timer:(keepalive,42sec,0)  <- keepalive armed, next probe in 42 s

# Observe the half-close in action -- note FIN then continued data:
$ sudo tcpdump -i lo -n 'port 8080' -c 6
# IP client > server: Flags [P.], length 28       <- request
# IP client > server: Flags [F.]                  <- shutdown(SHUT_WR)
# IP server > client: Flags [.], ack
# IP server > client: Flags [P.], length 519      <- response AFTER the FIN
# IP server > client: Flags [F.]
# IP client > server: Flags [.], ack

# Read all options on a live socket:
$ ss -tiom dst :443 | head -4
# ESTAB 0 0 10.0.0.9:51234 93.184.216.34:443
#   skmem:(r0,rb131072,t0,tb87040,...)  ts sack cubic wscale:7,7 rto:204
#          ^^^^^^^^^^^ receive buffer  ^^^^^^^ send buffer

Key takeaways

• close() tears down both directions and can send an RST that discards data; shutdown() closes one direction while keeping the descriptor usable.
• The half-close pattern — send, shutdown(SHUT_WR), then read until recv() returns 0 — is the correct way to delimit a request whose length the peer cannot predict.
• The four-way close means whoever closes first pays TIME_WAIT (~60 s), so busy servers should prefer to let clients close.
• Many CLOSE_WAIT sockets signal an application bug: the peer closed and your code never called close() — a file-descriptor leak.
• SO_KEEPALIVE defaults wait two hours; tune TCP_KEEPIDLE/TCP_KEEPINTVL/TCP_KEEPCNT or use TCP_USER_TIMEOUT, or implement an application heartbeat.
• SO_LINGER with l_linger=0 forces an RST and discards unsent data — a deliberate abort, not a normal close.
• Always set SO_RCVTIMEO/SO_SNDTIMEO so a stalled peer cannot hang a thread indefinitely.

🧪 Practice

1. Explain why shutdown(fd, SHUT_WR) is required for the cat file | nc host port pattern to work, and what happens if close() is used instead.
2. A server shows 40,000 sockets in TIME_WAIT and 300 in CLOSE_WAIT. Explain what each number indicates and which one is a bug.
3. Compute how long the kernel's default keepalive settings take to declare a peer dead, and give settings that detect it within two minutes.
4. Interview-style: A client uploads a file, the server reports success, and the last few kilobytes are missing. (Hint: which call did the client make when it finished writing, and what does that call do to data still sitting in the receive buffer?)

<a id="92-concurrency-models"></a>

9.2 Concurrency Models

A server that handles one connection at a time is useless: this subchapter traces the four approaches to handling many at once, and why each one replaced the last.

Process-per-Connection Servers

Theory

The problem is immediate. The accept() loop from 9.1 handles one connection, finishes with it, then accepts the next. If one client takes ten seconds, every other client waits ten seconds. Any server that does real work needs concurrency.

The oldest answer is the simplest: fork() a new process for every connection. The parent accepts and immediately forks; the child handles that one connection and exits; the parent loops back to accept().

Its appeal is genuine and worth stating, because it explains why this model survived for decades and still appears in security-sensitive software:

• Total isolation. Each connection has its own address space. A crash, a memory corruption, or a runaway allocation affects one client. inetd, older Apache, qmail, and OpenSSH's privilege-separated model all rely on this.
• No shared-state bugs. There is nothing to race on, so no locks, no data races, no reasoning about memory ordering. The code is ordinary sequential code.
• Security boundaries are enforced by the kernel, not by programmer discipline. A compromised child cannot read another connection's memory.

Its cost is equally real, and it is why the model was eventually replaced:

• Creation is expensive. A fork() costs roughly 0.5–2 ms even with copy-on-write, plus page-table setup. At 1,000 connections per second, that alone consumes most of a core.
• Memory per connection is large. Copy-on-write helps, but each process still needs its own page tables, kernel structures, and stack — on the order of 1–10 MB of real cost.
• Context switching is expensive, because it involves changing the address space and flushing TLB entries.

The essential fork() detail is that it returns twice: 0 in the child and the child's PID in the parent. And each process must close the descriptor it does not need — the parent closes the connection socket, the child closes the listening socket. Forgetting this leaks descriptors and, worse, means the connection is never fully closed because a reference remains open in the other process.

Zombie processes are the other classic pitfall. A terminated child remains in the process table until the parent reaps it with wait(). The standard fix is to handle SIGCHLD or set it to SIG_IGN so the kernel reaps automatically.

Pre-forking is the practical refinement: create a pool of worker processes at start-up, all blocked in accept() on the same listening socket, and let the kernel distribute connections. This removes per-connection fork cost while keeping isolation — and introduces the thundering herd problem discussed later in this subchapter.

Examples

Process-per-connection

  parent (listening fd 3)
     |
   accept() -> fd 4
     |
   fork() -----------------------+
     |                           |
   PARENT                      CHILD
   close(4)  <- does not need   close(3)  <- does not need
   loop to accept()              handle fd 4 to completion
     |                           exit()
   accept() -> fd 5              |
   fork() ---------------+       v
     |                   |    reaped by SIGCHLD handler
   ...                CHILD     (otherwise: ZOMBIE)

  Each child has its OWN address space:
   +-------------+  +-------------+  +-------------+
   | child A     |  | child B     |  | child C     |
   | heap, stack |  | heap, stack |  | heap, stack |
   | fd 4        |  | fd 5        |  | fd 6        |
   +-------------+  +-------------+  +-------------+
   A segfault in B does not touch A or C. That is the whole value proposition.


 PRE-FORKING: pay the fork cost once, at start-up

   parent creates listening fd, then forks N workers BEFORE any client arrives

        [listening fd 3]
         /    |    |    \
    worker worker worker worker      all blocked in accept() on fd 3
       1     2     3     4
                                     kernel hands each connection to one
                                     -> no per-connection fork cost
                                     -> but see "thundering herd" below

#include <sys/socket.h>
#include <sys/wait.h>
#include <netinet/in.h>
#include <signal.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>

static void handle_connection(int fd) {
    char buf[1024];
    ssize_t n = recv(fd, buf, sizeof buf, 0);
    if (n > 0) send(fd, buf, (size_t)n, 0);      /* echo */
    close(fd);
}

int main(void) {
    /* Reap children automatically; otherwise every exited child becomes
     * a ZOMBIE occupying a process-table slot until the parent wait()s. */
    signal(SIGCHLD, SIG_IGN);

    int listen_fd = socket(AF_INET, SOCK_STREAM, 0);
    int yes = 1;
    setsockopt(listen_fd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof yes);

    struct sockaddr_in addr = { .sin_family = AF_INET,
                                .sin_port = htons(8080),
                                .sin_addr.s_addr = htonl(INADDR_ANY) };
    bind(listen_fd, (struct sockaddr *)&addr, sizeof addr);
    listen(listen_fd, 128);

    for (;;) {
        int conn_fd = accept(listen_fd, NULL, NULL);
        if (conn_fd < 0) continue;

        pid_t pid = fork();          /* RETURNS TWICE: 0 in child, pid in parent */

        if (pid == 0) {
            /* ---- CHILD ---- */
            close(listen_fd);        /* child does not accept; MUST close it, or
                                      * the listening socket leaks into every
                                      * child and cannot be rebound later */
            handle_connection(conn_fd);
            _exit(0);                /* _exit, not exit: skip parent's atexit
                                      * handlers and buffer flushes */
        }
        /* ---- PARENT ---- */
        close(conn_fd);              /* parent does not serve; MUST close, or the
                                      * connection never truly closes (the child's
                                      * close() leaves the parent's ref open) */
    }
}

import os, socket, signal, time, resource

# Measure what fork() actually costs -- this is the model's core limitation.
def measure_fork_cost(n=200):
    signal.signal(signal.SIGCHLD, signal.SIG_IGN)   # auto-reap
    t0 = time.perf_counter()
    for _ in range(n):
        pid = os.fork()
        if pid == 0:
            os._exit(0)                             # child does nothing
    elapsed = time.perf_counter() - t0
    return elapsed / n * 1000                        # ms per fork

per_fork_ms = measure_fork_cost()
print(f"fork() cost: {per_fork_ms:.3f} ms each")
# fork() cost: 0.412 ms each

# What that means for throughput:
print(f"\n{'conns/sec':>11}{'CPU spent forking':>20}")
for rate in (100, 1_000, 10_000):
    cpu_frac = rate * per_fork_ms / 1000
    print(f"{rate:>11,}{cpu_frac:>18.1%}")
#   conns/sec   CPU spent forking
#         100               4.1%
#       1,000              41.2%
#      10,000             412.0%
#   At 10k connections/sec, forking alone needs four cores before your
#   application does any work at all.

# Memory: copy-on-write helps, but per-process overhead is not free.
def memory_model(connections, rss_mb_per_proc=8, shared_fraction=0.7):
    private = rss_mb_per_proc * (1 - shared_fraction)
    return connections * private, connections * rss_mb_per_proc

print(f"\n{'connections':>13}{'real (CoW)':>14}{'naive estimate':>17}")
for n in (100, 1_000, 10_000):
    real, naive = memory_model(n)
    print(f"{n:>13,}{real:>11,.0f} MB{naive:>14,.0f} MB")
#   connections    real (CoW)   naive estimate
#           100        240 MB          800 MB
#         1,000      2,400 MB        8,000 MB
#        10,000     24,000 MB       80,000 MB
#   Even with 70% sharing, 10k connections needs 24 GB. This is the wall.

# The isolation benefit, stated concretely.
print("\nWhen a connection handler segfaults:")
print("  process-per-connection : ONE client sees a reset; server keeps running")
print("  thread-per-connection  : the WHOLE process dies; all clients dropped")
print("  event loop             : the WHOLE process dies; all clients dropped")
#   This is why OpenSSH, qmail, and browser tab sandboxes still use processes.

# Pre-forking: pay the cost once.
def preforked_server(workers=4, port=8080):
    srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    srv.bind(("0.0.0.0", port))
    srv.listen(128)

    for i in range(workers):
        if os.fork() == 0:
            # Child: loop forever on the INHERITED listening socket.
            while True:
                conn, addr = srv.accept()     # all workers block here
                conn.sendall(b"hello\n")
                conn.close()
            os._exit(0)
    return f"{workers} workers sharing one listening socket"

print(f"\n{preforked_server.__doc__ or 'pre-fork'}: fork cost paid once at boot")
#   No per-connection fork. The kernel distributes accepts among workers.
#   Cost: all workers wake when a connection arrives -> thundering herd.

# Zombie processes: what happens without reaping.
print("\nWithout SIGCHLD handling:")
print("  each exited child stays in the process table as a ZOMBIE")
print("  -> `ps` shows <defunct>")
print("  -> eventually EAGAIN from fork(): process table exhausted")

# Watch processes appear per connection:
$ ps --forest -o pid,ppid,stat,cmd -C myserver
#   PID  PPID STAT CMD
#  4821     1 Ss   ./myserver
#  4890  4821 S     \_ ./myserver      <- one child per connection
#  4891  4821 S     \_ ./myserver
#  4892  4821 S     \_ ./myserver

# Zombies, if SIGCHLD is not handled:
$ ps -eo pid,stat,cmd | grep defunct | head -3
# 4893 Z    [myserver] <defunct>
# 4894 Z    [myserver] <defunct>
#      ^ Z = zombie: exited, but the parent never wait()ed

$ ps -eo stat | grep -c '^Z'
# 1204        <- approaching the process-table limit
$ sysctl kernel.pid_max
# kernel.pid_max = 4194304

# Measure fork cost directly:
$ python3 -c "
import os,time,signal
signal.signal(signal.SIGCHLD, signal.SIG_IGN)
t=time.perf_counter()
for _ in range(500):
    if os.fork()==0: os._exit(0)
print(f'{(time.perf_counter()-t)/500*1000:.3f} ms per fork')"
# 0.418 ms per fork

# Real memory cost with copy-on-write accounted for:
$ ps -o pid,rss,cmd -C myserver | head -4
#   PID   RSS CMD
#  4821  8192 ./myserver
#  4890  8320 ./myserver       <- RSS counts SHARED pages in every process
$ smem -c 'pid command uss pss rss' -P myserver | head -4
#   PID Command                USS      PSS      RSS
#  4890 ./myserver            2432     3180     8320
#        ^^^ USS = the genuinely PRIVATE cost per connection

# Pre-forked servers: many processes, one listening socket:
$ ss -tlnp | grep 8080
# LISTEN 0 128 0.0.0.0:8080 0.0.0.0:* users:(("worker",pid=101,fd=3),
#                                             ("worker",pid=102,fd=3),
#                                             ("worker",pid=103,fd=3))
#   One socket, shared by inheritance across four processes.

Key takeaways

• fork() per connection gives complete isolation: a crash, leak, or compromise affects exactly one client, enforced by the kernel rather than by programmer discipline.
• It requires no locks and no shared-state reasoning, so handler code is ordinary sequential code.
• The costs are ~0.5–2 ms per fork, 1–10 MB of real memory per connection, and expensive address-space context switches.
• fork() returns twice; the parent must close the connection socket and the child must close the listening socket, or descriptors leak and connections never fully close.
• Unreaped children become zombies — handle SIGCHLD or set it to SIG_IGN.
• Pre-forking moves the fork cost to start-up while keeping isolation, at the price of the thundering herd.
• The model remains correct where isolation matters more than density: OpenSSH, qmail, and browser process sandboxes.

🧪 Practice

1. Explain what each of the parent and child must close after fork(), and what goes wrong if either is omitted.
2. At 0.5 ms per fork, compute the CPU fraction consumed by forking alone at 500, 2,000, and 8,000 connections per second.
3. Describe how a zombie process arises and give two ways to prevent it.
4. Interview-style: Why does OpenSSH still use a process per connection when event loops are far more efficient? (Hint: ask what the server is protecting against, and which model enforces that boundary without trusting the code to be bug-free.)

Thread-per-Connection Servers

Theory

Threads are the obvious refinement. A thread is a separate execution context inside the same address space: its own stack, registers, and program counter, but shared heap, globals, and file descriptors.

Sharing the address space removes the two biggest costs of the process model:

• Creation is roughly 10× cheaper — tens of microseconds rather than hundreds, because there are no page tables to duplicate.
• Memory per connection is far lower — the dominant cost is the thread stack, typically 8 MB of virtual address space on Linux but only the pages actually touched in physical memory, often tens of kilobytes.
• Context switches are cheaper because the address space does not change, so the TLB is not flushed.

But sharing the address space is also exactly what creates the model's problems, and they are not incidental — they are the same trade-off in both directions:

• Data races. Any shared mutable state needs synchronisation. Getting this wrong produces bugs that appear only under load, on particular hardware, in production.
• No fault isolation. One thread's segfault or uncaught exception kills the entire process and every connection it was serving. The thing the process model bought you is precisely what you gave up.
• Lock contention can make a threaded server slower than a single-threaded one, because threads spend their time waiting on each other rather than doing work.

Even with cheaper threads, the model has a hard ceiling. Each thread needs a stack, a kernel task structure, and a scheduler entry. At 10,000 threads the scheduler itself becomes a bottleneck — the run queue is long, cache locality is destroyed, and context switching dominates. This is a direct statement of the C10K problem discussed later.

Thread pools are the standard refinement, and for the same reason pre-forking was: creation cost is paid once. A fixed pool of workers pulls connections from a queue. This bounds resource use and makes the server's behaviour under overload predictable — the queue grows rather than the thread count, which is the property you want, because unbounded thread creation turns a traffic spike into a crash.

The pool size question has a useful rule of thumb. For CPU-bound work, the optimum is near the core count, since more threads only add switching overhead. For I/O-bound work — which most network servers are — threads spend most of their time blocked, so the useful count is roughly cores × (1 + wait_time/service_time). A handler that waits 90 ms on a database for every 10 ms of computation can usefully run about 10 threads per core.

Virtual threads (Java 21+, Go's goroutines) change the arithmetic substantially: they are scheduled in userspace onto a small pool of OS threads, cost a few kilobytes each, and make thread-per-connection viable at hundreds of thousands of connections. They keep the sequential programming model while getting event-loop-like efficiency, which is why Go servers are written this way.

Examples

Processes vs threads: what is shared

  PROCESS MODEL                      THREAD MODEL
  +----------------+                 +--------------------------------+
  | process A      |                 |          one process           |
  |  heap  stack   |                 |  SHARED: heap, globals, fds    |
  |  fds           |                 |  +--------+--------+--------+  |
  +----------------+                 |  |thread 1|thread 2|thread 3|  |
  +----------------+                 |  | stack  | stack  | stack  |  |
  | process B      |                 |  +--------+--------+--------+  |
  |  heap  stack   |                 +--------------------------------+
  +----------------+
   isolated: a crash                  shared: a crash in ANY thread
   kills one connection               kills EVERY connection


 THE POOL, and why unbounded threads are dangerous

  UNBOUNDED                           POOL
  10,000 clients -> 10,000 threads    10,000 clients -> 16 threads + a queue
  -> 10,000 stacks                    -> bounded memory
  -> scheduler thrashes               -> predictable behaviour
  -> OOM or collapse                  -> queue grows; latency rises gracefully

  A traffic spike should make you SLOWER, not DEAD.


 POOL SIZING

   CPU-bound:  threads ~= cores
               (more only adds context-switch overhead)

   I/O-bound:  threads ~= cores x (1 + wait/service)
               10 ms CPU + 90 ms DB wait on 8 cores
               = 8 x (1 + 90/10) = 80 threads

#include <pthread.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

static void *worker(void *arg) {
    int fd = *(int *)arg;
    free(arg);                       /* the fd was heap-allocated per thread:
                                      * passing &conn_fd directly would be a
                                      * RACE -- the loop overwrites it */
    char buf[1024];
    ssize_t n = recv(fd, buf, sizeof buf, 0);
    if (n > 0) send(fd, buf, (size_t)n, 0);
    close(fd);
    return NULL;
}

int main(void) {
    int listen_fd = socket(AF_INET, SOCK_STREAM, 0);
    int yes = 1;
    setsockopt(listen_fd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof yes);
    struct sockaddr_in addr = { .sin_family = AF_INET,
                                .sin_port = htons(8080),
                                .sin_addr.s_addr = htonl(INADDR_ANY) };
    bind(listen_fd, (struct sockaddr *)&addr, sizeof addr);
    listen(listen_fd, 128);

    /* Smaller stacks let you run far more threads before exhausting memory. */
    pthread_attr_t attr;
    pthread_attr_init(&attr);
    pthread_attr_setstacksize(&attr, 256 * 1024);       /* 256 KB, not 8 MB */
    pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);
    /* DETACHED: the thread cleans itself up. Without this you must
     * pthread_join() every thread or leak its resources. */

    for (;;) {
        int *conn_fd = malloc(sizeof *conn_fd);
        *conn_fd = accept(listen_fd, NULL, NULL);
        if (*conn_fd < 0) { free(conn_fd); continue; }

        pthread_t tid;
        if (pthread_create(&tid, &attr, worker, conn_fd) != 0) {
            /* Thread creation FAILS under load -- handle it, do not ignore it. */
            close(*conn_fd);
            free(conn_fd);
        }
    }
}

import socket, threading, time, concurrent.futures, os

# Thread creation cost, versus the fork cost measured earlier.
def measure_thread_cost(n=500):
    t0 = time.perf_counter()
    threads = [threading.Thread(target=lambda: None) for _ in range(n)]
    for t in threads: t.start()
    for t in threads: t.join()
    return (time.perf_counter() - t0) / n * 1000

print(f"thread creation: {measure_thread_cost():.3f} ms each")
print(f"fork (measured earlier): ~0.412 ms each")
# thread creation: 0.048 ms each
# fork (measured earlier): ~0.412 ms each
#   Roughly 10x cheaper -- the model's whole justification.

# Memory: virtual reservation vs what is actually resident.
def thread_memory(threads, stack_virtual_mb=8, stack_touched_kb=32):
    return (threads * stack_virtual_mb,
            threads * stack_touched_kb / 1024)

print(f"\n{'threads':>9}{'virtual':>13}{'resident':>12}")
for n in (100, 1_000, 10_000):
    v, r = thread_memory(n)
    print(f"{n:>9,}{v:>10,.0f} GB{r:>9,.0f} MB")
#    threads      virtual     resident
#        100         1 GB        3 MB
#      1,000         8 GB       31 MB
#     10,000        78 GB      313 MB
#   Virtual address space is cheap; the real constraint is the SCHEDULER.

# Why a thread pool, not unbounded threads.
class ThreadPoolServer:
    def __init__(self, workers=16, queue_limit=1000):
        self.pool = concurrent.futures.ThreadPoolExecutor(max_workers=workers)
        self.queue_limit = queue_limit
        self.pending = 0
        self.lock = threading.Lock()
        self.rejected = 0

    def submit(self, conn):
        with self.lock:
            if self.pending >= self.queue_limit:
                self.rejected += 1
                conn.close()                 # SHED LOAD deliberately
                return False
            self.pending += 1
        self.pool.submit(self._run, conn)
        return True

    def _run(self, conn):
        try:
            data = conn.recv(4096)
            if data: conn.sendall(data)
        finally:
            conn.close()
            with self.lock:
                self.pending -= 1

# Pool sizing for I/O-bound work -- the formula that actually matters.
def optimal_pool(cores, service_ms, wait_ms):
    return round(cores * (1 + wait_ms / service_ms))

print(f"\n{'workload':<34}{'cores':>7}{'threads':>9}")
for name, svc, wait in (("CPU-bound (compression)", 100, 0),
                        ("balanced", 50, 50),
                        ("I/O-bound (DB query)", 10, 90),
                        ("very I/O-bound (slow API)", 5, 495)):
    print(f"{name:<34}{8:>7}{optimal_pool(8, svc, wait):>9}")
# workload                            cores  threads
# CPU-bound (compression)                 8        8
# balanced                                8       16
# I/O-bound (DB query)                    8       80
# very I/O-bound (slow API)               8      800
#   Note the last row: 800 OS threads is where this model starts to break,
#   and where event loops or virtual threads become the right answer.

# The isolation you gave up, demonstrated.
def crash_blast_radius():
    return {
        "process-per-connection": "1 connection lost, server survives",
        "thread-per-connection":  "ENTIRE PROCESS dies, all connections lost",
        "thread pool":            "ENTIRE PROCESS dies, all connections lost",
    }
print()
for model, effect in crash_blast_radius().items():
    print(f"  {model:<26} {effect}")

# The race that malloc'ing the fd prevents, shown in Python terms.
def racy_handoff_demo():
    """Passing a loop variable by reference to a thread is the classic bug."""
    results = []
    def worker(get_value):
        results.append(get_value())
    threads = []
    shared = [0]
    for i in range(3):
        shared[0] = i                                  # overwritten each loop
        t = threading.Thread(target=worker, args=(lambda: shared[0],))
        threads.append(t); t.start()
    for t in threads: t.join()
    return results

print(f"\nracy handoff produced: {sorted(racy_handoff_demo())}")
# racy handoff produced: [2, 2, 2]     <- expected [0, 1, 2]
#   Each thread read the variable AFTER the loop had advanced it.
#   In C this is why the fd is heap-allocated per connection.

# Count threads in a process -- the number that hits the scheduler:
$ ps -o nlwp= -p $(pgrep -n myserver)
# 847
$ ls /proc/$(pgrep -n myserver)/task | wc -l
# 847

# Per-thread CPU: find the one that is hot:
$ top -H -p $(pgrep -n myserver) -b -n1 | head -10
#   PID USER  PR NI  %CPU  COMMAND
#  4821 www   20  0  98.2  myserver      <- one thread pegged: likely a lock
#  4822 www   20  0   0.3  myserver

# Context-switch rate: the cost of too many threads:
$ pidstat -w -p $(pgrep -n myserver) 1 3
# UID  PID   cswch/s  nvcswch/s  Command
# 33   4821   1204.0    8912.0   myserver
#                       ^^^^^^^^ INVOLUNTARY: threads preempted mid-work
#                                high values mean the scheduler is thrashing

# Thread stack size -- the default reserves a lot of virtual space:
$ ulimit -s
# 8192        <- 8 MB per thread, VIRTUAL
$ grep -c 'stack' /proc/$(pgrep -n myserver)/maps
# 847

# Virtual vs resident, to see what threads actually cost:
$ ps -o pid,vsz,rss,nlwp -p $(pgrep -n myserver)
#   PID     VSZ    RSS NLWP
#  4821 7012352 284160  847
#       ^^^^^^^ 7 GB virtual   ^^^^^^ 284 MB resident
#   Virtual is a reservation; resident is the real cost.

# Lock contention -- where threaded servers actually lose:
$ perf record -e 'syscalls:sys_enter_futex' -p $(pgrep -n myserver) -- sleep 5
$ perf report --stdio | head -5
#  62.41%  myserver  [.] pthread_mutex_lock
#   ^ the majority of time spent waiting, not working

# Thread limits you will hit:
$ sysctl kernel.threads-max
# kernel.threads-max = 255432
$ ulimit -u
# 63216       <- per-user process/thread limit; hit this and pthread_create fails

Key takeaways

• Threads share one address space, making creation ~10× cheaper than fork() and context switches cheaper because the TLB is not flushed.
• The same sharing removes fault isolation: one thread's crash kills every connection in the process.
• Shared mutable state requires synchronisation, and lock contention can make a threaded server slower than a serial one.
• Thread stacks reserve large virtual space (8 MB default) but only touched pages are resident — the real ceiling is the scheduler, not memory.
• Thread pools bound resource use so that overload increases latency instead of causing collapse; unbounded thread creation turns a spike into a crash.
• Pool sizing: ≈ cores for CPU-bound, ≈ cores × (1 + wait/service) for I/O-bound work.
• Virtual threads and goroutines are userspace-scheduled and cost kilobytes, making thread-per-connection viable again at very high connection counts.

🧪 Practice

1. List two advantages and two disadvantages of threads over processes for a connection handler.
2. A handler uses 15 ms of CPU and waits 135 ms on a remote API, on a 4-core machine. Compute a sensible pool size.
3. Explain why passing &conn_fd directly to pthread_create is a bug and what to do instead.
4. Interview-style: Your threaded server handles 2,000 connections comfortably and collapses at 5,000, although CPU sits at 40%. (Hint: if the cores are idle, the threads are not running — look at what they are waiting on and how often they are being switched.)

select and poll Multiplexing

Theory

Both previous models spend most of their resources waiting. A thread blocked in recv() consumes a stack, a kernel task structure, and a scheduler slot, while doing nothing at all. With 10,000 mostly-idle connections — which is the normal case for chat, long-polling, and keep-alive HTTP — that is an enormous amount of machinery devoted to idleness.

I/O multiplexing inverts the arrangement. Instead of one thread per connection blocking on one descriptor, one thread watches many descriptors and is told which ones are ready. It then services exactly those, in one thread, without blocking. Concurrency comes from interleaving rather than from parallelism.

The key insight is that a network server is almost never CPU-bound; it is readiness-bound. If the kernel tells you which sockets have data, a single thread can service thousands, because each one needs only microseconds of actual work.

select() was the first such interface (BSD, 1983). You pass three fd_set bitmaps — read, write, exception — plus a timeout. It blocks until at least one descriptor is ready, then modifies the sets in place to indicate which. Two limitations are severe:

• FD_SETSIZE is 1024 on most systems, and it is a compile-time constant. Descriptor 1024 or above cannot be watched at all, and passing one is undefined behaviour, not an error — often a stack corruption.
• The sets are destroyed by each call, so you must rebuild them every iteration.

poll() (POSIX, 1986) fixes both: it takes an array of struct pollfd, so there is no fixed limit, and it separates the requested events (events) from the returned ones (revents), so the array is reusable without rebuilding.

Both, however, share the defect that ultimately motivated epoll: they are O(n) in the number of watched descriptors, twice over. The kernel scans every descriptor to determine readiness, and your code scans every descriptor to find which ones the kernel flagged. With 10,000 connections of which 10 are active, you do 10,000 units of work to discover 10 events — and you do it on every single call, because the kernel keeps no state between calls.

	select	poll	epoll/kqueue
Max descriptors	1024 (FD_SETSIZE)	Unlimited	Unlimited
Per-call cost	O(n)	O(n)	O(active)
Rebuild each call	Yes	No	No
Kernel keeps state	No	No	Yes
Portability	Everywhere	POSIX	Linux / BSD only

The event structure itself is worth internalising, because it is the same in every model that follows and in every async runtime: wait for readiness, then perform non-blocking I/O on exactly the ready descriptors, then loop. Sockets must be set non-blocking, because readiness can be spurious — a packet may be discarded by a checksum failure between the readiness notification and your read(), and on a blocking socket that would hang the entire server.

select remains useful for two reasons: it is portable to everything, and its timeout has microsecond resolution. For a handful of descriptors it is perfectly adequate, and it is often the right choice in a small client.

Examples

One thread, many connections

  BLOCKING MODEL                     MULTIPLEXED MODEL
  thread 1 -> recv(fd 4) BLOCKED
  thread 2 -> recv(fd 5) BLOCKED          +-------------------+
  thread 3 -> recv(fd 6) BLOCKED          |   one thread      |
  ...                                     |                   |
  thread N -> recv(fd N) BLOCKED          | select/poll on    |
                                          | fds 4,5,6,...,N   |
  N stacks, N scheduler entries,          |        |          |
  all doing nothing                       |   "5 and 9 ready" |
                                          |        v          |
                                          | recv(5); recv(9)  |
                                          +-------------------+
                                           1 stack, services only
                                           what is actually ready


 WHY select/poll ARE O(n)

   watching 10,000 fds, 10 are active

   your code:  build the set of 10,000        <- O(n)
   kernel:     scan all 10,000 for readiness  <- O(n)
   kernel:     return "10 are ready"
   your code:  scan all 10,000 to find which  <- O(n)
               -> 30,000 units of work to discover 10 events
               -> repeated on EVERY call, because no state is kept


 THE UNIVERSAL EVENT LOOP SHAPE

   loop forever:
       ready = wait_for_readiness(watched, timeout)   <- blocks here
       for fd in ready:
           if fd is the listener:  accept() a new connection
           elif readable:          read non-blocking, buffer it
           elif writable:          write pending output
           if closed or error:     clean up and unwatch

   Sockets MUST be non-blocking: readiness can be spurious, and a
   blocking read on a falsely-ready socket stalls EVERY connection.

#include <sys/select.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <fcntl.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>

static void set_nonblocking(int fd) {
    int flags = fcntl(fd, F_GETFL, 0);
    fcntl(fd, F_SETFL, flags | O_NONBLOCK);   /* MANDATORY in an event loop */
}

int main(void) {
    int listen_fd = socket(AF_INET, SOCK_STREAM, 0);
    int yes = 1;
    setsockopt(listen_fd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof yes);
    struct sockaddr_in addr = { .sin_family = AF_INET,
                                .sin_port = htons(8080),
                                .sin_addr.s_addr = htonl(INADDR_ANY) };
    bind(listen_fd, (struct sockaddr *)&addr, sizeof addr);
    listen(listen_fd, 128);
    set_nonblocking(listen_fd);

    fd_set master;                    /* the authoritative set we maintain */
    FD_ZERO(&master);
    FD_SET(listen_fd, &master);
    int fd_max = listen_fd;

    for (;;) {
        /* select() DESTROYS the sets it is given, so copy them every call.
         * This copy is itself part of why select is O(n). */
        fd_set read_fds = master;
        struct timeval tv = { .tv_sec = 1, .tv_usec = 0 };

        int n = select(fd_max + 1, &read_fds, NULL, NULL, &tv);
        if (n < 0) { perror("select"); break; }
        if (n == 0) continue;                       /* timeout: do periodic work */

        /* Scan EVERY descriptor to find the ready ones -- the second O(n). */
        for (int fd = 0; fd <= fd_max; fd++) {
            if (!FD_ISSET(fd, &read_fds)) continue;

            if (fd == listen_fd) {
                int conn = accept(listen_fd, NULL, NULL);
                if (conn < 0) continue;
                if (conn >= FD_SETSIZE) {           /* fd >= 1024 CANNOT be watched.
                                                     * Using it is UNDEFINED
                                                     * BEHAVIOUR, not an error. */
                    fprintf(stderr, "fd %d exceeds FD_SETSIZE\n", conn);
                    close(conn);
                    continue;
                }
                set_nonblocking(conn);
                FD_SET(conn, &master);
                if (conn > fd_max) fd_max = conn;
            } else {
                char buf[4096];
                ssize_t r = recv(fd, buf, sizeof buf, 0);
                if (r <= 0) {                        /* 0 = peer closed; <0 = error */
                    close(fd);
                    FD_CLR(fd, &master);
                } else {
                    send(fd, buf, (size_t)r, 0);     /* echo (ignores partial sends) */
                }
            }
        }
    }
    return 0;
}

#include <poll.h>
#include <sys/socket.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>

/* poll(): no FD_SETSIZE limit, and the array is REUSABLE because
 * `events` (what we want) is separate from `revents` (what happened). */
int poll_server(int listen_fd) {
    struct pollfd fds[4096];
    nfds_t nfds = 1;

    fds[0].fd = listen_fd;
    fds[0].events = POLLIN;               /* we want: readable */

    for (;;) {
        int n = poll(fds, nfds, 1000);    /* timeout in MILLISECONDS, not timeval */
        if (n <= 0) continue;

        for (nfds_t i = 0; i < nfds; i++) {
            if (fds[i].revents == 0) continue;        /* nothing for this one */

            /* Always check errors BEFORE data: a socket can be both. */
            if (fds[i].revents & (POLLERR | POLLHUP | POLLNVAL)) {
                close(fds[i].fd);
                fds[i] = fds[--nfds];                  /* compact the array */
                continue;
            }

            if (fds[i].revents & POLLIN) {
                if (fds[i].fd == listen_fd) {
                    int conn = accept(listen_fd, NULL, NULL);
                    if (conn >= 0 && nfds < 4096) {
                        fds[nfds].fd = conn;
                        fds[nfds].events = POLLIN;
                        fds[nfds].revents = 0;
                        nfds++;
                    }
                } else {
                    char buf[4096];
                    ssize_t r = recv(fds[i].fd, buf, sizeof buf, 0);
                    if (r <= 0) {
                        close(fds[i].fd);
                        fds[i] = fds[--nfds];
                    } else {
                        send(fds[i].fd, buf, (size_t)r, 0);
                    }
                }
            }
        }
    }
}

import select, socket, time

# A complete multiplexed echo server in the canonical shape.
def poll_server(port=8080):
    srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    srv.bind(("0.0.0.0", port)); srv.listen(128)
    srv.setblocking(False)                       # non-blocking is MANDATORY

    poller = select.poll()
    poller.register(srv.fileno(), select.POLLIN)
    conns = {}                                    # fd -> socket
    outbox = {}                                   # fd -> pending bytes

    while True:
        for fd, event in poller.poll(1000):      # timeout in milliseconds
            if event & (select.POLLERR | select.POLLHUP | select.POLLNVAL):
                poller.unregister(fd)
                conns.pop(fd).close()
                continue

            if fd == srv.fileno():
                conn, _ = srv.accept()
                conn.setblocking(False)
                conns[conn.fileno()] = conn
                poller.register(conn.fileno(), select.POLLIN)

            elif event & select.POLLIN:
                conn = conns[fd]
                data = conn.recv(4096)
                if not data:                      # orderly shutdown
                    poller.unregister(fd)
                    conns.pop(fd).close()
                else:
                    outbox[fd] = outbox.get(fd, b"") + data
                    # Now interested in WRITABILITY too, so we can flush.
                    poller.modify(fd, select.POLLIN | select.POLLOUT)

            elif event & select.POLLOUT:
                conn = conns[fd]
                pending = outbox.get(fd, b"")
                if pending:
                    sent = conn.send(pending)     # may be PARTIAL
                    outbox[fd] = pending[sent:]
                if not outbox.get(fd):
                    poller.modify(fd, select.POLLIN)   # nothing left to write

# The O(n) cost that motivates epoll.
def scan_cost(watched, active, ns_per_fd=20):
    """Kernel scans all; your code scans all; the set is rebuilt each call."""
    kernel = watched * ns_per_fd
    userspace = watched * ns_per_fd
    return (kernel + userspace) / 1000, active

print(f"{'watched':>9}{'active':>8}{'us per call':>14}{'us per event':>15}")
for watched in (10, 100, 1_000, 10_000, 50_000):
    active = max(1, watched // 1000)
    us, act = scan_cost(watched, active)
    print(f"{watched:>9,}{act:>8}{us:>13.1f}{us/act:>15.1f}")
#   watched  active   us per call   us per event
#        10       1           0.4            0.4
#       100       1           4.0            4.0
#     1,000       1          40.0           40.0
#    10,000      10         400.0           40.0
#    50,000      50       2,000.0           40.0
#   Cost per USEFUL event stays flat, but per-call cost grows without bound.
#   At 50,000 fds you burn 2 ms of CPU per call just to find out what happened.

# The FD_SETSIZE cliff -- silent, not an error.
FD_SETSIZE = 1024
def select_safe(fd):
    if fd >= FD_SETSIZE:
        return f"fd {fd}: UNDEFINED BEHAVIOUR (stack corruption), not an error"
    return f"fd {fd}: ok"

print()
for fd in (3, 1023, 1024, 5000):
    print(f"  {select_safe(fd)}")
#   fd 3: ok
#   fd 1023: ok
#   fd 1024: UNDEFINED BEHAVIOUR (stack corruption), not an error
#   fd 5000: UNDEFINED BEHAVIOUR (stack corruption), not an error
#   This is why poll() exists and why select() must never be used on a
#   server that can exceed 1024 descriptors.

# Why non-blocking is mandatory: spurious readiness.
print("\nselect() says fd 7 is readable. You call recv(fd 7).")
print("  blocking socket   : if the data was discarded (bad checksum),")
print("                      recv() BLOCKS -> the entire server freezes")
print("  non-blocking socket: recv() returns EAGAIN -> loop continues")

# Watch a multiplexed server: ONE thread, thousands of connections.
$ ps -o nlwp= -p $(pgrep -n eventserver)
# 1
$ ss -tn state established '( sport = :8080 )' | wc -l
# 9847
#   One thread serving ~10,000 connections. That is the whole point.

# Which multiplexing syscall is it actually using?
$ sudo strace -c -p $(pgrep -n myserver) -f 2>&1 | head -8
# % time     seconds  usecs/call     calls    syscall
# ------ ----------- ----------- --------- ----------------
#  71.20    2.914021         291     10012 poll
#  18.44    0.754812          75      9987 recvfrom
#   ^^^^^ 291 us PER CALL: the O(n) scan dominating the workload

# Compare with an epoll-based server under the same load:
$ sudo strace -c -p $(pgrep -n nginx) -f 2>&1 | head -5
# % time     seconds  usecs/call     calls    syscall
#  12.10    0.048201           4     12044 epoll_wait
#                              ^ 4 us: O(active), not O(watched)

# See the actual fd sets being passed:
$ sudo strace -e trace=poll -p $(pgrep -n myserver) 2>&1 | head -2
# poll([{fd=3, events=POLLIN}, {fd=7, events=POLLIN}, ...], 9848, 1000) = 2
#                                                        ^^^^ 9848 entries
#                                                              copied EVERY call

# The FD_SETSIZE limit in the headers:
$ grep -r 'define __FD_SETSIZE' /usr/include/ 2>/dev/null
# /usr/include/bits/typesizes.h:#define __FD_SETSIZE  1024

# Raise the descriptor limit before a multiplexed server can be useful:
$ ulimit -n
# 1024
$ ulimit -n 65536 && ulimit -n
# 65536
$ cat /proc/sys/fs/file-max
# 9223372036854775807

Key takeaways

• Multiplexing replaces "one blocked thread per connection" with one thread watching many descriptors, because servers are readiness-bound rather than CPU-bound.
• select() is limited to FD_SETSIZE (1024) and exceeding it is undefined behaviour, not an error; it also destroys its sets, forcing a rebuild each call.
• poll() removes the limit and separates events from revents, making the array reusable.
• Both are O(n) twice over — the kernel scans all descriptors and so does your code — and keep no state between calls.
• Sockets must be non-blocking: readiness can be spurious, and one blocking call stalls every connection in the loop.
• The event loop shape — wait for readiness, service ready descriptors, repeat — is identical in every model that follows.
• select remains appropriate for portability and small descriptor counts; poll for moderate ones; beyond that you need epoll/kqueue.

🧪 Practice

1. Explain the two separate O(n) costs incurred by every select() call and why they persist across calls.
2. State what happens if a program calls select() on file descriptor 2000, and why this is worse than an error.
3. Explain why every socket in an event loop must be non-blocking, giving a concrete failure scenario.
4. Interview-style: A poll()-based server serving 20,000 mostly idle connections burns 60% CPU while handling almost no traffic. (Hint: count the work done per call versus the number of events returned, and ask what the kernel has to redo every time.)

epoll and kqueue

Theory

The defect in select and poll is structural: the kernel is told the entire watch list on every call, and therefore must examine all of it every time. Nothing is remembered. If you watch the same 10,000 sockets a thousand times per second, you describe those 10,000 sockets a thousand times per second.

epoll (Linux 2.5.44, 2002) and kqueue (FreeBSD 4.1, 2000) fix this by moving the watch list into the kernel. You register interest once; the kernel maintains the set and, crucially, maintains a ready list updated by the network stack as events occur. When you call epoll_wait(), the kernel returns the ready list directly — O(number of ready events), entirely independent of how many descriptors are being watched.

This is the change that made 10,000+ connections routine, and it is why every modern server and async runtime is built on one of these interfaces.

epoll has three calls:

• epoll_create1() — create an epoll instance, itself a file descriptor.
• epoll_ctl() — ADD, MOD, or DEL a descriptor and the events you care about. Called once per change, not once per loop.
• epoll_wait() — block until events occur; returns only the ready ones.

The genuinely important concept is the trigger mode:

Level-triggered (LT), the default and the same semantics as poll: report readiness as long as the condition holds. If 1,000 bytes are readable and you read 100, the next epoll_wait() reports it readable again. Forgiving, harder to misuse, and the right default.

Edge-triggered (ET): report only on a transition — when new data arrives. If you do not drain the socket completely, you will not be told again, and the connection silently hangs. ET requires that you loop reading until EAGAIN, and it mandates non-blocking sockets. In exchange it produces fewer wakeups, which matters at very high event rates.

	Level-triggered	Edge-triggered
Notifies	While the condition holds	Only on a change
Must drain fully	No	Yes, until EAGAIN
Risk	Extra wakeups	Silent stalls if you under-read
Non-blocking required	Recommended	Mandatory
Used by	nginx (default), most code	High-performance event loops

kqueue is the BSD and macOS equivalent, and is arguably the better design: a single kevent() call both registers interest and retrieves events, and it handles more than sockets — file changes, process exits, signals, and timers all flow through the same mechanism. epoll needs signalfd, timerfd, and inotify to cover the same ground.

Because these interfaces are platform-specific, portable code uses a wrapper — libevent, libev, or libuv (which powers Node.js) — that selects epoll, kqueue, or poll at runtime. Almost nobody writes directly against epoll outside of the runtimes themselves.

Examples

Where the watch list lives

  poll(): described in full, every call
     userspace                          kernel
     [10,000 pollfd entries] --copy--> scan all 10,000
                            <-returns- "3 are ready"
     ^ 10,000 entries crossing the boundary, every single call

  epoll: registered ONCE; kernel maintains a READY LIST
     userspace                          kernel
     epoll_ctl(ADD, fd) x 10,000 -----> +-------------------+
        (once, at registration)         | interest set      |
                                        |   (red-black tree)|
     epoll_wait() -------------------->  | ready list        |
                                        |   [fd 7][fd 12]   | <- updated by the
                 <-- returns 2 entries -+-------------------+    network stack
     ^ only the READY ones cross the boundary


 COST COMPARISON: 10,000 watched, 3 active

   select/poll : 10,000 units of work -> 3 events    (3,333 units per event)
   epoll       :      3 units of work -> 3 events    (    1 unit  per event)


 LEVEL-TRIGGERED vs EDGE-TRIGGERED

  1000 bytes arrive; your handler reads only 100.

  LEVEL-TRIGGERED                     EDGE-TRIGGERED
  epoll_wait -> fd readable           epoll_wait -> fd readable
  read 100 (900 remain)               read 100 (900 remain)
  epoll_wait -> fd readable AGAIN     epoll_wait -> ...BLOCKS...
  read 100 (800 remain)               ^ NO new data arrived, so no new EDGE.
  ...                                   The 900 bytes sit there forever.
  eventually drained                    THE CONNECTION HANGS SILENTLY.

  ET is correct only with:
     while ((n = read(fd, buf, size)) > 0) { process(buf, n); }
     if (n < 0 && errno == EAGAIN) { /* NOW it is genuinely drained */ }

#include <sys/epoll.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <fcntl.h>
#include <errno.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>

#define MAX_EVENTS 512

static void set_nonblocking(int fd) {
    int flags = fcntl(fd, F_GETFL, 0);
    fcntl(fd, F_SETFL, flags | O_NONBLOCK);
}

int main(void) {
    int listen_fd = socket(AF_INET, SOCK_STREAM, 0);
    int yes = 1;
    setsockopt(listen_fd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof yes);
    struct sockaddr_in addr = { .sin_family = AF_INET,
                                .sin_port = htons(8080),
                                .sin_addr.s_addr = htonl(INADDR_ANY) };
    bind(listen_fd, (struct sockaddr *)&addr, sizeof addr);
    listen(listen_fd, 512);
    set_nonblocking(listen_fd);

    /* The epoll instance is itself a file descriptor. */
    int ep = epoll_create1(0);

    struct epoll_event ev = { .events = EPOLLIN, .data.fd = listen_fd };
    epoll_ctl(ep, EPOLL_CTL_ADD, listen_fd, &ev);   /* registered ONCE */

    struct epoll_event events[MAX_EVENTS];

    for (;;) {
        /* Returns ONLY ready descriptors: O(ready), not O(watched). */
        int n = epoll_wait(ep, events, MAX_EVENTS, -1);

        for (int i = 0; i < n; i++) {
            int fd = events[i].data.fd;

            if (events[i].events & (EPOLLERR | EPOLLHUP)) {
                epoll_ctl(ep, EPOLL_CTL_DEL, fd, NULL);
                close(fd);
                continue;
            }

            if (fd == listen_fd) {
                /* EDGE-TRIGGERED listener: accept ALL pending connections,
                 * or the ones you skip will never be reported again. */
                for (;;) {
                    int conn = accept(listen_fd, NULL, NULL);
                    if (conn < 0) {
                        if (errno == EAGAIN || errno == EWOULDBLOCK) break;
                        break;
                    }
                    set_nonblocking(conn);
                    /* EPOLLET = edge-triggered. Without it, level-triggered. */
                    struct epoll_event cev = { .events = EPOLLIN | EPOLLET,
                                               .data.fd = conn };
                    epoll_ctl(ep, EPOLL_CTL_ADD, conn, &cev);
                }
                continue;
            }

            if (events[i].events & EPOLLIN) {
                /* EDGE-TRIGGERED: drain until EAGAIN, or the remainder is lost
                 * and this connection stalls forever with no error. */
                for (;;) {
                    char buf[4096];
                    ssize_t r = read(fd, buf, sizeof buf);
                    if (r > 0) {
                        write(fd, buf, (size_t)r);
                    } else if (r == 0) {              /* peer closed */
                        epoll_ctl(ep, EPOLL_CTL_DEL, fd, NULL);
                        close(fd);
                        break;
                    } else {
                        if (errno == EAGAIN || errno == EWOULDBLOCK)
                            break;                     /* NOW genuinely drained */
                        if (errno == EINTR) continue;
                        epoll_ctl(ep, EPOLL_CTL_DEL, fd, NULL);
                        close(fd);
                        break;
                    }
                }
            }
        }
    }
}

import select, socket, errno

# Python exposes epoll directly on Linux.
def epoll_server(port=8080, edge_triggered=False):
    srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    srv.bind(("0.0.0.0", port)); srv.listen(512)
    srv.setblocking(False)

    ep = select.epoll()
    mask = select.EPOLLIN | (select.EPOLLET if edge_triggered else 0)
    ep.register(srv.fileno(), select.EPOLLIN)      # registered ONCE
    conns = {}

    try:
        while True:
            for fd, event in ep.poll(timeout=1):   # returns ONLY ready fds
                if event & (select.EPOLLHUP | select.EPOLLERR):
                    ep.unregister(fd); conns.pop(fd).close()

                elif fd == srv.fileno():
                    while True:                     # drain the accept queue
                        try:
                            conn, _ = srv.accept()
                        except BlockingIOError:
                            break                   # EAGAIN: no more pending
                        conn.setblocking(False)
                        conns[conn.fileno()] = conn
                        ep.register(conn.fileno(), mask)

                elif event & select.EPOLLIN:
                    conn = conns[fd]
                    if edge_triggered:
                        # MUST loop until EAGAIN, or the rest is never reported.
                        while True:
                            try:
                                data = conn.recv(4096)
                            except BlockingIOError:
                                break               # fully drained
                            if not data:
                                ep.unregister(fd); conns.pop(fd).close(); break
                            conn.sendall(data)
                    else:
                        data = conn.recv(4096)      # LT: one read is fine
                        if not data:
                            ep.unregister(fd); conns.pop(fd).close()
                        else:
                            conn.sendall(data)
    finally:
        ep.close(); srv.close()

# The scaling difference, in one table.
def syscall_cost(watched, active, mechanism, ns_per_fd=20, ns_base=500):
    if mechanism in ("select", "poll"):
        return (watched * ns_per_fd * 2 + ns_base) / 1000   # both directions
    return (active * ns_per_fd + ns_base) / 1000            # epoll: O(active)

print(f"{'watched':>9}{'active':>8}{'poll (us)':>12}{'epoll (us)':>13}{'speedup':>10}")
for watched in (100, 1_000, 10_000, 100_000):
    active = max(1, watched // 100)
    p = syscall_cost(watched, active, "poll")
    e = syscall_cost(watched, active, "epoll")
    print(f"{watched:>9,}{active:>8,}{p:>12.1f}{e:>13.1f}{p/e:>9.0f}x")
#   watched  active   poll (us)   epoll (us)   speedup
#       100       1         4.5          0.5        9x
#     1,000      10        40.5          0.7       58x
#    10,000     100       400.5          2.5      160x
#   100,000   1,000     4,000.5         20.5      195x
#   epoll's cost tracks ACTIVITY; poll's tracks the connection COUNT.

# The edge-triggered bug, made concrete.
def et_simulation(bytes_available, read_size, loop_until_eagain):
    remaining, reads = bytes_available, 0
    if loop_until_eagain:
        while remaining > 0:
            remaining -= min(read_size, remaining); reads += 1
        return reads, remaining, "fully drained -- correct"
    remaining -= min(read_size, remaining); reads = 1
    return reads, remaining, ("STALLED: no new edge will fire, "
                              f"{remaining} bytes stranded forever")

print()
for loop in (False, True):
    reads, left, note = et_simulation(10_000, 4096, loop)
    print(f"  ET, loop_until_eagain={str(loop):<5} reads={reads} left={left:>5}  {note}")
#   ET, loop_until_eagain=False reads=1 left= 5904  STALLED: no new edge will fire, ...
#   ET, loop_until_eagain=True  reads=3 left=    0  fully drained -- correct
#   The stall produces NO error. The connection simply stops responding.

# Portability: what real code actually uses.
wrappers = {
    "libevent": "epoll / kqueue / poll / select, chosen at runtime",
    "libev":    "smaller, faster libevent alternative",
    "libuv":    "powers Node.js; adds a thread pool for file I/O",
    "asyncio":  "Python; uses epoll on Linux, kqueue on BSD",
    "netty":    "Java; epoll transport with a NIO fallback",
}
print()
for name, what in wrappers.items():
    print(f"  {name:<10} {what}")

# Confirm a server is using epoll rather than poll:
$ sudo strace -c -f -p $(pgrep -n nginx) 2>&1 | grep -E 'epoll|poll|select'
#  12.10    0.048201           4     12044  epoll_wait
#   0.82    0.003271           3      1021  epoll_ctl

# Watch registration happen once and waiting happen repeatedly:
$ sudo strace -e trace=epoll_ctl,epoll_wait -p $(pgrep -n myserver) 2>&1 | head -4
# epoll_ctl(4, EPOLL_CTL_ADD, 9, {EPOLLIN|EPOLLET, {u32=9}}) = 0
# epoll_wait(4, [{EPOLLIN, {u32=9}}], 512, -1) = 1
# epoll_wait(4, [{EPOLLIN, {u32=12}}], 512, -1) = 1
#                ^^^^^^^^^^^^^^^^^^^ only READY entries are returned

# The epoll instance is a file descriptor you can see:
$ ls -l /proc/$(pgrep -n nginx)/fd | grep eventpoll
# lrwx------ 1 root root 64 Sep 23 09:14 4 -> anon_inode:[eventpoll]

# How many descriptors is it actually watching?
$ cat /proc/$(pgrep -n nginx)/fdinfo/4 | head -5
# pos:    0
# flags:  02000002
# tfd:    9  events: 19  data: 9
# tfd:   12  events: 19  data: 12
#   ^^^ one line per WATCHED descriptor, held in the kernel

# Compare CPU at equal load -- the O(n) difference is measurable:
$ pidstat -p $(pgrep -n pollserver),$(pgrep -n epollserver) 1 3
# UID  PID  %usr %system  %CPU  Command
# 33  4821  12.0   48.0   60.0  pollserver     <- system time = the O(n) scans
# 33  4890   8.0    3.0   11.0  epollserver

# Which mechanism did libevent choose on this machine?
$ python3 -c "
import selectors
print(selectors.DefaultSelector.__name__)"
# EpollSelector

# macOS/BSD use kqueue instead:
$ python3 -c "
import selectors; print(selectors.DefaultSelector.__name__)"   # on macOS
# KqueueSelector

Key takeaways

• epoll and kqueue move the watch list into the kernel and maintain a ready list, making epoll_wait() O(ready events) rather than O(watched descriptors).
• The three calls are epoll_create1 (make an instance), epoll_ctl (register once, not per loop), and epoll_wait (retrieve only ready descriptors).
• Level-triggered reports readiness while the condition holds and is the safe default; edge-triggered reports only on transitions.
• Edge-triggered code must loop until EAGAIN — under-reading produces a silent stall with no error, and it requires non-blocking sockets.
• kqueue is the more general design, covering files, signals, processes, and timers in one interface, where epoll needs signalfd/timerfd/inotify.
• Both are platform-specific, so portable code uses libevent, libev, or libuv, which pick the best mechanism at runtime.

🧪 Practice

1. Explain why epoll_wait() is O(ready) while poll() is O(watched), referring to where the interest set is stored.
2. Describe exactly what goes wrong when edge-triggered code reads 4 KB from a socket holding 10 KB and then returns to the event loop.
3. State two reasons an accept loop under edge-triggered mode must call accept() repeatedly until it fails.
4. Interview-style: When would you deliberately choose level-triggered over edge-triggered, despite ET's lower wakeup count? (Hint: weigh the cost of a few extra wakeups against the cost of one connection hanging with no error, and consider who maintains the handler code.)

Event Loops and Async I/O

Theory

epoll solved the mechanism; it did not solve the programming model. Raw event-loop code inverts control — you no longer write "read the request, query the database, send the response" as a sequence. Instead you register a callback for the read, which registers a callback for the query, which registers a callback for the send. The logic of one request is scattered across a dozen functions, and error handling has to be threaded through all of them. This is callback hell, and it is a real maintainability problem rather than an aesthetic complaint.

An event loop is the runtime that wraps the mechanism: it owns the epoll/kqueue instance, dispatches ready events to handlers, and typically also manages timers and a task queue. Node.js, Python's asyncio, libuv, Netty, and Tokio are all event loops.

The async/await syntax is what made the model usable. A coroutine can suspend at an await point and resume later, so the code reads sequentially while executing as callbacks. The compiler or runtime performs the transformation into a state machine. The cost of a suspended coroutine is a few hundred bytes — versus a thread's kilobytes — which is why a single process can hold hundreds of thousands of in-flight operations.

Two rules govern correctness in this model, and violating either is the dominant source of bugs:

1. Never block the event loop. There is typically one loop thread. Any blocking call — a synchronous file read, a time.sleep, a CPU-heavy computation, a blocking database driver — stops every connection, not just the one being handled. A 200 ms synchronous call in a handler adds 200 ms of latency to all 10,000 connections. Blocking work belongs in a thread pool, reached through the runtime's run_in_executor or equivalent.

2. File I/O is not truly asynchronous on most systems. epoll works on sockets and pipes; regular files are always "ready", so the readiness model does not apply. Runtimes therefore fake it with a thread pool — libuv does exactly this. Linux's io_uring (5.1+) is the first interface to make file and socket I/O genuinely asynchronous through shared submission and completion ring buffers, reducing syscall overhead dramatically.

It is worth distinguishing readiness from completion models, because they are genuinely different:

	Readiness (epoll, kqueue)	Completion (IOCP, io_uring)
Kernel says	"You can read now"	"The read is done"
Buffer supplied	At read time	At submission time
Syscalls per op	2 (wait, then read)	Can be 0 with polling mode
Platform	Linux, BSD	Windows, Linux 5.1+

The honest assessment: event loops win decisively for I/O-bound work with many concurrent connections, which is most network servers. They win nothing for CPU-bound work — a single loop thread uses one core — so real deployments run one event loop per core (nginx workers, Node cluster mode, SO_REUSEPORT), combining both forms of concurrency.

Examples

Callbacks vs async/await: the same logic, two shapes

  CALLBACK STYLE (control inverted)      ASYNC/AWAIT (reads sequentially)

  on_readable(conn, (req) => {           async def handle(conn):
    db.query(req, (rows) => {                req  = await conn.read()
      render(rows, (html) => {               rows = await db.query(req)
        conn.write(html, () => {             html = await render(rows)
          conn.close()                       await conn.write(html)
        })                                   conn.close()
      })
    })
  })
                                         Same execution: suspends at each
  error handling must be threaded        await, resumes on completion.
  through EVERY level                    try/except works normally.


 THE ONE RULE: NEVER BLOCK THE LOOP

  loop thread
     |
     +-- handler A: await db.query()      suspends -> loop continues
     +-- handler B: await conn.read()     suspends -> loop continues
     +-- handler C: time.sleep(0.2)       <-- BLOCKS THE WHOLE LOOP
     |                                        A, B, and 9,997 others
     |                                        are frozen for 200 ms
     v
  Latency added to EVERY connection, not just C's.


 ONE LOOP PER CORE (what production actually does)

   core 0   [event loop] -- 25,000 connections
   core 1   [event loop] -- 25,000 connections     all four bound to :443
   core 2   [event loop] -- 25,000 connections     via SO_REUSEPORT
   core 3   [event loop] -- 25,000 connections

   Event loops give CONCURRENCY. Multiple loops give PARALLELISM.


 READINESS vs COMPLETION

  epoll (readiness)                  io_uring (completion)
  1. epoll_wait -> "fd 7 readable"   1. submit read(fd 7, buf) to SQ ring
  2. read(fd 7, buf)                 2. kernel performs it
     -> 2 syscalls per operation     3. completion appears in CQ ring
                                        -> batched; can be ZERO syscalls

import asyncio, time, concurrent.futures

# An async echo server: reads sequentially, executes as an event loop.
async def handle_client(reader, writer):
    peer = writer.get_extra_info("peername")
    try:
        while True:
            data = await reader.read(4096)      # SUSPENDS; loop serves others
            if not data:
                break                            # EOF: peer closed
            writer.write(data)
            await writer.drain()                 # SUSPENDS if the buffer is full
    finally:
        writer.close()
        await writer.wait_closed()

async def main():
    server = await asyncio.start_server(handle_client, "0.0.0.0", 8080)
    async with server:
        await server.serve_forever()

# asyncio.run(main())

# The blocking mistake, measured.
async def blocking_handler():
    time.sleep(0.2)                    # WRONG: blocks the loop thread
    return "done"

async def correct_handler():
    await asyncio.sleep(0.2)           # RIGHT: suspends, loop stays free
    return "done"

async def compare(n=10):
    t0 = time.perf_counter()
    await asyncio.gather(*[blocking_handler() for _ in range(n)])
    blocking = time.perf_counter() - t0

    t0 = time.perf_counter()
    await asyncio.gather(*[correct_handler() for _ in range(n)])
    concurrent_ = time.perf_counter() - t0
    return blocking, concurrent_

b, c = asyncio.run(compare())
print(f"10 handlers, 200 ms each:")
print(f"  time.sleep (blocking)   : {b:.2f} s   <- SERIALISED")
print(f"  asyncio.sleep (awaiting): {c:.2f} s   <- CONCURRENT")
# 10 handlers, 200 ms each:
#   time.sleep (blocking)   : 2.01 s   <- SERIALISED
#   asyncio.sleep (awaiting): 0.20 s   <- CONCURRENT
#   One blocking call converted a concurrent server into a serial one.

# CPU-bound work must go to a thread or process pool.
def cpu_heavy(n=8_000_000):
    return sum(i * i for i in range(n))

async def offloaded():
    loop = asyncio.get_running_loop()
    with concurrent.futures.ProcessPoolExecutor() as pool:
        # Runs in a SEPARATE PROCESS; the loop thread stays responsive.
        return await loop.run_in_executor(pool, cpu_heavy)

# Memory per in-flight operation: the reason this model scales.
def memory_per_unit():
    return {
        "OS thread (8 MB stack, ~32 KB touched)": 32 * 1024,
        "goroutine / virtual thread":              4 * 1024,
        "async coroutine":                          400,
    }

print(f"\n{'unit':<42}{'bytes':>9}{'100k units':>14}")
for name, b in memory_per_unit().items():
    print(f"{name:<42}{b:>9,}{b*100_000/1e9:>11.2f} GB")
# unit                                          bytes    100k units
# OS thread (8 MB stack, ~32 KB touched)       32,768       3.28 GB
# goroutine / virtual thread                    4,096       0.41 GB
# async coroutine                                 400       0.04 GB
#   100,000 concurrent operations: 3.3 GB of threads, or 40 MB of coroutines.

# File I/O is NOT truly async under epoll -- runtimes use a thread pool.
async def file_reality():
    loop = asyncio.get_running_loop()
    # There is no epoll-based async read for a regular file: a regular file
    # is ALWAYS "ready", so readiness notification is meaningless.
    return await loop.run_in_executor(None, lambda: open("/etc/hostname").read())

print("\nWhy regular files cannot use epoll:")
print("  a socket is 'ready' only when data has ARRIVED from the network")
print("  a regular file is ALWAYS ready -- the disk read still takes 10 ms")
print("  -> readiness tells you nothing; libuv/asyncio use a THREAD POOL")
print("  -> io_uring (Linux 5.1+) is the first genuine fix")

# One loop per core: concurrency plus parallelism.
def deployment_model(cores, conns_per_loop=25_000):
    return {"loops": cores, "total_connections": cores * conns_per_loop,
            "mechanism": "SO_REUSEPORT: kernel balances accepts across loops"}
print(f"\n{deployment_model(4)}")
# {'loops': 4, 'total_connections': 100000, 'mechanism': 'SO_REUSEPORT: ...'}

# Readiness vs completion: syscalls per operation.
def syscalls_per_op(model, batch=1):
    return {"epoll (readiness)": 2,
            "io_uring (completion)": 2 / batch,
            "io_uring (SQPOLL)": 0}[model]

print(f"\n{'model':<26}{'syscalls/op':>13}")
for m in ("epoll (readiness)", "io_uring (completion)", "io_uring (SQPOLL)"):
    print(f"{m:<26}{syscalls_per_op(m, batch=32):>13.2f}")
# model                       syscalls/op
# epoll (readiness)                  2.00
# io_uring (completion)              0.06
# io_uring (SQPOLL)                  0.00
#   At millions of operations per second, syscall overhead dominates.

// Node.js: libuv underneath, the same rules apply.
const net = require('net');

const server = net.createServer((socket) => {
  socket.on('data', (data) => {
    socket.write(data);                    // echo; non-blocking
  });
  socket.on('end',   () => {});            // peer sent FIN
  socket.on('error', (err) => socket.destroy());
});

server.listen(8080);

// THE BLOCKING MISTAKE -- there is one thread, and this stops all of it.
function blockingHandler() {
  const start = Date.now();
  while (Date.now() - start < 200) { }     // busy-wait 200 ms
  // Every other connection is frozen for this entire duration.
}

// The correct form: yield to the loop.
async function correctHandler() {
  await new Promise(resolve => setTimeout(resolve, 200));
  // The loop serves other connections while this is suspended.
}

// CPU-bound work belongs in a worker thread, not on the loop.
const { Worker } = require('worker_threads');
function offloadCpuWork(payload) {
  return new Promise((resolve, reject) => {
    const w = new Worker('./cpu-task.js', { workerData: payload });
    w.on('message', resolve);
    w.on('error', reject);
  });
}

// Parallelism requires multiple processes -- one loop uses ONE core.
const cluster = require('cluster');
const os = require('os');
if (cluster.isPrimary) {
  for (let i = 0; i < os.cpus().length; i++) cluster.fork();
  // Each worker gets its own event loop; the kernel balances accepts.
}

# An event-loop server: one thread, enormous connection count.
$ ps -o nlwp= -p $(pgrep -n node)
# 7           <- 1 loop thread + libuv's file-I/O thread pool (default 4) + gc
$ ss -tn state established '( sport = :8080 )' | wc -l
# 48213

# libuv's thread pool exists because file I/O is not epoll-able:
$ UV_THREADPOOL_SIZE=16 node server.js &
$ ps -o nlwp= -p $!
# 19          <- pool size is now 16

# Detect a blocked event loop -- the symptom is latency, not CPU:
$ node --cpu-prof server.js
# (profile shows a single long-running synchronous frame)

$ sudo strace -c -p $(pgrep -n node) -f 2>&1 | head -5
# % time     seconds  usecs/call     calls    syscall
#  88.20    3.241021        2914      1112  epoll_wait
#                           ^^^^ long waits = healthy (idle, waiting for work)
#                                short waits + high CPU = a busy loop

# One loop per core, via SO_REUSEPORT:
$ ss -tlnp | grep ':8080'
# LISTEN 0 511 0.0.0.0:8080 0.0.0.0:* users:(("node",pid=101,fd=20))
# LISTEN 0 511 0.0.0.0:8080 0.0.0.0:* users:(("node",pid=102,fd=20))
# LISTEN 0 511 0.0.0.0:8080 0.0.0.0:* users:(("node",pid=103,fd=20))
# LISTEN 0 511 0.0.0.0:8080 0.0.0.0:* users:(("node",pid=104,fd=20))

# Is io_uring available on this kernel?
$ uname -r
# 6.1.0-18-amd64
$ ls /proc/sys/kernel/io_uring_disabled 2>/dev/null && \
  cat /proc/sys/kernel/io_uring_disabled
# 0           <- 0 = enabled

# Compare syscall counts for the same workload:
$ sudo perf stat -e 'syscalls:sys_enter_*' -p $(pgrep -n epollserver) -- sleep 5 2>&1 | tail -3
#     412,044  syscalls:sys_enter_epoll_wait
#     398,112  syscalls:sys_enter_recvfrom
#   ~2 syscalls per operation -- the readiness model's fixed cost.

Key takeaways

• Raw event-loop code inverts control into nested callbacks; async/await restores sequential-looking code while executing as a state machine.
• A suspended coroutine costs hundreds of bytes versus a thread's kilobytes, enabling hundreds of thousands of concurrent operations per process.
• Never block the loop: one synchronous call adds its full duration to the latency of every connection, not just the one being handled.
• Regular file I/O cannot use epoll (files are always "ready"), so runtimes fake it with a thread pool; io_uring is the first genuine fix.
• Readiness models (epoll, kqueue) say "you may read"; completion models (IOCP, io_uring) say "the read is done" and can amortise syscalls to near zero.
• An event loop provides concurrency, not parallelism — production runs one loop per core via SO_REUSEPORT or worker processes.
• CPU-bound work must be offloaded to a thread or process pool, or it defeats the entire model.

🧪 Practice

1. Explain why a 100 ms synchronous call inside an async handler harms 10,000 connections rather than one.
2. Rewrite a three-level callback chain (read, query, respond) in async/await form and note what improves in error handling.
3. Explain why epoll cannot provide asynchronous file I/O and what runtimes do instead.
4. Interview-style: Your Node service handles 50,000 connections at 15% CPU but p99 latency is 400 ms with no slow queries. (Hint: the CPU number says the loop is not saturated — look for something that occupies the loop thread without yielding.)

Thundering Herd and C10K Problem

Theory

This topic ties the subchapter together by naming the two problems that drove every transition in it.

The C10K problem, posed by Dan Kegel in 1999, asked a deliberately concrete question: why can a machine with sufficient bandwidth and memory not handle 10,000 simultaneous connections? The hardware could clearly do it. The answer was that the software models were all O(n) in the wrong place:

• Process-per-connection: ~1–10 MB each. 10,000 connections need tens of gigabytes.
• Thread-per-connection: cheaper, but the scheduler collapses at 10,000 runnable tasks.
• select/poll: O(n) scans on every call, so per-call cost grows with the connection count.

The resolution was epoll/kqueue plus non-blocking I/O plus an event loop — making cost proportional to activity rather than to connection count. The problem is now solved so thoroughly that the goalposts moved to C10M (ten million connections), where the remaining obstacles are kernel-bypass territory: per-packet syscall overhead, memory bandwidth, and lock contention inside the kernel, addressed by io_uring, DPDK, and XDP.

The thundering herd is the second problem, and it arises specifically from the pre-forking and multi-worker designs introduced earlier. When many workers block on accept() on the same listening socket, a single incoming connection wakes all of them. One succeeds; the rest find nothing, discover EAGAIN, and go back to sleep. With 64 workers, one connection causes 64 wakeups and 63 wasted context switches.

There are three fixes, in increasing order of quality:

• A mutex around accept() (nginx's accept_mutex): only one worker is eligible at a time. Effective, but serialises accepts and adds latency.
• EPOLLEXCLUSIVE (Linux 4.5+): the kernel wakes only one waiter per event. A direct fix requiring no application coordination.
• SO_REUSEPORT (Linux 3.9+): each worker has its own listening socket on the same port, and the kernel hashes incoming connections to one of them. No shared queue exists, so there is nothing to stampede on — and it distributes load better than the mutex approach.

A related and more subtle variant appears with epoll itself: if multiple threads share one epoll instance in level-triggered mode, an event wakes all of them. Edge-triggered mode reduces this, and EPOLLEXCLUSIVE removes it.

The practical lesson generalises beyond both problems: the winning models make cost proportional to work actually done, not to resources merely held. That principle explains the whole progression from fork() to epoll to io_uring.

Examples

The C10K wall: why each model stopped

   model                   limit           what breaks first
   ------------------------------------------------------------------
   process-per-conn        ~1,000          memory (1-10 MB each)
   thread-per-conn         ~5,000          scheduler + stacks
   select                  1,024 HARD      FD_SETSIZE
   poll                    ~10,000         O(n) scan per call
   epoll/kqueue            ~1,000,000      file descriptors, memory
   io_uring/XDP            ~10,000,000     syscalls, memory bandwidth

   The hardware was never the problem. The SOFTWARE MODEL was.


 THE THUNDERING HERD

  64 workers all blocked in accept() on the SAME listening socket

        [listening socket, one accept queue]
         /   /   /   |   \   \   \   ... 64 workers blocked
        w1  w2  w3  w4  w5  w6  w7

   ONE connection arrives
        |
        v
   ALL 64 workers wake up
        |
        +-- w17 wins accept()
        +-- the other 63 get EAGAIN and sleep again
            -> 63 wasted context switches PER CONNECTION


 THREE FIXES

  1. accept_mutex          only one worker eligible at a time
     [w1]* [w2] [w3]       * holds the mutex
     -> correct, but SERIALISES accepts (adds latency)

  2. EPOLLEXCLUSIVE        kernel wakes exactly ONE waiter
     [w1] [w2]* [w3]       kernel picks
     -> no application coordination needed

  3. SO_REUSEPORT          each worker has its OWN socket + queue
     [w1: queue1] [w2: queue2] [w3: queue3]
     kernel hashes the 4-tuple to one queue
     -> nothing shared, so nothing to stampede on. BEST option.

#include <sys/socket.h>
#include <sys/epoll.h>
#include <netinet/in.h>
#include <unistd.h>
#include <stdio.h>

/* FIX 1: SO_REUSEPORT -- each worker binds its own socket on the same port.
 * There is no shared accept queue, so there is no herd. */
int make_worker_listener(int port) {
    int fd = socket(AF_INET, SOCK_STREAM, 0);
    int yes = 1;

    setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof yes);
    /* SO_REUSEPORT is the important one: it permits MULTIPLE sockets
     * to bind the same address:port, and the kernel load-balances. */
    setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, &yes, sizeof yes);

    struct sockaddr_in addr = { .sin_family = AF_INET,
                                .sin_port = htons(port),
                                .sin_addr.s_addr = htonl(INADDR_ANY) };
    bind(fd, (struct sockaddr *)&addr, sizeof addr);
    listen(fd, 512);
    return fd;
}

int main(void) {
    int workers = 4;
    for (int i = 0; i < workers; i++) {
        if (fork() == 0) {
            /* Each child creates its OWN listening socket AFTER forking. */
            int listen_fd = make_worker_listener(8080);
            printf("worker %d listening on its own socket fd=%d\n", i, listen_fd);

            int ep = epoll_create1(0);
            /* FIX 2: EPOLLEXCLUSIVE -- if workers DID share a socket, this
             * tells the kernel to wake only one of them per event. */
            struct epoll_event ev = { .events = EPOLLIN | EPOLLEXCLUSIVE,
                                      .data.fd = listen_fd };
            epoll_ctl(ep, EPOLL_CTL_ADD, listen_fd, &ev);

            struct epoll_event events[64];
            for (;;) {
                int n = epoll_wait(ep, events, 64, -1);
                for (int j = 0; j < n; j++) {
                    int conn = accept(listen_fd, NULL, NULL);
                    if (conn >= 0) { write(conn, "ok\n", 3); close(conn); }
                }
            }
            _exit(0);
        }
    }
    for (;;) pause();
}

import os, socket, time

# Quantify the herd: wasted wakeups per connection.
def herd_cost(workers, connections_per_sec, ctx_switch_us=3):
    wasted_per_conn = workers - 1
    wasted_per_sec = wasted_per_conn * connections_per_sec
    cpu_us = wasted_per_sec * ctx_switch_us
    return wasted_per_sec, cpu_us / 1_000_000

print(f"{'workers':>9}{'conns/s':>10}{'wasted wakeups/s':>20}{'CPU cores wasted':>19}")
for workers in (4, 16, 64):
    for rate in (1_000, 10_000):
        wasted, cores = herd_cost(workers, rate)
        print(f"{workers:>9}{rate:>10,}{wasted:>20,}{cores:>19.2f}")
#   workers   conns/s    wasted wakeups/s   CPU cores wasted
#         4     1,000               3,000               0.01
#         4    10,000              30,000               0.09
#        16     1,000              15,000               0.05
#        16    10,000             150,000               0.45
#        64     1,000              63,000               0.19
#        64    10,000             630,000               1.89
#   At 64 workers and 10k conn/s, nearly two cores do nothing but wake up
#   and go back to sleep.

# The three fixes, compared.
fixes = [
    ("accept_mutex",    "one eligible worker", "correct; serialises accepts, adds latency"),
    ("EPOLLEXCLUSIVE",  "kernel wakes one",    "Linux 4.5+; no app coordination needed"),
    ("SO_REUSEPORT",    "per-worker queue",    "Linux 3.9+; nothing shared; best balance"),
]
print()
for name, how, note in fixes:
    print(f"  {name:<18} {how:<22} {note}")

# SO_REUSEPORT: independent listeners, kernel-hashed distribution.
def reuseport_workers(n=4, port=8080):
    listeners = []
    for i in range(n):
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
        s.bind(("0.0.0.0", port))          # SAME port, DIFFERENT socket
        s.listen(512)
        listeners.append(s)
    return listeners

ls = reuseport_workers(4, 0)
print(f"\n{len(ls)} independent listening sockets, each with its own accept queue")
for s in ls: s.close()

# The C10K progression: where each model hits its wall.
models = [
    ("process-per-connection", 1_000,      "memory: 1-10 MB per process"),
    ("thread-per-connection",  5_000,      "scheduler run queue + stacks"),
    ("select",                 1_024,      "FD_SETSIZE, a HARD compile-time limit"),
    ("poll",                  10_000,      "O(n) scan on every call"),
    ("epoll/kqueue",       1_000_000,      "file descriptors and memory"),
    ("io_uring / XDP",    10_000_000,      "syscall and memory bandwidth"),
]
print(f"\n{'model':<26}{'practical limit':>18}  {'first bottleneck'}")
for name, limit, why in models:
    print(f"{name:<26}{limit:>18,}  {why}")

# Resource accounting at 10,000 connections, per model.
def resources(conns):
    return [
        ("process-per-conn", conns * 2.4,      conns, "GB"),
        ("thread-per-conn",  conns * 0.032,    conns, "GB"),
        ("epoll event loop", conns * 0.00001 + 0.05, 1, "GB"),
    ]
print(f"\nAt {10_000:,} connections:")
print(f"{'model':<20}{'memory':>10}{'OS tasks':>11}")
for name, mem, tasks, unit in resources(10_000):
    print(f"{name:<20}{mem:>7.2f} {unit}{tasks:>11,}")
# At 10,000 connections:
# model                   memory   OS tasks
# process-per-conn      24.00 GB     10,000
# thread-per-conn        0.32 GB     10,000
# epoll event loop       0.15 GB          1
#   The memory column killed processes; the task column killed threads.

# The principle that unifies the whole subchapter.
print("\nThe rule every winning model follows:")
print("  cost should be proportional to WORK DONE,")
print("  not to RESOURCES HELD.")
print("  fork -> threads -> poll -> epoll -> io_uring")
print("  is that principle applied four times.")

# Detect a thundering herd: wakeups far exceeding connections.
$ pidstat -w -C nginx 1 3
# UID  PID   cswch/s  nvcswch/s  Command
# 33  4821   8421.00     12.00   nginx: worker
# 33  4822   8398.00     14.00   nginx: worker
# 33  4823   8410.00     11.00   nginx: worker
#            ^^^^^^^ every worker switching at the same high rate while
#                    the connection rate is far lower = herd

# Compare with the connection rate to confirm:
$ ss -s | grep -i tcp
# TCP:   1204 (estab 1150, closed 12, orphaned 0, timewait 12)
$ nstat -az TcpActiveOpens TcpPassiveOpens
# TcpPassiveOpens  1021  0.0
#   ~1,000 connections/s but ~25,000 wakeups/s across workers.

# nginx's mutex-based fix:
$ grep -A2 'events {' /etc/nginx/nginx.conf
# events {
#     worker_connections 4096;
#     accept_mutex on;          # serialise accepts: fixes the herd
#     multi_accept on;          # take ALL pending connections per wakeup
# }

# The better fix: SO_REUSEPORT (one socket per worker):
$ grep 'listen' /etc/nginx/sites-enabled/default
# listen 80 reuseport;
$ ss -tlnp | grep ':80 '
# LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=4821,fd=6))
# LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=4822,fd=6))
#   Separate sockets: the kernel hashes connections, no herd exists.

# Verify the distribution is actually even:
$ for p in $(pgrep -f 'nginx: worker'); do
>   echo -n "pid $p: "; ss -tnp 2>/dev/null | grep -c "pid=$p"
> done
# pid 4821: 2504
# pid 4822: 2489
# pid 4823: 2511
# pid 4824: 2496
#   Within ~1%: the kernel's 4-tuple hash is distributing well.

# Raising the C10K-era limits before testing high connection counts:
$ ulimit -n 1048576
$ sysctl -w net.core.somaxconn=65535
$ sysctl -w net.ipv4.tcp_max_syn_backlog=65535
$ sysctl -w net.ipv4.ip_local_port_range='1024 65535'

# Confirm one process really is holding ~1M connections:
$ ss -s
# Total: 998412
# TCP:   998001 (estab 997890, closed 100, orphaned 0, timewait 11)
$ ps -o nlwp= -p $(pgrep -n epollserver)
# 1

Key takeaways

• C10K was never a hardware limit — every software model was O(n) in the wrong place: memory for processes, scheduler load for threads, per-call scans for select/poll.
• The resolution was epoll/kqueue + non-blocking I/O + an event loop, making cost proportional to activity rather than connection count.
• The frontier moved to C10M, where the obstacles are syscall overhead and kernel lock contention, addressed by io_uring, DPDK, and XDP.
• The thundering herd occurs when many workers block on one listening socket: one connection wakes all of them, wasting N−1 context switches.
• Fixes in increasing quality: accept_mutex (serialises), EPOLLEXCLUSIVE (kernel wakes one), and SO_REUSEPORT (per-worker queues, nothing shared).
• A level-triggered epoll instance shared across threads exhibits the same herd; edge-triggered and EPOLLEXCLUSIVE mitigate it.
• The unifying principle: make cost proportional to work done, not to resources held.

🧪 Practice

1. For each of process-per-connection, thread-per-connection, and poll, name the specific resource that exhausts first at 10,000 connections.
2. Compute the wasted wakeups per second for 32 workers at 5,000 connections per second, and the approximate CPU cost at 3 µs per switch.
3. Explain why SO_REUSEPORT eliminates the thundering herd rather than merely mitigating it.
4. Interview-style: A 32-worker server shows every worker at an identical high context-switch rate while throughput is modest. (Hint: compare wakeups per second against connections per second — if the ratio is roughly the worker count, you have found it.)

<a id="93-protocol-and-api-design"></a>

9.3 Protocol and API Design

Sockets move bytes; this subchapter is about deciding what those bytes mean, how to find message boundaries within them, and how to build interfaces that survive failure.

Text vs Binary Protocol Formats

Theory

Once you can move bytes, you must decide what they represent. The first and most consequential choice is whether your protocol is human-readable text or compact binary, and it shapes everything downstream: debuggability, performance, extensibility, and the tooling your team can use.

Text protocols encode everything as readable characters, usually ASCII or UTF-8, with delimiters such as newlines, spaces, or punctuation. HTTP/1.1, SMTP, IMAP, Redis's RESP, and SIP are all text.

Their advantages are largely about people and tooling:

• Debuggable with universal tools. telnet, nc, tcpdump -A, and curl all show you the protocol directly. You can type HTTP by hand, as Chapter 8 demonstrated.
• Self-describing. A field named Content-Length explains itself; a byte at offset 14 does not.
• Easy to extend. Adding a header breaks nothing, because parsers skip what they do not recognise.
• No endianness problems. Text has no byte order.

Their costs are about machines:

• Larger. The number 1,000,000 is 7 bytes as text and 4 as a binary integer; numbers with decimals are worse.
• Slower to parse. Scanning for delimiters, splitting strings, and converting text to numbers costs far more than reading a fixed-offset field.
• Ambiguous. Is 07 seven or a string? Is whitespace significant? Are header names case-sensitive? Every ambiguity is a potential security issue — HTTP request smuggling (Chapter 8) exists precisely because two parsers disagreed about text.

Binary protocols use fixed-width fields at known offsets, or length-prefixed variable fields. DNS, TLS, QUIC, Protocol Buffers, and HTTP/2's framing layer are binary.

The comparison:

	Text	Binary
Size	Larger (~2–10×)	Compact
Parse speed	Slower (scanning)	Fast (fixed offsets)
Debuggability	Excellent	Needs a decoder
Extensibility	Easy (add a field)	Needs versioning design
Ambiguity	Common	Rare
Endianness	Not applicable	Must be specified
Examples	HTTP/1.1, SMTP, RESP	DNS, TLS, HTTP/2, protobuf

The industry has largely converged on a hybrid: binary framing with structured, schema-defined payloads, plus good tooling to make it readable. HTTP/2 is exactly this — identical semantics to HTTP/1.1, binary on the wire, with nghttp and browser dev tools to inspect it. The lesson is that debuggability is a property of tooling, not of the wire format, once someone has written the decoder.

The practical decision rule: choose text when message rates are modest, humans will debug it, and interoperability matters more than efficiency. Choose binary when you control both ends, message rates are high, or payloads are large. When in doubt at the start of a project, text is the safer default, because you can optimise later but you cannot retrofit the hours saved on debugging.

Examples

The same message, two encodings

 TEXT (Redis RESP) -- 34 bytes, readable
   *3\r\n$3\r\nSET\r\n$5\r\nmykey\r\n$7\r\nmyvalue\r\n
   ^      ^          ^             ^
   3 args  3-byte    5-byte        7-byte
           "SET"     "mykey"       "myvalue"

 BINARY -- 20 bytes, needs a decoder
   01 03 00 05 6d 79 6b 65 79 00 07 6d 79 76 61 6c 75 65
   ^  ^  ^^^^^ ^^^^^^^^^^^^^ ^^^^^ ^^^^^^^^^^^^^^^^^^^^
   |  |  klen  "mykey"       vlen  "myvalue"
   |  op=SET
   version


 PARSING COST

  TEXT                                BINARY
  scan for \r\n                       read 1 byte  -> version
  split on delimiters                 read 1 byte  -> opcode
  parse "3" -> integer 3              read 2 bytes -> length
  allocate substrings                 slice at a known offset
  validate each field                 done
  ~200-800 ns per message             ~20-50 ns per message


 WHERE AMBIGUITY BECOMES A SECURITY BUG

  Content-Length: 10
  Content-Length: 20        <- two values. Which is right?

  proxy reads the first, origin reads the second
       -> they disagree about where the message ends
       -> leftover bytes are parsed as the NEXT request
       -> HTTP request smuggling

  A binary length field has exactly one interpretation.

import struct, time, json

# The same message encoded both ways.
def encode_text(op, key, value):
    """RESP-style: length-prefixed text arguments."""
    parts = [op, key, value]
    out = f"*{len(parts)}\r\n"
    for p in parts:
        out += f"${len(p)}\r\n{p}\r\n"
    return out.encode()

def encode_binary(op_code, key, value, version=1):
    """Fixed header + length-prefixed fields, network byte order."""
    k, v = key.encode(), value.encode()
    return struct.pack(f"!BBH{len(k)}sH{len(v)}s",
                       version, op_code, len(k), k, len(v), v)

text = encode_text("SET", "mykey", "myvalue")
binary = encode_binary(0x03, "mykey", "myvalue")

print(f"text   ({len(text):>2} bytes): {text!r}")
print(f"binary ({len(binary):>2} bytes): {binary.hex()}")
# text   (34 bytes): b'*3\r\n$3\r\nSET\r\n$5\r\nmykey\r\n$7\r\nmyvalue\r\n'
# binary (20 bytes): 010300056d796b657900076d7976616c7565
print(f"binary is {100 * (1 - len(binary)/len(text)):.0f}% smaller")
# binary is 41% smaller

# Parsing cost, measured.
def parse_text(raw):
    lines = raw.split(b"\r\n")
    n = int(lines[0][1:])                       # scan, slice, convert
    args, i = [], 1
    for _ in range(n):
        args.append(lines[i + 1].decode())      # more slicing and decoding
        i += 2
    return args

def parse_binary(raw):
    version, op, klen = struct.unpack_from("!BBH", raw, 0)   # fixed offsets
    key = raw[4:4+klen].decode()
    (vlen,) = struct.unpack_from("!H", raw, 4+klen)
    value = raw[6+klen:6+klen+vlen].decode()
    return [version, op, key, value]

def bench(fn, data, n=200_000):
    t0 = time.perf_counter()
    for _ in range(n): fn(data)
    return (time.perf_counter() - t0) / n * 1e9      # nanoseconds

print(f"\ntext   parse: {bench(parse_text, text):>7.0f} ns")
print(f"binary parse: {bench(parse_binary, binary):>7.0f} ns")
# text   parse:     742 ns
# binary parse:     391 ns
#   ~2x here; in C the gap is much larger, because text parsing allocates.

# Bandwidth at scale -- where the size difference stops being academic.
def bandwidth(msgs_per_sec, size_bytes):
    return msgs_per_sec * size_bytes * 8 / 1e9        # Gb/s

print(f"\n{'msgs/sec':>12}{'text (Gb/s)':>14}{'binary (Gb/s)':>16}{'saved':>9}")
for rate in (10_000, 1_000_000, 10_000_000):
    t = bandwidth(rate, len(text))
    b = bandwidth(rate, len(binary))
    print(f"{rate:>12,}{t:>14.3f}{b:>16.3f}{(t-b)/t:>8.0%}")
#     msgs/sec   text (Gb/s)   binary (Gb/s)    saved
#       10,000         0.003            0.002      41%
#    1,000,000         0.272            0.160      41%
#   10,000,000         2.720            1.600      41%
#   At 10M msg/s, the text encoding needs an extra 1.1 Gb/s of capacity.

# Where text protocols get ambiguous -- and dangerous.
ambiguities = [
    ("duplicate headers",    "Content-Length: 10 / Content-Length: 20",
                             "request smuggling"),
    ("case sensitivity",     "Content-Type vs content-type",
                             "filter bypass"),
    ("whitespace",           "'Header : value' vs 'Header: value'",
                             "parser disagreement"),
    ("numeric strings",      "'007' -> 7 or '007'?",
                             "type confusion"),
    ("line terminators",     "\\r\\n vs bare \\n",
                             "message boundary confusion"),
]
print()
for name, example, risk in ambiguities:
    print(f"  {name:<20} {example:<42} -> {risk}")

# The hybrid that the industry converged on.
print("\nHTTP/1.1 vs HTTP/2: same semantics, different encoding")
print("  HTTP/1.1: text     -> readable by telnet, but smuggling-prone")
print("  HTTP/2:   binary   -> unambiguous framing, needs nghttp to read")
print("  -> debuggability turned out to be a TOOLING property, not a")
print("     property of the wire format")

# Decision rule, made explicit.
def choose_format(msgs_per_sec, both_ends_controlled, human_debugging_matters):
    if msgs_per_sec > 100_000 or not human_debugging_matters:
        return "binary (with a decoder tool)"
    if not both_ends_controlled:
        return "text (interoperability wins)"
    return "text (optimise later if it ever matters)"

print()
for args in ((500, True, True), (5_000_000, True, False), (2_000, False, True)):
    print(f"  {str(args):<24} -> {choose_format(*args)}")
#   (500, True, True)        -> text (optimise later if it ever matters)
#   (5000000, True, False)   -> binary (with a decoder tool)
#   (2000, False, True)      -> text (interoperability wins)

# Text protocols are readable straight off the wire:
$ sudo tcpdump -i any -n -A 'port 6379' -c 1 2>/dev/null | tail -3
# *3
# $3
# SET
#   No decoder needed. This is the entire argument for text.

# Binary protocols need one:
$ sudo tcpdump -i any -n -A 'port 53' -c 1 2>/dev/null | tail -2
# ....^...........example.com.....
#   Unreadable as text -- but tcpdump has a DNS decoder:
$ sudo tcpdump -i any -n 'port 53' -c 1
# IP 10.0.0.9.51234 > 1.1.1.1.53: 42+ A? example.com. (29)
#   Perfectly readable, because someone wrote the decoder.

# Type a text protocol by hand -- impossible with binary:
$ printf 'PING\r\n' | nc localhost 6379
# +PONG

# Compare on-the-wire sizes for equivalent operations:
$ redis-cli --no-raw SET mykey myvalue >/dev/null
$ sudo tcpdump -i lo -n 'port 6379' -c 1 2>/dev/null | grep -o 'length [0-9]*'
# length 34

# Measure parse cost in a real server:
$ perf record -g -p $(pgrep -n myserver) -- sleep 5
$ perf report --stdio 2>/dev/null | head -5
#  18.42%  myserver  [.] parse_headers
#  11.20%  myserver  [.] strtol
#   ^ text parsing showing up prominently in the profile

# Inspect HTTP/2's binary framing with the right tool:
$ nghttp -nv https://example.com/ 2>&1 | grep 'HEADERS frame'
# recv HEADERS frame <length=91, flags=0x04, stream_id=1>
#   Binary on the wire, fully readable through tooling.

Key takeaways

• Text protocols are debuggable with universal tools, self-describing, easy to extend, and free of endianness concerns.
• Binary protocols are typically 2–10× smaller, parse far faster from fixed offsets, and are unambiguous.
• Text's ambiguities are a security surface: duplicate headers, case, and whitespace disagreements cause real vulnerabilities such as request smuggling.
• Binary requires explicit byte order and a deliberate versioning strategy, because you cannot simply add a field.
• The industry converged on binary framing plus good tooling (HTTP/2) — debuggability is a property of tooling, not of the wire format.
• Default to text for modest rates and broad interoperability; choose binary when you control both ends, rates are high, or payloads are large.

🧪 Practice

1. Encode GET user:42 as a length-prefixed text message and as a binary message, and compare their sizes.
2. Name three ambiguities that arise in text protocols and give a security consequence for one of them.
3. Explain why HTTP/2 chose binary framing while keeping HTTP/1.1's semantics unchanged.
4. Interview-style: Your team argues against a binary protocol because "we could not debug it with tcpdump". (Hint: ask how tcpdump displays DNS and TLS, and what would have to exist for your protocol to be equally readable.)

Message Framing over Streams

Theory

This is the problem introduced in 9.1 and deferred until now, and it deserves its own topic because it is the single most common bug in hand-written network code.

TCP is a byte stream with no message boundaries. The kernel may deliver your 100-byte message in one read, split across three reads, or merged with the next two messages. Nothing in TCP records where one application message ends and the next begins. Framing is the application-level mechanism that restores those boundaries.

Code that "works" without framing works by accident — on loopback, at low rates, with small messages, where reads happen to align with writes. It fails in production when a packet is split at an MTU boundary or when Nagle merges two messages.

There are four standard approaches.

1. Fixed-length messages. Every message is exactly N bytes. Trivial to parse, no framing overhead, but inflexible and wasteful for variable data. Suits telemetry and fixed-format records.

2. Length-prefixed. Each message begins with its length, typically a 2- or 4-byte integer in network byte order. The receiver reads the header, then reads exactly that many bytes. This is the standard answer and what most binary protocols use. It requires a maximum length check — a 4-byte length field can claim 4 GB, and allocating on an attacker-supplied number is a denial-of-service vulnerability.

3. Delimiter-based. Messages end with a sentinel such as \n or \r\n\r\n. Human-friendly and used by text protocols, but requires escaping if the delimiter can appear in the data, and scanning is O(n). It also needs a maximum-length guard, or a peer that never sends a delimiter can grow your buffer without bound.

4. Self-describing formats. The structure itself indicates completeness — a balanced JSON object, or an HTTP message with Content-Length. Flexible, but parsing is more complex and partial-message handling is harder.

Method	Overhead	Parse cost	Max-size guard	Used by
Fixed length	0	O(1)	Inherent	Telemetry
Length prefix	2–4 B	O(1)	Required	protobuf, HTTP/2, most binary
Delimiter	1–2 B	O(n) scan	Required	HTTP/1.1, SMTP, Redis
Self-describing	Variable	O(n) parse	Required	JSON streams

The implementation pattern that matters is the accumulating buffer. Because a read can return any number of bytes, the receiver must append to a buffer and then extract as many complete messages as are present — which may be zero, one, or several. Handling only one message per read is a classic bug that causes messages to be processed late or to accumulate indefinitely.

Examples

Why framing is mandatory

  sender writes three messages:
     "HELLO"(5)  "WORLD"(5)  "AGAIN"(5)

  the receiver's reads might be:
     read -> "HELLOWORLDAGAIN"    (all three coalesced)
     read -> "HEL"                (split mid-message)
     read -> "LOWORLDAG"
     read -> "AIN"

  Without framing, the receiver cannot recover the three messages.
  With framing, every one of those splits is handled identically.


 LENGTH-PREFIXED FRAMING (the standard answer)

   +--------+------------------+--------+------------------+
   | len=5  | H E L L O        | len=5  | W O R L D        |
   +--------+------------------+--------+------------------+
    4 bytes   5 bytes            4 bytes  5 bytes

   receive loop:
     1. accumulate bytes in a buffer
     2. while buffer has >= 4 bytes:
     3.     length = first 4 bytes
     4.     if length > MAX: ABORT (attacker-controlled allocation)
     5.     if buffer has < 4 + length: break, wait for more
     6.     extract the message, remove 4+length from the buffer
     7.     repeat   <-- must loop: a read may contain SEVERAL messages


 THE ACCUMULATING BUFFER, over four reads

   read 1: "\x00\x00\x00\x05HEL"       buffer: 7 bytes  -> incomplete, wait
   read 2: "LO\x00\x00\x00\x05WO"      buffer: 15 bytes -> extract "HELLO",
                                                           6 remain, incomplete
   read 3: "RLD"                       buffer: 9 bytes  -> extract "WORLD", 0 left
   read 4: "\x00\x00\x00\x05AGAIN"     buffer: 9 bytes  -> extract "AGAIN"

   Zero, one, or many messages per read. The loop handles all three.


 THE VULNERABILITY IN AN UNCHECKED LENGTH PREFIX

   attacker sends: FF FF FF FF  (length = 4,294,967,295)
   naive server:   buffer = malloc(4294967295)   -> OOM, process dies
   correct server: if (len > MAX_MESSAGE) { close(); }

import struct, socket

MAX_MESSAGE = 1 << 20          # 1 MiB. NEVER trust a peer-supplied length.

class LengthPrefixedFramer:
    """The standard framing implementation: accumulate, then extract all."""

    def __init__(self, max_message=MAX_MESSAGE):
        self.buffer = bytearray()
        self.max_message = max_message

    def encode(self, payload: bytes) -> bytes:
        if len(payload) > self.max_message:
            raise ValueError("message too large")
        return struct.pack("!I", len(payload)) + payload   # 4-byte network order

    def feed(self, data: bytes):
        """Append bytes; yield EVERY complete message now available."""
        self.buffer.extend(data)
        while True:
            if len(self.buffer) < 4:
                return                                   # header incomplete
            (length,) = struct.unpack_from("!I", self.buffer, 0)

            if length > self.max_message:                # THE CRITICAL CHECK
                raise ValueError(f"declared length {length} exceeds maximum")

            if len(self.buffer) < 4 + length:
                return                                   # body incomplete
            message = bytes(self.buffer[4:4 + length])
            del self.buffer[:4 + length]                 # consume it
            yield message                                 # loop: there may be more

# Prove it handles every possible split.
framer = LengthPrefixedFramer()
wire = b"".join(framer.encode(m) for m in (b"HELLO", b"WORLD", b"AGAIN"))
print(f"wire ({len(wire)} bytes): {wire.hex()}")
# wire (27 bytes): 0000000548454c4c4f00000005574f524c440000000541474149 4e

splits = {
    "all at once":      [wire],
    "byte by byte":     [wire[i:i+1] for i in range(len(wire))],
    "awkward chunks":   [wire[0:7], wire[7:15], wire[15:18], wire[18:]],
    "two messages then one": [wire[:18], wire[18:]],
}
for name, chunks in splits.items():
    f = LengthPrefixedFramer()
    out = []
    for chunk in chunks:
        out.extend(f.feed(chunk))
    print(f"  {name:<24} -> {out}")
#   all at once              -> [b'HELLO', b'WORLD', b'AGAIN']
#   byte by byte             -> [b'HELLO', b'WORLD', b'AGAIN']
#   awkward chunks           -> [b'HELLO', b'WORLD', b'AGAIN']
#   two messages then one    -> [b'HELLO', b'WORLD', b'AGAIN']
#   Identical output for every split. That is the correctness criterion.

# The attack an unchecked length prefix permits.
malicious = struct.pack("!I", 0xFFFFFFFF) + b"x"
f = LengthPrefixedFramer()
try:
    list(f.feed(malicious))
except ValueError as e:
    print(f"\nrejected: {e}")
# rejected: declared length 4294967295 exceeds maximum
#   Without the check: buffer.reserve(4 GB) -> OOM -> process killed.
#   One 5-byte message takes down the server.

# Delimiter framing, with the guard it also needs.
class DelimiterFramer:
    def __init__(self, delimiter=b"\r\n", max_line=8192):
        self.buffer = bytearray()
        self.delimiter = delimiter
        self.max_line = max_line

    def feed(self, data):
        self.buffer.extend(data)
        while (idx := self.buffer.find(self.delimiter)) != -1:
            line = bytes(self.buffer[:idx])
            del self.buffer[:idx + len(self.delimiter)]
            yield line
        # A peer that never sends the delimiter must not grow the buffer forever.
        if len(self.buffer) > self.max_line:
            raise ValueError("line too long without a delimiter")

d = DelimiterFramer()
print(f"\ndelimiter framing: {list(d.feed(b'GET /\r\nHost: x\r\nincomplete'))}")
print(f"  retained in buffer: {bytes(d.buffer)!r}")
# delimiter framing: [b'GET /', b'Host: x']
#   retained in buffer: b'incomplete'

# Comparing the four methods on a 100-byte payload.
def overhead(method, payload_len=100):
    return {"fixed length": 0, "length prefix (4B)": 4,
            "delimiter (\\r\\n)": 2, "self-describing (JSON)": 2}[method]

print(f"\n{'method':<26}{'overhead':>10}{'parse':>12}{'guard needed':>15}")
rows = [("fixed length", "O(1)", "inherent"),
        ("length prefix (4B)", "O(1)", "REQUIRED"),
        ("delimiter (\\r\\n)", "O(n) scan", "REQUIRED"),
        ("self-describing (JSON)", "O(n) parse", "REQUIRED")]
for m, parse, guard in rows:
    print(f"{m:<26}{overhead(m):>9}B{parse:>12}{guard:>15}")

# The classic bug: processing only ONE message per read.
def broken_feed(buffer, data):
    """Handles one message and returns -- leftover messages pile up."""
    buffer.extend(data)
    if len(buffer) >= 4:
        (length,) = struct.unpack_from("!I", buffer, 0)
        if len(buffer) >= 4 + length:
            msg = bytes(buffer[4:4+length])
            del buffer[:4+length]
            return [msg]                      # RETURNS after ONE
    return []

buf = bytearray()
print(f"\nbroken (one per read): {broken_feed(buf, wire)}")
print(f"  left unprocessed: {len(buf)} bytes of complete messages")
# broken (one per read): [b'HELLO']
#   left unprocessed: 18 bytes of complete messages
#   Those two messages are delayed until the NEXT read arrives -- and if the
#   peer is waiting for a reply before sending more, this DEADLOCKS.

#include <stdint.h>
#include <string.h>
#include <arpa/inet.h>
#include <stdio.h>
#include <stdlib.h>

#define MAX_MESSAGE (1u << 20)      /* 1 MiB cap on any declared length */

typedef struct {
    uint8_t *data;
    size_t   len, cap;
} Buffer;

/* Extract as many complete messages as the buffer currently holds.
 * Returns the number extracted; calls handler() for each. */
int extract_messages(Buffer *b, void (*handler)(const uint8_t *, uint32_t)) {
    int count = 0;
    for (;;) {
        if (b->len < 4) break;                      /* header incomplete */

        uint32_t net_len;
        memcpy(&net_len, b->data, 4);
        uint32_t length = ntohl(net_len);           /* ALWAYS convert */

        if (length > MAX_MESSAGE) {                 /* THE CRITICAL CHECK:
                                                     * without it, a 4-byte
                                                     * field requests 4 GB */
            fprintf(stderr, "length %u exceeds max; closing\n", length);
            return -1;
        }
        if (b->len < 4 + (size_t)length) break;     /* body incomplete */

        handler(b->data + 4, length);
        memmove(b->data, b->data + 4 + length,      /* consume the message */
                b->len - 4 - length);
        b->len -= 4 + length;
        count++;                                     /* LOOP: more may follow */
    }
    return count;
}

/* The receive loop that uses it. */
void recv_loop(int fd, void (*handler)(const uint8_t *, uint32_t)) {
    Buffer b = { .data = malloc(65536), .len = 0, .cap = 65536 };
    for (;;) {
        /* Grow if needed, bounded by 4 + MAX_MESSAGE. */
        if (b.cap - b.len < 4096) {
            size_t want = b.cap * 2;
            if (want > 4 + MAX_MESSAGE) want = 4 + MAX_MESSAGE;
            b.data = realloc(b.data, want);
            b.cap = want;
        }
        ssize_t n = recv(fd, b.data + b.len, b.cap - b.len, 0);
        if (n <= 0) break;                           /* 0 = EOF, <0 = error */
        b.len += (size_t)n;

        if (extract_messages(&b, handler) < 0) break;  /* protocol violation */
    }
    free(b.data);
}

# See framing on the wire -- a length prefix is visible as a size field:
$ sudo tcpdump -i any -n -X 'port 9000' -c 1 2>/dev/null | head -3
# 0x0000:  0000 0005 4845 4c4c 4f00 0000 0557 4f52
#          ^^^^^^^^^ length=5  ^^^^^^^^^^ "HELLO"
#                                        ^^^^^^^^^ length=5, next message
#   Two messages in ONE TCP segment. Without framing they are indistinguishable.

# Demonstrate coalescing forcing the issue:
$ python3 -c "
import socket
s = socket.create_connection(('127.0.0.1', 9000))
s.sendall(b'MSG1'); s.sendall(b'MSG2'); s.sendall(b'MSG3')"
$ sudo tcpdump -i lo -n 'port 9000' -c 1
# IP 127.0.0.1.44122 > 127.0.0.1.9000: Flags [P.], length 12
#                                                       ^^ 3 writes -> 1 segment

# Force a split at an MTU boundary to test your framer:
$ sudo tc qdisc add dev lo root netem delay 10ms
$ sudo ip link set dev lo mtu 1280        # smaller MTU = more splits
# ... run your tests ...
$ sudo tc qdisc del dev lo root
$ sudo ip link set dev lo mtu 65536

# Test framing robustness by sending one byte at a time:
$ python3 -c "
import socket, time, struct
payload = b'HELLO'
wire = struct.pack('!I', len(payload)) + payload
s = socket.create_connection(('127.0.0.1', 9000))
for b in wire:
    s.sendall(bytes([b])); time.sleep(0.01)   # worst-case fragmentation
print('sent byte-by-byte')"
#   A correct framer reassembles this perfectly. A broken one hangs.

# Probe for the unchecked-length vulnerability (on YOUR OWN service):
$ python3 -c "
import socket
s = socket.create_connection(('127.0.0.1', 9000))
s.sendall(b'\xff\xff\xff\xff')            # declare a 4 GB message
print('sent oversized length prefix')"
$ dmesg | tail -2
# Out of memory: Killed process 4821 (myserver)
#   If this kills your server, add the MAX_MESSAGE check.

Key takeaways

• TCP provides no message boundaries; framing is an application responsibility, and code that works without it works by accident.
• The four methods are fixed length, length prefix, delimiter, and self-describing; length prefix is the standard choice for binary protocols.
• The receiver must use an accumulating buffer and extract all complete messages per read — handling only one causes delays and can deadlock.
• A read may yield zero, one, or several messages; correct code produces identical results for every possible split.
• Always enforce a maximum message size: an unchecked 4-byte length field lets a 5-byte message request a 4 GB allocation.
• Delimiter framing also needs a maximum, or a peer that never sends the delimiter grows your buffer without bound; it additionally needs escaping when the delimiter can appear in data.

🧪 Practice

1. Show the exact bytes for the messages OK and DONE under 4-byte length-prefix framing.
2. Write pseudocode for a receive loop that correctly extracts multiple messages from a single read.
3. Explain the denial-of-service risk in a 4-byte length prefix and the one check that removes it.
4. Interview-style: A colleague says framing is unnecessary because their tests always receive exactly one message per read(). (Hint: ask what their tests run over, how large their messages are, and what happens when a message crosses an MTU boundary.)

Serialization: JSON, Protocol Buffers, MessagePack

Theory

Framing tells you where a message ends; serialisation determines how structured data — objects, nested records, lists — becomes bytes and back again. The three questions that separate the options are: how large is the output, how fast is encoding and decoding, and what happens when the two sides run different versions of the schema.

That third question is the one teams underestimate and the one that causes production incidents, so it deserves emphasis up front. Any long-lived system will at some point run old senders against new receivers and vice versa, during deployment if nowhere else. A format that cannot tolerate this forces lockstep upgrades across every service.

JSON is text, schemaless, and self-describing. Its advantages are overwhelming for interoperability: every language parses it, humans read it, and browsers speak it natively. Its costs are size (field names repeat in every message), parse speed, and a genuinely weak type system — no integer/float distinction, no binary type (base64 at +33%), and no standard date type. Numbers above 2^53 lose precision silently in JavaScript, which is why APIs return large IDs as strings.

Protocol Buffers are binary and schema-driven: you define messages in a .proto file and generate code. Each field has a number, and it is the number — not the name — that appears on the wire. This is what makes evolution work: renaming a field is harmless, and unknown fields are skipped rather than rejected. Output is typically 3–10× smaller than JSON and considerably faster to parse. The cost is tooling: you need the compiler, generated code, and the schema to interpret bytes at all.

MessagePack is best described as binary JSON: the same schemaless, self-describing model, encoded compactly. It needs no schema or code generation, is a near drop-in replacement for JSON, and is roughly 20–50% smaller with faster parsing — but field names still appear in every message, so it never reaches protobuf's compactness.

	JSON	Protocol Buffers	MessagePack
Encoding	Text	Binary	Binary
Schema	None	Required (.proto)	None
Size (relative)	1.0×	0.2–0.4×	0.6–0.8×
Parse speed	1.0×	3–10×	2–3×
Human readable	Yes	No	No
Binary data	base64 (+33%)	Native	Native
Schema evolution	Ad hoc	Designed for it	Ad hoc
Tooling needed	None	Compiler + codegen	Library only

Protobuf's evolution rules are worth memorising, because they are the payoff for the schema burden:

• Never reuse a field number. Reserve retired numbers explicitly with reserved.
• Adding optional fields is always safe — old readers skip what they do not know.
• Renaming a field is safe; the wire format carries numbers.
• Changing a field's type is generally unsafe, except between specific compatible pairs.
• Removing a required field breaks old readers — which is why proto3 removed required entirely.

The practical guidance: use JSON for public APIs and anything a browser or third party consumes, protobuf for internal service-to-service traffic at volume, and MessagePack when you want JSON's flexibility with less overhead and no build-step.

Examples

The same record, three encodings

  {id: 12345, name: "alice", active: true, score: 99.5}

 JSON -- 62 bytes, readable, field names repeated
   {"id":12345,"name":"alice","active":true,"score":99.5}
    ^^^^        ^^^^^^         ^^^^^^^^      ^^^^^^^
    field names cost bytes in EVERY message

 MessagePack -- 41 bytes, binary, field names still present
   84 a2 69 64 cd 30 39 a4 6e 61 6d 65 a5 61 6c 69 63 65 ...
   ^^ map(4)  ^^^^^ "id"  ^^^^^^^^ 12345  ^^^^^ "name"

 Protobuf -- 21 bytes, binary, field NUMBERS not names
   08 b9 60 12 05 61 6c 69 63 65 18 01 25 00 00 c7 42
   ^^ field 1, varint      ^^ field 2, length-delimited
      ^^^^^^ 12345            ^^^^^ len=5, "alice"
   Field names appear NOWHERE on the wire. That is where the size goes.


 PROTOBUF WIRE FORMAT: tag = (field_number << 3) | wire_type

   0x08 = 0000 1000
          ^^^^^ field 1
                ^^^ wire type 0 (varint)

   wire types:  0 varint (int32/64, bool, enum)
                1 64-bit (double, fixed64)
                2 length-delimited (string, bytes, nested)
                5 32-bit (float, fixed32)

   An unknown field number is SKIPPED using its wire type --
   which is exactly why forward compatibility works.


 SCHEMA EVOLUTION: old reader, new message

   new sender: {id:1, name:"a", email:"a@x"}  <- field 4 added
   old reader: knows fields 1,2,3
               reads 1, reads 2, sees tag for 4 -> UNKNOWN
               -> reads its wire type, SKIPS it, continues
               -> no error, no data loss for known fields

   JSON: same behaviour if the parser ignores unknown keys,
         but nothing in the format ENFORCES it.

syntax = "proto3";

package example;

message User {
  // The NUMBER is the wire identity. Never change or reuse it.
  int32  id     = 1;
  string name   = 2;
  bool   active = 3;
  float  score  = 4;

  // Added in v2: old readers skip it harmlessly.
  string email  = 5;

  // Field 6 held `legacy_token`, removed in v3.
  // Reserving it prevents a future developer from reusing the number
  // and silently misinterpreting old stored messages.
  reserved 6;
  reserved "legacy_token";

  repeated string roles = 7;          // repeated = a list
  map<string, string> labels = 8;     // map support is built in
}

import json, struct, time

record = {"id": 12345, "name": "alice", "active": True, "score": 99.5}

# --- JSON ---
json_bytes = json.dumps(record, separators=(",", ":")).encode()
print(f"JSON        {len(json_bytes):>3} bytes  {json_bytes}")
# JSON         54 bytes  b'{"id":12345,"name":"alice","active":true,"score":99.5}'

# --- Protobuf, encoded by hand to show the wire format ---
def varint(n):
    """Base-128 encoding: 7 bits per byte, high bit = 'more follows'."""
    out = bytearray()
    while True:
        b = n & 0x7F
        n >>= 7
        out.append(b | (0x80 if n else 0))
        if not n:
            return bytes(out)

def tag(field_number, wire_type):
    return varint((field_number << 3) | wire_type)

pb = (tag(1, 0) + varint(12345) +                       # id: varint
      tag(2, 2) + varint(5) + b"alice" +                 # name: length-delimited
      tag(3, 0) + varint(1) +                            # active: varint(bool)
      tag(4, 5) + struct.pack("<f", 99.5))               # score: 32-bit float
print(f"protobuf    {len(pb):>3} bytes  {pb.hex()}")
# protobuf     21 bytes  08b96012056 16c6963651801250000c742

# --- MessagePack, encoded by hand ---
def msgpack_encode(obj):
    if isinstance(obj, dict):
        out = bytearray([0x80 | len(obj)])               # fixmap
        for k, v in obj.items():
            out += msgpack_encode(k); out += msgpack_encode(v)
        return bytes(out)
    if isinstance(obj, str):
        b = obj.encode()
        return bytes([0xA0 | len(b)]) + b                # fixstr
    if isinstance(obj, bool):
        return bytes([0xC3 if obj else 0xC2])
    if isinstance(obj, int):
        return bytes([0xCD]) + struct.pack(">H", obj) if obj > 127 else bytes([obj])
    if isinstance(obj, float):
        return bytes([0xCB]) + struct.pack(">d", obj)
    raise TypeError(type(obj))

mp = msgpack_encode(record)
print(f"messagepack {len(mp):>3} bytes  {mp.hex()[:40]}...")
# messagepack  45 bytes  84a26964cd3039a46e616d65a5616c696365...

print(f"\n{'format':<14}{'bytes':>7}{'relative':>11}")
for name, data in (("JSON", json_bytes), ("MessagePack", mp), ("protobuf", pb)):
    print(f"{name:<14}{len(data):>7}{len(data)/len(json_bytes):>10.2f}x")
# format          bytes   relative
# JSON               54       1.00x
# MessagePack        45       0.83x
# protobuf           21       0.39x

# Bandwidth at scale.
def monthly_gb(msgs_per_sec, size):
    return msgs_per_sec * size * 86400 * 30 / 1e9

print(f"\nAt 50,000 msg/s:")
for name, data in (("JSON", json_bytes), ("MessagePack", mp), ("protobuf", pb)):
    print(f"  {name:<14}{monthly_gb(50_000, len(data)):>10,.0f} GB/month")
#   JSON                 6,998 GB/month
#   MessagePack          5,832 GB/month
#   protobuf             2,722 GB/month
#   ~4.3 TB/month saved by the encoding choice alone.

# Forward compatibility: an old reader meets a new field.
def parse_protobuf_skipping_unknown(data, known_fields={1, 2, 3, 4}):
    """Demonstrates WHY protobuf evolution works: unknown fields are skippable."""
    pos, out, skipped = 0, {}, []
    while pos < len(data):
        # Read the tag varint.
        t, shift = 0, 0
        while True:
            b = data[pos]; pos += 1
            t |= (b & 0x7F) << shift; shift += 7
            if not (b & 0x80): break
        field, wire = t >> 3, t & 0x07

        # The wire type tells us the LENGTH even when the field is unknown.
        if wire == 0:                                    # varint
            v, shift = 0, 0
            while True:
                b = data[pos]; pos += 1
                v |= (b & 0x7F) << shift; shift += 7
                if not (b & 0x80): break
        elif wire == 2:                                  # length-delimited
            ln, shift = 0, 0
            while True:
                b = data[pos]; pos += 1
                ln |= (b & 0x7F) << shift; shift += 7
                if not (b & 0x80): break
            v = data[pos:pos+ln]; pos += ln
        elif wire == 5:                                  # 32-bit
            v = struct.unpack_from("<f", data, pos)[0]; pos += 4
        elif wire == 1:                                  # 64-bit
            v = struct.unpack_from("<d", data, pos)[0]; pos += 8
        else:
            raise ValueError(f"unknown wire type {wire}")

        if field in known_fields:
            out[field] = v
        else:
            skipped.append(field)                        # SKIPPED, not an error
    return out, skipped

new_message = pb + tag(5, 2) + varint(7) + b"a@x.com"    # field 5 added
parsed, skipped = parse_protobuf_skipping_unknown(new_message)
print(f"\nold reader parsed fields {sorted(parsed)}, skipped unknown {skipped}")
# old reader parsed fields [1, 2, 3, 4], skipped unknown [5]
#   No error. Known data intact. This is the whole point of the schema.

# The evolution rules that make that safe.
rules = [
    ("add an optional field",        "SAFE",   "old readers skip it"),
    ("rename a field",               "SAFE",   "the wire carries NUMBERS"),
    ("remove a field",               "SAFE*",  "*reserve the number afterwards"),
    ("REUSE a retired field number", "BREAKS", "old data is silently misread"),
    ("change int32 -> string",       "BREAKS", "incompatible wire type"),
    ("change int32 -> int64",        "SAFE",   "both are varints"),
]
print()
for change, verdict, why in rules:
    print(f"  {change:<32} {verdict:<7} {why}")

# JSON's precision trap, which drives a common API convention.
big_id = 9007199254740993                                # 2^53 + 1
print(f"\npython int : {big_id}")
print(f"through JSON parsed as a JS double: {float(big_id):.0f}")
# python int : 9007199254740993
# through JSON parsed as a JS double: 9007199254740992
#   Off by one, silently. This is why APIs return large IDs as STRINGS.

# Compile a schema and inspect the generated code:
$ protoc --python_out=. --go_out=. user.proto
$ ls
# user.proto  user_pb2.py  user.pb.go

# Decode protobuf bytes WITHOUT the schema -- field numbers only:
$ printf '\x08\xb9\x60\x12\x05alice\x18\x01' | protoc --decode_raw
# 1: 12345
# 2: "alice"
# 3: 1
#   Note: names are absent. Without the .proto you get numbers, not meaning.

# Decode WITH the schema:
$ printf '\x08\xb9\x60\x12\x05alice\x18\x01' | protoc --decode=example.User user.proto
# id: 12345
# name: "alice"
# active: true

# Compare sizes for a real payload:
$ cat record.json | wc -c
# 54
$ protoc --encode=example.User user.proto < record.txt | wc -c
# 21

# Check a schema change for compatibility BEFORE shipping it:
$ buf breaking --against '.git#branch=main'
# user.proto:12:3: Field "6" with name "legacy_token" on message "User"
#   was deleted without reserving the number.
#   ^ exactly the mistake that silently corrupts stored data later

# Benchmark encode/decode in your own language:
$ python3 -m timeit -s "
import json
d = {'id':12345,'name':'alice','active':True,'score':99.5}" \
  "json.dumps(d)"
# 500000 loops, best of 5: 812 nsec per loop

$ python3 -m timeit -s "
import msgpack
d = {'id':12345,'name':'alice','active':True,'score':99.5}" \
  "msgpack.packb(d)"
# 1000000 loops, best of 5: 291 nsec per loop

# Measure what the encoding costs you in production bandwidth:
$ ss -tin dst :50051 | grep -o 'bytes_sent:[0-9]*'
# bytes_sent:4820113920

Key takeaways

• Serialisation is judged on size, speed, and schema evolution — the third matters most in long-lived systems, because versions always skew during deployment.
• JSON is schemaless, universal, and human-readable, but repeats field names, lacks a binary type (base64 costs +33%), and loses integer precision above 2^53.
• Protocol Buffers are schema-driven and put field numbers, not names, on the wire, giving 3–10× smaller messages and explicit evolution rules.
• MessagePack is "binary JSON": schemaless and drop-in, 20–50% smaller than JSON, but field names still appear in every message.
• Protobuf rules: never reuse a field number (use reserved), adding optional fields and renaming are safe, and type changes are generally unsafe.
• Unknown protobuf fields are skipped using their wire type, which is the mechanism behind forward compatibility.
• Use JSON for public APIs, protobuf for high-volume internal traffic, MessagePack for JSON-like flexibility without a build step.

🧪 Practice

1. Encode {id: 7, name: "bo"} as JSON and as protobuf by hand, and compare the byte counts.
2. State which of these are safe: renaming field 3, adding field 9, reusing retired number 4, changing int32 to int64.
3. Explain why large numeric IDs are commonly returned as strings in JSON APIs.
4. Interview-style: A new service version adds a field, and an older consumer starts rejecting messages entirely. (Hint: protobuf skips unknown fields by design — so ask what the consumer is actually parsing with, and whether its parser was configured to be strict.)

REST Design Principles

Theory

REST (Representational State Transfer) was Roy Fielding's 2000 description of the architectural style the web already embodied. It is not a protocol or a standard — it is a set of constraints, and following them yields systems that cache well, scale horizontally, and evolve without breaking clients.

The constraints that matter in practice:

1. Resources, identified by URIs. The unit of design is a noun, not a verb. /users/42 is a resource; /getUser?id=42 is a remote procedure call wearing a URI. The distinction is not pedantry — it determines whether caching, conditional requests, and standard method semantics apply at all.

2. Uniform interface via HTTP methods. The methods from Chapter 8 carry meaning that every intermediary understands: GET is safe and cacheable, PUT and DELETE are idempotent, POST is neither. A cache can store a GET because the method promises it has no side effects. Overloading POST for reads discards that entire layer of infrastructure.

3. Statelessness. Every request carries everything needed to process it. No server-side session affinity, so any server can handle any request — the property that makes load balancing and horizontal scaling trivial.

4. Representations. A resource is abstract; what you transfer is a representation, negotiated with Accept and Content-Type. One resource can be JSON, HTML, or CSV.

5. HATEOAS — responses include links to available next actions. This is the least-adopted constraint, and honesty requires saying so: most "REST" APIs are not hypermedia-driven, and that is a defensible engineering choice rather than a failure.

The design decisions that cause the most real-world pain:

Versioning. URI-based (/v1/users) is explicit and cache-friendly but proliferates paths; header-based (Accept: application/vnd.api+json;version=1) is cleaner but harder to test with a browser. Most teams choose URI versioning for its sheer obviousness.

Pagination. Offset-based (?page=3&size=20) is simple but degrades on large offsets — the database must still scan the skipped rows — and skips or repeats items when the data changes between requests. Cursor-based pagination (?after=<opaque-cursor>) is stable under concurrent modification and performs consistently, and is the right default for anything large.

Error responses should be machine-parseable and consistent. RFC 9457 (formerly 7807) defines application/problem+json with type, title, status, detail, and instance — worth adopting rather than inventing another error shape.

Partial updates. PUT replaces the whole resource; PATCH modifies part of it. Sending a partial body with PUT and expecting a merge is a common and damaging mistake — strictly interpreted, it deletes every omitted field.

Examples

Resource-oriented vs RPC-style URIs

  RPC-STYLE (verbs in the path)        RESTful (nouns + methods)
  POST /getUser?id=42                  GET    /users/42
  POST /createUser                     POST   /users
  POST /updateUserEmail                PATCH  /users/42
  POST /deleteUser?id=42               DELETE /users/42
  POST /listUserOrders?id=42           GET    /users/42/orders

  Everything is POST, so:              Methods carry meaning, so:
   - nothing is cacheable               - GET responses cache
   - nothing is safely retryable        - PUT/DELETE retry safely
   - proxies cannot help                - proxies and CDNs work
   - the URI count grows per operation  - one URI, several operations


 PAGINATION: why cursors beat offsets

  OFFSET (?page=2&size=3) on a list where a row is INSERTED between calls

   t=0  page 1: [A][B][C]              t=1  a new row X is inserted at the top
   t=2  page 2 (offset 3):  [C][D][E]
                             ^ C IS REPEATED -- the window shifted

  CURSOR (?after=cursor_C)

   t=0  page 1: [A][B][C]  next_cursor = "C"
   t=1  X inserted at the top -- irrelevant, the cursor names a POSITION
   t=2  page 2 (after=C):  [D][E][F]     correct, no repeat, no skip

  Also: OFFSET 1000000 makes the database scan a million rows.
        A cursor is an indexed seek.


 STATUS CODE SELECTION

   POST /users        -> 201 Created + Location: /users/42
   GET  /users/42     -> 200 OK
   GET  /users/999    -> 404 Not Found
   PUT  /users/42     -> 200 OK  (or 204 No Content)
   DELETE /users/42   -> 204 No Content
   POST /users (dup)  -> 409 Conflict
   POST /users (bad)  -> 422 Unprocessable Content
   any, rate limited  -> 429 Too Many Requests + Retry-After

import json, base64, hmac, hashlib
from datetime import datetime, timezone

# Resource-oriented routing: nouns and methods, not verbs.
ROUTES = {
    ("GET",    "/users"):        "list users (paginated)",
    ("POST",   "/users"):        "create -> 201 + Location",
    ("GET",    "/users/{id}"):   "retrieve one",
    ("PUT",    "/users/{id}"):   "REPLACE entirely (idempotent)",
    ("PATCH",  "/users/{id}"):   "partial update",
    ("DELETE", "/users/{id}"):   "remove (idempotent)",
    ("GET",    "/users/{id}/orders"): "sub-resource collection",
}
for (method, path), desc in ROUTES.items():
    print(f"  {method:<7}{path:<22}{desc}")

# PUT vs PATCH: the difference that destroys data when confused.
existing = {"id": 42, "name": "alice", "email": "a@x.com", "role": "admin"}

def apply_put(resource, body):
    """REPLACE: omitted fields are DELETED. This is the correct semantics."""
    return {"id": resource["id"], **body}

def apply_patch(resource, body):
    """MERGE: only the supplied fields change (RFC 7396 merge-patch)."""
    merged = dict(resource)
    for k, v in body.items():
        if v is None: merged.pop(k, None)     # null means DELETE this field
        else:         merged[k] = v
    return merged

partial = {"email": "new@x.com"}
print(f"\nexisting      : {existing}")
print(f"PUT {partial}  -> {apply_put(existing, partial)}")
print(f"PATCH {partial}-> {apply_patch(existing, partial)}")
# existing      : {'id': 42, 'name': 'alice', 'email': 'a@x.com', 'role': 'admin'}
# PUT {'email': 'new@x.com'}  -> {'id': 42, 'email': 'new@x.com'}
#                                 ^ name and role are GONE. Correct per spec.
# PATCH {'email': 'new@x.com'}-> {'id': 42, 'name': 'alice', 'email': 'new@x.com',
#                                 'role': 'admin'}

# Cursor pagination: stable under concurrent modification.
SECRET = b"cursor-signing-key"

def make_cursor(last_id, last_sort_value):
    """Opaque and SIGNED, so clients cannot forge positions or probe data."""
    payload = json.dumps({"id": last_id, "v": last_sort_value}).encode()
    sig = hmac.new(SECRET, payload, hashlib.sha256).digest()[:8]
    return base64.urlsafe_b64encode(payload + sig).decode().rstrip("=")

def parse_cursor(cursor):
    raw = base64.urlsafe_b64decode(cursor + "==")
    payload, sig = raw[:-8], raw[-8:]
    expected = hmac.new(SECRET, payload, hashlib.sha256).digest()[:8]
    if not hmac.compare_digest(sig, expected):
        raise ValueError("invalid cursor")
    return json.loads(payload)

cur = make_cursor(1042, "2026-09-23T09:14:15Z")
print(f"\ncursor: {cur[:40]}...")
print(f"parsed: {parse_cursor(cur)}")
# cursor: eyJpZCI6IDEwNDIsICJ2IjogIjIwMjYtMDktMjNU...
# parsed: {'id': 1042, 'v': '2026-09-23T09:14:15Z'}

# Offset vs cursor, on a changing dataset.
def offset_page(items, offset, size):
    return items[offset:offset+size]

def cursor_page(items, after_id, size):
    start = next((i + 1 for i, x in enumerate(items) if x["id"] == after_id), 0)
    return items[start:start+size]

items = [{"id": i} for i in range(1, 7)]
p1 = offset_page(items, 0, 3)
items.insert(0, {"id": 0})                      # a row is inserted between calls
p2_offset = offset_page(items, 3, 3)
p2_cursor = cursor_page(items, p1[-1]["id"], 3)
print(f"\npage 1              : {[x['id'] for x in p1]}")
print(f"page 2 by offset    : {[x['id'] for x in p2_offset]}  <- id 3 REPEATED")
print(f"page 2 by cursor    : {[x['id'] for x in p2_cursor]}  <- correct")
# page 1              : [1, 2, 3]
# page 2 by offset    : [3, 4, 5]  <- id 3 REPEATED
# page 2 by cursor    : [4, 5, 6]  <- correct

# RFC 9457 problem+json: one consistent, machine-parseable error shape.
def problem(status, title, detail, type_uri="about:blank", **extra):
    return {"type": type_uri, "title": title, "status": status,
            "detail": detail, "instance": extra.pop("instance", None), **extra}

err = problem(422, "Validation failed",
              "The 'email' field is not a valid address.",
              type_uri="https://api.example.com/problems/validation",
              instance="/users", errors=[{"field": "email", "code": "format"}])
print(f"\nContent-Type: application/problem+json")
print(json.dumps(err, indent=2)[:260])
# {
#   "type": "https://api.example.com/problems/validation",
#   "title": "Validation failed",
#   "status": 422,
#   "detail": "The 'email' field is not a valid address.",
#   "instance": "/users",
#   "errors": [ { "field": "email", "code": "format" } ]

# Versioning strategies, with their real trade-offs.
strategies = [
    ("URI path",    "/v1/users",                          "obvious, cacheable, browser-testable; path proliferation"),
    ("Accept header","Accept: application/vnd.api+json;version=1", "clean URIs; hard to test by hand, cache needs Vary"),
    ("Query param", "/users?version=1",                   "easy; pollutes the cache key"),
]
print()
for name, example, tradeoff in strategies:
    print(f"  {name:<14} {example:<44} {tradeoff}")

# A well-behaved creation: 201 with a Location header.
$ curl -si -X POST https://api.example.com/users \
    -H 'Content-Type: application/json' \
    -d '{"name":"alice","email":"a@x.com"}' | head -4
# HTTP/2 201
# location: /users/42
# content-type: application/json
#   201, not 200: the client learns WHERE the new resource lives.

# GET is cacheable because the method promises no side effects:
$ curl -si https://api.example.com/users/42 | grep -iE 'cache-control|etag'
# cache-control: private, max-age=60
# etag: "a7f3c91e"

# Conditional request using the ETag -- the mechanism from Chapter 8:
$ curl -si -H 'If-None-Match: "a7f3c91e"' https://api.example.com/users/42 | head -1
# HTTP/2 304

# Optimistic concurrency: refuse a write based on a stale version.
$ curl -si -X PUT -H 'If-Match: "a7f3c91e"' \
    -H 'Content-Type: application/json' \
    -d '{"name":"alice","email":"new@x.com","role":"admin"}' \
    https://api.example.com/users/42 | head -1
# HTTP/2 412            <- Precondition Failed: someone else updated it first

# Cursor pagination in practice:
$ curl -s 'https://api.example.com/users?limit=3' | jq -c '{data:[.data[].id],next:.next_cursor}'
# {"data":[1,2,3],"next":"eyJpZCI6M30"}
$ curl -s 'https://api.example.com/users?limit=3&after=eyJpZCI6M30' | jq -c '[.data[].id]'
# [4,5,6]

# Errors in a consistent, parseable shape:
$ curl -s -X POST https://api.example.com/users -d '{"email":"bad"}' | jq .
# {
#   "type": "https://api.example.com/problems/validation",
#   "title": "Validation failed",
#   "status": 422,
#   "errors": [{"field": "email", "code": "format"}]
# }

# Check which methods a resource supports:
$ curl -si -X OPTIONS https://api.example.com/users/42 | grep -i '^allow'
# allow: GET, PUT, PATCH, DELETE, OPTIONS

# Verify idempotency by repeating a DELETE:
$ for i in 1 2 3; do
>   curl -s -o /dev/null -w "%{http_code} " -X DELETE https://api.example.com/users/42
> done; echo
# 204 404 404
#   Different codes, same END STATE. That is what idempotent means.

Key takeaways

• REST is a set of constraints, not a protocol: resources as nouns, a uniform interface of HTTP methods, statelessness, and negotiated representations.
• Method semantics are what let caches, proxies, and retry logic work — overloading POST for reads discards that infrastructure.
• PUT replaces the whole resource (omitted fields are deleted); PATCH merges — confusing them destroys data.
• Cursor pagination is stable under concurrent modification and performs consistently; offset pagination skips or repeats rows and degrades at large offsets.
• Use RFC 9457 application/problem+json for consistent, machine-parseable errors rather than inventing a shape.
• URI versioning is the common choice for obviousness; header versioning is cleaner but harder to test and needs Vary.
• HATEOAS is rarely implemented, and omitting it is a defensible engineering decision.

🧪 Practice

1. Rewrite these as resource-oriented endpoints: POST /getOrderHistory, POST /cancelOrder, POST /addItemToCart.
2. Explain what a client receives if it sends {"email":"x@y.com"} via PUT to a resource with four fields, and why.
3. Give two distinct reasons cursor pagination is preferable to offset pagination for a large, actively-written table.
4. Interview-style: An API returns 200 OK with {"success": false, "error": "not found"} for missing resources. Critique this. (Hint: think about every intermediary between client and server that reads only the status line, and what each of them now believes.)

gRPC and RPC Semantics

Theory

RPC (Remote Procedure Call) aims to make calling a function on another machine look like calling one locally. You write user = client.GetUser(id) and the framework handles serialisation, transport, and deserialisation.

The appeal is obvious, and so is the danger. The classic paper A Note on Distributed Computing (Waldo et al., 1994) makes the essential argument: a remote call is fundamentally different from a local one, and pretending otherwise produces fragile systems. A local call cannot partially fail, has nanosecond latency, and either returns or throws. A remote call can hang forever, fail after succeeding, or succeed after the caller gave up. Good RPC frameworks therefore expose timeouts, retries, and errors rather than hiding them.

	Local call	Remote call
Latency	~1 ns	0.1–100 ms (10⁵–10⁸× slower)
Failure modes	Exception	Timeout, partial failure, network partition
Argument passing	Reference	Copy (serialised)
Can hang forever	No	Yes
Succeeds after you gave up	No	Yes

gRPC is Google's framework and the current default for internal service communication. Its design choices are worth understanding individually:

• Protocol Buffers for the interface definition and wire format, giving schema evolution and compact encoding.
• HTTP/2 as transport, which brings multiplexing (many concurrent calls on one connection, no head-of-line blocking at the application layer), header compression, and binary framing — all from Chapter 8.
• Code generation in many languages from one .proto, so the contract is enforced at compile time rather than discovered at runtime.

gRPC supports four call patterns, and the last three are what genuinely distinguish it from REST:

Pattern	Shape	Use
Unary	1 request → 1 response	Ordinary calls
Server streaming	1 request → N responses	Feeds, large result sets
Client streaming	N requests → 1 response	Uploads, batched telemetry
Bidirectional	N ↔ N, independently	Chat, real-time sync

Because streams map onto HTTP/2 streams, all of these run concurrently over a single connection.

Two features deserve emphasis because they encode good distributed-systems practice directly into the framework:

Deadlines, not timeouts. A gRPC deadline is an absolute point in time that propagates across service boundaries. If A calls B with a 500 ms deadline and B calls C, C receives the remaining budget — not a fresh 500 ms. This prevents the cascading amplification where a chain of five services each waits its own full timeout. Deadlines are how you bound total request latency in a call graph.

Status codes are an explicit, finite enumeration: OK, CANCELLED, DEADLINE_EXCEEDED, NOT_FOUND, ALREADY_EXISTS, PERMISSION_DENIED, RESOURCE_EXHAUSTED, FAILED_PRECONDITION, ABORTED, UNAVAILABLE, UNIMPLEMENTED, INTERNAL. Crucially, they distinguish retryable conditions (UNAVAILABLE, RESOURCE_EXHAUSTED) from non-retryable ones (INVALID_ARGUMENT, NOT_FOUND) — which is exactly the information a client needs and which HTTP status codes convey only loosely.

The honest comparison with REST: gRPC wins on performance, streaming, and compile-time contracts; REST wins on browser support, human debuggability, and caching. Browsers cannot speak raw gRPC, requiring gRPC-Web and a proxy. The common architecture is therefore REST at the edge, gRPC between internal services.

Examples

The four call patterns, mapped onto HTTP/2 streams

  UNARY                          SERVER STREAMING
  client        server           client        server
    |--request---->|               |--request---->|
    |<--response---|               |<--response---|
                                   |<--response---|
                                   |<--response---|
                                   |<--(end)------|

  CLIENT STREAMING               BIDIRECTIONAL
  client        server           client        server
    |--request---->|               |--request---->|
    |--request---->|               |<--response---|
    |--request---->|               |--request---->|
    |--(end)------>|               |--request---->|
    |<--response---|               |<--response---|
                                   (independent, interleaved)

  All four ride HTTP/2 streams, so many calls share ONE connection
  with no application-layer head-of-line blocking.


 DEADLINE PROPAGATION -- the feature that prevents cascading timeouts

  WITHOUT propagation (each service has its own timeout)
    A (timeout 500ms) -> B (timeout 500ms) -> C (timeout 500ms)
    C takes 450ms, B adds 60ms -> A has already given up at 500ms
    -> B and C did all that work for NOTHING, and keep doing it

  WITH gRPC deadlines (absolute time, propagated)
    A sets deadline = now + 500ms
    B receives "440ms remaining"  -> passes "380ms remaining" to C
    C knows it has 380ms; if it cannot finish, it fails FAST
    -> no wasted work, and the total is bounded by A's deadline


 STATUS CODES THAT ENCODE RETRY POLICY

   RETRYABLE                        NOT RETRYABLE
   UNAVAILABLE       (try again)    INVALID_ARGUMENT  (fix the request)
   RESOURCE_EXHAUSTED(back off)     NOT_FOUND         (it does not exist)
   ABORTED           (contention)   PERMISSION_DENIED (you are not allowed)
   DEADLINE_EXCEEDED (maybe)        UNIMPLEMENTED     (never will work)

   The client does not have to guess. That is the design win over HTTP codes.

syntax = "proto3";

package users.v1;

service UserService {
  // Unary: the ordinary case.
  rpc GetUser(GetUserRequest) returns (User);

  // Server streaming: one request, a stream of results.
  rpc ListUsers(ListUsersRequest) returns (stream User);

  // Client streaming: a stream of uploads, one summary back.
  rpc ImportUsers(stream User) returns (ImportSummary);

  // Bidirectional: independent streams in both directions.
  rpc SyncUsers(stream UserEvent) returns (stream UserEvent);
}

message GetUserRequest { int32 id = 1; }

message User {
  int32  id    = 1;
  string name  = 2;
  string email = 3;
}

message ListUsersRequest {
  int32  page_size  = 1;
  string page_token = 2;      // cursor pagination, as in the previous topic
}

message ImportSummary {
  int32 imported = 1;
  int32 failed   = 2;
}

message UserEvent {
  enum Kind { UNSPECIFIED = 0; CREATED = 1; UPDATED = 2; DELETED = 3; }
  Kind  kind = 1;
  User  user = 2;
}

import time, grpc                      # illustrative; requires generated stubs

# --- Server: the four patterns ---
class UserService:
    def GetUser(self, request, context):
        """Unary. Always check the deadline before doing expensive work."""
        remaining = context.time_remaining()
        if remaining is not None and remaining < 0.05:
            context.abort(grpc.StatusCode.DEADLINE_EXCEEDED,
                          "insufficient budget remaining")
        user = db_lookup(request.id)
        if user is None:
            # NOT_FOUND is explicitly non-retryable: the client must not retry.
            context.abort(grpc.StatusCode.NOT_FOUND, f"user {request.id}")
        return user

    def ListUsers(self, request, context):
        """Server streaming: yield results as they are produced."""
        for user in db_scan(limit=request.page_size):
            if not context.is_active():         # client cancelled: stop working
                return
            yield user

    def ImportUsers(self, request_iterator, context):
        """Client streaming: consume the stream, return one summary."""
        imported = failed = 0
        for user in request_iterator:
            try:
                db_insert(user); imported += 1
            except Exception:
                failed += 1
        return ImportSummary(imported=imported, failed=failed)

    def SyncUsers(self, request_iterator, context):
        """Bidirectional: read and write independently."""
        for event in request_iterator:
            for out in apply_event(event):
                yield out

# --- Client: deadlines and retry policy ---
def call_with_deadline(stub, user_id, budget_ms=500):
    try:
        # A DEADLINE, not a timeout: absolute, and propagated downstream.
        return stub.GetUser(GetUserRequest(id=user_id),
                            timeout=budget_ms / 1000)
    except grpc.RpcError as e:
        code = e.code()
        retryable = {grpc.StatusCode.UNAVAILABLE,
                     grpc.StatusCode.RESOURCE_EXHAUSTED,
                     grpc.StatusCode.ABORTED}
        return f"{code.name} ({'retryable' if code in retryable else 'permanent'})"

# Deadline propagation, modelled.
def propagate(total_budget_ms, hops):
    remaining, log = total_budget_ms, []
    for i, cost in enumerate(hops, 1):
        log.append((f"service {i}", remaining, cost))
        remaining -= cost
        if remaining <= 0:
            log.append(("ABORT", remaining, 0))
            break
    return log, remaining

print(f"{'hop':<12}{'budget in':>12}{'own cost':>11}")
log, left = propagate(500, [60, 80, 200, 120])
for name, budget, cost in log:
    print(f"{name:<12}{budget:>11}ms{cost:>10}ms")
print(f"remaining after the chain: {left} ms")
# hop            budget in   own cost
# service 1            500ms       60ms
# service 2            440ms       80ms
# service 3            360ms      200ms
# service 4            160ms      120ms
# remaining after the chain: 40 ms
#   Each service knows its REAL budget and can fail fast instead of
#   doing work whose result nobody is still waiting for.

# Without propagation: the cascading-timeout amplification.
def no_propagation(per_service_timeout_ms, depth):
    return per_service_timeout_ms * depth

print(f"\n{'depth':>7}{'with deadlines':>17}{'without':>12}")
for depth in (2, 3, 5, 8):
    print(f"{depth:>7}{500:>15}ms{no_propagation(500, depth):>10}ms")
#   depth   with deadlines     without
#       2            500ms       1000ms
#       3            500ms       1500ms
#       5            500ms       2500ms
#       8            500ms       4000ms
#   Without propagation, worst-case latency multiplies by call depth.

# Status codes drive retry policy without guessing.
RETRYABLE = {"UNAVAILABLE", "RESOURCE_EXHAUSTED", "ABORTED"}
CODES = ["OK", "CANCELLED", "INVALID_ARGUMENT", "DEADLINE_EXCEEDED", "NOT_FOUND",
         "ALREADY_EXISTS", "PERMISSION_DENIED", "RESOURCE_EXHAUSTED",
         "FAILED_PRECONDITION", "ABORTED", "UNAVAILABLE", "UNIMPLEMENTED",
         "INTERNAL"]
print()
for c in CODES:
    print(f"  {c:<22}{'RETRY' if c in RETRYABLE else 'do not retry'}")

# gRPC vs REST, honestly.
comparison = [
    ("wire format",      "protobuf (binary)",  "JSON (text)"),
    ("transport",        "HTTP/2 only",        "HTTP/1.1 or 2"),
    ("streaming",        "4 patterns, native", "SSE/WebSocket bolt-on"),
    ("contract",         "compile-time",       "runtime (OpenAPI helps)"),
    ("browser support",  "needs gRPC-Web+proxy","native"),
    ("caching",          "none",               "full HTTP caching"),
    ("debugging",        "grpcurl needed",     "curl"),
    ("payload size",     "0.2-0.4x",           "1.0x"),
]
print(f"\n{'dimension':<18}{'gRPC':<24}{'REST'}")
for dim, g, r in comparison:
    print(f"{dim:<18}{g:<24}{r}")

# The distributed-computing warning, quantified.
def call_cost():
    return {"local function call": 1e-9, "same-host IPC": 5e-5,
            "same-datacenter RPC": 5e-4, "cross-region RPC": 1.5e-1}
print()
base = 1e-9
for name, seconds in call_cost().items():
    print(f"  {name:<24}{seconds*1e6:>12,.1f} us   {seconds/base:>15,.0f}x local")
#   local function call              0.0 us                 1x local
#   same-host IPC                   50.0 us            50,000x local
#   same-datacenter RPC            500.0 us           500,000x local
#   cross-region RPC           150,000.0 us       150,000,000x local
#   An API that hides this difference invites callers to make it in a loop.

# Explore a gRPC service without writing a client (reflection must be enabled):
$ grpcurl -plaintext localhost:50051 list
# users.v1.UserService
# grpc.reflection.v1alpha.ServerReflection

$ grpcurl -plaintext localhost:50051 list users.v1.UserService
# users.v1.UserService.GetUser
# users.v1.UserService.ListUsers
# users.v1.UserService.ImportUsers
# users.v1.UserService.SyncUsers

# Call a unary method:
$ grpcurl -plaintext -d '{"id": 42}' localhost:50051 users.v1.UserService/GetUser
# {
#   "id": 42,
#   "name": "alice",
#   "email": "a@x.com"
# }

# Set a deadline and observe the status code:
$ grpcurl -plaintext -max-time 0.05 -d '{"id": 42}' \
    localhost:50051 users.v1.UserService/GetUser
# ERROR:
#   Code: DeadlineExceeded
#   Message: context deadline exceeded

# Consume a server stream:
$ grpcurl -plaintext -d '{"page_size": 3}' \
    localhost:50051 users.v1.UserService/ListUsers
# { "id": 1, "name": "alice" }
# { "id": 2, "name": "bob" }
# { "id": 3, "name": "carol" }

# gRPC rides HTTP/2 -- confirm multiplexing on ONE connection:
$ ss -tnp | grep 50051 | wc -l
# 1
$ nghttp -nv http://localhost:50051 2>&1 | grep -c 'stream_id'
# 47        <- 47 concurrent streams, one TCP connection

# Deadline propagation is a header on the wire:
$ sudo tcpdump -i lo -n -A 'port 50051' -c 20 2>/dev/null | grep -o 'grpc-timeout[^ ]*'
# grpc-timeout:440m
#                ^^^ 440 MILLISECONDS remaining, passed downstream

# Check the status code returned for a failure:
$ grpcurl -plaintext -d '{"id": 999999}' \
    localhost:50051 users.v1.UserService/GetUser 2>&1 | grep Code
#   Code: NotFound
#   ^ explicitly non-retryable: the client knows not to try again

# Benchmark against the REST equivalent:
$ ghz --insecure --proto user.proto --call users.v1.UserService/GetUser \
      -d '{"id":42}' -c 50 -n 10000 localhost:50051 | grep -E 'Requests|Average'
# Requests/sec: 48214.02
# Average:      1.03 ms

Key takeaways

• RPC makes a remote call look local, but a remote call is fundamentally different: it can hang forever, fail partially, or succeed after the caller gave up.
• gRPC = Protocol Buffers + HTTP/2 + code generation, giving compact binary messages, multiplexed streams, and compile-time contracts.
• The four call patterns — unary, server streaming, client streaming, bidirectional — all ride HTTP/2 streams on one connection.
• Deadlines are absolute and propagate across service boundaries, so a call chain's total latency is bounded; without propagation, worst-case latency multiplies by call depth.
• gRPC status codes distinguish retryable from non-retryable failures explicitly, removing the guesswork that HTTP codes leave to the client.
• Browsers cannot speak raw gRPC (gRPC-Web plus a proxy is required), and gRPC gives up HTTP caching entirely.
• The common architecture is REST at the edge, gRPC between internal services.

🧪 Practice

1. Choose the appropriate gRPC pattern for: a live log tail, a bulk import, a single lookup, and a collaborative editing session.
2. Explain how deadline propagation prevents cascading timeouts in a four-service call chain, with numbers.
3. Classify these as retryable or not: UNAVAILABLE, INVALID_ARGUMENT, RESOURCE_EXHAUSTED, PERMISSION_DENIED.
4. Interview-style: Why is it a design risk for an RPC framework to make remote calls look exactly like local ones? (Hint: think about what a developer does when a call looks cheap — then reread the latency comparison.)

Idempotency, Retries, and Timeouts

Theory

This topic addresses the hardest fact in distributed systems: when a request times out, you do not know whether it succeeded. The response may have been lost, the server may still be processing, or nothing may have happened at all. The client has exactly one bit of information — "no answer" — and three possible realities behind it.

Retrying is the obvious response and is safe only if the operation is idempotent. Retrying a payment that actually succeeded charges the customer twice.

Idempotency means that performing an operation N times has the same effect as performing it once. Some operations are naturally idempotent: PUT /users/42 with a fixed body, DELETE /users/42, setting a value. Others are not: POST /payments, appending to a log, incrementing a counter.

For operations that are not naturally idempotent, the standard mechanism is an idempotency key: the client generates a unique identifier, sends it with the request, and the server records it. If the same key arrives again, the server returns the stored original response instead of performing the work a second time. Stripe, PayPal, and every serious payment API work this way.

Three details make the implementation correct rather than merely plausible:

• The key must be stored atomically with the effect, in the same transaction. Otherwise a crash between the two leaves a key recorded for work that never happened, or work done with no key.
• The original response must be stored, not just the key — the retry needs the same answer.
• Keys need a TTL, typically 24 hours, or the table grows without bound.

Retry strategy matters as much as retry safety. Naive fixed-interval retries by many clients produce a synchronised thundering herd against an already-struggling server. The correct approach is exponential backoff with jitter:

  delay = random(0, min(cap, base * 2^attempt))

The jitter is not optional. Without it, all clients that failed at the same moment retry at the same moment, repeatedly, converting a brief outage into a sustained one. Full jitter — a uniform random value between zero and the computed delay — is the standard recommendation.

Timeouts must be set at every layer, because the default is usually "wait forever". A missing timeout is how one slow dependency exhausts a connection pool and takes down an entire service. Timeout budgets should decrease with depth in a call chain — the gRPC deadline propagation from the previous topic is exactly this principle made explicit.

The circuit breaker completes the picture. After repeated failures it stops sending requests entirely for a cooling-off period, failing fast instead of queueing doomed work. Its three states are closed (normal), open (failing fast), and half-open (probing with a few requests to test recovery).

Examples

The fundamental ambiguity

   client sends POST /payments  -->  server
                                       |
                                    processes it
                                    CHARGES THE CARD
                                       |
                    X  <---- response LOST in the network
   client sees: TIMEOUT

   Three possibilities, indistinguishable from the client:
     (a) the request never arrived            -> safe to retry
     (b) it arrived, failed, no charge        -> safe to retry
     (c) IT SUCCEEDED, the reply was lost     -> retrying CHARGES TWICE

   The client cannot tell which. Only an idempotency key resolves it.


 IDEMPOTENCY KEY FLOW

   attempt 1:  POST /payments
               Idempotency-Key: 7f3a-91c4
               |
               v  server: key unseen -> perform the work
                          store (key, response) ATOMICALLY with the effect
               X  <-- response lost

   attempt 2:  POST /payments
               Idempotency-Key: 7f3a-91c4   <- SAME key
               |
               v  server: key EXISTS -> return the STORED response
                          no second charge
               <-- 200 OK (the original response)


 BACKOFF WITHOUT AND WITH JITTER

  NO JITTER: 1000 clients fail at t=0
    t=1s   [################] all 1000 retry together
    t=2s   [################] all 1000 retry together
    t=4s   [################] all 1000 retry together
    -> the server never recovers; the herd re-forms every round

  FULL JITTER: delay = random(0, base * 2^attempt)
    t=0-1s  [##  #   ## #  #] spread out
    t=0-2s  [ # ## #  #  # #] spread out
    -> load is smooth; the server can recover


 CIRCUIT BREAKER STATES

        failures >= threshold
   CLOSED ------------------> OPEN
     ^                          |  cooling-off period elapses
     |                          v
     +------- HALF-OPEN <-------+
       success    |  failure
                  +------> OPEN (reset the timer)

   CLOSED    : requests pass through normally
   OPEN      : fail IMMEDIATELY; do not queue doomed work
   HALF-OPEN : allow a few probes to test whether it recovered

import random, time, hashlib, uuid
from dataclasses import dataclass, field

# --- Idempotency keys ---
class IdempotentStore:
    """Key -> stored response. Must be written ATOMICALLY with the effect."""
    def __init__(self, ttl_seconds=86400):
        self.entries = {}                  # key -> (response, expires_at)
        self.ttl = ttl_seconds

    def get(self, key, now=None):
        now = now or time.time()
        entry = self.entries.get(key)
        if not entry:
            return None
        response, expires = entry
        if now >= expires:
            del self.entries[key]          # TTL: or the table grows forever
            return None
        return response

    def put(self, key, response, now=None):
        now = now or time.time()
        self.entries[key] = (response, now + self.ttl)

class PaymentService:
    def __init__(self):
        self.store = IdempotentStore()
        self.charges = []                  # the real side effect

    def charge(self, amount, idempotency_key):
        existing = self.store.get(idempotency_key)
        if existing is not None:
            return {**existing, "replayed": True}    # DO NOT charge again

        # In production this and the store.put MUST share one transaction.
        charge_id = f"ch_{len(self.charges)+1}"
        self.charges.append({"id": charge_id, "amount": amount})
        response = {"charge_id": charge_id, "amount": amount, "status": "succeeded"}
        self.store.put(idempotency_key, response)
        return {**response, "replayed": False}

svc = PaymentService()
key = str(uuid.uuid4())
for attempt in (1, 2, 3):
    r = svc.charge(5000, key)
    print(f"attempt {attempt}: {r['charge_id']} replayed={r['replayed']}")
print(f"total charges actually made: {len(svc.charges)}")
# attempt 1: ch_1 replayed=False
# attempt 2: ch_1 replayed=True
# attempt 3: ch_1 replayed=True
# total charges actually made: 1
#   Three requests, one charge. Without the key: three charges.

# --- Exponential backoff with jitter ---
def backoff_delay(attempt, base=0.1, cap=30.0, strategy="full_jitter"):
    exp = min(cap, base * (2 ** attempt))
    if strategy == "none":            return exp
    if strategy == "full_jitter":     return random.uniform(0, exp)
    if strategy == "equal_jitter":    return exp/2 + random.uniform(0, exp/2)
    if strategy == "decorrelated":    return min(cap, random.uniform(base, exp*3))
    raise ValueError(strategy)

random.seed(7)
print(f"\n{'attempt':>8}{'no jitter':>12}{'full jitter':>14}{'equal jitter':>15}")
for a in range(6):
    print(f"{a:>8}{backoff_delay(a,strategy='none'):>11.2f}s"
          f"{backoff_delay(a,strategy='full_jitter'):>13.2f}s"
          f"{backoff_delay(a,strategy='equal_jitter'):>14.2f}s")
#  attempt   no jitter   full jitter   equal jitter
#        0       0.10s         0.02s          0.06s
#        1       0.20s         0.15s          0.19s
#        2       0.40s         0.01s          0.36s
#        3       0.80s         0.44s          0.57s
#        4       1.60s         1.25s          1.31s
#        5       3.20s         0.75s          2.20s

# Why jitter matters: retry concurrency at a given instant.
def herd_concentration(clients, strategy, attempt=3, buckets=20, base=0.1):
    hist = [0] * buckets
    for _ in range(clients):
        d = backoff_delay(attempt, base=base, strategy=strategy)
        hist[min(buckets - 1, int(d / (base * 2**attempt) * buckets))] += 1
    return max(hist), hist

for strategy in ("none", "full_jitter"):
    peak, _ = herd_concentration(1000, strategy)
    print(f"\n{strategy:<12} peak simultaneous retries: {peak:>4} of 1000")
# none         peak simultaneous retries: 1000 of 1000
# full_jitter  peak simultaneous retries:   68 of 1000
#   Jitter converts a 1000-request spike into a smooth 68-per-bucket flow.

# --- The complete retry policy ---
RETRYABLE = {"UNAVAILABLE", "RESOURCE_EXHAUSTED", "ABORTED", "TIMEOUT", 503, 429, 502, 504}
NON_RETRYABLE = {"INVALID_ARGUMENT", "NOT_FOUND", "PERMISSION_DENIED", 400, 401, 403, 404, 422}

def should_retry(error, method, attempt, max_attempts=5, has_idempotency_key=False):
    if attempt >= max_attempts:
        return False, "attempt limit reached"
    if error in NON_RETRYABLE:
        return False, "permanent error: retrying cannot help"
    if error not in RETRYABLE:
        return False, "unknown error: do not retry blindly"
    if method in {"GET", "HEAD", "PUT", "DELETE", "OPTIONS"}:
        return True, "method is idempotent"
    if has_idempotency_key:
        return True, "idempotency key makes it safe"
    return False, "non-idempotent without a key: MUST NOT retry"

print()
cases = [("TIMEOUT","GET",1,False), ("TIMEOUT","POST",1,False),
         ("TIMEOUT","POST",1,True), (404,"GET",1,False), (503,"PUT",1,False),
         ("TIMEOUT","GET",5,False)]
for err, method, attempt, key_ in cases:
    ok, why = should_retry(err, method, attempt, has_idempotency_key=key_)
    print(f"  {str(err):<10}{method:<7}attempt={attempt} key={str(key_):<5} "
          f"-> {str(ok):<5} ({why})")
#   TIMEOUT   GET    attempt=1 key=False -> True  (method is idempotent)
#   TIMEOUT   POST   attempt=1 key=False -> False (non-idempotent without a key: MUST NOT retry)
#   TIMEOUT   POST   attempt=1 key=True  -> True  (idempotency key makes it safe)
#   404       GET    attempt=1 key=False -> False (permanent error: retrying cannot help)
#   503       PUT    attempt=1 key=False -> True  (method is idempotent)
#   TIMEOUT   GET    attempt=5 key=False -> False (attempt limit reached)

# --- Circuit breaker ---
@dataclass
class CircuitBreaker:
    failure_threshold: int = 5
    cooldown_s: float = 30.0
    half_open_probes: int = 3
    state: str = "closed"
    failures: int = 0
    opened_at: float = 0.0
    probes_ok: int = 0

    def allow(self, now):
        if self.state == "closed":
            return True
        if self.state == "open":
            if now - self.opened_at >= self.cooldown_s:
                self.state, self.probes_ok = "half_open", 0
                return True
            return False                      # FAIL FAST: no doomed work queued
        return True                            # half_open: allow probes

    def record(self, success, now):
        if self.state == "half_open":
            if success:
                self.probes_ok += 1
                if self.probes_ok >= self.half_open_probes:
                    self.state, self.failures = "closed", 0
            else:
                self.state, self.opened_at = "open", now
        elif success:
            self.failures = 0
        else:
            self.failures += 1
            if self.failures >= self.failure_threshold:
                self.state, self.opened_at = "open", now

cb = CircuitBreaker()
t = 0.0
timeline = [False]*6 + [True]*2 + [True]*4
print()
for i, success in enumerate(timeline):
    t += 5
    if not cb.allow(t):
        print(f"  t={t:>5.0f}s state={cb.state:<10} REQUEST BLOCKED (fail fast)")
        continue
    cb.record(success, t)
    print(f"  t={t:>5.0f}s state={cb.state:<10} call {'ok' if success else 'FAILED'}")
#   t=    5s state=closed     call FAILED
#   ...
#   t=   30s state=open       call FAILED      <- threshold reached
#   t=   35s state=open       REQUEST BLOCKED (fail fast)
#   t=   40s state=open       REQUEST BLOCKED (fail fast)
#   ...
#   t=   65s state=half_open  call ok          <- cooldown elapsed, probing

# --- Timeout budgets must shrink with depth ---
def timeout_budget(total_ms, depth):
    return [round(total_ms * (0.8 ** i)) for i in range(depth)]
print(f"\ntimeout budget for a 4-deep call chain (1000 ms total): "
      f"{timeout_budget(1000, 4)} ms")
# timeout budget for a 4-deep call chain (1000 ms total): [1000, 800, 640, 512] ms
#   Each layer leaves room for its own error handling and response.

# Send an idempotency key (the Stripe convention):
$ curl -s -X POST https://api.example.com/payments \
    -H 'Idempotency-Key: 7f3a91c4-2b8e-4d5f-9a1c-3e6d8b1a4f7c' \
    -H 'Content-Type: application/json' \
    -d '{"amount": 5000, "currency": "usd"}' | jq -c
# {"charge_id":"ch_1","amount":5000,"status":"succeeded"}

# Repeat the SAME key -- the original response comes back, no second charge:
$ curl -si -X POST https://api.example.com/payments \
    -H 'Idempotency-Key: 7f3a91c4-2b8e-4d5f-9a1c-3e6d8b1a4f7c' \
    -d '{"amount": 5000, "currency": "usd"}' | grep -iE '^HTTP|idempot'
# HTTP/2 200
# idempotent-replay: true

# curl's built-in retry, with backoff:
$ curl --retry 5 --retry-delay 1 --retry-max-time 60 \
       --retry-all-errors --max-time 10 https://api.example.com/users/42
#   --max-time bounds the WHOLE operation; --retry-max-time bounds retries.

# Always set connect AND total timeouts -- the default is often infinite:
$ curl --connect-timeout 3 --max-time 10 https://slow.example.com/
# curl: (28) Operation timed out after 10001 milliseconds

# Honour Retry-After rather than guessing:
$ curl -si https://api.example.com/users | grep -iE '^HTTP|retry-after'
# HTTP/2 429
# retry-after: 30

# Observe a circuit breaker in a service mesh (Envoy):
$ curl -s localhost:15000/stats | grep -E 'circuit_breakers.*(open|remaining)'
# cluster.users_service.circuit_breakers.default.rq_open: 1
# cluster.users_service.circuit_breakers.default.remaining_rq: 0
#   rq_open=1 means the breaker is OPEN and shedding requests.

# Check for the duplicate-charge symptom in your own data:
$ psql -c "SELECT idempotency_key, COUNT(*)
           FROM charges GROUP BY 1 HAVING COUNT(*) > 1;"
#  idempotency_key | count
# -----------------+-------
# (0 rows)
#   Any rows here mean the key and the effect were not written atomically.

# Verify idempotency keys are being expired:
$ redis-cli --scan --pattern 'idem:*' | head -3
# idem:7f3a91c4-2b8e-4d5f-9a1c-3e6d8b1a4f7c
$ redis-cli TTL idem:7f3a91c4-2b8e-4d5f-9a1c-3e6d8b1a4f7c
# (integer) 83291       <- ~23 hours remaining

Key takeaways

• A timed-out request has an unknowable outcome: it may have never arrived, failed, or succeeded with the response lost.
• Retrying is safe only for idempotent operations; for others, an idempotency key makes the retry safe by returning the stored original response.
• The key and the side effect must be written atomically in one transaction, the response must be stored (not just the key), and keys need a TTL.
• Exponential backoff requires jitter — without it, all clients retry in lockstep and convert a brief outage into a sustained one.
• Retry only on retryable errors; 400, 404, and INVALID_ARGUMENT will never succeed on a second attempt.
• Set timeouts at every layer and shrink the budget with call depth, or one slow dependency exhausts a connection pool.
• A circuit breaker (closed → open → half-open) fails fast during an outage instead of queueing doomed work.

🧪 Practice

1. Explain why PUT /users/42 is safely retryable but POST /users is not, and what makes the latter safe.
2. Compute full-jitter backoff delays for attempts 0 through 5 with a 100 ms base and a 10 s cap.
3. Describe what breaks if the idempotency key is stored in a separate transaction from the payment it guards.
4. Interview-style: A brief 30-second outage turns into a 20-minute one after the service comes back up. (Hint: think about what every failed client did at the same moment, and what they all did again exactly one second later.)

Backpressure and Rate Limiting

Theory

The final topic addresses what happens when demand exceeds capacity — which it eventually will, and the difference between a graceful and a catastrophic response is entirely a matter of design.

The naive behaviour is to queue everything. It feels accommodating and is precisely wrong. An unbounded queue converts an overload into a collapse: memory grows until the process is killed, and long before that, latency rises until every queued request has already been abandoned by its client. The server ends up spending 100% of its capacity computing answers nobody is waiting for. This failure mode has a name — congestive collapse — and it is the same phenomenon Chapter 7 described for networks.

Backpressure is the alternative: propagate the overload backwards to the producer, so it slows down rather than the consumer falling over. TCP already does this at the transport layer — a full receive window stops the sender — and the same principle must be implemented at the application layer, because TCP knows nothing about your queue depth.

The mechanisms, from most to least graceful:

• Blocking or bounded queues. A producer that cannot enqueue must wait. Natural backpressure, but it can deadlock if the producer is also the consumer of something downstream.
• Reactive streams (request(n)): the consumer explicitly requests how many items it can handle. This is demand-driven rather than supply-driven, and is the model behind RSocket and Reactive Streams.
• Load shedding: reject work immediately with 503 or 429. Counterintuitively this is often the kindest response — a fast failure lets the client retry elsewhere, whereas a slow success is indistinguishable from an outage.

Rate limiting is the deliberate, policy-driven form of the same idea, applied before overload occurs. Four algorithms, with genuinely different behaviour:

Algorithm	Bursts	Memory	Characteristic
Fixed window	Allows 2× at boundaries	Lowest	Simple, flawed
Sliding window log	Exact	High (timestamp per request)	Precise, costly
Sliding window counter	Approximate	Low	Good compromise
Token bucket	Allows controlled bursts	Low	The usual choice
Leaky bucket	None — smooths output	Low	Constant output rate

The fixed window boundary problem is worth understanding because it is a common production surprise: with a limit of 100 per minute, a client can send 100 requests at 11:59:59 and another 100 at 12:00:00 — 200 requests in one second, entirely within the rules. Token bucket avoids this by refilling continuously rather than resetting.

Token bucket is the standard choice because it expresses two independent parameters that map onto real requirements: a rate (sustained throughput) and a capacity (how large a burst is tolerated). Real traffic is bursty, and a limiter that forbids all bursts rejects legitimate work.

Two practices make rate limiting usable rather than merely enforced. First, return the right signals: 429 Too Many Requests with a Retry-After header and RateLimit-* headers telling the client its limit, remaining quota, and reset time. A client that knows its budget can self-regulate; one that does not will hammer you. Second, shed load by priority — under pressure, reject batch and analytics traffic before interactive requests, because not all work is equally valuable.

Examples

Why unbounded queues are worse than rejection

  UNBOUNDED QUEUE                       BOUNDED QUEUE + SHEDDING
  arrivals 1000/s, capacity 500/s       arrivals 1000/s, capacity 500/s

  t=10s   queue=5,000    latency 10s    queue=500 (max)  latency 1s
  t=60s   queue=30,000   latency 60s    queue=500        latency 1s
  t=300s  queue=150,000  OOM -> killed  queue=500        latency 1s
                                        500/s served, 500/s rejected FAST

  In the left column, by t=60s EVERY queued client has already timed out.
  The server is spending 100% of its capacity on work nobody wants.
  This is CONGESTIVE COLLAPSE.


 THE FIXED-WINDOW BOUNDARY PROBLEM

   limit: 100 requests per minute

   11:59:00 -------- window 1 -------- 12:00:00 ------- window 2 -------
                            [100 reqs]  [100 reqs]
                            at 11:59:59  at 12:00:00

   -> 200 requests in ONE SECOND, and both windows are "within limits"

   TOKEN BUCKET refills continuously, so this cannot happen.


 TOKEN BUCKET

   capacity = 10 tokens        refill = 5 tokens/second
   +----------------+
   | o o o o o      |  <- tokens accumulate up to CAPACITY (burst allowance)
   +----------------+
          |  each request takes 1 token
          v
     token available? -> ALLOW
     bucket empty?    -> REJECT (429) or wait

   rate     = sustained throughput
   capacity = how large a burst is tolerated
   Two independent knobs matching two real requirements.


 BACKPRESSURE PROPAGATES UPSTREAM

   [producer] --> [queue FULL] --> [consumer at capacity]
        ^                                   |
        +-------- "slow down" --------------+

   TCP does this with the receive window.
   Your application must do it with bounded queues or explicit demand.

import time, math, random
from collections import deque

# --- Token bucket: the standard choice ---
class TokenBucket:
    def __init__(self, rate, capacity):
        self.rate = rate                    # tokens per second (sustained)
        self.capacity = capacity            # max burst
        self.tokens = float(capacity)
        self.last = time.monotonic()

    def allow(self, cost=1, now=None):
        now = now if now is not None else time.monotonic()
        # Refill CONTINUOUSLY -- this is what avoids the boundary problem.
        self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
        self.last = now
        if self.tokens >= cost:
            self.tokens -= cost
            return True, 0.0
        deficit = cost - self.tokens
        return False, deficit / self.rate   # tell the client WHEN to retry

bucket = TokenBucket(rate=5, capacity=10)
t = 0.0
print(f"{'t':>6}{'tokens':>9}{'verdict':>10}{'retry after':>14}")
for i in range(16):
    t += 0.1 if i < 12 else 1.0
    ok, wait = bucket.allow(now=t)
    print(f"{t:>5.1f}s{bucket.tokens:>9.2f}{'ALLOW' if ok else 'REJECT':>10}"
          f"{('-' if ok else f'{wait:.2f}s'):>14}")
#      t   tokens   verdict   retry after
#    0.1s     9.50     ALLOW             -
#    ...                                        (burst of 10 absorbed)
#    1.0s     0.00    REJECT         0.20s      (bucket drained)
#    2.0s     4.00     ALLOW             -      (refilled at 5/s)

# --- The fixed-window flaw, demonstrated ---
class FixedWindow:
    def __init__(self, limit, window_s):
        self.limit, self.window = limit, window_s
        self.counts = {}

    def allow(self, key, now):
        w = int(now // self.window)
        self.counts.setdefault(key, {})
        self.counts[key] = {w: self.counts[key].get(w, 0)}   # drop old windows
        if self.counts[key][w] < self.limit:
            self.counts[key][w] += 1
            return True
        return False

fw = FixedWindow(limit=100, window_s=60)
allowed = 0
for _ in range(100): allowed += fw.allow("client", now=59.9)   # end of window 1
for _ in range(100): allowed += fw.allow("client", now=60.1)   # start of window 2
print(f"\nfixed window (100/min): {allowed} requests allowed within 0.2 seconds")
# fixed window (100/min): 200 requests allowed within 0.2 seconds
#   Double the intended rate, entirely "within the rules".

tb = TokenBucket(rate=100/60, capacity=100)
allowed = sum(tb.allow(now=59.9 + i*0.001)[0] for i in range(200))
print(f"token bucket (100/min): {allowed} requests allowed in the same span")
# token bucket (100/min): 100 requests allowed in the same span

# --- Sliding window log: exact but memory-hungry ---
class SlidingWindowLog:
    def __init__(self, limit, window_s):
        self.limit, self.window = limit, window_s
        self.events = deque()

    def allow(self, now):
        while self.events and self.events[0] <= now - self.window:
            self.events.popleft()                   # evict expired timestamps
        if len(self.events) < self.limit:
            self.events.append(now)
            return True
        return False

    def memory_bytes(self):
        return len(self.events) * 8                 # one timestamp per request

swl = SlidingWindowLog(limit=1000, window_s=60)
for i in range(1000): swl.allow(now=i * 0.01)
print(f"\nsliding window log: {swl.memory_bytes():,} bytes for ONE client")
print(f"  at 100,000 clients: {swl.memory_bytes()*100_000/1e6:,.0f} MB")
# sliding window log: 8,000 bytes for ONE client
#   at 100,000 clients: 800 MB
#   Exact, but it does not scale. Hence sliding-window COUNTER approximations.

# --- Bounded queue vs unbounded: the collapse, modelled ---
def simulate(arrival_rate, service_rate, seconds, queue_limit=None):
    queue, rejected, served, max_latency = 0, 0, 0, 0.0
    for _ in range(seconds):
        queue += arrival_rate
        if queue_limit is not None and queue > queue_limit:
            rejected += queue - queue_limit
            queue = queue_limit                    # SHED the excess
        done = min(queue, service_rate)
        queue -= done
        served += done
        max_latency = queue / service_rate
    return queue, rejected, served, max_latency

print(f"\n{'model':<22}{'queue':>10}{'rejected':>11}{'served':>9}{'latency':>11}")
for name, limit in (("unbounded", None), ("bounded (500)", 500)):
    q, r, s, lat = simulate(1000, 500, 300, limit)
    print(f"{name:<22}{q:>10,}{r:>11,}{s:>9,}{lat:>9.1f}s")
# model                      queue    rejected   served    latency
# unbounded                150,000           0  150,000     300.0s
# bounded (500)                500     149,500  150,000       1.0s
#   IDENTICAL useful throughput. But the unbounded version has 150,000
#   requests queued whose clients timed out minutes ago, and 300 s latency
#   for anyone still waiting. Rejecting fast is the KINDER outcome.

# --- Priority-based load shedding ---
PRIORITIES = {"interactive": 0, "api": 1, "batch": 2, "analytics": 3}

def shed(load_factor, request_class):
    """Drop the least valuable work first as pressure rises."""
    threshold = {0: 0.98, 1: 0.90, 2: 0.75, 3: 0.60}[PRIORITIES[request_class]]
    return load_factor > threshold

print(f"\n{'load':>7}", "".join(f"{c:>14}" for c in PRIORITIES))
for load in (0.5, 0.7, 0.85, 0.95, 0.99):
    row = "".join(f"{'SHED' if shed(load, c) else 'serve':>14}" for c in PRIORITIES)
    print(f"{load:>6.0%}{row}")
#    load    interactive           api         batch     analytics
#     50%          serve         serve         serve         serve
#     70%          serve         serve         serve         serve
#     85%          serve         serve          SHED          SHED
#     95%          serve          SHED          SHED          SHED
#     99%           SHED          SHED          SHED          SHED
#   Under pressure, protect what users are waiting on.

# --- The headers that let clients behave well ---
def rate_limit_headers(limit, remaining, reset_seconds, retry_after=None):
    h = {"RateLimit-Limit": str(limit),
         "RateLimit-Remaining": str(remaining),
         "RateLimit-Reset": str(reset_seconds)}
    if retry_after is not None:
        h["Retry-After"] = str(retry_after)
    return h

print(f"\nallowed  -> {rate_limit_headers(1000, 847, 43)}")
print(f"rejected -> HTTP 429 + {rate_limit_headers(1000, 0, 43, retry_after=43)}")
# allowed  -> {'RateLimit-Limit': '1000', 'RateLimit-Remaining': '847', 'RateLimit-Reset': '43'}
# rejected -> HTTP 429 + {..., 'Retry-After': '43'}
#   A client that can see its budget self-regulates. One that cannot, retries blindly.

import asyncio

# Backpressure in an async pipeline: a BOUNDED queue is the mechanism.
async def producer(queue, n):
    for i in range(n):
        # await put() BLOCKS when the queue is full -- this IS backpressure,
        # propagating the consumer's slowness back to the producer.
        await queue.put(i)
    await queue.put(None)

async def slow_consumer(queue, delay=0.01):
    processed = 0
    while (item := await queue.get()) is not None:
        await asyncio.sleep(delay)
        processed += 1
    return processed

async def demo(maxsize):
    q = asyncio.Queue(maxsize=maxsize)        # maxsize=0 means UNBOUNDED
    t0 = asyncio.get_event_loop().time()
    _, processed = await asyncio.gather(producer(q, 200), slow_consumer(q))
    return processed, q.qsize(), asyncio.get_event_loop().time() - t0

for maxsize in (10, 0):
    processed, left, elapsed = asyncio.run(demo(maxsize))
    label = f"bounded({maxsize})" if maxsize else "UNBOUNDED"
    print(f"{label:<14} processed={processed} peak memory bounded={'yes' if maxsize else 'NO'}")
# bounded(10)    processed=200 peak memory bounded=yes
# UNBOUNDED      processed=200 peak memory bounded=NO
#   Same result here because the run is short. In production, the unbounded
#   queue grows until the process is OOM-killed.

# Rate limit headers a well-behaved API returns:
$ curl -si https://api.example.com/users | grep -i '^ratelimit'
# ratelimit-limit: 1000
# ratelimit-remaining: 847
# ratelimit-reset: 43

# What a rejection looks like -- note Retry-After:
$ curl -si https://api.example.com/users | grep -iE '^HTTP|retry-after'
# HTTP/2 429
# retry-after: 43

# nginx token-bucket rate limiting (burst = bucket capacity):
$ grep -A3 'limit_req_zone' /etc/nginx/nginx.conf
# limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
# limit_req zone=api burst=20 nodelay;
#                    ^^^^^^^^ capacity 20: absorbs a burst, sustains 10/s
#                             ^^^^^^^ serve burst immediately, do not queue

# Bound the connection queue so overload sheds instead of piling up:
$ grep -E 'worker_connections|listen' /etc/nginx/nginx.conf
# worker_connections 4096;
# listen 80 backlog=511;

# Watch queue depth on a live server -- Recv-Q is unprocessed work:
$ ss -tln sport = :8080
# State  Recv-Q Send-Q Local Address:Port
# LISTEN 128    128          0.0.0.0:8080
#        ^^^ accept queue FULL: connections are being dropped

# Count shed connections:
$ nstat -az TcpExtListenOverflows
# TcpExtListenOverflows  18472  0.0
#   Rising = you are shedding. Better than queueing, but tune capacity.

# Redis-based distributed rate limiting, atomically:
$ redis-cli EVAL "
  local tokens = tonumber(redis.call('GET', KEYS[1]) or ARGV[2])
  if tokens > 0 then
    redis.call('DECR', KEYS[1])
    redis.call('EXPIRE', KEYS[1], ARGV[3])
    return 1
  end
  return 0" 1 rl:client42 1 100 60
# (integer) 1
#   The Lua script is atomic: no check-then-act race across instances.

# Envoy circuit breaker and shed statistics:
$ curl -s localhost:15000/stats | grep -E 'upstream_rq_pending_overflow|rq_retry'
# cluster.users.upstream_rq_pending_overflow: 412
#   ^ requests rejected because the pending queue was full -- shedding working

# Verify latency stays bounded while shedding:
$ curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' https://api.example.com/users
# 429 0.008s
#   Fast rejection. Compare with a 30-second queue wait ending in a timeout.

Key takeaways

• Unbounded queues turn overload into collapse: memory grows until OOM, and long before that the server serves only requests whose clients have already given up.
• Backpressure propagates slowness upstream — TCP does it with the receive window; applications must do it with bounded queues or explicit demand signalling.
• Fast rejection is kinder than slow success: a 503/429 lets a client retry elsewhere, whereas a 30-second wait is indistinguishable from an outage.
• Fixed-window rate limiting allows 2× the intended rate at window boundaries; token bucket refills continuously and avoids it.
• Token bucket is the usual choice because rate and capacity are independent knobs matching sustained throughput and tolerated burst.
• Sliding window log is exact but stores a timestamp per request, which does not scale across many clients.
• Return 429 with Retry-After and RateLimit-* headers so clients can self-regulate, and shed by priority so interactive work survives longest.

🧪 Practice

1. Explain how a 100-per-minute fixed window permits 200 requests in one second, and which algorithm prevents it.
2. A token bucket has rate 10/s and capacity 50. Describe the behaviour for a client that is idle for 10 seconds then sends 60 requests instantly.
3. Give two reasons rejecting a request immediately is better than queueing it for 30 seconds.
4. Interview-style: Under load your service's latency rises steadily until every request times out, though CPU never exceeds 60%. (Hint: if the CPU is not saturated, requests are waiting rather than working — ask what they are waiting in, and whether anything bounds its size.)

---

<a id="10-wireless-and-mobile-networking"></a>