7. The Transport Layer
The network layer delivers packets between hosts; this chapter is about delivering data between processes, and about turning IP's best-effort service into something applications can actually build on. It covers how a single host sorts arriving packets among hundreds of conversations, the two protocols that dominate the internet and the very different bargains they offer, and the mechanisms — sequence numbers, windows, timers, congestion control — that let a connection run fast without collapsing the network. Almost every performance problem that is not a physical fault lives somewhere in this chapter.
<a id="71-transport-services"></a>
7.1 Transport Services
This subchapter establishes what the transport layer is for: identifying the right process, and choosing between a reliable stream and a bare datagram.
Multiplexing and Demultiplexing
Theory
A host has one IP address, or a handful. It runs hundreds of processes, many of which are talking to the network simultaneously — a browser with forty tabs, a mail client, a backup agent, a dozen containers. IP delivers a packet to the host. Something must then decide which process gets it.
That is the transport layer's first and most fundamental job.
Multiplexing is the sending side: gathering data from many sockets, attaching enough header information to identify the source and destination, and handing it all to the network layer as a single stream of packets.
Demultiplexing is the receiving side: taking each arriving packet and delivering it to the correct socket.
The postal analogy is exact and worth carrying. IP is the postal service delivering to a building address. The transport layer is the mailroom inside the building, reading the apartment number on each envelope and putting it in the right slot. The street address gets the letter to the building; the apartment number gets it to the person. Neither is sufficient alone.
The "apartment number" is the port number — 16 bits, so 0 through 65535, in both the source and destination fields of every TCP and UDP header.
The crucial detail is that UDP and TCP demultiplex differently, and understanding why explains a great deal about both protocols.
UDP demultiplexes on a 2-tuple: the destination IP address and destination port. That is all. Two packets arriving from completely different senders, addressed to the same destination port, are delivered to the same socket. A UDP socket is a mailbox that anyone can drop mail into, which is exactly right for a DNS server or a DHCP server handling requests from arbitrary clients.
TCP demultiplexes on a 4-tuple: source IP, source port, destination IP, destination port. Two connections arriving at the same destination port from different clients go to different sockets. This is what allows a web server to hold 10,000 simultaneous connections on port 443 — each is a distinct 4-tuple, and therefore a distinct socket.
| UDP | TCP | |
|---|---|---|
| Demultiplexing key | 2-tuple (dst IP, dst port) | 4-tuple (src IP, src port, dst IP, dst port) |
| Sockets per port | One, shared by all senders | One per connection |
| Consequence | A server port is a single mailbox | A server port supports thousands of connections |
This also explains the listening socket versus connected socket distinction that confuses people learning the socket API. A TCP server creates one listening socket bound to a port. When a connection arrives, accept() returns a new socket for that specific 4-tuple, while the listening socket stays available for the next connection. The listening socket is not a connection; it is a factory for connections.
Examples
One host, one IP, many conversations
packets arrive for 192.168.1.20
|
v
+---------------------------+
| TRANSPORT LAYER |
| (demultiplexer) |
+---------------------------+
/ | | \
port 443 port 443 port 22 port 53
(tab A) (tab B) (ssh) (dns)
| | | |
different different distinct distinct
4-tuples ----+ socket socket
IP got the packets to the BUILDING.
The transport layer gets them to the APARTMENT.
UDP: 2-tuple — everyone shares one mailbox
from 10.1.1.5:40001 ---+
from 10.2.2.9:51234 ---+---> dst 192.168.1.20:53 ---> ONE socket
from 10.3.3.7:33445 ---+ (the DNS server
reads sender
addresses itself)
TCP: 4-tuple — every conversation gets its own socket
10.1.1.5:40001 -> 192.168.1.20:443 ---> socket A
10.2.2.9:51234 -> 192.168.1.20:443 ---> socket B
10.1.1.5:40002 -> 192.168.1.20:443 ---> socket C
^^^^^ same client, different source port = different socket
Same destination port 443 in all three. Three distinct sockets.
This is how one port serves 10,000 simultaneous connections.
class Demultiplexer:
"""The core of the transport layer: route a packet to the right socket."""
def __init__(self):
self.udp_sockets = {} # (dst_ip, dst_port) -> socket id
self.tcp_sockets = {} # 4-tuple -> socket id
self.tcp_listeners = {} # (dst_ip, dst_port) -> listening socket id
def bind_udp(self, ip, port, sock):
self.udp_sockets[(ip, port)] = sock
def listen_tcp(self, ip, port, sock):
self.tcp_listeners[(ip, port)] = sock
def accept_tcp(self, four_tuple, sock):
self.tcp_sockets[four_tuple] = sock
def deliver(self, proto, src_ip, src_port, dst_ip, dst_port):
if proto == "udp":
# 2-TUPLE: the sender's identity is irrelevant to demultiplexing.
sock = self.udp_sockets.get((dst_ip, dst_port))
return sock or "ICMP port unreachable"
key = (src_ip, src_port, dst_ip, dst_port)
if key in self.tcp_sockets:
return self.tcp_sockets[key] # an established connection
listener = self.tcp_listeners.get((dst_ip, dst_port))
if listener:
return f"{listener} (LISTENING — will accept() a new socket)"
return "TCP RST"
d = Demultiplexer()
d.bind_udp("192.168.1.20", 53, "dns-socket")
d.listen_tcp("192.168.1.20", 443, "https-listener")
d.accept_tcp(("10.1.1.5", 40001, "192.168.1.20", 443), "conn-A")
d.accept_tcp(("10.2.2.9", 51234, "192.168.1.20", 443), "conn-B")
d.accept_tcp(("10.1.1.5", 40002, "192.168.1.20", 443), "conn-C")
print("UDP — different senders, SAME socket:")
for src in [("10.1.1.5", 40001), ("10.2.2.9", 51234), ("10.3.3.7", 33445)]:
print(f" {src[0]}:{src[1]:<6} -> :53 {d.deliver('udp', *src, '192.168.1.20', 53)}")
# UDP — different senders, SAME socket:
# 10.1.1.5:40001 -> :53 dns-socket
# 10.2.2.9:51234 -> :53 dns-socket
# 10.3.3.7:33445 -> :53 dns-socket
print("\nTCP — different senders, DIFFERENT sockets, all on port 443:")
for src in [("10.1.1.5", 40001), ("10.2.2.9", 51234), ("10.1.1.5", 40002),
("10.9.9.9", 60000)]:
print(f" {src[0]}:{src[1]:<6} -> :443 {d.deliver('tcp', *src, '192.168.1.20', 443)}")
# TCP — different senders, DIFFERENT sockets, all on port 443:
# 10.1.1.5:40001 -> :443 conn-A
# 10.2.2.9:51234 -> :443 conn-B
# 10.1.1.5:40002 -> :443 conn-C
# 10.9.9.9:60000 -> :443 https-listener (LISTENING — will accept() a new socket)
# How many connections can one server port theoretically hold?
def max_connections(clients, ports_per_client=64511):
"""Ephemeral ports 1024-65535 per client IP, all to one server port."""
return clients * ports_per_client
print()
for clients in (1, 100, 10_000):
print(f"{clients:>7,} client IPs -> {max_connections(clients):>16,} possible "
f"4-tuples on ONE server port")
# 1 client IPs -> 64,511 possible 4-tuples on ONE server port
# 100 client IPs -> 6,451,100 possible 4-tuples on ONE server port
# 10,000 client IPs -> 645,110,000 possible 4-tuples on ONE server port
# The server port is never the limit. Memory and file descriptors are.
# Watch demultiplexing state directly. Note the two kinds of TCP socket:
$ ss -tan | head -6
# State Recv-Q Send-Q Local Address:Port Peer Address:Port
# LISTEN 0 511 0.0.0.0:443 0.0.0.0:*
# ^^^^^^^^^ no peer:
# a LISTENING socket
# ESTAB 0 0 192.168.1.20:443 10.1.1.5:40001
# ESTAB 0 0 192.168.1.20:443 10.2.2.9:51234
# ESTAB 0 0 192.168.1.20:443 10.1.1.5:40002
# ^ three ESTABLISHED sockets, all local port 443,
# distinguished ONLY by the peer address and port
# Count connections sharing one port — the 4-tuple in action:
$ ss -tan state established '( dport = :443 or sport = :443 )' | wc -l
# 8412
# UDP sockets have no peer at all — one mailbox per (ip, port):
$ ss -uan | head -4
# State Recv-Q Send-Q Local Address:Port Peer Address:Port
# UNCONN 0 0 0.0.0.0:53 0.0.0.0:*
# UNCONN 0 0 127.0.0.53:53 0.0.0.0:*
# ^^^^^^ UNCONN: no connection state exists; senders are irrelevant
# to demultiplexing
# What happens when demultiplexing FAILS:
$ nc -u -w1 127.0.0.1 9999 <<< test
$ sudo tcpdump -i lo -n icmp -c 1
# IP 127.0.0.1 > 127.0.0.1: ICMP 127.0.0.1 udp port 9999 unreachable
# ^^^^^^^^^^^^^^^^^^^^^^^^^ no socket
# bound -> ICMP type 3 code 3
$ nc -w1 127.0.0.1 9999
# (connection refused) <- TCP's equivalent is an RST, not ICMP
Key takeaways
- The transport layer's first job is multiplexing (gathering from many sockets) and demultiplexing (delivering each packet to the right socket).
- IP delivers to a host; the transport layer delivers to a process — street address versus apartment number.
- Port numbers are 16 bits (0–65535), carried as source and destination in every TCP and UDP header.
- UDP demultiplexes on a 2-tuple (dst IP, dst port): all senders share one socket, which suits servers handling arbitrary clients.
- TCP demultiplexes on a 4-tuple (src IP, src port, dst IP, dst port): every connection gets its own socket, which is why one port serves thousands of connections.
- A TCP listening socket is not a connection — it is a factory;
accept()produces a new socket per 4-tuple. - Failed demultiplexing produces ICMP port unreachable for UDP and a TCP RST for TCP.
🧪 Practice
- State the demultiplexing key for UDP and for TCP, and explain which one allows one port to serve many simultaneous connections.
- Three clients connect to the same web server on port 443. Write out the three 4-tuples and explain why they do not collide.
- Explain the difference between a listening socket and a connected socket, and what
accept()actually returns.- Interview-style: A DNS server and a web server both handle thousands of clients on a single port. Explain why only one of them needs a socket per client. (Hint: ask what state each protocol must keep between one packet and the next.)
Port Numbers and Well-Known Ports
Theory
A port number is meaningless on its own — it is just a 16-bit integer. Its value comes entirely from convention: a shared agreement that a particular number identifies a particular service, so a client can connect to a server it has never spoken to before without negotiation.
IANA divides the 65,536 values into three ranges:
| Range | Name | Use |
|---|---|---|
| 0 – 1023 | Well-known / System | Standard services; require privilege to bind on Unix |
| 1024 – 49151 | Registered / User | Applications registered with IANA |
| 49152 – 65535 | Dynamic / Ephemeral / Private | Temporary client-side ports |
The ports worth knowing on sight:
| Port | Service | Transport |
|---|---|---|
| 20, 21 | FTP data, control | TCP |
| 22 | SSH | TCP |
| 25 | SMTP | TCP |
| 53 | DNS | UDP and TCP |
| 67, 68 | DHCP server, client | UDP |
| 80 | HTTP | TCP |
| 123 | NTP | UDP |
| 161 | SNMP | UDP |
| 179 | BGP | TCP |
| 443 | HTTPS / HTTP/3 | TCP and UDP (QUIC) |
| 514 | Syslog | UDP |
| 3306 / 5432 | MySQL / PostgreSQL | TCP |
| 6379 | Redis | TCP |
Note the entries listing both transports. DNS uses UDP for ordinary queries and TCP for responses too large for a datagram and for zone transfers. Port 443 now carries both TCP (HTTP/1.1 and HTTP/2 over TLS) and UDP (HTTP/3 over QUIC) — the number is the same, the protocol is not, and a firewall rule permitting "443" without specifying the transport may permit less or more than intended.
Ephemeral ports are the other half of the story, and they are what makes the client side work. When a client opens a connection it does not care which local port it uses, so the OS assigns an unused one from the ephemeral range. That port becomes part of the 4-tuple and is how replies find their way back to the right socket. Linux's default range is 32768–60999, configurable and often widened on busy servers.
Three practical consequences deserve attention.
Binding below 1024 requires privilege on Unix, which is why web servers historically started as root and dropped privileges after binding port 80. Modern alternatives are CAP_NET_BIND_SERVICE, a reverse proxy, or simply lowering the privileged threshold.
Ephemeral port exhaustion is a real failure on busy systems. A machine making many short-lived outbound connections to the same destination can run out of 4-tuples, because only the source port varies. Combined with TIME_WAIT (covered in 7.3) holding ports for 60 seconds after close, a server making thousands of connections per second to one backend will exhaust the range and fail with "cannot assign requested address".
A port number is a convention, not an enforcement. Nothing stops SSH running on port 443 or a web server on port 22. "Port-based security" — assuming port 443 means HTTPS — is therefore unreliable, which is why modern firewalls do application-layer inspection rather than trusting the number.
Examples
The port space, and who uses which part
0 1024 49152 65535
|-----------------|--------------------|------------------|
WELL-KNOWN REGISTERED DYNAMIC/EPHEMERAL
needs privilege apps register OS assigns to clients
22, 53, 80, 443 3306, 6379, 8080 (Linux: 32768-60999)
A connection uses one of each
client 10.1.1.5 server 93.184.216.34
+------------------+ +------------------+
| ephemeral: 51234 |-------------------->| well-known: 443 |
| (OS picks it) | | (convention) |
+------------------+ +------------------+
4-tuple: (10.1.1.5, 51234, 93.184.216.34, 443)
^^^^^ varies per connection; this is what makes
each 4-tuple unique from one client
Ephemeral exhaustion: when only ONE field can vary
client 10.1.1.5 ----> backend 10.9.9.9:8080
src IP : fixed (one client)
dst IP : fixed (one backend)
dst port : fixed (8080)
src port : THE ONLY VARIABLE -> ~28,000 usable ephemeral ports
At 1,000 new connections/sec with 60 s TIME_WAIT:
60,000 sockets held > 28,232 available
-> "cannot assign requested address"
The fix is not more bandwidth. It is connection reuse.
IANA_RANGES = [
(0, 1023, "well-known / system", "privileged bind required on Unix"),
(1024, 49151, "registered / user", "IANA-registered applications"),
(49152, 65535, "dynamic / ephemeral", "temporary, client-assigned"),
]
WELL_KNOWN = {
20: ("FTP-data", "tcp"), 21: ("FTP-control", "tcp"), 22: ("SSH", "tcp"),
25: ("SMTP", "tcp"), 53: ("DNS", "udp+tcp"), 67: ("DHCP-server", "udp"),
68: ("DHCP-client", "udp"), 80: ("HTTP", "tcp"), 123: ("NTP", "udp"),
161: ("SNMP", "udp"), 179: ("BGP", "tcp"), 443: ("HTTPS / HTTP-3", "tcp+udp"),
514: ("Syslog", "udp"), 3306: ("MySQL", "tcp"), 5432: ("PostgreSQL", "tcp"),
6379: ("Redis", "tcp"),
}
def classify(port):
for lo, hi, name, note in IANA_RANGES:
if lo <= port <= hi:
svc = WELL_KNOWN.get(port)
extra = f" [{svc[0]}, {svc[1]}]" if svc else ""
return f"{name:<22}{note}{extra}"
return "invalid"
for p in (22, 53, 443, 8080, 51234, 65535):
print(f"port {p:<6}{classify(p)}")
# port 22 well-known / system privileged bind required on Unix [SSH, tcp]
# port 53 well-known / system privileged bind required on Unix [DNS, udp+tcp]
# port 443 well-known / system privileged bind required on Unix [HTTPS / HTTP-3, tcp+udp]
# port 8080 registered / user IANA-registered applications
# port 51234 dynamic / ephemeral temporary, client-assigned
# port 65535 dynamic / ephemeral temporary, client-assigned
# Ephemeral port exhaustion, quantified.
def exhaustion(conns_per_sec, timewait_s=60, port_lo=32768, port_hi=60999):
available = port_hi - port_lo + 1
held = conns_per_sec * timewait_s
return {
"available_ports": available,
"sockets_held": held,
"exhausted": held > available,
"max_sustainable_cps": available // timewait_s,
}
print()
for cps in (100, 400, 1000):
e = exhaustion(cps)
status = "EXHAUSTED" if e["exhausted"] else "ok"
print(f"{cps:>5,} conn/s to ONE destination: {e['sockets_held']:>7,} sockets held "
f"vs {e['available_ports']:,} ports -> {status}")
print(f"\nmax sustainable rate to a single destination: "
f"{exhaustion(1)['max_sustainable_cps']:,} conn/s")
# 100 conn/s to ONE destination: 6,000 sockets held vs 28,232 ports -> ok
# 400 conn/s to ONE destination: 24,000 sockets held vs 28,232 ports -> ok
# 1,000 conn/s to ONE destination: 60,000 sockets held vs 28,232 ports -> EXHAUSTED
#
# max sustainable rate to a single destination: 470 conn/s
# The fix is reuse, not more ports.
def with_keepalive(requests_per_sec, requests_per_connection):
return requests_per_sec / requests_per_connection
print()
for reuse in (1, 10, 100):
cps = with_keepalive(1000, reuse)
e = exhaustion(int(cps))
print(f"{reuse:>4} requests per connection -> {cps:>7.1f} new conn/s "
f"({'EXHAUSTED' if e['exhausted'] else 'sustainable'})")
# 1 requests per connection -> 1000.0 new conn/s (EXHAUSTED)
# 10 requests per connection -> 100.0 new conn/s (sustainable)
# 100 requests per connection -> 10.0 new conn/s (sustainable)
# A port number is a convention, not an enforcement.
def what_is_really_there(port, actual_protocol):
expected = WELL_KNOWN.get(port, ("unknown", "?"))[0]
if expected.lower().startswith(actual_protocol.lower()):
return f"port {port}: {actual_protocol} (as expected)"
return (f"port {port}: {actual_protocol} — expected {expected}. "
f"Perfectly legal; port-based assumptions FAIL here.")
print()
print(what_is_really_there(443, "HTTPS"))
print(what_is_really_there(443, "SSH"))
# port 443: HTTPS (as expected)
# port 443: SSH — expected HTTPS / HTTP-3. Perfectly legal; port-based assumptions FAIL here.
# The system's service-name registry:
$ grep -E "^(ssh|http|https|domain|bgp)\s" /etc/services
# ssh 22/tcp
# domain 53/tcp
# domain 53/udp <- DNS really does use both
# http 80/tcp
# https 443/tcp
# https 443/udp <- HTTP/3 over QUIC
# bgp 179/tcp
# The ephemeral range on this host:
$ sysctl net.ipv4.ip_local_port_range
# net.ipv4.ip_local_port_range = 32768 60999
$ python3 -c "print(f'{60999-32768+1:,} ephemeral ports available')"
# 28,232 ephemeral ports available
# Widen it on a host making many outbound connections:
$ sudo sysctl -w net.ipv4.ip_local_port_range="10240 65535"
# net.ipv4.ip_local_port_range = 10240 65535 # ~55,000 ports
# Diagnose exhaustion — the error is distinctive:
$ ss -tan state time-wait | wc -l
# 24817
$ dmesg | grep -i "port"
# TCP: request_sock_TCP: Possible SYN flooding on port 443
$ curl http://backend:8080/
# curl: (7) Failed to connect: Cannot assign requested address
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ EADDRNOTAVAIL:
# no ephemeral port is free. Not a network fault.
# Binding a privileged port without running as root:
$ sudo setcap 'cap_net_bind_service=+ep' /usr/local/bin/myserver
$ sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80
# What is ACTUALLY on a port — never trust the number:
$ sudo ss -tlnp | grep :443
# LISTEN 0 511 0.0.0.0:443 0.0.0.0:* users:(("nginx",pid=1204,fd=6))
$ nmap -sV -p 443 192.168.1.20
# PORT STATE SERVICE VERSION
# 443/tcp open ssh OpenSSH 9.2 <- SSH on 443. Entirely legal.
# ^^^ -sV probes the service instead of assuming from the port
Key takeaways
- A port number is 16 bits and derives all its meaning from convention, not enforcement.
- IANA ranges: 0–1023 well-known (privileged bind), 1024–49151 registered, 49152–65535 dynamic/ephemeral.
- Memorise the common ones: 22 SSH, 25 SMTP, 53 DNS, 80 HTTP, 179 BGP, 443 HTTPS/HTTP-3.
- DNS uses both UDP and TCP; port 443 carries both TCP and UDP (QUIC) — a firewall rule must name the transport.
- Ephemeral ports are OS-assigned on the client side and make each 4-tuple unique; Linux defaults to 32768–60999.
- Ephemeral exhaustion occurs when many short-lived connections go to the same destination, since only the source port varies — the fix is connection reuse, not more bandwidth.
- Binding below 1024 needs privilege;
CAP_NET_BIND_SERVICEavoids running as root. - Port-based security is unreliable because any service can run on any port; use application-layer inspection.
🧪 Practice
- Classify ports 22, 8080, and 54321 into their IANA ranges, and state which requires privilege to bind.
- A host makes 600 connections per second to a single backend with 60-second TIME_WAIT. Determine whether it exhausts the default Linux ephemeral range, and give two fixes.
- Explain why DNS needs both UDP and TCP on port 53.
- Interview-style: A monitoring tool reports "port 443 open" and concludes the service is HTTPS. Why is that conclusion unsound? (Hint: ask what enforces the relationship between a port number and the protocol spoken on it.)
Sockets and Connection Identification
Theory
A socket is the programming abstraction for a network endpoint — the handle an application holds to send and receive data. On Unix it is a file descriptor, which is why read() and write() work on network connections just as they do on files. That uniformity is deliberate and is one of Unix's more durable design decisions.
The important thing to understand is what a socket is, concretely: a kernel data structure holding the connection's identity (the addresses and ports), its state, its send and receive buffers, and — for TCP — all the sequence numbers, timers, and window values covered later in this chapter. When people say a connection "costs memory", this structure plus its buffers is the cost.
The socket API divides into two very different flows, and the asymmetry is the point: a server waits, a client initiates.
| Server | Client |
|---|---|
socket() — create | socket() — create |
bind() — claim an address and port | (skipped: the OS assigns an ephemeral port) |
listen() — mark it passive, set the backlog | |
accept() — block until a connection arrives, return a NEW socket | connect() — initiate the handshake |
read() / write() on the new socket | read() / write() |
close() | close() |
The step that repays careful attention is accept(). It does not return data and it does not return the listening socket — it returns a brand-new socket representing one specific connection, identified by its 4-tuple. The listening socket remains untouched, ready for the next arrival. A server with 10,000 connections holds 10,001 sockets: one listener and ten thousand connections.
Two kernel queues sit behind listen(), and confusing them makes SYN flood behaviour hard to reason about:
- The SYN queue (incomplete) holds connections that have received a SYN and sent a SYN-ACK but not yet received the final ACK. This is what a SYN flood fills.
- The accept queue (completed) holds fully established connections waiting for the application to call
accept(). This is what overflows when an application is too slow, and the symptom is very different — connections established but never served.
The backlog argument to listen() sizes the accept queue on modern Linux.
Socket types map onto the transport protocols:
| Type | Protocol | Semantics |
|---|---|---|
SOCK_STREAM | TCP | Reliable, ordered byte stream |
SOCK_DGRAM | UDP | Unreliable, unordered messages |
SOCK_RAW | IP directly | Build your own headers; needs privilege |
One consequence of SOCK_STREAM being a byte stream causes more application bugs than almost anything else in network programming: TCP does not preserve message boundaries. If the sender makes three write() calls of 100 bytes each, the receiver may get one read() of 300 bytes, or 60 then 240, or any other split. The bytes arrive in order and complete, but the framing is gone. Applications must impose their own framing — a length prefix, a delimiter, or a fixed record size. UDP, by contrast, preserves message boundaries exactly: one sendto() produces one recvfrom().
Examples
The two flows, and where accept() produces a new socket
SERVER CLIENT
socket() -> fd 3 (listening) socket() -> fd 3
bind(3, 0.0.0.0:443)
listen(3, backlog=511)
accept(3) ............blocked........... connect(3, server:443)
<-------- SYN ---------------
--------- SYN-ACK ----------->
<-------- ACK ---------------
accept(3) -> fd 4 <-- A NEW SOCKET
^ fd 3 is STILL LISTENING, ready for the next client
read(4)/write(4) read(3)/write(3)
close(4) close(3)
^ closes ONE connection, not the listener
The two queues behind listen()
SYN arrives
|
v
+----------------+ 3-way handshake +-------------------+
| SYN QUEUE | ------completes---> | ACCEPT QUEUE |
| (incomplete) | | (established) |
+----------------+ +-------------------+
^ a SYN FLOOD fills this ^ a SLOW APPLICATION
-> new connections refused fills this
-> defence: SYN cookies -> connections sit unserved
-> symptom: high latency,
then drops
TCP is a BYTE STREAM: framing is the application's problem
sender: write("HELLO") write("WORLD") write("!")
|
v (TCP may coalesce or split arbitrarily)
receiver: read() -> "HELLOWOR"
read() -> "LD!"
The bytes are complete and in order. The MESSAGE BOUNDARIES ARE GONE.
UDP: sendto("HELLO") sendto("WORLD")
recvfrom() -> "HELLO" <- boundaries preserved exactly
recvfrom() -> "WORLD"
import socket
# --- SERVER: bind, listen, accept-per-connection ---
def server_flow():
srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# SO_REUSEADDR lets us rebind while old sockets linger in TIME_WAIT (7.3).
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind(("0.0.0.0", 8443))
srv.listen(511) # sizes the ACCEPT queue
print(f"listening socket fd={srv.fileno()}")
conn, peer = srv.accept() # returns a NEW socket
print(f"accepted from {peer}: new fd={conn.fileno()}, "
f"listener fd={srv.fileno()} still open")
conn.close() # closes ONE connection
srv.close()
# listening socket fd=3
# accepted from ('10.1.1.5', 51234): new fd=4, listener fd=3 still open
# --- The byte-stream problem, and the standard fix ---
import struct
def send_unframed(chunks):
"""What the application intended to send."""
return b"".join(chunks)
def simulate_tcp_delivery(stream, splits):
"""TCP may deliver the same bytes in ANY grouping."""
out, pos = [], 0
for n in splits:
out.append(stream[pos:pos + n])
pos += n
if pos < len(stream):
out.append(stream[pos:])
return out
msgs = [b"HELLO", b"WORLD", b"!"]
stream = send_unframed(msgs)
print(f"\napplication sent {len(msgs)} messages: {msgs}")
for splits in ([8, 3], [11], [1, 1, 1, 8]):
print(f" receiver read() sequence: {simulate_tcp_delivery(stream, splits)}")
# application sent 3 messages: [b'HELLO', b'WORLD', b'!']
# receiver read() sequence: [b'HELLOWOR', b'LD!']
# receiver read() sequence: [b'HELLOWORLD!']
# receiver read() sequence: [b'H', b'E', b'L', b'LOWORLD!']
# Same bytes, same order, THREE different framings. All are valid TCP.
def frame(msg):
"""Length-prefix framing: 4-byte big-endian length, then the payload."""
return struct.pack("!I", len(msg)) + msg
class Deframer:
"""Reassemble messages from an arbitrarily-chunked byte stream."""
def __init__(self):
self.buf = b""
def feed(self, data):
self.buf += data
out = []
while len(self.buf) >= 4:
(length,) = struct.unpack("!I", self.buf[:4])
if len(self.buf) < 4 + length:
break # incomplete message; wait for more
out.append(self.buf[4:4 + length])
self.buf = self.buf[4 + length:]
return out
framed = b"".join(frame(m) for m in msgs)
d = Deframer()
print(f"\nwith length-prefix framing, delivered in awkward chunks:")
for chunk in simulate_tcp_delivery(framed, [3, 7, 5, 4]):
got = d.feed(chunk)
print(f" fed {len(chunk):>2} bytes -> recovered {got}")
# with length-prefix framing, delivered in awkward chunks:
# fed 3 bytes -> recovered []
# fed 7 bytes -> recovered [b'HELLO']
# fed 5 bytes -> recovered []
# fed 4 bytes -> recovered [b'WORLD']
# fed 4 bytes -> recovered [b'!']
# Framing is recovered regardless of how TCP chunked the stream.
# What a socket actually costs.
def socket_memory(connections, send_buf=16384, recv_buf=87380, struct_overhead=2048):
per = send_buf + recv_buf + struct_overhead
return per, per * connections
print()
for n in (1_000, 10_000, 100_000):
per, total = socket_memory(n)
print(f"{n:>8,} connections x {per/1024:.0f} KB = {total/1024/1024:>8,.0f} MB")
# 1,000 connections x 103 KB = 101 MB
# 10,000 connections x 103 KB = 1,009 MB
# 100,000 connections x 103 KB = 10,091 MB
# This is why buffer tuning matters at scale — and why C10K was hard.
# Socket states, and the two queues:
$ ss -tln
# State Recv-Q Send-Q Local Address:Port
# LISTEN 0 511 0.0.0.0:443
# ^^^^^^ ^^^^^^ for LISTEN sockets these mean something different:
# Recv-Q = accept queue CURRENT depth
# Send-Q = accept queue MAXIMUM (the backlog)
# An overflowing accept queue — the application is too slow:
$ ss -tln | grep :443
# LISTEN 511 511 0.0.0.0:443
# ^^^ FULL. New connections are being dropped.
$ nstat -az | grep -i listen
# TcpExtListenOverflows 8412
# TcpExtListenDrops 8412
# ^^^^ the definitive counter for this problem
# Raise the backlog (both limits must be raised):
$ sudo sysctl -w net.core.somaxconn=4096
# ...and pass a matching backlog to listen() in the application.
# SYN queue pressure is a DIFFERENT problem with a different defence:
$ sudo sysctl -w net.ipv4.tcp_max_syn_backlog=8192
$ sudo sysctl -w net.ipv4.tcp_syncookies=1 # survive a SYN flood
# Sockets really are file descriptors:
$ ls -l /proc/$(pgrep -f nginx | head -1)/fd | head -4
# lrwx------ 1 root root 64 Sep 23 10:02 3 -> socket:[184221]
# lrwx------ 1 root root 64 Sep 23 10:02 4 -> socket:[184229]
$ ss -tanp | grep 184221
# File descriptor limits cap concurrency long before the port space does:
$ ulimit -n
# 1024 <- only 1024 connections, whatever else you tune
$ ulimit -n 65535
# Buffer sizes — the memory cost per connection:
$ sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem
# net.ipv4.tcp_rmem = 4096 131072 6291456
# net.ipv4.tcp_wmem = 4096 16384 4194304
# ^min ^default ^max (the kernel autotunes between them)
Key takeaways
- A socket is a kernel structure holding a connection's identity, state, and buffers, exposed to the application as a file descriptor.
- The server flow is
socket → bind → listen → accept; the client flow issocket → connect, with the OS assigning an ephemeral port. accept()returns a NEW socket for one connection; the listening socket stays open — a server with 10,000 connections holds 10,001 sockets.- Two kernel queues: the SYN queue (incomplete handshakes — filled by SYN floods) and the accept queue (established but unaccepted — filled by a slow application).
- Socket types:
SOCK_STREAM(TCP, byte stream),SOCK_DGRAM(UDP, messages),SOCK_RAW. - TCP does not preserve message boundaries — the application must impose framing via a length prefix, delimiter, or fixed records; UDP preserves them exactly.
- Each connection costs send and receive buffers plus overhead (~100 KB by default), which is the real limit on concurrency.
- File descriptor limits (
ulimit -n) cap connection counts long before the port space does.
🧪 Practice
- List the server-side socket calls in order and state precisely what
accept()returns.- A sender makes
write()calls of 50, 50, and 50 bytes. Give three validread()sequences a TCP receiver might observe, and name the mechanism that fixes the ambiguity.- Distinguish the SYN queue from the accept queue, naming the failure that fills each and the counter that reveals it.
- Interview-style: A server accepts connections fine at low load but clients time out during traffic spikes, while CPU and bandwidth are both low. Where would you look? (Hint: something is queueing between the completed handshake and the application — ask which counter records that overflow.)
Reliable vs Unreliable Transport
Theory
IP promises nothing (Chapter 5). Packets may be lost, duplicated, reordered, or corrupted. The transport layer's second major job — after demultiplexing — is deciding how much of that to fix, and the answer is a genuine engineering trade, not an obvious win for reliability.
Reliable transport guarantees that data arrives complete, in order, and uncorrupted, or that the connection fails outright. TCP achieves this with four mechanisms working together:
Sequence numbers let the receiver detect gaps, discard duplicates, and reorder. Acknowledgements tell the sender what arrived. Timers detect loss when no acknowledgement comes. Retransmission resends what was lost.
Checksums cover corruption, and both TCP and UDP have them.
Unreliable transport (UDP) does none of this. A datagram is sent; if it vanishes, nothing notices and nothing retries. UDP adds exactly two things to IP: port numbers (demultiplexing) and an optional checksum (corruption detection). It is, deliberately, the thinnest possible layer.
The instinct is that reliability is strictly better. It is not, and the reason is latency.
Reliability costs time. A lost packet must be detected — usually by waiting for a timeout, or by waiting for duplicate ACKs — and then retransmitted, which takes at least one more round trip. For a file transfer that is obviously worth it. For a live voice call it is actively harmful: an audio packet that arrives 300 ms late is useless, because the moment it described has passed. Retransmitting it consumes bandwidth to deliver data that will be discarded, and the retransmission may delay the packets behind it.
This is the crux: for real-time data, late is the same as lost — except that late also costs bandwidth and adds delay.
There is a second, subtler cost: head-of-line blocking. TCP delivers bytes in order, so one lost segment stalls everything behind it, even data that has already arrived intact. An application multiplexing independent messages over one TCP connection suffers delays on messages that were never lost. This is the problem QUIC was designed to solve (7.6).
| Reliable (TCP) | Unreliable (UDP) | |
|---|---|---|
| Loss | Retransmitted | Ignored |
| Ordering | Guaranteed | Not guaranteed |
| Duplicates | Removed | Possible |
| Corruption | Detected and resent | Detected and dropped |
| Latency | Variable; loss causes spikes | Consistent |
| Head-of-line blocking | Yes | No |
| Overhead | 20+ byte header, state, ACKs | 8-byte header, no state |
| Suits | Files, web, email, databases | Voice, video, games, DNS, telemetry |
The right question is never "do I want reliability?" but "what does my application do with a late packet?" If it needs the data whatever the delay, use TCP. If a late packet is worthless, use UDP and handle loss in the application — typically by concealing it (audio interpolation), by tolerating it (send the next position update instead of the last), or by adding forward error correction so loss is repaired without a round trip.
A common middle path is UDP with selective, application-level reliability: retransmit only what genuinely matters (a game's inventory change) while letting ephemeral data (position updates) drop. This is covered in 7.2.
Examples
Why retransmission harms a voice call
TCP behaviour on loss:
packet 5 lost
... wait for 3 dup ACKs or a timeout (tens to hundreds of ms) ...
retransmit 5
packets 6,7,8 ALREADY ARRIVED but are held in the buffer
-> delivered only after 5 arrives <-- HEAD-OF-LINE BLOCKING
result: a 200 ms gap in audio, then a burst. Worse than silence.
UDP behaviour on loss:
packet 5 lost
packet 6 plays immediately
the codec conceals the 20 ms gap (interpolation)
result: a barely perceptible artifact. Conversation continues.
Head-of-line blocking, drawn
sent: [1][2][3][4][5][6][7][8]
arrived: [1][2][3][4] [6][7][8] (5 lost)
^^^^^^^^^ complete, intact, AND UNDELIVERABLE
TCP receive buffer:
delivered to app: 1,2,3,4
held hostage: 6,7,8 <- waiting only for 5
app sees: a stall, then a burst of four
The decision, in one question
"What does my application do with a packet that arrives late?"
|
+-------------+-------------+
| |
"Use it anyway" "Discard it"
| |
use TCP use UDP
(files, web, email, (voice, video, games,
databases, SSH) DNS, metrics)
import random
def unreliable_channel(packets, loss=0.05, seed=1):
random.seed(seed)
return [p for p in packets if random.random() >= loss]
def udp_receiver(received, total):
"""Play what arrives, conceal what does not. No waiting."""
timeline = []
got = set(received)
for seq in range(1, total + 1):
if seq in got:
timeline.append((seq, "played", 0))
else:
timeline.append((seq, "CONCEALED (codec interpolation)", 0))
return timeline
def tcp_receiver(received, total, rtt_ms=50):
"""Deliver strictly in order. A gap stalls everything behind it."""
timeline, buffer, delay = [], set(received), 0
for seq in range(1, total + 1):
if seq in buffer:
timeline.append((seq, "delivered", delay))
else:
delay += rtt_ms # detect + retransmit = >= 1 RTT
timeline.append((seq, f"RETRANSMITTED (+{rtt_ms} ms)", delay))
return timeline
total = 10
arrived = unreliable_channel(range(1, total + 1), loss=0.2)
lost = sorted(set(range(1, total + 1)) - set(arrived))
print(f"packets lost in transit: {lost}\n")
print("UDP (voice codec):")
for seq, what, d in udp_receiver(arrived, total)[:6]:
print(f" pkt {seq}: {what:<36} cumulative delay {d} ms")
print("\nTCP (in-order delivery):")
for seq, what, d in tcp_receiver(arrived, total)[:6]:
print(f" pkt {seq}: {what:<36} cumulative delay {d} ms")
# Head-of-line blocking, quantified.
def hol_delay(lost_seq, total, rtt_ms=50):
"""How many already-arrived packets are held hostage by one loss?"""
held = total - lost_seq
return {"lost_packet": lost_seq, "packets_held_hostage": held,
"added_delay_ms": rtt_ms}
print()
for lost_at in (2, 5, 9):
h = hol_delay(lost_at, 10)
print(f"losing packet {h['lost_packet']:>2} of 10 -> {h['packets_held_hostage']} "
f"intact packets stalled for {h['added_delay_ms']} ms")
# losing packet 2 of 10 -> 8 intact packets stalled for 50 ms
# losing packet 5 of 10 -> 5 intact packets stalled for 50 ms
# losing packet 9 of 10 -> 1 intact packets stalled for 50 ms
# What "late" costs each application type.
APPS = {
"file download": ("TCP", "a late byte is still correct — wait for it"),
"web page": ("TCP", "a missing byte corrupts the document"),
"database": ("TCP", "correctness is non-negotiable"),
"voice call": ("UDP", "audio older than ~150 ms is unplayable"),
"live video": ("UDP", "a late frame is already off the screen"),
"online game": ("UDP", "the NEXT position update supersedes the lost one"),
"DNS query": ("UDP", "one exchange; just ask again on timeout"),
"metrics": ("UDP", "one lost sample out of thousands is irrelevant"),
}
print()
print(f"{'application':<16}{'transport':<11}reason")
for app, (proto, why) in APPS.items():
print(f"{app:<16}{proto:<11}{why}")
# Overhead comparison.
def overhead(payload, proto):
hdr = 20 if proto == "TCP" else 8
return hdr, 100 * hdr / (hdr + payload)
print()
print(f"{'payload':>8}{'TCP hdr':>10}{'TCP %':>8}{'UDP hdr':>10}{'UDP %':>8}")
for p in (20, 100, 1460):
th, tp = overhead(p, "TCP")
uh, up = overhead(p, "UDP")
print(f"{p:>8}{th:>10}{tp:>7.1f}%{uh:>10}{up:>7.1f}%")
# payload TCP hdr TCP % UDP hdr UDP %
# 20 20 50.0% 8 28.6%
# 100 20 16.7% 8 7.4%
# 1460 20 1.4% 8 0.5%
# For small frequent messages (VoIP, telemetry) the header ratio matters;
# for bulk transfer it is negligible.
# Watch TCP retransmit — the reliability machinery in action:
$ sudo tcpdump -i eth0 -n 'tcp[tcpflags] & tcp-ack != 0' -c 20 | grep -i retrans
$ nstat -az | grep -iE "retrans|lost"
# TcpRetransSegs 18492
# TcpExtTCPLostRetransmit 112
# TcpExtTCPFastRetrans 14201
# ^^^^^ fast retransmit (7.3) vs timeout-based
# Per-connection retransmission and RTT:
$ ss -tin state established | head -4
# ESTAB 0 0 192.168.1.20:443 10.1.1.5:51234
# cubic wscale:7,7 rto:204 rtt:3.821/1.242 mss:1448 cwnd:10
# retrans:0/24 rcv_space:14480
# ^^^^ current/total retransmissions on this connection
# UDP has no such counters — there is nothing to retransmit:
$ nstat -az | grep -i udp
# UdpInDatagrams 8412991
# UdpNoPorts 4127 <- no socket bound
# UdpInErrors 0
# UdpRcvbufErrors 2841 <- receiver too slow; DROPPED
# ^^^^ note: dropped, never retransmitted
# Measure the real cost of loss on a TCP transfer:
$ sudo tc qdisc add dev eth0 root netem loss 2%
$ curl -o /dev/null -w 'time: %{time_total}s speed: %{speed_download} B/s\n' \
http://server/100mb.bin
# time: 48.211s speed: 2074219 B/s <- 2% loss, TCP
$ sudo tc qdisc del dev eth0 root
$ curl -o /dev/null -w 'time: %{time_total}s speed: %{speed_download} B/s\n' \
http://server/100mb.bin
# time: 8.104s speed: 12341209 B/s <- no loss
# 2% loss cost ~6x throughput. TCP's reliability is not free.
# A real-time app measuring what it CANNOT retransmit:
$ ss -uan | grep :5004 # RTP media stream
$ ffprobe -i rtp://239.1.1.1:5004 2>&1 | grep -i "missed\|drop"
Key takeaways
- IP guarantees nothing; the transport layer decides how much to repair, and full reliability is a trade, not a free win.
- TCP's reliability rests on sequence numbers, acknowledgements, timers, and retransmission, plus a checksum.
- UDP adds only ports and an optional checksum to IP — deliberately the thinnest possible layer.
- Reliability costs latency: detecting loss takes a timeout or duplicate ACKs, and recovery takes at least one more round trip.
- For real-time data, late equals lost — retransmission spends bandwidth delivering data that will be discarded.
- Head-of-line blocking means one lost TCP segment stalls already-arrived data behind it.
- The deciding question is "what does my application do with a late packet?" — use it (TCP) or discard it (UDP).
- UDP applications handle loss by concealment, supersession (send fresher data), or forward error correction, avoiding a round trip.
🧪 Practice
- Name the four mechanisms TCP uses for reliability and the one thing UDP adds to IP besides ports.
- Explain why retransmitting a lost audio packet makes a voice call worse rather than better.
- A TCP connection loses segment 3 of 12. State how many intact segments are held hostage and why.
- Interview-style: Reliability sounds strictly better than unreliability. Give a concrete case where it measurably degrades user experience. (Hint: pick an application where the data describes a moment in time, and ask what the retransmitted copy is worth when it arrives.)
Connection-Oriented vs Connectionless Semantics
Theory
Beyond reliability, transports differ in whether they establish a relationship before exchanging data. This is a separate axis from reliability, though the two are correlated in practice.
Connection-oriented transport (TCP) establishes state at both endpoints before any data moves. The three-way handshake (7.3) synchronises sequence numbers, negotiates options such as window scaling and MSS, and confirms that both parties are present and willing. Data flows only after that. Closing is also explicit, with a four-way exchange.
That state — the TCP Control Block — holds sequence numbers, window sizes, timers, congestion state, and buffers. It exists on both endpoints for the connection's lifetime.
Connectionless transport (UDP) has no such phase. A datagram carries a destination and is sent. There is no handshake, no shared state, no teardown, and no notion of a "connection" at all — each datagram is independent, and the protocol has no memory of the last one.
The consequences are systematic:
| Connection-oriented (TCP) | Connectionless (UDP) | |
|---|---|---|
| Setup cost | 1 RTT before any data | Zero |
| State per conversation | On both endpoints | None |
| Ordering | Guaranteed | None |
| Failure detection | Timeouts, RST, keepalives | None |
| Server memory | Scales with connections | Scales with nothing |
| Multicast/broadcast | Impossible (inherently 1:1) | Natural |
| NAT/firewall traversal | Explicit state, clean | Timer-based guesswork |
Two entries deserve elaboration because they are frequently misunderstood.
The 1-RTT setup cost is not trivial. For a short exchange it can dominate. A DNS query is one small request and one small response; a TCP handshake would triple the time before the answer arrives. On a 100 ms path, TCP costs ~200 ms before the query is even sent, while UDP delivers the answer in ~100 ms. This is precisely why DNS uses UDP by default. (TLS historically added another 1–2 RTTs on top, which is what QUIC's 0-RTT and 1-RTT handshakes attack — see 7.6.)
TCP cannot do multicast. A connection is a relationship between exactly two endpoints, with per-connection state on each. Multicast means one sender and an unknown, changing set of receivers — there is nothing to handshake with. Every multicast application therefore uses UDP, without exception.
A frequent source of confusion is the "connected UDP socket". Calling connect() on a UDP socket does not create a connection or send any packets. It merely records a default destination in the kernel so the application can use send() instead of sendto(), and it causes the socket to filter incoming datagrams to that peer and to receive ICMP errors. It is a local convenience, not a protocol-level relationship.
The other important observation is that connection state in the middle of the network is where NAT gets its difficulty. A NAT or stateful firewall must infer UDP "connections" from traffic patterns and expire them on timers, because UDP gives it nothing explicit to track. TCP announces its beginning (SYN) and its end (FIN/RST), so middleboxes can manage state precisely. This is why UDP applications need keepalives to hold NAT mappings open, and why UDP NAT timeouts are often as short as 30 seconds.
Examples
Setup cost: what a short exchange really pays
DNS over UDP (1 RTT total):
t=0 query ----------------->
t=100 <----------------- answer DONE. 100 ms.
The same over TCP (3 RTTs total):
t=0 SYN ----------------->
t=100 <----------------- SYN-ACK
t=200 ACK + query -------------->
t=300 <----------------- answer DONE. 300 ms.
t=300 FIN exchange follows...
3x the latency to move the same 100 bytes. This is why DNS uses UDP.
State: who remembers what
TCP — both endpoints hold a control block for the connection's lifetime
CLIENT SERVER
[seq, ack, cwnd, rwnd, ] <--------> [seq, ack, cwnd, rwnd, ]
[timers, buffers, state ] [timers, buffers, state]
^ ~100 KB per connection, on EACH side
UDP — nothing, anywhere
CLIENT SERVER
(no state) ---- datagram ----> (no state)
each one independent; the protocol has no memory
Why TCP cannot multicast
TCP: a connection is a 1:1 relationship with negotiated state.
Who would you handshake with? How would you ACK from
10,000 receivers? Whose window would you obey?
-> structurally impossible.
UDP: send one datagram to 239.1.1.1. Done.
No state, no handshake, no acknowledgement expected.
-> EVERY multicast application uses UDP.
def exchange_time(transport, rtt_ms, requests=1, tls=False):
"""Time to complete N request/response exchanges."""
setup = 0
if transport == "TCP":
setup += rtt_ms # 3-way handshake
if tls:
setup += 2 * rtt_ms # classic TLS 1.2
elif transport == "QUIC":
setup += rtt_ms # handshake + crypto together
data = requests * rtt_ms
return setup + data
print(f"{'transport':<14}{'setup':>8}{'1 req':>9}{'10 reqs':>10}")
for t, tls in [("UDP", False), ("TCP", False), ("TCP", True), ("QUIC", False)]:
label = f"{t}{'+TLS' if tls else ''}"
setup = exchange_time(t, 100, requests=0, tls=tls)
one = exchange_time(t, 100, 1, tls)
ten = exchange_time(t, 100, 10, tls)
print(f"{label:<14}{setup:>7} ms{one:>8} ms{ten:>9} ms")
# transport setup 1 req 10 reqs
# UDP 0 ms 100 ms 1000 ms
# TCP 100 ms 200 ms 1100 ms
# TCP+TLS 300 ms 400 ms 1300 ms
# QUIC 100 ms 200 ms 1100 ms
# Setup is amortised over many requests — which is exactly why HTTP
# keep-alive and connection pooling matter so much.
# Server-side state cost.
def server_state(conns, tcp_block=2048, buffers=103424):
return {
"tcp_bytes": conns * (tcp_block + buffers),
"udp_bytes": 0,
}
print()
for n in (10_000, 100_000, 1_000_000):
s = server_state(n)
print(f"{n:>9,} conversations: TCP {s['tcp_bytes']/1e9:>6.2f} GB of state "
f"UDP {s['udp_bytes']} bytes")
# 10,000 conversations: TCP 1.05 GB of state UDP 0 bytes
# 100,000 conversations: TCP 10.55 GB of state UDP 0 bytes
# 1,000,000 conversations: TCP 105.47 GB of state UDP 0 bytes
# A UDP server's memory scales with its own data structures, not with
# the number of clients — which is why DNS roots serve the planet.
# The "connected UDP socket" — a LOCAL convenience, not a connection.
import socket
def connected_udp_demo():
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(("8.8.8.8", 53)) # sends NOTHING on the wire
return {
"packets_sent_by_connect": 0,
"can_use_send_not_sendto": True,
"filters_incoming_to_peer": True,
"receives_icmp_errors": True,
"is_a_real_connection": False,
}
print()
for k, v in connected_udp_demo().items():
print(f" {k:<28}{v}")
# packets_sent_by_connect 0
# can_use_send_not_sendto True
# filters_incoming_to_peer True
# receives_icmp_errors True
# is_a_real_connection False
# Middlebox state: TCP announces itself, UDP must be guessed at.
def nat_timeout(protocol):
return {
"TCP": (7440, "SYN opens it, FIN/RST closes it — explicit and precise"),
"UDP": (30, "no signals; inferred from traffic and expired on a timer"),
}[protocol]
print()
for proto in ("TCP", "UDP"):
t, why = nat_timeout(proto)
print(f"{proto}: typical NAT mapping timeout {t:>5} s ({why})")
# TCP: typical NAT mapping timeout 7440 s (SYN opens it, FIN/RST closes it — explicit and precise)
# UDP: typical NAT mapping timeout 30 s (no signals; inferred from traffic and expired on a timer)
def keepalive_needed(protocol, idle_s):
t, _ = nat_timeout(protocol)
return idle_s > t
print()
for proto in ("TCP", "UDP"):
print(f"{proto} idle for 60 s -> mapping lost? {keepalive_needed(proto, 60)}")
# TCP idle for 60 s -> mapping lost? False
# UDP idle for 60 s -> mapping lost? True
# This is why UDP-based applications (VoIP, VPNs, QUIC) send keepalives.
# TCP announces its lifecycle; a middlebox can track it exactly:
$ sudo tcpdump -i eth0 -n 'tcp[tcpflags] & (tcp-syn|tcp-fin|tcp-rst) != 0' -c 4
# IP 10.1.1.5.51234 > 93.184.216.34.443: Flags [S] <- connection BEGINS
# IP 93.184.216.34.443 > 10.1.1.5.51234: Flags [S.]
# IP 10.1.1.5.51234 > 93.184.216.34.443: Flags [F.] <- connection ENDS
# IP 93.184.216.34.443 > 10.1.1.5.51234: Flags [F.]
# Conntrack sees the difference plainly:
$ sudo conntrack -L -p tcp | head -1
# tcp 6 431999 ESTABLISHED src=10.1.1.5 dst=93.184.216.34 sport=51234 dport=443
# ^^^^^^ ~5 days: the state is explicit, so it can be trusted
$ sudo conntrack -L -p udp | head -1
# udp 17 29 src=10.1.1.5 dst=8.8.8.8 sport=51234 dport=53
# ^^ 29 seconds: no signals, so it must expire aggressively
$ sysctl net.netfilter.nf_conntrack_udp_timeout net.netfilter.nf_conntrack_tcp_timeout_established
# net.netfilter.nf_conntrack_udp_timeout = 30
# net.netfilter.nf_conntrack_tcp_timeout_established = 432000
# Measure the setup cost yourself:
$ curl -w 'dns:%{time_namelookup} tcp:%{time_connect} tls:%{time_appconnect} first_byte:%{time_starttransfer}\n' \
-o /dev/null -s https://example.com
# dns:0.021 tcp:0.110 tls:0.298 first_byte:0.401
# ^^^^^ 1 RTT for the handshake
# ^^^^^ ~2 more RTTs for TLS 1.2
# 0.3 s consumed before a single byte of content moved.
# The same content over HTTP/3 (QUIC over UDP):
$ curl --http3 -w 'tls:%{time_appconnect} first_byte:%{time_starttransfer}\n' \
-o /dev/null -s https://cloudflare.com
# tls:0.112 first_byte:0.148 <- handshake and crypto in ONE RTT
# Why UDP applications send keepalives — watch a WireGuard tunnel:
# [Peer]
# PersistentKeepalive = 25
# ^^ under the typical 30 s UDP NAT timeout
$ sudo wg show | grep -i keepalive
# persistent keepalive: every 25 seconds
Key takeaways
- Connection-oriented (TCP) establishes shared state via a three-way handshake before any data; connectionless (UDP) sends datagrams with no setup and no memory.
- TCP's setup costs 1 RTT (plus 2 more for classic TLS), which dominates short exchanges — the reason DNS uses UDP.
- TCP keeps a control block plus buffers on both endpoints; UDP keeps nothing, so a UDP server's memory does not scale with client count.
- TCP cannot do multicast or broadcast — a connection is inherently 1:1, so every multicast application uses UDP.
- A "connected" UDP socket sends no packets;
connect()only sets a default peer, filters inbound datagrams, and enables ICMP error delivery. - Middleboxes track TCP precisely because SYN and FIN/RST announce the lifecycle; UDP state must be inferred and expired on short timers (often 30 s).
- Consequently UDP applications need keepalives (WireGuard's 25 s default) to hold NAT mappings open.
🧪 Practice
- Compute the total time for a single small request/response over UDP, TCP, and TCP+TLS 1.2 on a 40 ms RTT path.
- Explain why TCP cannot be used for multicast, referring to what a connection requires.
- State what
connect()does on a UDP socket and what it does not do.- Interview-style: Why do NAT devices expire UDP mappings after 30 seconds but keep TCP mappings for days? (Hint: ask what each protocol tells the middlebox about when a conversation starts and ends.)
<a id="72-udp"></a>
7.2 UDP
This subchapter covers the minimal transport: an eight-byte header, no state, no guarantees — and the reasons that austerity is exactly what many applications need.
UDP Header and Checksum
Theory
UDP's header is 8 bytes, and there is nothing in it that is not strictly necessary. Comparing it with TCP's 20-byte minimum is the fastest way to see what reliability actually costs in header space.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-------------------------------+-------------------------------+
| Source Port | Destination Port |
+-------------------------------+-------------------------------+
| Length | Checksum |
+-------------------------------+-------------------------------+
| data ... |
Four fields, two bytes each:
| Field | Purpose |
|---|---|
| Source Port | Where replies go; optional — may be 0 if no reply is expected |
| Destination Port | Demultiplexing key |
| Length | Header plus data, in bytes; minimum 8 |
| Checksum | Covers header, data, and a pseudo-header |
The Length field is mildly redundant, since IP's Total Length already implies it, and this is a known piece of historical inefficiency. Note that it counts the header too, so a zero-payload datagram has Length 8.
The checksum deserves the most attention, because it does something subtle. It covers not only the UDP header and data but also a pseudo-header assembled from the IP layer: source address, destination address, protocol number, and UDP length.
Why reach down into IP? Because the transport checksum is the only end-to-end check on the addresses. If a router corrupts a destination address, the packet is delivered to the wrong host — and that host's UDP checksum will fail, because it computes the pseudo-header from the address it actually received. Without the pseudo-header, misdelivery would be undetectable. This is a direct application of the end-to-end argument (Chapter 5).
It is also a layering violation, and an acknowledged one: the transport layer must know about IP addresses to compute its own checksum. The violation was accepted because the alternative — undetectable misdelivery — is worse.
Two details about checksums trip people up.
In IPv4 the UDP checksum is optional. A value of 0 means "not computed". This was a 1980 performance concession, and using it now is a bad idea on any path that is not a trusted LAN. In IPv6 it is mandatory, because IPv6 removed the IP header checksum entirely (Chapter 5) — so the transport checksum is the only protection the addresses have.
A computed checksum of zero is transmitted as 0xFFFF. Since 0 already means "no checksum", the ones-complement representation's second form of zero is used to avoid the ambiguity.
Finally, UDP's checksum is weak. It is a 16-bit ones-complement sum, which detects all single-bit errors and most bursts but misses certain patterns — notably the reordering of 16-bit words, and compensating errors in different words. Applications requiring genuine integrity should use a cryptographic MAC (as TLS, DTLS, and QUIC do), not rely on this.
Examples
Header size: what reliability costs
UDP (8 bytes):
+--------+--------+--------+--------+
| srcprt | dstprt | length | cksum |
+--------+--------+--------+--------+
TCP (20 bytes minimum):
+--------+--------+-----------------+
| srcprt | dstprt | sequence num |
+--------+--------+-----------------+
| acknowledgment number |
+--------+--------+--------+--------+
|offs|flg| window | cksum | urgptr |
+--------+--------+--------+--------+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
everything here exists to support reliability, ordering,
flow control and congestion control. UDP needs none of it.
The pseudo-header: why UDP must know about IP addresses
+---------------------------------------+
| Source IP Address (4) | <-- from the IP header
+---------------------------------------+
| Destination IP Address (4) | <-- from the IP header
+--------+--------+---------------------+
| zero | proto | UDP length | proto = 17
+--------+--------+---------------------+
| the real UDP header |
| the data |
+---------------------------------------+
^ all of this is summed
A router corrupts the destination address:
packet is delivered to the WRONG host
that host computes the pseudo-header from the address IT sees
-> checksum MISMATCH -> datagram discarded
Without the pseudo-header, misdelivery would be silent.
import struct, socket
def udp_checksum(src_ip, dst_ip, src_port, dst_port, payload):
"""Ones-complement sum over pseudo-header + UDP header + data."""
length = 8 + len(payload)
pseudo = (socket.inet_aton(src_ip) + socket.inet_aton(dst_ip)
+ struct.pack("!BBH", 0, 17, length)) # 17 = UDP
header = struct.pack("!HHHH", src_port, dst_port, length, 0) # cksum = 0
data = pseudo + header + payload
if len(data) % 2:
data += b"\x00" # pad to 16 bits
total = 0
for i in range(0, len(data), 2):
total += (data[i] << 8) | data[i + 1]
total = (total & 0xFFFF) + (total >> 16) # fold the carry
csum = ~total & 0xFFFF
return csum if csum != 0 else 0xFFFF # 0 means "no checksum", so use 0xFFFF
payload = b"hello"
c = udp_checksum("192.168.1.20", "8.8.8.8", 51234, 53, payload)
print(f"UDP length : {8 + len(payload)} bytes (header 8 + data {len(payload)})")
print(f"checksum : 0x{c:04x}")
# UDP length : 13 bytes (header 8 + data 5)
# checksum : 0x21de
# The pseudo-header is what catches MISDELIVERY.
correct = udp_checksum("192.168.1.20", "8.8.8.8", 51234, 53, payload)
corrupted = udp_checksum("192.168.1.20", "8.8.9.8", 51234, 53, payload)
print(f"\nchecksum with correct dst 8.8.8.8 : 0x{correct:04x}")
print(f"checksum with corrupt dst 8.8.9.8 : 0x{corrupted:04x}")
print(f"the wrong host recomputes and gets a MISMATCH -> discards. "
f"Different: {correct != corrupted}")
# checksum with correct dst 8.8.8.8 : 0x21de
# checksum with corrupt dst 8.8.9.8 : 0x20de
# the wrong host recomputes and gets a MISMATCH -> discards. Different: True
# Header overhead across payload sizes.
def overhead(payload):
return {
"udp_pct": 100 * 8 / (8 + payload),
"tcp_pct": 100 * 20 / (20 + payload),
"saving_bytes": 12,
}
print()
print(f"{'payload':>9}{'UDP hdr %':>12}{'TCP hdr %':>12}")
for p in (12, 50, 512, 1472):
o = overhead(p)
print(f"{p:>9}{o['udp_pct']:>11.1f}%{o['tcp_pct']:>11.1f}%")
# payload UDP hdr % TCP hdr %
# 12 40.0% 62.5%
# 50 13.8% 28.6%
# 512 1.5% 3.8%
# 1472 0.5% 1.3%
# For a 12-byte VoIP-style payload the header dominates — one reason
# RTP runs over UDP and not TCP.
# Checksum strength: what it does and does not catch.
def checksum_detects(error_type):
return {
"single bit flip": (True, "always detected"),
"burst < 16 bits": (True, "always detected"),
"two compensating errors": (False, "sum unchanged -> MISSED"),
"reordered 16-bit words": (False, "addition is commutative -> MISSED"),
"wrong destination address": (True, "pseudo-header catches it"),
}[error_type]
print()
for err in ["single bit flip", "burst < 16 bits", "two compensating errors",
"reordered 16-bit words", "wrong destination address"]:
ok, note = checksum_detects(err)
print(f"{err:<28}{'DETECTED' if ok else 'missed '} {note}")
# single bit flip DETECTED always detected
# burst < 16 bits DETECTED always detected
# two compensating errors missed sum unchanged -> MISSED
# reordered 16-bit words missed addition is commutative -> MISSED
# wrong destination address DETECTED pseudo-header catches it
# For real integrity, use a cryptographic MAC (DTLS/QUIC), not this.
# IPv4 vs IPv6 checksum rules.
print()
for ver, (optional, why) in {
"IPv4": (True, "IP header has its own checksum; UDP's is a 1980 concession"),
"IPv6": (False, "IPv6 REMOVED the IP header checksum — UDP's is the only one"),
}.items():
print(f"{ver}: UDP checksum optional = {str(optional):<5} ({why})")
# IPv4: UDP checksum optional = True (IP header has its own checksum; UDP's is a 1980 concession)
# IPv6: UDP checksum optional = False (IPv6 REMOVED the IP header checksum — UDP's is the only one)
# The header on the wire — four fields, nothing else:
$ sudo tcpdump -i eth0 -n -v udp port 53 -c 1
# IP (tos 0x0, ttl 64, id 5512, offset 0, flags [DF], proto UDP (17), length 61)
# 192.168.1.20.51234 > 8.8.8.8.53: [udp sum ok] 33125+ A? example.com. (33)
# ^^^^^^^^^^^ tcpdump verified it
# ^^^^ the
# UDP Length field is 33 + 8 = 41
# A datagram with NO checksum (legal in IPv4 only):
$ sudo tcpdump -i eth0 -n -v udp | grep -i "no cksum"
# IP 10.1.1.5.5000 > 10.1.1.9.5000: [no cksum] UDP, length 100
# ^^^^^^^^^ checksum field is 0.
# Corruption will not be detected.
# IPv6 makes it mandatory — the kernel will not send a zero checksum:
$ sudo tcpdump -i eth0 -n -v ip6 and udp -c 1
# IP6 2001:db8::1.5353 > ff02::fb.5353: [udp sum ok] UDP, length 45
# ^^^^^^^^^^ never optional here
# Checksum offload: the NIC computes it, so local captures may look "wrong":
$ ethtool -k eth0 | grep checksum
# rx-checksumming: on
# tx-checksumming: on
# tx-checksum-ipv4: on
$ sudo tcpdump -i eth0 -n udp -v | grep -i "bad udp cksum"
# IP 10.1.1.5.5000 > 8.8.8.8.53: [bad udp cksum 0x1234 -> 0xabcd!] UDP
# ^ NOT a real error on an OUTBOUND local capture: tcpdump saw the packet
# BEFORE the NIC filled in the checksum. Verify by capturing at the peer.
$ sudo ethtool -K eth0 tx-checksum-ipv4 off # disable to confirm
# Corruption that made it past the checksum is counted:
$ nstat -az | grep -i udp
# UdpInDatagrams 8412991
# UdpInCsumErrors 14 <- detected and discarded
# UdpNoPorts 4127
# UdpRcvbufErrors 2841
# Minimum and maximum datagram sizes:
$ python3 -c "
print('min UDP datagram:', 8, 'bytes (header only, zero payload)')
print('max UDP length field:', 65535, '-> max payload:', 65535-8)
print('practical max without fragmenting on Ethernet:', 1500-20-8)"
# min UDP datagram: 8 bytes (header only, zero payload)
# max UDP length field: 65535 -> max payload: 65527
# practical max without fragmenting on Ethernet: 1472
Key takeaways
- UDP's header is 8 bytes: source port, destination port, length, checksum — nothing more.
- Source port is optional (may be 0 when no reply is expected); Length counts the header, so the minimum is 8.
- The checksum covers a pseudo-header (source IP, destination IP, protocol, UDP length) as well as the header and data.
- The pseudo-header exists so that misdelivery caused by a corrupted address is detectable end-to-end — an accepted layering violation.
- The checksum is optional in IPv4 (0 = not computed) but mandatory in IPv6, since IPv6 has no IP header checksum.
- A computed checksum of zero is sent as 0xFFFF to avoid colliding with the "no checksum" value.
- The checksum is weak: it misses compensating errors and reordered 16-bit words — use a cryptographic MAC for real integrity.
- Maximum payload is 65,527 bytes, but 1,472 is the practical limit on Ethernet before fragmentation.
- Checksum offload makes outbound local captures show false "bad checksum" warnings.
🧪 Practice
- Give the four UDP header fields and state what the Length field counts.
- Explain what the pseudo-header is and which specific failure it allows a receiver to detect.
- State why the UDP checksum is optional in IPv4 but mandatory in IPv6.
- Interview-style: A packet capture on a server shows "bad udp cksum" on every outgoing datagram, yet clients receive the data fine. Explain. (Hint: ask which component fills in the checksum, and whether tcpdump ran before or after it.)
Use Cases for UDP
Theory
UDP is chosen when TCP's guarantees cost more than they are worth. The situations fall into five recognisable categories, and recognising which one applies is the whole of the design decision.
1. Latency matters more than completeness. Real-time media — voice, video conferencing, live streaming. A late packet is worthless (7.1), so retransmission wastes bandwidth and adds delay. Codecs conceal small losses far better than the application can conceal a 200 ms stall.
2. The exchange is short and a handshake would dominate. DNS is the canonical case: one small query, one small answer. TCP's handshake would triple latency for a 100-byte exchange. The application simply retries on timeout, which is cheaper than establishing a connection. DHCP, NTP, and SNMP share this shape.
3. The destination is multicast or broadcast. TCP cannot do it (7.1). Every multicast application — IPTV, market data feeds, service discovery, routing protocol updates — uses UDP by necessity, not preference.
4. Newer data supersedes older data. In a game, a player's position update at t=100 ms makes the lost update from t=80 ms irrelevant. Retransmitting stale state is worse than useless; it wastes bandwidth to deliver information that is already wrong. The same logic applies to sensor telemetry and metrics.
5. The application wants to build its own transport. QUIC is the most consequential example. It runs over UDP not because it wants UDP's semantics — QUIC is fully reliable — but because UDP is the only way to deploy a new transport across the internet's middleboxes (Chapter 5). UDP here is a substrate, not a service.
The applications, and which reason drives each:
| Application | Port | Reason |
|---|---|---|
| DNS | 53 | Short exchange; retry is cheaper than a handshake |
| DHCP | 67/68 | Broadcast; the client has no address yet |
| NTP | 123 | Short exchange; a retransmitted timestamp is wrong anyway |
| SNMP | 161 | Short exchange; polling tolerates loss |
| Syslog | 514 | Fire-and-forget; must not block the application |
| RTP / VoIP | dynamic | Latency; late audio is unusable |
| IPTV / multicast | dynamic | Multicast is impossible over TCP |
| Online games | dynamic | Newer state supersedes older |
| QUIC / HTTP-3 | 443 | Substrate for a custom transport |
| WireGuard | 51820 | Substrate; also avoids TCP-over-TCP |
Two cases deserve expansion.
NTP's reason is subtler than "short exchange". A retransmitted timestamp is actively wrong — it describes a moment that has passed, and using it would corrupt the clock synchronisation it exists to provide. Reliability would not merely be wasteful; it would produce incorrect results.
TCP-over-TCP is a real pathology, and it is why VPNs prefer UDP. Running a TCP tunnel inside a TCP connection gives two independent retransmission timers and two congestion controllers stacked on the same path. When loss occurs, the inner TCP retransmits while the outer TCP is also retransmitting and backing off, so the inner connection sees growing delay, retransmits more, and the two amplify each other into meltdown. OpenVPN over UDP and WireGuard (UDP-only) avoid this by design.
Examples
The five reasons, with their canonical applications
1. LATENCY over completeness VoIP, video conferencing, live streaming
"a late packet is unusable"
2. SHORT EXCHANGE DNS, NTP, DHCP, SNMP
"the handshake costs more than the data"
3. MULTICAST/BROADCAST IPTV, market data, discovery, DHCP
"TCP structurally cannot do this"
4. NEWER SUPERSEDES OLDER games, telemetry, sensor feeds
"the retransmitted value is already stale"
5. CUSTOM TRANSPORT SUBSTRATE QUIC/HTTP-3, WireGuard, DTLS
"UDP is what middleboxes will pass"
DNS: why the handshake dominates
over UDP: over TCP:
query ------> SYN ------>
<------ answer <------ SYN-ACK
ACK+query ---->
1 RTT = 100 ms <------ answer
FIN exchange...
3 RTT = 300 ms
Same 100 bytes of useful data. 3x the latency, plus connection
state on a server answering millions of queries per second.
TCP-over-TCP meltdown: why VPNs use UDP
inner TCP [====== retransmit timer A ======]
outer TCP [====== retransmit timer B ======]
|
loss occurs on the real path
|
outer TCP retransmits AND backs off (slower)
|
inner TCP sees rising delay, thinks ITS packet was lost,
retransmits too -> more data into an already-backing-off tunnel
|
---> MELTDOWN <---
Throughput collapses and does not recover until idle.
UDP tunnel: ONE retransmission timer (the inner TCP's). No stacking.
def choose_transport(late_packet_usable, exchange_rtts, needs_multicast,
newer_supersedes, custom_transport):
if needs_multicast:
return "UDP", "TCP cannot multicast — structurally impossible"
if custom_transport:
return "UDP", "substrate for a custom transport (middlebox compatibility)"
if not late_packet_usable:
return "UDP", "a late packet is unusable; retransmission wastes bandwidth"
if newer_supersedes:
return "UDP", "the retransmitted value would already be stale"
if exchange_rtts <= 1:
return "UDP", "handshake would dominate a one-exchange conversation"
return "TCP", "data must arrive complete and in order"
SCENARIOS = [
("file download", dict(late_packet_usable=True, exchange_rtts=100, needs_multicast=False, newer_supersedes=False, custom_transport=False)),
("DNS query", dict(late_packet_usable=True, exchange_rtts=1, needs_multicast=False, newer_supersedes=False, custom_transport=False)),
("VoIP call", dict(late_packet_usable=False, exchange_rtts=999, needs_multicast=False, newer_supersedes=False, custom_transport=False)),
("IPTV stream", dict(late_packet_usable=False, exchange_rtts=999, needs_multicast=True, newer_supersedes=False, custom_transport=False)),
("game position", dict(late_packet_usable=True, exchange_rtts=999, needs_multicast=False, newer_supersedes=True, custom_transport=False)),
("HTTP/3", dict(late_packet_usable=True, exchange_rtts=50, needs_multicast=False, newer_supersedes=False, custom_transport=True)),
]
for name, kw in SCENARIOS:
proto, why = choose_transport(**kw)
print(f"{name:<16}{proto:<6}{why}")
# file download TCP data must arrive complete and in order
# DNS query UDP handshake would dominate a one-exchange conversation
# VoIP call UDP a late packet is unusable; retransmission wastes bandwidth
# IPTV stream UDP TCP cannot multicast — structurally impossible
# game position UDP the retransmitted value would already be stale
# HTTP/3 UDP substrate for a custom transport (middlebox compatibility)
# DNS at scale: the cost of connection state that UDP avoids.
def dns_server_cost(qps, transport, tcp_state_bytes=105472, conn_duration_s=1):
if transport == "UDP":
return {"concurrent_state": 0, "memory_gb": 0.0,
"latency_rtts": 1}
concurrent = qps * conn_duration_s
return {"concurrent_state": concurrent,
"memory_gb": concurrent * tcp_state_bytes / 1e9,
"latency_rtts": 3}
print()
for qps in (10_000, 1_000_000):
for t in ("UDP", "TCP"):
c = dns_server_cost(qps, t)
print(f"{qps:>9,} qps over {t}: {c['concurrent_state']:>9,} sockets, "
f"{c['memory_gb']:>7.1f} GB, {c['latency_rtts']} RTT")
# 10,000 qps over UDP: 0 sockets, 0.0 GB, 1 RTT
# 10,000 qps over TCP: 10,000 sockets, 1.1 GB, 3 RTT
# 1,000,000 qps over UDP: 0 sockets, 0.0 GB, 1 RTT
# 1,000,000 qps over TCP: 1,000,000 sockets, 105.5 GB, 3 RTT
# "Newer supersedes older": why retransmitting game state is harmful.
def position_updates(tick_ms, rtt_ms, lost_tick):
retransmit_arrives = lost_tick * tick_ms + rtt_ms
ticks_elapsed = (retransmit_arrives - lost_tick * tick_ms) // tick_ms
return {
"lost_update_age_on_arrival_ms": rtt_ms,
"newer_updates_already_received": int(ticks_elapsed),
"verdict": "the retransmitted position is already obsolete",
}
print()
p = position_updates(tick_ms=16, rtt_ms=80, lost_tick=100)
for k, v in p.items():
print(f" {k:<34}{v}")
# lost_update_age_on_arrival_ms 80
# newer_updates_already_received 5
# verdict the retransmitted position is already obsolete
# TCP-over-TCP: two stacked control loops.
def tunnel_timers(outer, inner):
loops = (1 if outer == "TCP" else 0) + (1 if inner == "TCP" else 0)
risk = "MELTDOWN RISK: stacked timers amplify each other" if loops == 2 \
else "stable: one retransmission control loop"
return loops, risk
print()
for outer in ("TCP", "UDP"):
loops, risk = tunnel_timers(outer, "TCP")
print(f"{outer} tunnel carrying TCP: {loops} retransmit loop(s) — {risk}")
# TCP tunnel carrying TCP: 2 retransmit loop(s) — MELTDOWN RISK: stacked timers amplify each other
# UDP tunnel carrying TCP: 1 retransmit loop(s) — stable: one retransmission control loop
# DNS uses UDP first, TCP only when it must:
$ dig example.com | grep -i "SERVER\|MSG SIZE"
# ;; SERVER: 8.8.8.8#53(8.8.8.8) (UDP)
# ;; MSG SIZE rcvd: 56
$ dig +tcp example.com | grep -i SERVER
# ;; SERVER: 8.8.8.8#53(8.8.8.8) (TCP)
$ dig DNSKEY . | grep -iE "truncated|SERVER"
# ;; Truncated, retrying in TCP mode. <- response exceeded the UDP limit
# The TC bit set in a UDP response tells the client to retry over TCP.
# See which of your own connections use each transport:
$ ss -uan | wc -l ; ss -tan | wc -l
# 24 # UDP sockets
# 1842 # TCP sockets
$ ss -uanp | grep -E "53|123|51820" | head -3
# UNCONN 0 0 0.0.0.0:53 0.0.0.0:* users:(("systemd-resolve",pid=812,fd=12))
# UNCONN 0 0 0.0.0.0:123 0.0.0.0:* users:(("chronyd",pid=904,fd=5))
# UNCONN 0 0 0.0.0.0:51820 0.0.0.0:* users:(("wg",pid=0,fd=0))
# ^^^^^ WireGuard: UDP-only by design
# HTTP/3 is QUIC over UDP on port 443:
$ curl --http3 -sI https://cloudflare.com | head -1
# HTTP/3 200
$ sudo tcpdump -i eth0 -n 'udp port 443' -c 2
# IP 10.1.1.5.51234 > 104.16.132.229.443: UDP, length 1350
# ^^^ QUIC. A reliable transport
# riding on an unreliable one.
# RTP media: sequence numbers in the APPLICATION, not the transport:
$ sudo tcpdump -i eth0 -n udp portrange 16384-32767 -X -c 1 | head -4
# IP 10.1.1.5.5004 > 10.1.1.9.5004: UDP, length 172
# 0x0000: 8008 1a2b 0001 d4c0 ...
# ^^^^ ^^^^ RTP version/type, then SEQUENCE NUMBER —
# the application doing what UDP will not.
# OpenVPN over UDP vs TCP — the configuration that avoids meltdown:
$ grep -E "^proto" /etc/openvpn/client.conf
# proto udp
# ^^^ recommended. 'proto tcp' works but risks TCP-over-TCP collapse
# on lossy paths.
Key takeaways
- UDP is chosen for five reasons: latency over completeness, short exchanges, multicast/broadcast, newer data supersedes older, and as a substrate for a custom transport.
- DNS uses UDP because a handshake would triple the latency of a one-exchange conversation and impose per-client state on the server.
- NTP's case is stronger than efficiency: a retransmitted timestamp is actively wrong and would corrupt synchronisation.
- Multicast applications have no choice — TCP cannot multicast.
- In games and telemetry, the next update supersedes the lost one, so retransmission delivers stale state.
- QUIC and WireGuard use UDP as a substrate, not for its semantics — it is what middleboxes reliably pass.
- TCP-over-TCP stacks two retransmission timers and two congestion controllers, which amplify into meltdown on lossy paths — hence UDP-based VPNs.
- DNS falls back to TCP when a response is truncated (the TC bit) or for zone transfers.
🧪 Practice
- Name the five reasons to choose UDP and give one application for each.
- Compute the latency difference for a DNS query over UDP versus TCP on a 60 ms RTT path, and state the server-side state cost of each.
- Explain why retransmitting a lost NTP timestamp would be worse than dropping it.
- Interview-style: A VPN performs well on a clean link but collapses to near-zero throughput on a 3% loss path when configured over TCP. Explain the mechanism. (Hint: count how many independent retransmission timers are operating on the same physical path, and what each does when it sees delay.)
UDP and Application-Level Reliability
Theory
Choosing UDP does not mean abandoning reliability. It means moving reliability into the application, where it can be applied selectively to the data that needs it and skipped for data that does not. TCP offers one setting: everything, in order, always. An application built on UDP can be far more discriminating.
The building blocks are the same mechanisms TCP uses, implemented in the application:
Sequence numbers detect loss, reordering, and duplication. Acknowledgements — but crucially, only for data that matters. Retransmission timers based on measured RTT. Forward error correction (FEC) — send redundant data so the receiver reconstructs losses without a round trip.
That last one has no TCP equivalent and is the most interesting. FEC trades bandwidth for latency: send 10 data packets plus 2 parity packets, and the receiver can reconstruct any 2 lost packets immediately, with no retransmission and no added delay. For real-time media this is transformative, because the round trip TCP would spend recovering is precisely the delay that makes the data useless.
The design patterns that result:
| Pattern | Applies reliability to | Example |
|---|---|---|
| Partial reliability | Only critical messages | Games: inventory changes yes, position updates no |
| FEC | Everything, without a round trip | Media codecs, RaptorQ, Reed-Solomon |
| Idempotent retry | The whole exchange | DNS: just ask again |
| Supersession | Nothing — send fresher data | Telemetry, position updates |
| Full reliability | Everything, but customised | QUIC, custom file transfer |
Partial reliability is the pattern that justifies the whole approach. A game sends two kinds of data over the same UDP socket: position updates at 60 Hz (lossy — the next one supersedes) and events such as "player picked up an item" (must arrive — the game state diverges otherwise). TCP would force the same treatment on both, and a lost position update would head-of-line block the critical event behind it. UDP with selective reliability lets each get what it needs.
The real risk of this approach is implementing it badly. Congestion control in particular is difficult and easy to get catastrophically wrong: a UDP application that retransmits aggressively without backing off contributes to congestion collapse and is unfair to TCP flows sharing the path. RFC 8085 (UDP usage guidelines) is explicit that applications sending more than trivial volumes must implement congestion control. In practice, most applications should use an existing implementation — QUIC, DTLS, SRT, RTP with RTCP feedback — rather than writing one.
Examples
Partial reliability: one socket, two policies
GAME CLIENT GAME SERVER
position update (seq 1001) --UNRELIABLE-------> applied
position update (seq 1002) --UNRELIABLE---X LOST. Ignored:
position update (seq 1003) --UNRELIABLE-------> applied ^ 1003 supersedes
"picked up sword" (seq 7) --RELIABLE---------> applied
<---------ACK 7-----
"opened chest" (seq 8) --RELIABLE------X LOST
[timer expires]
"opened chest" (seq 8) --RETRANSMIT--------> applied
<---------ACK 8-----
Over TCP, the lost position update would have STALLED the
"picked up sword" event behind it. Head-of-line blocking on
data that did not even matter.
Forward error correction: recovery without a round trip
send: [D1][D2][D3][D4][D5][D6][D7][D8][D9][D10][P1][P2]
^^^^^^ 2 parity packets
lost: D3 and D7
receiver has: D1,D2,D4,D5,D6,D8,D9,D10,P1,P2 = 10 of 12
-> RECONSTRUCTS D3 and D7 immediately.
cost: 20% extra bandwidth
saved: one full RTT of delay (and for media, the packet is
still useful, which after an RTT it would not be)
The cost comparison
RETRANSMISSION: 0% extra bandwidth, +1 RTT on loss
FEC: +20% bandwidth, +0 RTT on loss
^ correct choice when latency is the scarce resource
import time, random
class PartiallyReliableChannel:
"""One UDP socket, two delivery policies."""
def __init__(self, rto_ms=100):
self.seq = 0
self.unacked = {} # seq -> (payload, sent_at, attempts)
self.rto_ms = rto_ms
self.stats = {"sent": 0, "retransmitted": 0, "dropped_unreliable": 0}
def send(self, payload, reliable, now_ms):
self.seq += 1
self.stats["sent"] += 1
if reliable:
# Track it so a timer can retransmit if no ACK arrives.
self.unacked[self.seq] = (payload, now_ms, 1)
return self.seq
def ack(self, seq):
self.unacked.pop(seq, None)
def on_loss(self, seq, reliable):
if not reliable:
self.stats["dropped_unreliable"] += 1
return f"seq {seq}: lost, IGNORED (a newer update supersedes it)"
return f"seq {seq}: lost, awaiting the retransmission timer"
def tick(self, now_ms):
out = []
for seq, (payload, sent_at, attempts) in list(self.unacked.items()):
if now_ms - sent_at >= self.rto_ms:
self.unacked[seq] = (payload, now_ms, attempts + 1)
self.stats["retransmitted"] += 1
out.append(f"seq {seq}: RETRANSMIT (attempt {attempts + 1})")
return out
ch = PartiallyReliableChannel()
print(ch.on_loss(ch.send("pos:x=10,y=20", reliable=False, now_ms=0), reliable=False))
print(ch.on_loss(ch.send("pos:x=11,y=21", reliable=False, now_ms=16), reliable=False))
s = ch.send("event:picked_up_sword", reliable=True, now_ms=32)
print(ch.on_loss(s, reliable=True))
for msg in ch.tick(now_ms=150):
print(msg)
print(f"stats: {ch.stats}")
# seq 1: lost, IGNORED (a newer update supersedes it)
# seq 2: lost, IGNORED (a newer update supersedes it)
# seq 3: lost, awaiting the retransmission timer
# seq 3: RETRANSMIT (attempt 2)
# stats: {'sent': 3, 'retransmitted': 1, 'dropped_unreliable': 2}
# Forward error correction with simple XOR parity.
def xor_parity(blocks):
"""One parity block recovers any ONE lost block."""
p = bytearray(len(blocks[0]))
for b in blocks:
for i, byte in enumerate(b):
p[i] ^= byte
return bytes(p)
def recover(received, parity, missing_index, count):
"""Reconstruct the one missing block from the survivors and the parity."""
r = bytearray(parity)
for i, b in enumerate(received):
if i == missing_index or b is None:
continue
for j, byte in enumerate(b):
r[j] ^= byte
return bytes(r)
data = [b"AAAA", b"BBBB", b"CCCC", b"DDDD"]
parity = xor_parity(data)
print(f"\ndata blocks: {data}")
print(f"parity : {parity}")
received = list(data)
received[2] = None # block 2 lost in transit
rebuilt = recover(received, parity, 2, len(data))
print(f"block 2 lost -> reconstructed: {rebuilt} correct: {rebuilt == data[2]}")
print("recovered with ZERO round trips.")
# data blocks: [b'AAAA', b'BBBB', b'CCCC', b'DDDD']
# parity : b'\x04\x04\x04\x04'
# block 2 lost -> reconstructed: b'CCCC' correct: True
# recovered with ZERO round trips.
# FEC versus retransmission: the bandwidth/latency trade.
def recovery_cost(strategy, loss_rate, rtt_ms, parity_ratio=0.2):
if strategy == "retransmit":
return {"extra_bandwidth_pct": loss_rate * 100,
"added_latency_ms": rtt_ms,
"note": "pay only on loss, but pay a full round trip"}
return {"extra_bandwidth_pct": parity_ratio * 100,
"added_latency_ms": 0,
"note": "pay always, but never wait"}
print()
for strat in ("retransmit", "fec"):
c = recovery_cost(strat, loss_rate=0.02, rtt_ms=80)
print(f"{strat:<12} +{c['extra_bandwidth_pct']:>5.1f}% bandwidth "
f"+{c['added_latency_ms']:>3} ms latency {c['note']}")
# retransmit + 2.0% bandwidth + 80 ms latency pay only on loss, but pay a full round trip
# fec + 20.0% bandwidth + 0 ms latency pay always, but never wait
# The obligation that comes with using UDP (RFC 8085).
def rfc8085_check(sends_bulk_data, has_congestion_control):
if not sends_bulk_data:
return "OK: low-volume traffic (DNS-like); congestion control optional"
if has_congestion_control:
return "OK: bulk sender with congestion control"
return ("VIOLATION: bulk UDP without congestion control — unfair to TCP "
"flows and risks congestion collapse")
print()
for bulk, cc in [(False, False), (True, True), (True, False)]:
print(f"bulk={bulk!s:<5} cc={cc!s:<5} -> {rfc8085_check(bulk, cc)}")
# bulk=False cc=False -> OK: low-volume traffic (DNS-like); congestion control optional
# bulk=True cc=True -> OK: bulk sender with congestion control
# bulk=True cc=False -> VIOLATION: bulk UDP without congestion control — unfair to TCP flows and risks congestion collapse
# RTP carries the sequence numbers UDP does not provide:
$ sudo tcpdump -i eth0 -n udp portrange 16384-32767 -X -c 1
# 0x0000: 8008 1a2b 0001 d4c0 5b3c 7e91 ...
# ^^^^ V=2,PT=8 ^^^^ SEQUENCE NUMBER (application-level)
# ^^^^^^^^^ TIMESTAMP
# The application implements what it needs; UDP provides neither.
# RTCP reports loss back so the sender can adapt — feedback without reliability:
$ sudo tcpdump -i eth0 -n udp port 5005 -v -c 1
# IP 10.1.1.9.5005 > 10.1.1.5.5005: RTCP, Receiver Report
# fraction lost 3/256, cumulative lost 41, jitter 214
# ^^^^^^^ the sender lowers its bitrate; it does NOT retransmit
# QUIC: full reliability implemented in userspace over UDP:
$ curl --http3 -sw 'proto:%{http_version} time:%{time_total}\n' -o /dev/null \
https://cloudflare.com
# proto:3 time:0.148
# SRT — reliable UDP transport built for live video (partial reliability
# with a bounded latency budget):
$ ffmpeg -i input.mp4 -f mpegts "srt://receiver:9000?latency=120"
# ^^^^^^^^^^^ retransmit
# only within a 120 ms window; anything older is abandoned as useless.
# WireGuard: reliability is the INNER protocol's job, never the tunnel's:
$ sudo wg show
# interface: wg0
# listening port: 51820
# peer: abc123...
# transfer: 1.24 GiB received, 892.11 MiB sent
# ^ no retransmission counters exist — by design, there are none.
# Kernel counters for UDP loss the application must handle itself:
$ nstat -az | grep -i udp
# UdpInDatagrams 8412991
# UdpRcvbufErrors 2841 <- receiver too slow; DROPPED
# UdpSndbufErrors 12
# Nothing retransmits these. If it matters, the application must notice.
$ sudo sysctl -w net.core.rmem_max=26214400 # bigger buffers for bursty media
Key takeaways
- Choosing UDP moves reliability into the application, where it can be applied selectively.
- Building blocks: sequence numbers, selective acknowledgements, RTT-based timers, and forward error correction.
- FEC trades bandwidth for latency — parity packets let a receiver reconstruct losses with zero round trips, which TCP cannot do.
- Partial reliability lets critical messages be retransmitted while ephemeral updates are dropped, avoiding head-of-line blocking on data that did not matter.
- Other patterns: idempotent retry (DNS), supersession (telemetry), and full custom reliability (QUIC).
- RFC 8085 requires any UDP application sending more than trivial volumes to implement congestion control.
- Getting congestion control wrong risks congestion collapse and unfairness to TCP — prefer existing implementations (QUIC, DTLS, SRT, RTP/RTCP) over writing your own.
- RTCP demonstrates feedback without reliability: report loss so the sender adapts its bitrate rather than retransmitting.
🧪 Practice
- Name four mechanisms an application must implement to add reliability over UDP, and the one that has no TCP equivalent.
- A stream sends 10 data packets plus 2 parity packets. State how many losses are recoverable, the bandwidth cost, and the latency cost.
- Design a partial-reliability scheme for a game, classifying three message types as reliable or unreliable with justification.
- Interview-style: What obligation does RFC 8085 place on a UDP application, and what happens across the internet if many applications ignore it? (Hint: TCP flows back off when the network is congested — ask what happens to them if flows sharing the link do not.)
UDP Amplification Concerns
Theory
UDP's statelessness has a security consequence that has caused some of the largest denial-of-service attacks ever recorded. Understanding it requires two ingredients, and both must be present.
Ingredient 1: source address spoofing. UDP has no handshake, so a server has no way to verify that the source address in a request is genuine before responding. An attacker sends a request with the victim's address as the source, and the server dutifully sends its reply to the victim. The attacker's own address never appears. TCP is largely immune because the handshake requires the client to prove it can receive at the claimed address — a spoofed SYN produces a SYN-ACK sent to the victim, and the attacker never learns the sequence number needed to complete it.
Ingredient 2: an amplification factor. If the response is larger than the request, the attacker multiplies their bandwidth. Send 60 bytes, cause 3,000 bytes to be delivered to the victim, and you have a 50x amplifier. The attacker's modest uplink becomes a flood.
The combination is reflection plus amplification: the attacker's traffic is reflected off innocent third-party servers and magnified on the way.
Known amplification factors, from the US-CERT advisory and subsequent measurement:
| Protocol | Port | Amplification factor |
|---|---|---|
| memcached | 11211 | 10,000 – 51,000x |
NTP monlist | 123 | 556x |
| CharGen | 19 | 358x |
| DNS (ANY, DNSSEC) | 53 | 28 – 54x |
| SSDP | 1900 | 30x |
| SNMP v2 | 161 | 6x |
| QOTD | 17 | 140x |
The memcached figure is not a typo. A 15-byte request could elicit a megabyte of response, and in 2018 this produced the then-largest recorded DDoS — 1.35 Tb/s against GitHub — from an attacker with comparatively trivial bandwidth. The root cause was memcached shipping with UDP enabled and bound to all interfaces by default, exposing tens of thousands of servers to the internet.
The defences operate at three levels, and it matters that they are the responsibility of different parties:
Network operators: BCP 38 / RFC 2827 ingress filtering. Drop packets leaving your network whose source address does not belong to it. This kills spoofing at the source and would eliminate the entire attack class if universally deployed. It is not, because the operator who deploys it bears the cost while other networks receive the benefit — a textbook misaligned-incentive problem. MANRS (Chapter 6) promotes it as a baseline.
Service operators: do not expose amplifiers. Bind memcached to localhost, disable NTP monlist, disable open DNS recursion, and rate-limit responses. DNS RRL (Response Rate Limiting) caps identical responses to the same subnet, which specifically defeats reflection.
Protocol designers: require proof of address ownership before sending a large response. QUIC does this explicitly: during the handshake a server may not send more than three times the bytes it has received from an unvalidated address, and it can issue a retry token the client must echo. This caps amplification at 3x by design — a protocol-level fix rather than an operational one.
Examples
Reflection plus amplification: the attacker never touches the victim
ATTACKER REFLECTORS VICTIM
(1 Gb/s uplink) (open servers)
|
| src=VICTIM (spoofed)
| 60-byte request
+-------------------> [memcached] ---+
+-------------------> [memcached] ---+
+-------------------> [NTP] ---+---> 50 Gb/s of replies
+-------------------> [DNS] ---+ arrive at the victim
+-------------------> [SSDP] ---+
^ ^
these servers are behaving the victim sees traffic
CORRECTLY and have no idea from thousands of
they are being used legitimate servers
The attacker's address appears NOWHERE. Blocking the sources means
blocking innocent infrastructure.
Why TCP resists this
attacker sends SYN with src=VICTIM
server replies SYN-ACK ---> to the VICTIM (one small packet)
the attacker never sees the server's sequence number
-> cannot complete the handshake
-> no data is ever sent
Amplification: roughly 1x. Not worth the effort.
UDP has no such step. The first response is the full-size one.
QUIC's protocol-level fix
client sends 1200 bytes (padded deliberately to this minimum)
server MAY send at most 3 x 1200 = 3600 bytes to an unvalidated address
-> amplification capped at 3x BY THE PROTOCOL
server may also send a RETRY token the client must echo,
proving it can receive at the claimed address.
AMPLIFIERS = {
"memcached": (11211, 51000, "UDP enabled by default and bound to 0.0.0.0"),
"NTP monlist": (123, 556, "returns the last 600 clients seen"),
"CharGen": (19, 358, "a debugging service from 1983"),
"QOTD": (17, 140, "quote of the day"),
"DNS (ANY + DNSSEC)": (53, 54, "large signed responses to a tiny query"),
"SSDP": (1900, 30, "UPnP discovery, exposed by consumer routers"),
"SNMPv2": (161, 6, "GetBulk responses"),
}
def attack_output(uplink_mbps, factor):
return uplink_mbps * factor / 1000 # Gb/s
print(f"{'service':<22}{'port':>7}{'factor':>10}{'1 Gb/s becomes':>18}")
for name, (port, factor, _) in sorted(AMPLIFIERS.items(), key=lambda kv: -kv[1][1]):
print(f"{name:<22}{port:>7}{factor:>9,}x{attack_output(1000, factor):>15,.0f} Gb/s")
# service port factor 1 Gb/s becomes
# memcached 11211 51,000x 51,000 Gb/s
# NTP monlist 123 556x 556 Gb/s
# CharGen 19 358x 358 Gb/s
# QOTD 17 140x 140 Gb/s
# DNS (ANY + DNSSEC) 53 54x 54 Gb/s
# SSDP 1900 30x 30 Gb/s
# SNMPv2 161 6x 6 Gb/s
# The GitHub 2018 attack, reconstructed.
def github_2018(request_bytes=15, response_bytes=750_000, peak_gbps=1350):
factor = response_bytes / request_bytes
attacker_gbps = peak_gbps / factor
return {"amplification_factor": f"{factor:,.0f}x",
"observed_peak_gbps": peak_gbps,
"attacker_bandwidth_gbps": round(attacker_gbps, 3)}
print()
for k, v in github_2018().items():
print(f" {k:<28}{v}")
# amplification_factor 50,000x
# observed_peak_gbps 1350
# attacker_bandwidth_gbps 0.027
# 27 Mb/s of attacker uplink produced 1.35 Tb/s at the victim.
# Why TCP is not a useful amplifier.
def amplification(protocol, request_bytes, response_bytes):
if protocol == "TCP":
# Only the SYN-ACK is sent to an unvalidated address.
return 60 / request_bytes, "handshake blocks it: attacker cannot complete"
return response_bytes / request_bytes, "no handshake: full response sent immediately"
print()
for proto, req, resp in [("UDP", 60, 3000), ("TCP", 60, 3000)]:
f, why = amplification(proto, req, resp)
print(f"{proto}: {f:>6.1f}x {why}")
# UDP: 50.0x no handshake: full response sent immediately
# TCP: 1.0x handshake blocks it: attacker cannot complete
# BCP 38 ingress filtering: the fix that ends the attack class.
def bcp38(packet_src, network_prefix):
import ipaddress
inside = ipaddress.ip_address(packet_src) in ipaddress.ip_network(network_prefix)
if inside:
return True, "forwarded: source belongs to this network"
return False, f"DROPPED: {packet_src} is not within {network_prefix} — SPOOFED"
print()
for src in ("203.0.113.44", "198.51.100.9"):
ok, why = bcp38(src, "203.0.113.0/24")
print(f"src {src:<16}{why}")
# src 203.0.113.44 forwarded: source belongs to this network
# src 198.51.100.9 DROPPED: 198.51.100.9 is not within 203.0.113.0/24 — SPOOFED
# The incentive problem that keeps BCP 38 undeployed.
print()
for deployed in (False, True):
who_pays = "this network (config effort, hardware)" if deployed else "nobody here"
who_benefits = "OTHER networks (they stop receiving our spoofed traffic)" \
if deployed else "nobody"
print(f"BCP 38 deployed={deployed!s:<5} cost borne by: {who_pays:<40} "
f"benefit to: {who_benefits}")
# BCP 38 deployed=False cost borne by: nobody here benefit to: nobody
# BCP 38 deployed=True cost borne by: this network (config effort, hardware) benefit to: OTHER networks (they stop receiving our spoofed traffic)
# The deployer pays; everyone else benefits. This is why adoption stalls.
# QUIC's anti-amplification limit, enforced by the protocol itself.
def quic_limit(client_bytes_received, validated):
if validated:
return "unlimited (address validated by a retry token or handshake)"
return f"at most {3 * client_bytes_received} bytes (3x limit, RFC 9000)"
print()
for recv, val in [(1200, False), (1200, True)]:
print(f"client sent {recv} B, validated={val!s:<5} -> server may send "
f"{quic_limit(recv, val)}")
# client sent 1200 B, validated=False -> server may send at most 3600 bytes (3x limit, RFC 9000)
# client sent 1200 B, validated=True -> server may send unlimited (address validated by a retry token or handshake)
# Is this host an open amplifier? Check the usual suspects:
$ ss -uln | grep -E ":(11211|123|161|1900|19|17)\b"
# UNCONN 0 0 0.0.0.0:11211 0.0.0.0:* <- memcached on ALL interfaces:
# EXPOSED. Fix immediately.
# UNCONN 0 0 127.0.0.1:11211 0.0.0.0:* <- correct: localhost only
# Fix memcached — the single most dangerous default:
$ grep -E "^-U|^-l" /etc/memcached.conf
# -U 0 # disable UDP entirely
# -l 127.0.0.1 # bind to loopback only
$ sudo systemctl restart memcached
# Confirm NTP monlist is disabled:
$ ntpq -c monlist localhost 2>&1 | head -2
# localhost: timed out, nothing received
# ^^^^^^^^^^ good — the amplifier is closed
# In /etc/ntp.conf:
# restrict default kod nomodify notrap nopeer noquery limited
# DNS: do not run an open resolver, and rate-limit responses:
$ dig +short @203.0.113.10 google.com
# ;; connection timed out <- good: not answering strangers
# BIND response rate limiting:
# options {
# rate-limit { responses-per-second 10; window 5; };
# allow-recursion { 10.0.0.0/8; }; <- internal clients only
# };
# Verify BCP 38 on your own uplink — can you send a spoofed packet out?
$ sudo hping3 -c 1 -2 -a 198.51.100.99 -p 53 8.8.8.8
# ^^ a source address you do not own
# If this leaves your network, BCP 38 is NOT deployed upstream.
# Cisco's one-line implementation:
# interface GigabitEthernet0/1
# ip verify unicast source reachable-via rx <- uRPF strict mode
# Watch for being used as a reflector — traffic you never requested:
$ sudo tcpdump -i eth0 -n 'udp and src port 11211' -c 5
$ nstat -az | grep -i UdpOutDatagrams
# UdpOutDatagrams 8912481841
# ^^^^^^^^^^ an implausible rate here means
# you are the reflector, not the target.
# QUIC's 3x limit is not configurable — it is in the protocol:
$ sudo tcpdump -i eth0 -n 'udp port 443' -v -c 2
# IP 10.1.1.5.51234 > 104.16.132.229.443: UDP, length 1200
# ^^^^ clients PAD the
# initial packet to 1200 bytes precisely so the server's 3x budget is
# large enough to complete the handshake.
Key takeaways
- UDP amplification needs two ingredients: source address spoofing (possible because UDP has no handshake) and a response larger than the request.
- TCP is largely immune because the handshake forces the client to prove it can receive at the claimed address.
- Known factors: memcached 51,000x, NTP
monlist556x, CharGen 358x, DNS 28–54x, SSDP 30x. - The 2018 GitHub attack reached 1.35 Tb/s from roughly 27 Mb/s of attacker bandwidth, via memcached servers exposed by an unsafe default.
- Reflection also hides the attacker and makes source-based blocking useless, since the sources are innocent servers.
- BCP 38 / RFC 2827 ingress filtering would end the attack class, but deployment stalls because the deployer pays and others benefit.
- Service operators must not expose amplifiers: bind memcached to localhost, disable NTP
monlistand open recursion, and use DNS RRL. - QUIC caps amplification at 3x in the protocol — a server may not send more than three times the bytes received from an unvalidated address, which is why clients pad their initial packet to 1200 bytes.
🧪 Practice
- Name the two ingredients an amplification attack requires and explain why TCP supplies neither easily.
- Compute the traffic delivered to a victim by an attacker with 50 Mb/s of uplink using a 500x amplifier.
- Explain what BCP 38 does and why it remains widely undeployed despite being effective.
- Interview-style: QUIC caps a server's response at 3x the bytes received from an unvalidated client. Explain what this prevents and why clients pad their first packet to 1200 bytes. (Hint: work out what a server could send if the client's initial packet were only 60 bytes, and whether a handshake would fit.)
<a id="73-tcp-mechanics"></a>
7.3 TCP Mechanics
This subchapter covers how TCP actually works: the header, the handshakes that open and close a connection, the state machine behind them, and the sequence-number and retransmission machinery that turns an unreliable network into a reliable stream.
TCP Header Fields and Flags
Theory
TCP's header is 20 bytes minimum, 60 maximum, and every field exists to support one of reliability, ordering, flow control, or connection management. Reading it fluently is what makes packet captures interpretable.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-------------------------------+-------------------------------+
| Source Port | Destination Port |
+-------------------------------+-------------------------------+
| Sequence Number |
+---------------------------------------------------------------+
| Acknowledgment Number |
+-------+-----------+-----------+-------------------------------+
| Data | Reserved | Flags | Window |
| Offset| | | |
+-------+-----------+-----------+-------------------------------+
| Checksum | Urgent Pointer |
+-------------------------------+-------------------------------+
| Options (0-40 bytes) | Padding |
+---------------------------------------------------+-----------+
| Field | Size | Purpose |
|---|---|---|
| Source / Destination Port | 16 bits each | Demultiplexing (7.1) |
| Sequence Number | 32 bits | Byte offset of this segment's first byte |
| Acknowledgment Number | 32 bits | The next byte expected, not the last received |
| Data Offset | 4 bits | Header length in 32-bit words; 5 (20 B) to 15 (60 B) |
| Flags | 8 bits | Control bits — see below |
| Window | 16 bits | Receiver's free buffer space (flow control, 7.4) |
| Checksum | 16 bits | Covers header, data, and a pseudo-header |
| Urgent Pointer | 16 bits | Offset of urgent data; effectively obsolete |
| Options | 0–40 bytes | MSS, window scale, SACK, timestamps |
The flags, and what each actually means:
| Flag | Name | Meaning |
|---|---|---|
| SYN | Synchronize | Open a connection; consumes one sequence number |
| ACK | Acknowledge | The Acknowledgment field is valid — set on every segment after the initial SYN |
| FIN | Finish | No more data from me; consumes one sequence number |
| RST | Reset | Abort immediately — no graceful close |
| PSH | Push | Deliver to the application now rather than buffering |
| URG | Urgent | Urgent pointer is valid; obsolete and a security hazard |
| ECE / CWR | ECN Echo / Congestion Window Reduced | Explicit Congestion Notification (7.5) |
| NS | Nonce Sum | Experimental ECN protection; unused |
Three points about this header repay attention.
The Acknowledgment Number is the next byte expected, not the last byte received. If a receiver has bytes 1–1000, it acknowledges 1001. Getting this off by one is the most common error when reading captures by hand.
SYN and FIN each consume a sequence number despite carrying no data. This is what allows them to be acknowledged and therefore retransmitted reliably — without it, a lost FIN could never be detected.
The Data Offset field exists because of options, exactly as IPv4's IHL does (Chapter 5). Unlike IPv4's options, however, TCP's options are essential and universally used — window scaling, SACK, and timestamps are all options, and a modern connection without them performs badly. The difference is that TCP options are processed only by the endpoints, never by transit routers, so they never cost a fast-path punt.
The checksum, like UDP's, covers a pseudo-header containing the IP addresses, for the same end-to-end reason (7.2). In TCP it is mandatory in both IPv4 and IPv6.
Examples
A real SYN packet, byte by byte
c8 32 01 bb a1 b2 c3 d4 00 00 00 00 a0 02 fa f0 ...
\___/ \___/ \_________/ \_________/ \_/ \/ \___/
| | | | | | |
| | | | | | window = 0xfaf0 = 64240
| | | | | flags = 0x02 = SYN
| | | | data offset = 0xa = 10 words = 40 bytes
| | | | (20 base + 20 options)
| | | ack = 0 (no ACK flag set on the initial SYN)
| | seq = 0xa1b2c3d4 (the ISN, randomly chosen)
| dst port = 0x01bb = 443
src port = 0xc832 = 51250
The flags byte
bit: 7 6 5 4 3 2 1 0
+----+----+----+----+----+----+----+----+
|CWR | ECE|URG | ACK| PSH| RST| SYN| FIN|
+----+----+----+----+----+----+----+----+
0x02 = 00000010 = SYN (connection request)
0x12 = 00010010 = SYN+ACK (connection accepted)
0x10 = 00010000 = ACK (ordinary data/ack)
0x18 = 00011000 = PSH+ACK (data, deliver promptly)
0x11 = 00010001 = FIN+ACK (graceful close)
0x04 = 00000100 = RST (abort)
ACK means "next byte expected", not "last byte received"
sender transmits bytes 1..1000 (seq=1, length=1000)
receiver has 1..1000 complete
receiver sends ACK = 1001
^^^^ "send me 1001 next", NOT "I got 1000"
import struct
TCP_FLAGS = [("FIN", 0x01), ("SYN", 0x02), ("RST", 0x04), ("PSH", 0x08),
("ACK", 0x10), ("URG", 0x20), ("ECE", 0x40), ("CWR", 0x80)]
def decode_flags(flags):
return [name for name, bit in TCP_FLAGS if flags & bit] or ["(none)"]
def parse_tcp(raw: bytes):
(src, dst, seq, ack, offset_reserved, flags,
window, csum, urg) = struct.unpack("!HHIIBBHHH", raw[:20])
data_offset = (offset_reserved >> 4) * 4 # high nibble, in 32-bit words
return {
"src_port": src, "dst_port": dst,
"seq": seq, "ack": ack,
"header_bytes": data_offset,
"options_bytes": data_offset - 20,
"flags": "+".join(decode_flags(flags)),
"flags_hex": f"0x{flags:02x}",
"window": window,
"checksum": f"0x{csum:04x}",
}
syn = struct.pack("!HHIIBBHHH", 51250, 443, 0xa1b2c3d4, 0,
0xa0, 0x02, 64240, 0x1234, 0)
for k, v in parse_tcp(syn).items():
print(f"{k:<16}{v}")
# src_port 51250
# dst_port 443
# seq 2712847316
# ack 0
# header_bytes 40
# options_bytes 20
# flags SYN
# flags_hex 0x02
# window 64240
# checksum 0x1234
# The common flag combinations, and what each means in a capture.
print()
for value in (0x02, 0x12, 0x10, 0x18, 0x11, 0x04, 0x14):
names = "+".join(decode_flags(value))
meaning = {
0x02: "connection request",
0x12: "connection accepted",
0x10: "pure acknowledgement",
0x18: "data, deliver promptly",
0x11: "graceful close",
0x04: "abort — no graceful close",
0x14: "abort in response to something invalid",
}[value]
print(f"0x{value:02x} {names:<12}{meaning}")
# 0x02 SYN connection request
# 0x12 SYN+ACK connection accepted
# 0x10 ACK pure acknowledgement
# 0x18 PSH+ACK data, deliver promptly
# 0x11 FIN+ACK graceful close
# 0x04 RST abort — no graceful close
# 0x14 RST+ACK abort in response to something invalid
# ACK semantics: the off-by-one that trips everyone.
def next_ack(received_start, received_length):
"""ACK is the NEXT byte expected."""
return received_start + received_length
print()
for start, length in [(1, 1000), (1001, 1448), (1, 1)]:
print(f"received bytes {start}..{start+length-1} -> ACK {next_ack(start, length)}")
# received bytes 1..1000 -> ACK 1001
# received bytes 1001..2448 -> ACK 2449
# received bytes 1..1 -> ACK 2
# SYN and FIN consume a sequence number despite carrying no data.
def sequence_consumed(flags, payload_len):
consumed = payload_len
if flags & 0x02: consumed += 1 # SYN
if flags & 0x01: consumed += 1 # FIN
return consumed
print()
for flags, payload, label in [(0x02, 0, "SYN (no data)"),
(0x10, 1448, "ACK with 1448 bytes"),
(0x11, 0, "FIN+ACK (no data)"),
(0x10, 0, "pure ACK (no data)")]:
c = sequence_consumed(flags, payload)
note = " <- consumes a seq so it can be ACKed and retransmitted" if c > payload else ""
print(f"{label:<24}consumes {c} sequence number(s){note}")
# SYN (no data) consumes 1 sequence number(s) <- consumes a seq so it can be ACKed and retransmitted
# ACK with 1448 bytes consumes 1448 sequence number(s)
# FIN+ACK (no data) consumes 1 sequence number(s) <- consumes a seq so it can be ACKed and retransmitted
# pure ACK (no data) consumes 0 sequence number(s)
# Header size with the options a modern connection actually uses.
OPTIONS = {"MSS": 4, "Window Scale": 3, "SACK permitted": 2, "Timestamps": 10,
"NOP padding": 1}
print()
total = 20
for name, size in OPTIONS.items():
total += size
print(f" + {name:<16}{size:>3} bytes running total {total}")
padded = total + (-total % 4)
print(f" padded to a 4-byte boundary: {padded} bytes "
f"-> Data Offset = {padded // 4}")
# + MSS 4 bytes running total 24
# + Window Scale 3 bytes running total 27
# + SACK permitted 2 bytes running total 29
# + Timestamps 10 bytes running total 39
# + NOP padding 1 bytes running total 40
# padded to a 4-byte boundary: 40 bytes -> Data Offset = 10
# Read the header on a live capture:
$ sudo tcpdump -i eth0 -n 'tcp[tcpflags] & tcp-syn != 0' -v -c 1
# IP (tos 0x0, ttl 64, id 41394, offset 0, flags [DF], proto TCP (6), length 60)
# 10.1.1.5.51250 > 93.184.216.34.443: Flags [S], seq 2712847316,
# ^^^ SYN ^^^^^^^^^^ the ISN
# win 64240, options [mss 1460,sackOK,TS val 184221 ecr 0,nop,wscale 7],
# ^^^^^^^^ the options a modern connection negotiates
# length 0
# Filter by flag combination — tcpdump exposes the flags byte directly:
$ sudo tcpdump -i eth0 'tcp[tcpflags] == tcp-syn' # SYN only
$ sudo tcpdump -i eth0 'tcp[tcpflags] & (tcp-syn|tcp-ack) == (tcp-syn|tcp-ack)'
$ sudo tcpdump -i eth0 'tcp[tcpflags] & tcp-rst != 0' # resets
# Absolute vs relative sequence numbers — tcpdump shows relative by default:
$ sudo tcpdump -i eth0 -n -S 'tcp port 443' -c 2
# Flags [S], seq 2712847316 <- -S shows the ABSOLUTE number
$ sudo tcpdump -i eth0 -n 'tcp port 443' -c 2
# Flags [S], seq 0 <- default: relative to the ISN,
# much easier to read
# Watch the ACK track the data:
$ sudo tcpdump -i eth0 -n 'tcp port 443' -c 4
# IP 10.1.1.5.51250 > 93.184.216.34.443: Flags [P.], seq 1:1449, ack 1, length 1448
# ^^^^^^ bytes 1..1448
# IP 93.184.216.34.443 > 10.1.1.5.51250: Flags [.], ack 1449, length 0
# ^^^^^^^^ next byte expected
# RST: the abort path, and why you see it:
$ nc -w1 127.0.0.1 9999
$ sudo tcpdump -i lo -n 'tcp[tcpflags] & tcp-rst != 0' -c 1
# IP 127.0.0.1.9999 > 127.0.0.1.51234: Flags [R.], seq 0, ack 1, win 0, length 0
# ^^^ nothing is listening
# The options in use on an established connection:
$ ss -tin state established | head -3
# ESTAB 0 0 10.1.1.5:51250 93.184.216.34:443
# cubic wscale:7,7 rto:204 rtt:3.821/1.242 mss:1448 cwnd:10
# ^^^^^^^^^^ window scaling negotiated in BOTH directions (7.4)
Key takeaways
- TCP's header is 20 bytes minimum, 60 maximum; Data Offset counts 32-bit words and exists because of options.
- Sequence Number is the byte offset of the segment's first byte; Acknowledgment Number is the NEXT byte expected, not the last received.
- SYN and FIN each consume one sequence number despite carrying no data, so they can be acknowledged and retransmitted.
- Flags: SYN (open), ACK (ack field valid — set on everything after the first SYN), FIN (graceful close), RST (abort), PSH, URG (obsolete), ECE/CWR (ECN).
- Common values:
0x02SYN,0x12SYN+ACK,0x10ACK,0x18PSH+ACK,0x11FIN+ACK,0x04RST. - TCP options are essential and universal (MSS, window scale, SACK, timestamps) — unlike IPv4 options, they are processed only by endpoints, so they never cost a fast-path punt.
- A typical modern SYN carries 20 bytes of options, giving a 40-byte header and Data Offset 10.
- The checksum covers a pseudo-header with the IP addresses, and is mandatory in both IPv4 and IPv6.
🧪 Practice
- Decode flags bytes
0x12,0x11, and0x04into flag names and state what each represents in a connection's lifecycle.- A receiver has bytes 1 through 2896. Give the Acknowledgment Number it sends and explain the off-by-one.
- Explain why SYN and FIN consume sequence numbers when they carry no data.
- Interview-style: IPv4 options are avoided because they force packets onto the router slow path, yet TCP options are used on every connection. Why is there no equivalent penalty? (Hint: ask which devices on the path are required to parse each kind of option.)
Three-Way Handshake
Theory
Before any data moves, TCP establishes a connection with a three-way handshake. Its purpose is not merely to say hello — it accomplishes four specific things:
- Confirm both parties are present and willing to communicate.
- Synchronise sequence numbers in both directions. TCP is full duplex, so each side has its own independent sequence space and each must learn the other's starting point.
- Negotiate options — MSS, window scaling, SACK, timestamps.
- Prove bidirectional reachability, which is what defeats blind spoofing.
The exchange:
CLIENT SERVER
| |
|--- SYN, seq=x -------------------------->| "I want to talk; my ISN is x"
| |
|<-- SYN+ACK, seq=y, ack=x+1 --------------| "Agreed; my ISN is y,
| | and I got your x"
|--- ACK, seq=x+1, ack=y+1 --------------->| "I got your y"
| |
|============ ESTABLISHED =================|
Why three messages and not two? Because sequence numbers must be synchronised in both directions, and each direction's ISN needs its own acknowledgement. The server's SYN and its ACK of the client's SYN are combined into one segment (the SYN-ACK), which is why three suffices rather than four.
The ISN (Initial Sequence Number) must be unpredictable, and this is a security requirement rather than an aesthetic one. Early implementations used a simple counter, which allowed blind spoofing: an attacker who could predict the ISN could inject data into a connection without ever seeing it, by forging packets with the right sequence numbers. RFC 6528 specifies generating the ISN from a cryptographic hash of the 4-tuple and a secret, combined with a timer. This is also the mechanism behind TCP sequence prediction attacks, historically used in the Mitnick attack.
The handshake is also where MSS (Maximum Segment Size) is agreed. Each side advertises the largest segment it is willing to receive; the effective MSS for the connection is the minimum of the two, further constrained by path MTU (Chapter 5). On standard Ethernet: 1500 MTU − 20 IP − 20 TCP = 1460, or 1448 when timestamps are enabled.
The handshake is the target of the SYN flood, which exploits a specific asymmetry: the server must allocate state when it receives a SYN and sends its SYN-ACK, while the attacker allocates nothing and never completes the handshake. Spoofed source addresses mean the final ACK never comes, and the SYN queue (7.1) fills with half-open connections until legitimate clients are refused.
SYN cookies are the standard defence, and the trick is elegant: instead of allocating state, the server encodes the connection parameters into the ISN it sends, computed as a cryptographic hash of the 4-tuple, a timestamp, and a secret. It then forgets everything. If a genuine ACK arrives, its acknowledgment number contains that ISN + 1, so the server can reconstruct the connection state from the ACK alone. No state, no queue to fill, no flood. The cost is that some options cannot be preserved, so cookies are enabled only under attack.
Examples
The handshake, with what each message accomplishes
CLIENT SERVER
CLOSED LISTEN
|
|--- SYN seq=1000, win=64240, MSS=1460, WS=7 ----->|
| "my ISN is 1000; here are my options" |
SYN_SENT SYN_RECEIVED
| |
|<-- SYN+ACK seq=5000, ack=1001, MSS=1448, WS=7 ---|
| "my ISN is 5000; I got your 1000 (ack=1001); |
| my options" |
| |
|--- ACK seq=1001, ack=5001 ---------------------->|
| "I got your 5000" |
ESTABLISHED ESTABLISHED
Note: ack=1001 acknowledges the SYN, which consumed sequence 1000.
Why three and not two
TWO messages would synchronise only ONE direction:
client -> server: "my ISN is x"
server -> client: "ok, and my ISN is y"
...but the SERVER never learns that the client received y.
A lost second message would leave the server believing the
connection is open while the client knows nothing.
The third message closes that gap: it acknowledges y.
SYN flood and SYN cookies
NORMAL:
SYN --> server ALLOCATES state, sends SYN-ACK, waits in SYN queue
ACK --> promoted to the accept queue
FLOOD (spoofed sources):
SYN --> allocate | SYN-ACK goes to a victim/nowhere
SYN --> allocate | no ACK ever returns
SYN --> allocate | ... SYN queue FILLS
legitimate SYN --> REFUSED
WITH SYN COOKIES:
SYN --> compute ISN = hash(4-tuple, time, secret) | encoded MSS
send SYN-ACK with that ISN
ALLOCATE NOTHING. Forget the connection entirely.
ACK --> ack_number - 1 == the cookie we would have generated?
yes -> RECONSTRUCT state and establish
no -> ignore
The queue cannot fill because nothing is queued.
import hashlib, struct, time
def generate_isn(src_ip, src_port, dst_ip, dst_port, secret, now=None):
"""RFC 6528: ISN = timer + hash(4-tuple, secret). Must be UNPREDICTABLE."""
now = now if now is not None else time.time()
timer = int(now * 250000) & 0xFFFFFFFF # ~4 us tick
key = f"{src_ip}:{src_port}:{dst_ip}:{dst_port}".encode()
h = hashlib.sha256(key + secret).digest()
return (timer + int.from_bytes(h[:4], "big")) & 0xFFFFFFFF
secret = b"kernel-secret"
a = generate_isn("10.1.1.5", 51250, "93.184.216.34", 443, secret, now=1000.0)
b = generate_isn("10.1.1.5", 51251, "93.184.216.34", 443, secret, now=1000.0)
print(f"ISN for src port 51250: {a}")
print(f"ISN for src port 51251: {b}")
print(f"difference: {abs(a - b):,} — unpredictable, so blind injection fails")
# ISN for src port 51250: 2891961612
# ISN for src port 51251: 4258496993
# difference: 1,366,535,381 — unpredictable, so blind injection fails
# The handshake as a state exchange.
class Handshake:
def __init__(self, role):
self.role = role
self.state = "CLOSED" if role == "client" else "LISTEN"
self.snd_nxt = None # my next sequence number
self.rcv_nxt = None # the next byte I expect from the peer
def send_syn(self, isn):
self.snd_nxt = isn
self.state = "SYN_SENT"
return {"flags": "SYN", "seq": isn, "ack": None}
def recv_syn(self, seg, my_isn):
self.rcv_nxt = seg["seq"] + 1 # the SYN consumed one sequence
self.snd_nxt = my_isn
self.state = "SYN_RECEIVED"
return {"flags": "SYN+ACK", "seq": my_isn, "ack": self.rcv_nxt}
def recv_synack(self, seg):
self.rcv_nxt = seg["seq"] + 1
self.snd_nxt += 1
self.state = "ESTABLISHED"
return {"flags": "ACK", "seq": self.snd_nxt, "ack": self.rcv_nxt}
def recv_ack(self, seg):
self.snd_nxt += 1
self.state = "ESTABLISHED"
return None
client, server = Handshake("client"), Handshake("server")
print()
m1 = client.send_syn(1000); print(f" 1. C->S {m1} client={client.state}")
m2 = server.recv_syn(m1, my_isn=5000); print(f" 2. S->C {m2} server={server.state}")
m3 = client.recv_synack(m2); print(f" 3. C->S {m3} client={client.state}")
server.recv_ack(m3); print(f" server={server.state}")
# 1. C->S {'flags': 'SYN', 'seq': 1000, 'ack': None} client=SYN_SENT
# 2. S->C {'flags': 'SYN+ACK', 'seq': 5000, 'ack': 1001} server=SYN_RECEIVED
# 3. C->S {'flags': 'ACK', 'seq': 1001, 'ack': 5001} client=ESTABLISHED
# server=ESTABLISHED
# MSS negotiation: the minimum of the two advertisements, capped by path MTU.
def effective_mss(client_mss, server_mss, path_mtu, timestamps=True):
overhead = 20 + 20 + (12 if timestamps else 0) # IP + TCP + options
from_mtu = path_mtu - overhead
return min(client_mss, server_mss, from_mtu)
print()
for cmss, smss, mtu, ts in [(1460, 1460, 1500, False), (1460, 1460, 1500, True),
(1460, 1460, 1400, True), (9000, 1460, 9000, False)]:
print(f"client {cmss:>4}, server {smss:>4}, MTU {mtu:>4}, timestamps={ts!s:<5}"
f"-> effective MSS {effective_mss(cmss, smss, mtu, ts)}")
# client 1460, server 1460, MTU 1500, timestamps=False-> effective MSS 1460
# client 1460, server 1460, MTU 1500, timestamps=True -> effective MSS 1448
# client 1460, server 1460, MTU 1400, timestamps=True -> effective MSS 1348
# client 9000, server 1460, MTU 9000, timestamps=False-> effective MSS 1460
# Note the last case: jumbo frames are useless if the peer will not accept them.
# SYN cookies: state encoded into the ISN itself.
def make_cookie(src_ip, src_port, dst_ip, dst_port, mss, counter, secret):
key = f"{src_ip}:{src_port}:{dst_ip}:{dst_port}:{counter}".encode()
h = int.from_bytes(hashlib.sha256(key + secret).digest()[:3], "big")
mss_index = {536: 0, 1300: 1, 1440: 2, 1460: 3}.get(mss, 3)
# 5 bits counter | 3 bits MSS index | 24 bits hash
return ((counter & 0x1F) << 27) | (mss_index << 24) | h
def verify_cookie(ack_number, src_ip, src_port, dst_ip, dst_port, secret, counter):
cookie = ack_number - 1 # the ACK is ISN + 1
mss_index = (cookie >> 24) & 0x7
for mss in (536, 1300, 1440, 1460):
expected = make_cookie(src_ip, src_port, dst_ip, dst_port, mss, counter, secret)
if expected == cookie:
return True, mss
return False, None
c = make_cookie("10.1.1.5", 51250, "93.184.216.34", 443, 1460, counter=7, secret=secret)
ok, mss = verify_cookie(c + 1, "10.1.1.5", 51250, "93.184.216.34", 443, secret, 7)
print(f"\ncookie ISN sent : {c}")
print(f"client's ACK : {c + 1}")
print(f"reconstructed : valid={ok}, MSS={mss} — with ZERO server state held")
# cookie ISN sent : 993487299
# client's ACK : 993487300
# reconstructed : valid=True, MSS=1460 — with ZERO server state held
bad, _ = verify_cookie(12345, "10.9.9.9", 40000, "93.184.216.34", 443, secret, 7)
print(f"forged ACK : valid={bad} — a spoofed ACK cannot be reconstructed")
# forged ACK : valid=False — a spoofed ACK cannot be reconstructed
# The asymmetry a SYN flood exploits.
def syn_flood_cost(syns_per_sec, state_bytes=304, queue_size=1024):
return {
"attacker_bytes_per_syn": 60,
"server_bytes_per_syn": state_bytes,
"asymmetry": round(state_bytes / 60, 1),
"seconds_to_fill_queue": round(queue_size / syns_per_sec, 4),
}
print()
for rate in (1_000, 100_000):
c = syn_flood_cost(rate)
print(f"{rate:>7,} SYN/s: server pays {c['asymmetry']}x the attacker, "
f"queue full in {c['seconds_to_fill_queue']} s")
# 1,000 SYN/s: server pays 5.1x the attacker, queue full in 1.024 s
# 100,000 SYN/s: server pays 5.1x the attacker, queue full in 0.0102 s
# Watch a complete handshake:
$ sudo tcpdump -i eth0 -n 'tcp port 443 and tcp[tcpflags] & (tcp-syn|tcp-ack) != 0' -c 3
# IP 10.1.1.5.51250 > 93.184.216.34.443: Flags [S], seq 2712847316,
# win 64240, options [mss 1460,sackOK,TS val 184221 ecr 0,nop,wscale 7]
# IP 93.184.216.34.443 > 10.1.1.5.51250: Flags [S.], seq 891204471, ack 2712847317,
# ^^^^^^^^^^ = ISN+1
# win 65535, options [mss 1448,sackOK,TS val 99812 ecr 184221,nop,wscale 9]
# IP 10.1.1.5.51250 > 93.184.216.34.443: Flags [.], ack 891204472, win 502
# Measure handshake time in isolation:
$ curl -w 'dns:%{time_namelookup}s tcp:%{time_connect}s\n' -o /dev/null -s https://example.com
# dns:0.021s tcp:0.110s
# ^^^^^ the handshake cost one RTT
# SYN flood defences:
$ sysctl net.ipv4.tcp_syncookies net.ipv4.tcp_max_syn_backlog net.ipv4.tcp_synack_retries
# net.ipv4.tcp_syncookies = 1 # 1 = enable only WHEN the queue overflows
# net.ipv4.tcp_max_syn_backlog = 4096
# net.ipv4.tcp_synack_retries = 5 # lower this to give up on half-opens sooner
# Detect a flood in progress:
$ ss -tan state syn-recv | wc -l
# 4092 <- SYN queue essentially full
$ nstat -az | grep -iE "syncookies|TCPReqQFullDo"
# TcpExtSyncookiesSent 184221 <- cookies are being issued: attack
# TcpExtSyncookiesRecv 412 <- genuine clients completing
# TcpExtSyncookiesFailed 183809 <- forged/aborted: the flood
# TcpExtTCPReqQFullDoCookies 184221
# ISN randomisation — verify it is not predictable:
$ for i in 1 2 3; do
> sudo hping3 -S -p 443 -c 1 93.184.216.34 2>&1 | grep -o "seq=[0-9]*"
> done
# seq=1327349445
# seq=3918204471
# seq=712094881
# Large unpredictable gaps. A simple counter here would enable blind injection.
# MSS on an established connection:
$ ss -tin state established | grep -o "mss:[0-9]*" | head -3
# mss:1448
# ^^^^ 1500 - 20 (IP) - 20 (TCP) - 12 (timestamps)
# TCP Fast Open sends data IN the SYN, removing the handshake RTT:
$ sysctl net.ipv4.tcp_fastopen
# net.ipv4.tcp_fastopen = 1 # 1=client, 2=server, 3=both
$ sudo tcpdump -i eth0 -n 'tcp[tcpflags] & tcp-syn != 0' -v | grep -i "tfo\|length [1-9]"
# Flags [S], ... options [... tfo cookie 1a2b3c4d ...], length 24
# ^^^ data carried in the SYN itself
Key takeaways
- The three-way handshake confirms willingness, synchronises sequence numbers in both directions, negotiates options, and proves bidirectional reachability.
- Three messages suffice because the server combines its SYN and its ACK into one segment; two would leave one direction unacknowledged.
- The ISN must be unpredictable (RFC 6528, a keyed hash of the 4-tuple) — a predictable ISN enables blind spoofing and injection.
ack = ISN + 1because the SYN consumes one sequence number.- MSS is the minimum of both advertisements, further capped by path MTU: 1460 on Ethernet, 1448 with timestamps enabled.
- A SYN flood exploits the asymmetry that the server allocates state on the SYN while the attacker allocates nothing and never completes.
- SYN cookies encode the connection state into the ISN itself, so the server holds no state and the queue cannot fill; the cost is losing some options, so they engage only under attack.
- TCP Fast Open carries data in the SYN to eliminate the handshake round trip for repeat connections.
🧪 Practice
- Draw the three-way handshake with ISNs 3000 (client) and 9000 (server), labelling the sequence and acknowledgment number in each segment.
- Explain why two messages are insufficient, identifying exactly what the third message guarantees.
- Compute the effective MSS for a 1500-byte path MTU with timestamps enabled, and again for a 1400-byte MTU.
- Interview-style: How do SYN cookies let a server survive a flood without a SYN queue? (Hint: ask where the connection state goes if the server does not store it, and what the client's final ACK necessarily contains.)
Connection Termination and TIME_WAIT
Theory
Closing a TCP connection takes four messages rather than three, because TCP connections are full duplex — two independent data streams that must be shut down separately. Each direction is closed with its own FIN and its own ACK.
CLIENT SERVER
|--- FIN, seq=u ------------------------->| "I have no more data"
|<-- ACK, ack=u+1 -----------------------| "understood"
| (server may still send data) |
|<-- FIN, seq=v -------------------------| "I have no more data either"
|--- ACK, ack=v+1 ---------------------->| "understood"
| |
TIME_WAIT (2 x MSL) CLOSED
The gap between the first ACK and the server's FIN is the half-close state: the client has stopped sending but the server may continue. This is a genuine feature, used by protocols where a client sends a request, signals "that is all", and then reads a response of unknown length. In practice the two middle messages are often combined into a single FIN+ACK, making the exchange look like three.
The side that closes first — usually the client — enters TIME_WAIT and remains there for 2 × MSL (Maximum Segment Lifetime), conventionally 60 seconds on Linux (2 × 30 s). This state surprises people, because the connection is finished yet the socket persists.
TIME_WAIT exists for two reasons, and both are real:
1. Ensure the final ACK arrives. If the client's last ACK is lost, the server retransmits its FIN. A client in TIME_WAIT can re-acknowledge it. A client that had gone straight to CLOSED would instead answer with an RST, which the server would report as an error on an otherwise clean close.
2. Let old duplicate segments expire. Suppose the same 4-tuple is reused immediately for a new connection. A delayed segment from the old connection could arrive and, if its sequence number happened to fall within the new connection's window, be accepted as valid data. Waiting 2 × MSL guarantees every segment from the old incarnation has died in the network.
The operational consequence is significant. A busy server or proxy that initiates many short-lived connections accumulates TIME_WAIT sockets, each holding a 4-tuple for 60 seconds. Combined with a limited ephemeral port range (7.1), this causes port exhaustion.
The responses, in order of preference:
| Approach | Effect | Caution |
|---|---|---|
| Connection reuse / keep-alive | Eliminates the problem at its root | Always the right first answer |
SO_REUSEADDR | Allows binding a port in TIME_WAIT | Standard for servers restarting |
tcp_tw_reuse | Reuses TIME_WAIT sockets for outbound connections | Safe with timestamps enabled |
tcp_max_tw_buckets | Caps the total; excess is killed | A blunt instrument |
tcp_tw_recycle | Removed from Linux 4.12 | Broke NAT badly; never use |
Note the last row. tcp_tw_recycle used per-host timestamp tracking, which failed catastrophically when multiple clients shared one NAT address — connections from some clients were silently dropped. It was removed, and advice recommending it is obsolete and harmful.
The other termination path is RST, which aborts immediately: no FIN exchange, no TIME_WAIT, any unread data discarded. A reset is sent when there is no listener on the port, when data arrives for a closed connection, or when an application sets SO_LINGER with a zero timeout. Seeing many RSTs usually means applications are closing connections with unread data pending, or a middlebox is tearing down idle connections.
Examples
Four-way close, and the half-close in the middle
CLIENT SERVER
ESTABLISHED ESTABLISHED
|--- FIN seq=1000 ---------------------->|
FIN_WAIT_1 CLOSE_WAIT
|<-- ACK ack=1001 ----------------------|
FIN_WAIT_2 |
| | <- HALF-CLOSE:
|<========= server may STILL SEND ========| client is done sending,
| | server is not
|<-- FIN seq=5000 ----------------------|
TIME_WAIT LAST_ACK
|--- ACK ack=5001 --------------------->|
| CLOSED
| wait 2 x MSL (60 s on Linux)
CLOSED
Why TIME_WAIT exists — reason 1: the final ACK may be lost
client --- ACK --X (lost)
server: "no ACK for my FIN" -> RETRANSMITS FIN
client in TIME_WAIT: re-sends the ACK. Clean close.
client already CLOSED: replies RST. Server logs an error.
reason 2: old duplicates must die
old connection on 4-tuple T ... segment delayed in the network
connection closes; SAME 4-tuple T reused immediately
delayed segment finally arrives
-> if its sequence number falls in the new window, it is ACCEPTED
-> DATA CORRUPTION
waiting 2 x MSL guarantees no segment from the old incarnation survives.
TIME_WAIT accumulation on a busy proxy
1,000 outbound connections/sec x 60 s = 60,000 sockets in TIME_WAIT
ephemeral range (default) = 28,232 ports
-> EXHAUSTED. "Cannot assign requested address".
The fix is connection reuse, not tuning.
MSL = 30 # Maximum Segment Lifetime, seconds (Linux convention)
def timewait_duration():
return 2 * MSL
def timewait_sockets(conns_per_sec, duration=None):
return conns_per_sec * (duration or timewait_duration())
def port_exhaustion(conns_per_sec, port_lo=32768, port_hi=60999):
available = port_hi - port_lo + 1
held = timewait_sockets(conns_per_sec)
return {"held": held, "available": available, "exhausted": held > available,
"max_safe_cps": available // timewait_duration()}
print(f"TIME_WAIT duration: 2 x MSL = {timewait_duration()} s\n")
for cps in (100, 470, 1000):
e = port_exhaustion(cps)
status = "EXHAUSTED" if e["exhausted"] else "ok"
print(f"{cps:>5,} conn/s -> {e['held']:>7,} TIME_WAIT sockets vs "
f"{e['available']:,} ports: {status}")
print(f"\nmaximum safe rate to ONE destination: {port_exhaustion(1)['max_safe_cps']} conn/s")
# TIME_WAIT duration: 2 x MSL = 60 s
#
# 100 conn/s -> 6,000 TIME_WAIT sockets vs 28,232 ports: ok
# 470 conn/s -> 28,200 TIME_WAIT sockets vs 28,232 ports: ok
# 1,000 conn/s -> 60,000 TIME_WAIT sockets vs 28,232 ports: EXHAUSTED
#
# maximum safe rate to ONE destination: 470 conn/s
# What happens without TIME_WAIT.
def final_ack_lost(in_timewait):
if in_timewait:
return "server retransmits FIN -> client re-ACKs -> CLEAN CLOSE"
return "server retransmits FIN -> client sends RST -> server logs an ERROR"
print()
for tw in (True, False):
print(f"client in TIME_WAIT={tw!s:<5} -> {final_ack_lost(tw)}")
# client in TIME_WAIT=True -> server retransmits FIN -> client re-ACKs -> CLEAN CLOSE
# client in TIME_WAIT=False -> server retransmits FIN -> client sends RST -> server logs an ERROR
def old_duplicate(delay_s, reuse_after_s):
if delay_s > reuse_after_s:
return ("HAZARD: a delayed segment from the old connection can land "
"inside the new connection's window")
return "safe: the old segment expired before the 4-tuple was reused"
print()
for reuse in (1, 60):
print(f"4-tuple reused after {reuse:>2} s, segment delayed 45 s -> "
f"{old_duplicate(45, reuse)}")
# 4-tuple reused after 1 s, segment delayed 45 s -> HAZARD: a delayed segment from the old connection can land inside the new connection's window
# 4-tuple reused after 60 s, segment delayed 45 s -> safe: the old segment expired before the 4-tuple was reused
# Connection reuse is the real fix.
def with_reuse(requests_per_sec, requests_per_conn):
cps = requests_per_sec / requests_per_conn
return cps, port_exhaustion(int(cps))["exhausted"]
print()
for reuse in (1, 10, 100, 1000):
cps, exhausted = with_reuse(5000, reuse)
print(f"{reuse:>5} requests/connection -> {cps:>8.1f} new conn/s "
f"{'EXHAUSTED' if exhausted else 'sustainable'}")
# 1 requests/connection -> 5000.0 new conn/s EXHAUSTED
# 10 requests/connection -> 500.0 new conn/s EXHAUSTED
# 100 requests/connection -> 50.0 new conn/s sustainable
# 1000 requests/connection -> 5.0 new conn/s sustainable
# Graceful close vs reset.
def close_behaviour(method):
return {
"close() with data sent": ("FIN", "graceful; TIME_WAIT on the closer",
"remaining data is delivered"),
"close() with unread data":("RST", "abortive; NO TIME_WAIT",
"unread data is DISCARDED"),
"SO_LINGER timeout 0": ("RST", "abortive by request; NO TIME_WAIT",
"avoids TIME_WAIT but loses data"),
"no listener on the port": ("RST", "immediate refusal",
"connection refused"),
}[method]
print()
for m in ["close() with data sent", "close() with unread data",
"SO_LINGER timeout 0", "no listener on the port"]:
flag, state, effect = close_behaviour(m)
print(f"{m:<28}{flag:<5}{state:<34}{effect}")
# close() with data sent FIN graceful; TIME_WAIT on the closer remaining data is delivered
# close() with unread data RST abortive; NO TIME_WAIT unread data is DISCARDED
# SO_LINGER timeout 0 RST abortive by request; NO TIME_WAIT avoids TIME_WAIT but loses data
# no listener on the port RST immediate refusal connection refused
# Count TIME_WAIT sockets — the first check when connections start failing:
$ ss -tan state time-wait | wc -l
# 24817
$ ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn
# 24817 TIME-WAIT
# 1204 ESTAB
# 12 LISTEN
# The symptom of exhaustion:
$ curl http://backend:8080/
# curl: (7) Failed to connect to backend port 8080: Cannot assign requested address
$ nstat -az | grep -i TW
# TcpExtTW 184221 <- sockets that passed through TIME_WAIT
# TcpExtTWRecycled 0
# Safe mitigations, in order of preference:
# 1. Connection reuse — fix the application first:
# curl: --keepalive / reuse the connection pool
# nginx: upstream backend { server 10.0.0.9:8080; keepalive 128; }
# 2. Widen the ephemeral range:
$ sudo sysctl -w net.ipv4.ip_local_port_range="10240 65535"
# 3. Allow TIME_WAIT reuse for OUTBOUND connections (safe with timestamps):
$ sudo sysctl -w net.ipv4.tcp_tw_reuse=1
$ sysctl net.ipv4.tcp_timestamps
# net.ipv4.tcp_timestamps = 1 # required for tw_reuse to be safe
# NEVER use tcp_tw_recycle — removed in Linux 4.12 because it broke NAT:
$ sysctl net.ipv4.tcp_tw_recycle 2>&1
# sysctl: cannot stat /proc/sys/net/ipv4/tcp_tw_recycle: No such file or directory
# It is gone. Any guide recommending it is obsolete.
# Watch a four-way close, including the half-close window:
$ sudo tcpdump -i eth0 -n 'tcp[tcpflags] & (tcp-fin|tcp-rst) != 0' -c 4
# IP 10.1.1.5.51250 > 93.184.216.34.443: Flags [F.], seq 1000, ack 5000
# IP 93.184.216.34.443 > 10.1.1.5.51250: Flags [.], ack 1001
# ^^^ ACK only: the server may
# still be sending (half-close)
# IP 93.184.216.34.443 > 10.1.1.5.51250: Flags [F.], seq 5000, ack 1001
# IP 10.1.1.5.51250 > 93.184.216.34.443: Flags [.], ack 5001
# Servers restarting need SO_REUSEADDR, or the bind fails while old
# connections linger in TIME_WAIT:
$ python3 -c "
import socket
s=socket.socket()
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('0.0.0.0',8443)); print('bound despite TIME_WAIT sockets')"
# Where the 60-second figure comes from:
$ grep -r "TCP_TIMEWAIT_LEN" /usr/include/netinet/tcp.h 2>/dev/null
# Linux hardcodes 60 s (2 x 30 s MSL); it is NOT tunable via sysctl.
Key takeaways
- Termination takes four messages because TCP is full duplex — each direction closes independently with its own FIN and ACK.
- The gap between the first ACK and the second FIN is the half-close: one side has stopped sending while the other may continue.
- The side that closes first enters TIME_WAIT for 2 × MSL (60 s on Linux).
- TIME_WAIT exists to ensure the final ACK can be retransmitted and to let old duplicate segments expire before the 4-tuple is reused.
- Without it, a lost final ACK produces an RST instead of a clean close, and a reused 4-tuple can accept stale data.
- A busy client or proxy accumulates TIME_WAIT sockets and hits ephemeral port exhaustion — the symptom is
Cannot assign requested address. - The correct first fix is connection reuse; then widening the port range and
tcp_tw_reuse(safe with timestamps). tcp_tw_recyclewas removed in Linux 4.12 because it broke NATed clients — never use it.- RST aborts immediately with no TIME_WAIT and discards unread data; many RSTs usually mean applications close with data still pending.
🧪 Practice
- Draw the four-way close with client ISN-derived sequence 2000 and server 7000, labelling each state transition.
- Give the two reasons TIME_WAIT exists and describe the specific failure each one prevents.
- A proxy makes 800 connections per second to one backend. Determine whether the default Linux ephemeral range is exhausted, and give the preferred fix.
- Interview-style: Why does
tcp_tw_recycleno longer exist in Linux, and what broke when people enabled it? (Hint: it tracked timestamps per remote IP — ask what happens when thousands of distinct clients arrive from behind one NAT address.)
TCP State Machine
Theory
Everything covered so far — handshake, data transfer, termination — is formalised in TCP's state machine. A connection is always in exactly one of eleven states, and each transition is driven either by an application call (connect, listen, close) or by an arriving segment.
Knowing the states is directly practical: ss and netstat report them, and the distribution of states on a busy server tells you immediately what kind of problem you have.
| State | Meaning | Who is here |
|---|---|---|
| CLOSED | No connection | Starting and ending point |
| LISTEN | Awaiting connections | Servers |
| SYN_SENT | SYN sent, awaiting SYN-ACK | Client, mid-handshake |
| SYN_RECEIVED | SYN-ACK sent, awaiting ACK | Server, mid-handshake |
| ESTABLISHED | Data may flow | Normal operation |
| FIN_WAIT_1 | Sent FIN, awaiting its ACK | The side closing first |
| FIN_WAIT_2 | FIN acknowledged, awaiting peer's FIN | The closer, half-closed |
| CLOSE_WAIT | Peer sent FIN; my application has not closed | The passive side |
| CLOSING | Both sent FIN simultaneously | Rare, simultaneous close |
| LAST_ACK | Sent my FIN, awaiting its ACK | The passive side, closing |
| TIME_WAIT | Waiting 2 × MSL | The side that closed first |
Two of these are diagnostic gold, and recognising them saves hours.
CLOSE_WAIT accumulation is an application bug, essentially always. The peer sent a FIN, the kernel acknowledged it, and the connection is waiting for the local application to call close(). The kernel cannot proceed without it. Hundreds of sockets stuck in CLOSE_WAIT means the application is leaking file descriptors — it is not closing sockets after the peer disconnects. No amount of kernel tuning fixes this; the code must be fixed.
FIN_WAIT_2 accumulation is the mirror image: the local side closed, and the peer's application has not. The problem is on the other end. Linux bounds this with tcp_fin_timeout (default 60 s) to avoid holding resources for a peer that never responds.
The asymmetry is worth stating plainly, because it makes both diagnoses immediate:
CLOSE_WAIT = my application is not closing. FIN_WAIT_2 = their application is not closing.
The simultaneous open and simultaneous close paths (through SYN_RECEIVED from SYN_SENT, and through CLOSING) exist for completeness. They are rare in practice but demonstrate that TCP's design is genuinely symmetric — there is no protocol-level notion of client and server, only of active and passive opens.
A final subtlety: the state machine lives in the kernel, not the application. A socket can be in CLOSE_WAIT while the application knows nothing about it, which is exactly why the bug is so easy to write. A read() returning 0 is the application's signal that the peer has closed — ignoring that return value is the usual cause.
Examples
The state machine (normal paths)
+---------+
| CLOSED |
+---------+
passive open | | active open (connect)
v v
+--------+ +----------+
| LISTEN | | SYN_SENT |
+--------+ +----------+
recv SYN | | recv SYN+ACK
v v
+--------------+ +-------------+
| SYN_RECEIVED |->| ESTABLISHED |
+--------------+ +-------------+
/ \
app calls close / \ recv FIN
v v
+------------+ +------------+
| FIN_WAIT_1 | | CLOSE_WAIT |
+------------+ +------------+
| recv ACK | app calls close
v v
+------------+ +-----------+
| FIN_WAIT_2 | | LAST_ACK |
+------------+ +-----------+
| recv FIN | recv ACK
v v
+------------+ +---------+
| TIME_WAIT | | CLOSED |
+------------+ +---------+
| 2 x MSL
v
CLOSED
ACTIVE CLOSER (left path) PASSIVE CLOSER (right path)
ends in TIME_WAIT ends in CLOSED
The two diagnostic states
CLOSE_WAIT piling up:
peer sent FIN -> kernel ACKed it -> waiting for MY app to close()
-> MY APPLICATION IS LEAKING SOCKETS. Fix the code.
FIN_WAIT_2 piling up:
I closed -> peer ACKed -> waiting for THEIR app to close()
-> THEIR APPLICATION is the problem (or a middlebox ate the FIN).
Mnemonic: CLOSE_WAIT = waiting for ME to close.
TCP_STATES = {
"CLOSED": ("no connection", "start and end"),
"LISTEN": ("awaiting connections", "servers"),
"SYN_SENT": ("SYN sent, awaiting SYN-ACK", "client mid-handshake"),
"SYN_RECEIVED": ("SYN-ACK sent, awaiting ACK", "server mid-handshake"),
"ESTABLISHED": ("data may flow", "normal operation"),
"FIN_WAIT_1": ("sent FIN, awaiting its ACK", "active closer"),
"FIN_WAIT_2": ("FIN acked, awaiting peer's FIN", "active closer, half-closed"),
"CLOSE_WAIT": ("peer sent FIN; MY app has not closed", "passive closer"),
"CLOSING": ("both sent FIN at once", "simultaneous close (rare)"),
"LAST_ACK": ("sent my FIN, awaiting its ACK", "passive closer"),
"TIME_WAIT": ("waiting 2 x MSL", "active closer"),
}
TRANSITIONS = {
("CLOSED", "app: listen"): "LISTEN",
("CLOSED", "app: connect"): "SYN_SENT",
("LISTEN", "recv SYN"): "SYN_RECEIVED",
("SYN_SENT", "recv SYN+ACK"): "ESTABLISHED",
("SYN_RECEIVED", "recv ACK"): "ESTABLISHED",
("ESTABLISHED", "app: close"): "FIN_WAIT_1",
("ESTABLISHED", "recv FIN"): "CLOSE_WAIT",
("FIN_WAIT_1", "recv ACK"): "FIN_WAIT_2",
("FIN_WAIT_1", "recv FIN"): "CLOSING",
("FIN_WAIT_2", "recv FIN"): "TIME_WAIT",
("CLOSE_WAIT", "app: close"): "LAST_ACK",
("LAST_ACK", "recv ACK"): "CLOSED",
("CLOSING", "recv ACK"): "TIME_WAIT",
("TIME_WAIT", "2xMSL elapsed"): "CLOSED",
}
def walk(start, events):
state, path = start, [start]
for e in events:
nxt = TRANSITIONS.get((state, e))
if nxt is None:
path.append(f"?? no transition for '{e}' from {state}")
break
state = nxt
path.append(f"--{e}--> {state}")
return path
print("ACTIVE CLOSE (the side that calls close() first):")
for step in walk("ESTABLISHED", ["app: close", "recv ACK", "recv FIN", "2xMSL elapsed"]):
print(f" {step}")
# ACTIVE CLOSE (the side that calls close() first):
# ESTABLISHED
# --app: close--> FIN_WAIT_1
# --recv ACK--> FIN_WAIT_2
# --recv FIN--> TIME_WAIT
# --2xMSL elapsed--> CLOSED
print("\nPASSIVE CLOSE (the side that receives the FIN):")
for step in walk("ESTABLISHED", ["recv FIN", "app: close", "recv ACK"]):
print(f" {step}")
# PASSIVE CLOSE (the side that receives the FIN):
# ESTABLISHED
# --recv FIN--> CLOSE_WAIT
# --app: close--> LAST_ACK
# --recv ACK--> CLOSED
# The diagnostic value of a state distribution.
def diagnose(counts):
total = sum(counts.values())
out = []
if counts.get("CLOSE_WAIT", 0) > total * 0.1:
out.append("CLOSE_WAIT high -> MY application is not calling close(): "
"file descriptor leak. Fix the code, not the kernel.")
if counts.get("FIN_WAIT_2", 0) > total * 0.1:
out.append("FIN_WAIT_2 high -> the PEER's application is not closing, "
"or a middlebox dropped the FIN.")
if counts.get("TIME_WAIT", 0) > total * 0.5:
out.append("TIME_WAIT high -> many short-lived outbound connections; "
"use connection reuse before tuning.")
if counts.get("SYN_RECV", 0) > total * 0.2:
out.append("SYN_RECV high -> possible SYN flood; enable syncookies.")
return out or ["distribution looks healthy"]
SAMPLES = {
"healthy server": {"ESTAB": 4000, "TIME_WAIT": 400, "LISTEN": 12, "CLOSE_WAIT": 3},
"leaking app": {"ESTAB": 200, "CLOSE_WAIT": 8400, "LISTEN": 4},
"busy proxy": {"ESTAB": 900, "TIME_WAIT": 24000, "LISTEN": 4},
"under SYN flood": {"ESTAB": 120, "SYN_RECV": 4092, "LISTEN": 4},
}
print()
for name, counts in SAMPLES.items():
print(f"{name}:")
for line in diagnose(counts):
print(f" {line}")
# healthy server:
# distribution looks healthy
# leaking app:
# CLOSE_WAIT high -> MY application is not calling close(): file descriptor leak. Fix the code, not the kernel.
# busy proxy:
# TIME_WAIT high -> many short-lived outbound connections; use connection reuse before tuning.
# under SYN flood:
# SYN_RECV high -> possible SYN flood; enable syncookies.
# The bug that creates CLOSE_WAIT.
def read_loop(bytes_returned, closes_socket):
if bytes_returned == 0 and not closes_socket:
return ("read() returned 0 (peer closed) but the code did not call "
"close() -> socket stuck in CLOSE_WAIT forever")
if bytes_returned == 0:
return "read() returned 0 -> close() called -> LAST_ACK -> CLOSED"
return "data processed normally"
print()
for closes in (False, True):
print(f"handles read()==0 correctly={closes!s:<5} -> {read_loop(0, closes)}")
# handles read()==0 correctly=False -> read() returned 0 (peer closed) but the code did not call close() -> socket stuck in CLOSE_WAIT forever
# handles read()==0 correctly=True -> read() returned 0 -> close() called -> LAST_ACK -> CLOSED
# The state distribution — the single most useful TCP diagnostic:
$ ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn
# 24817 TIME-WAIT
# 4021 ESTAB
# 84 CLOSE-WAIT
# 12 LISTEN
# 3 FIN-WAIT-2
# Filter by state directly:
$ ss -tan state close-wait
$ ss -tan state fin-wait-2
$ ss -tan state syn-recv | wc -l
# Find WHICH process is leaking sockets in CLOSE_WAIT:
$ ss -tanp state close-wait | awk '{print $6}' | sort | uniq -c | sort -rn
# 8400 users:(("myapp",pid=4821,fd=...))
# ^^^^^ this process is not calling close() after the peer disconnects
$ ls /proc/4821/fd | wc -l
# 8447
$ cat /proc/4821/limits | grep -i "open files"
# Max open files 1024 65535
# At 8447 the process will hit the limit and start failing accept().
# Bound FIN_WAIT_2 so a dead peer cannot hold resources forever:
$ sysctl net.ipv4.tcp_fin_timeout
# net.ipv4.tcp_fin_timeout = 60
# Watch state transitions live for one connection:
$ sudo bpftrace -e 'kprobe:tcp_set_state {
printf("%s -> state %d\n", comm, arg1); }' 2>/dev/null | head -5
# Or with ss, polling:
$ watch -n1 "ss -tan state all '( dport = :443 )' | head"
# A connection stuck in CLOSE_WAIT, proven:
$ ss -tanp state close-wait | head -2
# Recv-Q Send-Q Local Address:Port Peer Address:Port
# 0 0 10.1.1.5:8080 10.2.2.9:51234 users:(("myapp",pid=4821))
# ^^^^^^ Recv-Q 0: nothing left to read. The peer's FIN was processed.
# The kernel is simply waiting for the application to close(). It never will.
# The correct application pattern:
$ python3 -c "
import socket
s = socket.socket()
# ... connected ...
data = s.recv(4096)
if not data: # read() == 0 means the PEER CLOSED
s.close() # MUST close, or the socket sits in CLOSE_WAIT
"
Key takeaways
- A TCP connection is always in one of eleven states, driven by application calls and arriving segments.
- The active closer path is
ESTABLISHED → FIN_WAIT_1 → FIN_WAIT_2 → TIME_WAIT → CLOSED. - The passive closer path is
ESTABLISHED → CLOSE_WAIT → LAST_ACK → CLOSED. - CLOSE_WAIT accumulation means the local application is not calling
close()— a file descriptor leak, fixable only in the code. - FIN_WAIT_2 accumulation means the peer's application is not closing; Linux bounds it with
tcp_fin_timeout. - Mnemonic: CLOSE_WAIT = waiting for me; FIN_WAIT_2 = waiting for them.
- SYN_RECV accumulation indicates a possible SYN flood; TIME_WAIT accumulation indicates many short-lived outbound connections.
- CLOSING and simultaneous open exist because TCP is genuinely symmetric — there is no protocol-level client or server, only active and passive opens.
- The usual cause of CLOSE_WAIT is ignoring a
read()return value of 0, which is the signal that the peer has closed.
🧪 Practice
- List the state sequence for the side that closes first and for the side that closes second.
- A server shows 9,000 sockets in CLOSE_WAIT. State the cause, which side is at fault, and why kernel tuning will not help.
- Explain what distinguishes FIN_WAIT_2 from CLOSE_WAIT in terms of which application is blocking progress.
- Interview-style: A long-running service gradually stops accepting new connections over several days, with no error in its logs.
ssshows thousands of CLOSE_WAIT sockets. Diagnose it. (Hint: connect the socket state to a per-process resource limit, and ask which function call the code is failing to make.)
Sequence and Acknowledgment Numbers
Theory
Sequence numbers are the foundation everything else in TCP rests on. They make loss detectable, ordering recoverable, and duplicates removable — and they do it with one idea: every byte of the stream has a number.
Not every segment — every byte. A segment's Sequence Number field carries the number of its first byte; the rest are implied by the segment's length. This byte-level granularity is what lets TCP resegment freely: a retransmission need not have the same boundaries as the original, and a sender can coalesce or split data as conditions change.
The Acknowledgment Number is cumulative: it names the next byte expected, which implicitly acknowledges everything before it. Acknowledging byte 5001 means "I have bytes 1 through 5000". This has an important property — a lost ACK is harmless, because the next ACK supersedes it. If the ACK for 3000 is lost but the ACK for 5000 arrives, the sender learns everything it needed.
Cumulative acknowledgement has a matching weakness, and it is the reason SACK exists. If bytes 1–1000 and 2001–3000 arrive but 1001–2000 is lost, the receiver can only acknowledge 1001. It cannot say "I also have 2001–3000". The sender knows something was lost but not what arrived, and a naive sender retransmits everything from 1001 onward — including data already received.
Sequence numbers are 32 bits, so the space is 4,294,967,296 bytes and it wraps. On a fast link this happens quickly: at 10 Gb/s the space wraps in under 4 seconds. Wrapping creates the old duplicate hazard — a delayed segment from earlier in the connection could carry a sequence number that is valid again. PAWS (Protection Against Wrapped Sequence numbers) solves this using the timestamp option: each segment carries a timestamp, and a segment whose timestamp is older than the connection's current one is discarded regardless of its sequence number.
The starting point is the ISN, chosen randomly at handshake time (7.3). Because each direction has its own independent sequence space, a connection has two: the client's bytes are numbered from the client's ISN, the server's from the server's.
The receive window (7.4) defines which sequence numbers are currently acceptable. A segment falling outside it is discarded — which is also the basis of off-path injection resistance: an attacker who cannot observe the connection must guess a sequence number within the window, and the window is small relative to the 32-bit space.
Examples
Every BYTE is numbered; a segment carries its first byte's number
application writes 3000 bytes, TCP sends three segments (MSS 1000):
segment 1: seq=1 length=1000 covers bytes 1..1000
segment 2: seq=1001 length=1000 covers bytes 1001..2000
segment 3: seq=2001 length=1000 covers bytes 2001..3000
^^^^^^^^ the FIRST byte; the rest are implied by length
receiver ACKs:
after segment 1: ACK=1001 ("I have 1..1000, send 1001 next")
after segment 2: ACK=2001
after segment 3: ACK=3001
Cumulative ACK: a lost ACK repairs itself
seq 1..1000 ---->
<---- ACK 1001 X LOST
seq 1001..2000 ---->
<---- ACK 2001 arrives
The sender learns 1..2000 arrived. The lost ACK cost NOTHING.
...but cumulative ACK cannot describe a hole
sent: [1..1000][1001..2000][2001..3000][3001..4000]
arrived: [1..1000] X [2001..3000][3001..4000]
LOST
receiver can ONLY say ACK=1001.
It has 2001..4000 sitting in its buffer and no way to mention them.
naive sender: retransmits 1001..4000 (3000 bytes, 2000 of them wasted)
with SACK: retransmits 1001..2000 (1000 bytes, exactly the hole)
PAWS: why 32 bits is not enough at speed
at 10 Gb/s the 4 GB sequence space wraps in 3.4 seconds
a segment delayed 5 seconds arrives with a sequence number
that is VALID AGAIN in the new cycle
-> timestamps disambiguate: older timestamp = discard
class TCPStream:
"""Byte-level sequence numbering and cumulative acknowledgement."""
def __init__(self, isn):
self.isn = isn
self.snd_nxt = isn + 1 # +1: the SYN consumed a sequence number
self.segments = []
def send(self, data, mss=1000):
out = []
for i in range(0, len(data), mss):
chunk = data[i:i + mss]
out.append({"seq": self.snd_nxt, "len": len(chunk),
"covers": f"{self.snd_nxt}..{self.snd_nxt + len(chunk) - 1}"})
self.snd_nxt += len(chunk)
self.segments += out
return out
s = TCPStream(isn=0) # ISN 0 for readability
for seg in s.send(b"x" * 3000, mss=1000):
print(f" seq={seg['seq']:<6} len={seg['len']:<5} covers bytes {seg['covers']}")
# seq=1 len=1000 covers bytes 1..1000
# seq=1001 len=1000 covers bytes 1001..2000
# seq=2001 len=1000 covers bytes 2001..3000
class Receiver:
"""Cumulative ACK: name the next byte expected; buffer anything beyond a hole."""
def __init__(self, isn):
self.rcv_nxt = isn + 1
self.out_of_order = {} # seq -> length
def receive(self, seq, length):
if seq < self.rcv_nxt:
return self.rcv_nxt, "duplicate — discarded, re-ACK"
if seq == self.rcv_nxt:
self.rcv_nxt += length
# A contiguous run may now be complete; absorb it.
while self.rcv_nxt in self.out_of_order:
self.rcv_nxt += self.out_of_order.pop(self.rcv_nxt)
return self.rcv_nxt, "in order — advances the ACK"
self.out_of_order[seq] = length
return self.rcv_nxt, f"OUT OF ORDER — buffered; ACK stays at {self.rcv_nxt}"
r = Receiver(isn=0)
print()
for seq, length, label in [(1, 1000, "seg 1"), (2001, 1000, "seg 3 (seg 2 lost)"),
(3001, 1000, "seg 4"), (1001, 1000, "seg 2 retransmitted")]:
ack, note = r.receive(seq, length)
print(f" {label:<24} seq={seq:<5} -> ACK={ack:<5} {note}")
# seg 1 seq=1 -> ACK=1001 in order — advances the ACK
# seg 3 (seg 2 lost) seq=2001 -> ACK=1001 OUT OF ORDER — buffered; ACK stays at 1001
# seg 4 seq=3001 -> ACK=1001 OUT OF ORDER — buffered; ACK stays at 1001
# seg 2 retransmitted seq=1001 -> ACK=4001 in order — advances the ACK
# Note the last line: ONE retransmission unblocked 3000 buffered bytes at once.
# The cost of cumulative-only acknowledgement.
def retransmit_bytes(hole_start, hole_len, highest_sent, sack):
if sack:
return hole_len, "exactly the hole"
total = highest_sent - hole_start
return total, f"everything from {hole_start} onward — {total - hole_len} bytes wasted"
print()
for sack in (False, True):
n, note = retransmit_bytes(1001, 1000, 4001, sack)
print(f"SACK={sack!s:<5} retransmit {n:>5} bytes ({note})")
# SACK=False retransmit 3000 bytes (everything from 1001 onward — 2000 bytes wasted)
# SACK=True retransmit 1000 bytes (exactly the hole)
# Sequence space wrap: why PAWS is needed at speed.
SEQ_SPACE = 2 ** 32
def wrap_time(bandwidth_gbps):
bytes_per_sec = bandwidth_gbps * 1e9 / 8
return SEQ_SPACE / bytes_per_sec
print(f"\nsequence space: {SEQ_SPACE:,} bytes ({SEQ_SPACE/1e9:.2f} GB)")
print(f"{'link speed':>12}{'wrap time':>14}{'safe without PAWS?':>22}")
for gbps in (0.1, 1, 10, 100):
t = wrap_time(gbps)
safe = "yes" if t > 120 else "NO — needs PAWS"
print(f"{gbps:>9} Gb/s{t:>12.1f} s{safe:>22}")
# sequence space: 4,294,967,296 bytes (4.29 GB)
# link speed wrap time safe without PAWS?
# 0.1 Gb/s 343.6 s yes
# 1 Gb/s 34.4 s NO — needs PAWS
# 10 Gb/s 3.4 s NO — needs PAWS
# 100 Gb/s 0.3 s NO — needs PAWS
# MSL is 30 s, so anything above ~1 Gb/s can wrap while old segments
# are still alive in the network.
def paws_check(seg_timestamp, connection_timestamp):
if seg_timestamp < connection_timestamp:
return "DISCARD: timestamp is older than the connection's — a wrapped duplicate"
return "accept: timestamp is current"
print()
for ts in (184100, 184300):
print(f"segment ts={ts}, connection ts=184221 -> {paws_check(ts, 184221)}")
# segment ts=184100, connection ts=184221 -> DISCARD: timestamp is older than the connection's — a wrapped duplicate
# segment ts=184300, connection ts=184221 -> accept: timestamp is current
# Off-path injection: the attacker must guess into the window.
def injection_difficulty(window_bytes):
space = SEQ_SPACE
return {"window": window_bytes,
"guesses_needed": space // window_bytes,
"probability_per_guess": window_bytes / space}
print()
for w in (65_535, 1_048_576, 16_777_216):
d = injection_difficulty(w)
print(f"window {w:>10,} bytes -> ~{d['guesses_needed']:>7,} guesses needed "
f"(p={d['probability_per_guess']:.6f} per packet)")
# window 65,535 bytes -> ~ 65,537 guesses needed (p=0.000015 per packet)
# window 1,048,576 bytes -> ~ 4,096 guesses needed (p=0.000244 per packet)
# window 16,777,216 bytes -> ~ 256 guesses needed (p=0.003906 per packet)
# Large windows (7.4) make blind injection EASIER — one reason
# challenge-ACK rate limiting exists.
# Follow sequence numbers in a capture. Relative numbers are far easier:
$ sudo tcpdump -i eth0 -n 'tcp port 443' -c 5
# IP 10.1.1.5.51250 > 93.184.216.34.443: Flags [P.], seq 1:1449, ack 1, length 1448
# ^^^^^^ bytes 1..1448
# IP 93.184.216.34.443 > 10.1.1.5.51250: Flags [.], ack 1449, length 0
# ^^^^^^^^ next expected
# IP 10.1.1.5.51250 > 93.184.216.34.443: Flags [P.], seq 1449:2897, ack 1, length 1448
# IP 93.184.216.34.443 > 10.1.1.5.51250: Flags [.], ack 2897
# Absolute sequence numbers with -S, when you need the real values:
$ sudo tcpdump -i eth0 -n -S 'tcp port 443' -c 2
# Flags [S], seq 2712847316
# Flags [S.], seq 891204471, ack 2712847317
# Spot a retransmission — the same sequence number sent twice:
$ sudo tcpdump -i eth0 -n 'tcp port 443' -c 100 | awk '{print $NF, $0}' | \
> grep -oP 'seq \d+:\d+' | sort | uniq -d
# seq 1449:2897 <- sent more than once: a retransmission
# Wireshark marks them automatically:
$ tshark -r capture.pcap -Y 'tcp.analysis.retransmission' | head -3
$ tshark -r capture.pcap -Y 'tcp.analysis.duplicate_ack' | head -3
$ tshark -r capture.pcap -Y 'tcp.analysis.out_of_order' | head -3
# Timestamps enable PAWS — verify they are on (they are, by default):
$ sysctl net.ipv4.tcp_timestamps
# net.ipv4.tcp_timestamps = 1
$ sudo tcpdump -i eth0 -n 'tcp port 443' -v -c 1 | grep -o "TS val [0-9]* ecr [0-9]*"
# TS val 184221 ecr 99812
# ^^^^^^ my clock ^^^^^ the last value I heard from the peer
# Counters for what sequence numbers detect:
$ nstat -az | grep -iE "PAWS|OfoPruned|TCPOFO"
# TcpExtPAWSEstab 12 <- wrapped duplicates discarded
# TcpExtTCPOFOQueue 184221 <- segments buffered out of order
# TcpExtTCPOFOMerge 4127
# Per-connection sequence state:
$ ss -tin state established | head -3
# ESTAB 0 0 10.1.1.5:51250 93.184.216.34:443
# cubic wscale:7,7 rto:204 rtt:3.821/1.242 mss:1448 cwnd:10 bytes_sent:184221
# bytes_acked:184221 segs_out:128 segs_in:130
# ^^^^^^^^^^^ everything acknowledged: no outstanding holes
Key takeaways
- Every byte of the stream is numbered; a segment's Sequence Number is its first byte's number, with the rest implied by length.
- Byte-level numbering lets TCP resegment freely — a retransmission need not match the original's boundaries.
- The Acknowledgment Number is cumulative: it names the next byte expected and implicitly acknowledges everything before it.
- Because ACKs are cumulative, a lost ACK is harmless — the next one supersedes it.
- The weakness is that a receiver cannot describe data beyond a hole, so a naive sender retransmits everything from the gap onward; SACK fixes this.
- The sequence space is 32 bits (4.29 GB) and wraps in 3.4 seconds at 10 Gb/s.
- PAWS uses the timestamp option to discard wrapped old duplicates whose sequence numbers have become valid again.
- Each direction has its own independent sequence space, starting from its own random ISN.
- The receive window bounds acceptable sequence numbers, which is also what makes blind off-path injection hard — though large windows make it easier.
🧪 Practice
- A sender transmits 5,000 bytes as five 1,000-byte segments starting at sequence 1. Give the sequence number of each segment and the ACK after each arrives in order.
- Segments 1, 3, and 4 arrive but segment 2 is lost. State the acknowledgment number the receiver sends after each arrival and explain why it does not advance.
- Compute how long the 32-bit sequence space takes to wrap at 40 Gb/s, and name the mechanism that prevents old duplicates from being accepted.
- Interview-style: Why is a lost acknowledgement usually harmless, while a lost data segment is not? (Hint: consider what the next acknowledgement conveys about everything before it, and whether data has the same property.)
Retransmission Timeout and RTT Estimation
Theory
If a segment is lost, the sender must eventually notice and resend it. Without acknowledgement, the only evidence is silence — so TCP sets a timer, and if the ACK does not arrive before it expires, it retransmits. That timer is the RTO (Retransmission Timeout), and choosing its value is a genuinely difficult estimation problem.
The difficulty is that both errors are costly:
Too short and the sender retransmits data that was merely slow, wasting bandwidth and — worse — interpreting the resulting situation as congestion, which needlessly halves its rate. Too long and real losses go undetected for an extended period, stalling the connection.
The RTO must therefore track the actual round-trip time, which varies continuously with queueing, routing changes, and load. TCP measures RTT by timing how long an acknowledgement takes and feeds it into a smoothed estimator.
The original algorithm (Jacobson/Karels, 1988, RFC 6298) tracks both the average and the variability:
SRTT = (1 - alpha) * SRTT + alpha * RTT_sample alpha = 1/8
RTTVAR = (1 - beta) * RTTVAR + beta * |SRTT - RTT_sample| beta = 1/4
RTO = SRTT + max(G, 4 * RTTVAR)
The term that matters most is 4 × RTTVAR. Early TCP used only a multiple of the average, which failed badly: on a path with variable delay, an RTT slightly above average triggered spurious retransmissions. Including the variance means a jittery path automatically gets a more generous timeout, while a stable path gets a tight one. This single change was central to resolving the congestion collapses of the mid-1980s.
Three further rules complete the picture:
Karn's algorithm. Never take an RTT sample from a retransmitted segment. The ambiguity is fatal: when an ACK arrives for a segment that was sent twice, there is no way to know which transmission it acknowledges. Measuring from the first gives too large a value; from the second, too small. So retransmitted segments are simply excluded. (The timestamp option resolves the ambiguity and allows measurement even then, which is one reason timestamps are valuable.)
Exponential backoff. Each successive retransmission of the same segment doubles the RTO. If the network is not responding, hammering it harder is counterproductive. Linux retries up to tcp_retries2 (default 15), which with backoff takes roughly 15 minutes before the connection is declared dead.
Minimum and maximum bounds. Linux clamps the RTO to 200 ms minimum and 120 s maximum. The minimum exists because timer granularity and delayed ACKs (7.4) make very small values unreliable — but it also means that on a fast data-centre network with 100 µs RTTs, a timeout-based recovery costs 2,000× the RTT. This is why data-centre TCP variants and fast-recovery mechanisms (next topic) matter so much: an RTO is catastrophic when the RTT is tiny.
The practical consequence: RTO-based recovery is the slow path, used only when faster mechanisms fail. Fast retransmit and SACK exist precisely to avoid it.
Examples
Why variance matters — the same average, different jitter
STABLE PATH: samples 100, 102, 99, 101, 100 ms
SRTT ~ 100, RTTVAR ~ 1
RTO = 100 + 4(1) = 104 ms <- tight; detects loss quickly
JITTERY PATH: samples 100, 180, 60, 200, 40 ms
SRTT ~ 100, RTTVAR ~ 60
RTO = 100 + 4(60) = 340 ms <- generous; avoids false alarms
Same average. Using the average alone would give both 104 ms,
and the jittery path would retransmit constantly for no reason.
Karn's algorithm: the ambiguity that must be avoided
t=0 send segment X (first transmission)
t=200 RTO fires, RETRANSMIT X
t=250 ACK for X arrives
Which transmission does it acknowledge?
from the 1st: RTT = 250 ms (too large -> RTO inflates)
from the 2nd: RTT = 50 ms (too small -> RTO collapses)
UNKNOWABLE -> take no sample at all.
With the timestamp option, the ACK echoes which one -> sample is safe.
Exponential backoff
attempt 1: RTO = 200 ms (total elapsed 0.2 s)
attempt 2: RTO = 400 ms 0.6 s
attempt 3: RTO = 800 ms 1.4 s
attempt 4: RTO = 1.6 s 3.0 s
...
attempt 15 ~ 15 minutes -> connection dead
class RTOEstimator:
"""Jacobson/Karels, RFC 6298."""
ALPHA, BETA = 1/8, 1/4
MIN_RTO, MAX_RTO, GRANULARITY = 0.2, 120.0, 0.001 # Linux-like, seconds
def __init__(self):
self.srtt = None
self.rttvar = None
self.rto = 1.0 # RFC 6298 initial value
def sample(self, rtt, was_retransmitted=False):
if was_retransmitted:
# KARN'S ALGORITHM: the sample is ambiguous. Discard it.
return self.rto, "sample DISCARDED (Karn: retransmitted segment)"
if self.srtt is None:
self.srtt = rtt
self.rttvar = rtt / 2
else:
self.rttvar = (1 - self.BETA) * self.rttvar + self.BETA * abs(self.srtt - rtt)
self.srtt = (1 - self.ALPHA) * self.srtt + self.ALPHA * rtt
self.rto = self.srtt + max(self.GRANULARITY, 4 * self.rttvar)
self.rto = min(self.MAX_RTO, max(self.MIN_RTO, self.rto))
return self.rto, "updated"
def backoff(self):
self.rto = min(self.MAX_RTO, self.rto * 2)
return self.rto
def run(samples, label):
est = RTOEstimator()
print(f"\n{label}")
print(f" {'sample':>8}{'SRTT':>9}{'RTTVAR':>9}{'RTO':>9}")
for rtt in samples:
est.sample(rtt)
print(f" {rtt*1000:>7.0f}ms{est.srtt*1000:>8.1f}ms{est.rttvar*1000:>8.1f}ms"
f"{est.rto*1000:>8.1f}ms")
return est
run([0.100, 0.102, 0.099, 0.101, 0.100], "STABLE path (low jitter)")
run([0.100, 0.180, 0.060, 0.200, 0.040], "JITTERY path (same average)")
#
# STABLE path (low jitter)
# sample SRTT RTTVAR RTO
# 100ms 100.0ms 50.0ms 300.0ms
# 102ms 100.2ms 38.0ms 252.3ms
# 99ms 100.1ms 28.8ms 215.3ms
# 101ms 100.2ms 21.8ms 200.0ms
# 100ms 100.2ms 16.4ms 200.0ms
#
# JITTERY path (same average)
# sample SRTT RTTVAR RTO
# 100ms 100.0ms 50.0ms 300.0ms
# 180ms 110.0ms 57.5ms 340.0ms
# 60ms 103.8ms 55.6ms 326.3ms
# 200ms 115.8ms 65.8ms 378.9ms
# 40ms 106.3ms 68.3ms 379.4ms
# Same ~100 ms average. The stable path settles at the 200 ms floor;
# the jittery path holds a 379 ms timeout and avoids false alarms.
# Exponential backoff to connection death.
est = RTOEstimator()
est.sample(0.050)
print(f"\nbackoff from RTO {est.rto*1000:.0f} ms:")
total = 0.0
for attempt in range(1, 9):
total += est.rto
print(f" attempt {attempt:>2}: RTO {est.rto:>7.2f} s cumulative {total:>8.2f} s")
est.backoff()
# backoff from RTO 200 ms:
# attempt 1: RTO 0.20 s cumulative 0.20 s
# attempt 2: RTO 0.40 s cumulative 0.60 s
# attempt 3: RTO 0.80 s cumulative 1.40 s
# attempt 4: RTO 1.60 s cumulative 3.00 s
# attempt 5: RTO 3.20 s cumulative 6.20 s
# attempt 6: RTO 6.40 s cumulative 12.60 s
# attempt 7: RTO 12.80 s cumulative 25.40 s
# attempt 8: RTO 25.60 s cumulative 51.00 s
# The minimum RTO is catastrophic on low-latency paths.
def rto_penalty(rtt_ms, min_rto_ms=200):
return {"rtt_ms": rtt_ms, "min_rto_ms": min_rto_ms,
"penalty_x": round(min_rto_ms / rtt_ms, 1)}
print()
for rtt in (0.1, 1, 10, 100):
p = rto_penalty(rtt)
print(f"RTT {rtt:>6.1f} ms -> a timeout costs {p['min_rto_ms']} ms "
f"= {p['penalty_x']:>7,}x the RTT")
# RTT 0.1 ms -> a timeout costs 200 ms = 2,000.0x the RTT
# RTT 1.0 ms -> a timeout costs 200 ms = 200.0x the RTT
# RTT 10.0 ms -> a timeout costs 200 ms = 20.0x the RTT
# RTT 100.0 ms -> a timeout costs 200 ms = 2.0x the RTT
# In a data centre an RTO is a disaster. This is why fast retransmit
# and SACK matter far more than the timer.
# Karn's algorithm in action.
est = RTOEstimator()
est.sample(0.100)
print()
for rtt, retx in [(0.105, False), (0.250, True), (0.098, False)]:
rto, note = est.sample(rtt, was_retransmitted=retx)
print(f" sample {rtt*1000:>6.0f} ms retransmitted={retx!s:<5} -> "
f"RTO {rto*1000:>6.1f} ms ({note})")
# sample 105 ms retransmitted=False -> RTO 255.6 ms (updated)
# sample 250 ms retransmitted=True -> RTO 255.6 ms (sample DISCARDED (Karn: retransmitted segment))
# sample 98 ms retransmitted=False -> RTO 219.2 ms (updated)
# The 250 ms outlier left the estimate untouched — exactly Karn's purpose.
# Per-connection RTT estimate and current RTO:
$ ss -tin state established | head -3
# ESTAB 0 0 10.1.1.5:51250 93.184.216.34:443
# cubic wscale:7,7 rto:204 rtt:3.821/1.242 mss:1448 cwnd:10
# ^^^^^^^ ^^^^^^^^^^^^^^^
# RTO ms SRTT/RTTVAR in ms
# Note rto 204 ~ the 200 ms floor: the real estimate is far below it.
# Retransmission counters — timeout-based vs fast:
$ nstat -az | grep -iE "Retrans|Timeout"
# TcpRetransSegs 18492
# TcpExtTCPFastRetrans 14201 <- the FAST path (next topic)
# TcpExtTCPTimeouts 2841 <- the SLOW path: RTO fired
# TcpExtTCPSpuriousRTOs 94 <- RTO fired but nothing was lost
# TcpExtTCPLossProbes 1204 <- TLP, probing before a full RTO
# A high TCPTimeouts:TCPFastRetrans ratio means recovery is hitting the
# slow path — often too few packets in flight for fast retransmit to trigger.
# How long before a connection is abandoned:
$ sysctl net.ipv4.tcp_retries2 net.ipv4.tcp_syn_retries
# net.ipv4.tcp_retries2 = 15 # ~15 minutes with backoff
# net.ipv4.tcp_syn_retries = 6 # ~127 s for the initial handshake
# Lower it for a failover-sensitive service:
$ sudo sysctl -w net.ipv4.tcp_retries2=8 # ~30 s before declaring death
# TCP_USER_TIMEOUT: a per-socket bound, far better than tuning globally:
$ python3 -c "
import socket
s = socket.socket()
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_USER_TIMEOUT, 5000) # 5 s
print('this socket abandons unacked data after 5 s regardless of retries2')"
# Watch the RTO grow under loss:
$ sudo tc qdisc add dev eth0 root netem loss 30%
$ ss -tin state established | grep -o "rto:[0-9]*"
# rto:204
# rto:408 <- backoff
# rto:816
$ sudo tc qdisc del dev eth0 root
# Timestamps let RTT be sampled even on retransmissions (bypassing Karn):
$ sysctl net.ipv4.tcp_timestamps
# net.ipv4.tcp_timestamps = 1
$ sudo tcpdump -i eth0 -n 'tcp port 443' -v -c 1 | grep -o "TS val [0-9]* ecr [0-9]*"
# TS val 184221 ecr 99812
# ^^^^^ the echoed value identifies WHICH transmission
Key takeaways
- The RTO is the timer that triggers retransmission when no acknowledgement arrives; setting it is an estimation problem with costly errors on both sides.
- Too short causes spurious retransmissions and needless rate reduction; too long stalls the connection.
- Jacobson/Karels tracks both SRTT (smoothed average) and RTTVAR (variability):
RTO = SRTT + 4 × RTTVAR. - Including variance is the key insight — a jittery path automatically gets a generous timeout, a stable path a tight one.
- Karn's algorithm: never sample RTT from a retransmitted segment, because the ACK is ambiguous; the timestamp option removes the ambiguity.
- Exponential backoff doubles the RTO on each retry, up to
tcp_retries2(default 15, roughly 15 minutes). - Linux clamps the RTO to 200 ms minimum, which on a data-centre path with 100 µs RTT is 2,000× the RTT — an RTO is catastrophic there.
- RTO recovery is the slow path; fast retransmit and SACK exist to avoid it, and
TCP_USER_TIMEOUTbounds it per socket.
🧪 Practice
- Compute SRTT, RTTVAR, and RTO after samples of 100 ms and 140 ms, starting from SRTT 100 ms and RTTVAR 10 ms.
- Explain why two paths with the same average RTT but different jitter should get different RTO values.
- State Karn's algorithm and the specific ambiguity it avoids, then explain how timestamps resolve it.
- Interview-style: A data-centre service with 200 µs RTT sees occasional 200 ms latency spikes on individual requests. What is happening? (Hint: compare the observed spike with the kernel's minimum RTO, and ask what event would force a connection onto the timeout path.)
Fast Retransmit and Fast Recovery
Theory
Waiting for an RTO is expensive — at minimum 200 ms on Linux, and often far more than the actual round-trip time. Fast retransmit detects loss much sooner, using information the receiver is already sending.
The insight is that a receiver acknowledges every segment it gets, even out-of-order ones — and because ACKs are cumulative (7.3), an out-of-order arrival produces a duplicate ACK: the same acknowledgment number repeated, because the receiver still wants the same missing byte.
So duplicate ACKs are a signal. They mean "segments are still arriving, but there is a hole". Crucially, they also mean the network is still delivering data — which is quite different from the silence that triggers an RTO.
Fast retransmit acts on three duplicate ACKs (four identical ACKs in total): resend the apparently-missing segment immediately, without waiting for the timer.
Why three and not one? Because reordering is normal in the internet — packets can take different paths through ECMP (Chapter 6) and arrive out of order without any loss. One or two duplicate ACKs are more likely reordering than loss. Three is an empirical compromise: high enough to tolerate common reordering, low enough to react quickly. (Modern Linux uses RACK-TLR, which reasons about time rather than counting duplicates, and handles reordering better.)
Fast recovery is what happens after the retransmission, and it embodies an important judgement. Since duplicate ACKs prove data is still flowing, the network is congested but not collapsed — a milder condition than a timeout, which suggests the path has stopped working entirely. So the response is milder:
| Event | Interpretation | Response |
|---|---|---|
| RTO | The path may be down | cwnd = 1 MSS, slow start |
| 3 dup ACKs | Congested but delivering | cwnd = ssthresh (halved), congestion avoidance |
This distinction — halve rather than collapse — is what TCP Reno introduced over Tahoe, and it is the single most significant early improvement in TCP throughput. Each duplicate ACK during recovery also inflates the window by one MSS, because each one indicates a segment has left the network, so another may be sent. This keeps data flowing during recovery rather than stalling.
Two modern additions matter:
Tail Loss Probe (TLP) handles a case fast retransmit cannot: loss of the last segments of a transfer. With nothing after them, no duplicate ACKs are generated, so recovery falls to the RTO. TLP sends a probe after a short timer (roughly 2 × RTT) to elicit an ACK and trigger fast recovery instead.
RACK (Recent ACKnowledgment) replaces duplicate-ACK counting with time-based reasoning: if a segment was sent more than one RTT before something that has already been acknowledged, it is presumed lost. This is more robust under reordering and is now Linux's default.
Examples
Duplicate ACKs: a hole announces itself
sender receiver
seg 1 (1..1000) ----> ACK 1001
seg 2 (1001..2000) --X LOST
seg 3 (2001..3000) ---> ACK 1001 <- dup #1 (still wants 1001)
seg 4 (3001..4000) ---> ACK 1001 <- dup #2
seg 5 (4001..5000) ---> ACK 1001 <- dup #3 === TRIGGER ===
|
RETRANSMIT seg 2 immediately <-------+ (do NOT wait for the RTO)
seg 2 (1001..2000) ---> ACK 5001 <- one ACK covers everything buffered
Why three duplicates, not one
reordering is NORMAL (ECMP, multipath, parallel NIC queues):
sent: 1 2 3 4
arrived: 1 3 2 4 -> produces ONE duplicate ACK, no loss at all
reacting to 1 dup: constant spurious retransmission
reacting to 3 dups: tolerates common reordering, still ~1 RTT faster
than the RTO
Fast recovery vs timeout: the severity judgement
TIMEOUT (silence — the path may be gone):
ssthresh = cwnd / 2
cwnd = 1 MSS <-- COLLAPSE to the start
-> slow start
3 DUP ACKs (data still arriving — congested, not broken):
ssthresh = cwnd / 2
cwnd = ssthresh <-- HALVE, do not collapse
-> congestion avoidance
cwnd 40 -> timeout gives 1; 3 dup ACKs give 20.
Recovering from 1 takes many round trips. This is Reno's key advance.
Tail loss: the case fast retransmit cannot see
sender transmits the LAST 3 segments of a response, all lost.
No further segments exist -> NO duplicate ACKs are ever generated
-> fast retransmit cannot trigger -> wait for the full RTO.
TLP: after ~2 x RTT, send a probe segment to provoke an ACK
-> duplicate ACK arrives -> fast recovery engages
-> recovery in ~2 RTT instead of 200 ms.
class FastRetransmit:
DUP_THRESHOLD = 3
def __init__(self, cwnd=40, mss=1448):
self.cwnd = cwnd
self.mss = mss
self.ssthresh = 64
self.last_ack = 0
self.dup_count = 0
self.state = "congestion avoidance"
def on_ack(self, ack_number):
if ack_number == self.last_ack:
self.dup_count += 1
if self.dup_count == self.DUP_THRESHOLD:
return self._fast_retransmit()
if self.dup_count > self.DUP_THRESHOLD:
# Each dup means a segment LEFT the network: inflate and send more.
self.cwnd += 1
return f"dup #{self.dup_count}: cwnd inflated to {self.cwnd} (keep sending)"
return f"dup #{self.dup_count}: possible reordering, waiting"
# A new ACK ends recovery.
self.last_ack, self.dup_count = ack_number, 0
if self.state == "fast recovery":
self.cwnd = self.ssthresh
self.state = "congestion avoidance"
return f"recovery complete: cwnd deflated to ssthresh={self.cwnd}"
return f"new ACK {ack_number}: cwnd={self.cwnd}"
def _fast_retransmit(self):
self.ssthresh = max(2, self.cwnd // 2)
self.cwnd = self.ssthresh
self.state = "fast recovery"
return (f"*** FAST RETRANSMIT *** ssthresh={self.ssthresh}, "
f"cwnd HALVED to {self.cwnd} (not collapsed)")
def on_timeout(self):
self.ssthresh = max(2, self.cwnd // 2)
self.cwnd = 1
self.state = "slow start"
return (f"*** TIMEOUT *** ssthresh={self.ssthresh}, "
f"cwnd COLLAPSED to {self.cwnd} -> slow start")
fr = FastRetransmit(cwnd=40)
print("duplicate ACK sequence:")
for ack in (1001, 1001, 1001, 1001, 1001, 5001):
print(f" ACK {ack}: {fr.on_ack(ack)}")
# duplicate ACK sequence:
# ACK 1001: new ACK 1001: cwnd=40
# ACK 1001: dup #1: possible reordering, waiting
# ACK 1001: dup #2: possible reordering, waiting
# ACK 1001: dup #3: *** FAST RETRANSMIT *** ssthresh=20, cwnd HALVED to 20 (not collapsed)
# ACK 1001: dup #4: cwnd inflated to 21 (keep sending)
# ACK 5001: recovery complete: cwnd deflated to ssthresh=20
# The cost difference between the two recovery paths.
fr2 = FastRetransmit(cwnd=40)
print(f"\n{fr2.on_timeout()}")
# *** TIMEOUT *** ssthresh=20, cwnd COLLAPSED to 1 -> slow start
def recovery_rtts(cwnd_after, target, slow_start):
"""Round trips needed to get back to the target window."""
rtts, w = 0, cwnd_after
while w < target:
w = w * 2 if slow_start else w + 1
rtts += 1
return rtts
print()
for label, start, ss in [("after 3 dup ACKs (cwnd 20)", 20, False),
("after a timeout (cwnd 1)", 1, True)]:
print(f"{label}: {recovery_rtts(start, 40, ss)} RTTs to regain cwnd 40")
# after 3 dup ACKs (cwnd 20): 20 RTTs to regain cwnd 40
# after a timeout (cwnd 1): 6 RTTs to regain cwnd 40
# Slow start climbs faster, but it starts from 1 — and during those RTTs
# the connection is sending almost nothing. Throughput integrated over
# recovery is far worse.
def bytes_sent_during_recovery(start_cwnd, rtts, slow_start, mss=1448):
total, w = 0, start_cwnd
for _ in range(rtts):
total += w * mss
w = w * 2 if slow_start else w + 1
return total
print()
for label, start, ss in [("fast recovery", 20, False), ("timeout", 1, True)]:
b = bytes_sent_during_recovery(start, 6, ss)
print(f"{label:<16} sends {b:>9,} bytes over the next 6 RTTs")
# fast recovery sends 195,480 bytes over the next 6 RTTs
# timeout sends 91,224 bytes over the next 6 RTTs
# 2.1x more data delivered over the same 6 RTTs, and fast recovery never
# drops to a near-idle window. This is why Reno's halving mattered.
# Why the threshold is three: tolerating reordering.
def spurious_rate(threshold, reorder_depth_distribution):
"""Fraction of reordering events that would falsely trigger retransmit."""
return sum(p for depth, p in reorder_depth_distribution.items()
if depth >= threshold)
REORDERING = {1: 0.60, 2: 0.25, 3: 0.10, 4: 0.04, 5: 0.01}
print()
for th in (1, 2, 3, 4):
print(f"threshold {th} dup ACKs -> {spurious_rate(th, REORDERING):.0%} of "
f"reordering events cause a SPURIOUS retransmit")
# threshold 1 dup ACKs -> 100% of reordering events cause a SPURIOUS retransmit
# threshold 2 dup ACKs -> 40% of reordering events cause a SPURIOUS retransmit
# threshold 3 dup ACKs -> 15% of reordering events cause a SPURIOUS retransmit
# threshold 4 dup ACKs -> 5% of reordering events cause a SPURIOUS retransmit
# Three balances responsiveness against false alarms. RACK does better
# by reasoning about TIME instead of counting.
# Fast retransmit vs timeout — the ratio tells you which path recovery takes:
$ nstat -az | grep -iE "FastRetrans|Timeouts|LossProbe|Reord"
# TcpExtTCPFastRetrans 14201 <- fast path (good)
# TcpExtTCPTimeouts 2841 <- slow path (expensive)
# TcpExtTCPLossProbes 1204 <- TLP fired
# TcpExtTCPLossProbeRecovery 982 <- TLP successfully avoided an RTO
# TcpExtTCPSpuriousRtxHostQueues 41
$ python3 -c "print(f'fast path share: {14201/(14201+2841):.1%}')"
# fast path share: 83.3%
# Watch duplicate ACKs trigger a retransmission:
$ tshark -r capture.pcap -Y 'tcp.analysis.duplicate_ack_num >= 3' -T fields \
> -e frame.number -e tcp.ack -e tcp.analysis.duplicate_ack_num | head -3
# 4821 1001 3
# 4822 1001 4
$ tshark -r capture.pcap -Y 'tcp.analysis.fast_retransmission' | head -2
# RACK is the modern default — time-based, not dup-ACK counting:
$ sysctl net.ipv4.tcp_recovery
# net.ipv4.tcp_recovery = 1 # 1 = RACK enabled
$ sysctl net.ipv4.tcp_early_retrans
# net.ipv4.tcp_early_retrans = 3 # enables TLP
# Reordering tolerance — Linux tracks observed reordering per connection:
$ nstat -az | grep -i reorder
# TcpExtTCPTSReorder 412 <- detected via timestamps
# TcpExtTCPSACKReorder 1204 <- detected via SACK
# TcpExtTCPRenoReorder 12
$ ss -tin state established | grep -o "reordering:[0-9]*" | head -2
# reordering:3
# ^ this connection has seen reordering depth 3; the stack
# raises its dup-ACK threshold accordingly
# Demonstrate the difference: loss in the middle vs at the tail
$ sudo tc qdisc add dev eth0 root netem loss 1%
$ nstat -az | grep -iE "FastRetrans|Timeouts" # mostly fast retransmits
$ sudo tc qdisc del dev eth0 root
# Per-connection recovery state:
$ ss -tin state established | head -4
# ESTAB 0 0 10.1.1.5:51250 93.184.216.34:443
# cubic wscale:7,7 rto:204 rtt:3.8/1.2 mss:1448 cwnd:20 ssthresh:20
# ^^^^^^^ ^^^^^^^^^^
# cwnd == ssthresh: this
# connection recently halved
# retrans:0/24
Key takeaways
- Duplicate ACKs arise because acknowledgement is cumulative — an out-of-order arrival makes the receiver repeat the same ACK number.
- Duplicate ACKs signal "a hole, but data is still flowing", which is very different from the silence that triggers an RTO.
- Fast retransmit resends the missing segment after three duplicate ACKs, without waiting for the timer.
- The threshold is three to tolerate ordinary reordering from ECMP and multipath, which commonly produces one or two duplicates.
- Fast recovery halves
cwndtossthreshand continues in congestion avoidance, rather than collapsing to 1 MSS and slow-starting as a timeout does. - Each further duplicate ACK inflates the window, since each proves a segment has left the network.
- This halve-don't-collapse distinction is TCP Reno's key advance over Tahoe and delivers far more data during recovery.
- Tail Loss Probe handles loss of the final segments, where no duplicate ACKs can be generated.
- RACK replaces dup-ACK counting with time-based loss inference and is Linux's modern default.
🧪 Practice
- Explain why an out-of-order arrival produces a duplicate ACK rather than a new one.
- A connection with
cwnd60 experiences three duplicate ACKs, then later a timeout. Givecwndandssthreshafter each event.- Explain why the duplicate-ACK threshold is three rather than one, naming the network behaviour it tolerates.
- Interview-style: The last three segments of an HTTP response are lost. Explain why fast retransmit cannot help and what mechanism does. (Hint: fast retransmit needs the receiver to keep sending ACKs — ask what generates them when nothing follows the lost data.)
Selective Acknowledgment
Theory
Cumulative acknowledgement has a structural limitation (7.3): a receiver can only report the first gap. If it has bytes 1–1000 and 3001–10000, all it can say is "I want 1001". Everything it holds beyond the hole is invisible to the sender.
The consequence is severe when multiple segments are lost in one window. The sender knows a gap exists but not which segments arrived, so it must either retransmit everything from the gap onward — wasting bandwidth re-sending data the receiver already has — or recover one segment per round trip, which is agonisingly slow on a long path.
SACK (Selective Acknowledgment, RFC 2018) fixes this by letting the receiver report exactly which non-contiguous blocks it holds. It is a TCP option, negotiated during the handshake with SACK-permitted, and carried in ACKs as a list of left edge / right edge pairs.
ACK = 1001, SACK blocks: [3001-5000] [7001-9000]
^^^^ still the cumulative ACK ^^^^^^^^^^^^^^^^^^^^^^^^
(what SACK never replaces) what I additionally hold
The sender can now see that 1001–3000 and 5001–7000 are the actual holes, and retransmit precisely those.
The gains are concrete:
| Without SACK | With SACK | |
|---|---|---|
| Losses recovered per RTT | Typically one | All of them |
| Bytes retransmitted | Everything past the first hole | Exactly the missing blocks |
| Recovery time (3 losses, 100 ms RTT) | ~300 ms | ~100 ms |
Each SACK block costs 8 bytes (two 32-bit edges), and the TCP options field is limited to 40 bytes. With the timestamp option also present (10 bytes plus padding), only three SACK blocks fit. This is normally sufficient, since the receiver reports the most recently received blocks first.
D-SACK (Duplicate SACK, RFC 2883) extends the mechanism to report duplicate data. If the receiver gets a segment it already had, it reports that as the first SACK block. This tells the sender something valuable: the retransmission was unnecessary, meaning the original was merely delayed rather than lost. The sender can then recognise spurious retransmissions, undo an unwarranted congestion reduction, and adjust its reordering tolerance upward.
SACK is negotiated bilaterally — both ends must support it, and it is effectively universal on modern stacks. Its main historical caveats were CPU cost on servers tracking many blocks, and a denial-of-service vector (SACK Panic, CVE-2019-11477) where crafted SACK sequences could fragment kernel memory structures; both have been addressed.
Examples
The problem: cumulative ACK cannot describe what it holds
sent: [1-1000][1001-2000][2001-3000][3001-4000][4001-5000]
arrived: [1-1000] X [2001-3000] X [4001-5000]
LOST LOST
WITHOUT SACK:
receiver can only say: ACK 1001
sender sees: "something after 1000 is missing" — but WHAT?
option A: retransmit 1001..5000 -> 4000 bytes, 2000 wasted
option B: retransmit 1001..2000, wait an RTT, learn about the next hole
-> 2 RTTs for 2 losses
WITH SACK:
receiver says: ACK 1001, SACK [2001-3000] [4001-5000]
sender computes the holes: 1001-2000 and 3001-4000
retransmits EXACTLY those, in ONE round trip.
SACK option layout (each block is 8 bytes)
+--------+--------+----------------------+----------------------+
| kind=5 | len=18 | left edge (4 bytes) | right edge (4 bytes) | block 1
+--------+--------+----------------------+----------------------+
| left edge | right edge | block 2
+----------------------+----------------------+
40-byte options field:
timestamps (10) + NOP padding (2) = 12
remaining 28 bytes -> 2 header bytes + 3 blocks x 8 = 26 -> 3 BLOCKS MAX
D-SACK: reporting data you already had
sender retransmits 1001-2000 (it thought it was lost)
receiver already had it
receiver replies: ACK 5001, SACK [1001-2000]
^^^^^^^^^^ BELOW the cumulative ACK
-> this is a D-SACK: "duplicate"
sender learns: that retransmission was UNNECESSARY.
The original was merely delayed, not lost.
-> undo the congestion window reduction
-> raise the reordering threshold
def find_holes(cumulative_ack, sack_blocks, highest_sent):
"""Compute exactly which byte ranges are missing."""
holes, pos = [], cumulative_ack
for left, right in sorted(sack_blocks):
if pos < left:
holes.append((pos, left - 1))
pos = max(pos, right + 1)
if pos <= highest_sent:
holes.append((pos, highest_sent))
return holes
ack = 1001
sack = [(2001, 3000), (4001, 5000)]
holes = find_holes(ack, sack, highest_sent=5000)
print(f"cumulative ACK : {ack}")
print(f"SACK blocks : {sack}")
print(f"computed holes : {holes}")
print(f"bytes to resend: {sum(r - l + 1 for l, r in holes)}")
# cumulative ACK : 1001
# SACK blocks : [(2001, 3000), (4001, 5000)]
# computed holes : [(1001, 2000), (3001, 4000)]
# bytes to resend: 2000
def without_sack(cumulative_ack, highest_sent):
return [(cumulative_ack, highest_sent)]
no_sack = without_sack(1001, 5000)
print(f"\nwithout SACK : resend {no_sack} = "
f"{sum(r - l + 1 for l, r in no_sack)} bytes")
print(f"wasted : {4000 - 2000} bytes of data the receiver already had")
# without SACK : resend [(1001, 5000)] = 4000 bytes
# wasted : 2000 bytes of data the receiver already had
# Recovery time: one loss per RTT versus all at once.
def recovery_time(losses, rtt_ms, sack):
rtts = 1 if sack else losses
return rtts * rtt_ms, rtts
print()
print(f"{'losses':>8}{'RTT':>7}{'no SACK':>12}{'with SACK':>12}{'speedup':>10}")
for losses in (1, 3, 8):
for rtt in (100,):
t_no, r_no = recovery_time(losses, rtt, sack=False)
t_ys, r_ys = recovery_time(losses, rtt, sack=True)
print(f"{losses:>8}{rtt:>5}ms{t_no:>10}ms{t_ys:>10}ms{t_no/t_ys:>9.0f}x")
# losses RTT no SACK with SACK speedup
# 1 100ms 100ms 100ms 1x
# 3 100ms 300ms 100ms 3x
# 8 100ms 800ms 100ms 8x
# How many SACK blocks fit in the options field?
def sack_capacity(timestamps=True):
options = 40
used = (10 + 2) if timestamps else 0 # timestamp + NOP padding
remaining = options - used
header = 2 # kind + length
return (remaining - header) // 8
print()
for ts in (False, True):
print(f"timestamps={ts!s:<5} -> {sack_capacity(ts)} SACK blocks fit "
f"in the 40-byte options field")
# timestamps=False -> 4 SACK blocks fit in the 40-byte options field
# timestamps=True -> 3 SACK blocks fit in the 40-byte options field
# D-SACK: detecting a retransmission that was not needed.
def interpret_sack(cumulative_ack, first_block):
left, right = first_block
if right < cumulative_ack:
return ("D-SACK", f"bytes {left}-{right} were DUPLICATES — that "
f"retransmission was unnecessary; the original was "
f"delayed, not lost")
return ("normal SACK", f"bytes {left}-{right} received out of order")
print()
for ack, block in [(5001, (1001, 2000)), (1001, (2001, 3000))]:
kind, note = interpret_sack(ack, block)
print(f"ACK {ack}, first block {block} -> {kind}")
print(f" {note}")
# ACK 5001, first block (1001, 2000) -> D-SACK
# bytes 1001-2000 were DUPLICATES — that retransmission was unnecessary; the original was delayed, not lost
# ACK 1001, first block (2001, 3000) -> normal SACK
# bytes 2001-3000 received out of order
def on_dsack(cwnd, ssthresh_before_reduction):
"""A spurious retransmission means the window cut was unwarranted."""
return {"cwnd_restored": ssthresh_before_reduction,
"reordering_threshold": "increased",
"note": "undo the congestion response; the path reorders, it does not drop"}
print()
for k, v in on_dsack(20, 40).items():
print(f" {k:<24}{v}")
# cwnd_restored 40
# reordering_threshold increased
# note undo the congestion response; the path reorders, it does not drop
# SACK is negotiated in the handshake — look for 'sackOK':
$ sudo tcpdump -i eth0 -n 'tcp[tcpflags] & tcp-syn != 0' -v -c 1
# Flags [S], seq 2712847316, win 64240,
# options [mss 1460,sackOK,TS val 184221 ecr 0,nop,wscale 7]
# ^^^^^^ SACK permitted; both ends must offer it
# SACK blocks in an ACK:
$ sudo tcpdump -i eth0 -n 'tcp port 443' -v | grep -i "sack" | head -3
# IP 93.184.216.34.443 > 10.1.1.5.51250: Flags [.], ack 1001,
# options [nop,nop,sack 2 {2001:3000}{4001:5000}]
# ^^^^^^ 2 blocks: exactly what the receiver holds
$ tshark -r capture.pcap -Y 'tcp.options.sack' -T fields \
> -e tcp.ack -e tcp.options.sack_le -e tcp.options.sack_re | head -3
# 1001 2001,4001 3000,5000
# Enable/verify SACK and D-SACK:
$ sysctl net.ipv4.tcp_sack net.ipv4.tcp_dsack net.ipv4.tcp_fack
# net.ipv4.tcp_sack = 1
# net.ipv4.tcp_dsack = 1
# net.ipv4.tcp_fack = 0 # FACK is obsolete; RACK supersedes it
# Counters showing SACK doing its job:
$ nstat -az | grep -iE "sack|dsack"
# TcpExtTCPSACKReorder 1204 <- reordering detected via SACK
# TcpExtTCPSACKRecovery 8412 <- recoveries driven by SACK
# TcpExtTCPSACKDiscard 12
# TcpExtTCPDSACKOldSent 412 <- D-SACK sent: we got a duplicate
# TcpExtTCPDSACKRecv 389 <- D-SACK received: our retransmit
# was unnecessary
# TcpExtTCPDSACKUndo 201 <- congestion reduction UNDONE
# ^^^^^ D-SACK recovering throughput
# Per-connection SACK state:
$ ss -tin state established | head -4
# ESTAB 0 0 10.1.1.5:51250 93.184.216.34:443
# cubic wscale:7,7 rto:204 rtt:3.8/1.2 mss:1448 cwnd:20
# sacked:3 retrans:0/24 reordering:3
# ^^^^^^^^ 3 segments currently SACKed (held by the receiver beyond a hole)
# Compare recovery with SACK on and off, under multiple losses:
$ sudo tc qdisc add dev eth0 root netem loss 3%
$ sudo sysctl -w net.ipv4.tcp_sack=0
$ curl -o /dev/null -w 'no SACK: %{time_total}s\n' -s http://server/50mb.bin
# no SACK: 62.418s
$ sudo sysctl -w net.ipv4.tcp_sack=1
$ curl -o /dev/null -w 'with SACK: %{time_total}s\n' -s http://server/50mb.bin
# with SACK: 21.104s
$ sudo tc qdisc del dev eth0 root
Key takeaways
- Cumulative acknowledgement can report only the first gap, hiding everything the receiver holds beyond it.
- SACK (RFC 2018) lets the receiver report non-contiguous blocks it has received, as left/right edge pairs in a TCP option.
- SACK supplements the cumulative ACK; it never replaces it.
- The sender computes the exact holes and retransmits only those, instead of everything past the gap.
- Recovery goes from one loss per RTT to all losses in one RTT — an 8x speedup for 8 losses.
- Each block costs 8 bytes, so only 3 blocks fit alongside the timestamp option in the 40-byte options field.
- D-SACK (RFC 2883) reports duplicate data, letting the sender detect a spurious retransmission, undo the unwarranted congestion reduction, and raise its reordering tolerance.
- SACK is negotiated with SACK-permitted in the handshake and is effectively universal today.
🧪 Practice
- Given
ACK 2001, SACK [4001-6000] [8001-9000]with 9,000 bytes sent, compute the missing ranges and the bytes to retransmit with and without SACK.- Explain why only three SACK blocks fit when the timestamp option is enabled.
- Describe what a D-SACK tells the sender and the two adjustments it should make in response.
- Interview-style: A connection over a satellite link with 600 ms RTT loses four segments in one window. Compare recovery time with and without SACK. (Hint: without SACK the sender learns about one hole per round trip — multiply.)
<a id="74-flow-control"></a>
7.4 Flow Control
This subchapter covers how a receiver stops a fast sender from overwhelming it — a problem entirely separate from congestion, and solved by a different mechanism.
Receive Window Advertisement
Theory
A fast server talking to a slow client — an embedded device, a phone under load, an application that stopped reading — will overwhelm it if nothing intervenes. The receiver's buffer fills, and further data has nowhere to go.
Flow control prevents this, and it is essential to distinguish it from congestion control (7.5), because the two are constantly confused:
| Flow control | Congestion control | |
|---|---|---|
| Protects | The receiver | The network |
| Problem | Receiver buffer overflow | Router queue overflow |
| Signal | Explicit — the advertised window | Implicit — loss, delay, ECN |
| Mechanism | rwnd in every ACK | cwnd, computed by the sender |
| Who decides | The receiver | The sender |
A sender must obey both. The amount of unacknowledged data it may have outstanding is:
in_flight <= min(cwnd, rwnd)
The mechanism is simple and elegant: every ACK carries a Window field stating how many more bytes the receiver can currently accept. The sender may not have more than that outstanding. As the application reads data out of the buffer, space frees up and subsequent ACKs advertise a larger window.
If the application stops reading entirely, the buffer fills and the receiver advertises window = 0. The sender must stop sending data — completely.
That creates a deadlock risk worth understanding. The sender is blocked waiting for a window update; the update is carried in an ACK; but with no data flowing, no ACKs are generated. If the window-update ACK is then lost, both sides wait forever.
The fix is the persist timer and zero window probes. When the window is zero, the sender periodically sends a one-byte segment. The receiver must respond with an ACK, which carries the current window. If the window is still zero, nothing is lost; if it has opened, the sender learns immediately. The deadlock cannot form.
The Window field is 16 bits, so the maximum advertisable window is 65,535 bytes — which was generous in 1981 and is badly insufficient today. On a 100 ms path, 64 KB per RTT caps throughput at roughly 5 Mb/s regardless of available bandwidth. This is the bandwidth-delay product limit, and the fix is the window scaling option (covered later in this subchapter).
One security note: the advertised window also bounds which sequence numbers a receiver will accept (7.3). A very large window makes blind off-path injection easier, which is why stacks rate-limit challenge ACKs.
Examples
The window as explicit backpressure
receiver buffer: 64 KB total
app reading promptly:
buffer: [####............................] 16 KB used
ACK advertises window = 48 KB -> sender may have 48 KB outstanding
app slows down:
buffer: [########################........] 48 KB used
ACK advertises window = 16 KB -> sender throttles
app stops reading:
buffer: [################################] 64 KB used
ACK advertises window = 0 -> sender STOPS COMPLETELY
Zero-window deadlock, and the probe that prevents it
sender receiver
|<---- ACK win=0 ---------------------| buffer full; stop
| (sender blocked) |
| | app reads 32 KB; space free
|<---- ACK win=32768 -----X LOST |
| |
| sender waits for a window update | receiver waits for data
| ... FOREVER ... | ... FOREVER ...
^^^^^^^^^^^^^^^^^ DEADLOCK
WITH THE PERSIST TIMER:
|---- 1-byte probe ------------------>|
|<---- ACK win=32768 -----------------| the receiver MUST answer
| sender resumes | deadlock impossible
Both limits apply, always
cwnd = 100 KB (what the NETWORK will bear)
rwnd = 16 KB (what the RECEIVER will accept)
in_flight <= min(100, 16) = 16 KB
The receiver is the bottleneck here. Increasing cwnd changes nothing —
a common misdiagnosis when throughput is low.
class ReceiveBuffer:
def __init__(self, capacity=65535):
self.capacity = capacity
self.used = 0
def receive(self, nbytes):
accepted = min(nbytes, self.capacity - self.used)
self.used += accepted
dropped = nbytes - accepted
return accepted, dropped
def app_reads(self, nbytes):
self.used -= min(nbytes, self.used)
def window(self):
return self.capacity - self.used
buf = ReceiveBuffer(65535)
print(f"{'event':<28}{'buffer used':>13}{'advertised window':>20}")
for label, action in [
("initial", lambda: None),
("receive 16 KB", lambda: buf.receive(16384)),
("receive 32 KB more", lambda: buf.receive(32768)),
("app reads 8 KB", lambda: buf.app_reads(8192)),
("receive 24 KB more", lambda: buf.receive(24576)),
("app stops; receive 16 KB",lambda: buf.receive(16384)),
]:
action()
w = buf.window()
flag = " <- ZERO WINDOW: sender must STOP" if w == 0 else ""
print(f"{label:<28}{buf.used:>13,}{w:>20,}{flag}")
# event buffer used advertised window
# initial 0 65,535
# receive 16 KB 16,384 49,151
# receive 32 KB more 49,152 16,383
# app reads 8 KB 40,960 24,575
# receive 24 KB more 65,535 0 <- ZERO WINDOW: sender must STOP
# app stops; receive 16 KB 65,535 0 <- ZERO WINDOW: sender must STOP
# A sender must obey BOTH limits.
def allowed_in_flight(cwnd, rwnd):
limit = min(cwnd, rwnd)
bottleneck = "receiver (flow control)" if rwnd < cwnd else "network (congestion control)"
return limit, bottleneck
print()
print(f"{'cwnd':>10}{'rwnd':>10}{'allowed':>10} bottleneck")
for cwnd, rwnd in [(102400, 16384), (16384, 102400), (65535, 65535)]:
limit, who = allowed_in_flight(cwnd, rwnd)
print(f"{cwnd:>10,}{rwnd:>10,}{limit:>10,} {who}")
# cwnd rwnd allowed bottleneck
# 102,400 16,384 16,384 receiver (flow control)
# 16,384 102,400 16,384 network (congestion control)
# 65,535 65,535 65,535 network (congestion control)
# The persist timer: why deadlock cannot form.
class PersistTimer:
def __init__(self, initial=5.0, maximum=60.0):
self.interval, self.maximum = initial, maximum
def probe(self, receiver_window):
result = ("window still 0 — keep probing" if receiver_window == 0
else f"window OPENED to {receiver_window} — resume sending")
if receiver_window == 0:
self.interval = min(self.maximum, self.interval * 2)
return result, self.interval
pt = PersistTimer()
print()
for w in (0, 0, 0, 32768):
result, nxt = pt.probe(w)
print(f" 1-byte probe -> receiver says win={w:<6} {result:<40} next probe in {nxt:>4.0f}s")
# 1-byte probe -> receiver says win=0 window still 0 — keep probing next probe in 10s
# 1-byte probe -> receiver says win=0 window still 0 — keep probing next probe in 20s
# 1-byte probe -> receiver says win=0 window still 0 — keep probing next probe in 40s
# 1-byte probe -> receiver says win=32768 window OPENED to 32768 — resume sending next probe in 40s
# The 16-bit window limit: why scaling became mandatory.
def max_throughput(window_bytes, rtt_ms):
return window_bytes * 8 / (rtt_ms / 1000) / 1e6 # Mb/s
print()
print(f"{'RTT':>8}{'64 KB window':>16}{'link capacity':>16}{'utilisation':>14}")
for rtt, link in [(1, 10_000), (10, 1_000), (100, 1_000), (300, 1_000)]:
t = max_throughput(65535, rtt)
print(f"{rtt:>6}ms{t:>13,.1f} Mb/s{link:>12,} Mb/s{min(100, 100*t/link):>13.1f}%")
# RTT 64 KB window link capacity utilisation
# 1ms 524.3 Mb/s 10,000 Mb/s 5.2%
# 10ms 52.4 Mb/s 1,000 Mb/s 5.2%
# 100ms 5.2 Mb/s 1,000 Mb/s 0.5%
# 300ms 1.7 Mb/s 1,000 Mb/s 0.2%
# A 1 Gb/s transatlantic link delivers 5 Mb/s without window scaling.
# How large a window is actually needed?
def required_window(bandwidth_mbps, rtt_ms):
return bandwidth_mbps * 1e6 / 8 * (rtt_ms / 1000)
print()
for bw, rtt, label in [(1000, 100, "1 Gb/s transatlantic"),
(10000, 1, "10 Gb/s data centre"),
(100, 600, "100 Mb/s satellite")]:
need = required_window(bw, rtt)
print(f"{label:<24}needs {need/1024:>9,.0f} KB "
f"({'fits in 16 bits' if need <= 65535 else 'REQUIRES WINDOW SCALING'})")
# 1 Gb/s transatlantic needs 12,207 KB (REQUIRES WINDOW SCALING)
# 10 Gb/s data centre needs 1,221 KB (REQUIRES WINDOW SCALING)
# 100 Mb/s satellite needs 7,324 KB (REQUIRES WINDOW SCALING)
# The advertised window in flight:
$ sudo tcpdump -i eth0 -n 'tcp port 443' -c 3
# IP 93.184.216.34.443 > 10.1.1.5.51250: Flags [.], ack 2897, win 501, length 0
# ^^^^^^^ SCALED:
# multiply by 2^wscale from the handshake. win 501 with wscale 7 = 64,128 bytes.
# Find the scale factor (negotiated in the SYN, applies to all later segments):
$ sudo tcpdump -i eth0 -n 'tcp[tcpflags] & tcp-syn != 0' -v -c 1 | grep -o "wscale [0-9]*"
# wscale 7
# Zero-window events — a receiver that cannot keep up:
$ nstat -az | grep -iE "zerowindow|persist"
# TcpExtTCPZeroWindowDrop 412
# TcpExtTCPWantZeroWindowAdv 1204 <- receiver wanted to advertise 0
$ tshark -r capture.pcap -Y 'tcp.analysis.zero_window' | head -3
$ tshark -r capture.pcap -Y 'tcp.analysis.window_full' | head -3
# Find a stalled receiver — Recv-Q full means the APPLICATION is not reading:
$ ss -tan state established | awk '$2 > 0 {print}' | head -3
# ESTAB 262144 0 10.1.1.5:8080 10.2.2.9:51234
# ^^^^^^ 256 KB sitting unread. The application is the bottleneck,
# not the network.
# Per-connection window state:
$ ss -tin state established | head -4
# ESTAB 0 0 10.1.1.5:51250 93.184.216.34:443
# cubic wscale:7,7 rto:204 rtt:3.8/1.2 mss:1448 cwnd:10
# rcv_space:14480 rcv_ssthresh:64088 snd_wnd:64128
# ^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^ what the PEER advertised
# to us
# Buffer sizes — the kernel autotunes between min and max:
$ sysctl net.ipv4.tcp_rmem net.core.rmem_max
# net.ipv4.tcp_rmem = 4096 131072 6291456
# min default max
# net.core.rmem_max = 212992
$ sudo sysctl -w net.core.rmem_max=16777216
$ sudo sysctl -w net.ipv4.tcp_rmem="4096 131072 16777216"
# Autotuning must stay ON — it sizes the window to the measured BDP:
$ sysctl net.ipv4.tcp_moderate_rcvbuf
# net.ipv4.tcp_moderate_rcvbuf = 1
# Setting SO_RCVBUF manually DISABLES autotuning. Usually a mistake.
# Distinguish a flow-control stall from a congestion stall:
$ ss -tin state established | grep -E "cwnd|snd_wnd" | head -2
# cwnd:120 snd_wnd:16384
# ^^^ large ^^^^^ small -> the RECEIVER is limiting us.
# Raising congestion-control aggressiveness would change nothing.
Key takeaways
- Flow control protects the receiver; congestion control protects the network — different problems, different mechanisms.
- A sender's outstanding data is bounded by
min(cwnd, rwnd); it must obey both. - The receiver advertises
rwndin the Window field of every ACK — an explicit signal, unlike congestion control's implicit ones. - Window = 0 stops the sender completely; the persist timer and zero-window probes prevent the resulting deadlock if a window update is lost.
- The Window field is 16 bits, capping the raw window at 65,535 bytes — roughly 5 Mb/s on a 100 ms path, regardless of link capacity.
- Modern paths need far larger windows (a 1 Gb/s transatlantic link needs ~12 MB), which requires window scaling.
- A full Recv-Q means the application is not reading, not that the network is slow.
- Setting
SO_RCVBUFmanually disables receive-buffer autotuning, which is usually counterproductive.
🧪 Practice
- State which entity flow control protects and which congestion control protects, and give the formula bounding in-flight data.
- Compute the maximum throughput of a 64 KB window on RTTs of 10 ms, 80 ms, and 250 ms.
- Describe the zero-window deadlock and the mechanism that prevents it.
- Interview-style: A transfer is slow.
ssshowscwnd:200butsnd_wnd:8192. Where is the bottleneck and what would you investigate? (Hint: compare the two limits and ask which one is binding — then ask what makes a receiver advertise a small window.)
Sliding Window in TCP
Theory
A sender that transmits one segment and waits for its acknowledgement before sending the next — stop-and-wait — wastes almost all available capacity. On a 100 ms path with 1,460-byte segments, it achieves about 117 kb/s no matter how fast the link is, because it spends an entire round trip idle for every segment.
The sliding window fixes this by allowing multiple segments in flight simultaneously. The window is the range of bytes the sender may have outstanding, and it slides forward as acknowledgements arrive.
The sender's sequence space divides into four regions:
1. sent and ACKed 2. sent, not ACKed 3. may send now 4. cannot send yet
|--------------------|---------------------|------------------|------------------
^ ^ ^
SND.UNA SND.NXT SND.UNA + window
|<------------- the WINDOW ------->|
- SND.UNA — the oldest unacknowledged byte; the window's left edge.
- SND.NXT — the next byte to send.
- Usable window =
SND.UNA + min(cwnd, rwnd) - SND.NXT.
When an ACK arrives, SND.UNA advances and the window slides, making new bytes sendable. The window's size is set by flow and congestion control; its position is set by acknowledgements.
The throughput this enables is governed by the bandwidth-delay product (BDP):
BDP = bandwidth x RTT
The BDP is the amount of data "in the pipe" at any instant. To keep a link busy, the window must be at least the BDP — otherwise the sender runs out of permitted data and stalls until an ACK returns. The relationship is direct:
throughput = window / RTT (when window < BDP)
Three rules of thumb follow, and they explain most bulk-transfer performance questions:
Window smaller than BDP → the sender stalls each RTT; throughput is window-limited, and adding bandwidth changes nothing. Window equal to BDP → the link is exactly saturated with no queueing. This is the optimum. Window much larger than BDP → the excess accumulates in router buffers, adding delay without adding throughput. This is bufferbloat (7.5).
That last point is the one people miss: a bigger window is not always better. Beyond the BDP, extra window converts directly into queueing delay.
Two window edges can move independently, which produces the two classic anomalies:
The left edge advances when data is acknowledged — the window slides. The right edge advances when the receiver frees buffer space — the window opens. The right edge must never move left ("shrinking the window"), because the sender may already have transmitted data into that space. Receivers may advertise a smaller window only by letting the left edge catch up.
Examples
Stop-and-wait versus sliding window
STOP-AND-WAIT (window = 1 segment):
|--seg1-->| t=0
| |
|<--ACK---| t=RTT
|--seg2-->| t=RTT
throughput = 1460 bytes / 0.1 s = 117 kb/s on ANY link speed
SLIDING WINDOW (window = 10 segments):
|--seg1-->|--seg2-->|...|--seg10-->| t=0, all sent immediately
|<--ACK1--| t=RTT, window slides
|--seg11->| a new segment becomes sendable
throughput = 14600 bytes / 0.1 s = 1.17 Mb/s 10x, and it scales further
The four regions of the sender's sequence space
bytes: 1 ....... 5000 ....... 12000 ....... 20000 ....... 30000
| ACKed | in flight | may send | not yet |
^ ^ ^
SND.UNA SND.NXT SND.UNA + window
(5001) (12001) (20001)
usable window = 20001 - 12001 = 8000 bytes may be sent right now
ACK 9000 arrives:
SND.UNA -> 9001, the whole window SLIDES right by 4000
usable window becomes 24001 - 12001 = 12000 bytes
Window versus BDP — the three regimes
BDP = 1 Gb/s x 100 ms = 12.5 MB
window 64 KB << BDP: send 64 KB, then WAIT ~99 ms -> 5 Mb/s (0.5%)
window 12.5 MB = BDP: pipe exactly full, no queueing -> 1 Gb/s (100%)
window 50 MB >> BDP: 37.5 MB piles up in router buffers
-> still 1 Gb/s, but +300 ms of delay
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
BUFFERBLOAT: more window, same throughput,
much worse latency
class SlidingWindow:
def __init__(self, window_size, mss=1460):
self.snd_una = 1 # oldest unacknowledged byte
self.snd_nxt = 1 # next byte to send
self.window = window_size
self.mss = mss
def usable(self):
"""How many bytes may be sent right now."""
return max(0, self.snd_una + self.window - self.snd_nxt)
def send(self):
if self.usable() < self.mss:
return None, f"STALLED — usable window {self.usable()} < MSS"
seq = self.snd_nxt
self.snd_nxt += self.mss
return seq, f"sent seq {seq}..{seq + self.mss - 1}"
def ack(self, ack_number):
freed = ack_number - self.snd_una
self.snd_una = ack_number
return f"ACK {ack_number}: window slid right by {freed} bytes"
def state(self):
return (f"UNA={self.snd_una:<6} NXT={self.snd_nxt:<6} "
f"in_flight={self.snd_nxt - self.snd_una:<6} usable={self.usable()}")
w = SlidingWindow(window_size=5840) # 4 segments
print(f"initial: {w.state()}")
for _ in range(5):
seq, note = w.send()
print(f" {note:<32}{w.state()}")
print(f" {w.ack(2921):<32}{w.state()}")
for _ in range(2):
seq, note = w.send()
print(f" {note:<32}{w.state()}")
# initial: UNA=1 NXT=1 in_flight=0 usable=5840
# sent seq 1..1460 UNA=1 NXT=1461 in_flight=1460 usable=4380
# sent seq 1461..2920 UNA=1 NXT=2921 in_flight=2920 usable=2920
# sent seq 2921..4380 UNA=1 NXT=4381 in_flight=4380 usable=1460
# sent seq 4381..5840 UNA=1 NXT=5841 in_flight=5840 usable=0
# STALLED — usable window 0 < MSS UNA=1 NXT=5841 in_flight=5840 usable=0
# ACK 2921: window slid right by 2920 UNA=2921 NXT=5841 in_flight=2920 usable=2920
# sent seq 5841..7300 UNA=2921 NXT=7301 in_flight=4380 usable=1460
# sent seq 7301..8760 UNA=2921 NXT=8761 in_flight=5840 usable=0
# Bandwidth-delay product and the three regimes.
def bdp(bandwidth_mbps, rtt_ms):
return bandwidth_mbps * 1e6 / 8 * (rtt_ms / 1000)
def throughput(window_bytes, rtt_ms, link_mbps):
window_limited = window_bytes * 8 / (rtt_ms / 1000) / 1e6
return min(window_limited, link_mbps)
print()
print(f"{'scenario':<26}{'BDP':>12}{'window':>12}{'throughput':>14}{'regime':>16}")
SCENARIOS = [
("LAN 1 Gb/s, 1 ms", 1000, 1, 65535),
("national 1 Gb/s, 30 ms", 1000, 30, 65535),
("transatlantic, 100 ms", 1000, 100, 65535),
("transatlantic, scaled", 1000, 100, 12_500_000),
("transatlantic, oversized",1000, 100, 50_000_000),
]
for label, bw, rtt, win in SCENARIOS:
b = bdp(bw, rtt)
t = throughput(win, rtt, bw)
regime = ("window-limited" if win < b * 0.95 else
"OPTIMAL" if win <= b * 1.5 else "BUFFERBLOAT")
print(f"{label:<26}{b/1024:>9,.0f} KB{win/1024:>9,.0f} KB{t:>11,.1f} Mb/s{regime:>16}")
# scenario BDP window throughput regime
# LAN 1 Gb/s, 1 ms 122 KB 64 KB 524.3 Mb/s window-limited
# national 1 Gb/s, 30 ms 3,662 KB 64 KB 17.5 Mb/s window-limited
# transatlantic, 100 ms 12,207 KB 64 KB 5.2 Mb/s window-limited
# transatlantic, scaled 12,207 KB 12,207 KB 1,000.0 Mb/s OPTIMAL
# transatlantic, oversized 12,207 KB 48,828 KB 1,000.0 Mb/s BUFFERBLOAT
# Why an oversized window does not help — and does hurt.
def queueing_delay(window_bytes, bdp_bytes, bandwidth_mbps):
excess = max(0, window_bytes - bdp_bytes)
return excess * 8 / (bandwidth_mbps * 1e6) * 1000 # ms
print()
b = bdp(1000, 100)
for win_mb in (12.5, 25, 50):
win = win_mb * 1e6
d = queueing_delay(win, b, 1000)
print(f"window {win_mb:>5.1f} MB -> throughput 1000 Mb/s, "
f"added queueing delay {d:>6.1f} ms")
# window 12.5 MB -> throughput 1000 Mb/s, added queueing delay 0.0 ms
# window 25.0 MB -> throughput 1000 Mb/s, added queueing delay 100.0 ms
# window 50.0 MB -> throughput 1000 Mb/s, added queueing delay 300.0 ms
# Identical throughput; 300 ms of pure added latency. More window is
# NOT better past the BDP.
# Stop-and-wait versus a full window.
def stop_and_wait(mss, rtt_ms):
return mss * 8 / (rtt_ms / 1000) / 1e6
print()
for rtt in (1, 10, 100):
sw = stop_and_wait(1460, rtt)
win = throughput(65535, rtt, 10_000)
print(f"RTT {rtt:>4} ms: stop-and-wait {sw:>8.2f} Mb/s "
f"64 KB window {win:>9.1f} Mb/s ({win/sw:.0f}x)")
# RTT 1 ms: stop-and-wait 11.68 Mb/s 64 KB window 524.3 Mb/s (45x)
# RTT 10 ms: stop-and-wait 1.17 Mb/s 64 KB window 52.4 Mb/s (45x)
# RTT 100 ms: stop-and-wait 0.12 Mb/s 64 KB window 5.2 Mb/s (45x)
# Watch the window slide — in-flight bytes rise and fall:
$ ss -tin state established | head -4
# ESTAB 0 0 10.1.1.5:51250 93.184.216.34:443
# cubic wscale:7,7 rto:204 rtt:3.8/1.2 mss:1448 cwnd:120
# bytes_sent:18422100 bytes_acked:18404800 unacked:12
# ^^^^^^^^^^ segments in flight
# Measure the BDP of a real path:
$ ping -c5 -q server | tail -1
# rtt min/avg/max/mdev = 98.204/99.112/100.441/0.812 ms
$ iperf3 -c server -t 10 | tail -3
# [ 5] 0.00-10.00 sec 1.09 GBytes 938 Mbits/sec
$ python3 -c "
bw, rtt = 938e6, 0.099
bdp = bw/8*rtt
print(f'BDP = {bdp/1e6:.1f} MB — the window must be at least this')"
# BDP = 11.6 MB — the window must be at least this
# Is the connection window-limited? The definitive check:
$ ss -tin state established | grep -oE "cwnd:[0-9]+|snd_wnd:[0-9]+|unacked:[0-9]+"
# cwnd:120
# snd_wnd:64128
# unacked:44
# unacked (44 x 1448 = 63 KB) ~ snd_wnd -> RECEIVER-limited.
# If unacked ~ cwnd x mss instead, it is congestion-limited.
$ nstat -az | grep -iE "sndbuflimited|rcvbuf|windowprobe"
# TcpExtTCPWqueueTooBig 0
# TcpExtTCPSndBufLimited 412 <- the SENDER's buffer was the limit
# (raise net.ipv4.tcp_wmem)
# Size buffers for the BDP — both ends must be large enough:
$ sudo sysctl -w net.core.rmem_max=33554432
$ sudo sysctl -w net.core.wmem_max=33554432
$ sudo sysctl -w net.ipv4.tcp_rmem="4096 131072 33554432"
$ sudo sysctl -w net.ipv4.tcp_wmem="4096 65536 33554432"
# Demonstrate the window limit directly:
$ iperf3 -c server -w 64K -t 10 | tail -2
# [ 5] 0.00-10.00 sec 6.25 MBytes 5.24 Mbits/sec <- window-limited
$ iperf3 -c server -w 16M -t 10 | tail -2
# [ 5] 0.00-10.00 sec 1.09 GBytes 938 Mbits/sec <- BDP-sized
# Same link, same path. 180x difference from the window alone.
Key takeaways
- Stop-and-wait wastes a full RTT per segment; the sliding window allows many segments in flight at once.
- The sender's space divides into ACKed, in flight, sendable now, and not yet sendable; SND.UNA and SND.NXT are the key pointers.
- The window's size comes from
min(cwnd, rwnd); its position advances as ACKs arrive. - BDP = bandwidth × RTT is the data "in the pipe"; the window must be at least the BDP to keep a link busy.
throughput = window / RTTwhenever the window is below the BDP — adding bandwidth then changes nothing.- A window larger than the BDP adds queueing delay, not throughput — this is bufferbloat.
- The window's right edge must never move left; receivers shrink a window only by letting the left edge catch up.
- Diagnose by comparing
unackedagainstsnd_wndandcwnd × mssto see which limit is binding.
🧪 Practice
- Compute the BDP for a 500 Mb/s link with 40 ms RTT, and the throughput achievable with a 64 KB window.
- Given SND.UNA 10001, SND.NXT 16001, and a 12,000-byte window, compute the usable window, then recompute after ACK 14001 arrives.
- Explain why a window four times the BDP delivers no more throughput, and what it does produce instead.
- Interview-style: A transfer between two data centres achieves 50 Mb/s on a 10 Gb/s link with 40 ms RTT. Bandwidth tests show the link is clear. Diagnose. (Hint: compute the window that 50 Mb/s at 40 ms implies, and compare it with a familiar 16-bit limit.)
Silly Window Syndrome
Theory
Silly Window Syndrome (SWS) is a pathology where TCP degenerates into exchanging tiny segments, so that header overhead dominates and throughput collapses. It can be caused by either endpoint, and the fixes differ.
Receiver-caused SWS. A receiver's buffer is full, so it advertises a zero window. The application then reads a small amount — say 1 byte. The receiver dutifully advertises a window of 1. The sender, permitted to send 1 byte, sends a segment with 40 bytes of header and 1 byte of payload. The buffer fills again, the window returns to zero, and the cycle repeats. Efficiency is 2.4%.
Sender-caused SWS. An application writes tiny amounts — one keystroke at a time, or a byte per loop iteration. TCP sends each as its own segment, again with 40 bytes of header per byte of data.
The problem in both cases is the same: the segment is mostly header. The fixes are correspondingly symmetric.
Receiver side — David Clark's solution. Do not advertise a small window at all. Keep advertising zero until the available space is at least one full MSS or half the buffer, whichever is smaller. The sender stays blocked briefly, then receives a window worth using. A short delay is far better than a stream of 1-byte segments.
Sender side — Nagle's algorithm (RFC 896). Do not send a small segment while an earlier small segment remains unacknowledged. Buffer the data instead; when the ACK arrives, send everything accumulated as one larger segment. The rule is:
if data >= MSS and window allows: send immediately
elif nothing unacknowledged: send immediately
else: buffer until the ACK arrives
The elegance is that Nagle is self-clocking: on a fast path ACKs return quickly, so little buffering occurs; on a slow path more data accumulates per segment, which is exactly when coalescing helps most. It adapts without configuration.
The efficiency numbers make the motivation concrete:
| Payload | Total bytes | Efficiency |
|---|---|---|
| 1 byte | 41 | 2.4% |
| 10 bytes | 50 | 20% |
| 100 bytes | 140 | 71% |
| 1,460 bytes | 1,500 | 97.3% |
A 1-byte segment spends 40 bytes of header to move 1 byte of data — and each one also consumes a packet slot on every link and a forwarding decision at every router.
Nagle's interaction with delayed ACKs causes a well-known pathology, covered in the next topic. For interactive and latency-sensitive applications, TCP_NODELAY disables Nagle.
Examples
Receiver-caused SWS: the 1-byte death spiral
buffer full -> advertise window 0
app reads 1 byte -> 1 byte free
advertise window = 1
sender sends [40-byte header][1 byte] <- 2.4% efficiency
buffer full again -> advertise window 0
app reads 1 byte...
... repeat forever
CLARK'S FIX:
app reads 1 byte -> 1 byte free
free space (1) < min(MSS, buffer/2)? YES
-> KEEP ADVERTISING ZERO
... app reads more ...
free space reaches 1460 -> advertise 1460
sender sends one FULL segment <- 97.3% efficiency
Sender-caused SWS: the telnet problem
user types "hello", one character at a time.
WITHOUT NAGLE: 5 segments, 41 bytes each = 205 bytes for 5 bytes of data
[h][e][l][l][o] -> 5 packets on every link, 5 lookups per router
WITH NAGLE:
't' sent immediately (nothing outstanding)
'e','l','l','o' BUFFERED while 'h' is unacknowledged
ACK for 'h' arrives -> send "ello" as ONE segment
= 2 segments, 86 bytes total
Nagle is self-clocking
FAST path (1 ms RTT): ACKs return almost instantly
-> almost no buffering, almost no added delay
SLOW path (200 ms RTT): ACKs are slow
-> more data accumulates per segment
-> coalescing helps exactly where it matters
No tuning required. The network's own timing drives it.
HEADER = 40 # 20 IP + 20 TCP
def efficiency(payload):
return payload / (payload + HEADER)
print(f"{'payload':>9}{'on the wire':>14}{'efficiency':>13}")
for p in (1, 10, 100, 536, 1460):
total = p + HEADER
print(f"{p:>9}{total:>14}{efficiency(p):>12.1%}")
# payload on the wire efficiency
# 1 41 2.4%
# 10 50 20.0%
# 100 140 71.4%
# 536 576 93.1%
# 1460 1500 97.3%
class Receiver:
"""Clark's solution: never advertise a window too small to be useful."""
def __init__(self, capacity=65535, mss=1460):
self.capacity, self.mss = capacity, mss
self.used = capacity # start full
def app_reads(self, nbytes):
self.used -= min(nbytes, self.used)
def advertise(self, use_clark=True):
free = self.capacity - self.used
if not use_clark:
return free, "naive: advertise whatever is free"
threshold = min(self.mss, self.capacity // 2)
if free < threshold:
return 0, f"CLARK: {free} < threshold {threshold} -> keep advertising 0"
return free, f"CLARK: {free} >= threshold {threshold} -> advertise it"
print()
for clark in (False, True):
r = Receiver()
label = "WITH Clark's fix" if clark else "NAIVE (silly window)"
print(f"\n{label}:")
for read in (1, 1, 1, 1500):
r.app_reads(read)
win, note = r.advertise(use_clark=clark)
eff = f"{efficiency(win):.1%}" if win else "sender blocked"
print(f" app reads {read:>5} -> advertise {win:>5} ({note}) segment efficiency {eff}")
#
# NAIVE (silly window):
# app reads 1 -> advertise 1 (naive: advertise whatever is free) segment efficiency 2.4%
# app reads 1 -> advertise 2 (naive: advertise whatever is free) segment efficiency 4.8%
# app reads 1 -> advertise 3 (naive: advertise whatever is free) segment efficiency 7.0%
# app reads 1500 -> advertise 1503 (naive: advertise whatever is free) segment efficiency 97.4%
#
# WITH Clark's fix:
# app reads 1 -> advertise 0 (CLARK: 1 < threshold 1460 -> keep advertising 0) segment efficiency sender blocked
# app reads 1 -> advertise 0 (CLARK: 2 < threshold 1460 -> keep advertising 0) segment efficiency sender blocked
# app reads 1 -> advertise 0 (CLARK: 3 < threshold 1460 -> keep advertising 0) segment efficiency sender blocked
# app reads 1500 -> advertise 1503 (CLARK: 1503 >= threshold 1460 -> advertise it) segment efficiency 97.4%
class NagleSender:
"""RFC 896: no small segment while a small segment is unacknowledged."""
def __init__(self, mss=1460, nagle=True):
self.mss, self.nagle = mss, nagle
self.buffer = b""
self.unacked_small = False
self.sent = []
def write(self, data):
self.buffer += data
if len(self.buffer) >= self.mss:
return self._flush("full MSS available")
if not self.nagle:
return self._flush("TCP_NODELAY: send immediately")
if not self.unacked_small:
return self._flush("nothing outstanding: send immediately")
return None, f"BUFFERED ({len(self.buffer)} bytes waiting for an ACK)"
def _flush(self, why):
if not self.buffer:
return None, "nothing to send"
seg, self.buffer = self.buffer, b""
self.unacked_small = len(seg) < self.mss
self.sent.append(seg)
return seg, f"SENT {len(seg)} bytes ({why})"
def ack(self):
self.unacked_small = False
if self.buffer:
return self._flush("ACK arrived: flush the buffer")
return None, "ACK arrived, nothing buffered"
print()
for nagle in (False, True):
s = NagleSender(nagle=nagle)
label = "WITH Nagle" if nagle else "WITHOUT Nagle (TCP_NODELAY)"
print(f"\n{label} — typing 'hello' one character at a time:")
for ch in b"hello":
seg, note = s.write(bytes([ch]))
print(f" write {chr(ch)!r}: {note}")
if seg and nagle:
pass
if nagle:
seg, note = s.ack()
print(f" {note}")
total = sum(len(x) + HEADER for x in s.sent)
print(f" -> {len(s.sent)} segment(s), {total} bytes on the wire for 5 bytes of data")
#
# WITHOUT Nagle (TCP_NODELAY) — typing 'hello' one character at a time:
# write 'h': SENT 1 bytes (TCP_NODELAY: send immediately)
# write 'e': SENT 1 bytes (TCP_NODELAY: send immediately)
# write 'l': SENT 1 bytes (TCP_NODELAY: send immediately)
# write 'l': SENT 1 bytes (TCP_NODELAY: send immediately)
# write 'o': SENT 1 bytes (TCP_NODELAY: send immediately)
# -> 5 segment(s), 205 bytes on the wire for 5 bytes of data
#
# WITH Nagle — typing 'hello' one character at a time:
# write 'h': SENT 1 bytes (nothing outstanding: send immediately)
# write 'e': BUFFERED (1 bytes waiting for an ACK)
# write 'l': BUFFERED (2 bytes waiting for an ACK)
# write 'l': BUFFERED (3 bytes waiting for an ACK)
# write 'o': BUFFERED (4 bytes waiting for an ACK)
# SENT 4 bytes (ACK arrived: flush the buffer)
# -> 2 segment(s), 85 bytes on the wire for 5 bytes of data
# 205 bytes becomes 85: same data, 2.4x less wire traffic.
# Nagle's self-clocking behaviour.
def nagle_batching(rtt_ms, keystrokes_per_sec=5):
per_rtt = keystrokes_per_sec * rtt_ms / 1000
return max(1, round(per_rtt))
print()
for rtt in (1, 20, 200):
n = nagle_batching(rtt)
print(f"RTT {rtt:>4} ms -> ~{n} keystroke(s) coalesced per segment "
f"({'negligible delay' if rtt < 50 else 'meaningful coalescing'})")
# RTT 1 ms -> ~1 keystroke(s) coalesced per segment (negligible delay)
# RTT 20 ms -> ~1 keystroke(s) coalesced per segment (negligible delay)
# RTT 200 ms -> ~1 keystroke(s) coalesced per segment (meaningful coalescing)
# Spot silly-window behaviour: many tiny segments:
$ sudo tcpdump -i eth0 -n 'tcp and less 100' -c 10
$ tshark -r capture.pcap -Y 'tcp.len > 0 && tcp.len < 100' -T fields -e tcp.len \
> | sort -n | uniq -c | head -5
# 4821 1
# 412 2
# 94 5
# ^^^^ thousands of 1-byte segments: 2.4% efficiency
# Counters for window-probe and small-window behaviour:
$ nstat -az | grep -iE "zerowindow|wantzero"
# TcpExtTCPWantZeroWindowAdv 1204 <- receiver wanted to advertise small
# TcpExtTCPToZeroWindowAdv 412
# TcpExtTCPFromZeroWindowAdv 408
# Nagle is ON by default; disable it per socket for interactive traffic:
$ python3 -c "
import socket
s = socket.socket()
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
print('Nagle disabled: every write() goes out immediately')"
# Which sockets have it disabled:
$ ss -tin state established | grep -c nodelay
# See the difference in segment sizes:
$ tshark -r with_nagle.pcap -T fields -e tcp.len | awk '$1>0{s+=$1;n++} END{print "avg segment:", s/n}'
# avg segment: 1421.4
$ tshark -r nodelay.pcap -T fields -e tcp.len | awk '$1>0{s+=$1;n++} END{print "avg segment:", s/n}'
# avg segment: 38.2
# Same data volume, 37x more packets.
# Nginx and similar servers disable Nagle for response headers, then
# re-enable coalescing for the body:
$ grep -E "tcp_nodelay|tcp_nopush" /etc/nginx/nginx.conf
# tcp_nodelay on; # disable Nagle on keep-alive connections
# tcp_nopush on; # TCP_CORK: fill full segments for sendfile() bodies
# TCP_CORK: the opposite of NODELAY — hold data until a full segment exists:
$ python3 -c "
import socket
s = socket.socket()
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_CORK, 1) # coalesce aggressively
print('TCP_CORK: send only full segments (or on a 200 ms timer)')"
# Measure header overhead on a real capture:
$ tshark -r capture.pcap -T fields -e tcp.len | awk '
> $1>0 {payload+=$1; wire+=$1+40; n++}
> END {printf "segments: %d payload: %d wire: %d efficiency: %.1f%%\n",
> n, payload, wire, 100*payload/wire}'
# segments: 18422 payload: 26183424 wire: 26920304 efficiency: 97.3%
Key takeaways
- Silly Window Syndrome is the exchange of tiny segments where header overhead dominates — a 1-byte payload is 2.4% efficient.
- Receiver-caused SWS: advertising a tiny window after the application reads a small amount, producing a 1-byte death spiral.
- Clark's solution: keep advertising zero until free space reaches min(MSS, half the buffer).
- Sender-caused SWS: an application writing tiny amounts, each sent as its own segment.
- Nagle's algorithm (RFC 896): do not send a small segment while an earlier small segment is unacknowledged — buffer and coalesce instead.
- Nagle is self-clocking: fast paths coalesce little, slow paths coalesce more, with no tuning.
- Full segments reach 97.3% efficiency (1,460 payload in 1,500 bytes on the wire).
TCP_NODELAYdisables Nagle for latency-sensitive applications;TCP_CORKdoes the opposite, holding data until a full segment exists.
🧪 Practice
- Compute the wire efficiency of 1-byte, 20-byte, and 1,460-byte payloads with 40 bytes of header.
- Describe the receiver-side silly window cycle and state Clark's threshold rule.
- Trace Nagle's behaviour for an application writing 5 single bytes with a 100 ms RTT, giving the number of segments sent.
- Interview-style: Nagle needs no configuration to behave differently on fast and slow paths. Why? (Hint: ask what triggers the flush of buffered data, and how that trigger's timing varies with the path.)
Nagle's Algorithm and Delayed ACKs
Theory
Nagle's algorithm (previous topic) reduces small-segment overhead on the sender. Delayed ACKs (RFC 1122) reduce acknowledgement overhead on the receiver. Each is sensible alone. Together they interact badly, producing one of TCP's most notorious latency pathologies.
Delayed ACKs work on the observation that a pure ACK carries no data but costs a full 40-byte packet. So a receiver waits — up to 200 ms on Linux, up to 500 ms permitted by the RFC — before acknowledging, hoping either that more data arrives (so one ACK can cover several segments) or that the application produces a response the ACK can piggyback on.
The rules require an ACK to be sent:
- Immediately upon receiving two full-sized segments (the "ACK every other segment" rule).
- When the delay timer expires (40 ms typical on Linux, 200 ms maximum).
- Immediately when out-of-order data arrives (so fast retransmit can trigger).
- Whenever the receiver has data to send, piggybacked.
The typical saving is around 50% of ACK packets, which is worthwhile on a busy server.
The interaction. Consider a request-response application where the request is slightly larger than one MSS:
- The sender transmits a full segment, then has a small remainder.
- Nagle says: a small segment with data unacknowledged → buffer it.
- The receiver has one segment — not two — so delayed ACK says: wait.
- The receiver's application cannot respond because the request is incomplete, so there is nothing to piggyback.
- Both sides wait. The deadlock breaks only when the delayed ACK timer expires, typically 40 ms, sometimes 200 ms.
This produces the classic signature: an application that is fast on localhost but shows a consistent ~40 ms floor over the network, with the delay appearing only on certain message sizes. Because the delay is a timer rather than a queue, it is remarkably constant — which is itself the diagnostic clue.
The fixes, in order of preference:
| Fix | Effect | When |
|---|---|---|
| Write the whole message in one call | No small remainder exists | Best — fixes the cause |
TCP_NODELAY | Disables Nagle | Standard for RPC, interactive, request-response |
TCP_QUICKACK | Disables delayed ACK (temporarily) | Rarely; it resets itself |
TCP_CORK then uncork | Explicit control of when to flush | Batch senders |
The first is genuinely the right fix and is often overlooked: the interaction requires a small trailing segment, which exists only because the application made multiple writes or produced an awkward size. Using a single write(), or writev(), or buffering in the application, removes the condition entirely.
In practice most RPC frameworks, databases, and interactive servers set TCP_NODELAY unconditionally, because their traffic is request-response and latency matters more than the small overhead saving Nagle provides.
Examples
The interaction, step by step
request = 1500 bytes, MSS = 1460
t=0 sender: write(1500)
-> segment 1: 1460 bytes, sent immediately (full MSS)
-> remaining 40 bytes: NAGLE says wait (segment 1 unacknowledged)
t=0.1 receiver: has 1460 bytes
-> only ONE segment received, not two -> DELAYED ACK says wait
-> application cannot respond: the request is INCOMPLETE
-> nothing to piggyback on
t=0.1 .. 40 BOTH SIDES WAITING. Nothing is in flight. The link is idle.
t=40 receiver's delayed-ACK timer EXPIRES -> sends the ACK
t=40.1 sender receives it -> Nagle releases the 40-byte remainder
t=40.2 receiver has the full request -> application responds
Total: ~40 ms for a request that needed ~0.2 ms of network time.
The signature in measurements
localhost: 0.2 ms <- no delay: loopback ACKs immediately
over network: 40.2 ms <- the delayed ACK timer, almost exactly
over network: 40.1 ms
over network: 40.2 ms
^^^^ SUSPICIOUSLY CONSTANT. Queueing delay varies;
a TIMER does not. That constancy is the clue.
Why one write() fixes it
write(1500) -> 1460 + 40 remainder -> Nagle holds the 40 -> 40 ms stall
write(1460) -> exactly one full segment, no remainder -> no stall
two writes of 750 -> first sent immediately, second buffered -> 40 ms stall
^^^^ the number of write() calls matters
class NagleDelayedAck:
"""Simulate the interaction that produces the 40 ms stall."""
DELAYED_ACK_MS = 40
def __init__(self, mss=1460, nagle=True, delayed_ack=True):
self.mss, self.nagle, self.delayed_ack = mss, nagle, delayed_ack
def request_response(self, request_bytes, rtt_ms=0.2):
t = 0.0
log = []
full, remainder = divmod(request_bytes, self.mss)
for i in range(full):
log.append((t, f"send full segment {i+1} ({self.mss} B)"))
t += rtt_ms
if remainder == 0:
log.append((t, "request complete -> application responds"))
return t, log
# A small remainder exists. Does Nagle hold it?
if self.nagle and full > 0:
log.append((t, f"NAGLE holds the {remainder} B remainder "
f"(segment 1 unacknowledged)"))
if self.delayed_ack:
t += self.DELAYED_ACK_MS
log.append((t, f"DELAYED ACK timer expires after "
f"{self.DELAYED_ACK_MS} ms -> ACK sent"))
else:
log.append((t, "QUICKACK: receiver ACKs immediately"))
t += rtt_ms
log.append((t, f"Nagle releases the {remainder} B remainder"))
else:
log.append((t, f"NODELAY: {remainder} B remainder sent immediately"))
t += rtt_ms
log.append((t, "request complete -> application responds"))
return t, log
for nagle, dack, label in [(True, True, "Nagle ON + delayed ACK ON (the default)"),
(False, True, "TCP_NODELAY (Nagle off)"),
(True, False, "TCP_QUICKACK (delayed ACK off)")]:
sim = NagleDelayedAck(nagle=nagle, delayed_ack=dack)
total, log = sim.request_response(1500)
print(f"\n{label}: total {total:.1f} ms")
for t, event in log:
print(f" t={t:>6.1f} ms {event}")
#
# Nagle ON + delayed ACK ON (the default): total 40.6 ms
# t= 0.0 ms send full segment 1 (1460 B)
# t= 0.2 ms NAGLE holds the 40 B remainder (segment 1 unacknowledged)
# t= 40.2 ms DELAYED ACK timer expires after 40 ms -> ACK sent
# t= 40.4 ms Nagle releases the 40 B remainder
# t= 40.6 ms request complete -> application responds
#
# TCP_NODELAY (Nagle off): total 0.4 ms
# t= 0.0 ms send full segment 1 (1460 B)
# t= 0.2 ms NODELAY: 40 B remainder sent immediately
# t= 0.4 ms request complete -> application responds
#
# TCP_QUICKACK (delayed ACK off): total 0.6 ms
# t= 0.0 ms send full segment 1 (1460 B)
# t= 0.2 ms NAGLE holds the 40 B remainder (segment 1 unacknowledged)
# t= 0.2 ms QUICKACK: receiver ACKs immediately
# t= 0.4 ms Nagle releases the 40 B remainder
# t= 0.6 ms request complete -> application responds
#
# 0.4 ms becomes 40.6 ms — a 100x latency penalty from two individually
# sensible optimisations interacting.
# Message size determines whether the pathology triggers at all.
def stalls(request_bytes, mss=1460):
full, remainder = divmod(request_bytes, mss)
if full > 0 and remainder > 0:
return True, f"{full} full segment(s) + {remainder} B remainder -> STALLS"
if remainder > 0:
return False, f"only {remainder} B, nothing outstanding -> sent immediately"
return False, f"exactly {full} full segment(s), no remainder -> no stall"
print()
for size in (500, 1460, 1500, 2920, 3000):
bad, why = stalls(size)
print(f"{size:>6} byte request: {'STALL' if bad else 'ok '} {why}")
# 500 byte request: ok only 500 B, nothing outstanding -> sent immediately
# 1460 byte request: ok exactly 1 full segment(s), no remainder -> no stall
# 1500 byte request: STALL 1 full segment(s) + 40 B remainder -> STALLS
# 2920 byte request: ok exactly 2 full segment(s), no remainder -> no stall
# 3000 byte request: STALL 2 full segment(s) + 80 B remainder -> STALLS
# Only SOME sizes trigger it — which is why the bug looks intermittent.
# The ACK saving delayed ACKs actually provide.
def ack_overhead(segments, delayed):
acks = (segments + 1) // 2 if delayed else segments
return acks, acks * 40
print()
for segs in (100, 10_000):
for d in (False, True):
acks, bytes_ = ack_overhead(segs, d)
label = "delayed" if d else "immediate"
print(f"{segs:>7,} segments, {label:<10} ACKs -> {acks:>6,} ACKs, "
f"{bytes_:>9,} bytes of pure overhead")
# 100 segments, immediate ACKs -> 100 ACKs, 4,000 bytes of pure overhead
# 100 segments, delayed ACKs -> 50 ACKs, 2,000 bytes of pure overhead
# 10,000 segments, immediate ACKs -> 10,000 ACKs, 400,000 bytes of pure overhead
# 10,000 segments, delayed ACKs -> 5,000 ACKs, 200,000 bytes of pure overhead
# A real 50% saving — which is why delayed ACKs are not simply removed.
# The fix hierarchy.
FIXES = {
"single write() / writev()": (1, "removes the small remainder — fixes the CAUSE"),
"TCP_NODELAY": (2, "Nagle off; standard for RPC and interactive"),
"TCP_CORK then uncork": (3, "explicit control of when to flush"),
"TCP_QUICKACK": (4, "delayed ACK off, but it RESETS itself"),
}
print()
for fix, (rank, note) in sorted(FIXES.items(), key=lambda kv: kv[1][0]):
print(f" {rank}. {fix:<28}{note}")
# The signature: suspiciously constant ~40 ms latency:
$ for i in $(seq 1 5); do
> curl -s -o /dev/null -w '%{time_total}\n' http://server/api/request
> done
# 0.040821
# 0.040914
# 0.040772
# 0.040883
# 0.040801
# Queueing delay VARIES. A timer does not. This constancy is the clue.
# Confirm in a capture — look for the gap before the small segment:
$ tshark -r capture.pcap -T fields -e frame.time_delta -e tcp.len -e tcp.flags \
> | awk '$1 > 0.03 && $1 < 0.25 {print}' | head -3
# 0.040112 40 0x0018
# ^^ a 40-byte segment preceded by a 40 ms gap
# Check whether Nagle is disabled on the socket:
$ ss -tin state established | grep -o nodelay | head -1
# (empty means Nagle is ON)
# Apply TCP_NODELAY:
$ python3 -c "
import socket
s = socket.socket()
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
print('Nagle disabled')"
# Most servers set it for you — verify your configuration:
$ grep -rE "tcp_nodelay|TCP_NODELAY" /etc/nginx/nginx.conf /etc/redis/redis.conf 2>/dev/null
# /etc/nginx/nginx.conf: tcp_nodelay on;
# Redis, PostgreSQL, gRPC, and most RPC frameworks set it unconditionally.
# TCP_QUICKACK is temporary — it resets after one use:
$ python3 -c "
import socket
s = socket.socket()
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_QUICKACK, 1)
print('quickack set — but the kernel may clear it again; NODELAY is more reliable')"
# Delayed-ACK counters:
$ nstat -az | grep -iE "delayedack"
# TcpExtDelayedACKs 184221 <- ACKs that were delayed
# TcpExtDelayedACKLocked 412
# TcpExtDelayedACKLost 94
# Prove the cause by removing the small remainder:
$ python3 - <<'EOF'
import socket, time
s = socket.socket(); s.connect(("server", 8080))
payload = b"x" * 1500
# BAD: two writes -> a small second segment that Nagle holds
t0 = time.time(); s.sendall(payload[:1460]); s.sendall(payload[1460:])
print(f"two writes: {(time.time()-t0)*1000:.1f} ms")
# GOOD: one write -> the kernel segments it optimally
t0 = time.time(); s.sendall(payload)
print(f"one write: {(time.time()-t0)*1000:.1f} ms")
EOF
# Kernel delayed-ACK timing (not directly tunable; 40 ms min, 200 ms max):
$ grep -r "TCP_DELACK_MIN\|TCP_DELACK_MAX" /usr/src/linux/include/net/tcp.h 2>/dev/null
# #define TCP_DELACK_MAX ((unsigned)(HZ/5)) /* 200 ms */
# #define TCP_DELACK_MIN ((unsigned)(HZ/25)) /* 40 ms */
Key takeaways
- Delayed ACKs hold acknowledgements up to 40 ms (Linux) hoping to piggyback or to cover two segments, saving roughly 50% of ACK packets.
- An ACK must be sent immediately on two full segments, on out-of-order data, or when the receiver has data to piggyback on.
- Nagle + delayed ACK deadlock when a sender has a small trailing segment: Nagle waits for an ACK, the receiver waits for a second segment or a response, and nothing moves until the timer fires.
- The signature is a suspiciously constant ~40 ms latency that appears over the network but not on localhost, and only for certain message sizes.
- Constancy is the diagnostic clue — queueing delay varies, a timer does not.
- The best fix is a single
write(), which removes the small remainder that creates the condition. TCP_NODELAYis the standard practical fix and is set unconditionally by most RPC frameworks, databases, and interactive servers.TCP_QUICKACKis unreliable because the kernel resets it;TCP_CORKgives explicit flush control for batch senders.
🧪 Practice
- Explain why an ACK is delayed at all, and list the three conditions that force one immediately.
- Determine which of 1,000, 1,460, 2,100, and 2,920-byte requests trigger the Nagle/delayed-ACK stall, with MSS 1,460.
- Describe the measurement signature of this pathology and explain why the constancy of the delay is diagnostic.
- Interview-style: An RPC service shows 0.3 ms latency on localhost and exactly 40 ms over the network, with CPU and bandwidth both idle. Diagnose and give two fixes. (Hint: 40 ms is not a queue — ask which kernel timer has that value and what must be waiting on it.)
Window Scaling Option
Theory
The TCP Window field is 16 bits, capping the advertised window at 65,535 bytes. In 1981 that was ample. Today it is the single most common cause of disappointing throughput on long, fast paths.
The constraint is arithmetic. Throughput is bounded by window / RTT, so a 64 KB window on a 100 ms path yields about 5.2 Mb/s — on a 1 Gb/s link, that is 0.5% utilisation. The link is not the problem; the field width is.
Window scaling (RFC 7323, originally RFC 1323) fixes this without changing the header. It adds a scale factor negotiated during the handshake: a value S between 0 and 14, meaning the Window field should be left-shifted by S bits — multiplied by 2^S.
effective window = advertised_window x 2^scale
With the maximum scale of 14, the window reaches 65,535 × 16,384 ≈ 1 GB, which is sufficient for any realistic path.
Three properties of the mechanism matter operationally.
It is negotiated only in the SYN. Both sides must include the Window Scale option in their SYN or SYN-ACK. If either omits it, scaling is disabled in both directions for the life of the connection. There is no way to enable it later.
Each direction has its own factor. The client's scale applies to windows the client advertises; the server's to windows the server advertises. They are frequently different — a server sending bulk data may use a large scale while the client uses a small one.
Scaling does not apply to the SYN itself. The window in the SYN and SYN-ACK is unscaled, because the factor is still being negotiated at that moment.
This creates the classic middlebox failure. A firewall or load balancer that strips unknown TCP options removes the Window Scale option from the SYN. Both endpoints then believe scaling was not negotiated — but if the middlebox strips it in only one direction, or if it passes the option through but does not track the factor when rewriting sequence numbers, the two ends disagree about the window size by a factor of 2^S. The symptoms are bizarre: connections that establish fine, transfer a little data, then stall completely. Diagnosing this requires capturing the SYN at both ends and comparing.
Because scaling raises the window ceiling, the practical limit becomes buffer memory, and the kernel's receive buffer autotuning takes over — growing the buffer toward the measured bandwidth-delay product. This is why manually setting SO_RCVBUF is usually counterproductive: it disables autotuning and pins the window to a fixed value that is probably wrong.
Examples
The 16-bit ceiling, and what scaling changes
Window field: 16 bits -> max 65,535 bytes
throughput = window / RTT:
RTT 1 ms -> 524 Mb/s (acceptable on a LAN)
RTT 10 ms -> 52 Mb/s
RTT 100 ms -> 5.2 Mb/s <- on a 1 Gb/s link: 0.5% utilisation
RTT 300 ms -> 1.7 Mb/s <- satellite: essentially unusable
With scale 7 (x128):
RTT 100 ms -> 65535 x 128 = 8.4 MB window -> 671 Mb/s
Negotiation happens ONLY in the SYN
CLIENT SERVER
|--- SYN win=64240, wscale=7 -------->|
|<-- SYN+ACK win=65535, wscale=9 -----|
|--- ACK ----------------------------->|
client scale = 7 -> client's advertised windows are multiplied by 128
server scale = 9 -> server's advertised windows are multiplied by 512
DIFFERENT factors per direction. Both are fixed for the connection's life.
Later segment: "win 501" from the server
real window = 501 x 512 = 256,512 bytes
^^^^^^^^^ you MUST know the scale to read a capture
The middlebox failure
client [firewall strips server
unknown options]
|--- SYN wscale=7 ----->|--- SYN (no wscale) --->|
| |
server: "no scaling offered" -> scaling DISABLED
client: still believes scaling is active
client advertises "win 501" meaning 501 x 128 = 64,128
server reads it as 501 bytes
-> server sends 501 bytes and waits forever
-> connection establishes, transfers a little, then STALLS
def scaled_window(advertised, scale):
return advertised * (2 ** scale)
def max_window(scale):
return 65535 * (2 ** scale)
print(f"{'scale':>6}{'multiplier':>12}{'max window':>16}")
for s in (0, 1, 7, 9, 14):
print(f"{s:>6}{2**s:>12,}{max_window(s):>15,} B")
# scale multiplier max window
# 0 1 65,535 B
# 1 2 131,070 B
# 7 128 8,388,480 B
# 9 512 33,553,920 B
# 14 16,384 1,073,725,440 B
# Throughput with and without scaling.
def throughput(window_bytes, rtt_ms):
return window_bytes * 8 / (rtt_ms / 1000) / 1e6
print()
print(f"{'RTT':>7}{'no scaling':>15}{'scale 7':>15}{'scale 9':>15}")
for rtt in (1, 10, 100, 300):
print(f"{rtt:>5}ms{throughput(65535, rtt):>12,.1f} Mb/s"
f"{throughput(max_window(7), rtt):>12,.1f} Mb/s"
f"{throughput(max_window(9), rtt):>12,.1f} Mb/s")
# RTT no scaling scale 7 scale 9
# 1ms 524.3 Mb/s 67,107.8 Mb/s 268,431.4 Mb/s
# 10ms 52.4 Mb/s 6,710.8 Mb/s 26,843.1 Mb/s
# 100ms 5.2 Mb/s 671.1 Mb/s 2,684.3 Mb/s
# 300ms 1.7 Mb/s 223.7 Mb/s 894.8 Mb/s
# Choosing a scale factor for a given path.
import math
def required_scale(bandwidth_mbps, rtt_ms):
bdp = bandwidth_mbps * 1e6 / 8 * (rtt_ms / 1000)
if bdp <= 65535:
return 0, bdp
return math.ceil(math.log2(bdp / 65535)), bdp
print()
for bw, rtt, label in [(100, 10, "100 Mb/s LAN"),
(1000, 100, "1 Gb/s transatlantic"),
(10000, 1, "10 Gb/s data centre"),
(100, 600, "100 Mb/s satellite")]:
scale, bdp = required_scale(bw, rtt)
print(f"{label:<24}BDP {bdp/1024:>9,.0f} KB -> needs scale {scale} "
f"(window up to {max_window(scale)/1024/1024:>7,.1f} MB)")
# 100 Mb/s LAN BDP 122 KB -> needs scale 1 (window up to 0.1 MB)
# 1 Gb/s transatlantic BDP 12,207 KB -> needs scale 8 (window up to 16.0 MB)
# 10 Gb/s data centre BDP 1,221 KB -> needs scale 5 (window up to 2.0 MB)
# 100 Mb/s satellite BDP 7,324 KB -> needs scale 7 (window up to 8.0 MB)
# Reading a capture requires knowing the scale.
def interpret(advertised, scale):
return {"raw_field": advertised, "scale": scale,
"real_window": scaled_window(advertised, scale)}
print()
for adv, sc in [(501, 0), (501, 7), (501, 9)]:
i = interpret(adv, sc)
print(f" 'win {adv}' with wscale {sc} -> {i['real_window']:>9,} bytes")
# 'win 501' with wscale 0 -> 501 bytes
# 'win 501' with wscale 7 -> 64,128 bytes
# 'win 501' with wscale 9 -> 256,512 bytes
# The same capture line means three very different things.
# The middlebox mismatch.
def mismatch(client_thinks, server_thinks, advertised):
c = scaled_window(advertised, client_thinks)
s = scaled_window(advertised, server_thinks)
if c == s:
return "agreement: the connection works"
return (f"MISMATCH: client means {c:,} bytes, server reads {s:,} bytes "
f"({c // s if s else 0}x apart) -> establishes, then STALLS")
print()
for cs, ss in [(7, 7), (7, 0)]:
print(f" client scale {cs}, server scale {ss}: {mismatch(cs, ss, 501)}")
# client scale 7, server scale 7: agreement: the connection works
# client scale 7, server scale 0: MISMATCH: client means 64,128 bytes, server reads 501 bytes (128x apart) -> establishes, then STALLS
# Why manual SO_RCVBUF is usually a mistake.
def buffer_strategy(manual_size, measured_bdp):
if manual_size is None:
return "autotuning: the kernel grows the buffer toward the measured BDP"
if manual_size < measured_bdp:
return (f"PINNED at {manual_size/1024:.0f} KB < BDP "
f"{measured_bdp/1024:.0f} KB -> throughput capped, autotuning OFF")
return f"pinned at {manual_size/1024:.0f} KB -> works, but wastes memory and cannot adapt"
print()
bdp = 12_500_000
for manual in (None, 262144, 33554432):
print(f" SO_RCVBUF={str(manual):<10} {buffer_strategy(manual, bdp)}")
# SO_RCVBUF=None autotuning: the kernel grows the buffer toward the measured BDP
# SO_RCVBUF=262144 PINNED at 256 KB < BDP 12207 KB -> throughput capped, autotuning OFF
# SO_RCVBUF=33554432 pinned at 32768 KB -> works, but wastes memory and cannot adapt
# Is scaling negotiated? Check the SYN — it is the ONLY place it appears:
$ sudo tcpdump -i eth0 -n 'tcp[tcpflags] & tcp-syn != 0' -v -c 2 | grep -o "wscale [0-9]*"
# wscale 7
# wscale 9
# Two different factors, one per direction. Both must be present.
# The negotiated factors on a live connection:
$ ss -tin state established | head -3
# ESTAB 0 0 10.1.1.5:51250 93.184.216.34:443
# cubic wscale:7,9 rto:204 rtt:3.8/1.2 mss:1448 cwnd:120
# ^^^^^^^^^^ send scale 7, receive scale 9
# Reading a scaled window in a capture:
$ sudo tcpdump -i eth0 -n 'tcp port 443' -c 1
# IP 93.184.216.34.443 > 10.1.1.5.51250: Flags [.], ack 2897, win 501
# ^^^^^^^
$ python3 -c "print(f'real window: {501 * 2**9:,} bytes')"
# real window: 256,512 bytes
# tcpdump does NOT scale for you unless it saw the SYN in the same capture.
# Verify scaling is enabled system-wide (it is, by default):
$ sysctl net.ipv4.tcp_window_scaling
# net.ipv4.tcp_window_scaling = 1
# Size buffers so autotuning can reach the BDP:
$ sysctl net.ipv4.tcp_rmem net.core.rmem_max net.ipv4.tcp_moderate_rcvbuf
# net.ipv4.tcp_rmem = 4096 131072 6291456
# net.core.rmem_max = 212992
# net.ipv4.tcp_moderate_rcvbuf = 1 <- autotuning ON; keep it that way
$ sudo sysctl -w net.core.rmem_max=33554432
$ sudo sysctl -w net.ipv4.tcp_rmem="4096 131072 33554432"
# Diagnose the middlebox stripping failure — capture at BOTH ends:
# on the client:
$ sudo tcpdump -i eth0 -n 'tcp[tcpflags] & tcp-syn != 0' -v -c 1 -w client.pcap
# on the server:
$ sudo tcpdump -i eth0 -n 'tcp[tcpflags] & tcp-syn != 0' -v -c 1 -w server.pcap
$ tshark -r client.pcap -T fields -e tcp.options.wscale.shift
# 7
$ tshark -r server.pcap -T fields -e tcp.options.wscale.shift
# (empty) <- the option was STRIPPED in transit. Found it.
# The throughput difference, measured:
$ iperf3 -c server -w 64K -t 10 | tail -2
# [ 5] 0.00-10.00 sec 6.25 MBytes 5.24 Mbits/sec
$ iperf3 -c server -t 10 | tail -2 # let autotuning do its job
# [ 5] 0.00-10.00 sec 1.09 GBytes 938 Mbits/sec
# Counters for buffer-limited connections:
$ nstat -az | grep -iE "rcvpruned|sndbuflimited|rcvcollapsed"
# TcpExtRcvPruned 0
# TcpExtTCPSndBufLimited 412 <- sender buffer was the bottleneck
# TcpExtRcvCollapsed 1204
Key takeaways
- The Window field is 16 bits, capping the raw window at 65,535 bytes — about 5.2 Mb/s on a 100 ms path.
- Window scaling (RFC 7323) negotiates a shift factor S (0–14); the effective window is
advertised × 2^S, up to about 1 GB. - The option appears only in the SYN/SYN-ACK; if either side omits it, scaling is disabled in both directions permanently.
- Each direction has its own scale factor, and they are frequently different.
- The window in the SYN itself is unscaled, since the factor is still being negotiated.
- A middlebox that strips the option causes the two ends to disagree by 2^S — connections establish, transfer briefly, then stall; diagnosis requires capturing the SYN at both ends.
- Reading a capture requires knowing the scale:
win 501can mean 501, 64,128, or 256,512 bytes. - With scaling enabled, buffer memory becomes the limit, and autotuning grows it toward the measured BDP — setting
SO_RCVBUFmanually disables autotuning and usually hurts.
🧪 Practice
- Compute the effective window for an advertised value of 1,024 with scale factors 0, 7, and 12.
- Determine the scale factor needed for a 2 Gb/s link with 80 ms RTT, showing the BDP calculation.
- Explain why window scaling cannot be enabled after the handshake, and what happens if a middlebox strips the option one-way.
- Interview-style: A connection establishes normally, transfers about 64 KB, then stalls permanently. Captures at the client show
wscale 7in the SYN. What would you check next? (Hint: the client and server may not agree — ask what the server actually received, and what 64 KB corresponds to.)
<a id="75-congestion-control"></a>
7.5 Congestion Control
This subchapter covers how senders avoid collectively destroying the network they share — the algorithms that decide how fast to send when nobody tells you the answer.
Causes and Costs of Congestion
Theory
Flow control (7.4) stops a sender overwhelming a receiver. Congestion control stops senders collectively overwhelming the network — and it is a harder problem, because no single sender can see the network's state and there is no explicit signal telling anyone to slow down.
Congestion occurs when the aggregate arrival rate at a link exceeds its capacity. Packets queue, the queue grows, delay rises, and when the buffer fills, packets are dropped. The link itself is working perfectly; the problem is that more traffic wants to cross it than it can carry.
The costs escalate in three stages, and distinguishing them matters:
Stage 1 — queueing delay. Throughput is still fine, but latency rises as queues build. This is where bufferbloat lives: the network appears to be working, and interactive applications feel terrible.
Stage 2 — loss and retransmission. Buffers overflow. Senders retransmit, which puts more traffic onto an already-saturated link. Useful throughput (goodput) begins to fall below the raw throughput, because some of what is being carried is duplicate data.
Stage 3 — congestion collapse. The pathological state. Retransmissions dominate, the link is fully utilised, and almost nothing useful gets through. Crucially, this state is stable: it does not resolve itself, because every sender's response to poor performance is to retransmit more.
This actually happened. In October 1986 the link between Lawrence Berkeley Lab and UC Berkeley — 400 yards apart — collapsed from 32 kb/s to 40 bits per second, a factor of 1,000. Van Jacobson's investigation produced slow start, congestion avoidance, and the RTT estimator covered in 7.3, which were deployed in 4.3BSD and remain the foundation of TCP today.
The reason congestion collapse is possible at all is a tragedy of the commons. Each sender individually benefits from sending faster. The cost — queueing and loss — is shared by everyone. If every sender optimises selfishly, the shared resource is destroyed and everyone does worse than if all had restrained themselves. TCP's congestion control is, in effect, a voluntary restraint protocol: it works because the overwhelming majority of traffic implements it.
This is why UDP applications that ignore congestion control are a genuine problem (7.2), and why RFC 8085 requires them to implement it.
The other key insight is that loss is the signal. IP provides no explicit congestion notification by default, so TCP must infer congestion from its effects. A dropped packet is interpreted as "the network is full". This inference is usually right on wired networks, where loss is overwhelmingly caused by buffer overflow — but it is wrong on wireless, where loss is often caused by radio interference. A sender that halves its rate because of a corrupted frame is responding to a problem that does not exist, which is why wireless links do link-layer retransmission (Chapter 4) to hide such losses from TCP.
Examples
The three stages
throughput
^
| _______________
| / \___
| / \____________ <- STAGE 3: collapse
| / (retransmissions dominate)
| / STAGE 1 STAGE 2
| / queueing loss +
| / delay only retransmission
| /
+------------------------------------> offered load
| | |
capacity buffers collapse
reached overflow
delay
^ ______/
| _____/ <- unbounded
| ____/
| _______/
|_________/ <- STAGE 1 starts here
+------------------------------------> offered load
Why collapse is STABLE
link saturated -> packets dropped
-> senders time out
-> senders RETRANSMIT
-> MORE traffic on an already-full link
-> more drops
-> more retransmissions
... the feedback loop REINFORCES itself.
Nothing in the network breaks this. Only the SENDERS backing off can.
That is the entire reason congestion control exists.
The 1986 collapse, measured
LBL <-> UC Berkeley, 400 yards apart
before: 32,000 bit/s
during: 40 bit/s
ratio: 1/1000
The link hardware was fine throughout. The protocol was the problem.
def network_stage(offered_mbps, capacity_mbps, buffer_pkts=100):
"""Model the three stages as offered load rises past capacity."""
if offered_mbps <= capacity_mbps * 0.8:
return {"stage": 0, "name": "uncongested",
"goodput": offered_mbps, "queue_delay_ms": 0.1, "loss": 0.0}
if offered_mbps <= capacity_mbps:
util = offered_mbps / capacity_mbps
# M/M/1-style delay growth as utilisation approaches 1.
delay = 0.1 / max(0.01, 1 - util)
return {"stage": 1, "name": "queueing delay",
"goodput": offered_mbps, "queue_delay_ms": delay, "loss": 0.0}
overload = offered_mbps / capacity_mbps
loss = min(0.9, (overload - 1) * 0.5)
# Retransmissions consume capacity that carries no new data.
goodput = capacity_mbps * (1 - loss) ** 2
stage = 3 if loss > 0.4 else 2
return {"stage": stage, "name": "collapse" if stage == 3 else "loss + retransmission",
"goodput": goodput, "queue_delay_ms": 500.0, "loss": loss}
print(f"{'offered':>9}{'stage':>7}{'name':>24}{'goodput':>11}{'delay':>10}{'loss':>8}")
for offered in (50, 90, 99, 120, 200, 400):
r = network_stage(offered, 100)
print(f"{offered:>7} Mb{r['stage']:>7}{r['name']:>24}{r['goodput']:>8.1f} Mb"
f"{r['queue_delay_ms']:>8.1f}ms{r['loss']:>7.0%}")
# offered stage name goodput delay loss
# 50 Mb 0 uncongested 50.0 Mb 0.1ms 0%
# 90 Mb 1 queueing delay 90.0 Mb 1.0ms 0%
# 99 Mb 1 queueing delay 99.0 Mb 10.0ms 0%
# 120 Mb 2 loss + retransmission 81.0 Mb 500.0ms 10%
# 200 Mb 3 collapse 25.0 Mb 500.0ms 50%
# 400 Mb 3 collapse 1.0 Mb 500.0ms 90%
# Note the shape: 4x the offered load delivers 1/50th the goodput.
# The 1986 collapse.
def collapse_1986():
return {"before_bps": 32_000, "during_bps": 40,
"degradation": f"{32_000 // 40}x",
"distance": "400 yards",
"fix": "slow start + congestion avoidance (Jacobson, 1988)"}
print()
for k, v in collapse_1986().items():
print(f" {k:<16}{v}")
# before_bps 32000
# during_bps 40
# degradation 800x
# distance 400 yards
# fix slow start + congestion avoidance (Jacobson, 1988)
# The feedback loop that makes collapse self-sustaining.
def collapse_loop(rounds, initial_load=120, capacity=100):
load = initial_load
print(f"\n{'round':>6}{'offered':>10}{'loss':>8}{'retransmitted':>15}{'goodput':>10}")
for r in range(1, rounds + 1):
loss = min(0.9, max(0, (load / capacity - 1) * 0.5))
retrans = load * loss
goodput = capacity * (1 - loss) ** 2
print(f"{r:>6}{load:>8.0f} Mb{loss:>7.0%}{retrans:>12.0f} Mb{goodput:>7.1f} Mb")
load = load + retrans # retransmissions ADD to the load
collapse_loop(5)
# round offered loss retransmitted goodput
# 1 120 Mb 10% 12 Mb 81.0 Mb
# 2 132 Mb 16% 21 Mb 70.6 Mb
# 3 153 Mb 27% 41 Mb 53.9 Mb
# 4 194 Mb 47% 91 Mb 28.2 Mb
# 5 285 Mb 90% 256 Mb 1.0 Mb
# Each round's retransmissions become the next round's offered load.
# The tragedy of the commons.
def commons(senders, restrained):
"""If everyone restrains, all do well. Defectors gain — until all defect."""
greedy = senders - restrained
total_offered = restrained * 10 + greedy * 100
capacity = 100
if total_offered <= capacity:
return {"per_restrained": 10, "per_greedy": 100, "outcome": "stable"}
share = capacity / total_offered
return {"per_restrained": round(10 * share * share, 2),
"per_greedy": round(100 * share * share, 2),
"outcome": "congested — everyone's goodput falls"}
print()
for greedy_count in (0, 1, 5, 10):
c = commons(10, 10 - greedy_count)
print(f"{greedy_count:>3} greedy of 10: restrained get {c['per_restrained']:>6} Mb, "
f"greedy get {c['per_greedy']:>6} Mb ({c['outcome']})")
# 0 greedy of 10: restrained get 10 Mb, greedy get 100 Mb (stable)
# 1 greedy of 10: restrained get 2.77 Mb, greedy get 27.7 Mb (congested — everyone's goodput falls)
# 5 greedy of 10: restrained get 0.33 Mb, greedy get 3.31 Mb (congested — everyone's goodput falls)
# 10 greedy of 10: restrained get 0.1 Mb, greedy get 1.0 Mb (congested — everyone's goodput falls)
# All-greedy is far worse for EVERYONE than all-restrained.
# Loss as a congestion signal: right on wire, wrong on wireless.
def interpret_loss(medium):
return {
"wired": ("buffer overflow", True, "halving the rate is CORRECT"),
"wireless": ("radio interference", False,
"halving the rate is WRONG — the link is not congested"),
}[medium]
print()
for m in ("wired", "wireless"):
cause, correct, note = interpret_loss(m)
print(f"{m:<10} loss usually means {cause:<20} TCP's inference correct: "
f"{correct!s:<6} {note}")
# wired loss usually means buffer overflow TCP's inference correct: True halving the rate is CORRECT
# wireless loss usually means radio interference TCP's inference correct: False halving the rate is WRONG — the link is not congested
# This is why 802.11 retransmits at the LINK layer: to hide non-congestive
# loss from TCP entirely.
# Is this host experiencing congestion? Start with retransmission rate:
$ nstat -az | grep -iE "RetransSegs|OutSegs"
# TcpOutSegs 184221934
# TcpRetransSegs 1842219
$ python3 -c "print(f'retransmission rate: {1842219/184221934:.2%}')"
# retransmission rate: 1.00%
# Under 0.1% is healthy. Over 2% means real loss somewhere on the path.
# Distinguish queueing delay (stage 1) from loss (stage 2):
$ ping -c 100 -i 0.2 -q server | tail -2
# 100 packets transmitted, 100 received, 0% packet loss
# rtt min/avg/max/mdev = 12.104/48.221/312.884/61.204 ms
# ^^^^^^ empty queue ^^^^^^^ 300 ms of queueing
# Zero loss but 26x delay variation = STAGE 1: bufferbloat, not loss.
# Watch queues build in real time:
$ tc -s qdisc show dev eth0
# qdisc fq_codel 8001: root
# Sent 184221934 bytes 128412 pkt (dropped 412, overlimits 0 requeues 21)
# backlog 1284420b 892p
# ^^^^^^^^^^^^^^ 892 packets queued right now
# maxpacket 1514 drop_overlimit 0 new_flow_count 4821 ecn_mark 1204
# Per-connection evidence of congestion response:
$ ss -tin state established | head -4
# ESTAB 0 0 10.1.1.5:51250 93.184.216.34:443
# cubic wscale:7,7 rto:312 rtt:48.2/61.2 mss:1448 cwnd:12 ssthresh:12
# ^^^^^^^^^^^ huge RTT variance = queueing
# ^^^^^^^^^^^^^^^^^^
# cwnd == ssthresh: recently halved
# retrans:2/184 lost:1
# Congestion-related counters across the system:
$ nstat -az | grep -iE "Timeouts|FastRetrans|Lost|ECN"
# TcpExtTCPTimeouts 2841
# TcpExtTCPFastRetrans 14201
# TcpExtTCPLostRetransmit 112
# TcpExtTCPSynRetrans 94
# Simulate the stages and observe the collapse shape:
$ sudo tc qdisc add dev eth0 root netem rate 10mbit limit 1000
$ iperf3 -c server -b 5M -t 10 | tail -2 # stage 0: under capacity
$ iperf3 -c server -b 10M -t 10 | tail -2 # stage 1: at capacity
$ iperf3 -c server -b 50M -t 10 | tail -2 # stage 2/3: heavy overload
# [ 5] 0.00-10.00 sec 11.2 MBytes 9.41 Mbits/sec 1204 retransmits
# ^^^^ offered 5x,
# delivered the same, wasted the rest
$ sudo tc qdisc del dev eth0 root
# UDP flows that ignore congestion control crowd out TCP — observe it:
$ iperf3 -c server -u -b 100M -t 10 & # unresponsive UDP
$ iperf3 -c server -t 10 | tail -2 # TCP on the same path
# TCP backs off; UDP does not. TCP's share collapses. This is exactly
# why RFC 8085 requires UDP applications to implement congestion control.
Key takeaways
- Congestion is aggregate arrival rate exceeding link capacity; the link is fine, the demand is not.
- Three stages: queueing delay (throughput fine, latency rises), loss and retransmission (goodput falls below throughput), and congestion collapse.
- Collapse is stable and self-reinforcing — retransmissions add load to an already-full link, so it does not resolve itself.
- The 1986 LBL–Berkeley collapse dropped throughput from 32 kb/s to 40 bit/s, prompting Jacobson's slow start and congestion avoidance.
- Congestion control is a voluntary restraint protocol and a classic tragedy of the commons — it works because nearly all traffic implements it.
- This is why UDP applications must implement congestion control (RFC 8085); unresponsive flows crowd out well-behaved ones.
- Loss is the implicit signal, since IP gives no explicit one by default.
- That inference is correct on wired links (loss means buffer overflow) but wrong on wireless (loss often means interference), which is why 802.11 retransmits at the link layer to hide it.
- Goodput, not throughput, is the metric that matters — a saturated link carrying mostly retransmissions is delivering nothing.
🧪 Practice
- Name the three stages of congestion and state what degrades in each.
- Explain why congestion collapse is self-sustaining and what must happen to break it.
- Distinguish throughput from goodput and explain why the gap widens as loss increases.
- Interview-style: TCP treats packet loss as a congestion signal. On what kind of link is that inference wrong, and how do such links compensate? (Hint: ask what else besides a full buffer can destroy a frame, and which layer is best placed to repair it invisibly.)
Slow Start and Congestion Avoidance
Theory
A new connection faces a genuine dilemma: it has no idea how much capacity is available. Sending too fast immediately causes loss and possibly collapse. Sending too slowly wastes the link and the user's time.
TCP's answer is to probe: start conservatively and increase until the network pushes back. Two algorithms handle the two phases, and the switch between them is governed by ssthresh (slow start threshold).
Slow start grows the congestion window exponentially. It begins at the initial window — historically 1 MSS, now 10 MSS (RFC 6928) — and increases cwnd by one MSS for every ACK received. Since a full window's worth of ACKs arrives each RTT, the window doubles every round trip.
The name is misleading and worth addressing: slow start is not slow. It starts small but grows exponentially, reaching a 1 MB window in about seven round trips. "Slow" refers to starting from a small value rather than blasting at full rate, which is what earlier TCP did.
Congestion avoidance takes over once cwnd reaches ssthresh. Growth becomes linear: cwnd increases by roughly one MSS per RTT rather than per ACK. In practice each ACK adds MSS × MSS / cwnd, which sums to one MSS per round trip.
The rationale for switching is a change in what the sender knows. Below ssthresh, the sender has no evidence it is near capacity, so aggressive probing is appropriate. Above it, previous experience suggests capacity is close, so the sender approaches carefully — because overshooting causes loss that affects everyone.
The response to loss depends on how the loss was detected (7.3):
| Event | ssthresh | cwnd | Next phase |
|---|---|---|---|
| Timeout | cwnd / 2 | 1 MSS | Slow start |
| 3 dup ACKs | cwnd / 2 | ssthresh | Congestion avoidance |
The initial window of 10 (RFC 6928) is a significant modern change. Most web transfers are small — a typical page's critical resources fit in a handful of round trips — so the startup phase dominates. Raising the initial window from 3 to 10 segments (about 14 KB) lets a small response complete in one round trip instead of three, which measurably improved real-world page load times.
Slow start after idle is a related subtlety. If a connection sits idle, the network's state may have changed entirely, so the sender should not resume at its old rate. RFC 5681 says to reset cwnd after an idle period. Linux implements this via tcp_slow_start_after_idle, which is often disabled on long-lived internal connections where the path is known to be stable.
Examples
Slow start: exponential growth, one doubling per RTT
RTT 0: cwnd = 10 send 10 segments
RTT 1: 10 ACKs arrive, +1 MSS each -> cwnd = 20
RTT 2: cwnd = 40
RTT 3: cwnd = 80
RTT 4: cwnd = 160 <- reaches ssthresh (say 160): SWITCH
congestion avoidance: linear, +1 MSS per RTT
RTT 5: cwnd = 161
RTT 6: cwnd = 162
...
RTT n: LOSS -> back off
The classic sawtooth
cwnd
^
80 | /| /| /|
| / | / | / |
40 | / | / | ....../ | <- congestion avoidance
| / | / | / | (linear +1/RTT)
20 | / | / | / |
| / | / | / |
10 |__/ |____/ |___/ | <- slow start (exponential)
| v v v
+--------------------------------------> time
loss loss loss
cwnd halved each time (fast recovery)
Timeout vs 3 dup ACKs: very different severity
cwnd = 80 when loss occurs
3 DUP ACKS: ssthresh = 40, cwnd = 40 -> congestion avoidance
recovery: 40 RTTs to reach 80 again (linear)
TIMEOUT: ssthresh = 40, cwnd = 1 -> SLOW START
recovery: ~6 RTTs to reach 40 (exponential), then linear
but the connection was nearly IDLE during those RTTs
class CongestionControl:
def __init__(self, initial_window=10, ssthresh=64, mss=1448):
self.cwnd = initial_window
self.ssthresh = ssthresh
self.mss = mss
self.phase = "slow start"
def on_ack(self, acked_segments=1):
if self.cwnd < self.ssthresh:
self.phase = "slow start"
self.cwnd += acked_segments # +1 MSS per ACK -> doubles per RTT
else:
self.phase = "congestion avoidance"
self.cwnd += acked_segments / self.cwnd # sums to +1 MSS per RTT
return self.cwnd
def on_triple_dup_ack(self):
self.ssthresh = max(2, self.cwnd / 2)
self.cwnd = self.ssthresh
self.phase = "fast recovery -> congestion avoidance"
return f"3 dup ACKs: ssthresh={self.ssthresh:.0f}, cwnd={self.cwnd:.0f}"
def on_timeout(self):
self.ssthresh = max(2, self.cwnd / 2)
self.cwnd = 1
self.phase = "slow start"
return f"TIMEOUT: ssthresh={self.ssthresh:.0f}, cwnd=1 (collapsed)"
cc = CongestionControl(initial_window=10, ssthresh=64)
print(f"{'RTT':>5}{'cwnd':>9}{'phase':>26}{'bytes in flight':>18}")
for rtt in range(9):
in_flight = int(cc.cwnd) * cc.mss
print(f"{rtt:>5}{cc.cwnd:>9.1f}{cc.phase:>26}{in_flight:>15,} B")
acks = int(cc.cwnd)
for _ in range(acks):
cc.on_ack()
# RTT cwnd phase bytes in flight
# 0 10.0 slow start 14,480 B
# 1 20.0 slow start 28,960 B
# 2 40.0 slow start 57,920 B
# 3 64.2 congestion avoidance 92,672 B
# 4 65.2 congestion avoidance 94,120 B
# 5 66.2 congestion avoidance 95,568 B
# 6 67.2 congestion avoidance 97,016 B
# 7 68.2 congestion avoidance 98,464 B
# 8 69.2 congestion avoidance 99,912 B
# Doubling until ssthresh (64), then +1 per RTT. Note RTT 3: growth stops
# mid-doubling the moment cwnd crosses ssthresh.
# The two loss responses, compared.
print()
for event in ("dup", "timeout"):
cc = CongestionControl(initial_window=10, ssthresh=64)
for _ in range(200):
cc.on_ack()
before = cc.cwnd
msg = cc.on_triple_dup_ack() if event == "dup" else cc.on_timeout()
print(f" cwnd {before:.0f} -> {msg}")
# cwnd 66 -> 3 dup ACKs: ssthresh=33, cwnd=33
# cwnd 66 -> TIMEOUT: ssthresh=33, cwnd=1 (collapsed)
def rtts_to_recover(start, target, phase_ssthresh):
"""How many RTTs to climb back, given slow start below ssthresh."""
rtts, w = 0, start
while w < target:
w = w * 2 if w < phase_ssthresh else w + 1
rtts += 1
return rtts
print()
print(f" from cwnd 33 (dup ACK) -> 66: {rtts_to_recover(33, 66, 33)} RTTs")
print(f" from cwnd 1 (timeout) -> 66: {rtts_to_recover(1, 66, 33)} RTTs")
# from cwnd 33 (dup ACK) -> 66: 33 RTTs
# from cwnd 1 (timeout) -> 66: 8 RTTs
# The timeout LOOKS faster in RTT count, because slow start is exponential.
# But it spends those RTTs sending 1, 2, 4, 8 segments — almost nothing.
# Integrated over the recovery, fast recovery delivers far more data.
# Why the initial window matters for the web.
def rtts_for_transfer(size_bytes, initial_window, mss=1448):
sent, cwnd, rtts = 0, initial_window, 0
while sent < size_bytes:
sent += cwnd * mss
cwnd *= 2
rtts += 1
return rtts
print()
print(f"{'response size':>15}{'IW=1':>8}{'IW=3':>8}{'IW=10':>8}{'saving':>10}")
for size in (5_000, 14_000, 50_000, 200_000):
r1, r3, r10 = (rtts_for_transfer(size, iw) for iw in (1, 3, 10))
print(f"{size:>13,} B{r1:>8}{r3:>8}{r10:>8}{r3-r10:>8} RTT")
# response size IW=1 IW=3 IW=10 saving
# 5,000 B 3 2 1 1 RTT
# 14,000 B 4 3 1 2 RTT
# 50,000 B 6 4 3 1 RTT
# 200,000 B 8 6 4 2 RTT
# On a 100 ms path, saving 2 RTTs is 200 ms off every page load.
# Slow start is not slow: time to fill a large window.
def time_to_window(target_cwnd, initial=10, rtt_ms=100):
rtts, w = 0, initial
while w < target_cwnd:
w *= 2
rtts += 1
return rtts, rtts * rtt_ms
print()
for target in (100, 1000, 10000):
rtts, ms = time_to_window(target)
print(f" reach cwnd {target:>6,}: {rtts} RTTs = {ms:>5} ms on a 100 ms path")
# reach cwnd 100: 4 RTTs = 400 ms on a 100 ms path
# reach cwnd 1,000: 7 RTTs = 700 ms on a 100 ms path
# reach cwnd 10,000: 10 RTTs = 1000 ms on a 100 ms path
# Exponential growth reaches a 14 MB window in one second. Not slow.
# The initial window in use (RFC 6928 says 10):
$ ip route show | head -2
# default via 192.168.1.1 dev eth0 proto dhcp metric 100
$ ip route show | grep -o "initcwnd [0-9]*"
# (empty = kernel default, which is 10)
$ sudo ip route change default via 192.168.1.1 dev eth0 initcwnd 10 initrwnd 10
# Watch cwnd grow on a live connection:
$ while true; do
> ss -tin state established '( dport = :443 )' | grep -o "cwnd:[0-9]*" | head -1
> sleep 0.1
> done
# cwnd:10
# cwnd:20
# cwnd:40
# cwnd:80 <- doubling: slow start
# cwnd:81
# cwnd:82 <- +1 per RTT: congestion avoidance
# Per-connection phase indicator — cwnd vs ssthresh:
$ ss -tin state established | head -4
# ESTAB 0 0 10.1.1.5:51250 93.184.216.34:443
# cubic wscale:7,7 rto:204 rtt:3.8/1.2 mss:1448 cwnd:80 ssthresh:64
# ^^^^^^^ ^^^^^^^^^^^
# cwnd > ssthresh -> CONGESTION AVOIDANCE
# If ssthresh is absent, the connection is still in slow start.
# Slow start after idle — often disabled for long-lived internal connections:
$ sysctl net.ipv4.tcp_slow_start_after_idle
# net.ipv4.tcp_slow_start_after_idle = 1
$ sudo sysctl -w net.ipv4.tcp_slow_start_after_idle=0
# Safe on a stable internal path; risky across the internet, where the
# path may have changed completely while idle.
# Counters for the phases:
$ nstat -az | grep -iE "slowstart|cwnd"
# TcpExtTCPSlowStartRetrans 412 <- loss during slow start
# TcpExtTCPTimeouts 2841
# Observe the sawtooth directly:
$ ss -tin state established | grep -oE "cwnd:[0-9]+|ssthresh:[0-9]+" | paste - -
# cwnd:120 ssthresh:64
# cwnd:60 ssthresh:60 <- halved: loss detected via dup ACKs
# cwnd:61 ssthresh:60
# cwnd:62 ssthresh:60 <- linear climb resumes
# Measure the startup cost for a small response:
$ curl -w 'connect:%{time_connect} ttfb:%{time_starttransfer} total:%{time_total}\n' \
> -o /dev/null -s https://example.com/small.json
# connect:0.102 ttfb:0.208 total:0.210
# ^^^^^ ~1 RTT after the handshake: IW=10 covered it
# Compare with a larger response that needs several slow-start rounds:
$ curl -w 'ttfb:%{time_starttransfer} total:%{time_total}\n' \
> -o /dev/null -s https://example.com/1mb.bin
# ttfb:0.208 total:0.712
# ^^^^^ 5 extra RTTs of slow start to open the window
Key takeaways
- A new connection cannot know available capacity, so TCP probes: start small, grow until the network pushes back.
- Slow start grows
cwndby one MSS per ACK, doubling it every RTT — exponential, and not actually slow. - Congestion avoidance grows
cwndby one MSS per RTT — linear, cautious probing near suspected capacity. ssthreshis the boundary: below it, aggressive growth; above it, careful growth.- Loss responses differ by detection method: a timeout collapses
cwndto 1 MSS and re-enters slow start; 3 duplicate ACKs halve it tossthreshand continue in congestion avoidance. - The result is the characteristic sawtooth of TCP throughput.
- The initial window is now 10 MSS (RFC 6928, ~14 KB), letting most small web responses complete in one round trip instead of three.
- Slow start after idle resets
cwndafter inactivity because the path may have changed; it is often disabled on stable internal links.
🧪 Practice
- Starting at
cwnd10 withssthresh80, givecwndafter each of the first six RTTs and name the phase at each step.- A connection at
cwnd100 experiences a timeout, then later three duplicate ACKs. Givecwndandssthreshafter each.- Compute how many RTTs a 40 KB response needs with initial windows of 3 and 10, and the time saved on a 80 ms path.
- Interview-style: Why is slow start's exponential growth considered conservative rather than aggressive? (Hint: compare what it starts from with what the alternative was, and ask what it is trying to discover.)
AIMD and Fairness
Theory
The specific growth and reduction rules TCP uses are not arbitrary. AIMD (Additive Increase, Multiplicative Decrease) was chosen because it provably converges to a fair and efficient allocation among competing flows — a result from Chiu and Jain (1989) that is one of networking's genuinely elegant pieces of theory.
The rules:
no loss: cwnd = cwnd + 1 (additive increase, per RTT)
on loss: cwnd = cwnd x 0.5 (multiplicative decrease)
Why this combination and not another? Consider the alternatives:
| Policy | Convergence | Problem |
|---|---|---|
| AIAD (add/subtract) | Never converges to fairness | Differences persist forever |
| MIMD (multiply/divide) | Never converges to fairness | Ratios are preserved |
| MIAD (multiply up, subtract down) | Diverges | Unstable and unfair |
| AIMD | Converges to fairness | The one that works |
The intuition is geometric. Additive increase moves all flows the same absolute amount, which shifts the allocation toward equality in relative terms. Multiplicative decrease cuts flows proportionally, so a flow using more gives up more in absolute terms. Repeated, the two together drive any starting allocation toward the equal share.
On the standard two-flow diagram, additive increase moves along a 45° line (both gain equally) and multiplicative decrease moves along a line toward the origin (both cut proportionally). Alternating these two motions spirals in on the fairness line. No other combination of an increase and a decrease rule does this.
Fairness is measured with Jain's fairness index:
J = (sum of x_i)^2 / (n x sum of x_i^2)
It ranges from 1/n (one flow takes everything) to 1 (perfectly equal). It is convenient because it is scale-independent and has an intuitive reading: J = 0.5 with many flows roughly means half the flows are effectively starved.
Two important qualifications to TCP's "fairness" deserve emphasis, because the unqualified claim is misleading.
RTT unfairness. TCP increases cwnd once per RTT, so a flow with a short RTT grows faster and claims more bandwidth. Throughput is roughly proportional to 1/RTT. Two flows sharing a bottleneck with RTTs of 10 ms and 100 ms do not get equal shares — the short-RTT flow gets roughly 10 times more. This is why a nearby CDN beats a distant origin server on the same link, independent of any other factor.
Flow-count unfairness. Fairness is per flow, not per application or per user. An application opening 10 parallel connections gets 10 shares. This was the explicit motivation for browsers opening 6 connections per host in the HTTP/1.1 era, and one reason HTTP/2's single multiplexed connection was initially slower for some workloads — it was playing fair while competitors were not.
The Mathis equation captures TCP's throughput as a function of loss:
throughput ~ MSS / (RTT x sqrt(p))
This shows starkly why long, lossy paths perform badly: throughput falls with both RTT and the square root of loss rate. A 1% loss rate on a 100 ms path caps a single flow at roughly 1.2 Mb/s regardless of link capacity — which is precisely the problem that motivated CUBIC and BBR.
Examples
Why AIMD converges and the alternatives do not
Two flows sharing capacity C. Plot flow1 vs flow2.
flow2
^
C |\ fairness line (flow1 = flow2)
| \ /
| \ /
| \ / <- AIMD spirals INWARD toward this point
| \ /
| \ /
| \ /
| \ /
+--------X--------> flow1
\
efficiency line (flow1 + flow2 = C)
ADDITIVE INCREASE: move up-right at 45 degrees (both +1)
-> moves TOWARD the fairness line in relative terms
MULTIPLICATIVE DECREASE: move toward the origin along a ray
-> preserves the ratio but reduces magnitude
Alternating them converges on the intersection: FAIR and EFFICIENT.
AIAD: both moves are 45-degree lines -> the gap NEVER closes.
MIMD: both moves preserve the ratio -> the ratio NEVER changes.
RTT unfairness: the uncomfortable truth
flow A: RTT 10 ms -> +1 cwnd every 10 ms
flow B: RTT 100 ms -> +1 cwnd every 100 ms
A grows 10x faster -> A claims ~10x the bandwidth.
Same bottleneck, same algorithm, wildly different outcomes.
This is why CDN proximity matters so much.
def simulate_aimd(flows, capacity, rounds, policy="AIMD"):
"""Track convergence for different increase/decrease policies."""
rates = list(flows)
history = [list(rates)]
for _ in range(rounds):
# Increase
if policy in ("AIMD", "AIAD"):
rates = [r + 1 for r in rates] # additive
else:
rates = [r * 1.1 for r in rates] # multiplicative
# Decrease, only if over capacity
if sum(rates) > capacity:
if policy in ("AIMD", "MIMD"):
rates = [r * 0.5 for r in rates] # multiplicative
else:
rates = [max(1, r - 5) for r in rates] # additive
history.append(list(rates))
return history
def jain(rates):
n = len(rates)
return sum(rates) ** 2 / (n * sum(r ** 2 for r in rates))
print(f"{'policy':>7}{'start':>18}{'after 30 rounds':>24}{'fairness':>11}")
for policy in ("AIMD", "AIAD", "MIMD", "MIAD"):
h = simulate_aimd([40, 10], capacity=50, rounds=30, policy=policy)
final = [round(x, 1) for x in h[-1]]
print(f"{policy:>7}{str(h[0]):>18}{str(final):>24}{jain(h[-1]):>11.3f}")
# policy start after 30 rounds fairness
# AIMD [40, 10] [17.9, 14.1] 0.986
# AIAD [40, 10] [40, 10] 0.735
# MIMD [40, 10] [21.8, 5.5] 0.735
# MIAD [40, 10] [48.2, 1] 0.521
# AIMD climbs to 0.986 and keeps closing. AIAD and MIMD sit at the
# STARTING fairness of 0.735 — neither ever narrows the gap. MIAD gets worse.
# Watch AIMD converge step by step.
h = simulate_aimd([40, 10], capacity=50, rounds=12)
print(f"\n{'round':>6}{'flow A':>10}{'flow B':>10}{'ratio':>9}{'fairness':>11}")
for i, rates in enumerate(h[::2]):
ratio = rates[0] / rates[1] if rates[1] else float("inf")
print(f"{i*2:>6}{rates[0]:>10.1f}{rates[1]:>10.1f}{ratio:>9.2f}{jain(rates):>11.3f}")
# round flow A flow B ratio fairness
# 0 40.0 10.0 4.00 0.735
# 2 21.5 6.5 3.31 0.777
# 4 23.5 8.5 2.76 0.820
# 6 25.5 10.5 2.43 0.852
# 8 27.5 12.5 2.20 0.877
# 10 29.5 14.5 2.03 0.896
# 12 31.5 16.5 1.91 0.911
# The ratio falls monotonically: 4.00 -> 1.91. Each halving preserves the
# ratio, each +1 narrows it, and the +1 wins over time.
# Jain's fairness index across allocations.
print()
for label, rates in [("perfectly equal", [25, 25, 25, 25]),
("mild imbalance", [40, 30, 20, 10]),
("one dominant", [70, 10, 10, 10]),
("one flow only", [100, 0.001, 0.001, 0.001])]:
print(f"{label:<20}{str(rates):<28}J = {jain(rates):.3f}")
# perfectly equal [25, 25, 25, 25] J = 1.000
# mild imbalance [40, 30, 20, 10] J = 0.833
# one dominant [70, 10, 10, 10] J = 0.481
# one flow only [100, 0.001, 0.001, 0.001] J = 0.250
# RTT unfairness: throughput is proportional to 1/RTT.
def rtt_shares(rtts, capacity_mbps):
weights = [1 / r for r in rtts]
total = sum(weights)
return [capacity_mbps * w / total for w in weights]
print()
for rtts in ([10, 10], [10, 100], [10, 100, 200]):
shares = rtt_shares(rtts, 100)
pretty = ", ".join(f"{r}ms->{s:.1f}Mb" for r, s in zip(rtts, shares))
print(f" RTTs {str(rtts):<18} shares: {pretty} fairness {jain(shares):.3f}")
# RTTs [10, 10] shares: 10ms->50.0Mb, 10ms->50.0Mb fairness 1.000
# RTTs [10, 100] shares: 10ms->90.9Mb, 100ms->9.1Mb fairness 0.599
# RTTs [10, 100, 200] shares: 10ms->87.0Mb, 100ms->8.7Mb, 200ms->4.3Mb fairness 0.435
# Same algorithm, same bottleneck. RTT alone produces 10:1 unfairness.
# Flow-count unfairness: fairness is PER FLOW.
def flow_shares(connections_per_app, capacity_mbps):
total = sum(connections_per_app)
return [capacity_mbps * c / total for c in connections_per_app]
print()
for conns in ([1, 1], [6, 1], [10, 1]):
shares = flow_shares(conns, 100)
print(f" app A opens {conns[0]:>2} conn(s), app B opens {conns[1]}: "
f"A gets {shares[0]:>5.1f} Mb, B gets {shares[1]:>5.1f} Mb")
# app A opens 1 conn(s), app B opens 1: A gets 50.0 Mb, B gets 50.0 Mb
# app A opens 6 conn(s), app B opens 1: A gets 85.7 Mb, B gets 14.3 Mb
# app A opens 10 conn(s), app B opens 1: A gets 90.9 Mb, B gets 9.1 Mb
# This is exactly why browsers opened 6 connections per host.
# The Mathis equation: why long lossy paths are hopeless for one flow.
import math
def mathis(mss_bytes, rtt_ms, loss_rate):
return mss_bytes * 8 / ((rtt_ms / 1000) * math.sqrt(loss_rate)) / 1e6
print()
print(f"{'RTT':>7}{'loss':>8}{'max throughput':>18}")
for rtt in (10, 100, 300):
for loss in (0.0001, 0.001, 0.01):
print(f"{rtt:>5}ms{loss:>8.2%}{mathis(1448, rtt, loss):>15,.1f} Mb/s")
# RTT loss max throughput
# 10ms 0.01% 115.8 Mb/s
# 10ms 0.10% 36.6 Mb/s
# 10ms 1.00% 11.6 Mb/s
# 100ms 0.01% 11.6 Mb/s
# 100ms 0.10% 3.7 Mb/s
# 100ms 1.00% 1.2 Mb/s
# 300ms 0.01% 3.9 Mb/s
# 300ms 0.10% 1.2 Mb/s
# 300ms 1.00% 0.4 Mb/s
# A single Reno flow on a 300 ms path with 1% loss cannot exceed 0.4 Mb/s
# no matter how fast the link is. This motivated CUBIC and BBR.
# Observe the sawtooth that AIMD produces:
$ while true; do
> ss -tin state established '( dport = :5201 )' | grep -o "cwnd:[0-9]*"
> sleep 0.05
> done | head -20
# cwnd:120
# cwnd:121
# cwnd:122 <- additive increase, +1 per RTT
# cwnd:61 <- multiplicative decrease, x0.5
# cwnd:62
# cwnd:63
# Measure fairness between competing flows:
$ iperf3 -c server -P 4 -t 20 | tail -8
# [SUM] 0.00-20.00 sec 2.19 GBytes 940 Mbits/sec
# [ 5] 0.00-20.00 sec 561 MBytes 235 Mbits/sec
# [ 7] 0.00-20.00 sec 558 MBytes 234 Mbits/sec
# [ 9] 0.00-20.00 sec 563 MBytes 236 Mbits/sec
# [ 11] 0.00-20.00 sec 560 MBytes 235 Mbits/sec
$ python3 -c "
r=[235,234,236,235]; n=len(r)
print(f'Jain fairness: {sum(r)**2/(n*sum(x*x for x in r)):.4f}')"
# Jain fairness: 1.0000 <- equal RTTs, equal shares
# Demonstrate RTT unfairness by adding delay to one path:
$ sudo tc qdisc add dev eth0 root netem delay 100ms
$ iperf3 -c near-server -t 20 & # 10 ms RTT
$ iperf3 -c far-server -t 20 # 110 ms RTT
# The near flow takes roughly 10x the bandwidth. Same algorithm.
$ sudo tc qdisc del dev eth0 root
# Flow-count unfairness — one app taking more shares:
$ iperf3 -c server -P 1 -t 20 & # 1 connection
$ iperf3 -c server -P 6 -t 20 # 6 connections
# The 6-connection client gets ~6/7 of the bandwidth. Entirely "fair"
# by TCP's per-flow definition.
# Verify the Mathis prediction on a lossy path:
$ sudo tc qdisc add dev eth0 root netem loss 1% delay 50ms
$ iperf3 -c server -t 20 | tail -2
# [ 5] 0.00-20.00 sec 55.1 MBytes 23.1 Mbits/sec 1204 retransmits
$ python3 -c "
import math
print(f'Mathis predicts: {1448*8/((0.1)*math.sqrt(0.01))/1e6:.1f} Mb/s')"
# Mathis predicts: 11.6 Mb/s
# Same order of magnitude. CUBIC (the default) beats plain Reno here,
# which is exactly why it exists.
$ sudo tc qdisc del dev eth0 root
# Which algorithm is in use:
$ sysctl net.ipv4.tcp_congestion_control
# net.ipv4.tcp_congestion_control = cubic
$ sysctl net.ipv4.tcp_available_congestion_control
# net.ipv4.tcp_available_congestion_control = reno cubic bbr
Key takeaways
- AIMD — additive increase (+1 per RTT), multiplicative decrease (×0.5 on loss) — provably converges to a fair and efficient allocation (Chiu and Jain, 1989).
- The alternatives fail: AIAD and MIMD never close the gap between flows; MIAD diverges.
- Geometrically, additive increase moves toward the fairness line while multiplicative decrease cuts proportionally; alternating them spirals inward.
- Jain's fairness index
J = (Σx)² / (n·Σx²)ranges from1/nto1. - RTT unfairness: throughput is roughly proportional to 1/RTT, so a 10 ms flow beats a 100 ms flow about 10:1 on the same bottleneck.
- Flow-count unfairness: fairness is per flow, so an application opening 6 connections gets 6 shares — the reason browsers did exactly that.
- The Mathis equation
throughput ≈ MSS / (RTT × √p)shows throughput falling with both RTT and the square root of loss. - A single classic flow on a 300 ms path with 1% loss is capped near 4 Mb/s regardless of capacity — the motivation for CUBIC and BBR.
🧪 Practice
- Explain why AIAD and MIMD fail to converge to fairness while AIMD does.
- Compute Jain's fairness index for allocations
[50, 50],[80, 20], and[90, 5, 5].- Use the Mathis equation to compute maximum single-flow throughput for MSS 1,448 on a 200 ms path with 0.5% loss.
- Interview-style: Two flows share a bottleneck and both run standard TCP, yet one gets 8x the throughput. Give two explanations that involve no misconfiguration. (Hint: TCP's increase is clocked by something, and its fairness is defined per something — neither is what most people assume.)
TCP Reno, NewReno, and CUBIC
Theory
TCP's congestion control has been repeatedly revised, and each revision addressed a specific failure of the previous one. Understanding the sequence explains why today's default behaves as it does.
TCP Tahoe (1988) introduced slow start, congestion avoidance, and fast retransmit. Its weakness: any loss collapsed cwnd to 1 and re-entered slow start, even when duplicate ACKs proved data was still flowing.
TCP Reno (1990) added fast recovery (7.3): on three duplicate ACKs, halve cwnd rather than collapsing it. This was a large throughput improvement. Its weakness appears with multiple losses in one window — Reno exits recovery on the first new ACK, so a second loss in the same window typically causes a timeout.
TCP NewReno (RFC 6582) fixed exactly that with the concept of a partial ACK: an ACK that advances but does not cover the entire window outstanding when recovery began. NewReno stays in fast recovery, retransmits the next hole immediately, and repeats. It recovers multiple losses at one per RTT without a timeout — a major improvement, though still slower than SACK's all-at-once recovery.
TCP CUBIC (RFC 8312) is the current Linux default and addresses a different problem: on high bandwidth-delay product paths, AIMD's linear increase is far too slow. After a loss on a 10 Gb/s, 100 ms path, recovering the window at +1 MSS per RTT takes over an hour.
CUBIC replaces the linear growth with a cubic function of time since the last congestion event:
cwnd = C x (t - K)^3 + W_max
where W_max is the window before the loss, K is the time to return to it, and C is a scaling constant. The shape is the key idea:
- Immediately after loss — grows rapidly, quickly recovering most of the lost window.
- Near
W_max— flattens out, probing cautiously around the level that previously caused loss. This is the plateau, and it is where CUBIC spends most of its time. - Beyond
W_max— accelerates again to probe for newly available capacity.
Two further properties matter:
CUBIC is RTT-independent. Growth is a function of absolute time, not round trips, so flows with different RTTs grow at the same rate. This substantially reduces the RTT unfairness of classic AIMD.
CUBIC has a TCP-friendly region. On low-BDP paths where classic Reno would grow faster, CUBIC falls back to Reno-like behaviour so it does not starve legacy flows.
| Algorithm | Key idea | Weakness addressed |
|---|---|---|
| Tahoe | Slow start, fast retransmit | — |
| Reno | Fast recovery | Tahoe's collapse to 1 |
| NewReno | Partial ACKs | Reno's timeout on multiple losses |
| CUBIC | Cubic growth, RTT-independent | Reno's slow growth on high-BDP paths |
| BBR | Model bandwidth and RTT | Loss-based signalling entirely |
Other variants appear in specific niches: Vegas (delay-based, largely displaced), Compound TCP (Windows, hybrid), DCTCP (data centres, uses ECN for fine-grained feedback), and BBR (the next topic).
Examples
The sequence of fixes
TAHOE: any loss -> cwnd = 1, slow start
cwnd 80 -> 1. Recovery takes many RTTs at near-zero throughput.
RENO: 3 dup ACKs -> cwnd = cwnd/2, congestion avoidance
cwnd 80 -> 40. Much better.
BUT two losses in one window -> exits recovery too early
-> the second loss causes a TIMEOUT
NEWRENO: tracks the "recovery point" (highest sequence sent when
recovery began). A PARTIAL ACK (advances but < recovery point)
means another hole exists -> retransmit it, STAY in recovery.
Two losses -> 2 RTTs, no timeout.
CUBIC: the problem is not loss recovery but GROWTH RATE.
Why linear growth fails on fat long pipes
10 Gb/s, 100 ms RTT -> BDP = 125 MB = ~86,000 segments
after a loss: cwnd halves to 43,000 segments
Reno recovery at +1 segment per RTT:
43,000 RTTs x 100 ms = 4,300 seconds = 1.2 HOURS
CUBIC recovers most of it in a few seconds.
CUBIC's curve
cwnd
^
Wmax|. . . . . . . ___________ . . . . . . . . . <- the plateau:
| _/ \_ cautious probing
| _/ \ around Wmax
| _/ \
| _/ \_
| _/ \___
| _/ <- fast recovery of the \__ <- accelerates to
| / lost window probe for NEW capacity
|/
+--------------------------------------> time since loss
^
loss
Concave, then flat, then convex. Most time is spent near Wmax —
exactly where the network told us capacity lies.
import math
class Reno:
def __init__(self, cwnd=10, ssthresh=64):
self.cwnd, self.ssthresh = cwnd, ssthresh
self.name = "Reno"
def on_rtt(self):
if self.cwnd < self.ssthresh:
self.cwnd *= 2 # slow start
else:
self.cwnd += 1 # ADDITIVE increase
return self.cwnd
def on_loss(self):
self.ssthresh = max(2, self.cwnd / 2)
self.cwnd = self.ssthresh # MULTIPLICATIVE decrease
return self.cwnd
class Cubic:
C = 0.4
BETA = 0.7 # CUBIC reduces to 0.7, not 0.5
def __init__(self, cwnd=10):
self.cwnd = cwnd
self.w_max = cwnd
self.t = 0.0
self.k = 0.0
self.name = "CUBIC"
def on_rtt(self, rtt_s=0.1):
self.t += rtt_s
# cwnd = C(t - K)^3 + W_max — a function of TIME, not RTT count
self.cwnd = max(1, self.C * (self.t - self.k) ** 3 + self.w_max)
return self.cwnd
def on_loss(self):
self.w_max = self.cwnd
self.cwnd = self.cwnd * self.BETA
self.k = (self.w_max * (1 - self.BETA) / self.C) ** (1 / 3)
self.t = 0.0
return self.cwnd
# Recovery after a loss at cwnd 1000, on a 100 ms path.
print(f"{'RTT':>5}{'Reno cwnd':>12}{'CUBIC cwnd':>13}")
reno, cubic = Reno(cwnd=1000, ssthresh=1000), Cubic(cwnd=1000)
reno.on_loss(); cubic.on_loss()
for i in range(0, 61, 10):
print(f"{i:>5}{reno.cwnd:>12.0f}{cubic.cwnd:>13.0f}")
for _ in range(10):
reno.on_rtt(); cubic.on_rtt()
# RTT Reno cwnd CUBIC cwnd
# 0 500 700
# 10 510 789
# 20 520 858
# 30 530 910
# 40 540 947
# 50 550 973
# 60 560 988
# After 60 RTTs (6 s) CUBIC is back to 988 of its original 1000; Reno has
# crawled from 500 to 560 and needs ~440 more RTTs to finish.
# How long does Reno take to recover on a high-BDP path?
def reno_recovery_time(bdp_segments, rtt_ms):
to_recover = bdp_segments / 2 # halved by the loss
rtts = to_recover # +1 segment per RTT
seconds = rtts * rtt_ms / 1000
return rtts, seconds
print()
for bw_gbps, rtt in [(0.1, 10), (1, 50), (10, 100)]:
bdp_bytes = bw_gbps * 1e9 / 8 * (rtt / 1000)
segs = bdp_bytes / 1448
rtts, secs = reno_recovery_time(segs, rtt)
print(f"{bw_gbps:>5} Gb/s, {rtt:>3} ms: BDP {segs:>9,.0f} segments -> "
f"Reno needs {secs:>9,.0f} s ({secs/60:>6,.1f} min) to recover")
# 0.1 Gb/s, 10 ms: BDP 86 segments -> Reno needs 0 s ( 0.0 min) to recover
# 1 Gb/s, 50 ms: BDP 4,316 segments -> Reno needs 108 s ( 1.8 min) to recover
# 10 Gb/s, 100 ms: BDP 86,326 segments -> Reno needs 4,316 s ( 71.9 min) to recover
# Over an hour. This is the problem CUBIC was built to solve.
# NewReno's partial ACK handling.
class NewReno:
def __init__(self):
self.in_recovery = False
self.recovery_point = None
def enter_recovery(self, highest_sent):
self.in_recovery = True
self.recovery_point = highest_sent
return f"enter recovery, recovery_point={highest_sent}"
def on_ack(self, ack):
if not self.in_recovery:
return f"ACK {ack}: normal"
if ack < self.recovery_point:
return (f"ACK {ack}: PARTIAL ACK (< {self.recovery_point}) -> "
f"another hole exists; retransmit it, STAY in recovery")
self.in_recovery = False
return f"ACK {ack}: FULL ACK -> recovery complete, exit"
nr = NewReno()
print()
print(f" {nr.enter_recovery(10000)}")
for ack in (3000, 6000, 10001):
print(f" {nr.on_ack(ack)}")
# enter recovery, recovery_point=10000
# ACK 3000: PARTIAL ACK (< 10000) -> another hole exists; retransmit it, STAY in recovery
# ACK 6000: PARTIAL ACK (< 10000) -> another hole exists; retransmit it, STAY in recovery
# ACK 10001: FULL ACK -> recovery complete, exit
# Reno would have exited at ACK 3000 and timed out on the next hole.
# Recovery of multiple losses, by algorithm.
def recovery_rtts(losses, algorithm):
return {
"Reno": 1 if losses == 1 else "TIMEOUT (~200 ms)",
"NewReno": losses,
"SACK": 1,
}[algorithm]
print()
print(f"{'losses in one window':>22}{'Reno':>20}{'NewReno':>10}{'SACK':>7}")
for n in (1, 3, 5):
print(f"{n:>22}{str(recovery_rtts(n,'Reno')):>20}"
f"{str(recovery_rtts(n,'NewReno')):>10}{str(recovery_rtts(n,'SACK')):>7}")
# losses in one window Reno NewReno SACK
# 1 1 1 1
# 3 TIMEOUT (~200 ms) 3 1
# 5 TIMEOUT (~200 ms) 5 1
# The reduction factor differs too.
print()
for name, beta in [("Reno/NewReno", 0.5), ("CUBIC", 0.7)]:
print(f"{name:<14} on loss: cwnd x {beta} -> from 1000 becomes {1000*beta:.0f}")
# Reno/NewReno on loss: cwnd x 0.5 -> from 1000 becomes 500
# CUBIC on loss: cwnd x 0.7 -> from 1000 becomes 700
# CUBIC gives up less, recovers faster, and spends more time near capacity.
# Which algorithm is active, and what else is available:
$ sysctl net.ipv4.tcp_congestion_control net.ipv4.tcp_available_congestion_control
# net.ipv4.tcp_congestion_control = cubic
# net.ipv4.tcp_available_congestion_control = reno cubic bbr
# Load another module and switch:
$ sudo modprobe tcp_bbr
$ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr
# Per-connection algorithm — visible in ss:
$ ss -tin state established | head -3
# ESTAB 0 0 10.1.1.5:51250 93.184.216.34:443
# cubic wscale:7,7 rto:204 rtt:3.8/1.2 mss:1448 cwnd:120
# ^^^^^ this connection is using CUBIC
# Set it per socket rather than globally:
$ python3 -c "
import socket
s = socket.socket()
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_CONGESTION, b'bbr')
print('this socket uses BBR; the rest of the system is unaffected')"
# Or per route, which is often the cleanest:
$ sudo ip route change default via 192.168.1.1 dev eth0 congctl bbr
# Compare algorithms on a high-BDP lossy path:
$ sudo tc qdisc add dev eth0 root netem delay 100ms loss 0.5%
$ for algo in reno cubic bbr; do
> sudo sysctl -qw net.ipv4.tcp_congestion_control=$algo
> echo -n "$algo: "
> iperf3 -c server -t 15 | awk '/sender/{print $7, $8}'
> done
# reno: 18.2 Mbits/sec
# cubic: 94.1 Mbits/sec <- 5x better on this path
# bbr: 382.4 Mbits/sec <- loss-tolerant; see the next topic
$ sudo tc qdisc del dev eth0 root
# CUBIC's plateau, visible in cwnd samples:
$ while true; do
> ss -tin state established '( dport = :5201 )' | grep -o "cwnd:[0-9]*"
> sleep 0.1
> done | head -12
# cwnd:700 <- after loss (x0.7, not x0.5)
# cwnd:703
# cwnd:769 <- concave: fast recovery of the lost window
# cwnd:943
# cwnd:988
# cwnd:996 <- PLATEAU: cautious probing near Wmax
# cwnd:999
# cwnd:1001
# cwnd:1063 <- convex: probing for NEW capacity
# NewReno behaviour is folded into the kernel's recovery logic:
$ nstat -az | grep -iE "recovery|partial"
# TcpExtTCPSACKRecovery 8412
# TcpExtTCPRenoRecovery 41 <- non-SACK path (rare today)
# TcpExtTCPFullUndo 94
# TcpExtTCPPartialUndo 212
Key takeaways
- Tahoe collapsed
cwndto 1 on any loss; Reno added fast recovery to halve it instead. - Reno's weakness is multiple losses in one window — it exits recovery on the first new ACK and then times out.
- NewReno uses partial ACKs to detect further holes, staying in recovery and fixing one loss per RTT without a timeout.
- CUBIC addresses a different problem: AIMD's linear growth is far too slow on high-BDP paths — Reno needs over an hour to recover on a 10 Gb/s, 100 ms path.
- CUBIC grows as a cubic function of time since the last loss: fast recovery of the lost window, a plateau near
W_max, then acceleration to probe for new capacity. - CUBIC is RTT-independent (growth depends on absolute time), which reduces classic AIMD's RTT unfairness.
- CUBIC reduces by 0.7 rather than 0.5, giving up less on each loss.
- CUBIC includes a TCP-friendly region so it does not starve Reno flows on low-BDP paths.
- CUBIC is the Linux default; alternatives include DCTCP (data centres, ECN-based) and BBR.
🧪 Practice
- State the specific weakness each of Reno, NewReno, and CUBIC was designed to fix.
- Compute the time for Reno to recover its window after a loss on a 1 Gb/s link with 80 ms RTT.
- Describe the three phases of CUBIC's growth curve and what the sender is doing in each.
- Interview-style: CUBIC's growth depends on elapsed time rather than round trips. What fairness problem does that fix, and what new consideration does it introduce? (Hint: recall what classic AIMD's per-RTT increase does to flows with different path lengths — then ask how CUBIC coexists with those flows.)
BBR and Model-Based Congestion Control
Theory
Every algorithm so far treats loss as the congestion signal. That made sense when router buffers were small: a full buffer meant a congested link, and dropping a packet was prompt feedback.
Modern buffers broke that assumption. Routers, switches, and home gateways now ship with buffers far larger than any reasonable BDP, so a loss-based sender fills the buffer before it gets any feedback at all. Throughput is fine; latency is dreadful. The sender is optimising for the wrong thing — and it is the direct cause of bufferbloat (next topic).
Loss-based control also conflates two different things. On a wireless or long-haul path, loss may be random rather than congestive, and reacting to it needlessly collapses throughput (the Mathis equation, 7.5).
BBR (Bottleneck Bandwidth and Round-trip propagation time), developed at Google and deployed from 2016, takes a fundamentally different approach: build a model of the path and send according to the model, rather than reacting to loss.
BBR continuously estimates two quantities:
BtlBw (bottleneck bandwidth) — the maximum delivery rate observed, tracked as a windowed maximum over roughly 10 round trips. RTprop (round-trip propagation time) — the minimum RTT observed, tracked as a windowed minimum over about 10 seconds.
From these it derives the BDP = BtlBw × RTprop, and it paces sending at approximately that rate, keeping in flight about 2× the BDP as a ceiling.
The theoretical foundation is Kleinrock's optimal operating point: a connection achieves maximum throughput with minimum delay exactly when it sends at the bottleneck rate with just enough data in flight to keep the link busy — and no queue. Loss-based algorithms operate to the right of that point, filling the queue; BBR aims to sit on it.
BBR cycles through four states:
| State | Purpose |
|---|---|
| Startup | Exponential growth (like slow start) until bandwidth stops increasing |
| Drain | Send below the estimated rate to empty the queue created during startup |
| ProbeBW | Steady state: cycle pacing gain around 1.0 (up to 1.25, down to 0.75) to probe for more bandwidth |
| ProbeRTT | Every 10 s, cut the window to 4 packets briefly to re-measure the true minimum RTT |
ProbeRTT is necessary because RTprop can only be measured when the queue is empty — and if the connection keeps a queue, the minimum RTT estimate drifts upward and the model degrades. Briefly draining gives a clean measurement.
The practical results are striking on paths where loss-based control struggles:
| Path | CUBIC | BBR |
|---|---|---|
| Long-haul with 1% random loss | Collapses (Mathis) | Largely unaffected |
| Deep-buffered last mile | Fills the buffer; high latency | Keeps the queue near empty |
| Typical broadband | Comparable | Comparable or better |
The criticisms are real and worth knowing. BBRv1 could be unfair to CUBIC flows on shallow-buffered links, taking more than its share, and could induce persistent queues when multiple BBR flows shared a bottleneck. BBRv2 and v3 address these by incorporating loss and ECN signals alongside the model — a hybrid rather than a pure model-based design.
BBR is available in Linux (tcp_bbr) and is used extensively by Google and by YouTube.
Examples
Kleinrock's optimal point, and where each algorithm sits
delivery rate
^
BtlBw |........________________________
| /:
| / :
| / :
| / : <- BBR operates HERE (the knee)
+---/----:--------------------------> data in flight
| |
RTprop |....:________________________
round | : /
trip | : /
time | : / <- CUBIC/Reno operate HERE
^ | : / (queue full, latency high)
| |____:/
+--------------------------------> data in flight
^
BDP = BtlBw x RTprop
max throughput, min delay, NO QUEUE
Why loss-based control fails on a deep buffer
link: 10 Mb/s, RTT 20 ms -> BDP = 25 KB
router buffer: 1 MB (40x the BDP — entirely typical)
CUBIC: grows cwnd until LOSS.
Loss happens only when the 1 MB buffer is FULL.
-> 1 MB sitting in the queue
-> queueing delay = 1 MB / 10 Mb/s = 800 ms
-> throughput 10 Mb/s, latency 820 ms. Bufferbloat.
BBR: measures BtlBw = 10 Mb/s, RTprop = 20 ms
-> sends at 10 Mb/s with ~25 KB in flight
-> throughput 10 Mb/s, latency 22 ms.
SAME throughput. 37x better latency.
The four states
STARTUP ----> DRAIN ----> PROBE_BW <-----> PROBE_RTT
(double each (empty the (steady state, (every 10 s,
RTT until queue cycle gain cut to 4 pkts
bandwidth startup 1.25/0.75/1.0 to re-measure
plateaus) created) to probe) RTprop)
class BBR:
"""Model-based: estimate BtlBw and RTprop, then pace to the model."""
GAIN_CYCLE = [1.25, 0.75, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
def __init__(self):
self.btlbw = 0.0 # bytes/sec, windowed MAX
self.rtprop = float("inf") # seconds, windowed MIN
self.state = "STARTUP"
self.cycle_index = 0
self.bw_samples, self.rtt_samples = [], []
def update(self, delivery_rate, rtt):
# BtlBw: max over ~10 RTTs. RTprop: min over ~10 s.
self.bw_samples = (self.bw_samples + [delivery_rate])[-10:]
self.rtt_samples = (self.rtt_samples + [rtt])[-10:]
self.btlbw = max(self.bw_samples)
self.rtprop = min(self.rtt_samples)
def bdp(self):
return self.btlbw * self.rtprop
def pacing_rate(self):
if self.state == "STARTUP":
return self.btlbw * 2.89 # 2/ln2: exponential growth
if self.state == "DRAIN":
return self.btlbw * 0.35 # below the rate, to empty the queue
if self.state == "PROBE_RTT":
return self.btlbw * 0.1
gain = self.GAIN_CYCLE[self.cycle_index % len(self.GAIN_CYCLE)]
return self.btlbw * gain
def cwnd(self):
return self.bdp() * 2 # cap in-flight at 2 x BDP
def advance(self):
self.cycle_index += 1
bbr = BBR()
# Steady state on a 10 Mb/s, 20 ms path.
for _ in range(10):
bbr.update(delivery_rate=10e6 / 8, rtt=0.020)
bbr.state = "PROBE_BW"
print(f"BtlBw : {bbr.btlbw*8/1e6:.1f} Mb/s")
print(f"RTprop: {bbr.rtprop*1000:.0f} ms")
print(f"BDP : {bbr.bdp()/1024:.1f} KB")
print(f"cwnd : {bbr.cwnd()/1024:.1f} KB (2 x BDP)")
print(f"\n{'cycle':>6}{'gain':>7}{'pacing rate':>16}")
for i in range(8):
print(f"{i:>6}{BBR.GAIN_CYCLE[i]:>7}{bbr.pacing_rate()*8/1e6:>13.1f} Mb/s")
bbr.advance()
# BtlBw : 10.0 Mb/s
# RTprop: 20 ms
# BDP : 24.4 KB
# cwnd : 48.8 KB (2 x BDP)
#
# cycle gain pacing rate
# 0 1.25 12.5 Mb/s
# 1 0.75 7.5 Mb/s
# 2 1.0 10.0 Mb/s
# 3 1.0 10.0 Mb/s
# 4 1.0 10.0 Mb/s
# 5 1.0 10.0 Mb/s
# 6 1.0 10.0 Mb/s
# 7 1.0 10.0 Mb/s
# Probe up 25%, back off 25% to drain, then hold. Net queue: ~zero.
# Loss-based vs model-based on a deep-buffered link.
def compare(link_mbps, rtprop_ms, buffer_bytes):
bdp = link_mbps * 1e6 / 8 * (rtprop_ms / 1000)
cubic_inflight = bdp + buffer_bytes # fills the buffer, then loses
cubic_delay = rtprop_ms + buffer_bytes * 8 / (link_mbps * 1e6) * 1000
bbr_inflight = bdp # model says: just the BDP
bbr_delay = rtprop_ms
return {
"bdp_kb": bdp / 1024,
"buffer_kb": buffer_bytes / 1024,
"cubic": (link_mbps, cubic_delay, cubic_inflight / 1024),
"bbr": (link_mbps, bbr_delay, bbr_inflight / 1024),
}
print()
for link, rtt, buf, label in [(10, 20, 1_000_000, "deep-buffered DSL"),
(100, 10, 128_000, "modest buffer"),
(1000, 1, 256_000, "data centre")]:
c = compare(link, rtt, buf)
print(f"{label} ({link} Mb/s, {rtt} ms, {c['buffer_kb']:.0f} KB buffer, "
f"BDP {c['bdp_kb']:.0f} KB):")
print(f" CUBIC: {c['cubic'][0]:>5} Mb/s latency {c['cubic'][1]:>7.1f} ms")
print(f" BBR : {c['bbr'][0]:>5} Mb/s latency {c['bbr'][1]:>7.1f} ms "
f"({c['cubic'][1]/c['bbr'][1]:.0f}x better)")
# deep-buffered DSL (10 Mb/s, 20 ms, 977 KB buffer, BDP 24 KB):
# CUBIC: 10 Mb/s latency 820.0 ms
# BBR : 10 Mb/s latency 20.0 ms (41x better)
# modest buffer (100 Mb/s, 10 ms, 125 KB buffer, BDP 122 KB):
# CUBIC: 100 Mb/s latency 20.2 ms
# BBR : 100 Mb/s latency 10.0 ms (2x better)
# data centre (1000 Mb/s, 1 ms, 250 KB buffer, BDP 122 KB):
# CUBIC: 1000 Mb/s latency 3.0 ms
# BBR : 1000 Mb/s latency 1.0 ms (3x better)
# Identical throughput in every row. The buffer-to-BDP ratio is what
# decides how much latency the loss-based sender adds.
# Random loss: where loss-based control collapses and BBR does not.
import math
def throughput_under_loss(algorithm, mss, rtt_ms, loss):
if algorithm == "loss-based":
return mss * 8 / ((rtt_ms / 1000) * math.sqrt(loss)) / 1e6
return None # BBR's model ignores non-congestive loss
print()
print(f"{'loss rate':>11}{'CUBIC/Reno (Mathis)':>24}{'BBR':>22}")
for loss in (0.0001, 0.001, 0.01, 0.05):
t = throughput_under_loss("loss-based", 1448, 100, loss)
print(f"{loss:>10.2%}{t:>21,.1f} Mb/s{'unaffected by the model':>22}")
# loss rate CUBIC/Reno (Mathis) BBR
# 0.01% 11.6 Mb/s unaffected by the model
# 0.10% 3.7 Mb/s unaffected by the model
# 1.00% 1.2 Mb/s unaffected by the model
# 5.00% 0.5 Mb/s unaffected by the model
# BBR paces to measured bandwidth, so random loss does not collapse it.
# Why ProbeRTT is necessary.
def rtprop_drift(queue_ms, probe_rtt):
true_rtprop = 20
if probe_rtt:
return true_rtprop, "accurate: the queue was drained to measure it"
return true_rtprop + queue_ms, "INFLATED: a standing queue hides the true minimum"
print()
for probe in (True, False):
est, note = rtprop_drift(queue_ms=80, probe_rtt=probe)
print(f" ProbeRTT {'ON ' if probe else 'OFF'} -> RTprop estimate {est:>3} ms ({note})")
# ProbeRTT ON -> RTprop estimate 20 ms (accurate: the queue was drained to measure it)
# ProbeRTT OFF -> RTprop estimate 100 ms (INFLATED: a standing queue hides the true minimum)
# An inflated RTprop inflates the BDP, which inflates cwnd, which BUILDS
# a queue — a self-reinforcing error. ProbeRTT breaks the loop.
# The fairness criticism of BBRv1.
def bbr_v1_fairness(buffer_depth_bdp):
if buffer_depth_bdp < 1:
return "BBRv1 takes MORE than its share: it ignores the loss CUBIC reacts to"
return "roughly fair: both converge near the BDP"
print()
for depth in (0.25, 1.0, 4.0):
print(f" buffer = {depth:>4} x BDP -> {bbr_v1_fairness(depth)}")
# buffer = 0.25 x BDP -> BBRv1 takes MORE than its share: it ignores the loss CUBIC reacts to
# buffer = 1.0 x BDP -> roughly fair: both converge near the BDP
# buffer = 4.0 x BDP -> roughly fair: both converge near the BDP
# BBRv2/v3 add loss and ECN signals to the model to address exactly this.
# Enable BBR:
$ sudo modprobe tcp_bbr
$ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr
$ sudo sysctl -w net.core.default_qdisc=fq
# ^^ BBR needs a PACING qdisc.
# Without fq (or fq_codel), BBR sends in bursts and the model degrades.
$ sysctl net.ipv4.tcp_congestion_control net.core.default_qdisc
# net.ipv4.tcp_congestion_control = bbr
# net.core.default_qdisc = fq
# Per-connection BBR state — the model is visible:
$ ss -tin state established | head -4
# ESTAB 0 0 10.1.1.5:51250 93.184.216.34:443
# bbr wscale:7,7 rto:204 rtt:20.1/0.8 mss:1448 cwnd:22 pacing_rate 9.8Mbps
# ^^^ ^^^^^^^^ ^^^^^^ ^^^^^^^^^^^^^^^^^
# algorithm RTprop-ish ~2xBDP paced, not bursted
# bbr:(bw:10.1Mbps,mrtt:20.05,pacing_gain:1,cwnd_gain:2)
# ^^^^^^^^^^^^ BtlBw ^^^^^^^^^^^^ RTprop — the model, exposed
# The bufferbloat comparison, measured end to end:
$ sudo tc qdisc add dev eth0 root netem rate 10mbit delay 10ms limit 700
# ^^^^^ ~1 MB buffer
$ for algo in cubic bbr; do
> sudo sysctl -qw net.ipv4.tcp_congestion_control=$algo
> iperf3 -c server -t 15 >/dev/null &
> sleep 5
> echo -n "$algo latency under load: "
> ping -c 5 -q server | tail -1 | cut -d= -f2
> wait
> done
# cubic latency under load: 21.104/812.441/1204.882/402.1 ms
# ^^^^^^^ 812 ms average: the buffer is full
# bbr latency under load: 20.114/24.882/31.204/3.1 ms
# ^^^^^^ 25 ms: the queue stays near empty
$ sudo tc qdisc del dev eth0 root
# BBR on a lossy long-haul path — where Mathis predicts collapse:
$ sudo tc qdisc add dev eth0 root netem delay 100ms loss 1%
$ for algo in cubic bbr; do
> sudo sysctl -qw net.ipv4.tcp_congestion_control=$algo
> echo -n "$algo: "; iperf3 -c server -t 15 | awk '/sender/{print $7,$8}'
> done
# cubic: 8.4 Mbits/sec
# bbr: 287.1 Mbits/sec <- 34x, by ignoring non-congestive loss
$ sudo tc qdisc del dev eth0 root
# Watch the state machine — pacing_rate changes reveal the gain cycle:
$ while true; do
> ss -tin state established '( dport = :5201 )' | grep -o "pacing_rate [0-9.]*[MG]bps"
> sleep 0.2
> done | head -8
# pacing_rate 28.4Mbps <- STARTUP (gain 2.89)
# pacing_rate 12.1Mbps <- DRAIN
# pacing_rate 12.5Mbps <- PROBE_BW, gain 1.25
# pacing_rate 7.5Mbps <- PROBE_BW, gain 0.75
# pacing_rate 10.0Mbps <- PROBE_BW, gain 1.0
# Verify pacing is actually happening (bursts would break the model):
$ tc -s qdisc show dev eth0 | head -4
# qdisc fq 8001: root refcnt 2 limit 10000p flow_limit 100p
# Sent 184221934 bytes 128412 pkt (dropped 0, overlimits 0 requeues 0)
# throttled 8412 <- fq is spacing packets out
Key takeaways
- Loss-based control assumes loss means congestion, which fails on deep buffers (no feedback until the buffer is full) and on lossy links (loss is not congestive).
- BBR builds a model of the path instead: BtlBw (windowed max delivery rate) and RTprop (windowed min RTT).
- It paces at BtlBw with roughly 2 × BDP in flight, targeting Kleinrock's optimal point — maximum throughput, minimum delay, no queue.
- Four states: Startup (exponential), Drain (empty the startup queue), ProbeBW (cycle gain 1.25/0.75/1.0), ProbeRTT (every 10 s, drain briefly to re-measure).
- ProbeRTT is essential because a standing queue inflates the RTprop estimate, which inflates the BDP, which builds more queue — a self-reinforcing error.
- On a deep-buffered link BBR achieves the same throughput with dramatically lower latency; on lossy long-haul paths it avoids the Mathis collapse entirely.
- BBR requires a pacing qdisc (
fq) — without it, bursts degrade the model. - BBRv1 could be unfair to CUBIC on shallow buffers; BBRv2/v3 add loss and ECN signals to the model.
🧪 Practice
- Name the two quantities BBR estimates and how each is measured (max or min, over what window).
- A 20 Mb/s link has 15 ms RTprop and a 2 MB buffer. Compute the queueing delay a loss-based sender induces versus BBR.
- Explain why ProbeRTT exists and what goes wrong without it.
- Interview-style: BBR achieves the same throughput as CUBIC on a broadband link but much lower latency. Where does that latency go, and why does CUBIC incur it? (Hint: ask what CUBIC is waiting for before it stops increasing, and where the data sits in the meantime.)
Explicit Congestion Notification
Theory
Every congestion signal discussed so far is implicit: the sender infers congestion from loss or delay. That is a strange arrangement — the network knows perfectly well when it is congested, but its only way to say so is to destroy a packet.
ECN (Explicit Congestion Notification, RFC 3168) lets the network say it directly. A router experiencing congestion marks a packet instead of dropping it, the receiver echoes the mark back, and the sender reduces its rate — with no loss and no retransmission.
The mechanism uses the two ECN bits in the IP header's ToS/Traffic Class byte (Chapter 5):
| Codepoint | Bits | Meaning |
|---|---|---|
| Not-ECT | 00 | Endpoint does not support ECN |
| ECT(1) | 01 | ECN-capable transport |
| ECT(0) | 10 | ECN-capable transport |
| CE | 11 | Congestion Experienced — set by a router |
Two ECT codepoints exist so a sender can detect a router that erases marks (the "ECN nonce", now largely repurposed by L4S).
Feedback uses two TCP flags:
ECE (ECN Echo) — the receiver sets this in ACKs to tell the sender a CE mark was seen. It keeps setting it until acknowledged. CWR (Congestion Window Reduced) — the sender sets this to confirm it has reacted, which stops the ECE echoing.
Negotiation happens in the handshake: the SYN carries ECE+CWR, and the SYN-ACK carries ECE alone. Both ends must agree, or ECN is off for the connection.
The benefits are substantial:
No loss means no retransmission, no wasted bandwidth, and no retransmission delay. Earlier signalling — a router can mark when a queue starts to build, well before it overflows, so queues stay short. No head-of-line blocking from a recovery that never needed to happen.
Deployment was historically poor, for a specific and instructive reason: some early middleboxes dropped packets with ECN bits set, so enabling ECN broke connectivity entirely. This made operators reluctant to enable it, which made it not worth fixing the middleboxes. Linux's default of tcp_ecn=2 reflects the resolution: accept ECN when a peer requests it, but do not request it outbound. Failed negotiation is thus never fatal.
Two modern developments matter:
DCTCP (Data Center TCP) uses ECN much more aggressively. Instead of treating a mark like a loss (halve the window), it measures the fraction of marked packets and reduces proportionally. This gives fine-grained feedback and keeps data-centre queues extremely short. It requires a controlled environment, since it is not fair to standard TCP.
L4S (Low Latency, Low Loss, Scalable throughput, RFC 9330) generalises the DCTCP approach for the internet, using ECT(1) to identify L4S traffic so it can be queued separately from classic traffic. It is the current direction for achieving low latency without sacrificing throughput.
Examples
Without ECN: the network must destroy a packet to communicate
sender ---> [router queue filling] ---> receiver
|
queue FULL -> DROP a packet
|
sender detects loss (3 dup ACKs or timeout)
-> retransmit the lost data <- wasted bandwidth
-> halve cwnd <- the actual signal
-> receiver stalls waiting for the hole <- head-of-line blocking
Cost of communicating "slow down": one lost packet, one retransmission,
and one round trip of delay.
With ECN: the network says so directly
sender (ECT(0)) ---> [router queue building] ---> receiver
|
mark CE (11) — do NOT drop
|
packet arrives INTACT
|
receiver <--- ACK with ECE flag set ---
sender: halve cwnd, set CWR on the next segment
-> NO loss, NO retransmission, NO stall
Cost of communicating "slow down": two bits.
Negotiation in the handshake
SYN with ECE+CWR set "I support ECN"
SYN-ACK with ECE set "so do I"
-> ECN active for this connection
SYN with ECE+CWR set
SYN-ACK with neither "I do not"
-> ECN off; the connection proceeds normally
ECN_CODEPOINTS = {
0b00: ("Not-ECT", "endpoint does not support ECN — routers must DROP to signal"),
0b01: ("ECT(1)", "ECN-capable; used by L4S to identify scalable traffic"),
0b10: ("ECT(0)", "ECN-capable transport"),
0b11: ("CE", "CONGESTION EXPERIENCED — set by a router, not the sender"),
}
for bits, (name, meaning) in ECN_CODEPOINTS.items():
print(f" {bits:02b} {name:<9}{meaning}")
class ECNConnection:
def __init__(self, cwnd=100):
self.cwnd = cwnd
self.ecn_negotiated = False
self.ece_pending = False
def negotiate(self, syn_ece_cwr, synack_ece):
self.ecn_negotiated = syn_ece_cwr and synack_ece
return ("ECN ACTIVE for this connection" if self.ecn_negotiated
else "ECN off — falling back to loss-based signalling")
def router_marks(self):
if not self.ecn_negotiated:
return "router must DROP the packet (sender is not ECN-capable)"
self.ece_pending = True
return "router sets CE (11) — packet delivered INTACT"
def receiver_acks(self):
if self.ece_pending:
return "ACK with ECE flag set — 'I saw congestion'"
return "ordinary ACK"
def sender_reacts(self):
if not self.ece_pending:
return f"no action; cwnd stays {self.cwnd}"
before = self.cwnd
self.cwnd = max(2, self.cwnd // 2)
self.ece_pending = False
return (f"cwnd {before} -> {self.cwnd}, set CWR on the next segment. "
f"NO retransmission needed.")
c = ECNConnection(cwnd=100)
print()
print(f" {c.negotiate(True, True)}")
print(f" {c.router_marks()}")
print(f" {c.receiver_acks()}")
print(f" {c.sender_reacts()}")
# ECN ACTIVE for this connection
# router sets CE (11) — packet delivered INTACT
# ACK with ECE flag set — 'I saw congestion'
# cwnd 100 -> 50, set CWR on the next segment. NO retransmission needed.
# The cost of signalling with a drop versus with two bits.
def signal_cost(method, mss=1448, rtt_ms=50):
if method == "drop":
return {"bytes_wasted": mss, "retransmission": True,
"added_delay_ms": rtt_ms, "receiver_stall": True}
return {"bytes_wasted": 0, "retransmission": False,
"added_delay_ms": 0, "receiver_stall": False}
print()
for m in ("drop", "ecn mark"):
c = signal_cost(m)
print(f"{m:<10} wasted {c['bytes_wasted']:>5} B, retransmit={c['retransmission']!s:<6}"
f" delay +{c['added_delay_ms']:>3} ms, receiver stall={c['receiver_stall']}")
# drop wasted 1448 B, retransmit=True delay + 50 ms, receiver stall=True
# ecn mark wasted 0 B, retransmit=False delay + 0 ms, receiver stall=False
# Classic ECN vs DCTCP: coarse versus proportional response.
def react(marked_fraction, style, cwnd=100):
if style == "classic ECN":
# ANY mark in a window is treated like a loss.
return cwnd // 2 if marked_fraction > 0 else cwnd
# DCTCP: reduce in PROPORTION to the marked fraction.
alpha = marked_fraction
return max(2, int(cwnd * (1 - alpha / 2)))
print()
print(f"{'marked':>8}{'classic ECN':>14}{'DCTCP':>9}")
for frac in (0.0, 0.01, 0.10, 0.50, 1.0):
print(f"{frac:>7.0%}{react(frac,'classic ECN'):>14}{react(frac,'DCTCP'):>9}")
# marked classic ECN DCTCP
# 0% 100 100
# 1% 50 99
# 10% 50 95
# 50% 50 75
# 100% 50 50
# Classic ECN overreacts to light congestion; DCTCP's proportional
# response keeps queues short without sacrificing throughput.
# Why deployment stalled, and how Linux resolved it.
TCP_ECN_VALUES = {
0: ("disabled", "never use ECN"),
1: ("request and accept", "may break on hostile middleboxes"),
2: ("accept only (DEFAULT)", "safe: never initiates, so never breaks"),
}
print()
for v, (mode, note) in TCP_ECN_VALUES.items():
print(f" tcp_ecn={v} {mode:<24}{note}")
# tcp_ecn=0 disabled never use ECN
# tcp_ecn=1 request and accept may break on hostile middleboxes
# tcp_ecn=2 accept only (DEFAULT) safe: never initiates, so never breaks
def middlebox_outcome(sender_requests, middlebox_drops_ect):
if sender_requests and middlebox_drops_ect:
return "CONNECTION FAILS — the SYN with ECE+CWR is discarded"
if sender_requests:
return "ECN negotiated successfully"
return "ECN not requested; connection works regardless"
print()
for req, hostile in [(True, True), (True, False), (False, True)]:
print(f" requests={req!s:<5} hostile middlebox={hostile!s:<5} -> "
f"{middlebox_outcome(req, hostile)}")
# requests=True hostile middlebox=True -> CONNECTION FAILS — the SYN with ECE+CWR is discarded
# requests=True hostile middlebox=False -> ECN negotiated successfully
# requests=False hostile middlebox=True -> ECN not requested; connection works regardless
# This asymmetry is exactly why the default is "accept, never request".
# ECN configuration:
$ sysctl net.ipv4.tcp_ecn net.ipv4.tcp_ecn_fallback
# net.ipv4.tcp_ecn = 2 # accept if offered, never request
# net.ipv4.tcp_ecn_fallback = 1 # retry without ECN if the SYN fails
$ sudo sysctl -w net.ipv4.tcp_ecn=1 # request ECN outbound too
# ECN bits on the wire — they live in the IP ToS byte:
$ sudo tcpdump -i eth0 -n -v 'tcp port 443' -c 2
# IP (tos 0x2,ECT(0), ttl 64, ...) <- sender marked ECN-capable
# ^^^^^^^^^^
# IP (tos 0x3,CE, ttl 61, ...) <- a ROUTER set CE
# ^^^^^^^
# Filter for congestion marks specifically:
$ sudo tcpdump -i eth0 'ip[1] & 0x03 == 3' -c 5 # CE-marked packets
$ sudo tcpdump -i eth0 'tcp[tcpflags] & 0x40 != 0' -c 5 # ECE flag in ACKs
# ECN negotiation in the handshake:
$ tshark -r capture.pcap -Y 'tcp.flags.syn==1' -T fields \
> -e tcp.flags.ecn -e tcp.flags.cwr -e ip.dsfield.ecn | head -2
# 1 1 0 <- SYN with ECE+CWR: "I support ECN"
# 1 0 0 <- SYN-ACK with ECE: "so do I"
# ECN counters:
$ nstat -az | grep -iE "ecn|CE"
# TcpExtTCPECNSeen 4821 <- CE marks observed
# TcpExtTCPECNSentCWR 4812 <- our responses
# IpExtInECT0Pkts 184221934
# IpExtInCEPkts 4821 <- packets arriving CE-marked
# IpExtInNoECTPkts 12041
# Per-connection ECN state:
$ ss -tin state established | grep -o "ecn\|ecnseen" | head -2
# ecn
# ecnseen
# 'ecn' = negotiated; 'ecnseen' = a CE mark has actually been received.
# Make a router mark rather than drop — AQM with ECN enabled:
$ sudo tc qdisc replace dev eth0 root fq_codel ecn
$ tc -s qdisc show dev eth0
# qdisc fq_codel 8001: root ... ecn
# Sent 184221934 bytes (dropped 12, overlimits 0)
# maxpacket 1514 drop_overlimit 0 ecn_mark 4821 drop_overflow 0
# ^^^^^^^^^^^^ 4821 marked instead of dropped
$ sudo tc qdisc replace dev eth0 root fq_codel noecn # compare
# With noecn, the ecn_mark counter stays 0 and drop counts rise instead.
# DCTCP for a controlled data-centre fabric:
$ sudo modprobe tcp_dctcp
$ sudo sysctl -w net.ipv4.tcp_congestion_control=dctcp
$ sudo sysctl -w net.ipv4.tcp_ecn=1
$ sudo tc qdisc replace dev eth0 root red limit 1000000 min 30000 max 35000 \
> avpkt 1500 burst 20 ecn
# DCTCP needs a shallow, aggressive marking threshold — and it is NOT
# fair to standard TCP, so use it only inside a controlled network.
# Measure the benefit: latency under load, with and without ECN marking:
$ sudo tc qdisc replace dev eth0 root fq_codel noecn
$ iperf3 -c server -t 15 >/dev/null & sleep 5; ping -c5 -q server | tail -1
# rtt min/avg/max/mdev = 20.1/48.2/92.4/18.1 ms
$ sudo tc qdisc replace dev eth0 root fq_codel ecn
$ iperf3 -c server -t 15 >/dev/null & sleep 5; ping -c5 -q server | tail -1
# rtt min/avg/max/mdev = 20.1/24.8/31.2/3.1 ms <- shorter queues, no loss
Key takeaways
- Without ECN the network's only way to signal congestion is to destroy a packet — costing a retransmission, a round trip, and a receiver stall.
- ECN lets a router mark instead of drop, using the two ECN bits in the IP ToS byte: Not-ECT
00, ECT(1)01, ECT(0)10, CE11. - Feedback uses two TCP flags: ECE (receiver reports a CE mark) and CWR (sender confirms it reacted).
- ECN is negotiated in the handshake — SYN with ECE+CWR, SYN-ACK with ECE.
- Benefits: no loss, no retransmission, no head-of-line blocking, and signalling that can begin before the queue overflows.
- Deployment stalled because some middleboxes dropped ECN-marked SYNs, so Linux defaults to
tcp_ecn=2— accept if offered, never request. - DCTCP reacts proportionally to the fraction of marked packets rather than halving, keeping data-centre queues very short; it is not fair to standard TCP.
- L4S (RFC 9330) generalises this for the internet, using ECT(1) to separate scalable traffic into its own queue.
- A router only marks if an AQM is configured to do so (
fq_codel ecn,red ... ecn).
🧪 Practice
- Give the four ECN codepoints with their bit patterns and state which one only a router sets.
- Trace the full ECN signalling sequence from a router marking a packet to the sender reducing its window, naming both TCP flags.
- Compare classic ECN's response with DCTCP's for marked fractions of 2% and 60%.
- Interview-style: Linux accepts ECN but does not request it by default. Why is that asymmetry the safe choice? (Hint: consider what happens to a SYN that carries ECN bits through a middlebox that dislikes them, and who bears the cost of that failure.)
Active Queue Management and Bufferbloat
Theory
Bufferbloat is the pathology of excessively large network buffers. It is worth stating plainly because it inverts an intuition: adding memory to a router makes the network worse.
The mechanism follows directly from everything in this subchapter. A loss-based sender grows its window until it sees loss. Loss occurs only when a buffer overflows. So the sender fills whatever buffer exists before receiving any feedback at all. With a 1 MB buffer on a 10 Mb/s link, that is 800 ms of queued data sitting in front of every subsequent packet — including the interactive traffic that has nothing to do with the bulk transfer.
The result is the familiar experience of a household where one large download makes video calls stutter and web pages crawl, while a speed test still reports the full line rate. Throughput is fine. Latency is destroyed.
The cause was economics and misunderstanding: memory became cheap, "more buffer means fewer drops" sounded obviously correct, and nobody was measuring latency under load. Jim Gettys named and popularised the problem around 2010.
Active Queue Management (AQM) is the fix: drop or mark packets before the queue is full, so senders get feedback early and queues stay short.
The progression of algorithms:
RED (Random Early Detection, 1993) drops probabilistically as the average queue length grows between a minimum and maximum threshold. Randomness avoids global synchronisation, where all flows back off together and then ramp up together, producing oscillation. RED works, but its thresholds must be tuned per link, and mis-tuning is common enough that it was rarely deployed correctly.
CoDel (Controlled Delay, 2012) was designed to need no tuning. Its key insight is to measure the time a packet spends in the queue, not the queue's length in bytes — because delay is what actually matters and it is comparable across link speeds. If the minimum sojourn delay stays above a 5 ms target for a 100 ms interval, CoDel starts dropping, increasing the drop rate until delay falls. It distinguishes a good queue (a transient burst, which drains) from a bad queue (a standing backlog, which does not).
FQ-CoDel combines CoDel with fair queueing: traffic is hashed into many queues, each managed by CoDel, and served round-robin. This gives isolation as well as short queues — a bulk transfer cannot delay an interactive flow, because they are in different queues. It is the default on modern Linux and the standard recommendation for consumer gateways.
CAKE adds shaping, NAT awareness, DiffServ handling, and per-host fairness on top of FQ-CoDel, and is the usual choice for home routers where the bottleneck is a known-rate uplink.
The practical guidance:
| Situation | Recommendation |
|---|---|
| Home router / consumer gateway | CAKE with the uplink rate configured |
| Linux server or host | fq_codel (usually already the default) |
| Data centre | DCTCP with shallow ECN marking |
| Any bottleneck link | Enable ECN marking in the AQM |
The critical detail for home links is that AQM must run where the bottleneck is. If the bottleneck is the ISP's equipment, queueing happens there and a local AQM sees no queue. The workaround is to shape slightly below the actual line rate — typically 90–95% — so the bottleneck moves into your own router where the AQM can manage it.
Examples
Bufferbloat: throughput fine, latency destroyed
10 Mb/s link, 1 MB buffer, loss-based sender
sender grows cwnd -> no loss yet -> grows more -> ...
loss occurs ONLY when the 1 MB buffer overflows
steady state: 1 MB permanently queued
queueing delay = 1,000,000 x 8 / 10,000,000 = 800 ms
speed test: 10 Mb/s "my internet is fine"
video call: 800 ms delay "why is everything terrible"
The two observations are BOTH correct.
Buffer size versus delay
buffer queueing delay at 10 Mb/s
------ -------------------------
10 KB 8 ms <- fine
64 KB 51 ms <- noticeable
256 KB 205 ms <- bad
1 MB 800 ms <- unusable for interactive traffic
8 MB 6,400 ms <- entirely typical of old consumer gear
CoDel: measure TIME, not bytes
Why time? Because 100 packets means very different things:
on a 1 Gb/s link: 1.2 ms (harmless)
on a 1 Mb/s link: 1.2 s (catastrophic)
The same queue LENGTH, wildly different user experience.
Sojourn DELAY is comparable across link speeds and needs no tuning.
GOOD QUEUE (a burst): BAD QUEUE (a standing backlog):
delay spikes, then drains delay stays above 5 ms for > 100 ms
-> leave it alone -> start dropping, ramp up until it clears
FQ-CoDel: isolation as well as short queues
without FQ: [ bulk ][ bulk ][ bulk ][ VOIP ][ bulk ]...
^ waits behind everything
with FQ: queue 1: [ bulk ][ bulk ][ bulk ]
queue 2: [ VOIP ] <- served round-robin
queue 3: [ DNS ]
-> the VoIP packet leaves almost immediately, regardless
of how much bulk traffic exists
def queueing_delay(buffer_bytes, link_mbps):
return buffer_bytes * 8 / (link_mbps * 1e6) * 1000 # ms
print(f"{'buffer':>10}{'1 Mb/s':>12}{'10 Mb/s':>12}{'100 Mb/s':>12}{'1 Gb/s':>11}")
for buf in (10_240, 65_536, 262_144, 1_048_576, 8_388_608):
row = "".join(f"{queueing_delay(buf, s):>11,.0f}ms" for s in (1, 10, 100, 1000))
print(f"{buf//1024:>8} KB{row}")
# buffer 1 Mb/s 10 Mb/s 100 Mb/s 1 Gb/s
# 10 KB 82ms 8ms 1ms 0ms
# 64 KB 524ms 52ms 5ms 1ms
# 256 KB 2,097ms 210ms 21ms 2ms
# 1024 KB 8,389ms 839ms 84ms 8ms
# 8192 KB 67,109ms 6,711ms 671ms 67ms
# The SAME buffer is harmless at 1 Gb/s and catastrophic at 1 Mb/s.
# This is why sizing buffers in BYTES is the wrong approach.
# How big should a buffer be? The BDP, not "as big as memory allows".
def right_sized_buffer(link_mbps, rtt_ms):
bdp = link_mbps * 1e6 / 8 * (rtt_ms / 1000)
return bdp
print()
for link, rtt in [(10, 20), (100, 20), (1000, 20)]:
ideal = right_sized_buffer(link, rtt)
typical = 1_048_576
print(f"{link:>5} Mb/s, {rtt} ms RTT: ideal buffer {ideal/1024:>7,.0f} KB, "
f"typical shipped {typical/1024:.0f} KB "
f"({typical/ideal:>5,.0f}x too large)")
# 10 Mb/s, 20 ms RTT: ideal buffer 24 KB, typical shipped 1024 KB ( 42x too large)
# 100 Mb/s, 20 ms RTT: ideal buffer 244 KB, typical shipped 1024 KB ( 4x too large)
# 1000 Mb/s, 20 ms RTT: ideal buffer 2,441 KB, typical shipped 1024 KB ( 0x too large)
class CoDel:
TARGET_MS = 5.0 # acceptable standing queue delay
INTERVAL_MS = 100.0 # how long it must persist before acting
def __init__(self):
self.dropping = False
self.first_above_time = None
self.count = 0
def tick(self, sojourn_ms, now_ms):
"""One packet dequeued; decide whether to drop it."""
if sojourn_ms < self.TARGET_MS:
self.first_above_time = None
if self.dropping:
self.dropping = False
self.count = 0
return "deliver — delay back under target, stop dropping"
return "deliver — queue is healthy"
if self.first_above_time is None:
self.first_above_time = now_ms
return f"deliver — above target but only just ({sojourn_ms:.0f} ms)"
if now_ms - self.first_above_time < self.INTERVAL_MS:
return (f"deliver — above target for "
f"{now_ms - self.first_above_time:.0f} ms (< {self.INTERVAL_MS:.0f})")
self.dropping = True
self.count += 1
return f"DROP (or ECN-mark) — standing queue confirmed, drop #{self.count}"
cd = CoDel()
print()
for t, sojourn in [(0, 2), (10, 12), (50, 15), (120, 18), (160, 20), (200, 3)]:
print(f" t={t:>4} ms sojourn={sojourn:>3} ms -> {cd.tick(sojourn, t)}")
# t= 0 ms sojourn= 2 ms -> deliver — queue is healthy
# t= 10 ms sojourn= 12 ms -> deliver — above target but only just (12 ms)
# t= 50 ms sojourn= 15 ms -> deliver — above target for 40 ms (< 100)
# t= 120 ms sojourn= 18 ms -> DROP (or ECN-mark) — standing queue confirmed, drop #1
# t= 160 ms sojourn= 20 ms -> DROP (or ECN-mark) — standing queue confirmed, drop #2
# t= 200 ms sojourn= 3 ms -> deliver — delay back under target, stop dropping
# A 40 ms burst is tolerated; a 100 ms STANDING queue is not.
# Why time, not length, is the right metric.
def queue_meaning(packets, link_mbps, mss=1500):
delay = packets * mss * 8 / (link_mbps * 1e6) * 1000
verdict = "harmless" if delay < 5 else "noticeable" if delay < 50 else "catastrophic"
return delay, verdict
print()
for link in (1, 10, 100, 1000):
d, v = queue_meaning(100, link)
print(f" 100 packets queued on a {link:>4} Mb/s link = {d:>8,.1f} ms ({v})")
# 100 packets queued on a 1 Mb/s link = 1,200.0 ms (catastrophic)
# 100 packets queued on a 10 Mb/s link = 120.0 ms (catastrophic)
# 100 packets queued on a 100 Mb/s link = 12.0 ms (noticeable)
# 100 packets queued on a 1000 Mb/s link = 1.2 ms (harmless)
# FQ-CoDel: isolation between flows.
def latency_for_interactive(bulk_flows, fair_queueing, packet_ms=1.2):
if fair_queueing:
return packet_ms * 2, "own queue, served round-robin"
return packet_ms * bulk_flows * 100, "queued behind every bulk packet"
print()
for fq in (False, True):
for bulk in (1, 5):
lat, note = latency_for_interactive(bulk, fq)
label = "FQ-CoDel" if fq else "single FIFO"
print(f" {label:<12} {bulk} bulk flow(s): VoIP latency {lat:>8,.1f} ms ({note})")
# single FIFO 1 bulk flow(s): VoIP latency 120.0 ms (queued behind every bulk packet)
# single FIFO 5 bulk flow(s): VoIP latency 600.0 ms (queued behind every bulk packet)
# FQ-CoDel 1 bulk flow(s): VoIP latency 2.4 ms (own queue, served round-robin)
# FQ-CoDel 5 bulk flow(s): VoIP latency 2.4 ms (own queue, served round-robin)
# FQ-CoDel's latency is INDEPENDENT of how much bulk traffic exists.
# Shaping below line rate to move the bottleneck where you control it.
def where_is_the_bottleneck(local_shape_mbps, isp_rate_mbps):
if local_shape_mbps is None or local_shape_mbps >= isp_rate_mbps:
return ("ISP equipment", "your AQM sees no queue and can do nothing")
return ("your router", "your AQM manages the queue — bufferbloat controlled")
print()
for shape in (None, 100, 90):
where, note = where_is_the_bottleneck(shape, 100)
label = f"shape at {shape} Mb/s" if shape else "no shaping"
print(f" {label:<20} bottleneck: {where:<16} ({note})")
# no shaping bottleneck: ISP equipment (your AQM sees no queue and can do nothing)
# shape at 100 Mb/s bottleneck: ISP equipment (your AQM sees no queue and can do nothing)
# shape at 90 Mb/s bottleneck: your router (your AQM manages the queue — bufferbloat controlled)
# Measure bufferbloat: latency idle versus under load. This is THE test.
$ ping -c 10 -q 8.8.8.8 | tail -1
# rtt min/avg/max/mdev = 12.104/12.884/13.442/0.412 ms <- idle
$ iperf3 -c server -t 20 >/dev/null &
$ sleep 3 && ping -c 10 -q 8.8.8.8 | tail -1
# rtt min/avg/max/mdev = 14.201/612.884/1204.442/402.1 ms <- under load
# ^^^^^^^ 47x worse. That is bufferbloat.
$ wait
# Or use the purpose-built tools:
$ flent rrul -H server -o bufferbloat.png
$ dslreports.com/speedtest # reports a bufferbloat grade
# What queue discipline is in use?
$ tc qdisc show dev eth0
# qdisc fq_codel 0: root refcnt 2 limit 10240p flows 1024 quantum 1514
# target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
# ^^^^^^^^^^^^^^^^^^ CoDel's two parameters, rarely needing change
# Queue statistics — look for a standing backlog:
$ tc -s qdisc show dev eth0
# qdisc fq_codel 8001: root
# Sent 184221934 bytes 128412 pkt (dropped 412, overlimits 0 requeues 21)
# backlog 1284420b 892p
# ^^^^^^^^^^^^^^ 892 packets queued RIGHT NOW
# maxpacket 1514 drop_overlimit 0 new_flow_count 4821 ecn_mark 1204
# ^^^^^^^^^^^^ marking,
# not dropping
# Fix a Linux host (usually already the default on modern kernels):
$ sudo sysctl -w net.core.default_qdisc=fq_codel
$ sudo tc qdisc replace dev eth0 root fq_codel ecn
# Fix a home router — shape BELOW the line rate so you own the bottleneck:
$ sudo tc qdisc replace dev eth0 root cake bandwidth 90mbit docsis nat ack-filter
# ^^^^^^^^^ ~90-95% of the real rate
# ^^^^^^ link-layer overhead
# compensation
# ^^^ per-HOST fairness
# behind NAT
$ tc -s qdisc show dev eth0 | head -6
# qdisc cake 8001: root refcnt 2 bandwidth 90Mbit diffserv3 triple-isolate nat
# Sent 184221934 bytes 128412 pkt (dropped 4821, overlimits 184221 requeues 0)
# backlog 0b 0p requeues 0
# ^^^^^^ near-zero standing queue: exactly the goal
# Compare AQM options on the same link:
$ for q in pfifo_fast fq_codel cake; do
> sudo tc qdisc replace dev eth0 root $q 2>/dev/null
> iperf3 -c server -t 15 >/dev/null &
> sleep 5
> echo -n "$q: "; ping -c 5 -q server | tail -1 | cut -d/ -f5
> wait
> done
# pfifo_fast: 684.221 <- a plain FIFO: full bufferbloat
# fq_codel: 24.882 <- short queues plus flow isolation
# cake: 22.104 <- similar, plus shaping and per-host fairness
# Throughput is essentially identical in all three. Latency is not.
# Data centre: DCTCP with shallow ECN marking:
$ sudo tc qdisc replace dev eth0 root red limit 1000000 min 30000 max 35000 \
> avpkt 1500 burst 20 ecn
$ sudo sysctl -w net.ipv4.tcp_congestion_control=dctcp
Key takeaways
- Bufferbloat is excessive buffering: a loss-based sender fills whatever buffer exists before receiving any feedback, so a big buffer becomes permanent latency.
- The signature is full throughput with terrible latency — a speed test looks fine while interactive traffic is unusable.
- Buffers should be sized to the BDP, not to available memory; typical consumer gear ships buffers tens of times too large for low-speed links.
- AQM drops or marks before the queue fills, giving senders early feedback and keeping queues short.
- RED drops probabilistically by average queue length and avoids global synchronisation, but needs per-link tuning.
- CoDel measures sojourn time, not queue length, because time is comparable across link speeds; it acts when delay exceeds a 5 ms target for a 100 ms interval.
- CoDel distinguishes a good queue (a transient burst) from a bad queue (a standing backlog).
- FQ-CoDel adds flow isolation, so interactive latency becomes independent of how much bulk traffic exists; CAKE adds shaping, NAT awareness, and per-host fairness.
- AQM must run at the bottleneck — on a home link, shape to 90–95% of the line rate to move the bottleneck into your own router.
- Enabling ECN marking in the AQM gives the benefit without any loss at all.
🧪 Practice
- Compute the queueing delay a 512 KB buffer introduces at 5 Mb/s, 50 Mb/s, and 500 Mb/s, and explain why byte-based sizing is the wrong approach.
- Explain why CoDel measures time in queue rather than queue length, giving a concrete two-link comparison.
- Describe the test that reveals bufferbloat and what result indicates a problem.
- Interview-style: A home user installs fq_codel on their router and sees no improvement in latency under load. What is the most likely reason? (Hint: AQM can only manage a queue it actually holds — ask where the packets are really piling up.)
<a id="76-modern-transport-protocols"></a>
7.6 Modern Transport Protocols
This closing subchapter covers the transports built after TCP's limitations became structural: QUIC, SCTP, and Multipath TCP, and the constraints that shaped each.
QUIC Design Goals and Streams
Theory
TCP has accumulated four limitations that cannot be fixed within TCP itself, and QUIC (RFC 9000) was designed to address all of them at once.
1. Ossification. TCP is implemented in the kernel and inspected by middleboxes. Deploying a change requires operating system updates on billions of devices and tolerance from every firewall on every path. TCP Fast Open took a decade to reach partial deployment. The protocol is effectively frozen.
2. Handshake latency. A TCP connection costs 1 RTT, and TLS 1.2 adds 2 more. Three round trips before any application data moves. TLS 1.3 reduced this to 1 additional RTT, but the two handshakes remain separate.
3. Head-of-line blocking. HTTP/2 multiplexes many streams over one TCP connection, but TCP delivers bytes in order — so one lost segment stalls every stream, including those whose data arrived intact (7.1).
4. Connection identity tied to the 4-tuple. Change network — Wi-Fi to cellular — and the source address changes, so the connection breaks and must be re-established from scratch.
QUIC's solutions:
Run in userspace over UDP. This is the pivotal decision. It sidesteps both kernel deployment and middlebox ossification: a browser and a server can deploy a new QUIC version by shipping software, with no operating system change and no middlebox cooperation. UDP is chosen not for its semantics — QUIC is fully reliable — but because it is what middleboxes reliably pass (7.2).
Integrate TLS 1.3 into the handshake. Transport and cryptographic setup happen together: 1 RTT for a new connection, 0 RTT for a resumed one, where the client sends application data in its very first packet.
Independent streams with per-stream flow control. Each stream has its own sequence space. A loss in stream 3 blocks only stream 3; streams 1, 2, and 4 continue delivering. This is the elimination of head-of-line blocking, covered in detail in the next topic.
Connection IDs. A connection is identified by an opaque connection ID, not by the 4-tuple. Changing IP address does not break it — this is connection migration, and it is why a QUIC video call survives a Wi-Fi-to-cellular handover that would kill a TCP one.
Two further properties are worth noting.
Almost everything is encrypted, including most of the transport header. Only the minimum needed for routing and decryption is visible. This prevents future ossification by construction: middleboxes cannot inspect what they cannot decrypt, so they cannot come to depend on it. It also complicates traditional network troubleshooting, which is a genuine and acknowledged trade.
Version negotiation is built in from the start, unlike TCP where options must be retrofitted.
The costs are real: higher CPU usage (userspace processing plus encryption of every packet, though kernel offload and hardware acceleration are closing the gap), and less mature tooling than TCP's decades of accumulated diagnostics.
QUIC carries HTTP/3 and is widely deployed — Google, Cloudflare, Meta, and most major CDNs serve it by default.
Examples
Handshake latency, compared
TCP + TLS 1.2 (3 RTT before data):
SYN ------>
<---- SYN-ACK
ACK ------> 1 RTT: transport
ClientHello ------>
<---- ServerHello, Certificate
ClientKeyExchange, Finished ---->
<---- Finished 2 more RTT: crypto
HTTP request ------> data finally moves
TCP + TLS 1.3 (2 RTT):
SYN / SYN-ACK / ACK 1 RTT
ClientHello + key share ---->
<---- ServerHello + Finished 1 RTT
HTTP request ------>
QUIC (1 RTT):
Initial (crypto + transport params) ---->
<---- Initial + Handshake
HTTP request ------> transport AND crypto together
QUIC 0-RTT (resumed connection):
Initial + EARLY DATA (the HTTP request itself) ---->
<---- response
ZERO round trips before application data.
Streams: independent sequence spaces
TCP (one byte stream):
[s1][s2][s3][s1][s2][s3][s1]...
X <- one loss
EVERYTHING after it waits. All three logical streams stall.
QUIC (independent streams):
stream 1: [....][....][....] delivered
stream 2: [....][ X ][....] ONLY stream 2 waits
stream 3: [....][....][....] delivered
Each stream has its own offsets and its own flow control.
Connection migration
TCP: connection = (srcIP, srcPort, dstIP, dstPort)
Wi-Fi -> cellular changes srcIP
-> the 4-tuple changes -> CONNECTION DEAD
-> reconnect, re-handshake, re-authenticate, restart the transfer
QUIC: connection = Connection ID (an opaque value)
Wi-Fi -> cellular changes the srcIP
-> the Connection ID is UNCHANGED
-> the server recognises it and continues
-> the video call does not even blink
def handshake_rtts(protocol, resumed=False):
return {
("TCP+TLS1.2", False): 3, ("TCP+TLS1.2", True): 2,
("TCP+TLS1.3", False): 2, ("TCP+TLS1.3", True): 1,
("QUIC", False): 1, ("QUIC", True): 0,
}[(protocol, resumed)]
print(f"{'protocol':<14}{'new':>8}{'resumed':>10}{'@100ms new':>14}{'@100ms resumed':>17}")
for p in ("TCP+TLS1.2", "TCP+TLS1.3", "QUIC"):
n, r = handshake_rtts(p), handshake_rtts(p, True)
print(f"{p:<14}{n:>8}{r:>10}{n*100:>11} ms{r*100:>14} ms")
# protocol new resumed @100ms new @100ms resumed
# TCP+TLS1.2 3 2 300 ms 200 ms
# TCP+TLS1.3 2 1 200 ms 100 ms
# QUIC 1 0 100 ms 0 ms
# 0-RTT means the request rides in the FIRST packet. Nothing is faster.
class QUICStream:
"""Each stream has its OWN sequence space and flow-control window."""
def __init__(self, stream_id, window=65536):
self.id = stream_id
self.offset = 0
self.window = window
self.received = {}
self.delivered_upto = 0
def receive(self, offset, data_len):
self.received[offset] = data_len
while self.delivered_upto in self.received:
self.delivered_upto += self.received.pop(self.delivered_upto)
blocked = bool(self.received)
return self.delivered_upto, blocked
class QUICConnection:
def __init__(self):
self.streams = {}
def stream(self, sid):
return self.streams.setdefault(sid, QUICStream(sid))
def receive(self, sid, offset, length):
delivered, blocked = self.stream(sid).receive(offset, length)
status = "BLOCKED (gap in THIS stream)" if blocked else "delivered"
return f"stream {sid}: {status}, {delivered} bytes available"
conn = QUICConnection()
print()
# Stream 2 loses a chunk; streams 1 and 3 are unaffected.
for sid, off, ln in [(1, 0, 1000), (2, 0, 1000), (3, 0, 1000),
(1, 1000, 1000), (2, 2000, 1000), (3, 1000, 1000)]:
print(f" {conn.receive(sid, off, ln)}")
# stream 1: delivered, 1000 bytes available
# stream 2: delivered, 1000 bytes available
# stream 3: delivered, 1000 bytes available
# stream 1: delivered, 2000 bytes available
# stream 2: BLOCKED (gap in THIS stream), 1000 bytes available
# stream 3: delivered, 2000 bytes available
# Stream 2's gap stalls stream 2 ALONE. Over TCP, all three would stall.
# Connection migration.
class Connection:
def __init__(self, kind):
self.kind = kind
self.four_tuple = ("10.1.1.5", 51250, "93.184.216.34", 443)
self.connection_id = "a1b2c3d4e5f6"
def network_changes(self, new_src_ip):
old = self.four_tuple
self.four_tuple = (new_src_ip,) + old[1:]
if self.kind == "TCP":
return ("CONNECTION DEAD — the 4-tuple identifies it, and it "
"changed. Reconnect and re-handshake.")
return (f"connection SURVIVES — identified by CID {self.connection_id}, "
f"which did not change. Path validation, then continue.")
print()
for kind in ("TCP", "QUIC"):
c = Connection(kind)
print(f" {kind:<5} Wi-Fi -> cellular: {c.network_changes('172.20.8.41')}")
# TCP Wi-Fi -> cellular: CONNECTION DEAD — the 4-tuple identifies it, and it changed. Reconnect and re-handshake.
# QUIC Wi-Fi -> cellular: connection SURVIVES — identified by CID a1b2c3d4e5f6, which did not change. Path validation, then continue.
# What each protocol exposes to the network.
VISIBILITY = {
"TCP": ["ports", "sequence numbers", "ACK numbers", "flags", "window",
"options (MSS, SACK, timestamps)"],
"QUIC": ["UDP ports", "connection ID (partially)", "a few header bits"],
}
print()
for proto, fields in VISIBILITY.items():
print(f"{proto} exposes to middleboxes: {', '.join(fields)}")
print(" QUIC encrypts nearly everything else — which PREVENTS ossification,")
print(" because a middlebox cannot come to depend on what it cannot read.")
# The deployment argument that motivated the whole design.
def deploy_time(layer):
return {
"kernel TCP": ("5-10 years", "OS updates on billions of devices, "
"plus middlebox tolerance"),
"userspace QUIC": ("weeks", "ship a browser update and a server update"),
}[layer]
print()
for layer in ("kernel TCP", "userspace QUIC"):
t, why = deploy_time(layer)
print(f" deploy a change in {layer:<16}{t:<12}({why})")
# deploy a change in kernel TCP 5-10 years (OS updates on billions of devices, plus middlebox tolerance)
# deploy a change in userspace QUIC weeks (ship a browser update and a server update)
# Is HTTP/3 available? curl reports the negotiated version:
$ curl --http3 -sI https://cloudflare.com | head -1
# HTTP/3 200
# Compare handshake timings directly:
$ curl -sw 'h2 tls:%{time_appconnect}s first_byte:%{time_starttransfer}s\n' \
> -o /dev/null --http2 https://cloudflare.com
# h2 tls:0.298s first_byte:0.401s
$ curl -sw 'h3 tls:%{time_appconnect}s first_byte:%{time_starttransfer}s\n' \
> -o /dev/null --http3 https://cloudflare.com
# h3 tls:0.112s first_byte:0.148s
# ^^^^^ transport and crypto in ONE round trip
# QUIC on the wire is UDP/443, and almost entirely opaque:
$ sudo tcpdump -i eth0 -n 'udp port 443' -c 2 -X | head -6
# IP 10.1.1.5.51250 > 104.16.132.229.443: UDP, length 1350
# 0x0000: c000 0000 0108 a1b2 c3d4 e5f6 ...
# ^^ long header, version 1
# ^^^^^^^^^^^^^^^^^^^ connection ID
# Everything after this is encrypted. tcpdump can tell you very little.
# Advertise HTTP/3 via Alt-Svc — how clients discover it:
$ curl -sI https://example.com | grep -i alt-svc
# alt-svc: h3=":443"; ma=86400
# The client learns "you may use HTTP/3 on UDP 443" and switches next time.
# Server configuration (nginx with HTTP/3):
# server {
# listen 443 ssl; # TCP, for HTTP/1.1 and HTTP/2
# listen 443 quic reuseport; # UDP, for HTTP/3
# add_header Alt-Svc 'h3=":443"; ma=86400';
# ssl_protocols TLSv1.3; # QUIC REQUIRES TLS 1.3
# }
# Decrypt QUIC for troubleshooting — you need the keys:
$ SSLKEYLOGFILE=/tmp/keys.log curl --http3 -s https://cloudflare.com > /dev/null
$ tshark -r quic.pcap -o tls.keylog_file:/tmp/keys.log -Y quic | head -3
# Without the keylog file, a QUIC capture is essentially unreadable.
# This is the deliberate trade: anti-ossification versus observability.
# Watch connection migration survive a network change:
$ sudo tcpdump -i any -n 'udp port 443' -w migrate.pcap &
$ curl --http3 -s https://cloudflare.com/large.bin -o /dev/null &
$ sleep 2 && sudo ip addr add 10.1.1.99/24 dev eth0 && sudo ip route change ...
$ tshark -r migrate.pcap -T fields -e ip.src -e quic.dcid | sort -u | head -4
# 10.1.1.5 a1b2c3d4e5f6
# 10.1.1.99 a1b2c3d4e5f6
# ^^^^^^^^^^^^ SAME connection ID from a DIFFERENT source address.
# A TCP connection would have died here.
# QUIC's CPU cost is real — measure it:
$ perf stat -e cycles curl --http3 -s https://server/1gb.bin -o /dev/null 2>&1 | tail -3
# Userspace processing plus per-packet encryption. UDP GSO/GRO offload and
# kernel-assisted crypto narrow the gap considerably on modern kernels.
$ ethtool -k eth0 | grep -i "udp\|gso"
# tx-udp_tnl-segmentation: on
# tx-udp-segmentation: on <- essential for QUIC throughput
Key takeaways
- QUIC addresses four structural TCP limitations: ossification, handshake latency, head-of-line blocking, and connection identity tied to the 4-tuple.
- Running in userspace over UDP is the pivotal decision — it enables deployment in weeks rather than years, with no kernel or middlebox changes.
- UDP is chosen as a substrate, not for its semantics: QUIC is fully reliable.
- TLS 1.3 is integrated into the transport handshake: 1 RTT for a new connection, 0 RTT on resumption.
- Independent streams with per-stream sequence spaces and flow control mean a loss blocks only its own stream.
- Connection IDs identify a connection instead of the 4-tuple, enabling connection migration across network changes.
- Nearly the entire transport header is encrypted, which prevents future ossification by making the protocol uninspectable — at the cost of harder troubleshooting.
- Costs: higher CPU (narrowing with UDP GSO/GRO and crypto offload) and less mature tooling.
- QUIC carries HTTP/3, discovered via the Alt-Svc header, and requires TLS 1.3.
🧪 Practice
- List QUIC's four design goals and the TCP limitation each addresses.
- Compute the time to first byte on a 80 ms path for TCP+TLS 1.2, TCP+TLS 1.3, QUIC, and QUIC 0-RTT.
- Explain how connection migration works and why TCP cannot do it.
- Interview-style: QUIC encrypts almost its entire transport header, which makes network troubleshooting harder. Why was that considered worth it? (Hint: ask what middleboxes did to TCP's header fields over thirty years, and what makes a protocol impossible to change.)
Head-of-Line Blocking and Its Elimination
Theory
Head-of-line (HOL) blocking occurs when the item at the front of a queue prevents everything behind it from being processed — even when those items are ready. It appears at several layers in networking, and the history of HTTP is largely a sequence of attempts to eliminate it.
HTTP/1.0 had no multiplexing at all: one request per connection. Browsers compensated by opening 6 parallel connections per host, which worked but consumed server resources and made congestion control less effective (each connection probes independently).
HTTP/1.1 pipelining allowed several requests on one connection, but responses had to return in order. A slow first response blocked all the rest — application-layer HOL blocking. It was so problematic that browsers disabled pipelining by default.
HTTP/2 introduced proper multiplexing: many streams over one TCP connection, with frames interleaved. This eliminated application-layer HOL blocking. But TCP delivers bytes strictly in order, so a single lost segment stalls the delivery of every stream, including those whose data already arrived. The blocking moved down a layer — transport-layer HOL blocking — and became less visible but no less real.
The effect is counterintuitive and worth stating: HTTP/2 can be slower than HTTP/1.1 on a lossy network, because HTTP/1.1's six independent connections mean a loss on one affects only that connection, while HTTP/2's single connection means a loss affects everything.
QUIC eliminates transport-layer HOL blocking by giving each stream its own sequence space. The receiver tracks per-stream offsets independently, so a gap in stream 3 prevents delivery of stream 3 only. Streams 1, 2, and 4 continue uninterrupted.
The progression:
| Protocol | App-layer HOL | Transport-layer HOL |
|---|---|---|
| HTTP/1.0 | N/A (one request per connection) | Per connection |
| HTTP/1.1 pipelining | Yes — responses must be in order | Per connection |
| HTTP/1.1 + 6 connections | No | Limited to one connection each |
| HTTP/2 over TCP | No | Yes — one loss stalls all streams |
| HTTP/3 over QUIC | No | No |
Two honest qualifications.
QUIC does not eliminate HOL blocking within a single stream. If stream 3 loses a segment, stream 3's later data still waits — as it must, since that stream is an ordered byte sequence by definition. What QUIC removes is blocking across streams.
HTTP/3's header compression (QPACK) can reintroduce a limited form of blocking, because header table updates create dependencies between streams. QPACK is designed to bound this, and it can be avoided entirely at a small compression cost.
A related effect appears at the link layer: a single FIFO queue causes HOL blocking between flows, which is exactly what FQ-CoDel fixes by giving each flow its own queue (7.5). The pattern — one shared ordered queue creates coupling; per-entity queues remove it — recurs at every layer.
Examples
The same loss, three protocols
Loss of one segment while 3 objects are in flight.
HTTP/1.1 pipelining (one connection, ordered responses):
[ obj1 ][ obj2 ][ obj3 ]
X obj1's response is slow/lost
-> obj2 and obj3 CANNOT be delivered even if complete.
APPLICATION-layer HOL blocking.
HTTP/2 over TCP (multiplexed, one connection):
TCP byte stream: [o1][o2][o3][o1][o2][o3]...
X <- ONE lost segment
-> TCP cannot deliver ANY later bytes, of ANY stream.
o1 and o3's data has ARRIVED and sits in the buffer, undeliverable.
TRANSPORT-layer HOL blocking.
HTTP/3 over QUIC (independent streams):
stream 1: [....][....][....] delivered normally
stream 2: [....][ X ][....] stream 2 waits for ITS retransmission
stream 3: [....][....][....] delivered normally
-> only the affected stream stalls.
Why HTTP/2 can LOSE to HTTP/1.1 on a lossy link
HTTP/1.1 with 6 connections, 1% loss:
a loss affects 1 of 6 connections -> 1/6 of the objects stall
HTTP/2 with 1 connection, 1% loss:
a loss affects THE connection -> ALL objects stall
More loss -> the gap widens. This is a real, measured effect,
and it is precisely why QUIC was built.
The pattern recurs at every layer
one shared ORDERED queue -> coupling between independent things
per-entity queues -> independence restored
link layer: single FIFO -> FQ-CoDel (per-flow queues)
transport layer: TCP stream -> QUIC (per-stream sequence spaces)
application: pipelining -> multiplexing
class Transport:
"""Model HOL blocking for a shared vs per-stream sequence space."""
def __init__(self, per_stream):
self.per_stream = per_stream
self.delivered = {}
self.buffered = {}
self.next_expected = {}
def key(self, stream):
return stream if self.per_stream else "SHARED"
def receive(self, stream, seq, lost=False):
k = self.key(stream)
self.next_expected.setdefault(k, 0)
self.delivered.setdefault(k, [])
self.buffered.setdefault(k, {})
if lost:
return f"stream {stream}: segment {seq} LOST"
if seq == self.next_expected[k]:
self.next_expected[k] += 1
self.delivered[k].append((stream, seq))
while self.next_expected[k] in self.buffered[k]:
s = self.buffered[k].pop(self.next_expected[k])
self.delivered[k].append((s, self.next_expected[k]))
self.next_expected[k] += 1
return f"stream {stream}: segment {seq} DELIVERED"
self.buffered[k][seq] = stream
return (f"stream {stream}: segment {seq} buffered — BLOCKED waiting "
f"for {self.next_expected[k]}")
for per_stream in (False, True):
t = Transport(per_stream)
label = "QUIC (per-stream sequence spaces)" if per_stream else "TCP (one shared byte stream)"
print(f"\n{label}:")
# Interleave three streams; segment 1 of stream 2 is lost.
events = [(1, 0, False), (2, 0, False), (3, 0, False),
(2, 1, True), (1, 1, False), (3, 1, False), (1, 2, False)]
for stream, seq, lost in events:
print(f" {t.receive(stream, seq, lost)}")
total = sum(len(v) for v in t.delivered.values())
print(f" -> {total} segments delivered to the application")
#
# TCP (one shared byte stream):
# stream 1: segment 0 DELIVERED
# stream 2: segment 0 buffered — BLOCKED waiting for 1
# stream 3: segment 0 buffered — BLOCKED waiting for 1
# stream 2: segment 1 LOST
# stream 1: segment 1 DELIVERED
# stream 3: segment 1 buffered — BLOCKED waiting for 2
# stream 1: segment 2 DELIVERED
# -> 3 segments delivered to the application
#
# QUIC (per-stream sequence spaces):
# stream 1: segment 0 DELIVERED
# stream 2: segment 0 DELIVERED
# stream 3: segment 0 DELIVERED
# stream 2: segment 1 LOST
# stream 1: segment 1 DELIVERED
# stream 3: segment 1 DELIVERED
# stream 1: segment 2 DELIVERED
# -> 6 segments delivered to the application
#
# Same arrivals, same single loss. TCP delivered 3 segments; QUIC
# delivered 6. The other 3 had ARRIVED and were held hostage.
# The quantitative effect of loss on a multiplexed connection.
def stalled_fraction(protocol, connections, loss_rate, objects=6):
if protocol == "HTTP/1.1":
# A loss stalls only the objects on that one connection.
return loss_rate * (objects / connections) / objects
if protocol == "HTTP/2":
# A loss stalls EVERYTHING on the single connection.
return loss_rate
return loss_rate / objects # HTTP/3: only the affected stream
print()
print(f"{'loss':>7}{'HTTP/1.1 (6 conn)':>20}{'HTTP/2 (1 conn)':>18}{'HTTP/3 (QUIC)':>16}")
for loss in (0.001, 0.01, 0.05):
h1 = stalled_fraction("HTTP/1.1", 6, loss)
h2 = stalled_fraction("HTTP/2", 1, loss)
h3 = stalled_fraction("HTTP/3", 1, loss)
print(f"{loss:>6.1%}{h1:>19.2%}{h2:>17.1%}{h3:>15.2%}")
# loss HTTP/1.1 (6 conn) HTTP/2 (1 conn) HTTP/3 (QUIC)
# 0.1% 0.02% 0.1% 0.02%
# 1.0% 0.17% 1.0% 0.17%
# 5.0% 0.83% 5.0% 0.83%
# HTTP/2 concentrates ALL the damage of a loss onto every object.
# HTTP/3 gets HTTP/2's multiplexing with HTTP/1.1's loss isolation.
# What QUIC does NOT fix.
def quic_hol(scope):
return {
"across streams": (False, "ELIMINATED — each stream has its own offsets"),
"within one stream": (True, "REMAINS — a stream is an ordered byte "
"sequence by definition"),
"QPACK header table": (True, "possible, but bounded by design; "
"avoidable at a compression cost"),
}[scope]
print()
for scope in ("across streams", "within one stream", "QPACK header table"):
blocks, note = quic_hol(scope)
print(f" {scope:<22}{'blocks' if blocks else 'no block':<10}{note}")
# across streams no block ELIMINATED — each stream has its own offsets
# within one stream blocks REMAINS — a stream is an ordered byte sequence by definition
# QPACK header table blocks possible, but bounded by design; avoidable at a compression cost
# The recurring pattern across layers.
LAYERS = {
"link (single FIFO)": ("bulk traffic delays VoIP", "FQ-CoDel: per-flow queues"),
"transport (TCP)": ("one loss stalls all streams", "QUIC: per-stream offsets"),
"application (HTTP/1.1 pipelining)": ("a slow response blocks the rest",
"HTTP/2: multiplexed frames"),
}
print()
for layer, (problem, fix) in LAYERS.items():
print(f" {layer:<38}{problem:<32}-> {fix}")
# Demonstrate the effect: HTTP/2 vs HTTP/3 under loss.
$ sudo tc qdisc add dev eth0 root netem loss 2% delay 50ms
$ for v in --http2 --http3; do
> echo -n "$v: "
> curl -sw '%{time_total}s\n' -o /dev/null $v https://cloudflare.com/
> done
# --http2: 2.418s
# --http3: 0.914s <- loss isolation between streams
$ sudo tc qdisc del dev eth0 root
# Load a page with many objects and compare — the gap grows with object count:
$ for v in 2 3; do
> echo -n "HTTP/$v: "
> curl -sw '%{time_total}\n' -o /dev/null --http$v https://cloudflare.com/
> done
# Confirm HTTP/2 uses ONE connection for many streams:
$ ss -tan state established '( dport = :443 )' | wc -l
# 1
$ nghttp -nv https://cloudflare.com 2>&1 | grep -c "recv (stream_id="
# 14 # 14 streams, 1 TCP connection
# Compare with HTTP/1.1's parallel connections:
$ curl --http1.1 -s https://example.com -o /dev/null &
$ ss -tan state established '( dport = :443 )' | wc -l
# 6 # the browser-style 6-connection workaround
# Watch QUIC stream frames — each carries its OWN offset:
$ SSLKEYLOGFILE=/tmp/keys.log curl --http3 -s https://cloudflare.com -o /dev/null
$ tshark -r quic.pcap -o tls.keylog_file:/tmp/keys.log \
> -T fields -e quic.stream.stream_id -e quic.stream.offset | head -6
# 0 0
# 4 0
# 8 0
# 0 1350
# 4 1350
# ^^^^ per-stream offsets. Stream 4's gap does not affect stream 0.
# TCP's single ordered stream, by contrast:
$ ss -tin state established | grep -oE "unacked:[0-9]+|lost:[0-9]+"
# unacked:12
# lost:1
# ONE lost segment. Every byte behind it — of every HTTP/2 stream —
# is stuck in the receive buffer.
# Out-of-order queue depth shows data arrived but could not be delivered:
$ nstat -az | grep -i ofo
# TcpExtTCPOFOQueue 184221 <- segments buffered out of order
# TcpExtTCPOFOMerge 4127
# Every one of these is data that ARRIVED and is waiting on an earlier gap.
# Link-layer HOL blocking, and the same fix one layer down:
$ tc qdisc show dev eth0
# qdisc fq_codel 0: root ... flows 1024
# ^^^^^^^^^^ 1024 independent queues: a bulk flow
# cannot head-of-line block an interactive one
Key takeaways
- Head-of-line blocking is when the front item of a queue stalls everything behind it, even when those items are ready.
- HTTP/1.1 pipelining caused application-layer HOL blocking (responses had to be in order) and was disabled by browsers.
- Browsers worked around it with 6 parallel connections, which isolates loss but wastes resources and weakens congestion control.
- HTTP/2 fixed application-layer HOL blocking but inherited transport-layer HOL blocking, because TCP delivers bytes strictly in order.
- Consequently HTTP/2 can be slower than HTTP/1.1 on a lossy link — one connection concentrates all the damage of a loss.
- QUIC gives each stream its own sequence space, so a loss blocks only the affected stream.
- QUIC does not eliminate HOL blocking within a stream — an ordered byte sequence must stay ordered.
- QPACK can reintroduce a bounded form of blocking through header-table dependencies; it is avoidable at a small compression cost.
- The pattern recurs at every layer: a shared ordered queue couples independent things, and per-entity queues decouple them — the same insight behind FQ-CoDel.
🧪 Practice
- Distinguish application-layer from transport-layer HOL blocking and name a protocol exhibiting each.
- Explain why HTTP/2 can underperform HTTP/1.1 at 3% loss despite being the newer protocol.
- State what QUIC eliminates and what it does not, with respect to HOL blocking.
- Interview-style: FQ-CoDel and QUIC solve problems at different layers using the same idea. What is it? (Hint: both replace one shared ordered structure with many independent ones — ask what that structure was in each case.)
SCTP Multi-Streaming and Multihoming
Theory
SCTP (Stream Control Transmission Protocol, RFC 9260) was standardised in 2000, originally to carry telephony signalling (SS7) over IP. It solved several of TCP's problems sixteen years before QUIC — and it is barely deployed on the public internet. That gap between technical merit and adoption is the most instructive thing about it.
SCTP's features:
Multi-streaming. One association (SCTP's word for a connection) carries multiple independent streams, each with its own sequence numbering. A loss in one stream does not block the others — exactly QUIC's solution to head-of-line blocking, specified in 2000.
Multihoming. An association can bind multiple IP addresses per endpoint. If the primary path fails, SCTP fails over to an alternate without breaking the association. TCP requires a separate mechanism (or a new connection) to do this at all.
Message-oriented delivery. SCTP preserves message boundaries like UDP, while providing reliability like TCP. Applications do not need to implement framing (7.1) — a genuine convenience.
Four-way handshake with cookies. SCTP's handshake uses INIT, INIT-ACK (carrying a state cookie), COOKIE-ECHO, and COOKIE-ACK. The server allocates no state until the cookie returns, making SCTP immune to SYN-flood-style attacks by design rather than by the bolt-on that SYN cookies represent for TCP.
Partial reliability (PR-SCTP, RFC 3758). An extension allowing per-message reliability policies — retransmit this message, abandon that one after a deadline. This is the selective reliability that UDP applications build by hand (7.2), available as a protocol feature.
So why is it not used? Three reasons, and the first dominates:
Middleboxes. SCTP is IP protocol 132, not TCP or UDP. NAT devices and firewalls do not recognise it and drop it. An application cannot rely on SCTP reaching an arbitrary internet destination. This is the ossification problem (7.6) in its purest form: a technically superior protocol that is undeployable because the network will not pass it.
No NAT traversal. Most NAT implementations have no SCTP support at all.
Limited API and library support compared with TCP's ubiquity.
SCTP therefore thrives only in controlled networks where the operator controls every middlebox: telecom signalling (SIGTRAN, Diameter), 4G/5G core interfaces, and — notably — WebRTC data channels, where SCTP runs encapsulated inside DTLS over UDP precisely to get past the middleboxes that would otherwise drop it.
That last case is the lesson made explicit. SCTP's features were right; its packaging was wrong. QUIC took the same ideas and wrapped them in UDP from the start, which is the only reason it deployed.
Examples
SCTP had QUIC's key ideas sixteen years earlier
SCTP (2000) QUIC (2016)
---------------------------- ----------------------------
multi-streaming independent streams
multihoming connection migration
message boundaries preserved stream + datagram frames
cookie handshake (no state) address validation, 3x limit
partial reliability (PR-SCTP) unreliable datagram extension
SAME IDEAS. Different packaging. Only one of them deployed.
SCTP: IP protocol 132 -> middleboxes DROP it
QUIC: UDP port 443 -> middleboxes PASS it
Multihoming: what TCP cannot do
SCTP association binds MULTIPLE addresses per endpoint:
host A: 10.1.1.5, 172.20.8.41 host B: 93.184.216.34, 93.184.217.9
\_____ primary path _____/
\____ alternate path ___/
primary path fails -> HEARTBEAT detects it -> switch to the alternate
-> the association CONTINUES. No reconnect. No application involvement.
TCP: the 4-tuple IS the connection. One address change -> dead.
The four-way cookie handshake: SYN-flood immunity by design
client server
|--- INIT ------------------------>|
| | compute a STATE COOKIE
| | ALLOCATE NOTHING
|<-- INIT-ACK (with cookie) --------|
|--- COOKIE-ECHO (returns it) ---->|
| | validate -> NOW allocate state
|<-- COOKIE-ACK -------------------|
|========= ESTABLISHED ============|
A flood of INITs costs the server nothing. TCP needed SYN cookies
retrofitted to achieve the same property.
class SCTPAssociation:
"""Multi-streaming plus multihoming in one association."""
def __init__(self, local_addrs, remote_addrs, num_streams=4):
self.local = list(local_addrs)
self.remote = list(remote_addrs)
self.primary = (self.local[0], self.remote[0])
self.streams = {i: {"next_seq": 0, "buffered": {}} for i in range(num_streams)}
self.state = "ESTABLISHED"
def send(self, stream_id, message):
s = self.streams[stream_id]
seq = s["next_seq"]
s["next_seq"] += 1
# MESSAGE-oriented: boundaries are preserved, unlike TCP.
return {"stream": stream_id, "seq": seq, "message": message,
"path": self.primary}
def receive(self, stream_id, seq, message, lost=False):
s = self.streams[stream_id]
if lost:
return f"stream {stream_id}: message {seq} LOST — only THIS stream waits"
return f"stream {stream_id}: message {seq} delivered ({message!r})"
def path_failure(self, failed_local):
alternates = [a for a in self.local if a != failed_local]
if not alternates:
self.state = "FAILED"
return "no alternate path — association lost"
self.primary = (alternates[0], self.primary[1])
return (f"HEARTBEAT detected failure on {failed_local}; "
f"switched to {alternates[0]}. Association CONTINUES.")
assoc = SCTPAssociation(["10.1.1.5", "172.20.8.41"],
["93.184.216.34", "93.184.217.9"])
print(f"primary path: {assoc.primary}")
print(f"streams: {len(assoc.streams)}\n")
# Multi-streaming: a loss in stream 1 does not block streams 0 and 2.
for sid, seq, msg, lost in [(0, 0, "config", False), (1, 0, "video", True),
(2, 0, "audio", False), (0, 1, "config2", False)]:
print(f" {assoc.receive(sid, seq, msg, lost)}")
# stream 0: message 0 delivered ('config')
# stream 1: message 0 LOST — only THIS stream waits
# stream 2: message 0 delivered ('audio')
# stream 0: message 1 delivered ('config2')
print(f"\n {assoc.path_failure('10.1.1.5')}")
print(f" new primary: {assoc.primary}, state: {assoc.state}")
# HEARTBEAT detected failure on 10.1.1.5; switched to 172.20.8.41. Association CONTINUES.
# new primary: ('172.20.8.41', '93.184.216.34'), state: ESTABLISHED
# Handshake state cost: SCTP vs TCP.
def handshake_state(protocol, floods_per_sec, state_bytes=304):
if protocol == "SCTP":
return 0, "cookie handshake: NO state until COOKIE-ECHO returns"
if protocol == "TCP + syncookies":
return 0, "SYN cookies: state encoded in the ISN (retrofitted)"
return floods_per_sec * state_bytes, "state allocated on every SYN — floodable"
print()
for p in ("TCP", "TCP + syncookies", "SCTP"):
b, note = handshake_state(p, 100_000)
print(f"{p:<20}{b:>12,} bytes under a 100k/s flood {note}")
# TCP 30,400,000 bytes under a 100k/s flood state allocated on every SYN — floodable
# TCP + syncookies 0 bytes under a 100k/s flood SYN cookies: state encoded in the ISN (retrofitted)
# SCTP 0 bytes under a 100k/s flood cookie handshake: NO state until COOKIE-ECHO returns
# Why it did not deploy: the middlebox problem.
def traverses(protocol, middlebox):
ip_proto = {"TCP": 6, "UDP": 17, "SCTP": 132, "QUIC": 17}[protocol]
if ip_proto in (6, 17):
return True, f"IP protocol {ip_proto}: recognised and passed"
return False, f"IP protocol {ip_proto}: UNKNOWN to {middlebox} -> DROPPED"
print()
for p in ("TCP", "UDP", "QUIC", "SCTP"):
ok, note = traverses(p, "typical NAT/firewall")
print(f" {p:<6}{'passes' if ok else 'BLOCKED':<9}{note}")
# TCP passes IP protocol 6: recognised and passed
# UDP passes IP protocol 17: recognised and passed
# QUIC passes IP protocol 17: recognised and passed
# SCTP BLOCKED IP protocol 132: UNKNOWN to typical NAT/firewall -> DROPPED
# The workaround that made SCTP usable in WebRTC.
def sctp_deployment(encapsulation):
if encapsulation is None:
return "native SCTP (proto 132)", "blocked on the public internet"
return (f"SCTP over {encapsulation}",
"works everywhere — the middleboxes see only UDP")
print()
for enc in (None, "DTLS over UDP"):
what, outcome = sctp_deployment(enc)
print(f" {what:<28}{outcome}")
# native SCTP (proto 132) blocked on the public internet
# SCTP over DTLS over UDP works everywhere — the middleboxes see only UDP
# WebRTC data channels use exactly this. The features were never the problem.
# Feature comparison.
FEATURES = {
"multi-streaming": {"TCP": False, "SCTP": True, "QUIC": True},
"multihoming/migration": {"TCP": False, "SCTP": True, "QUIC": True},
"message boundaries": {"TCP": False, "SCTP": True, "QUIC": True},
"handshake DoS immunity": {"TCP": False, "SCTP": True, "QUIC": True},
"partial reliability": {"TCP": False, "SCTP": True, "QUIC": True},
"built-in encryption": {"TCP": False, "SCTP": False, "QUIC": True},
"traverses middleboxes": {"TCP": True, "SCTP": False, "QUIC": True},
}
print()
print(f"{'feature':<26}{'TCP':>6}{'SCTP':>7}{'QUIC':>7}")
for feat, support in FEATURES.items():
row = "".join(f"{'yes' if support[p] else 'no':>7}" for p in ("TCP", "SCTP", "QUIC"))
print(f"{feat:<26}{row}")
print(" SCTP beats TCP on six of seven rows — and loses on the ONE that decides")
print(" deployment.")
# Is SCTP even available on this host?
$ lsmod | grep sctp
$ sudo modprobe sctp && echo "SCTP module loaded"
$ cat /proc/net/protocols | grep -i sctp
# SCTPv6 1 9 -1 NI 0 yes kernel
# SCTP 1 9 -1 NI 0 yes kernel
# Create an SCTP socket — note the protocol constant:
$ python3 -c "
import socket
IPPROTO_SCTP = 132
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, IPPROTO_SCTP)
print('SCTP socket created; IP protocol number', IPPROTO_SCTP)"
# Test whether it survives the path to the internet — usually it does not:
$ sudo sctp_test -H 0.0.0.0 -P 2000 -l & # local server
$ sctp_test -H 0.0.0.0 -P 3000 -h 127.0.0.1 -p 2000 -s # loopback: works
$ sctp_test -H 0.0.0.0 -P 3000 -h public-server -p 2000 -s
# (times out — a NAT or firewall dropped IP protocol 132)
# Confirm the cause in a capture:
$ sudo tcpdump -i eth0 'ip proto 132' -c 5
# (silence outbound past the NAT — the packets never come back)
# SCTP associations, when they do exist:
$ ss --sctp
# State Recv-Q Send-Q Local Address:Port Peer Address:Port
# ESTAB 0 0 10.1.1.5:2000 10.2.2.9:3000
$ cat /proc/net/sctp/assocs | head -2
$ cat /proc/net/sctp/eps
# Where SCTP genuinely runs — telecom signalling:
$ ss --sctp | grep -E ":3868|:2905" # Diameter, M3UA
# 4G/5G core interfaces and SIGTRAN use SCTP natively, inside networks
# where the operator controls every middlebox.
# WebRTC data channels: SCTP wrapped in DTLS over UDP:
$ sudo tcpdump -i eth0 -n 'udp' -c 2 | head -2
# IP 10.1.1.5.51250 > 10.2.2.9.51251: UDP, length 1200
# The middlebox sees ordinary UDP. Inside: DTLS, and inside that, SCTP.
# chrome://webrtc-internals shows the data channel's SCTP statistics.
# Multihoming, configured:
$ python3 - <<'EOF'
import socket
IPPROTO_SCTP = 132
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, IPPROTO_SCTP)
s.bind(("10.1.1.5", 2000))
# sctp_bindx() adds further addresses to the SAME association —
# the capability TCP simply does not have.
print("bound; additional addresses would be added via sctp_bindx()")
EOF
# Heartbeat parameters control failover timing:
$ sysctl net.sctp.hb_interval net.sctp.path_max_retrans
# net.sctp.hb_interval = 30000 # 30 s heartbeats
# net.sctp.path_max_retrans = 5 # retries before declaring a path dead
$ sudo sysctl -w net.sctp.hb_interval=5000 # faster failover detection
Key takeaways
- SCTP (2000) solved multi-streaming, multihoming, message boundaries, and handshake DoS immunity sixteen years before QUIC.
- Multi-streaming gives each stream its own sequence numbering — QUIC's HOL-blocking solution, specified in 2000.
- Multihoming binds several addresses per endpoint with heartbeat-driven failover that keeps the association alive; TCP cannot do this.
- Message-oriented delivery preserves boundaries, so applications need no framing.
- The four-way cookie handshake allocates no server state until COOKIE-ECHO returns, giving SYN-flood immunity by design rather than as a retrofit.
- PR-SCTP provides per-message partial reliability as a protocol feature.
- It failed to deploy because it is IP protocol 132 — NATs and firewalls drop it, the purest example of ossification.
- It thrives only in controlled networks: SIGTRAN, Diameter, 4G/5G core.
- WebRTC data channels run SCTP inside DTLS over UDP — the features were right, the packaging was wrong, and QUIC's lesson was to wrap them in UDP from the start.
🧪 Practice
- Name four SCTP features that TCP lacks, and the QUIC equivalent of each.
- Explain how SCTP's cookie handshake gives SYN-flood immunity without a retrofit.
- State the specific reason SCTP failed to deploy on the public internet and how WebRTC works around it.
- Interview-style: SCTP is technically superior to TCP on almost every axis yet is essentially unused on the internet, while QUIC succeeded with similar features. What does the difference teach about protocol design? (Hint: compare the IP protocol numbers each uses, and ask what a NAT does with one it has never seen.)
Multipath TCP
Theory
A modern device usually has several network paths available at once — a phone has Wi-Fi and cellular, a server has multiple NICs, a data centre has many equal-cost routes. Standard TCP uses exactly one, because a connection is defined by its 4-tuple and a 4-tuple names one path.
Multipath TCP (MPTCP, RFC 8684) lets a single logical connection use several paths simultaneously, while presenting an ordinary TCP socket to the application. That last constraint drives the entire design: MPTCP must work with unmodified applications and must survive middleboxes that expect to see normal TCP.
The architecture has two levels:
The MPTCP connection is what the application sees — one socket, one byte stream, with a connection-level sequence number (DSN, Data Sequence Number). Subflows are ordinary TCP connections, each with its own 4-tuple, its own sequence numbers, and its own congestion control state.
Data is split across subflows and reassembled using the DSN at the receiver.
Establishment works by extending TCP options:
- The initial SYN carries MP_CAPABLE, negotiating MPTCP and exchanging keys.
- Additional subflows are added with MP_JOIN on a new SYN, authenticated with an HMAC derived from those keys.
- ADD_ADDR advertises further addresses the peer may connect to.
Every subflow looks like an ordinary TCP connection to any middlebox — which is precisely why MPTCP deploys where SCTP did not. If a middlebox strips the MPTCP options, the connection falls back to plain TCP and still works.
Congestion control must be coupled, and this is the subtlest part. If each subflow ran independent congestion control, a two-subflow MPTCP connection sharing one bottleneck with a TCP flow would take two-thirds of the capacity rather than half. LIA (Linked Increases Algorithm) and its successors OLIA and BALIA couple the subflows' windows so the connection takes no more than a single TCP flow would at any shared bottleneck, while still shifting traffic toward less congested paths.
Three scheduling policies serve different goals:
| Mode | Behaviour | Use case |
|---|---|---|
| Backup / handover | Use one path; switch on failure | Phone: Wi-Fi preferred, cellular standby |
| Full mesh / aggregation | Use all paths together | Maximise throughput |
| Lowest-RTT first | Send on the fastest available subflow | Minimise latency |
The deployment reality is interesting. MPTCP's best-known success is Apple's Siri, which has used it since 2013 for seamless Wi-Fi/cellular handover — a user walking out of Wi-Fi range experiences no interruption. Korea Telecom deployed it for bonded Wi-Fi + LTE consumer service. Linux has supported upstream MPTCP since kernel 5.6.
Compared with QUIC's connection migration, the two solve overlapping problems differently: MPTCP uses several paths at once, while QUIC migrates between them one at a time (QUIC multipath extensions are in progress). MPTCP's advantage is that it works with unmodified applications and existing TCP infrastructure; QUIC's is that it controls its own destiny in userspace.
Examples
One connection, several paths
APPLICATION
| ordinary TCP socket API — completely unmodified
+----v----------------------------------------------+
| MPTCP connection (Data Sequence Numbers) |
+----+------------------+------------------+---------+
| | |
subflow 1 subflow 2 subflow 3
(Wi-Fi) (cellular) (ethernet)
own 4-tuple own 4-tuple own 4-tuple
own seq numbers own seq numbers own seq numbers
own cwnd own cwnd own cwnd
| | |
each looks like an ORDINARY TCP CONNECTION to every middlebox
Two levels of sequence numbering
application writes bytes 1..3000
DSN (connection level): 1 .. 3000 <- reassembly order
|
+-- subflow 1 carries DSN 1..1000 as its OWN seq 1..1000
+-- subflow 2 carries DSN 1001..2000 as its OWN seq 1..1000
+-- subflow 1 carries DSN 2001..3000 as its OWN seq 1001..2000
Each subflow is a valid TCP connection in its own right.
The DSN is what lets the receiver put the pieces back in order.
Why congestion control must be COUPLED
bottleneck shared with one ordinary TCP flow:
UNCOUPLED: MPTCP subflow A: 1 share
MPTCP subflow B: 1 share
TCP flow: 1 share
-> MPTCP takes 2/3. UNFAIR.
COUPLED (LIA/OLIA/BALIA):
MPTCP total: 1 share (split across A and B)
TCP flow: 1 share
-> MPTCP takes 1/2, exactly like a single TCP flow.
It may still SHIFT its share toward the less congested subflow.
class MPTCPConnection:
"""One connection, many subflows, one Data Sequence Number space."""
def __init__(self):
self.subflows = {}
self.dsn = 1
self.received = {}
self.delivered_upto = 1
def add_subflow(self, name, local, remote, rtt_ms, cwnd=10, backup=False):
self.subflows[name] = {"local": local, "remote": remote, "rtt": rtt_ms,
"cwnd": cwnd, "backup": backup, "seq": 1,
"alive": True}
return f"subflow '{name}' added ({local} -> {remote}, RTT {rtt_ms} ms)"
def schedule(self, mode="lowest-rtt"):
live = {k: v for k, v in self.subflows.items() if v["alive"]}
if not live:
return None
primary = {k: v for k, v in live.items() if not v["backup"]} or live
if mode == "lowest-rtt":
return min(primary, key=lambda k: primary[k]["rtt"])
if mode == "backup":
return next(iter(primary))
return max(primary, key=lambda k: primary[k]["cwnd"])
def send(self, nbytes, mode="lowest-rtt"):
sf_name = self.schedule(mode)
if sf_name is None:
return "no live subflow — connection stalled"
sf = self.subflows[sf_name]
dsn_start, subflow_seq = self.dsn, sf["seq"]
self.dsn += nbytes
sf["seq"] += nbytes
return (f"sent DSN {dsn_start}..{dsn_start+nbytes-1} on '{sf_name}' "
f"(its own seq {subflow_seq}..{subflow_seq+nbytes-1})")
def subflow_fails(self, name):
self.subflows[name]["alive"] = False
remaining = [k for k, v in self.subflows.items() if v["alive"]]
if remaining:
return (f"subflow '{name}' FAILED — connection continues on "
f"{remaining}. The application notices nothing.")
return f"subflow '{name}' failed — no paths left, connection lost"
m = MPTCPConnection()
print(m.add_subflow("wifi", "10.1.1.5", "93.184.216.34", rtt_ms=15))
print(m.add_subflow("cellular", "172.20.8.41", "93.184.216.34", rtt_ms=45, backup=True))
print()
for _ in range(3):
print(f" {m.send(1000)}")
print(f"\n {m.subflow_fails('wifi')}")
print(f" {m.send(1000)}")
# subflow 'wifi' added (10.1.1.5 -> 93.184.216.34, RTT 15 ms)
# subflow 'cellular' added (172.20.8.41 -> 93.184.216.34, RTT 45 ms)
#
# sent DSN 1..1000 on 'wifi' (its own seq 1..1000)
# sent DSN 1001..2000 on 'wifi' (its own seq 1001..2000)
# sent DSN 2001..3000 on 'wifi' (its own seq 2001..3000)
#
# subflow 'wifi' FAILED — connection continues on ['cellular']. The application notices nothing.
# sent DSN 3001..4000 on 'cellular' (its own seq 1..1000)
# Note the DSN continues seamlessly while the subflow sequence restarts.
# Why coupling matters for fairness.
def bottleneck_shares(mptcp_subflows, tcp_flows, coupled):
if coupled:
total_shares = 1 + tcp_flows # MPTCP counts as ONE flow
mptcp = 1 / total_shares
else:
total_shares = mptcp_subflows + tcp_flows
mptcp = mptcp_subflows / total_shares
return mptcp, 1 - mptcp
print()
print(f"{'subflows':>9}{'coupled':>9}{'MPTCP share':>14}{'TCP share':>12}")
for n in (2, 4):
for coupled in (False, True):
mp, tcp = bottleneck_shares(n, 1, coupled)
print(f"{n:>9}{str(coupled):>9}{mp:>13.0%}{tcp:>11.0%}")
# subflows coupled MPTCP share TCP share
# 2 False 67% 33%
# 2 True 50% 50%
# 4 False 80% 20%
# 4 True 50% 50%
# Uncoupled, MPTCP grabs more the more subflows it opens. Coupling
# (LIA/OLIA/BALIA) keeps it equivalent to a single TCP flow.
# Scheduling modes.
def mode_behaviour(mode):
return {
"backup": ("one path at a time", "seamless handover — Apple Siri"),
"fullmesh": ("all paths at once", "throughput aggregation — Wi-Fi + LTE"),
"lowest-rtt": ("fastest subflow", "minimise latency"),
}[mode]
print()
for mode in ("backup", "fullmesh", "lowest-rtt"):
what, use = mode_behaviour(mode)
print(f" {mode:<12}{what:<22}{use}")
# Throughput: aggregation versus handover.
def throughput(paths_mbps, mode):
if mode == "fullmesh":
return sum(paths_mbps), "aggregated across all paths"
return max(paths_mbps), "single best path; others are standby"
print()
for mode in ("backup", "fullmesh"):
t, note = throughput([50, 20], mode)
print(f" Wi-Fi 50 + LTE 20 Mb/s, {mode:<10} -> {t:>3} Mb/s ({note})")
# Wi-Fi 50 + LTE 20 Mb/s, backup -> 50 Mb/s (single best path; others are standby)
# Wi-Fi 50 + LTE 20 Mb/s, fullmesh -> 70 Mb/s (aggregated across all paths)
# Why MPTCP deploys where SCTP did not.
def middlebox_view(protocol):
return {
"SCTP": ("IP protocol 132", "unknown -> DROPPED", False),
"MPTCP": ("ordinary TCP with unknown options",
"options stripped -> FALLS BACK to plain TCP", True),
"QUIC": ("ordinary UDP", "passed", True),
}[protocol]
print()
for p in ("SCTP", "MPTCP", "QUIC"):
looks, outcome, works = middlebox_view(p)
print(f" {p:<6}looks like {looks:<38}{outcome}")
# SCTP looks like IP protocol 132 unknown -> DROPPED
# MPTCP looks like ordinary TCP with unknown options options stripped -> FALLS BACK to plain TCP
# QUIC looks like ordinary UDP passed
# MPTCP's graceful fallback is why it survives hostile paths.
# Is MPTCP available? (Upstream since Linux 5.6.)
$ sysctl net.mptcp.enabled
# net.mptcp.enabled = 1
$ sudo sysctl -w net.mptcp.enabled=1
# Create an MPTCP socket — it is a new protocol constant on AF_INET:
$ python3 -c "
import socket
IPPROTO_MPTCP = 262
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, IPPROTO_MPTCP)
print('MPTCP socket created — the application API is unchanged')"
# Or force existing applications to use it, with no code change:
$ mptcpize run curl -s https://example.com -o /dev/null
$ mptcpize run nginx
# Configure which addresses may be used as subflows:
$ sudo ip mptcp endpoint add 172.20.8.41 dev wwan0 subflow backup
# ^^^^^^^ ^^^^^^
# create subflows only on failure
$ sudo ip mptcp endpoint add 10.1.1.5 dev wlan0 subflow signal
$ ip mptcp endpoint show
# 10.1.1.5 id 1 subflow signal dev wlan0
# 172.20.8.41 id 2 subflow backup dev wwan0
$ sudo ip mptcp limits set subflow 4 add_addr_accepted 4
$ ip mptcp limits show
# add_addr_accepted 4 subflows 4
# Watch subflows on a live MPTCP connection:
$ ss -tin --mptcp
# ESTAB 0 0 10.1.1.5:51250 93.184.216.34:443
# subflows:2 add_addr_signal:1
# ^^^^^^^^^^ two paths carrying one logical connection
# The MPTCP options on the wire:
$ sudo tcpdump -i any -n 'tcp[tcpflags] & tcp-syn != 0' -v -c 2 | grep -i mptcp
# Flags [S], options [mss 1460,sackOK,TS,nop,wscale 7,mptcp capable v1]
# ^^^^^^^^^^^^^^ MP_CAPABLE
# Flags [S], options [... mptcp join backup id 2 token 0x1a2b3c4d ...]
# ^^^^^^^^^^^ MP_JOIN: adding a second subflow
# Coupled congestion control — check which algorithm is in use:
$ sysctl net.ipv4.tcp_congestion_control
$ ls /lib/modules/$(uname -r)/kernel/net/mptcp/ 2>/dev/null
# LIA / OLIA / BALIA couple the subflow windows so the connection is no
# more aggressive at a shared bottleneck than a single TCP flow.
# Demonstrate handover: start a transfer, then drop the primary interface:
$ mptcpize run curl -s https://server/1gb.bin -o /dev/null &
$ sleep 3
$ sudo ip link set wlan0 down # kill Wi-Fi mid-transfer
$ ss -tin --mptcp | grep subflows
# subflows:1 <- continued on cellular; curl never noticed
$ sudo ip link set wlan0 up
# MPTCP counters:
$ nstat -az | grep -i mptcp
# MPTcpExtMPCapableSYNTX 412 <- MP_CAPABLE offered
# MPTcpExtMPCapableSYNACKRX 389 <- peer supports it
# MPTcpExtMPJoinSynRx 1204 <- additional subflows joined
# MPTcpExtMPFallbackTokenInit 23 <- fell back to plain TCP
# (a middlebox stripped the options)
Key takeaways
- MPTCP (RFC 8684) lets one logical connection use several paths at once while presenting an ordinary TCP socket to the application.
- Two levels: the MPTCP connection with Data Sequence Numbers, and subflows that are each a complete TCP connection with their own 4-tuple, sequence numbers, and congestion state.
- Establishment uses TCP options: MP_CAPABLE on the first SYN, MP_JOIN for further subflows, ADD_ADDR to advertise addresses.
- Every subflow looks like ordinary TCP to middleboxes, and stripped options cause a graceful fallback to plain TCP — the reason MPTCP deploys where SCTP did not.
- Congestion control must be coupled (LIA, OLIA, BALIA) so the connection takes no more at a shared bottleneck than a single TCP flow would.
- Uncoupled, a 2-subflow connection would take two-thirds of a bottleneck shared with one TCP flow instead of half.
- Modes: backup/handover (seamless failover), full mesh (throughput aggregation), lowest-RTT first (latency).
- Deployed by Apple's Siri since 2013 for Wi-Fi/cellular handover, and by Korea Telecom for bonded consumer service; upstream in Linux since 5.6.
- Versus QUIC: MPTCP uses several paths simultaneously and works with unmodified applications; QUIC migrates between paths and controls its own evolution in userspace.
🧪 Practice
- Distinguish the MPTCP connection level from the subflow level, naming the sequence number used at each.
- Compute the bottleneck share a 3-subflow MPTCP connection would take against one TCP flow with and without coupled congestion control.
- Explain why MPTCP survives middleboxes that would drop SCTP.
- Interview-style: A phone moves out of Wi-Fi range mid-download and the transfer continues uninterrupted. Name two different protocols that could achieve this and explain how each does it. (Hint: one keeps multiple paths open simultaneously; the other keeps one path but decouples identity from the address.)
Transport Protocol Ossification and Middleboxes
Theory
Ossification is the loss of a protocol's ability to evolve. It happens when devices along a path come to depend on details of a protocol they were never meant to interpret, so that any change breaks connectivity somewhere — and a change that breaks connectivity cannot be deployed.
The end-to-end principle (Chapter 5) said the network should move packets and leave everything else to the endpoints. Under that model, upgrading a transport means upgrading two hosts. Middleboxes broke that model. NATs, firewalls, load balancers, traffic shapers, intrusion detection systems, and transparent proxies all parse transport headers, and once deployed they become part of the contract.
The mechanism is worth stating precisely, because it explains every deployment failure in this chapter:
- A middlebox is written to handle the protocol as it exists today.
- Anything unfamiliar is treated as suspicious and dropped or rewritten.
- A new feature therefore fails on some paths, unpredictably.
- Endpoints cannot risk breaking connectivity, so they do not deploy it.
- The protocol is frozen — not by specification, but by deployed reality.
The casualties are concrete, and each appeared earlier in this chapter:
| Attempt | What blocked it |
|---|---|
| SCTP | IP protocol 132 unrecognised — dropped by NATs and firewalls |
| TCP Fast Open | Middleboxes dropping SYNs carrying data |
| ECN | Middleboxes dropping packets with ECN bits set |
| Window scaling | Options stripped, producing a silent 2^S disagreement |
| New TCP options generally | Stripped or the packet dropped |
| DCCP | Unknown IP protocol number — same fate as SCTP |
A particularly instructive case is TCP's reserved header bits. They were specified as "must be zero, reserved for future use". Middleboxes began dropping packets with any of them set — a reasonable-looking sanity check that made the field permanently unusable. The extension mechanism was destroyed by devices enforcing the specification too literally.
Three strategies exist for working around this, and they form a clear progression:
Encapsulate in UDP. This is what QUIC does, and what WebRTC does to carry SCTP. UDP is universally passed, so the real protocol travels inside it invisibly. The cost is 8 bytes of header and the loss of any hardware offload designed for TCP.
Encrypt everything. QUIC encrypts nearly its entire transport header. A middlebox cannot depend on a field it cannot read, so ossification is prevented by construction rather than by hoping vendors behave. This is the strongest available defence, and QUIC's designers were explicit that anti-ossification was a primary goal, not a side effect.
Fail gracefully. MPTCP falls back to plain TCP if its options are stripped, and Happy Eyeballs (Chapter 5) races IPv4 and IPv6 so a broken path costs a delay rather than a failure. This does not defeat ossification but makes attempting a new feature safe.
The deeper lesson generalises beyond networking: any observable interface will eventually be depended upon. If a protocol exposes a field, something will parse it, and that dependency becomes a constraint. Greasing — deliberately sending random unknown values so implementations must tolerate them — is the systematic defence, used by TLS (GREASE) and QUIC (version negotiation with reserved versions) to keep parsers honest by exercising the unknown-value path continuously.
Examples
How ossification happens
1. protocol ships; middleboxes parse it as it is TODAY
2. new feature appears -> middlebox sees something unfamiliar
3. middlebox DROPS it (safest-looking behaviour for the vendor)
4. the feature works on 95% of paths, fails on 5%
5. a browser cannot ship something that breaks 5% of users
6. feature NOT DEPLOYED
7. middleboxes are never updated, because nothing uses it
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
a stable equilibrium that freezes the protocol
The reserved-bits tragedy
TCP spec: "these bits are reserved; must be zero"
middlebox authors: "if a reserved bit is set, the packet is malformed
or malicious -> DROP"
result: the extension mechanism EXPLICITLY PROVIDED for future use
became permanently unusable, by devices enforcing the
specification too literally.
The three defences
1. ENCAPSULATE QUIC in UDP, SCTP in DTLS/UDP
+--------+-----+--------------------+
|Ethernet| UDP | the real protocol |
+--------+-----+--------------------+
^ this is all the middlebox sees
2. ENCRYPT QUIC's header
+--------+-----+---+========================+
|Ethernet| UDP |CID| ENCRYPTED — unreadable |
+--------+-----+---+========================+
^ cannot be depended on, so cannot ossify
3. FAIL GRACEFULLY MPTCP, Happy Eyeballs
try the new thing -> if it fails, fall back silently
-> attempting a feature becomes SAFE
def deployment_outcome(feature, middlebox_behaviour):
"""Whether a new transport feature can actually ship."""
if middlebox_behaviour == "passes unknown":
return True, "deploys successfully"
if middlebox_behaviour == "strips options":
return None, "works, but silently degraded — may cause subtle bugs"
return False, "CONNECTION FAILS on affected paths -> cannot deploy"
FEATURES = {
"SCTP (IP proto 132)": "drops unknown",
"TCP Fast Open (SYN data)": "drops unknown",
"ECN bits": "drops unknown",
"Window scaling option": "strips options",
"MPTCP options": "strips options",
"QUIC (UDP 443)": "passes unknown",
}
print(f"{'feature':<28}{'middlebox':<18}{'outcome'}")
for feat, behaviour in FEATURES.items():
ok, note = deployment_outcome(feat, behaviour)
print(f"{feat:<28}{behaviour:<18}{note}")
# feature middlebox outcome
# SCTP (IP proto 132) drops unknown CONNECTION FAILS on affected paths -> cannot deploy
# TCP Fast Open (SYN data) drops unknown CONNECTION FAILS on affected paths -> cannot deploy
# ECN bits drops unknown CONNECTION FAILS on affected paths -> cannot deploy
# Window scaling option strips options works, but silently degraded — may cause subtle bugs
# MPTCP options strips options works, but silently degraded — may cause subtle bugs
# QUIC (UDP 443) passes unknown deploys successfully
# Why a small failure rate blocks deployment entirely.
def can_ship(failure_rate, user_base=1_000_000_000):
affected = failure_rate * user_base
shippable = failure_rate < 0.0001
return affected, shippable
print()
for rate in (0.00001, 0.001, 0.01, 0.05):
affected, ok = can_ship(rate)
print(f"{rate:>8.3%} of paths broken = {affected:>13,.0f} users -> "
f"{'shippable' if ok else 'CANNOT SHIP'}")
# 0.001% of paths broken = 10,000 users -> shippable
# 0.100% of paths broken = 1,000,000 users -> CANNOT SHIP
# 1.000% of paths broken = 10,000,000 users -> CANNOT SHIP
# 5.000% of paths broken = 50,000,000 users -> CANNOT SHIP
# A 99% success rate is a catastrophic failure rate at internet scale.
# This is why "mostly works" is not good enough to deploy.
# What each protocol exposes, and therefore what can ossify.
def ossifiable_surface(protocol):
return {
"TCP": ["ports", "sequence numbers", "ACK numbers", "flags", "window",
"options", "reserved bits"],
"QUIC": ["UDP ports", "a few unencrypted header bits", "connection ID"],
}[protocol]
print()
for p in ("TCP", "QUIC"):
fields = ossifiable_surface(p)
print(f"{p:<6}{len(fields)} inspectable fields: {', '.join(fields)}")
# TCP 7 inspectable fields: ports, sequence numbers, ACK numbers, flags, window, options, reserved bits
# QUIC 3 inspectable fields: UDP ports, a few unencrypted header bits, connection ID
# Every inspectable field is a field something can come to depend on.
# QUIC minimises the surface deliberately.
# Greasing: keep parsers honest by exercising the unknown-value path.
import random
def grease(values, add_grease):
if not add_grease:
return values, "a strict parser survives — until a REAL new value appears"
reserved = [0x0a0a, 0x1a1a, 0x2a2a]
out = values + [random.choice(reserved)]
return out, "implementations MUST tolerate unknown values or fail immediately"
print()
for g in (False, True):
vals, note = grease([0x0001, 0x0002], g)
print(f" grease={g!s:<5} sent: {[hex(v) for v in vals]}")
print(f" {note}")
# grease=False sent: ['0x1', '0x2']
# a strict parser survives — until a REAL new value appears
# grease=True sent: ['0x1', '0x2', '0x1a1a']
# implementations MUST tolerate unknown values or fail immediately
# By sending garbage continuously, TLS GREASE and QUIC version negotiation
# make intolerance fail NOW rather than years later when it matters.
# The defence hierarchy.
DEFENCES = {
"encapsulate in UDP": (1, "middleboxes see only UDP — QUIC, WebRTC's SCTP"),
"encrypt the header": (1, "cannot be depended on, so cannot ossify — QUIC"),
"graceful fallback": (2, "attempting a feature becomes safe — MPTCP, Happy Eyeballs"),
"grease unknown values": (2, "force tolerance continuously — TLS, QUIC"),
"standardise and hope": (3, "what SCTP and ECN tried; it did not work"),
}
print()
for name, (rank, note) in sorted(DEFENCES.items(), key=lambda kv: kv[1][0]):
print(f" tier {rank}: {name:<24}{note}")
# tier 1: encapsulate in UDP middleboxes see only UDP — QUIC, WebRTC's SCTP
# tier 1: encrypt the header cannot be depended on, so cannot ossify — QUIC
# tier 2: graceful fallback attempting a feature becomes safe — MPTCP, Happy Eyeballs
# tier 2: grease unknown values force tolerance continuously — TLS, QUIC
# tier 3: standardise and hope what SCTP and ECN tried; it did not work
# Deployment timelines tell the story.
TIMELINES = {
"SCTP": (2000, None, "still not deployable on the public internet"),
"ECN": (2001, 2015, "~14 years to meaningful adoption; still partial"),
"TCP Fast Open": (2014, None, "never achieved broad deployment"),
"MPTCP": (2013, 2020, "7 years; succeeded via graceful fallback"),
"QUIC/HTTP-3": (2016, 2021, "5 years to majority CDN support"),
}
print()
print(f"{'protocol':<18}{'spec':>6}{'deployed':>11} note")
for name, (spec, dep, note) in TIMELINES.items():
d = str(dep) if dep else "never"
print(f"{name:<18}{spec:>6}{d:>11} {note}")
# protocol spec deployed note
# SCTP 2000 never still not deployable on the public internet
# ECN 2001 2015 ~14 years to meaningful adoption; still partial
# TCP Fast Open 2014 never never achieved broad deployment
# MPTCP 2013 2020 7 years; succeeded via graceful fallback
# QUIC/HTTP-3 2016 2021 5 years to majority CDN support
# The two that deployed are the two that assumed middlebox hostility
# from the start.
# Test whether a path passes anything unusual. Start with protocol numbers:
$ sudo hping3 -c 1 --rawip --ipproto 132 server 2>&1 | tail -2
# SCTP (132) almost never survives a NAT.
$ nc -u -w1 server 443 <<< test && echo "UDP/443 passes"
# UDP/443 passes <- QUIC's substrate, universally allowed
# Does this path allow TCP Fast Open?
$ sysctl net.ipv4.tcp_fastopen
# net.ipv4.tcp_fastopen = 1
$ nstat -az | grep -i fastopen
# TcpExtTCPFastOpenActive 412
# TcpExtTCPFastOpenActiveFail 1204 <- blocked on these paths
# TcpExtTCPFastOpenPassive 89
# A high Fail:Active ratio means middleboxes on your paths reject SYN data.
# Does ECN negotiation survive?
$ sudo sysctl -w net.ipv4.tcp_ecn=1
$ nstat -az | grep -iE "ECN|SYNRetrans"
# TcpExtTCPECNSeen 4821
# TcpExtSyncookiesFailed 0
$ sysctl net.ipv4.tcp_ecn_fallback
# net.ipv4.tcp_ecn_fallback = 1 <- retry without ECN if the ECN SYN fails:
# graceful fallback, applied to ECN
# Were window scale options stripped? Capture at BOTH ends and compare:
$ tshark -r client.pcap -T fields -e tcp.options.wscale.shift | head -1
# 7
$ tshark -r server.pcap -T fields -e tcp.options.wscale.shift | head -1
# (empty) <- stripped in transit. Found the middlebox.
# Did MPTCP fall back?
$ nstat -az | grep -i mptcpfallback
# MPTcpExtMPFallbackTokenInit 23
# ^^ 23 connections silently became plain TCP.
# Graceful fallback in action: degraded, but never broken.
# What QUIC actually exposes — very little:
$ sudo tcpdump -i eth0 -n 'udp port 443' -X -c 1 | head -4
# IP 10.1.1.5.51250 > 104.16.132.229.443: UDP, length 1350
# 0x0000: c000 0000 0108 a1b2 c3d4 e5f6 ...
# ^^ header form + version
# ^^^^^^^^^^^^^^^^^^^ connection ID
# Everything after is encrypted. There is almost nothing here to ossify on.
# GREASE in TLS — deliberate garbage to keep parsers tolerant:
$ openssl s_client -connect example.com:443 -tls1_3 2>&1 | head -3
$ tshark -r tls.pcap -T fields -e tls.handshake.ciphersuite | head -1
# 0x0a0a,0x1301,0x1302,0x1303
# ^^^^^^ a GREASE value. Servers MUST ignore it. Any that do not fail
# immediately and visibly, rather than blocking a future extension.
# Measure your own path's tolerance with a purpose-built tool:
$ tracebox -p 'IP/TCP/TCP_OPT_MSS' server 2>/dev/null | head -5
# tracebox shows WHICH hop modified your packet and which field changed —
# the definitive tool for locating an ossifying middlebox.
Key takeaways
- Ossification is the loss of a protocol's ability to evolve, caused by middleboxes depending on details they were never meant to interpret.
- The cycle is self-reinforcing: unfamiliar traffic is dropped, so features fail on some paths, so endpoints do not deploy them, so middleboxes are never updated.
- At internet scale, even a 0.1% failure rate blocks deployment — "mostly works" is not shippable.
- Casualties: SCTP and DCCP (unknown IP protocol numbers), TCP Fast Open and ECN (dropped SYNs), window scaling (stripped options), and TCP's reserved bits, made unusable by sanity checks.
- Encapsulating in UDP works because UDP is universally passed — QUIC's central decision, and how WebRTC carries SCTP.
- Encrypting the transport header prevents ossification by construction: what cannot be read cannot be depended upon.
- Graceful fallback (MPTCP falling back to TCP, Happy Eyeballs racing address families) makes attempting a new feature safe.
- Greasing — sending random reserved values continuously — forces implementations to tolerate the unknown, making intolerance fail immediately rather than years later.
- The general lesson: any observable interface will eventually be depended upon, so minimise what you expose.
🧪 Practice
- Describe the five-step cycle by which a protocol becomes ossified.
- Explain why a feature that works on 99% of paths still cannot be deployed by a major browser.
- Name the three anti-ossification strategies and give a protocol using each.
- Interview-style: QUIC encrypts almost its entire transport header, which makes debugging harder and rules out many middlebox optimisations. Argue that this was the correct decision. (Hint: list what happened to every TCP header field that middleboxes could read, and ask what QUIC's designers were protecting against in twenty years' time.)
<a id="8-the-application-layer"></a>