8. The Application Layer

Every layer so far existed to serve this one: the application layer is where the network stops being plumbing and starts being something a person uses — a page that loads, a message that arrives, a video that plays. This chapter covers how applications are structured across machines, how names become addresses, and the protocols that carry the overwhelming majority of internet traffic: DNS, HTTP, mail, SSH, and streaming media. The recurring theme is that application protocols are conventions about bytes, and that their design choices are shaped almost entirely by the transport guarantees, and round-trip costs, established in the previous chapters.

8.1 Application Architectures

Before any protocol, an application must decide which machine does what — this subchapter covers the structural choices and the service requirements that drive them.

Client-Server Model

Theory

Start with the problem. Two programs on different machines need to cooperate. Someone has to speak first, and for that to work, one side must be findable — it must be sitting at a known address, already running, waiting. The other side can be anywhere, appear at any time, and vanish without warning.

That asymmetry is the entire client-server model. It is not a technology; it is a division of roles:

The server is the passive, always-on party. It has a stable, publicly known address, binds a well-known port, and never initiates. Its job is to wait and respond.
The client is the active, intermittent party. It knows where the server is, initiates contact, makes requests, and may disappear the moment it is finished.

The restaurant analogy holds up well. The restaurant has a fixed address, published opening hours, and staff who arrive before any customer does. Customers know where the restaurant is; the restaurant does not know where its customers are, and does not need to. Communication is always customer-initiated — the kitchen never rings you at home to ask if you would like dessert.

Three consequences follow directly, and they explain most of what is distinctive about server software:

1. Clients never talk to each other. In a pure client-server system, all communication passes through the server. Two users of a chat application do not exchange packets; both exchange packets with the server, which relays. This centralises everything — state, policy, logging, abuse control — which is both the model's greatest strength and its main cost.

2. The server is a bottleneck and a single point of failure. Every client depends on it. Capacity must grow with the user population, and if it goes down, the application is gone. This is why real deployments are never literally "a server": they are a server farm behind a load balancer, with the single logical address mapped onto many physical machines.

3. Addressing is one-way. The server needs a permanent, well-known address; the client does not need one at all. This is precisely why NAT (Chapter 5) works as well as it does for consumer internet access — hundreds of millions of clients sit behind addresses that are neither stable nor globally reachable, and it does not matter, because clients only ever initiate.

The model's scaling story is worth stating plainly. If each client generates load L and there are N clients, the server must handle N × L while each client handles only L. Doubling users doubles server cost and changes client cost not at all. Every technique in large-scale web infrastructure — caching, replication, sharding, CDNs — exists to attack that multiplication.

Examples

Client-server: the star, with all traffic through the centre

        client A            client B            client C
        (laptop)            (phone)             (kiosk)
        NAT'd, mobile       NAT'd, mobile       NAT'd
             \                  |                  /
              \                 |                 /
               \                |                /
                v               v               v
              +-----------------------------------+
              |             SERVER                |
              |  stable public address            |
              |  well-known port (443)            |
              |  always running                   |
              |  holds ALL shared state           |
              +-----------------------------------+

  Arrows point ONE WAY at connection setup time: clients initiate.
  A and B never exchange a packet directly, even when chatting
  with each other -- the server relays.

  Load on server: N x L        Load on each client: L
  ^ this asymmetry is the whole story of web infrastructure

# The role difference expressed as state, not as code.
# A server's defining property is that it is REACHABLE and WAITING.

class Server:
    def __init__(self, address, port):
        self.address = address       # stable, published, resolvable via DNS
        self.port = port             # well-known: clients assume it
        self.always_on = True        # started before any client existed
        self.state = {}              # the shared truth of the application

    def handle(self, client_id, request):
        # The server NEVER initiates. Every action begins here,
        # triggered by a client that found it.
        self.state.setdefault(client_id, []).append(request)
        return f"ack:{request}"

class Client:
    def __init__(self, name):
        self.name = name
        self.address = None          # ephemeral, often NAT'd, may change
        self.always_on = False       # exists only while the user is present

    def request(self, server, text):
        # The client must KNOW the server's address. The reverse is not true.
        return server.handle(self.name, text)

srv = Client  # placeholder to show ordering matters
server = Server("93.184.216.34", 443)   # exists first
alice = Client("alice")                  # appears later
bob   = Client("bob")                    # appears later still

print(alice.request(server, "GET /index.html"))   # ack:GET /index.html
print(bob.request(server, "POST /message"))       # ack:POST /message

# Alice sends Bob a message -- note it goes THROUGH the server:
print(alice.request(server, "SEND to=bob 'hello'"))  # ack:SEND to=bob 'hello'
print(server.state)
# {'alice': ['GET /index.html', "SEND to=bob 'hello'"], 'bob': ['POST /message']}
#  ^ the server holds everything; the clients hold nothing shared

# The scaling consequence, quantified:
def load(clients, per_client_ops):
    return clients * per_client_ops, per_client_ops

for n in (1, 1_000, 1_000_000):
    s, c = load(n, 50)
    print(f"{n:>9,} clients -> server: {s:>12,} ops/s | each client: {c} ops/s")
#         1 clients -> server:           50 ops/s | each client: 50 ops/s
#     1,000 clients -> server:       50,000 ops/s | each client: 50 ops/s
# 1,000,000 clients -> server:   50,000,000 ops/s | each client: 50 ops/s
#   The client's job never gets harder. The server's grows without bound.

# A server is identifiable from the outside: it answers on a known port.
$ ss -tlnp | grep ':443'
# LISTEN 0 511 0.0.0.0:443 0.0.0.0:*  users:(("nginx",pid=812,fd=6))
#        ^ LISTEN = the passive posture that defines a server

# A client is identifiable by having no listening socket at all:
$ ss -tnp | grep firefox | head -2
# ESTAB 0 0 192.168.1.20:52814 93.184.216.34:443  users:(("firefox",...))
# ESTAB 0 0 192.168.1.20:52815 93.184.216.34:443  users:(("firefox",...))
#                        ^^^^^ ephemeral source ports, chosen per connection
#   No LISTEN line: nothing can contact Firefox first.

# "One server" is almost always a lie -- one name, many machines:
$ dig +short www.example-cdn.com
# 151.101.1.140
# 151.101.65.140
# 151.101.129.140     <- one logical server, several physical answers

Key takeaways

Client-server is a division of roles, not of hardware: the server waits at a known address, the client initiates from anywhere.
Servers need stable, publicly reachable addresses; clients do not — which is exactly why NAT is survivable for consumer access.
In a pure client-server system, clients never communicate directly; the server relays and holds all shared state.
Centralisation buys easy consistency, policy, logging, and abuse control, and costs scalability and availability.
Server load scales as N × L while client load stays at L — the asymmetry that motivates load balancers, caches, and CDNs.
A real "server" is normally a farm behind one name; the single logical endpoint is a naming illusion.

🧪 Practice

List the properties a machine must have to act as a server but not to act as a client, and explain why NAT breaks one role and not the other.

Two chat users are in the same room on the same Wi-Fi. Trace the path their messages take in a pure client-server design and count how many times each message crosses the internet.

A service has 200,000 users, each generating 4 requests per minute. Compute the server's required request rate, then state which two components you would replicate first and why.

Interview-style: Your client-server application is slow only for users in Australia, while the server in Frankfurt is at 20% CPU. (Hint: the server's load metric and the user's latency are measuring two different things — revisit which chapter's numbers dominate here.)

Peer-to-Peer Model

Theory

The client-server model has a structural flaw that becomes obvious at scale: the resource that is scarcest — server upload capacity — is the one that must grow fastest. Meanwhile, every client in the system has an idle upload link.

Peer-to-peer inverts the assumption. There are no permanently distinguished roles. Each participant is a peer: it consumes and it serves, simultaneously. The system's capacity is contributed by its participants, so adding a user adds both demand and supply.

This is the crucial property, and it is worth stating precisely. In client-server, capacity is fixed and demand grows with N, so per-user service degrades. In peer-to-peer, both grow with N, so per-user service can stay flat — the system is self-scaling. A file that becomes popular becomes easier to distribute, not harder, because every downloader turns into an uploader.

The classic quantitative argument is distribution time. To send a file of size F to N clients:

Client-server: the server must upload N × F bits through its own link u_s, so the time is at least N·F / u_s. It grows linearly with N, without bound.
Peer-to-peer: the server must upload the file at least once (F / u_s), every peer must download it (F / d_min), and the total upload capacity is shared (N·F / (u_s + Σu_i)). The binding constraint grows only logarithmically in practice, because the aggregate upload term grows with N too.

The costs are real and they are the reason P2P did not take over everything:

Problem	Why it appears	Typical mitigation
Peer discovery	No fixed address to ask	Trackers, DHTs, bootstrap nodes
Churn	Peers leave mid-transfer	Redundancy, re-request, many sources
NAT traversal	Peers must accept connections	STUN/TURN, hole punching, relays
Free-riding	Downloading without uploading	Tit-for-tat, choking algorithms
Trust	Content comes from strangers	Per-block hashing, Merkle trees
No central policy	Nobody can enforce anything	Reputation, out-of-band moderation

BitTorrent is the canonical design and solves several of these at once. A file is split into fixed-size pieces, each with a hash in the .torrent metadata. A peer can therefore verify every piece independently, from any source, without trusting the sender. Rarest-first piece selection keeps the swarm healthy by preventing any piece from going extinct. Tit-for-tat choking means a peer uploads preferentially to those who upload back, making free-riding slow rather than impossible.

The security model deserves emphasis because it generalises beyond file sharing: when you cannot trust the source, you verify the content. Hashing each piece turns an untrusted network of strangers into a reliable delivery system. Content-addressed storage, Git, and blockchain systems all reuse this exact idea.

Examples

Client-server vs peer-to-peer: where capacity comes from

 CLIENT-SERVER                          PEER-TO-PEER
 capacity is the server's               capacity is everyone's

      [SERVER u_s]                          [seed]
      /    |    \                          /      \
     /     |     \                    [peer A]---[peer B]
    v      v      v                       |  \   /  |
  [C1]   [C2]   [C3]                      |   \ /   |
                                          |    X    |
  server uploads N x F bits               |   / \   |
  each new client makes it WORSE      [peer C]---[peer D]

                                     every peer uploads what it has
                                     each new peer adds capacity too


 Distribution time as N grows (F=1 GB, u_s=100 Mb/s, peers 5 Mb/s up):

   N       client-server        peer-to-peer
   1            80 s                80 s
  10           800 s               160 s
 100         8,000 s               178 s
1000        80,000 s               180 s
             ^ linear              ^ flattens: supply grows with demand

def cs_time(N, F_bits, us, dmin):
    """Client-server: server uploads the whole file to every client."""
    return max(N * F_bits / us, F_bits / dmin)

def p2p_time(N, F_bits, us, dmin, u_peer):
    """P2P: bounded by seed upload, slowest download, OR aggregate capacity."""
    seed_once   = F_bits / us                      # seed must send it at least once
    slowest     = F_bits / dmin                    # everyone must receive it
    aggregate   = N * F_bits / (us + N * u_peer)   # shared upload pool grows with N
    return max(seed_once, slowest, aggregate)

F  = 1_000 * 8e6        # 1 GB in bits
us = 100e6              # server/seed upload 100 Mb/s
dn = 50e6               # peer download 50 Mb/s
up = 5e6                # peer upload 5 Mb/s

print(f"{'N':>6} {'client-server':>16} {'peer-to-peer':>14}")
for N in (1, 10, 100, 1000, 10000):
    print(f"{N:>6} {cs_time(N,F,us,dn):>14,.0f} s {p2p_time(N,F,us,dn,up):>12,.0f} s")
#      N    client-server   peer-to-peer
#      1             80 s           80 s
#     10            800 s          160 s
#    100          8,000 s          178 s
#   1000         80,000 s          180 s
#  10000        800,000 s          180 s
#   ^ client-server grows without bound; P2P converges to a constant

# Why verification replaces trust: hash every piece independently.
import hashlib

PIECE = 256 * 1024
def piece_hashes(data):
    return [hashlib.sha1(data[i:i+PIECE]).digest()
            for i in range(0, len(data), PIECE)]

original = bytes(range(256)) * 4096          # stand-in for a real file
manifest = piece_hashes(original)            # published in the .torrent

def accept_piece(index, blob):
    """A peer accepts data from a STRANGER only if the hash matches."""
    return hashlib.sha1(blob).digest() == manifest[index]

good = original[:PIECE]
evil = b"\x00" + original[1:PIECE]           # one byte tampered
print(accept_piece(0, good))   # True
print(accept_piece(0, evil))   # False
#   The sender's identity never mattered. The content authenticated itself.

# A swarm is visible as many simultaneous peer connections, no central server:
$ ss -tn | grep -c ESTAB
# 84
$ ss -tn | awk '{print $5}' | cut -d: -f1 | sort -u | head -5
# 24.5.113.9        <- residential
# 82.14.201.55      <- residential, different country
# 91.200.12.4
# 188.40.77.120
#   No single address dominates: capacity is coming from everywhere.

# Rarest-first in action (conceptual tracker view):
# piece  copies-in-swarm   priority
#   14        1            <- request FIRST: one departure and it is extinct
#   77        3
#    2       58            <- request last: plentiful

Key takeaways

In P2P every node is both consumer and provider, so adding a user adds demand and capacity together — the system is self-scaling.
Client-server distribution time grows linearly with N; P2P distribution time converges to a constant as aggregate upload grows.
P2P trades the server bottleneck for discovery, churn, NAT traversal, free-riding, and trust problems.
BitTorrent splits files into hash-verified pieces, uses rarest-first to keep pieces alive, and tit-for-tat choking to punish free-riders.
Verify content, not sources: per-piece hashing makes untrusted strangers a safe delivery mechanism — the same idea behind Git and content-addressed storage.
P2P is weakest exactly where client-server is strongest: enforcing policy and guaranteeing consistency.

🧪 Practice

Explain in one sentence why a popular file is easier to distribute over P2P but harder over client-server.

Using F = 8 Gb, u_s = 50 Mb/s, peer upload 2 Mb/s, peer download 20 Mb/s, compute client-server and P2P distribution times for N = 10 and N = 1000.

Describe how BitTorrent lets a peer safely accept data from a host it has never seen and has no reason to trust.

Interview-style: Why did P2P win for file distribution but lose for messaging, even though both are many-to-many? (Hint: consider which one needs a guaranteed-present participant and an authoritative ordering of events.)

Hybrid Architectures

Theory

Pure architectures are rare in production because the two models fail in complementary ways. Client-server is easy to control and hard to scale; P2P is easy to scale and hard to control. The obvious move is to use each where it is strong.

The organising principle of every hybrid is a split between:

the control plane — small, infrequent, needs authority: who exists, who is allowed, what the address is, what the current truth is; and
the data plane — large, sustained, needs bandwidth: the actual file, video, or media stream.

Put the control plane on a server and the data plane on peers. The server handles a few kilobytes per session and stays cheap; the peers handle gigabytes and cost the operator nothing.

Three recurring shapes:

1. Centralised directory, distributed transfer. The original Napster design, and still how BitTorrent trackers and most video calls work. A server knows who has what or who is online; the transfer happens directly between peers. Killing the server stops discovery but not transfers in progress.

2. Supernodes / hierarchical peers. Peers are not equal: those with good bandwidth, public addresses, and long uptime are promoted to supernodes that index and relay for ordinary peers. Skype's original design worked this way. It keeps the system serverless while avoiding the cost of flooding every query to everyone.

3. Server-assisted fallback. The system prefers the P2P path but keeps a server relay for cases where it fails — typically NAT traversal. WebRTC's ICE is the standard example: try direct, then STUN-assisted hole punching, and only if both fail use a TURN relay, which costs the operator real bandwidth. Roughly 10–20% of calls in practice need the relay.

The economics matter as much as the engineering. In WebRTC, a direct peer connection costs the operator nothing; a TURN relay costs the full media bitrate in both directions. Systems therefore invest heavily in raising the direct-connection success rate, because each percentage point is a line item.

Modern content delivery is also best understood as a hybrid, just with the "peers" owned by one company. A CDN (covered in 8.6) is a client-server system whose server has been replicated to hundreds of locations; the client-server contract is unchanged, but the capacity now scales by replication rather than by growing one machine.

Examples

The universal hybrid split: thin control plane, fat data plane

           +-------------------------------+
           |   SERVER (control plane)      |
           |   - who is online             |   a few KB per session
           |   - who has which piece       |   cheap, centralised,
           |   - authentication, policy    |   authoritative
           +-------------------------------+
              ^         ^          ^
              | signal  | signal   | signal          <- small
              |         |          |
           [peer A]  [peer B]   [peer C]
              |  \______  ______/  |
              |         \/         |
              +====== MEDIA =======+                 <- large
                 direct, peer-to-peer
                 gigabytes, never touches the server


WebRTC connectivity ladder (try cheapest first):

  1. HOST candidate    direct, same LAN            cost to operator: 0
        | fails
  2. SRFLX (STUN)      hole-punched through NAT    cost to operator: ~0
        | fails
  3. RELAY (TURN)      server relays all media     cost: FULL bitrate x2
                                                   ~10-20% of sessions

# Cost model for a hybrid: why raising direct-connection rate is a budget item.

def monthly_cost(sessions, minutes, bitrate_kbps, relay_fraction, usd_per_gb=0.09):
    """Only relayed sessions cost bandwidth; direct ones are free to us."""
    relayed = sessions * relay_fraction
    # Relay carries the stream in AND out -> x2
    gb = relayed * minutes * 60 * bitrate_kbps * 1000 * 2 / 8 / 1e9
    return gb, gb * usd_per_gb

for frac in (0.30, 0.20, 0.12, 0.05):
    gb, usd = monthly_cost(1_000_000, 20, 1500, frac)
    print(f"relay rate {frac:>5.0%} -> {gb:>10,.0f} GB  ${usd:>10,.0f}/month")
# relay rate   30% ->  1,350,000 GB  $   121,500/month
# relay rate   20% ->    900,000 GB  $    81,000/month
# relay rate   12% ->    540,000 GB  $    48,600/month
# relay rate    5% ->    225,000 GB  $    20,250/month
#   Each point of NAT-traversal success is worth real money.

# The control/data split expressed as a protocol trace:
session = [
    ("client->server", "LOGIN alice",              48),      # bytes
    ("server->client", "OK peers=[bob]",           36),
    ("client->server", "OFFER sdp...",            920),      # signalling only
    ("server->peer",   "RELAY OFFER",             920),
    ("peer->client",   "ANSWER sdp...",           880),
    ("client<->peer",  "MEDIA 20 min @1.5 Mb/s", 225_000_000),
]
ctrl = sum(b for who, _, b in session if "server" in who)
data = sum(b for who, _, b in session if "server" not in who)
print(f"\ncontrol plane: {ctrl:>12,} bytes via server")
print(f"data plane:    {data:>12,} bytes peer-to-peer")
print(f"server carries {ctrl/(ctrl+data):.6%} of the bytes")
# control plane:        2,804 bytes via server
# data plane:     225,000,000 bytes peer-to-peer
# server carries 0.001246% of the bytes
#   The server is authoritative over the session while being irrelevant to its cost.

# Inspect an ICE negotiation: which candidate type actually won.
# (chrome://webrtc-internals export, trimmed)
$ jq '.candidatePairs[] | select(.state=="succeeded") | {local,remote}' call.json
# { "local": "srflx 81.2.44.9:54312", "remote": "srflx 24.5.113.9:61003" }
#           ^^^^^ STUN-assisted hole punch succeeded: no relay needed

# A tracker is a control-plane server -- tiny responses, huge consequences:
$ curl -s "http://tracker.example:6969/announce?info_hash=...&peer_id=..." | wc -c
# 412
#   412 bytes of control data unlock gigabytes of peer-to-peer transfer.

Key takeaways

Hybrids split the control plane (small, authoritative, centralised) from the data plane (large, bandwidth-hungry, distributed).
Centralised directory + distributed transfer is the dominant pattern: the server knows who, the peers move what.
Supernodes promote well-connected peers into indexing/relay roles, avoiding both a central server and query flooding.
Server-assisted fallback (STUN then TURN in WebRTC) prefers the free direct path and pays for a relay only when NAT defeats it.
Relay fallback is an operating cost, so NAT-traversal success rate is an economic metric, not just a technical one.
A CDN is a hybrid in disguise: unchanged client-server semantics, capacity scaled by replication instead of by a bigger machine.

🧪 Practice

Classify these as control plane or data plane: login, piece request, video frames, peer list, TLS handshake to the signalling server.

Explain what still works and what immediately breaks if a BitTorrent tracker goes offline mid-swarm.

Using the cost model above, compute the monthly saving of reducing TURN usage from 18% to 9% for 500,000 sessions of 15 minutes at 1 Mb/s.

Interview-style: Design a file-sharing service for a company that must log every file access but cannot afford to carry the file traffic. (Hint: decide which plane the audit requirement actually lives in.)

Application Requirements: Loss, Delay, Throughput

Theory

Before choosing a transport or designing a protocol, an application should answer three questions. They are the only three that matter at this layer, and different applications answer them in dramatically different ways.

1. How much loss can it tolerate? Some applications are loss-intolerant: a file transfer, a bank transaction, an email. One missing byte corrupts the result, so every byte must arrive. Others are loss-tolerant: in a voice call, a lost 20 ms packet is a barely perceptible artefact. Retransmitting it would be worse than losing it, because by the time it arrived the moment for playing it would have passed.

2. How sensitive is it to delay? This splits into two distinct quantities that people routinely confuse:

Latency — how long one packet takes. Matters for interactivity.
Jitter — how much that latency varies. Matters for anything played back at a fixed rate.

A voice call at a rock-steady 200 ms is usable. The same call averaging 100 ms but swinging between 20 ms and 300 ms is not, because the playout buffer cannot keep up. This is why jitter, not average latency, is the metric that real-time media engineers watch.

The human thresholds are worth memorising:

Budget	Effect
< 150 ms one-way voice	Conversation feels natural
150–400 ms	Noticeable; people start talking over each other
> 400 ms	Conversation breaks down; walkie-talkie behaviour
~100 ms UI response	Feels instantaneous
~1 s page load	Attention holds
> 10 s	User abandons

3. How much throughput does it need, and is that need elastic?

Bandwidth-sensitive applications need a minimum to function at all. Video at 1080p needs roughly 5 Mb/s; below that, it does not degrade gracefully so much as it fails.
Elastic applications use whatever is available. A file download at 1 Mb/s is slow; at 1 Gb/s it is fast; both work.

Putting the three axes together gives the map that explains transport choice:

Application	Loss	Delay	Throughput	Transport
File transfer, email	Zero	Insensitive	Elastic	TCP
Web page	Zero	~1 s	Elastic, bursty	TCP / QUIC
Voice call	Tolerant (<1%)	< 150 ms, low jitter	~64 kb/s fixed	UDP + RTP
Video call	Tolerant (<1%)	< 150 ms, low jitter	0.5–5 Mb/s	UDP + RTP
Streaming video	Zero	Insensitive (buffered)	2–25 Mb/s	TCP / QUIC
Online game	Tolerant	< 50 ms	Low	UDP
DNS lookup	Tolerant (retry)	< 100 ms	Tiny	UDP

Two rows deserve attention because they look contradictory. Streaming video is loss-intolerant and delay-insensitive, even though it is video — because it is played from a buffer seconds deep, so there is time to retransmit, and a corrupted frame is visible for far longer than a delayed one. A video call is the exact opposite, and this single difference is why Netflix runs over TCP/QUIC and a video call does not.

The general rule that falls out: use TCP when correctness matters more than timing; use UDP when timing matters more than correctness. Everything else is detail.

Examples

The three-axis map of application requirements

  loss-intolerant
        ^
        |  [email]      [file transfer]      [streaming video]
        |  [web]                              (buffered: time to retransmit)
        |
        |-------------------------------------------------> delay-sensitive
        |
        |  [sensor telemetry]   [game state]  [voice call]
        |                                     [video call]
  loss-tolerant                               (no time to retransmit)

   TCP lives on the top-left. UDP lives on the bottom-right.


Why jitter matters more than latency for media playback:

  CONSTANT 200 ms latency          VARIABLE 20-300 ms latency
  arrivals:  | | | | | | | |       arrivals: ||  |    ||| |
  playout:   | | | | | | | |       playout:  | |x| | |x| |
                                                ^     ^
  smooth: buffer never starves     gaps: buffer underruns = audible dropouts

  Average latency of the second is LOWER. It still sounds broken.

# Jitter, not mean latency, predicts playback quality.
import statistics

def playout_analysis(arrivals_ms, buffer_ms, frame_ms=20):
    """Simulate a fixed-delay playout buffer; count underruns."""
    play_time = arrivals_ms[0] + buffer_ms      # first frame plays after buffering
    underruns = 0
    for i, arrival in enumerate(arrivals_ms):
        deadline = play_time + i * frame_ms      # when this frame MUST be ready
        if arrival > deadline:
            underruns += 1                       # arrived too late = dropout
    return underruns

# Stream A: high but STEADY latency
steady = [200 + i * 20 for i in range(100)]
# Stream B: lower average, but jittery
import random
random.seed(7)
jittery = [random.choice([20, 40, 300, 60, 280]) + i * 20 for i in range(100)]

for name, s in (("steady  ", steady), ("jittery ", jittery)):
    gaps = [s[i+1] - s[i] for i in range(len(s)-1)]
    print(f"{name} mean={statistics.mean(s):7.1f}ms "
          f"jitter={statistics.pstdev(gaps):6.2f}ms "
          f"underruns(60ms buffer)={playout_analysis(s, 60)}")
# steady   mean=1190.0ms jitter=  0.00ms underruns(60ms buffer)=0
# jittery  mean=1127.4ms jitter=147.65ms underruns(60ms buffer)=38
#   Lower mean latency, 38 audible dropouts. Jitter is the metric that matters.

# Does retransmission even help? Only if the deadline is far enough away.
def retransmit_useful(rtt_ms, buffer_depth_ms):
    """A lost packet can be recovered only if a round trip fits in the buffer."""
    return rtt_ms < buffer_depth_ms

cases = [("video call",      60,    150),    # 150 ms total delay budget
         ("streaming video", 60, 10_000),    # 10 s of buffer
         ("online game",     60,     50)]
for name, rtt, budget in cases:
    ok = retransmit_useful(rtt, budget)
    print(f"{name:<18} rtt={rtt}ms budget={budget:>6}ms -> "
          f"retransmit {'WORTH IT (use TCP)' if ok else 'TOO LATE (use UDP)'}")
# video call         rtt=60ms budget=   150ms -> retransmit WORTH IT (use TCP)
# streaming video    rtt=60ms budget= 10000ms -> retransmit WORTH IT (use TCP)
# online game        rtt=60ms budget=    50ms -> retransmit TOO LATE (use UDP)
#   Note: the video call is marginal -- one RTT eats 40% of the budget,
#   which is why it uses UDP with selective, deadline-aware recovery instead.

# Measure the three axes on a real path before designing for it.

# Delay AND jitter (mdev is the jitter figure):
$ ping -c 100 -i 0.2 target | tail -1
# rtt min/avg/max/mdev = 18.4/24.1/198.7/12.6 ms
#                                    ^^^^^ 198 ms outlier: a media killer
#                                          ^^^^ 12.6 ms jitter

# Loss, measured over a long enough run to be meaningful:
$ ping -c 1000 -q target | grep loss
# 1000 packets transmitted, 997 received, 0.3% packet loss
#   0.3% is invisible to voice, fatal to an unprotected file transfer.

# Achievable throughput (elastic demand finds the ceiling):
$ iperf3 -c target -t 10 | tail -3
# [ ID] Interval        Transfer     Bitrate
# [  5] 0.00-10.00 sec  1.09 GBytes  936 Mbits/sec  sender
# [  5] 0.00-10.00 sec  1.09 GBytes  934 Mbits/sec  receiver

# One-way delay budget check for voice (half of RTT, roughly):
$ ping -c 20 -q voip-gw | awk -F'/' '/rtt/ {print "one-way approx", $5/2, "ms"}'
# one-way approx 31.2 ms      <- well inside the 150 ms conversational budget

Key takeaways

Every application is characterised by three requirements: loss tolerance, delay sensitivity, and throughput need.
Latency and jitter are different metrics; for anything played at a fixed rate, jitter is the one that determines quality.
Human budgets worth remembering: 150 ms one-way for natural conversation, 100 ms for an instant-feeling UI, 400 ms for conversational breakdown.
Bandwidth-sensitive applications have a floor below which they fail; elastic applications simply go faster or slower.
Streaming video is loss-intolerant and delay-insensitive (deep buffer allows retransmission); a video call is the reverse — which is why they use different transports.
Retransmission is useful only when one RTT fits inside the delay budget — that inequality is the real TCP-versus-UDP decision rule.

🧪 Practice

For a multiplayer game with a 50 ms budget on a 70 ms RTT path, state whether retransmission can help and what the application should do instead.

Two paths: A has 40 ms mean / 35 ms jitter, B has 90 ms mean / 3 ms jitter. Choose one for a voice call and one for a file download, with reasons.

Classify a live sports stream with 3 s of buffer on all three axes, and choose its transport.

Interview-style: Netflix runs over TCP but Zoom does not, even though both send video. (Hint: ask how many milliseconds each one has before a frame becomes worthless.)

Request-Response vs Streaming Interaction

Theory

Independently of architecture and transport, an application protocol must choose an interaction pattern: the shape of the conversation over time. There are two fundamental shapes, and most real protocols are one, the other, or a documented hybrid.

Request-response is a strict alternation. The client asks, the server answers, the exchange is complete. Each transaction is self-contained, which produces a set of very attractive properties: it is trivially stateless (the server can forget everything after replying), trivially cacheable (a response can be stored and replayed for the same request), trivially load-balanced (any server can handle any request), and trivially retryable (if a response never arrives, ask again). HTTP, DNS, and most REST APIs work this way.

The cost is that it is client-driven. The server cannot speak unless spoken to. If the client wants to know about events as they happen, it must ask repeatedly — polling — which is wasteful when events are rare and slow when they are frequent.

Streaming is a continuous flow along an open channel. After setup, either side may send at any time, and messages are not paired. This buys server push, low per-message overhead (no repeated headers or handshakes), and instant delivery. It costs connection state on the server (memory per connection, a hard scaling ceiling), no natural caching, harder load balancing (the connection is pinned to one server), and explicit reconnection logic because a broken stream is the application's problem.

The intermediate patterns exist precisely to buy push without paying full streaming cost:

Pattern	Direction	Connection	Typical use
Request-response	Client asks	Per request	REST, DNS
Polling	Client asks repeatedly	Per request	Legacy status checks
Long polling	Client asks, server holds	Held until event	Pre-WebSocket push
Server-Sent Events	Server pushes	One long-lived HTTP	Feeds, notifications
WebSocket	Both directions	One long-lived, upgraded	Chat, collaboration, games
gRPC streaming	Either or both	HTTP/2 stream	Service-to-service

The decision rule is a ratio. Let f be the rate at which meaningful events occur and p the polling rate needed for acceptable freshness. Polling wastes roughly (p − f)/p of its requests. When events are rare relative to the freshness requirement — chat, notifications, live prices — polling wastes almost everything and a stream wins decisively. When events are frequent relative to the acceptable staleness, polling is nearly free and keeps all the statelessness benefits.

The other half of the decision is who initiates. If only the client ever needs to start an exchange, request-response is simpler and should be the default. The moment the server must initiate, request-response requires an awkward workaround, and that is the real signal to reach for a stream.

Examples

Interaction patterns over time (t runs downward)

 REQUEST-RESPONSE          POLLING                 LONG POLLING           STREAMING
 C          S              C          S            C          S           C          S
 |--req---->|              |--req---->|            |--req---->|           |--open--->|
 |<--resp---|              |<--none---|            |          | (held)    |<--evt----|
 (done)                    |--req---->|            |          |           |<--evt----|
                           |<--none---|            |<--EVENT--|           |--msg---->|
 stateless, cacheable      |--req---->|            |--req---->|           |<--evt----|
 server never initiates    |<-EVENT---|            |          | (held)    (open for hours)
                           ^ 2 of 3 wasted         one response per event  state on server


 Cost of freshness, 1 event/hour, 1 s freshness requirement:

   polling:     3,600 requests/hour, 1 useful    -> 99.97% waste
   long poll:       1 request/hour,  1 useful    -> 0% waste, 1 held connection
   streaming:       1 message/hour               -> 0% waste, 1 open connection

# The ratio that decides the pattern.

def polling_cost(event_rate_per_hour, freshness_s, req_bytes=800):
    polls = 3600 / freshness_s
    useful = min(event_rate_per_hour, polls)
    waste = 1 - useful / polls
    return polls, waste, polls * req_bytes

scenarios = [
    ("chat message",      2,      1),     # rare events, need instant delivery
    ("stock price",    3600,      1),     # frequent events
    ("build status",      4,      5),     # rare events, tolerant freshness
    ("sensor reading",  720,     30),     # steady events, tolerant freshness
]
print(f"{'scenario':<16}{'polls/h':>9}{'waste':>9}{'bytes/h':>12}  verdict")
for name, rate, fresh in scenarios:
    polls, waste, bytes_ = polling_cost(rate, fresh)
    verdict = "STREAM" if waste > 0.9 else "poll is fine"
    print(f"{name:<16}{polls:>9,.0f}{waste:>8.1%}{bytes_:>12,.0f}  {verdict}")
# scenario          polls/h    waste     bytes/h  verdict
# chat message        3,600   99.9%   2,880,000  STREAM
# stock price         3,600    0.0%   2,880,000  poll is fine
# build status          720   99.4%     576,000  STREAM
# sensor reading        120    0.0%      96,000  poll is fine
#   The deciding factor is the RATIO of event rate to required freshness,
#   not whether the events feel "real-time".

# The hidden cost of streaming: state the server cannot forget.
def server_capacity(ram_gb, kb_per_connection, req_per_sec_stateless=50_000):
    conns = ram_gb * 1024 * 1024 / kb_per_connection
    return int(conns)

for kb in (10, 40, 120):
    print(f"{kb:>4} KB/conn -> {server_capacity(64, kb):>10,} concurrent streams "
          f"on a 64 GB box")
#   10 KB/conn ->  6,710,886 concurrent streams on a 64 GB box
#   40 KB/conn ->  1,677,721 concurrent streams on a 64 GB box
#  120 KB/conn ->    559,240 concurrent streams on a 64 GB box
#   A stateless request-response server has NO such ceiling --
#   it is limited by request rate, not by user count.

# Request-response: each exchange is complete and independent.
$ curl -s -o /dev/null -w 'status=%{http_code} reused=%{num_connects}\n' \
    https://api.example.com/items
# status=200 reused=1

# Server-Sent Events: one request, many responses, no client polling.
$ curl -N -H 'Accept: text/event-stream' https://api.example.com/events
# event: build
# data: {"id":91,"state":"running"}
#
# event: build
# data: {"id":91,"state":"passed"}
#   ^ connection stays open; the server writes whenever it has something.
#     Note the blank line: it is the event delimiter, not padding.

# WebSocket: the HTTP request that stops being HTTP.
$ curl -i -N -H 'Connection: Upgrade' -H 'Upgrade: websocket' \
       -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: x3JJHMbDL1EzLkh9GBhXDw==' \
       https://ws.example.com/socket
# HTTP/1.1 101 Switching Protocols
# Upgrade: websocket
# Connection: Upgrade
# Sec-WebSocket-Accept: HSmrc0sMlYUkAGmm5OPpG2HaGWk=
#   101 means: this TCP connection is no longer request-response.

# Count long-lived connections -- the resource streaming actually consumes:
$ ss -tn state established '( sport = :443 )' | wc -l
# 48211      <- each one is memory the server cannot release

Key takeaways

Request-response alternates strictly and is stateless, cacheable, load-balanced, and retryable — the default unless something forces otherwise.
Its limitation is that it is client-driven: the server cannot initiate, so push requires polling or a workaround.
Streaming keeps a channel open for server push and low per-message overhead, paying with per-connection server state, weak caching, and pinned load balancing.
Long polling and SSE are intermediate patterns that buy push over ordinary HTTP; WebSocket buys full bidirectionality via an HTTP 101 upgrade.
The decision is a ratio: compare event rate to required freshness — when polling waste exceeds roughly 90%, switch to a stream.
The second signal is who initiates: the moment the server must speak first, request-response needs a workaround.

🧪 Practice

A dashboard shows a value that changes about twice an hour and must be no more than 2 seconds stale. Compute the polling waste and recommend a pattern.

Explain why a cache can store a request-response exchange but not a WebSocket session.

Give one scenario where SSE is the right choice over WebSocket, and one where it is not, with reasons.

Interview-style: Your WebSocket service works perfectly in testing and falls over at 50,000 users, while your REST service handles ten times that traffic on the same hardware. (Hint: compare what each server must remember between one message and the next.)

8.2 Naming and the Domain Name System

DNS is the internet's directory: this subchapter covers how the namespace is organised, how a name is resolved, what can be stored in it, and how it was retrofitted with authenticity and privacy.

DNS Namespace Hierarchy

Theory

Humans remember names; routers forward on addresses. Something must translate, and that something has to work for billions of names, across every organisation on earth, with no single party in charge of the contents.

The first attempt was a single file. In the early ARPANET, every host copied HOSTS.TXT from one machine at SRI. It worked for a few hundred hosts and then failed for three reasons that any centralised naming scheme eventually hits: the file grew faster than distribution could keep up, every addition needed a human at a single organisation, and one machine's outage broke naming for everyone.

DNS's answer is hierarchical delegation. The namespace is a tree; authority over any subtree can be handed to someone else; and the only thing the parent keeps is a pointer to who now answers.

Read a name right to left, because that is the direction of authority:

        www.eng.example.co.uk.
        ^   ^   ^       ^  ^  ^
        |   |   |       |  |  +-- root (the empty label, usually invisible)
        |   |   |       |  +----- TLD: uk
        |   |   |       +-------- second level: co.uk
        |   |   +---------------- example.co.uk  <- an organisation's zone
        |   +-------------------- eng subdomain, possibly delegated
        +------------------------ host or service label

The terminology that people mix up, stated precisely:

A label is one component between dots, max 63 bytes.
A domain name is the full dotted sequence, max 255 bytes total.
A fully qualified domain name (FQDN) ends with the root dot — www.example.com. is fully qualified, www is not.
A domain is a subtree of the namespace: everything under example.com.
A zone is an administrative unit: the part of a domain a particular server is authoritative for, excluding any subtrees delegated away.

The domain/zone distinction is the one that matters operationally. example.com the domain includes eng.example.com. If the engineering department runs its own nameservers, then eng.example.com is a separate zone, and the example.com zone contains only a delegation pointing at it. One is about the shape of the tree; the other is about who answers questions.

Delegation is what makes the system scale administratively, not just technically. Nobody needs permission from ICANN to create api.example.com, because authority over everything below example.com was handed over once, when the domain was registered. The registry's job ends at storing a pointer.

The analogy that fits best is postal addressing, read backwards: country, then region, then city, then street, then house. Each level only needs to know enough to hand the letter to the next level down; no single office knows every address in the world.

Examples

The DNS tree: delegation is the only thing that crosses a boundary

                              . (root)
                              |
        +---------+-----------+-----------+---------+
        |         |           |           |         |
       com       org          uk         net       arpa
        |         |           |
   +----+----+   ...       +--+--+
   |         |             |     |
example   google          co    ac          <- uk uses a second level
   |                       |
   |                   example              <- example.co.uk
   +------+--------+
   |      |        |
  www    api      eng        <- eng may be DELEGATED to another server
                   |
              +----+----+
             www       git


  ZONE vs DOMAIN for example.com:

  domain example.com = everything in the subtree (incl. eng.example.com)

  zone example.com   = www, api, and a DELEGATION record for eng
                       (NOT the contents of eng -- that is a separate zone)

  +------------------- zone: example.com -------------------+
  |  www   A 93.184.216.34                                  |
  |  api   A 93.184.216.35                                  |
  |  eng   NS ns1.eng.example.com.   <-- the boundary        |
  +----------------------------|-----------------------------+
                               v
                    +--- zone: eng.example.com ---+
                    |  www  A 10.1.1.5            |
                    |  git  A 10.1.1.6            |
                    +-----------------------------+

# Parsing a name the way DNS does: right to left, by authority.

def analyse(name):
    fqdn = name if name.endswith('.') else name + '.'
    labels = fqdn.rstrip('.').split('.')
    print(f"{name}")
    print(f"  FQDN          : {fqdn}")
    print(f"  labels        : {labels}  ({len(labels)} labels)")
    print(f"  total bytes   : {len(fqdn)}  (limit 255)")
    print(f"  longest label : {max(len(l) for l in labels)}  (limit 63)")
    # Authority is discovered right to left -- print the delegation chain:
    chain = ['.'] + ['.'.join(labels[i:]) + '.' for i in range(len(labels)-1, -1, -1)]
    print(f"  lookup order  : {' -> '.join(chain)}")

analyse("www.eng.example.co.uk")
# www.eng.example.co.uk
#   FQDN          : www.eng.example.co.uk.
#   labels        : ['www', 'eng', 'example', 'co', 'uk']  (5 labels)
#   total bytes   : 22  (limit 255)
#   longest label : 7  (limit 63)
#   lookup order  : . -> uk. -> co.uk. -> example.co.uk. -> eng.example.co.uk. -> www.eng.example.co.uk.

# Why the limits exist: the wire format prefixes each label with its length.
def wire_format(name):
    out = b''
    for label in name.rstrip('.').split('.'):
        out += bytes([len(label)]) + label.encode()   # 1 length byte + label
    return out + b'\x00'                              # root = zero-length label

w = wire_format("www.example.com")
print(w)
# b'\x03www\x07example\x03com\x00'
#   ^ the length byte is 1 octet -> 63 is the max a 6-bit length can express
#     (the top 2 bits are reserved for compression pointers)
print(f"{len(w)} bytes on the wire")   # 17 bytes on the wire

# See the delegation boundary directly -- where one zone ends and another begins:
$ dig +norecurse @a.gtld-servers.net example.com NS
# ;; AUTHORITY SECTION:
# example.com.  172800  IN  NS  a.iana-servers.net.
# example.com.  172800  IN  NS  b.iana-servers.net.
#   ^ the .com zone holds only a POINTER. It has no idea what www.example.com is.

# Confirm which zone a name actually lives in:
$ dig +short example.com SOA
# ns.icann.org. noc.dns.icann.org. 2024081401 7200 3600 1209600 3600
#               ^ SOA marks the apex of a zone: the start of authority

# A name is not fully qualified until the trailing dot -- watch the difference:
$ dig +short www.example.com.search-domain.local
# (NXDOMAIN: the resolver appended your search domain to an unqualified name)
$ cat /etc/resolv.conf | grep search
# search corp.internal
#   ^ this is why "ping www" sometimes resolves and sometimes does not

Key takeaways

A single HOSTS.TXT failed on distribution volume, administrative bottleneck, and single point of failure — DNS solves all three with hierarchical delegation.
Names are read right to left, from the root toward the specific host, because that is the direction authority flows.
A label is max 63 bytes, a full name max 255 bytes, because the wire format prefixes each label with a one-octet length.
A domain is a subtree of the namespace; a zone is the part of it one server actually answers for, excluding delegated subtrees.
The parent zone stores only a delegation (NS record) — it knows who is authoritative, not what the answers are.
An FQDN ends in a dot; without it, resolvers may append search domains, which is a common source of surprising resolution behaviour.

🧪 Practice

For db.internal.corp.example.com, list the delegation chain from the root and identify which labels could plausibly be separate zones.

Explain the difference between the example.org domain and the example.org zone if dev.example.org is delegated elsewhere.

State the maximum label length and total name length, and explain what in the wire format imposes each.

Interview-style: Why can you create api.yourcompany.com instantly, but registering yourcompany.com requires a registrar? (Hint: ask which zone has to be edited in each case, and who holds the pen for it.)

Root, TLD, and Authoritative Servers

Theory

The hierarchy is only useful if someone answers at each level. DNS defines three kinds of server along the delegation chain, plus a fourth that does the walking.

Root servers are the entry point. They are authoritative for the root zone, which is tiny — roughly 1,500 records, containing nothing but delegations to the TLDs. Ask a root server for www.example.com and it will not know; it will tell you who runs .com.

There are 13 root server addresses, named a.root-servers.net through m.root-servers.net. This is the single most misunderstood fact in DNS: 13 is a count of addresses, not machines. The limit came from fitting the full root referral inside a 512-byte UDP DNS message. Each of those 13 addresses is announced by anycast (Chapter 6) from hundreds of physical sites worldwide, so there are well over 1,500 actual root server instances. Your packet reaches whichever one is topologically closest, and you cannot tell which.

TLD servers are authoritative for a top-level domain: .com, .org, .uk, .dev. They are run by registry operators (Verisign for .com, for instance) and, like the root, they mostly contain delegations rather than answers. The .com zone holds roughly 160 million delegations and almost no addresses.

TLDs come in categories worth knowing: gTLDs (generic: .com, .org, .info), ccTLDs (country code, two letters: .uk, .de, .jp), new gTLDs (post-2013: .dev, .app, .cloud), and infrastructure (.arpa, used for reverse lookups).

Authoritative servers are where answers actually live. They hold the zone file for a domain and give definitive responses — the AA (Authoritative Answer) bit set in the header. Every zone should have at least two, on separate networks, precisely so that one failure does not make the domain vanish. One is the primary (holds the editable master copy), the others are secondaries that pull the zone via zone transfer (AXFR for a full copy, IXFR for incremental), coordinated by the serial number in the SOA record.

Recursive resolvers are the fourth kind, and the one clients actually talk to. A resolver is not authoritative for anything; its job is to walk the chain on the client's behalf and cache the results. Your ISP runs one; so do public services such as 1.1.1.1, 8.8.8.8, and 9.9.9.9.

The division of labour is the point: root and TLD servers are referral machines that stay small and fast; authoritative servers hold the data; resolvers do the work and absorb the traffic through caching. Without resolver caches, the root servers would receive every lookup on earth.

Examples

Who answers what, and how little the top of the tree knows

  +-----------------------------------------------------------+
  | ROOT SERVERS   a-m.root-servers.net                        |
  | 13 ADDRESSES, 1500+ anycast instances                      |
  | knows: "com is over there"        ~1,500 records total     |
  +-----------------------------------------------------------+
                          | referral
                          v
  +-----------------------------------------------------------+
  | TLD SERVERS    a-m.gtld-servers.net (.com registry)        |
  | knows: "example.com is over there"  ~160,000,000 delegations|
  | does NOT know any host addresses                            |
  +-----------------------------------------------------------+
                          | referral
                          v
  +-----------------------------------------------------------+
  | AUTHORITATIVE   ns1.example.com, ns2.example.com           |
  | knows: "www.example.com is 93.184.216.34"   <- REAL ANSWER |
  | sets the AA bit. Primary + secondaries via zone transfer.  |
  +-----------------------------------------------------------+
                          ^
                          | does all the walking, caches everything
  +-----------------------------------------------------------+
  | RECURSIVE RESOLVER   1.1.1.1 / ISP / corporate             |
  | authoritative for NOTHING. Absorbs 90%+ of traffic via cache|
  +-----------------------------------------------------------+
                          ^
                          | one question, one answer
                       [ stub resolver on your machine ]


  Why 13? The original UDP DNS message limit:

   512 bytes  - header and question  = ~500 usable
   each root NS + glue A record      = ~30-40 bytes
   500 / ~38                         = 13 servers that fit in ONE packet

# Anycast: why "13 root servers" and "1500 root servers" are both true.

roots = {c: sites for c, sites in zip("abcdefghijklm",
         [16, 2, 6, 150, 350, 8, 12, 4, 200, 80, 300, 10, 180])}
print(f"{'letter':<8}{'address':<22}{'anycast sites':>14}")
for letter, sites in roots.items():
    print(f"{letter:<8}{letter + '.root-servers.net':<22}{sites:>14}")
print(f"\n{len(roots)} ADDRESSES, {sum(roots.values())} physical instances")
# letter  address                anycast sites
# a       a.root-servers.net                16
# b       b.root-servers.net                 2
# ...
# 13 ADDRESSES, 1318 physical instances
#   The 13 limit is a PROTOCOL artefact (512-byte UDP), not a capacity limit.

# Why root traffic stays manageable: caching collapses it.
def root_queries(lookups_per_day, cache_hit_rate, tld_ttl_hours=48):
    """Root is consulted only on a cache miss for the TLD referral."""
    misses = lookups_per_day * (1 - cache_hit_rate)
    # And even misses reuse the cached TLD referral for its TTL:
    return misses / (tld_ttl_hours * 3600 / 86400 * 100)

for hit in (0.0, 0.80, 0.95, 0.99):
    print(f"cache hit {hit:>5.0%} -> ~{root_queries(1e9, hit):>15,.0f} root queries/day")
# cache hit    0% -> ~     500,000,000 root queries/day
# cache hit   80% -> ~     100,000,000 root queries/day
# cache hit   95% -> ~      25,000,000 root queries/day
# cache hit   99% -> ~       5,000,000 root queries/day
#   Caching is not an optimisation here -- it is what makes the root survivable.

# Zone transfer coordination: the SOA serial decides whether to pull.
def should_transfer(secondary_serial, primary_serial):
    return primary_serial > secondary_serial     # serial arithmetic, RFC 1982

print(should_transfer(2024081401, 2024081402))   # True  -> AXFR/IXFR
print(should_transfer(2024081402, 2024081402))   # False -> already current

# Walk the chain by hand -- every referral is visible:
$ dig +norecurse @a.root-servers.net www.example.com
# ;; AUTHORITY SECTION:
# com.  172800  IN  NS  a.gtld-servers.net.
#   ^ a REFERRAL, not an answer. Note: no AA bit, empty ANSWER section.

$ dig +norecurse @a.gtld-servers.net www.example.com
# ;; AUTHORITY SECTION:
# example.com.  172800  IN  NS  a.iana-servers.net.
#   ^ still a referral. The .com registry does not know host addresses.

$ dig +norecurse @a.iana-servers.net www.example.com
# ;; flags: qr aa; ANSWER: 1
#              ^^ AUTHORITATIVE ANSWER
# www.example.com.  86400  IN  A  93.184.216.34

# dig can do the whole walk for you and show each step:
$ dig +trace www.example.com | grep -E 'NS|A\s' | head -8

# Which anycast instance answered? Ask the server itself:
$ dig +short CH TXT hostname.bind @k.root-servers.net
# "ns1.lon.k.ripe.net"        <- London instance today
$ dig +short CH TXT id.server @k.root-servers.net
# "sin.k.root-servers.org"    <- a different instance from another network

# Check a domain's authoritative servers and that they agree:
$ dig +short example.com NS
# a.iana-servers.net.
# b.iana-servers.net.
$ for ns in a.iana-servers.net b.iana-servers.net; do
>   echo -n "$ns: "; dig +short @$ns example.com SOA | awk '{print $3}'
> done
# a.iana-servers.net: 2024081401
# b.iana-servers.net: 2024081401     <- serials match: replication is healthy

Key takeaways

Root servers hold only the root zone (~1,500 records) and answer every query with a referral to a TLD.
"13 root servers" counts addresses, not machines — the limit came from fitting a referral in a 512-byte UDP message; anycast provides 1,500+ real instances.
TLD servers are run by registry operators and also mostly store delegations, not addresses.
Authoritative servers hold the zone data and set the AA bit; a primary holds the master copy and secondaries replicate by AXFR/IXFR, coordinated by the SOA serial.
Recursive resolvers are authoritative for nothing: they walk the chain, cache aggressively, and absorb the overwhelming majority of query traffic.
Caching is structural, not optional — without it, root and TLD infrastructure could not carry the load.

🧪 Practice

Explain precisely why there are 13 root server addresses and why that number does not limit capacity.

A domain's two authoritative servers return different SOA serials. State what has gone wrong and what you would check next.

Perform a manual delegation walk for a domain of your choice using dig +norecurse, and identify at which step the AA bit first appears.

Interview-style: If every recursive resolver on the internet lost its cache simultaneously, what would break first and why? (Hint: compare the query volume each tier is provisioned for against the volume it would suddenly see.)

Iterative vs Recursive Resolution

Theory

Given a hierarchy of servers, somebody has to walk it. The two ways of doing that are the source of endless confusion, mostly because the words describe who does the work, not what path the query takes.

Recursive resolution: the client asks one question and receives one final answer. "Tell me the address of www.example.com." The server takes full responsibility — it will chase referrals as far as needed and come back only when it has the answer or a definitive failure. The client does nothing but wait.

Iterative resolution: the server answers with the best it has. "I do not know, but ask a.gtld-servers.net." The asker must then repeat the question to the referred server, and again, until it reaches something authoritative. The work stays with the asker.

The real system uses both, in sequence, and this is the part worth internalising:

Stub resolver → recursive resolver: recursive. Your operating system's stub resolver is deliberately stupid. It sends one query with the RD (Recursion Desired) bit set and waits.
Recursive resolver → root/TLD/authoritative: iterative. The resolver sends queries with RD clear, collects referrals, and walks down the tree itself.

The asymmetry is deliberate. If every client walked the tree iteratively, the root and TLD servers would see every lookup from every device on earth, and no cache would accumulate anywhere useful. By concentrating the walking in a few thousand resolvers, the work is done once and the result serves everybody behind that resolver.

Note also why the servers refuse recursion. An authoritative server that performed recursion for strangers would be doing unbounded work on behalf of anyone who asked — and would be an open resolver, usable as a DNS amplification reflector. Authoritative servers therefore answer only for their own zones and set RA (Recursion Available) to zero.

A full cold lookup for www.example.com therefore costs four round trips:

stub -> resolver          (1 RTT, and the resolver then does:)
    resolver -> root      (referral to .com)
    resolver -> TLD       (referral to example.com)
    resolver -> auth      (the answer)

That is why DNS latency is so visible in page-load waterfalls, and why everything downstream — caching, prefetching, connection reuse — is designed to avoid paying it twice.

Two mechanisms make the walk cheaper than it looks. Glue records: when example.com's nameserver is ns1.example.com, resolving the nameserver requires resolving a name inside the zone it serves — circular. The parent zone therefore ships the nameserver's A record alongside the NS record. QNAME minimisation (RFC 9156): rather than sending the full name to every server, a modern resolver sends only as much as that server needs — it asks the root for com., not www.example.com. — which stops the root and TLD operators from seeing the full query stream.

Examples

Who does the work: recursive on the left, iterative on the right

  STUB -> RESOLVER (recursive)          RESOLVER -> SERVERS (iterative)
  "just give me the answer"             "I'll walk it myself"

   [stub]                                 [resolver]
      |                                       |--(1) com NS? ------> [root]
      |--- A? www.example.com --------->      |<---- referral -------|
      |    RD=1                               |
      |                                       |--(2) example.com NS?-> [TLD]
      |    (waits, one RTT from               |<---- referral -------|
      |     its point of view)                |
      |                                       |--(3) www.example.com A?-> [auth]
      |<-- 93.184.216.34 ---------------      |<---- ANSWER (AA=1) --|
                                              |
   one query, one answer                   three queries, all RD=0


  Why not let every client do this?

    1 billion devices x iterative walk = every lookup hits the root
    1 billion devices behind ~50,000 resolvers = root sees cache misses only

  Flags that encode the relationship:
    RD (Recursion Desired)   set by the ASKER: "do the work for me"
    RA (Recursion Available) set by the SERVER: "I am willing to"
    AA (Authoritative Answer) "this is from the zone itself, not a cache"

# A recursive resolver, written out: the loop IS the algorithm.

ROOT = ["198.41.0.4"]   # a.root-servers.net

ZONES = {   # a toy stand-in for the real servers
    "198.41.0.4":   {"type": "root", "delegations": {"com.": "192.5.6.30"}},
    "192.5.6.30":   {"type": "tld",  "delegations": {"example.com.": "199.43.135.53"}},
    "199.43.135.53":{"type": "auth", "records": {"www.example.com.": "93.184.216.34"}},
}

def query(server, qname):
    """One iterative query: an ANSWER or a REFERRAL, never both."""
    z = ZONES[server]
    if "records" in z and qname in z["records"]:
        return ("answer", z["records"][qname])
    for zone, ns in z.get("delegations", {}).items():
        if qname.endswith(zone):
            return ("referral", ns)
    return ("nxdomain", None)

def resolve(qname, trace=True):
    """The resolver's job: keep following referrals until authoritative."""
    server, hops = ROOT[0], 0
    while True:
        hops += 1
        kind, value = query(server, qname)
        if trace:
            print(f"  hop {hops}: asked {server:<15} -> {kind} {value or ''}")
        if kind == "answer":
            return value, hops
        if kind == "nxdomain":
            return None, hops
        server = value                       # follow the referral and repeat

print("Cold resolution of www.example.com:")
addr, hops = resolve("www.example.com.")
print(f"  result: {addr} after {hops} queries")
#   hop 1: asked 198.41.0.4      -> referral 192.5.6.30
#   hop 2: asked 192.5.6.30      -> referral 199.43.135.53
#   hop 3: asked 199.43.135.53   -> answer 93.184.216.34
#   result: 93.184.216.34 after 3 queries

# The latency this costs, and what caching removes:
def lookup_latency(rtt_stub=2, rtt_root=30, rtt_tld=25, rtt_auth=40, cached=False):
    return rtt_stub if cached else rtt_stub + rtt_root + rtt_tld + rtt_auth

print(f"\ncold:   {lookup_latency():>3} ms   (4 round trips)")
print(f"cached: {lookup_latency(cached=True):>3} ms   (1 round trip)")
# cold:    97 ms   (4 round trips)
# cached:   2 ms   (1 round trip)
#   ~50x. This is why DNS caching is the single highest-leverage web optimisation.

# QNAME minimisation: send only what each server needs to know.
def qname_min(fqdn):
    labels = fqdn.rstrip('.').split('.')
    return ['.'.join(labels[i:]) + '.' for i in range(len(labels)-1, -1, -1)]

for step, q in enumerate(qname_min("www.secret-project.example.com"), 1):
    print(f"  query {step} sends: {q}")
#   query 1 sends: com.
#   query 2 sends: example.com.
#   query 3 sends: secret-project.example.com.
#   query 4 sends: www.secret-project.example.com.
#   The root learns only "com." -- not that you are looking up secret-project.

# RD=1: ask a resolver to do the work (the normal case).
$ dig @1.1.1.1 www.example.com
# ;; flags: qr rd ra; ANSWER: 1
#              ^^ ^^ recursion desired AND available
# www.example.com. 3512 IN A 93.184.216.34
#                  ^^^^ note: NOT the zone's 86400 -- this is a cached remainder

# RD=0 against a resolver: "answer only from cache, do not go and look".
$ dig +norecurse @1.1.1.1 www.example.com
# ;; flags: qr ra; ANSWER: 1      <- present = it was cached
$ dig +norecurse @1.1.1.1 never-queried-before.example.com
# ;; flags: qr ra; ANSWER: 0      <- empty = not in cache, and it refused to fetch

# An authoritative server REFUSES recursion -- note the missing ra flag:
$ dig @a.iana-servers.net www.google.com
# ;; flags: qr; ANSWER: 0
# ;; WARNING: recursion requested but not available
#   Correct behaviour. A server that answered this would be an open resolver.

# Test whether something is an open resolver (a scan-worthy misconfiguration):
$ dig +short @<target> google.com A
# (an answer here from a non-public resolver = amplification risk)

# Watch the full iterative walk with timing:
$ dig +trace +stats www.example.com | tail -3
# ;; Query time: 97 msec

Key takeaways

Recursive means the server does the work and returns a final answer; iterative means the asker follows referrals itself.
The internet uses both: stub → resolver is recursive (RD=1), resolver → root/TLD/authoritative is iterative (RD=0).
Concentrating the walk in a few thousand resolvers keeps root/TLD load bounded and makes caching effective for many users at once.
Authoritative servers refuse recursion (RA=0); one that accepts it is an open resolver and an amplification risk.
A cold lookup costs about four round trips (stub, root, TLD, authoritative) — roughly 100 ms versus ~2 ms cached.
Glue records break the circular dependency of in-zone nameservers; QNAME minimisation stops upper-tier servers from seeing full query names.

🧪 Practice

State which party performs the work in recursive versus iterative resolution, and name which flag in the header requests it.

Explain why dig +norecurse @1.1.1.1 <name> returning an empty answer section is not an error.

Explain the circular dependency that glue records solve, using ns1.example.com serving example.com as the example.

Interview-style: What privacy property does QNAME minimisation provide, and what does it cost? (Hint: list exactly which name each tier of server sees with and without it, then count the queries.)

Resource Record Types

Theory

DNS is not an address book — it is a general-purpose distributed key-value store whose keys are domain names. Addresses are just its most common value type. Once you see it that way, the proliferation of record types stops looking arbitrary.

Every record, regardless of type, has the same five fields:

NAME        TTL     CLASS   TYPE    RDATA
www.example.com. 86400 IN   A       93.184.216.34
^the key      ^cache ^always ^what   ^the value, format depends on TYPE

CLASS is effectively always IN (Internet); the others are historical curiosities. The interesting part is TYPE, which determines how RDATA is interpreted.

The records that carry most of the internet:

Type	Purpose	RDATA example
A	IPv4 address	`93.184.216.34`
AAAA	IPv6 address	`2606:2800:220:1:248:1893:25c8:1946`
CNAME	Alias to another name	`target.example.com.`
MX	Mail exchanger, with priority	`10 mail.example.com.`
NS	Delegation to a nameserver	`ns1.example.com.`
TXT	Arbitrary text	`"v=spf1 include:_spf.example.com ~all"`
SOA	Zone apex metadata	serial, refresh, retry, expire, minimum
PTR	Reverse lookup (address → name)	`www.example.com.`
SRV	Service location: port and host	`10 60 5060 sip.example.com.`
CAA	Which CAs may issue certificates	`0 issue "letsencrypt.org"`
HTTPS/SVCB	Service parameters, ALPN, hints	`1 . alpn="h3,h2" ipv4hint=...`

Three of these have rules that trip people up constantly.

CNAME is exclusive and terminal. A name that has a CNAME may have no other records — no A, no MX, nothing. The reason is that a CNAME says "this name is that name"; the resolver restarts the lookup at the target, so any other record at this name would be unreachable and ambiguous. The direct consequence: you cannot put a CNAME at a zone apex, because the apex must carry SOA and NS records. This is why example.com cannot CNAME to a CDN while www.example.com can, and why providers invented non-standard workarounds (ALIAS, ANAME, CNAME flattening).

MX points to a name, never an address. MX 10 192.0.2.1 is invalid. The target must be a hostname with its own A/AAAA record, and that target must not be a CNAME. Lower priority numbers are preferred; equal values load-balance.

PTR lives in a special tree. Reverse lookups do not search; they use a constructed name in in-addr.arpa (IPv4) or ip6.arpa (IPv6), with the address written backwards so that the hierarchy runs from general to specific like every other DNS name. 93.184.216.34 becomes 34.216.184.93.in-addr.arpa. Delegation then follows network ownership, which is why your ISP, not you, controls the PTR for your home address.

SRV and HTTPS/SVCB are the modern additions that let DNS answer "where and how do I connect?" rather than just "what is the address?". SRV carries a port, which A records cannot; HTTPS/SVCB carries ALPN protocol hints, allowing a browser to know a site supports HTTP/3 before it connects, and to skip an entire round trip of upgrade negotiation.

Examples

A zone file: the whole key-value store in one page

$ORIGIN example.com.
$TTL 3600

@     IN SOA  ns1.example.com. admin.example.com. (
                 2024081401   ; serial  -- secondaries compare this
                 7200         ; refresh -- how often to check
                 3600         ; retry   -- if the check failed
                 1209600      ; expire  -- give up being authoritative
                 3600 )       ; minimum -- negative-caching TTL

@     IN NS   ns1.example.com.        ; the apex MUST have NS
@     IN NS   ns2.example.com.
@     IN A    93.184.216.34           ; apex can have A, never CNAME
@     IN MX   10 mail.example.com.    ; lower number = higher preference
@     IN MX   20 backup.example.com.
@     IN TXT  "v=spf1 mx -all"
@     IN CAA  0 issue "letsencrypt.org"

ns1   IN A    199.43.135.53           ; glue: in-zone nameserver
ns2   IN A    199.43.133.53
www   IN A    93.184.216.34
www   IN AAAA 2606:2800:220:1::1946
mail  IN A    93.184.216.40
shop  IN CNAME shops.cdn-provider.net.   ; alias -- NOTHING else at "shop"

_sip._tcp IN SRV 10 60 5060 sip.example.com.
;              ^  ^   ^    ^
;         priority weight port target


THE CNAME RULE, drawn:

  LEGAL                             ILLEGAL
  shop  CNAME  cdn.example.net.     shop  CNAME cdn.example.net.
  (nothing else at "shop")          shop  A     1.2.3.4       <- conflict
                                    shop  MX    10 mail...    <- conflict

  ILLEGAL AT APEX                   WHY
  @     CNAME  cdn.example.net.     the apex needs SOA + NS,
                                    and CNAME forbids siblings

# Validating the rules that actually cause outages.

def validate_zone(records):
    """records: list of (name, type, rdata)"""
    problems = []
    by_name = {}
    for name, rtype, rdata in records:
        by_name.setdefault(name, []).append((rtype, rdata))

    for name, entries in by_name.items():
        types = [t for t, _ in entries]
        # RULE 1: CNAME excludes all siblings.
        if "CNAME" in types and len(types) > 1:
            problems.append(f"{name}: CNAME coexists with {set(types) - {'CNAME'}}")
        # RULE 2: the apex cannot be a CNAME (it must hold SOA and NS).
        if name == "@" and "CNAME" in types:
            problems.append(f"{name}: CNAME at zone apex is forbidden")
        # RULE 3: MX must target a hostname, not an address literal.
        for t, data in entries:
            if t == "MX":
                target = data.split()[-1]
                if target.replace('.', '').isdigit():
                    problems.append(f"{name}: MX points at an address literal")
    return problems

zone = [
    ("@",    "SOA",   "ns1.example.com. admin.example.com. 2024081401 ..."),
    ("@",    "NS",    "ns1.example.com."),
    ("@",    "MX",    "10 mail.example.com."),
    ("www",  "A",     "93.184.216.34"),
    ("shop", "CNAME", "cdn-provider.net."),
    ("shop", "A",     "1.2.3.4"),                 # BUG: sibling of a CNAME
    ("mail2","MX",    "10 192.0.2.5"),            # BUG: address literal
]
for p in validate_zone(zone):
    print("ERROR:", p)
# ERROR: shop: CNAME coexists with {'A'}
# ERROR: mail2: MX points at an address literal

# Reverse DNS: the name is CONSTRUCTED, not searched.
def to_ptr_name(ip):
    if ':' in ip:                                  # IPv6 -> nibble by nibble
        import ipaddress
        full = ipaddress.IPv6Address(ip).exploded.replace(':', '')
        return '.'.join(reversed(full)) + '.ip6.arpa.'
    return '.'.join(reversed(ip.split('.'))) + '.in-addr.arpa.'

print(to_ptr_name("93.184.216.34"))
# 34.216.184.93.in-addr.arpa.
#   Reversed so the hierarchy runs general -> specific, like every other name:
#   93.in-addr.arpa is delegated to whoever owns 93.0.0.0/8.
print(to_ptr_name("2001:db8::1")[:40] + "...")
# 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0....

# SRV solves what A cannot: it carries a PORT.
def parse_srv(rdata):
    pri, weight, port, target = rdata.split()
    return {"priority": int(pri), "weight": int(weight),
            "port": int(port), "target": target}
print(parse_srv("10 60 5060 sip.example.com."))
# {'priority': 10, 'weight': 60, 'port': 5060, 'target': 'sip.example.com.'}
#   priority: try lowest first.  weight: relative share among equal priorities.

# Query specific types -- the store is only as useful as the key you ask for:
$ dig +short example.com MX
# 10 mail.example.com.
# 20 backup.example.com.

$ dig +short example.com TXT
# "v=spf1 -all"

$ dig +short _sip._tcp.example.com SRV
# 10 60 5060 sip.example.com.

# HTTPS/SVCB: the browser learns HTTP/3 support BEFORE connecting:
$ dig +short cloudflare.com HTTPS
# 1 . alpn="h3,h2" ipv4hint=104.16.132.229 ipv6hint=2606:4700::6810:84e5
#       ^^^^^^^^^^ saves a round trip of Alt-Svc discovery

# Reverse lookup -- note dig builds the in-addr.arpa name for you:
$ dig +short -x 93.184.216.34
# (often empty: forward and reverse are separate, independently owned zones)

# The CNAME chain a resolver has to follow -- each hop is a fresh lookup:
$ dig www.github.com | grep -A3 'ANSWER SECTION'
# www.github.com.  3600 IN CNAME github.com.
# github.com.      60   IN A     140.82.121.4
#   The resolver restarted resolution at the CNAME target.

# Check who may issue certificates for a domain:
$ dig +short example.com CAA
# 0 issue "letsencrypt.org"
#   A CA that honours CAA will refuse to issue if it is not listed here.

Key takeaways

DNS is a distributed key-value store keyed by domain name; every record is NAME TTL CLASS TYPE RDATA, and TYPE decides how RDATA is read.
A/AAAA give addresses, NS delegates, MX routes mail, SOA defines a zone apex, TXT carries policy strings, SRV/HTTPS carry connection parameters.
A CNAME excludes all sibling records, which is why a CNAME cannot exist at the zone apex — the apex must hold SOA and NS.
MX must point to a hostname, never an address literal, and that hostname must not itself be a CNAME; lower priority values win.
PTR records live in the constructed in-addr.arpa / ip6.arpa tree with the address reversed, and are delegated by address ownership, not domain ownership.
SRV carries a port and HTTPS/SVCB carries ALPN hints, letting DNS answer "how do I connect" and saving round trips before the first packet.

🧪 Practice

Write the in-addr.arpa name for 203.0.113.45 and explain why the octets are reversed.

A colleague wants example.com (the apex) to CNAME to cdn.provider.net. Explain why this is invalid and give two workable alternatives.

Write the records needed for a domain that serves web over IPv4 and IPv6, accepts mail at two servers with a preferred one, and publishes an SPF policy.

Interview-style: What can an SRV or HTTPS record do that an A record fundamentally cannot, and what round trip does that save? (Hint: think about everything a client must know before it can open a connection, and where it would otherwise learn it.)

DNS Caching and TTL

Theory

The resolution walk described above costs four round trips. If every lookup paid that, the web would be unusable and the root servers would melt. Caching is what makes DNS work, and the TTL is the single knob that controls it.

Every record carries a TTL in seconds: how long a cache may keep it before discarding it. The TTL is set by the zone owner, not the cache, which means an administrator can decide in advance how quickly a change propagates — and, unavoidably, is also deciding how much query load the authoritative servers will carry.

That is the fundamental trade-off, and it is genuinely a trade-off with no correct answer:

TTL	Propagation of changes	Authoritative load	Failure blast radius
Low (60 s)	Fast — minutes	High	Small, but auth outage hurts fast
High (86400 s)	Slow — up to a day	Low	Large; a bad record persists

Caching happens at every level, and the layers are why "it works on my machine" is so common in DNS troubleshooting: the browser has its own cache, the OS stub has one, the corporate resolver has one, and the upstream public resolver has another. A record can be simultaneously fresh in one and stale in another.

Two further behaviours are less obvious and cause a disproportionate share of real incidents.

Negative caching. A NXDOMAIN (name does not exist) is also cached, and its TTL comes from the minimum field of the SOA record, not from the query. This is why creating a record that someone just tried to look up can appear not to work: the absence was cached. RFC 2308 caps this at the SOA minimum or the SOA record's own TTL, whichever is smaller.

TTL countdown is visible. A resolver decrements the TTL it reports as the record ages in its cache. Query the same name twice and watch the number fall — if it does not, you are talking to an authoritative server, not a cache. This is a genuinely useful diagnostic.

The practical procedure for a planned migration follows directly from the model. Lower the TTL well in advance — at least one old-TTL period before the change, so that every cache has picked up the short value — then make the change, then raise the TTL again once traffic has moved. Lowering the TTL at the same time as making the change accomplishes nothing, because caches are still holding the old record under the old, long TTL.

The one thing TTL cannot control is a resolver that ignores it. Some public resolvers enforce a minimum TTL (discarding your 30-second value), some browsers cache for a fixed period regardless, and some applications resolve once at start-up and never again — a very common cause of "DNS changed but the app still connects to the old address".

Examples

The cache hierarchy: four places a stale answer can hide

  [browser cache]      ~60 s, ignores your TTL entirely (Chrome, Firefox)
        |
  [OS stub cache]      systemd-resolved / dnsmasq / nscd, honours TTL
        |
  [corporate resolver] honours TTL, shared by thousands of users
        |
  [public resolver]    1.1.1.1 -- may enforce a MINIMUM TTL
        |
  [authoritative]      the truth, TTL as published


TTL countdown in a cache (query the same name repeatedly):

  t=0s    dig -> 300   <- fetched fresh from authoritative
  t=45s   dig -> 255   <- the cache is counting down
  t=120s  dig -> 180
  t=300s  dig -> 300   <- expired and re-fetched; the cycle restarts
          ^ a DECREASING ttl proves you are reading a cache


Planned migration timeline (the order matters):

  day -2  lower TTL 86400 -> 60      | wait ONE FULL OLD TTL (24 h)
  day -1  ...caches expire old 86400 | every cache now holds ttl=60
  day  0  change the A record        | propagates in <= 60 s
  day  0  verify, watch traffic move |
  day +1  raise TTL back to 86400    | restore low query load

# A DNS cache: the whole mechanism in twenty lines.
import time

class DNSCache:
    def __init__(self, min_ttl=0):
        self.store = {}            # (name, type) -> (rdata, expiry)
        self.min_ttl = min_ttl     # some resolvers override short TTLs
        self.hits = self.misses = 0

    def put(self, name, rtype, rdata, ttl, now=None):
        now = now or time.time()
        ttl = max(ttl, self.min_ttl)              # resolver policy wins
        self.store[(name, rtype)] = (rdata, now + ttl)

    def get(self, name, rtype, now=None):
        now = now or time.time()
        entry = self.store.get((name, rtype))
        if not entry:
            self.misses += 1
            return None, None
        rdata, expiry = entry
        if now >= expiry:
            del self.store[(name, rtype)]         # expired: must re-query
            self.misses += 1
            return None, None
        self.hits += 1
        return rdata, int(expiry - now)           # the COUNTING-DOWN ttl

c = DNSCache()
t0 = 1_000_000
c.put("www.example.com", "A", "93.184.216.34", 300, now=t0)
for offset in (0, 45, 120, 299, 301):
    rdata, ttl = c.get("www.example.com", "A", now=t0 + offset)
    print(f"t={offset:>4}s  -> {rdata or 'MISS (re-query authoritative)':<20} "
          f"ttl={ttl if ttl is not None else '-'}")
# t=   0s  -> 93.184.216.34        ttl=300
# t=  45s  -> 93.184.216.34        ttl=255
# t= 120s  -> 93.184.216.34        ttl=180
# t= 299s  -> 93.184.216.34        ttl=1
# t= 301s  -> MISS (re-query authoritative) ttl=-

# The TTL trade-off, quantified.
def auth_qps(clients, lookups_per_client_per_hour, ttl_s, resolvers=5000):
    """Each RESOLVER must refresh once per TTL; clients behind it are free."""
    refreshes_per_hour = 3600 / ttl_s
    return resolvers * refreshes_per_hour / 3600

print(f"\n{'TTL':>8}{'auth qps':>12}{'worst-case staleness':>24}")
for ttl in (60, 300, 3600, 86400):
    print(f"{ttl:>8}{auth_qps(1e6, 20, ttl):>12,.1f}{ttl/60:>20,.0f} min")
#      TTL    auth qps    worst-case staleness
#       60     83.3                   1 min
#      300     16.7                   5 min
#     3600      1.4                  60 min
#    86400      0.1               1,440 min
#   Every factor-of-10 in agility costs a factor-of-10 in query load.

# Negative caching: absence is cached too, using the SOA MINIMUM.
soa = {"serial": 2024081401, "refresh": 7200, "retry": 3600,
       "expire": 1209600, "minimum": 3600}
print(f"\nNXDOMAIN will be cached for {soa['minimum']}s "
      f"({soa['minimum']//60} min)")
# NXDOMAIN will be cached for 3600s (60 min)
#   Create the record now and it may still 'not exist' for an hour.

# Prove you are reading a cache: the TTL counts down.
$ dig +short +ttlid @1.1.1.1 www.example.com A | head -1; sleep 5
$ dig @1.1.1.1 www.example.com | grep -E '^www'
# www.example.com.  3487  IN  A  93.184.216.34
$ sleep 10; dig @1.1.1.1 www.example.com | grep -E '^www'
# www.example.com.  3477  IN  A  93.184.216.34
#                   ^^^^ decreasing => cached. Constant => authoritative.

# The authoritative value, bypassing every cache:
$ dig +norecurse @a.iana-servers.net www.example.com | grep -E '^www'
# www.example.com.  86400  IN  A  93.184.216.34
#                   ^^^^^ the PUBLISHED ttl, never decrements

# Flush the local caches while debugging (Linux / systemd):
$ sudo resolvectl flush-caches
$ resolvectl statistics | grep -A3 Cache
# Cache
#   Current Cache Size: 0
#   Cache Hits: 18422
#   Cache Misses: 1290          <- 93% hit rate: normal and healthy

# Browsers keep their own cache that ignores your TTL:
#   chrome://net-internals/#dns  ->  "Clear host cache"

# Check negative caching before panicking about a new record:
$ dig example.com SOA +short | awk '{print "negative TTL:", $NF"s"}'
# negative TTL: 3600s

Key takeaways

TTL is set by the zone owner and decides both how fast changes propagate and how much load authoritative servers carry — a direct, unavoidable trade-off.
Caches exist at four or more layers (browser, OS stub, corporate resolver, public resolver), so a record can be fresh and stale at the same time.
Negative responses are cached too, with a TTL taken from the SOA minimum — which is why a newly created record can appear missing for an hour.
A decreasing TTL across repeated queries proves you are reading a cache; a constant TTL means you reached an authoritative server.
Migration procedure: lower the TTL one full old-TTL period in advance, then change, then raise it back — lowering it at change time accomplishes nothing.
TTL is advisory in practice: resolvers may enforce minimums and applications that resolve once at start-up ignore it entirely.

🧪 Practice

A record has TTL 86400 and you must migrate it on Friday. Write the timeline of actions starting from Wednesday, with reasons for each step.

You create a new record and it still returns NXDOMAIN ten minutes later. Explain the most likely cause and how you would confirm it.

Using the load model above, compute authoritative query rate for TTL 30, 600, and 7200 with 20,000 resolvers, and recommend a value for a record that changes weekly.

Interview-style: An application keeps connecting to a decommissioned server hours after DNS was updated and every cache shows the new value. (Hint: DNS caching is not the only place a resolved address can be stored.)

DNSSEC Fundamentals

Theory

DNS was designed in 1983 with no security whatsoever. A response is accepted if it arrives at the right port with the right transaction ID — which means anyone who can forge a UDP packet, or win a race against the real server, can put whatever they like in your cache. Cache poisoning turns that into a persistent, cache-wide redirection affecting every user behind the resolver.

DNSSEC's answer is not encryption. This is the point people most often get wrong: DNSSEC provides authenticity and integrity, not confidentiality. DNSSEC-signed responses are still fully readable by anyone on the path. What DNSSEC guarantees is that the data came from the zone owner and was not modified.

It works by signing records with public-key cryptography, then chaining the trust the same way the namespace itself is chained.

The record types involved:

Record	Role
DNSKEY	The zone's public keys — a KSK (key-signing key) and a ZSK (zone-signing key)
RRSIG	A signature over a record set, produced with the ZSK
DS	A hash of the child's KSK, published in the parent zone
NSEC / NSEC3	Authenticated proof that a name does not exist

The chain of trust runs downward and mirrors delegation exactly:

The resolver starts with the root's KSK, configured out of band — the trust anchor, the one key that must be known in advance.
The root zone's DS record for com is signed by the root's keys, so the resolver learns com's KSK hash.
com's DNSKEY is validated against that DS; com's DS record for example.com is signed by com's ZSK.
example.com's DNSKEY validates against that DS, and the A record's RRSIG validates against example.com's ZSK.

Every link is verified, so the answer is trustworthy end to end — provided every zone on the path is signed. A single unsigned zone breaks the chain, and the result is insecure, not invalid: DNSSEC degrades to plain DNS rather than failing.

Proving non-existence is the genuinely clever part. You cannot sign an answer you did not precompute, so how do you sign "this name does not exist" without an online key? NSEC answers with the next name in sorted order: "there is nothing between alpha.example.com and gamma.example.com", which is a precomputed, signable statement that proves beta.example.com is absent. The side effect is zone walking — an attacker can enumerate every name in the zone by following NSEC records. NSEC3 fixes this by sorting and proving over hashed names, at a cost in CPU.

DNSSEC's practical difficulties are why adoption stalled around 30–40% of queries: key rollovers are error-prone, an expired RRSIG takes the domain completely offline (signatures have an explicit validity window, unlike TTLs), responses grow large enough to need EDNS0 and TCP fallback, and the DS record must be handed to the registrar through a separate channel.

Examples

The chain of trust: every link verified, rooted in one known key

  [ TRUST ANCHOR: root KSK ]  <- configured out of band, the ONLY pre-shared key
            |
            | root DNSKEY (validated against the anchor)
            | root ZSK signs ->  DS record for "com"
            v
  +-------------------------+
  | . (root zone)           |   DS(com) = hash of com's KSK
  +-------------------------+
            | verify: does com's DNSKEY hash to the DS above?  YES
            v
  +-------------------------+
  | com                     |   DS(example.com) = hash of example.com's KSK
  +-------------------------+
            | verify: does example.com's DNSKEY hash to that DS?  YES
            v
  +-------------------------+
  | example.com             |   RRSIG(www A) signed by example.com's ZSK
  +-------------------------+
            | verify signature over the A record set:  VALID
            v
      www.example.com A 93.184.216.34   <- AD bit set: authenticated

  Break ANY link -> "insecure" (falls back to plain DNS), not "invalid".
  Bad signature   -> SERVFAIL: the resolver refuses to return the answer.


NSEC: proving a name does NOT exist, without an online key

  zone contents, sorted:   alpha  --  gamma  --  omega

  query: beta.example.com?
  answer: NSEC alpha.example.com. gamma.example.com.
               ^ "nothing exists between these two"
          + RRSIG over that NSEC   <- precomputed and signable

  Side effect: follow the NSEC chain and you have enumerated the whole zone.
  NSEC3 signs over HASHED names instead, so the chain reveals only hashes.

# The chain of trust as code: each level validates the next.
import hashlib, hmac

def keypair(zone):
    """Stand-in for real asymmetric keys -- the STRUCTURE is what matters."""
    priv = hashlib.sha256(f"private:{zone}".encode()).digest()
    pub  = hashlib.sha256(priv).hexdigest()[:16]
    return priv, pub

def sign(priv, data):
    return hmac.new(priv, data.encode(), hashlib.sha256).hexdigest()[:16]

def verify(priv, data, sig):
    return hmac.compare_digest(sign(priv, data), sig)

def ds_record(pub_ksk):
    """The DS published in the PARENT is a hash of the child's KSK."""
    return hashlib.sha256(pub_ksk.encode()).hexdigest()[:16]

# Each zone has its own keys.
zones = {z: keypair(z) for z in (".", "com", "example.com")}

# The parent publishes a DS for each child.
parent_ds = {
    "com":         ds_record(zones["com"][1]),
    "example.com": ds_record(zones["example.com"][1]),
}

def validate_chain(anchor_pub, answer, answer_sig):
    steps = []
    # 1. Trust anchor: the root key we already know.
    steps.append(("root anchor", anchor_pub == zones["."][1]))
    # 2. Root's DS for com must match com's actual KSK.
    steps.append(("root -> com DS", parent_ds["com"] == ds_record(zones["com"][1])))
    # 3. com's DS for example.com must match.
    steps.append(("com -> example.com DS",
                  parent_ds["example.com"] == ds_record(zones["example.com"][1])))
    # 4. The RRSIG over the answer must verify with example.com's key.
    steps.append(("RRSIG over A record",
                  verify(zones["example.com"][0], answer, answer_sig)))
    return steps

answer = "www.example.com. 86400 IN A 93.184.216.34"
sig = sign(zones["example.com"][0], answer)

print("Legitimate answer:")
for name, ok in validate_chain(zones["."][1], answer, sig):
    print(f"  {name:<24} {'OK' if ok else 'FAIL'}")
# Legitimate answer:
#   root anchor              OK
#   root -> com DS           OK
#   com -> example.com DS    OK
#   RRSIG over A record      OK
#   -> AD bit set, answer returned

forged = "www.example.com. 86400 IN A 6.6.6.6"        # attacker's address
print("\nForged answer (cache-poisoning attempt):")
for name, ok in validate_chain(zones["."][1], forged, sig):
    print(f"  {name:<24} {'OK' if ok else 'FAIL'}")
# Forged answer (cache-poisoning attempt):
#   root anchor              OK
#   root -> com DS           OK
#   com -> example.com DS    OK
#   RRSIG over A record      FAIL      <- resolver returns SERVFAIL, not the lie

# Signature validity windows are NOT TTLs -- expiry takes the domain OFFLINE.
from datetime import datetime, timedelta
inception = datetime(2026, 9, 1)
expiration = inception + timedelta(days=30)
for day in (5, 29, 31):
    now = inception + timedelta(days=day)
    valid = inception <= now <= expiration
    print(f"day {day:>2}: {'valid' if valid else 'EXPIRED -> whole zone SERVFAIL'}")
# day  5: valid
# day 29: valid
# day 31: EXPIRED -> whole zone SERVFAIL
#   An expired RRSIG is worse than no DNSSEC: the domain disappears entirely.

# Is a response authenticated? Look for the AD (Authenticated Data) flag:
$ dig +dnssec @1.1.1.1 cloudflare.com A | grep -E 'flags|RRSIG'
# ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2
#                    ^^ AD = the resolver validated the chain
# cloudflare.com. 300 IN RRSIG A 13 2 300 20260930... (signature)

# A deliberately broken signature -- note SERVFAIL, not a wrong answer:
$ dig @1.1.1.1 dnssec-failed.org A
# ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL
#   Validation failure is FAIL-CLOSED: no answer is better than a forged one.

$ dig +cd @1.1.1.1 dnssec-failed.org A   # +cd = checking disabled
# ;; status: NOERROR
# dnssec-failed.org. 7200 IN A 69.252.80.75
#   +cd proves the failure was validation, not a missing record.

# Inspect the chain link by link:
$ dig +short example.com DS          # published in the PARENT (.com)
# 31589 8 2 CDE0D742D6998AA554A92D890F8184C698CFAC8A26FA59875A990C03E576343C

$ dig +short example.com DNSKEY | head -2
# 256 3 8 AwEAAb...   <- ZSK (flags 256): signs the records
# 257 3 8 AwEAAa...   <- KSK (flags 257): signs the DNSKEY set, hashed into DS

# Full validation trace, showing each link being checked:
$ delv +vtrace example.com A 2>&1 | grep -E 'validating|fully'
# ; fully validated
#   example.com. 86400 IN A 93.184.216.34

# DNSSEC responses are big -- check EDNS0 buffer negotiation:
$ dig +dnssec +bufsize=1232 example.com DNSKEY | grep -E 'MSG SIZE|EDNS'
# ; EDNS: version: 0, flags: do; udp: 1232
# ;; MSG SIZE  rcvd: 1097        <- would not fit in the old 512-byte limit

Key takeaways

DNSSEC provides authenticity and integrity, never confidentiality — signed responses are still plaintext on the wire.
The record set is DNSKEY (public keys: KSK and ZSK), RRSIG (signatures), DS (child key hash published in the parent), and NSEC/NSEC3 (proof of non-existence).
The chain of trust mirrors delegation, rooted in the root KSK trust anchor — the only key configured out of band.
A validation failure is fail-closed (SERVFAIL); an unsigned zone merely yields an insecure result, so DNSSEC degrades gracefully but protects strictly.
NSEC proves non-existence by naming the sorted neighbours, which permits zone walking; NSEC3 hashes names to prevent enumeration.
Operationally the hazards are expired RRSIGs (which take the whole zone offline), key rollovers, larger responses needing EDNS0/TCP, and out-of-band DS publication at the registrar.

🧪 Practice

Explain why DNSSEC does not protect the privacy of your queries, and name the technology in the next topic that does.

Trace the DS/DNSKEY validation steps from the root to www.example.com and state what a resolver does if com were unsigned.

Describe how NSEC proves a name does not exist, the enumeration weakness this creates, and how NSEC3 addresses it.

Interview-style: Argue that a misconfigured DNSSEC deployment is more dangerous to availability than no DNSSEC at all. (Hint: compare the failure mode of a wrong answer with the failure mode of a signature that expired at 3 a.m.)

DNS over TLS and DNS over HTTPS

Theory

DNSSEC closed the authenticity gap and left the other one wide open. Classic DNS travels over UDP port 53 in cleartext, so every device on the path — your ISP, the café Wi-Fi, a transit provider — can read every name you look up, and can block or rewrite the answers.

That leak is more revealing than people assume. Even with HTTPS protecting page contents, the sequence of names you resolve reconstructs your browsing history: which sites, in what order, at what time. DNS was, for most of the web's history, the last plaintext record of user behaviour.

Two encrypted transports emerged, differing mainly in how much they try to blend in.

DNS over TLS (DoT), RFC 7858, is the straightforward answer: ordinary DNS messages inside a TLS session on port 853. Clean, purpose-built, easy to implement in a resolver. Because it has a dedicated port, it is trivially identifiable on the network — which is good for network operators who want to audit or enforce policy, and bad for users in networks that would rather block it.

DNS over HTTPS (DoH), RFC 8484, wraps DNS queries in HTTPS requests on port 443, indistinguishable from any other web traffic. A query is an HTTP GET with a base64url-encoded DNS message, or a POST with the message as the body, using content type application/dns-message. Because it is indistinguishable from web traffic, it cannot be selectively blocked without blocking HTTPS — a property its advocates consider the point and its critics consider the problem.

	Classic DNS	DoT	DoH
Port	53	853	443
Encryption	None	TLS	TLS (inside HTTP/2 or HTTP/3)
Blockable selectively	Yes	Yes (port)	No (looks like web traffic)
Network visibility	Full	Metadata only	None
Implemented in	OS stub	OS stub	Often the browser
Overhead	~1 RTT	+TLS handshake	+TLS +HTTP framing

DoQ (DNS over QUIC), RFC 9250, is the newer addition on port 853/UDP, giving DoT's encryption without TCP head-of-line blocking — a natural fit since DNS was always a datagram protocol.

The genuine controversy is not cryptographic; it is about where trust moves. Encrypting DNS does not remove the observer — it relocates it. Your ISP can no longer see your lookups; the DoH provider can see all of them, now conveniently centralised across millions of users. If a browser ships with a default DoH provider, that provider gains a view of user behaviour that no single ISP ever had.

The operational objections are concrete and worth taking seriously:

Split-horizon DNS breaks. Corporate names that resolve only on the internal resolver stop resolving when the browser bypasses it.
Security filtering is bypassed. Malware-blocking and parental controls implemented at the resolver stop applying.
Troubleshooting gets harder. The OS no longer knows what the browser resolved.

The mitigations are policy-shaped rather than technical: Canary domains (use-application-dns.net, which a network answers NXDOMAIN to signal "do not use DoH here"), enterprise policy that disables application-level DoH, and the emerging Discovery of Designated Resolvers (DDR) for advertising a network's own encrypted resolver so clients can encrypt to the right place rather than bypassing it.

The realistic summary: encrypted DNS is strictly better than plaintext against on-path observers, and the remaining argument is entirely about which single party you would rather have the complete log.

Examples

What each transport exposes to whom

  CLASSIC DNS (port 53, plaintext)
  [you] --"www.medical-site.com?"--> [ISP] --> [root/TLD/auth]
              ^ visible to: ISP, Wi-Fi operator, every transit hop,
                and anyone running tcpdump on the path

  DoT (port 853, TLS)
  [you] ===encrypted===> [resolver] --> [root/TLD/auth]
         ^ ISP sees: "TLS to 1.1.1.1:853, 240 bytes"
           It knows you are doing DNS, and with whom, not WHAT.
           Selectively blockable: just drop port 853.

  DoH (port 443, HTTPS)
  [you] ===encrypted===> [resolver] --> [root/TLD/auth]
         ^ ISP sees: "HTTPS to 1.1.1.1, 240 bytes"
           Indistinguishable from any web request.
           To block it, you must block HTTPS to that address.

  IN ALL THREE CASES the resolver still sees everything,
  and the authoritative servers still see the queries they answer.
  Encryption relocates the observer; it does not remove one.

# Building a DoH query by hand -- it is genuinely just HTTP.
import base64, struct

def build_dns_query(name, qtype=1):
    """A wire-format DNS query: identical bytes for UDP/53, DoT, and DoH."""
    header = struct.pack('>HHHHHH',
                         0x0000,   # ID: MUST be 0 for DoH (HTTP handles matching)
                         0x0100,   # flags: RD=1
                         1, 0, 0, 0)
    qname = b''.join(bytes([len(l)]) + l.encode() for l in name.split('.')) + b'\x00'
    return header + qname + struct.pack('>HH', qtype, 1)   # QTYPE, QCLASS=IN

q = build_dns_query("www.example.com")
print(q)
# b'\x00\x00\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x03www\x07example\x03com\x00\x00\x01\x00\x01'

# GET form: base64url-encoded, unpadded, in the ?dns= parameter.
encoded = base64.urlsafe_b64encode(q).decode().rstrip('=')
print(f"\nGET https://cloudflare-dns.com/dns-query?dns={encoded}")
# GET https://cloudflare-dns.com/dns-query?dns=AAABAAABAAAAAAAAA3d3dwdleGFtcGxlA2NvbQAAAQAB
#   Note the stripped '=' padding: required by RFC 8484.
#   Note ID=0: DoH must zero it so identical queries share an HTTP cache entry.

# What each observer learns, side by side.
observers = {
    "classic/53": {"ISP": "full query names", "resolver": "full query names",
                   "cafe wifi": "full query names"},
    "DoT/853":    {"ISP": "you use DNS, and which resolver", "resolver": "full query names",
                   "cafe wifi": "you use DNS, and which resolver"},
    "DoH/443":    {"ISP": "you contacted an IP over HTTPS", "resolver": "full query names",
                   "cafe wifi": "you contacted an IP over HTTPS"},
}
for transport, who in observers.items():
    print(f"\n{transport}")
    for party, sees in who.items():
        print(f"   {party:<12} sees: {sees}")
# classic/53
#    ISP          sees: full query names
#    resolver     sees: full query names
#    cafe wifi    sees: full query names
# DoT/853
#    ISP          sees: you use DNS, and which resolver
#    resolver     sees: full query names
#    cafe wifi    sees: you use DNS, and which resolver
# DoH/443
#    ISP          sees: you contacted an IP over HTTPS
#    resolver     sees: full query names
#    cafe wifi    sees: you contacted an IP over HTTPS
#   The resolver column NEVER changes. That is the whole debate.

# Connection setup cost -- encrypted DNS is only expensive when cold.
costs = {"classic UDP": 1, "DoT (cold, TLS1.3)": 3, "DoT (warm, reused)": 1,
         "DoH/2 (cold)": 3, "DoH/2 (warm)": 1, "DoQ (cold, 0-RTT)": 1}
for name, rtts in costs.items():
    print(f"{name:<22} {rtts} RTT{'s' if rtts > 1 else ''}")
#   Warm connections are the common case: the cost is amortised to nearly zero.

# DoH via curl -- it is an ordinary HTTPS request with a special content type:
$ curl -s -H 'accept: application/dns-json' \
    'https://cloudflare-dns.com/dns-query?name=example.com&type=A' | jq -c '.Answer'
# [{"name":"example.com","type":1,"TTL":85652,"data":"93.184.216.34"}]

# The RFC 8484 binary form (what browsers actually send):
$ curl -s -H 'accept: application/dns-message' \
    'https://cloudflare-dns.com/dns-query?dns=AAABAAABAAAAAAAAA3d3dwdleGFtcGxlA2NvbQAAAQAB' \
    | xxd | head -2
# 00000000: 0000 8180 0001 0001 0000 0000 0377 7777  .............www

# DoT with a purpose-built client -- note the port and the certificate check:
$ kdig -d @1.1.1.1 +tls-ca +tls-host=cloudflare-dns.com example.com
# ;; DEBUG: TLS, peer certificate verified
# ;; DEBUG: TLS, handshake completed
# example.com.  85000  IN  A  93.184.216.34

# Configure the OS stub to use DoT system-wide (systemd-resolved):
$ resolvectl status | grep -E 'DNS Servers|DNSOverTLS'
# DNS Servers: 1.1.1.1 1.0.0.1
# +DNSOverTLS  +DNSSEC/supported          <- both on

# What an observer actually sees, per transport:
$ sudo tcpdump -i any -n port 53 -c 1 -A 2>/dev/null | grep -o 'example\.com'
# example.com          <- plaintext: the name is right there

$ sudo tcpdump -i any -n port 853 -c 1 -A 2>/dev/null | grep -o 'example\.com'
# (nothing: encrypted)

# The canary domain: how a network says "please do not use DoH here".
$ dig +short use-application-dns.net
# (an NXDOMAIN here signals Firefox to disable DoH on this network)

Key takeaways

Classic DNS is cleartext on UDP/53, so the sequence of names you resolve is readable by every on-path party — a browsing history that HTTPS does not hide.
DoT (port 853) puts DNS inside TLS: clean and purpose-built, but identifiable and therefore blockable by port.
DoH (port 443) puts DNS inside HTTPS: indistinguishable from web traffic, which is simultaneously its main feature and the core objection to it.
DoQ (RFC 9250) adds QUIC, removing TCP head-of-line blocking for a protocol that was always datagram-shaped.
Encryption relocates the observer rather than removing it: the resolver still sees every query, now centralised across far more users.
Real operational costs are broken split-horizon DNS, bypassed security filtering, and harder troubleshooting; mitigations include canary domains, enterprise policy, and DDR.

🧪 Practice

Build the table of what the ISP, the café Wi-Fi, and the resolver each learn under classic DNS, DoT, and DoH.

Explain why an enterprise might deliberately block port 853 but find DoH much harder to control.

A company's internal hostnames stop resolving after a browser update. Explain the likely cause and two possible remedies.

Interview-style: Does encrypted DNS improve a user's privacy, or merely change who is watching? (Hint: draw the path, mark every party that can read the query names before and after, and note which column of your table never changed.)

8.3 The Web and HTTP

HTTP carries more of the internet's traffic than any other application protocol: this subchapter builds it from the URL up through three major protocol versions and the push mechanisms layered on top.

URLs, URIs, and Resources

Theory

Before a protocol can fetch something, there must be a way to name it. The web's naming scheme is deliberately more general than "a file on a server", and understanding that generality explains a lot of otherwise puzzling design.

The vocabulary, from broadest to narrowest:

A URI (Uniform Resource Identifier) identifies a resource. That is all it promises.
A URL (Uniform Resource Locator) is a URI that also says how to locate it — it includes a scheme and enough information to access the thing.
A URN (Uniform Resource Name) is a URI that names without locating: urn:isbn:0140449132 identifies a book, but tells you nothing about where to get it.

In practice almost every URI you meet is a URL, and the modern WHATWG specification largely collapses the distinction. The concept worth keeping is resource: the abstract thing being identified, which is not the same as the bytes you get back. /weather/london is a resource; its representation changes every hour, and may be JSON or HTML depending on what you asked for. This separation is the foundation of content negotiation, caching, and REST.

The anatomy of a URL:

  https://user@www.example.com:8443/path/to/page?q=networking&p=2#section-3
  \___/   \__/ \_____________/ \__/\___________/ \______________/ \_______/
  scheme  user     host        port    path          query        fragment

Each component has behaviour worth knowing:

Scheme — the protocol, and it determines everything else's meaning. Case-insensitive.
Host — a domain name or address literal; case-insensitive, and IPv6 literals must be bracketed: http://[2001:db8::1]:80/.
Port — omitted means the scheme's default (80 for http, 443 for https).
Path — case-sensitive, hierarchical, and entirely up to the server to interpret. It need not correspond to a file.
Query — after ?, conventionally key=value pairs joined by &. Sent to the server.
Fragment — after #, never sent to the server. It is processed by the client alone, which is why single-page applications historically used it for routing and why analytics cannot see it server-side.

Percent-encoding is the part that causes real bugs. URLs may contain only a restricted ASCII set, so anything else — spaces, non-Latin characters, or a reserved character used literally — must be encoded as %XX. The trap is that the reserved set differs by component: / is legal and meaningful in a path but must be encoded inside a query value; + means a space in a form-encoded query but is a literal plus in a path. Encoding a whole URL with one function is a classic source of broken links, which is why JavaScript has both encodeURI (leaves structure alone) and encodeURIComponent (encodes everything).

Relative references resolve against a base URL by well-defined rules (RFC 3986): //host/path keeps the scheme, /path keeps scheme and host, path replaces the last path segment, and ?query keeps everything but the query.

Examples

URL anatomy and who sees what

  https://api.example.com:8443/v1/users/42?fields=name,email#bio
  \___/   \______________/\__/\___________/\_______________/\__/
    |            |          |        |             |          |
  scheme       host       port     path         query     fragment
    |            |          |        |             |          |
  chooses    resolved    default   opaque to    opaque to   NEVER
  transport  via DNS     if absent  the client  the client   SENT
  + default  case-       (80/443)  case-        (server      (client
    port     INSENSITIVE           SENSITIVE     parses)      only)

  What actually goes on the wire:

    GET /v1/users/42?fields=name,email HTTP/1.1
    Host: api.example.com
    ^ path + query only. The fragment "#bio" stayed in the browser.


  Same resource, different representations:

                    /weather/london          <- the RESOURCE (stable identity)
                     /        |      \
        Accept: json/    Accept: html   Accept-Language: fr
             |               |               |
      {"tempC":14}     <html>14C</html>  {"tempC":14,"desc":"nuageux"}
       ^-------------- three REPRESENTATIONS of one resource -----------^

from urllib.parse import urlparse, urlencode, quote, quote_plus, urljoin, parse_qs

url = "https://user@www.example.com:8443/path/to/page?q=networking&p=2#section-3"
p = urlparse(url)
for field in ("scheme", "username", "hostname", "port", "path", "query", "fragment"):
    print(f"  {field:<9}: {getattr(p, field)}")
#   scheme   : https
#   username : user
#   hostname : www.example.com      <- note: lowercased, host is case-insensitive
#   port     : 8443
#   path     : /path/to/page        <- NOT lowercased: paths are case-sensitive
#   query    : q=networking&p=2
#   fragment : section-3            <- parsed here, but never transmitted

# The encoding trap: the safe set DEPENDS ON THE COMPONENT.
value = "a/b c&d=e"
print(f"\nraw value           : {value}")
print(f"as a PATH segment   : {quote(value, safe='/')}")      # '/' kept structural
print(f"as a QUERY value    : {quote(value, safe='')}")       # everything encoded
print(f"as FORM data        : {quote_plus(value)}")           # space -> '+'
# raw value           : a/b c&d=e
# as a PATH segment   : a/b%20c%26d%3De
# as a QUERY value    : a%2Fb%20c%26d%3De
# as FORM data        : a%2Fb+c%26d%3De
#   Using the wrong one silently splits your value into two parameters.

# What happens when you get it wrong:
bad  = f"https://api.example.com/search?q={value}"
good = f"https://api.example.com/search?q={quote(value, safe='')}"
print(f"\nbad  -> {parse_qs(urlparse(bad).query)}")
print(f"good -> {parse_qs(urlparse(good).query)}")
# bad  -> {'q': ['a/b c'], 'd': ['e']}      <- one parameter became two
# good -> {'q': ['a/b c&d=e']}              <- intact

# Relative resolution (RFC 3986) -- each form keeps a different prefix.
base = "https://example.com/docs/guide/intro.html"
for ref in ("style.css", "/assets/app.js", "../api/v2.html",
            "//cdn.example.net/lib.js", "?page=2", "#top"):
    print(f"  {ref:<26} -> {urljoin(base, ref)}")
#   style.css                  -> https://example.com/docs/guide/style.css
#   /assets/app.js             -> https://example.com/assets/app.js
#   ../api/v2.html             -> https://example.com/docs/api/v2.html
#   //cdn.example.net/lib.js   -> https://cdn.example.net/lib.js
#   ?page=2                    -> https://example.com/docs/guide/intro.html?page=2
#   #top                       -> https://example.com/docs/guide/intro.html#top

# IPv6 literals must be bracketed, or the colons are ambiguous with the port:
print(urlparse("http://[2001:db8::1]:8080/x").hostname,
      urlparse("http://[2001:db8::1]:8080/x").port)
# 2001:db8::1 8080

# Prove the fragment never leaves the browser:
$ curl -v 'https://example.com/page#secret-section' 2>&1 | grep '^> GET'
# > GET /page HTTP/2
#   The fragment is absent. Server-side logs and analytics cannot see it.

# Paths are case-sensitive; hosts are not:
$ curl -s -o /dev/null -w '%{http_code}\n' https://EXAMPLE.COM/
# 200
$ curl -s -o /dev/null -w '%{http_code}\n' https://example.com/INDEX.HTML
# 404      <- same server, different path

# Let curl handle encoding rather than building strings by hand:
$ curl -G --data-urlencode 'q=a/b c&d=e' -w '\n%{url_effective}\n' \
       -o /dev/null https://example.com/search
# https://example.com/search?q=a%2Fb%20c%26d%3De

# Content negotiation: one URL, several representations:
$ curl -s -H 'Accept: application/json' https://api.example.com/weather/london
# {"tempC":14}
$ curl -s -H 'Accept: text/html' https://api.example.com/weather/london
# <html><body>14C</body></html>
#   Same resource. Same URL. Different representation.

Key takeaways

A URI identifies, a URL also locates, a URN names without locating — in practice nearly everything is a URL.
A resource is the abstract thing identified; a representation is the bytes returned — one resource can have many representations.
Host is case-insensitive, path is case-sensitive; the port defaults from the scheme; IPv6 literals require brackets.
The fragment is never sent to the server — it is client-side only, invisible to server logs and analytics.
Percent-encoding rules differ per component: encoding a whole URL with one function is a standard source of broken parameters.
Relative references resolve by RFC 3986 rules, each form inheriting a different amount of the base URL.

🧪 Practice

Break https://shop.example.co.uk/items/AB-12?sort=price&dir=asc#reviews into its components and state exactly which bytes appear in the request line.

A search value is 100% & rising. Show the correctly encoded query string and explain what breaks if it is not encoded.

Resolve ../images/logo.png and /images/logo.png against the base https://example.com/a/b/c/page.html.

Interview-style: Why can server-side analytics not tell you which tab of a single-page application a user opened, when the URL clearly shows it? (Hint: look again at which component crosses the network.)

HTTP Request and Response Structure

Theory

HTTP/1.1 is a text-based, stateless, request-response protocol, and its wire format is simple enough to type by hand — which is exactly why it won. You can debug it with telnet, and every proxy, cache, and load balancer in the world could be written without a specialised parser.

Both messages share one shape:

  <start line>
  <header>: <value>
  <header>: <value>
  <blank line>              <- CRLF CRLF: the critical delimiter
  [body]

A request start line is METHOD SP request-target SP HTTP-version:

GET /index.html HTTP/1.1

A response start line (the status line) is HTTP-version SP status-code SP reason-phrase:

HTTP/1.1 200 OK

Details that matter in practice:

Line endings are CRLF (\r\n), not bare LF. The blank line that separates headers from body is therefore \r\n\r\n, and this exact sequence is what every HTTP parser scans for. Ambiguity about it is the root of an entire class of request smuggling vulnerabilities, where a front-end proxy and a back-end server disagree about where one message ends and the next begins.

Host is mandatory in HTTP/1.1. This is the single change that made virtual hosting possible: one IP address can serve thousands of sites because the request names which one it wants. In HTTP/2 and HTTP/3 the same information moves into the :authority pseudo-header.

Body length must be unambiguous, and there are exactly two ways to express it:

Content-Length: N — the body is exactly N bytes. Simple, but requires knowing the size in advance.
Transfer-Encoding: chunked — the body arrives as a sequence of size-prefixed chunks, terminated by a zero-length chunk. This allows streaming a response whose length is not yet known, which is how server-generated content is sent without buffering it entirely first.

Sending both is forbidden, and a mismatch between what a proxy believes and what a server believes is, again, request smuggling.

Headers are case-insensitive in name, may repeat (combining as a comma-separated list where the field allows it), and are conventionally either standard (registered with IANA) or application-defined. The old X- prefix convention for non-standard headers was deprecated by RFC 6648.

Statelessness is the design decision with the widest consequences. The server is not required to remember anything between requests. That makes any server able to answer any request — the property that makes load balancing and horizontal scaling trivial — and it is why state has to be smuggled back in through cookies, tokens, or the URL.

Examples

An HTTP/1.1 exchange, byte for byte

REQUEST                                  bytes on the wire
+-----------------------------------+
| GET /api/users?limit=10 HTTP/1.1\r\n |  <- start line
| Host: api.example.com\r\n            |  <- MANDATORY in 1.1
| User-Agent: curl/8.4.0\r\n           |
| Accept: application/json\r\n         |
| \r\n                                 |  <- BLANK LINE ends the headers
+-----------------------------------+      (no body for a GET)

RESPONSE
+-----------------------------------+
| HTTP/1.1 200 OK\r\n                  |  <- status line
| Content-Type: application/json\r\n   |
| Content-Length: 27\r\n               |  <- body is EXACTLY 27 bytes
| \r\n                                 |
| {"users":[{"id":1}],"n":1}           |  <- body
+-----------------------------------+


CHUNKED: streaming a body of unknown length

  HTTP/1.1 200 OK
  Transfer-Encoding: chunked
  <blank>
  1a\r\n                       <- chunk size in HEXADECIMAL (26 bytes)
  {"partial":"first piece"}\n\r\n
  12\r\n                       <- next chunk, 18 bytes
  {"more":"data"}\n\r\n
  0\r\n                        <- zero-length chunk = end of body
  \r\n

  Why it exists: the server can start sending before it knows the total size.

# Speaking HTTP/1.1 by hand -- no library, just bytes on a socket.
import socket

def raw_http_get(host, path="/", port=80):
    request = (
        f"GET {path} HTTP/1.1\r\n"        # start line: METHOD TARGET VERSION
        f"Host: {host}\r\n"               # mandatory: selects the virtual host
        f"User-Agent: hand-written/1.0\r\n"
        f"Connection: close\r\n"          # ask the server to close when done
        f"\r\n"                           # BLANK LINE: headers are finished
    )
    s = socket.create_connection((host, port), timeout=5)
    s.sendall(request.encode())
    data = b""
    while chunk := s.recv(4096):
        data += chunk
    s.close()
    return data

# Parsing is equally mechanical: split on the CRLF CRLF delimiter.
def parse_response(raw):
    head, _, body = raw.partition(b"\r\n\r\n")   # THE critical split
    lines = head.decode("iso-8859-1").split("\r\n")
    version, code, reason = lines[0].split(" ", 2)
    headers = {}
    for line in lines[1:]:
        name, _, value = line.partition(":")
        headers[name.lower().strip()] = value.strip()   # names are case-insensitive
    return {"version": version, "status": int(code), "reason": reason,
            "headers": headers, "body": body}

sample = (b"HTTP/1.1 200 OK\r\n"
          b"Content-Type: application/json\r\n"
          b"Content-Length: 26\r\n"
          b"\r\n"
          b'{"users":[{"id":1}],"n":1}')
r = parse_response(sample)
print(r["status"], r["reason"], r["headers"]["content-type"], len(r["body"]))
# 200 OK application/json 26

# Decoding a chunked body: size line, data, repeat until a 0-size chunk.
def decode_chunked(body):
    out, pos = b"", 0
    while True:
        end = body.index(b"\r\n", pos)
        size = int(body[pos:end].split(b";")[0], 16)   # HEX size, optional ext
        if size == 0:
            return out                                  # terminating chunk
        start = end + 2
        out += body[start:start + size]
        pos = start + size + 2                          # skip the trailing CRLF

chunked = (b"1a\r\n" + b"{'partial':'first piece'}\n" + b"\r\n"
           b"f\r\n"  + b"{'more':'data'}"              + b"\r\n"
           b"0\r\n\r\n")
print(decode_chunked(chunked))
# b"{'partial':'first piece'}\n{'more':'data'}"

# Why sending BOTH length headers is forbidden -- the smuggling primitive:
front_end_reads = "Content-Length"      # proxy trusts one header
back_end_reads  = "Transfer-Encoding"   # origin trusts the other
print(f"\nproxy uses {front_end_reads}, origin uses {back_end_reads}")
print("-> they disagree about where the request ends")
print("-> leftover bytes are parsed as the START of the next request")
#   This is HTTP request smuggling. RFC 9112 requires rejecting such messages.

# Watch the raw bytes, request and response, with nothing hidden:
$ curl -v --http1.1 https://example.com/ 2>&1 | grep -E '^[><]'
# > GET / HTTP/1.1
# > Host: example.com
# > User-Agent: curl/8.4.0
# > Accept: */*
# >
# < HTTP/1.1 200 OK
# < Content-Type: text/html; charset=UTF-8
# < Content-Length: 1256
# <

# Type HTTP by hand -- it really is this simple:
$ printf 'GET / HTTP/1.1\r\nHost: example.com\r\nConnection: close\r\n\r\n' \
  | openssl s_client -quiet -connect example.com:443 2>/dev/null | head -5
# HTTP/1.1 200 OK
# Content-Type: text/html; charset=UTF-8

# Omit the mandatory Host header and a 1.1 server must reject you:
$ printf 'GET / HTTP/1.1\r\n\r\n' | nc example.com 80 | head -1
# HTTP/1.1 400 Bad Request

# Virtual hosting: same IP, different Host header, different site:
$ curl -s -H 'Host: site-a.example.com' http://93.184.216.34/ | grep -o '<title>.*'
# <title>Site A</title>
$ curl -s -H 'Host: site-b.example.com' http://93.184.216.34/ | grep -o '<title>.*'
# <title>Site B</title>

# Chunked in the wild -- note there is no Content-Length:
$ curl -sI https://api.example.com/stream | grep -iE 'transfer-encoding|content-length'
# transfer-encoding: chunked

Key takeaways

Every HTTP/1.1 message is start line, headers, blank line, optional body, with CRLF line endings and \r\n\r\n as the header/body delimiter.
The request start line is METHOD target HTTP/version; the response status line is HTTP/version code reason.
Host is mandatory in HTTP/1.1, which is what enables virtual hosting; HTTP/2 and HTTP/3 carry it as :authority.
Body length is expressed by Content-Length or Transfer-Encoding: chunked — never both, because disagreement between proxy and origin enables request smuggling.
Chunked encoding lets a server stream a response whose total size is not yet known.
HTTP is stateless by design: any server can serve any request, which makes horizontal scaling easy and forces state into cookies, tokens, or URLs.

🧪 Practice

Write, byte for byte including CRLFs, a valid HTTP/1.1 POST of a 12-byte JSON body to /api/v1/notes on api.example.com.

Decode this chunked body by hand: 5\r\nHello\r\n6\r\n World\r\n0\r\n\r\n.

Explain what a server must do if a request contains both Content-Length and Transfer-Encoding, and why.

Interview-style: HTTP is stateless, yet you stay logged in across requests. Where does the state actually live, and what does statelessness buy the operator? (Hint: think about what must be true for a load balancer to send your next request to a different machine.)

Methods, Status Codes, and Headers

Theory

The message format is a container; methods, status codes, and headers are the vocabulary that gives it meaning. Each has a small set of properties that determine how every intermediary on the path is allowed to treat the message — which is why getting them right matters far beyond style.

Methods declare intent. The two properties that govern them are:

Safe — the request does not change server state. A crawler, a prefetcher, or a browser preview may issue it without asking permission.
Idempotent — issuing it N times has the same effect as issuing it once. This is what makes automatic retry possible after a timeout.

Method	Safe	Idempotent	Body	Purpose
GET	Yes	Yes	No	Retrieve a representation
HEAD	Yes	Yes	No	GET's headers only
OPTIONS	Yes	Yes	No	Query capabilities; CORS preflight
POST	No	No	Yes	Submit data; create subordinate
PUT	No	Yes	Yes	Replace the resource entirely
PATCH	No	No	Yes	Partial modification
DELETE	No	Yes	Optional	Remove the resource

The distinction that matters operationally: PUT is idempotent, POST is not. PUT /users/42 with the same body ten times leaves one user in one state. POST /users ten times creates ten users. This is precisely why a browser warns before re-submitting a form, and why a client library may safely retry a PUT after a network timeout but must not retry a POST without an idempotency key.

Status codes are grouped by first digit, and the class is meaningful even when the specific code is not recognised:

Class	Meaning	Codes worth knowing
1xx	Informational	`100 Continue`, `101 Switching Protocols`
2xx	Success	`200 OK`, `201 Created`, `204 No Content`, `206 Partial Content`
3xx	Redirection	`301` permanent, `302`/`307` temporary, `304 Not Modified`
4xx	Client error	`400`, `401`, `403`, `404`, `409`, `418`, `429`
5xx	Server error	`500`, `502`, `503`, `504`

Three pairs are routinely confused and worth stating precisely:

401 vs 403. 401 Unauthorized actually means unauthenticated — "I do not know who you are, try again with credentials", and it must carry WWW-Authenticate. 403 Forbidden means authenticated but not permitted — "I know who you are and the answer is still no". Retrying with credentials helps only with 401.
301 vs 302 vs 307/308. 301 is permanent and aggressively cached, sometimes forever, by browsers — deploy one by mistake and users may be stuck with it. 302 is temporary but historically caused clients to change POST into GET; 307 and 308 were defined to preserve the method explicitly.
502 vs 503 vs 504. 502 Bad Gateway means an upstream returned garbage; 503 Service Unavailable means "I am up but cannot serve now" (and should carry Retry-After); 504 Gateway Timeout means an upstream did not answer in time. In a load-balanced deployment, these three point at three different components.

Headers carry the metadata that makes HTTP extensible without changing the protocol. They fall into rough categories: representation (Content-Type, Content-Encoding, Content-Language), negotiation (Accept, Accept-Encoding, Accept-Language), conditional (If-None-Match, If-Modified-Since), caching (Cache-Control, ETag, Age), authentication (Authorization, WWW-Authenticate), connection (Connection, Upgrade), and security (Strict-Transport-Security, Content-Security-Policy).

Content negotiation deserves highlighting because it is the mechanism behind the resource/representation split: the client states preferences with quality values (Accept: text/html;q=0.9, application/json;q=1.0), and the server picks, signalling what varies with the Vary header so caches do not serve the wrong representation to the next client.

Examples

Choosing a method: the decision that determines retry safety

  Does it change server state?
     |                         \
     NO                         YES
     |                            |
   SAFE                    Is the result the same if repeated?
   GET / HEAD / OPTIONS       |                    \
   (crawlers may fetch       YES                    NO
    freely, caches may       |                       |
    store)               IDEMPOTENT            NOT IDEMPOTENT
                         PUT / DELETE           POST / PATCH
                         (safe to retry         (retry needs an
                          after a timeout)       idempotency key)


Status code classes at a glance

  1xx  |  hold on, more is coming
  2xx  |  it worked
  3xx  |  it is somewhere else / you already have it
  4xx  |  YOU made a mistake       <- do not retry unchanged
  5xx  |  I made a mistake         <- retry MAY help

  The confusable pairs:

  401 "who are you?"        -> send credentials, retry helps
  403 "I know. Still no."   -> retry is pointless

  301 permanent   -> browser may cache FOREVER. Very hard to undo.
  302 temporary   -> some clients rewrite POST -> GET (historical bug)
  307 temporary   -> method PRESERVED, explicitly
  308 permanent   -> method PRESERVED, explicitly

  502 upstream returned nonsense   -> look at the backend
  503 I am here but refusing        -> look at capacity/maintenance
  504 upstream never answered       -> look at timeouts and network

# Retry logic driven entirely by method properties and status class.

SAFE       = {"GET", "HEAD", "OPTIONS", "TRACE"}
IDEMPOTENT = SAFE | {"PUT", "DELETE"}

def may_retry(method, status=None, timed_out=False, has_idempotency_key=False):
    """The rules every well-behaved HTTP client implements."""
    if timed_out:
        # No response at all: only safe-to-repeat requests may be re-sent.
        return method in IDEMPOTENT or has_idempotency_key
    if status is None:
        return False
    if 500 <= status < 600 and status != 501:
        return method in IDEMPOTENT or has_idempotency_key
    if status == 429:
        return True                     # rate limited: honour Retry-After
    return False                        # 4xx: the request itself is wrong

cases = [
    ("GET",    None, True,  False), ("POST",   None, True,  False),
    ("POST",   None, True,  True),  ("PUT",    503,  False, False),
    ("POST",   503,  False, False), ("GET",    404,  False, False),
    ("GET",    429,  False, False),
]
print(f"{'method':<8}{'status':<8}{'timeout':<9}{'idem-key':<10}retry?")
for m, s, t, k in cases:
    print(f"{m:<8}{str(s):<8}{str(t):<9}{str(k):<10}{may_retry(m, s, t, k)}")
# method  status  timeout  idem-key  retry?
# GET     None    True     False     True
# POST    None    True     False     False    <- may have already created a user
# POST    None    True     True      True     <- key makes it safe
# PUT     503     False    False     True
# POST    503     False    False     False
# GET     404     False    False     False    <- 4xx: retrying changes nothing
# GET     429     False    False     True     <- back off and retry

# Idempotency in practice: the same PUT twice vs the same POST twice.
users = {}
next_id = [1]

def put_user(uid, data):
    users[uid] = data                        # REPLACE: repeating changes nothing
    return 200 if uid in users else 201

def post_user(data):
    uid = next_id[0]; next_id[0] += 1        # CREATE: repeating creates again
    users[uid] = data
    return 201, uid

for _ in range(3):
    put_user(42, {"name": "alice"})
print(f"\nafter 3 identical PUTs : {len(users)} user(s)")   # after 3 identical PUTs : 1 user(s)
users.clear(); next_id[0] = 1
for _ in range(3):
    post_user({"name": "alice"})
print(f"after 3 identical POSTs: {len(users)} user(s)")     # after 3 identical POSTs: 3 user(s)

# Content negotiation with quality values.
def negotiate(accept_header, available):
    prefs = []
    for part in accept_header.split(","):
        media, _, params = part.strip().partition(";")
        q = 1.0
        for p in params.split(";"):
            if p.strip().startswith("q="):
                q = float(p.strip()[2:])
        prefs.append((media.strip(), q))
    prefs.sort(key=lambda x: -x[1])               # highest q wins
    for media, q in prefs:
        if media in available:
            return media, q
        if media == "*/*":
            return available[0], q
    return None, 0

hdr = "text/html;q=0.8, application/json;q=1.0, */*;q=0.1"
print(f"\nnegotiated: {negotiate(hdr, ['application/json', 'text/html'])}")
# negotiated: ('application/json', 1.0)
#   The server MUST then send `Vary: Accept` so caches key on it.

# HEAD: all the metadata, none of the bytes -- ideal for checking a large file:
$ curl -sI https://example.com/large.iso | grep -iE 'content-length|content-type'
# content-length: 4294967296
# content-type: application/octet-stream
#   4 GB of information obtained without downloading 4 GB.

# 401 vs 403 -- note what the 401 carries:
$ curl -si https://api.example.com/private | head -3
# HTTP/2 401
# www-authenticate: Bearer realm="api"
#   ^ present on 401 by requirement: it tells you HOW to authenticate
$ curl -si -H 'Authorization: Bearer valid-but-unprivileged' \
       https://api.example.com/admin | head -1
# HTTP/2 403
#   No www-authenticate: more credentials will not help.

# Follow a redirect chain and see each hop's code:
$ curl -sIL http://example.com/old | grep -E '^HTTP|^location'
# HTTP/1.1 301 Moved Permanently
# location: https://example.com/old
# HTTP/2 308
# location: https://example.com/new
# HTTP/2 200
#   308, not 302: the method is preserved across the redirect.

# 503 should always tell you when to come back:
$ curl -sI https://api.example.com/ | grep -iE '^HTTP|retry-after'
# HTTP/2 503
# retry-after: 120

# Content negotiation and the Vary header that makes it cache-safe:
$ curl -sI -H 'Accept-Encoding: gzip' https://example.com/ | grep -iE 'content-encoding|vary'
# content-encoding: gzip
# vary: Accept-Encoding
#   Without Vary, a cache would serve gzipped bytes to a client that cannot decode them.

Key takeaways

Methods are characterised by safe (no state change) and idempotent (repeating equals doing once); these properties drive caching, prefetching, and retry.
PUT is idempotent, POST is not — which is why POST needs an idempotency key to be safely retried after a timeout.
Status classes carry meaning on their own: 4xx means do not retry unchanged; 5xx means a retry may help.
401 means unauthenticated (and must carry WWW-Authenticate); 403 means authenticated but forbidden — only 401 is fixed by credentials.
301 is cached aggressively and hard to undo; 307/308 preserve the request method where 302 historically did not.
502/503/504 point at different components — bad upstream response, deliberate unavailability, and upstream timeout respectively.
Content negotiation uses Accept with q-values, and the server must send Vary so caches key on the right dimension.

🧪 Practice

Classify PUT, POST, PATCH, and DELETE as safe and/or idempotent, and state which may be retried automatically after a timeout.

An API returns 403 to a request with no credentials at all. Explain why 401 would be more correct and what header the 401 must carry.

A deployment accidentally served 301 to the wrong host for an hour. Explain why the impact outlasts the fix and what you would do about it.

Interview-style: A client sees 504 from your load balancer while every application server reports healthy and idle. (Hint: 504 names which component failed to answer — check what sits between the balancer and the servers it is timing out on.)

Persistent Connections and Pipelining

Theory

HTTP/1.0 opened a TCP connection, sent one request, read one response, and closed. On a page with eighty images, that is eighty connections. The cost is not the connections themselves but the round trips each one requires before a single byte of content moves.

Count them for one HTTPS request on a cold connection:

  TCP SYN -> SYN/ACK -> ACK          1 RTT
  TLS ClientHello -> ... -> Finished 1 RTT (TLS 1.3) or 2 (TLS 1.2)
  HTTP request -> response           1 RTT
                                     ----
                                     3 RTTs before the first content byte

At 80 ms RTT that is 240 ms per resource, before transferring anything. There is also a slower, subtler cost: every new TCP connection restarts congestion control at the initial window (Chapter 7), so it begins slow even when the path is fast.

Persistent connections (keep-alive), default in HTTP/1.1, fix this by reusing one TCP connection for many request-response exchanges. The handshakes are paid once, and the congestion window stays warm and grows. Connection: close opts out; in HTTP/1.0 the opposite applied and you needed Connection: keep-alive to opt in.

Pipelining was the next step: send multiple requests without waiting for the responses in between. It works in principle and failed in practice, for one decisive reason — HTTP/1.1 responses must return in request order. A slow first response blocks every response behind it, even if they are ready. This is head-of-line blocking at the application layer, and combined with buggy proxies that mishandled pipelined requests, it led every major browser to disable pipelining.

What browsers did instead was open several parallel connections — conventionally six per origin. It is a workaround with real costs: six TCP handshakes, six TLS handshakes, six congestion windows competing with each other on the same bottleneck, and six times the server's per-connection memory. It also spawned the domain sharding hack, spreading assets across img1.example.com, img2.example.com and so on to get past the six-connection limit — an optimisation that becomes actively harmful under HTTP/2.

This is exactly the problem HTTP/2 was designed to solve, and the reason it is described in terms of multiplexing: many concurrent exchanges on one connection, without the ordering constraint that killed pipelining.

Examples

Three strategies, same eight resources, 80 ms RTT

 HTTP/1.0: connection per request
  [SYN..TLS..GET..resp][SYN..TLS..GET..resp][SYN..TLS..GET..resp]...
  |<-- 240 ms -->|<-- 240 ms -->|<-- 240 ms -->|  x8 = 1920 ms

 HTTP/1.1 keep-alive: one connection, sequential requests
  [SYN..TLS][GET..resp][GET..resp][GET..resp]...
  |<-160ms->|<- 80ms ->|<- 80ms ->|          160 + 8x80 = 800 ms

 HTTP/1.1 keep-alive x6 parallel connections
  conn1 [SYN..TLS][GET..resp][GET..resp]
  conn2 [SYN..TLS][GET..resp]
  ...                                        ~160 + 2x80 = 320 ms
  cost: 6 handshakes, 6 congestion windows competing


 PIPELINING and why it died:

  requests:   GET /a  GET /b  GET /c     (sent back to back, no waiting)
  server:     /a is a slow database query; /b and /c are ready instantly

  responses:  [....... /a (900 ms) .......][/b][/c]
              ^ /b and /c were READY but MUST wait: responses are ordered
                                                    ^^^^^^^^^^^^^^^^^^^^
              head-of-line blocking at the APPLICATION layer

# Quantify the round trips. This is the whole argument for keep-alive.

def page_load_ms(resources, rtt=80, tls=True, mode="keepalive", parallel=1):
    handshake = rtt + (rtt if tls else 0)      # TCP 1 RTT + TLS 1.3 1 RTT
    if mode == "close":                         # HTTP/1.0: reconnect every time
        total = resources * (handshake + rtt)
    else:                                       # handshake once per connection
        per_conn = -(-resources // parallel)    # ceil division
        total = handshake + per_conn * rtt
    return total

print(f"{'strategy':<34}{'ms':>8}")
print(f"{'HTTP/1.0 (new conn each)':<34}{page_load_ms(8, mode='close'):>8}")
print(f"{'HTTP/1.1 keep-alive, 1 conn':<34}{page_load_ms(8):>8}")
print(f"{'HTTP/1.1 keep-alive, 6 conns':<34}{page_load_ms(8, parallel=6):>8}")
print(f"{'HTTP/2 multiplexed, 1 conn':<34}{page_load_ms(8, parallel=8):>8}")
# strategy                                ms
# HTTP/1.0 (new conn each)              1920
# HTTP/1.1 keep-alive, 1 conn            800
# HTTP/1.1 keep-alive, 6 conns           320
# HTTP/2 multiplexed, 1 conn             240
#   Note HTTP/2 is only slightly better than 6 connections HERE --
#   the real win appears with dozens of resources and on lossy links.

# Head-of-line blocking: why pipelining failed.
def pipelined_completion(service_times_ms):
    """Responses must be returned IN ORDER, regardless of readiness."""
    ready, done, elapsed = [], [], 0
    for i, t in enumerate(service_times_ms):
        ready.append(t)                          # each is ready after its own time
    for i, t in enumerate(service_times_ms):
        elapsed = max(elapsed, t)                # cannot finish before it is ready
        done.append((i, elapsed))                # ...and cannot jump the queue
    return done

slow_first = [900, 10, 10, 10]
print("\npipelined, slow request first:")
for i, finish in pipelined_completion(slow_first):
    print(f"  response {i}: ready at {slow_first[i]:>4} ms, delivered at {finish:>4} ms")
# pipelined, slow request first:
#   response 0: ready at  900 ms, delivered at  900 ms
#   response 1: ready at   10 ms, delivered at  900 ms   <- blocked 890 ms
#   response 2: ready at   10 ms, delivered at  900 ms
#   response 3: ready at   10 ms, delivered at  900 ms
#   Three fast responses held hostage by one slow one. HTTP/2 removes this
#   constraint at the application layer (TCP-level HOL blocking remains).

# The hidden cost of parallel connections: congestion windows do not share.
def effective_start_rate(conns, iw_segments=10, mss=1460, rtt_ms=80):
    """Each new connection starts at the initial window, then must grow."""
    first_rtt_bytes = conns * iw_segments * mss
    return first_rtt_bytes * 8 / (rtt_ms / 1000) / 1e6   # Mb/s in the first RTT

for c in (1, 6):
    print(f"\n{c} connection(s): {effective_start_rate(c):.1f} Mb/s in the first RTT")
# 1 connection(s): 1.5 Mb/s in the first RTT
# 6 connection(s): 8.8 Mb/s in the first RTT
#   Faster start -- but 6 windows now probe the same bottleneck independently,
#   which is unfair to other flows and causes more loss.

# Prove reuse: 3 requests, 1 connection.
$ curl -s -o /dev/null -w 'connects=%{num_connects} time=%{time_total}\n' \
    https://example.com/a https://example.com/b https://example.com/c
# connects=1 time=0.412
#             ^ one TCP+TLS handshake served all three

# Force a new connection each time and watch the cost:
$ curl -s -o /dev/null -H 'Connection: close' \
    -w 'connects=%{num_connects} time=%{time_total}\n' \
    https://example.com/a https://example.com/b https://example.com/c
# connects=3 time=0.907

# Where the time actually goes on a cold connection:
$ curl -s -o /dev/null -w 'dns=%{time_namelookup} tcp=%{time_connect} \
tls=%{time_appconnect} first_byte=%{time_starttransfer}\n' https://example.com/
# dns=0.021 tcp=0.103 tls=0.288 first_byte=0.371
#           ^1 RTT     ^+1 RTT     ^+1 RTT      = 3 RTTs before content

# Server-side keep-alive settings decide how long reuse is possible:
$ curl -sI https://example.com/ | grep -i keep-alive
# keep-alive: timeout=5, max=100
#   5 s idle before close; 100 requests per connection.

# Count a browser's parallel connections to one origin:
$ ss -tn dst 93.184.216.34 | wc -l
# 6        <- the conventional per-origin limit for HTTP/1.1

Key takeaways

A cold HTTPS request costs about three round trips (TCP, TLS 1.3, HTTP) before the first content byte, plus a cold congestion window.
Persistent connections (default in HTTP/1.1) amortise handshakes across many exchanges and keep the congestion window warm.
Pipelining sends requests without waiting but requires in-order responses, causing application-layer head-of-line blocking — which, with buggy proxies, is why browsers disabled it.
Browsers worked around this with ~6 parallel connections per origin, paying six handshakes and running six competing congestion windows.
Domain sharding extended that hack further and becomes counterproductive under HTTP/2.
HTTP/2's multiplexing exists precisely to get pipelining's benefit without the ordering constraint.

🧪 Practice

Compute total load time for 20 resources at 60 ms RTT under HTTP/1.0, HTTP/1.1 single connection, and 6 parallel connections.

Explain why pipelining a slow request ahead of three fast ones is worse than not pipelining at all.

List three distinct costs of opening six parallel connections instead of one.

Interview-style: Domain sharding made sites faster under HTTP/1.1 and slower under HTTP/2. Explain both halves. (Hint: ask what sharding was buying, and whether HTTP/2 still has the limit it was buying around — then count handshakes and congestion windows.)

Cookies and Session State

Theory

HTTP is stateless: the server is not obliged to remember anything between requests. That is excellent for scaling and useless for anything involving a logged-in user. Cookies are the retrofit that reconciles the two.

The mechanism is deliberately minimal. The server sends Set-Cookie: name=value in a response; the browser stores it and returns Cookie: name=value on every subsequent matching request. The server remains stateless in the sense that any of its machines can handle any request — the state travels with the client.

There are two fundamentally different ways to use that channel, and confusing them causes both security and scaling problems:

Session identifier. The cookie holds an opaque random ID; the real state lives server-side in a session store. The cookie is small and reveals nothing, but every request requires a lookup, and the store becomes shared infrastructure that must be replicated.
Self-contained token. The cookie holds the state itself, signed or encrypted (a JWT, for example). No lookup is needed and servers stay genuinely stateless, but the token is larger, is sent on every request, and cannot be revoked before it expires without reintroducing server-side state.

The attributes are where the security lives, and each one closes a specific attack:

Attribute	Effect	Attack it addresses
`Secure`	Sent only over HTTPS	Network interception
`HttpOnly`	Invisible to JavaScript	XSS stealing the session
`SameSite=Strict`	Never sent cross-site	CSRF
`SameSite=Lax`	Sent on top-level GET navigation only	CSRF, with usable links
`SameSite=None`	Always sent; requires `Secure`	Deliberate cross-site use
`Domain`	Widens scope to subdomains	(Widening is a risk, not a defence)
`Path`	Restricts to a path prefix	Weak scoping
`Max-Age` / `Expires`	Lifetime; absent means session-only	Limits the theft window
`__Host-` prefix	Forces Secure, no Domain, Path=/	Subdomain cookie injection

Two subtleties that matter:

Cookies ignore ports and are only partly scoped by scheme. A cookie set by example.com:8080 is sent to example.com:443. The same-origin policy that governs JavaScript does not govern cookies, which is why Secure and __Host- prefixes exist as explicit corrections.

SameSite changed defaults. Modern browsers treat a cookie with no SameSite attribute as Lax, which broke a great deal of legitimate cross-site integration and is a common cause of "it works in one browser" reports.

The alternatives are worth knowing because they have different trade-offs rather than being strictly better. localStorage is not sent automatically, so it avoids CSRF entirely but is fully readable by JavaScript, so XSS defeats it — many teams switch to it for CSRF reasons and quietly worsen their XSS exposure. The Authorization: Bearer header must be attached explicitly, which is why it is the norm for APIs where there is no browser to manage cookie jars.

Examples

The cookie round trip, and what each attribute closes off

  1. Client logs in
     POST /login  { user, pass }

  2. Server responds with state the client must carry back
     HTTP/1.1 200 OK
     Set-Cookie: sid=9f2a...; Secure; HttpOnly; SameSite=Lax; Max-Age=3600; Path=/
                 \_______/   \____/  \______/  \___________/ \__________/
                 the value   HTTPS   no JS     no cross-site  expiry
                             only    access     sending

  3. Every later request carries it automatically
     GET /account
     Cookie: sid=9f2a...


  WHAT EACH ATTRIBUTE BLOCKS

  no Secure    -> [attacker on cafe wifi] reads sid from a plaintext request
  no HttpOnly  -> [XSS payload] document.cookie -> exfiltrates sid
  no SameSite  -> [evil.com] auto-submits a form to bank.com; browser
                  helpfully attaches YOUR cookie = CSRF


  SESSION ID vs SELF-CONTAINED TOKEN

  sid=9f2a...              (32 bytes)        JWT eyJhbGci... (800 bytes)
      |                                          |
      v                                          v
  [session store]                            [verify signature]
  lookup per request                         no lookup
  revoke instantly  <-- the key difference --> cannot revoke early
  shared infrastructure                      truly stateless servers

# Building a session cookie correctly, and showing what each flag prevents.
import secrets, hmac, hashlib, base64, json, time

SECRET = secrets.token_bytes(32)

def set_cookie_header(name, value, *, secure=True, http_only=True,
                      same_site="Lax", max_age=3600, path="/", domain=None):
    parts = [f"{name}={value}", f"Path={path}", f"Max-Age={max_age}"]
    if domain:     parts.append(f"Domain={domain}")   # widens scope: use sparingly
    if secure:     parts.append("Secure")             # HTTPS only
    if http_only:  parts.append("HttpOnly")           # invisible to document.cookie
    if same_site:  parts.append(f"SameSite={same_site}")
    return "Set-Cookie: " + "; ".join(parts)

sid = secrets.token_urlsafe(24)                        # opaque, unguessable
print(set_cookie_header("__Host-sid", sid))
# Set-Cookie: __Host-sid=Xk3...; Path=/; Max-Age=3600; Secure; HttpOnly; SameSite=Lax
#   The __Host- prefix makes the browser REJECT the cookie unless it is
#   Secure, has no Domain, and has Path=/ -- it cannot be set by a subdomain.

# Self-contained token: state in the cookie, integrity by signature.
def make_token(payload, ttl=3600):
    body = {**payload, "exp": int(time.time()) + ttl}
    raw = base64.urlsafe_b64encode(json.dumps(body).encode()).rstrip(b"=")
    sig = hmac.new(SECRET, raw, hashlib.sha256).digest()
    return raw.decode() + "." + base64.urlsafe_b64encode(sig).rstrip(b"=").decode()

def verify_token(token):
    raw, _, sig = token.partition(".")
    expected = base64.urlsafe_b64encode(
        hmac.new(SECRET, raw.encode(), hashlib.sha256).digest()).rstrip(b"=").decode()
    if not hmac.compare_digest(sig, expected):
        return None, "bad signature"                   # tampering detected
    body = json.loads(base64.urlsafe_b64decode(raw + "=="))
    if body["exp"] < time.time():
        return None, "expired"
    return body, "ok"

tok = make_token({"uid": 42, "role": "user"})
print(f"\ntoken length: {len(tok)} bytes (sent on EVERY request)")
print(verify_token(tok))
# token length: 131 bytes (sent on EVERY request)
# ({'uid': 42, 'role': 'user', 'exp': 1758...}, 'ok')

forged = tok[:-4] + "AAAA"                             # attacker edits the signature
print(verify_token(forged))
# (None, 'bad signature')

# The revocation problem, stated plainly:
print("\nUser clicks 'log out of all devices' at t=0, token expires at t=3600:")
print("  session ID : delete the row -> revoked INSTANTLY")
print("  JWT        : token stays valid for up to 3600 s unless you keep")
print("               a denylist -- which is server-side state again")

# Overhead comparison over a page load:
for name, size in (("session id", 32), ("JWT", 800)):
    print(f"\n{name:<12} {size:>4} B x 60 requests = {size*60/1024:>6.1f} KB uploaded")
# session id     32 B x 60 requests =    1.9 KB uploaded
# JWT           800 B x 60 requests =   46.9 KB uploaded
#   On a slow uplink, cookie size is a real page-load cost.

# Inspect the cookies a site sets and audit the flags:
$ curl -sI https://example.com/login | grep -i '^set-cookie'
# set-cookie: sid=9f2a8c...; Path=/; HttpOnly; Secure; SameSite=Lax; Max-Age=3600

# A jar shows scope: domain, path, secure flag, and expiry:
$ curl -s -c jar.txt https://example.com/login > /dev/null && cat jar.txt
# #HttpOnly_example.com  FALSE  /  TRUE  1790000000  sid  9f2a8c...
#  ^HttpOnly            ^domain   ^path ^secure ^expiry  ^name ^value

# Reuse the jar on a later request -- this is what the browser does for you:
$ curl -s -b jar.txt https://example.com/account | grep -o 'Welcome [a-z]*'
# Welcome alice

# Demonstrate that cookies ignore the PORT:
$ curl -s -b 'sid=x' http://example.com:8080/ -v 2>&1 | grep -i '^> cookie'
# > Cookie: sid=x
#   The same cookie goes to :80, :443, and :8080 alike.
#   Same-origin policy governs JavaScript, NOT the cookie jar.

# Audit a whole site's cookies for missing protections:
$ curl -sI https://example.com/ | grep -i '^set-cookie' | \
    grep -vE 'HttpOnly' && echo 'WARNING: cookie readable by JavaScript'

# Bearer tokens: nothing is automatic, which is the point for APIs:
$ curl -s -H "Authorization: Bearer $TOKEN" https://api.example.com/me
# {"uid":42,"role":"user"}
#   No cookie jar, no automatic attachment, therefore no CSRF surface.

Key takeaways

Cookies retrofit state onto a stateless protocol: Set-Cookie from the server, automatic Cookie on every matching request thereafter.
Session IDs keep state server-side (small, instantly revocable, needs a shared store); self-contained tokens keep state in the cookie (no lookup, but not revocable before expiry).
Secure blocks plaintext interception, HttpOnly blocks XSS theft, SameSite blocks CSRF — each attribute closes a specific, different attack.
Cookies ignore ports and are not governed by the same-origin policy — the __Host- prefix exists to impose strict scoping.
Modern browsers default absent SameSite to Lax, which silently breaks legitimate cross-site flows.
localStorage avoids CSRF but is fully exposed to XSS; Authorization: Bearer requires explicit attachment, which is why APIs prefer it.

🧪 Practice

Write a Set-Cookie header for a one-hour session cookie that is HTTPS-only, invisible to JavaScript, and not sent cross-site.

Explain precisely which attack each of Secure, HttpOnly, and SameSite prevents, and which one none of them prevent.

Compare session IDs and JWTs on revocation, request size, and infrastructure requirements; pick one for a banking app and justify it.

Interview-style: A team moves session tokens from cookies to localStorage "to avoid CSRF". Evaluate the change. (Hint: they removed one attack surface and enlarged another — ask what a successful XSS can now read that it previously could not.)

Web Caching and Conditional Requests

Theory

The fastest request is the one never sent; the second fastest is the one answered with "nothing changed". HTTP caching provides both, and it is probably the single highest-leverage performance mechanism on the web.

Caches exist at several layers, and a response may be stored at any of them: the browser cache (private, one user), a proxy or corporate cache (shared), a CDN edge (shared, geographically distributed), and a reverse proxy in front of the origin. The distinction between private and shared matters enormously — marking a personalised response cacheable by a shared cache leaks one user's data to another, one of the more serious misconfigurations in web operations.

There are two complementary models.

Freshness (no request at all). The server declares how long a response may be reused. While it is fresh, the cache serves it without contacting the origin — zero latency, zero traffic. Cache-Control directives govern this:

Directive	Meaning
`max-age=N`	Fresh for N seconds
`s-maxage=N`	Same, but for shared caches only (overrides max-age)
`public`	May be stored by shared caches
`private`	Browser only — never a CDN or proxy
`no-cache`	May store, but must revalidate before each reuse
`no-store`	Must not be written down at all
`must-revalidate`	Once stale, do not serve without checking
`immutable`	Will never change — do not even revalidate on reload
`stale-while-revalidate=N`	Serve stale for N s while refreshing in the background

no-cache and no-store are the pair people get wrong: no-cache means "check first", no-store means "do not keep it". Sensitive data needs no-store.

Validation (a cheap request). When freshness expires, the cache does not need to re-download — it asks whether anything changed, sending a validator:

ETag — an opaque version identifier, sent back in If-None-Match. Strong ("abc") means byte-identical; weak (W/"abc") means semantically equivalent.
Last-Modified — a timestamp, sent back in If-Modified-Since. Only one-second resolution, and useless for content that changes more often.

If nothing changed, the server replies 304 Not Modified with headers and no body. On a 2 MB image that turns a 2 MB transfer into roughly 200 bytes, at the cost of one round trip.

The strategy that falls out of this, and which every modern build tool implements, is the cache-busting / immutable pattern: give static assets content-hashed filenames (app.4f9a2c.js) and serve them with max-age=31536000, immutable. Because the name changes whenever the content does, the asset can be cached forever with no revalidation, and a deployment invalidates nothing — it simply references new names. The HTML that points at them stays short-lived or no-cache.

Vary is the final piece: it tells caches which request headers form part of the cache key. Serving gzipped content without Vary: Accept-Encoding means a client that cannot decode gzip may be handed the compressed bytes.

Examples

The decision a cache makes on every request

              request arrives at the cache
                          |
                is there a stored response?
                   /                  \
                  NO                   YES
                  |                      |
             fetch from origin      is it still FRESH?
                                   (age < max-age)
                                     /          \
                                   YES           NO
                                    |             |
                              SERVE FROM       do we have a validator?
                              CACHE            (ETag / Last-Modified)
                              0 RTT, 0 bytes    /          \
                                              YES           NO
                                               |             |
                                     conditional GET    full fetch
                                     If-None-Match      200 + body
                                        /      \
                                 304 (no body)  200 (new body)
                                 ~200 bytes     full transfer
                                 1 RTT          1 RTT


 Cost of the three outcomes for a 2 MB image on a 50 ms RTT link:

   fresh hit        0 ms,        0 bytes
   304 revalidate  50 ms,    ~200 bytes      <- 99.99% saved
   200 full fetch  ~370 ms, 2,000,000 bytes

# A cache implementing both freshness and validation.
import hashlib, time

class HTTPCache:
    def __init__(self):
        self.store = {}     # url -> dict(body, etag, stored_at, max_age, private)

    def _fresh(self, entry, now):
        return (now - entry["stored_at"]) < entry["max_age"]

    def fetch(self, url, origin, now=None, shared=False):
        now = now or time.time()
        entry = self.store.get(url)

        if entry:
            if shared and entry["private"]:
                pass                               # a shared cache MUST NOT use it
            elif self._fresh(entry, now):
                age = int(now - entry["stored_at"])
                return "HIT (fresh)", len(entry["body"]), 0, age
            else:
                # Stale: revalidate rather than re-download.
                status, body, etag = origin(url, if_none_match=entry["etag"])
                if status == 304:
                    entry["stored_at"] = now       # refreshed, body reused
                    return "HIT (304 revalidated)", 200, 1, 0
                entry.update(body=body, etag=etag, stored_at=now)
                return "MISS (changed)", len(body), 1, 0

        status, body, etag = origin(url)
        self.store[url] = {"body": body, "etag": etag, "stored_at": now,
                           "max_age": 300, "private": False}
        return "MISS (cold)", len(body), 1, 0

IMAGE = b"x" * 2_000_000
def origin(url, if_none_match=None):
    etag = '"' + hashlib.md5(IMAGE).hexdigest()[:12] + '"'
    if if_none_match == etag:
        return 304, b"", etag                      # unchanged: no body at all
    return 200, IMAGE, etag

c, t0 = HTTPCache(), 1_000_000
print(f"{'t':>6}  {'result':<24}{'bytes':>10}{'rtts':>6}")
for offset in (0, 60, 299, 400):
    result, nbytes, rtts, age = c.fetch("/img.png", origin, now=t0 + offset)
    print(f"{offset:>6}  {result:<24}{nbytes:>10,}{rtts:>6}")
#      t  result                       bytes  rtts
#      0  MISS (cold)              2,000,000     1
#     60  HIT (fresh)              2,000,000     0    <- no network at all
#    299  HIT (fresh)              2,000,000     0
#    400  HIT (304 revalidated)          200     1    <- 2 MB avoided

# The immutable pattern: the NAME encodes the content, so nothing expires.
def asset_url(path, content):
    digest = hashlib.sha256(content).hexdigest()[:8]
    stem, _, ext = path.rpartition(".")
    return f"/{stem}.{digest}.{ext}"

v1 = b"console.log('v1')"
v2 = b"console.log('v2')"
print(f"\nv1 -> {asset_url('app.js', v1)}   Cache-Control: max-age=31536000, immutable")
print(f"v2 -> {asset_url('app.js', v2)}   (a DIFFERENT url: nothing to invalidate)")
# v1 -> /app.9d4b1e02.js   Cache-Control: max-age=31536000, immutable
# v2 -> /app.4f2a7c81.js   (a DIFFERENT url: nothing to invalidate)
#   Deployment changes the HTML reference; the old file simply stops being requested.

# Why Vary exists -- the bug it prevents:
print("\nWithout `Vary: Accept-Encoding`:")
print("  client A (gzip capable)     -> cache stores GZIPPED bytes")
print("  client B (no gzip support)  -> cache serves the SAME gzipped bytes")
print("  -> client B receives unreadable binary")

# Read the caching policy of a response:
$ curl -sI https://example.com/static/app.4f9a2c.js | grep -iE 'cache-control|etag|age'
# cache-control: public, max-age=31536000, immutable
# etag: "4f9a2c81b3"
# age: 84213
#   ^ this response has been sitting in a shared cache for ~23 hours

# Perform a conditional request by hand and observe the 304:
$ ETAG=$(curl -sI https://example.com/img.png | awk '/etag/{print $2}' | tr -d '\r')
$ curl -sI -H "If-None-Match: $ETAG" https://example.com/img.png | head -1
# HTTP/2 304
$ curl -s -o /dev/null -w 'bytes=%{size_download}\n' \
       -H "If-None-Match: $ETAG" https://example.com/img.png
# bytes=0        <- a 304 carries NO body

# Measure what caching is actually worth:
$ curl -s -o /dev/null -w 'cold  %{time_total}s %{size_download} bytes\n' \
       https://example.com/img.png
# cold  0.371s 2000000 bytes
$ curl -s -o /dev/null -w '304   %{time_total}s %{size_download} bytes\n' \
       -H "If-None-Match: $ETAG" https://example.com/img.png
# 304   0.052s 0 bytes

# no-cache vs no-store -- check that sensitive pages use the right one:
$ curl -sI https://bank.example.com/statement | grep -i cache-control
# cache-control: no-store, private
#   no-store: never written to disk. `no-cache` would still be STORED.

# Confirm a personalised response is not marked public:
$ curl -sI -b jar.txt https://example.com/account | grep -iE 'cache-control|vary'
# cache-control: private, no-store
# vary: Cookie
#   `public` here would let a CDN serve alice's account page to bob.

Key takeaways

Caching offers two wins: freshness (no request at all) and validation (a ~200-byte 304 instead of the full body).
max-age/s-maxage set freshness; private vs public decides whether shared caches and CDNs may store a response — mismarking this leaks user data.
no-cache means revalidate before reuse; no-store means never write it down — sensitive data requires no-store.
ETag/If-None-Match and Last-Modified/If-Modified-Since are the validators; 304 Not Modified returns headers with no body.
The content-hashed immutable pattern (app.4f9a2c.js with max-age=31536000, immutable) makes static assets permanently cacheable and deployments invalidation-free.
Vary declares which request headers belong to the cache key — omitting Vary: Accept-Encoding can deliver unreadable compressed bytes.

🧪 Practice

Give the Cache-Control header you would set for: a content-hashed JS bundle, an HTML page, and a bank statement. Justify each.

Explain the difference between no-cache and no-store and give a scenario where using the wrong one is a security incident.

Compute bytes and round trips for 100 requests to a 500 KB asset with (a) no caching, (b) max-age=3600, (c) revalidation only.

Interview-style: A CDN is serving one customer's personalised dashboard to other customers. Name the two headers most likely misconfigured. (Hint: one decides who may store it, the other decides what makes two requests "the same".)

HTTP/2 Multiplexing and Header Compression

Theory

HTTP/1.1's problems were all consequences of one design choice: one connection carries one exchange at a time, in order. Pipelining could not fix it, and parallel connections only worked around it at real cost. HTTP/2 (RFC 7540, 2015, derived from Google's SPDY) changes the framing layer underneath while keeping HTTP's semantics — the same methods, status codes, and headers — completely intact.

Four changes matter.

1. Binary framing. HTTP/2 messages are binary frames, not text lines. Each frame has a 9-byte header: 24-bit length, 8-bit type, 8-bit flags, 31-bit stream identifier. This is less human-readable and far less ambiguous — the entire class of parsing discrepancies behind request smuggling largely disappears, because a frame's length is explicit.

2. Multiplexing over streams. A connection carries many independent streams, each with an ID; frames from different streams interleave freely. Because every frame is tagged, responses may arrive in any order — which is exactly the constraint that made pipelining useless. One connection now does what six were doing, without ordering the responses.

3. Header compression (HPACK). HTTP headers are extraordinarily repetitive: the same User-Agent, Accept, and Cookie on every one of a hundred requests. HPACK compresses them using a static table of 61 common header entries, a dynamic table of values seen earlier on this connection, and Huffman coding for literals. A repeated header can shrink to a single index byte, turning kilobytes of per-request overhead into tens of bytes.

4. Stream prioritisation and flow control. Streams carry dependency and weight information so a client can express that the stylesheet matters more than a below-the-fold image, and there is per-stream and per-connection flow control independent of TCP's.

Server push was the fifth feature and is the instructive failure. The idea — send the CSS before the browser asks for it — foundered because the server cannot know what the client already has cached, so pushes were frequently wasted bandwidth. Chrome removed support in 2022; the replacement is 103 Early Hints, which tells the client what to fetch and lets the cache decide.

The one problem HTTP/2 does not solve is the decisive one for HTTP/3: TCP-level head-of-line blocking. HTTP/2 removed the ordering constraint at the application layer, but all streams still ride a single TCP connection, and TCP delivers bytes strictly in order. Lose one segment and every stream stalls until it is retransmitted — even streams whose data already arrived. On a clean link HTTP/2 is a clear win; on a lossy mobile link it can be worse than HTTP/1.1 with six connections, because six connections at least fail independently.

A practical consequence: stop sharding domains. HTTP/2 wants one connection per origin so that HPACK's dynamic table, prioritisation, and a single congestion window all work. Sharding splits them back apart.

Examples

HTTP/1.1 vs HTTP/2 on one connection

 HTTP/1.1 (ordered, one at a time)
  conn: [--- req A ---][--- resp A ---][--- req B ---][--- resp B ---]
        responses MUST come back in request order

 HTTP/2 (multiplexed, interleaved, any order)
  conn: [A:HEADERS][B:HEADERS][C:HEADERS][B:DATA][C:DATA][A:DATA][B:DATA]
         \_______/  \_______/             \____/  \____/  \____/
         stream 1   stream 3              B finishes FIRST, even though
                                          it was requested second

 The 9-byte frame header that makes this possible:

   0                   1                   2                   3
   +-----------------------------------------------+
   |                 Length (24)                   |
   +---------------+---------------+---------------+
   |   Type (8)    |   Flags (8)   |
   +-+-------------+---------------+-------------------------------+
   |R|                 Stream Identifier (31)                      |
   +=+=============================================================+
   |                   Frame Payload                               |
   +---------------------------------------------------------------+
      ^ every frame is TAGGED with its stream -> interleaving is safe


 WHAT HTTP/2 DOES NOT FIX: TCP head-of-line blocking

  TCP byte stream:  [s1][s3][s5][XX][s1][s3][s5][s1]
                                  ^^ one lost segment
  TCP will not deliver ANY later bytes until XX is retransmitted.
  Streams 3 and 5 are stalled by a loss that belonged to stream 1.

  6 x HTTP/1.1 connections: a loss stalls ONE connection.
  1 x HTTP/2 connection:    a loss stalls ALL streams.
  This is why HTTP/2 can lose to HTTP/1.1 on a lossy mobile link.

# HPACK: why header compression matters so much on a real page.

STATIC_TABLE = {                       # abbreviated RFC 7541 static table
    ":method GET": 2, ":path /": 4, ":scheme https": 7,
    ":status 200": 8, "accept-encoding gzip, deflate": 16,
}

class HPACKEncoder:
    def __init__(self):
        self.dynamic = {}              # values learned on THIS connection
        self.next_index = 62           # dynamic table starts after the static one

    def encode(self, headers):
        out = 0
        for name, value in headers:
            key = f"{name} {value}"
            if key in STATIC_TABLE:
                out += 1                                   # a single index byte
            elif key in self.dynamic:
                out += 1                                   # seen before: 1 byte
            else:
                self.dynamic[key] = self.next_index        # remember for next time
                self.next_index += 1
                out += len(name) + len(value) + 2          # literal, roughly
        return out

request = [
    (":method", "GET"), (":scheme", "https"), (":authority", "example.com"),
    ("user-agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
                   "(KHTML, like Gecko) Chrome/120.0 Safari/537.36"),
    ("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"),
    ("accept-encoding", "gzip, deflate"),
    ("cookie", "sid=9f2a8c1d4b7e; prefs=dark; consent=all; ab_test=variant_b"),
]

enc = HPACKEncoder()
uncompressed = sum(len(n) + len(v) + 4 for n, v in request)
sizes = [enc.encode([(n, v if n != ":path" else f"/asset{i}.js")
                     for n, v in request]) for i in range(1, 6)]

print(f"uncompressed, per request : {uncompressed} bytes")
for i, s in enumerate(sizes, 1):
    print(f"  HPACK request {i}        : {s:>4} bytes")
print(f"\n100 requests uncompressed : {uncompressed*100:>7,} bytes")
print(f"100 requests with HPACK   : {sizes[0] + sizes[-1]*99:>7,} bytes")
# uncompressed, per request : 359 bytes
#   HPACK request 1        :  296 bytes
#   HPACK request 2        :    7 bytes
#   HPACK request 3        :    7 bytes
#   HPACK request 4        :    7 bytes
#   HPACK request 5        :    7 bytes
#
# 100 requests uncompressed :  35,900 bytes
# 100 requests with HPACK   :     989 bytes
#   ~97% saved. The first request pays; every later one is nearly free.

# TCP head-of-line blocking, measured.
def completion_times(n_streams, loss_at_ms=None, rtt=80, stream_ms=100):
    """All streams share one TCP connection: a loss stalls everything."""
    times = {}
    for s in range(n_streams):
        t = stream_ms
        if loss_at_ms is not None and loss_at_ms < stream_ms:
            t += rtt                                    # ALL streams wait for RTO
        times[s] = t
    return times

clean  = completion_times(5)
lossy  = completion_times(5, loss_at_ms=40)
print(f"\nclean link : all 5 streams finish at {max(clean.values())} ms")
print(f"1 loss     : all 5 streams finish at {max(lossy.values())} ms")
print("           ^ four streams were delayed by a loss that was not theirs")
# clean link : all 5 streams finish at 100 ms
# 1 loss     : all 5 streams finish at 180 ms

# Confirm the negotiated version (ALPN does this during the TLS handshake):
$ curl -sI https://example.com/ -w '%{http_version}\n' -o /dev/null
# 2

# Prove multiplexing: many resources, ONE connection:
$ curl -s -o /dev/null --http2 -w 'connects=%{num_connects}\n' \
    https://example.com/{a,b,c,d,e,f,g,h}.js
# connects=1
#   Under HTTP/1.1 this needed six connections and still serialised.

# Watch the frames and stream ids directly:
$ nghttp -nv https://example.com/ 2>&1 | grep -E 'recv|send' | head -8
# send HEADERS frame <length=42, flags=0x25, stream_id=1>
# send HEADERS frame <length=38, flags=0x25, stream_id=3>
# recv HEADERS frame <length=91, flags=0x04, stream_id=3>   <- 3 answers FIRST
# recv DATA frame <length=1024, flags=0x00, stream_id=3>
# recv HEADERS frame <length=88, flags=0x04, stream_id=1>
#   Stream ids are odd for client-initiated streams; interleaving is visible.

# Measure HPACK's effect:
$ nghttp -nv https://example.com/ 2>&1 | grep -E 'HEADERS.*length'
# send HEADERS frame <length=289, ...>   <- first request: literals
# send HEADERS frame <length=12, ...>    <- second: indexed from the dynamic table

# 103 Early Hints, the replacement for server push:
$ curl -sI --http2 https://example.com/ | head -4
# HTTP/2 103
# link: </style.css>; rel=preload; as=style
# HTTP/2 200
#   The client is told what to fetch and consults its own cache first.

# Check whether an origin still shards domains (counterproductive now):
$ curl -s https://example.com/ | grep -oE 'https://[a-z0-9]+\.example\.com' | sort -u
# https://static.example.com
# https://img1.example.com     <- legacy HTTP/1.1 sharding: remove under HTTP/2
# https://img2.example.com

Key takeaways

HTTP/2 changes the framing layer only — methods, status codes, and header semantics are unchanged from HTTP/1.1.
Binary frames tagged with a stream ID allow full multiplexing: many concurrent exchanges on one connection, responses in any order.
HPACK compresses repetitive headers with a static table, a per-connection dynamic table, and Huffman coding — often a 90%+ reduction after the first request.
Stream priorities and per-stream flow control let clients express what matters most.
Server push failed because the server cannot know the client's cache; 103 Early Hints replaced it.
HTTP/2 does not fix TCP head-of-line blocking: one lost segment stalls every stream, which is why it can underperform HTTP/1.1 on lossy links — and the reason HTTP/3 exists.
Domain sharding becomes harmful under HTTP/2: one connection per origin is what makes HPACK and prioritisation work.

🧪 Practice

Explain how HTTP/2 achieves what pipelining could not, naming the specific field in the frame header that makes it possible.

Compute header bytes for 50 requests with 400 bytes of headers each, uncompressed versus HPACK with a warm dynamic table.

Describe a network condition under which HTTP/1.1 with six connections beats HTTP/2, and explain the mechanism.

Interview-style: Your team shards assets across four subdomains and sees no improvement after enabling HTTP/2 — in fact it got slower. (Hint: count the connections, then count how many dynamic tables and congestion windows you now have.)

HTTP/3 over QUIC

Theory

HTTP/2 removed application-layer head-of-line blocking and ran straight into the transport-layer version. The fix cannot be made inside TCP, because TCP's defining guarantee is in-order byte delivery — a single reliable stream with no notion of independent substreams. If you want independent streams that fail independently, you need a different transport.

HTTP/3 (RFC 9114) is HTTP semantics over QUIC (RFC 9000), a transport built on UDP. Chapter 7 covered QUIC in detail; what matters here is what it changes for HTTP.

Independent streams. QUIC implements streams in the transport itself, each with its own sequence space and loss recovery. Lose a packet carrying stream 3's data and only stream 3 waits. Streams 1 and 5 continue to be delivered. This is the whole reason HTTP/3 exists, and the improvement is largest exactly where HTTP/2 was weakest: lossy mobile and satellite links.

Fewer round trips. QUIC integrates TLS 1.3 into the transport handshake rather than layering it on top. A cold connection completes in 1 RTT instead of TCP's 3; a resumed connection can send application data in the first flight — 0-RTT, with the known replay caveat that 0-RTT data must be idempotent.

Connection migration. A QUIC connection is identified by a connection ID, not by the 4-tuple. Move from Wi-Fi to cellular and your IP changes, but the connection ID does not, so the connection survives. Under TCP, the 4-tuple is the connection, so it dies and every request in flight must be re-issued.

Encryption by default and by construction. Almost the entire QUIC header is encrypted or authenticated, so middleboxes cannot read or modify it. This is the anti-ossification argument from Chapter 7: what cannot be inspected cannot be depended on, so the protocol remains free to evolve.

HTTP/3 also swaps HPACK for QPACK, because HPACK's dynamic table assumed ordered delivery — the very assumption QUIC removes. QPACK keeps compression while tolerating out-of-order header blocks, using a separate encoder/decoder stream to avoid reintroducing the blocking it was trying to escape.

Discovery is worth knowing operationally. A client cannot assume HTTP/3, so it learns about it either from the Alt-Svc response header (h3=":443"; ma=86400) after a first HTTP/1.1 or HTTP/2 connection, or — better — in advance from a DNS HTTPS/SVCB record carrying alpn="h3", which saves the discovery round trip entirely.

The costs are real: UDP is sometimes blocked or rate-limited by enterprise firewalls (so clients must keep a TCP fallback), per-packet CPU cost is higher because congestion control and loss recovery run in userspace, and the encrypted headers that prevent ossification also prevent the network-level troubleshooting many operators rely on.

Examples

The stack, and where the blocking lives

 HTTP/2                              HTTP/3
 +---------------------+             +---------------------+
 |       HTTP/2        |             |       HTTP/3        |
 +---------------------+             +---------------------+
 |      TLS 1.3        |             |        QUIC         |
 +---------------------+             |  (streams + TLS 1.3 |
 |        TCP          |             |   + loss recovery)  |
 +---------------------+             +---------------------+
 |         IP          |             |        UDP          |
 +---------------------+             +---------------------+
   streams are ABOVE                 |         IP          |
   the reliable stream               +---------------------+
   -> one loss blocks all              streams are INSIDE
                                       the transport
                                       -> one loss blocks one


 HEAD-OF-LINE BLOCKING, side by side (X = lost packet)

 HTTP/2 over TCP
   wire:   [s1][s3][ X][s5][s1][s3]
   deliver: s1, s3, ......... STALL ......... then s5, s1, s3
            ^ s5 ARRIVED but TCP will not hand it up: it is "out of order"

 HTTP/3 over QUIC
   wire:   [s1][s3][ X][s5][s1][s3]
   deliver: s1, s3, s5 immediately; only stream 1's missing piece waits
            ^ independent sequence spaces per stream


 HANDSHAKE COST (80 ms RTT)

  HTTP/2 cold:  TCP(1) + TLS1.3(1) + request(1) = 3 RTT = 240 ms
  HTTP/3 cold:  QUIC+TLS(1)        + request(0) = 1 RTT =  80 ms
  HTTP/3 0-RTT: data in the very first flight   = 0 RTT =   0 ms setup

# Head-of-line blocking: the same loss, two transports.

def deliver(packets, lost_index, transport):
    """packets: list of (stream_id, label). Returns delivery order with stalls."""
    delivered, buffered = [], []
    for i, (sid, label) in enumerate(packets):
        if i == lost_index:
            buffered.append((sid, label, "LOST -> retransmit"))
            continue
        if transport == "tcp":
            # TCP: strictly in order. Anything after a gap must wait.
            if any(b for b in buffered):
                buffered.append((sid, label, "held by HOL blocking"))
            else:
                delivered.append((sid, label))
        else:
            # QUIC: only the SAME stream is blocked.
            blocked_streams = {b[0] for b in buffered}
            if sid in blocked_streams:
                buffered.append((sid, label, "held (same stream)"))
            else:
                delivered.append((sid, label))
    return delivered, buffered

packets = [(1, "a"), (3, "b"), (1, "c"), (5, "d"), (3, "e"), (5, "f")]
for transport in ("tcp", "quic"):
    d, b = deliver(packets, lost_index=2, transport=transport)
    print(f"\n{transport.upper()}: delivered immediately -> {[x[1] for x in d]}")
    print(f"      waiting               -> {[(x[1], x[2]) for x in b]}")
# TCP: delivered immediately -> ['a', 'b']
#       waiting               -> [('c','LOST -> retransmit'), ('d','held by HOL blocking'),
#                                 ('e','held by HOL blocking'), ('f','held by HOL blocking')]
# QUIC: delivered immediately -> ['a', 'b', 'd', 'e', 'f']
#       waiting               -> [('c','LOST -> retransmit')]
#   Same loss. TCP stalls 3 unrelated packets; QUIC stalls none.

# When does HTTP/3 actually win? Loss rate is the variable that decides.
def page_time_ms(resources, rtt, loss_rate, http3=False):
    setup = rtt if http3 else 3 * rtt                 # 1 RTT vs TCP+TLS+request
    transfer = resources * 8                          # nominal per-resource ms
    # Each loss costs a retransmit RTT; under TCP it stalls EVERYTHING.
    losses = resources * loss_rate
    penalty = losses * rtt * (1 if http3 else 3)      # HOL blocking multiplier
    return setup + transfer + penalty

print(f"\n{'loss':>7}{'HTTP/2 ms':>12}{'HTTP/3 ms':>12}{'gain':>9}")
for loss in (0.0, 0.005, 0.02, 0.05):
    h2 = page_time_ms(60, 80, loss)
    h3 = page_time_ms(60, 80, loss, http3=True)
    print(f"{loss:>6.1%}{h2:>12,.0f}{h3:>12,.0f}{(h2-h3)/h2:>8.0%}")
#    loss   HTTP/2 ms   HTTP/3 ms     gain
#   0.0%         720         560      22%
#   0.5%         792         584      26%
#   2.0%       1,008         656      35%
#   5.0%       1,440         800      44%
#   The advantage GROWS with loss -- exactly the mobile-network case.

# Connection migration: what identifies the connection.
tcp_conn  = {"identified_by": ("192.168.1.20", 52814, "93.184.216.34", 443)}
quic_conn = {"identified_by": "cid=8f3a2b91c4d5e6f7"}
print("\nUser walks out of Wi-Fi range; IP changes to 10.45.2.7:")
print(f"  TCP  : 4-tuple {tcp_conn['identified_by'][0]} -> 10.45.2.7 : CONNECTION DEAD")
print(f"  QUIC : {quic_conn['identified_by']} unchanged        : CONNECTION SURVIVES")

# Negotiate HTTP/3 explicitly and confirm the version:
$ curl --http3 -sI https://cloudflare.com/ -w '%{http_version}\n' -o /dev/null
# 3

# Discovery method 1: Alt-Svc, learned AFTER a first TCP connection.
$ curl -sI https://example.com/ | grep -i alt-svc
# alt-svc: h3=":443"; ma=86400
#   The client must connect over TCP once before it learns this.

# Discovery method 2: DNS, learned BEFORE connecting -- strictly better.
$ dig +short cloudflare.com HTTPS
# 1 . alpn="h3,h2" ipv4hint=104.16.132.229
#       ^^ the client can open QUIC immediately: no discovery round trip

# Compare handshake costs on the same host:
$ curl --http2 -s -o /dev/null -w 'h2 connect=%{time_connect} tls=%{time_appconnect} \
first_byte=%{time_starttransfer}\n' https://cloudflare.com/
# h2 connect=0.082 tls=0.171 first_byte=0.255
$ curl --http3 -s -o /dev/null -w 'h3 connect=%{time_connect} \
first_byte=%{time_starttransfer}\n' https://cloudflare.com/
# h3 connect=0.081 first_byte=0.166
#   ~90 ms saved: the TLS handshake is folded into the transport handshake.

# HTTP/3 is UDP -- this is what a firewall sees (and sometimes blocks):
$ sudo tcpdump -i any -n 'udp port 443' -c 3
# IP 192.168.1.20.54021 > 104.16.132.229.443: UDP, length 1350
#   No TCP handshake at all. Enterprise filters that allow only TCP/443
#   force a fallback, which is why clients always keep a TCP path.

# Watch a connection migrate across networks (connection ID stays constant):
$ sudo tcpdump -i any -n 'udp port 443' | head -4
# IP 192.168.1.20.54021 > 104.16.132.229.443: UDP, length 1350
# IP 10.45.2.7.54021 > 104.16.132.229.443: UDP, length 82
#    ^^^^^^^^^ new source address, same QUIC connection: no reconnect, no lost requests

Key takeaways

HTTP/3 is HTTP semantics over QUIC on UDP, adopted because TCP head-of-line blocking cannot be fixed inside TCP.
QUIC puts streams in the transport, each with its own loss recovery, so a lost packet stalls only its own stream.
The handshake folds TLS 1.3 into the transport: 1 RTT cold, 0-RTT on resumption (idempotent data only).
Connection migration works because a connection is identified by a connection ID, not a 4-tuple — Wi-Fi to cellular no longer kills it.
QPACK replaces HPACK because HPACK's dynamic table assumed ordered delivery.
Discovery is via Alt-Svc (costs a prior TCP connection) or, better, a DNS HTTPS/SVCB record with alpn="h3".
Costs: UDP blocking in some networks (TCP fallback required), higher CPU per packet, and reduced network-level observability.

🧪 Practice

Explain why HTTP/2's multiplexing does not remove head-of-line blocking and what specifically had to change to remove it.

Count the round trips before the first response byte for HTTP/1.1, HTTP/2, HTTP/3 cold, and HTTP/3 0-RTT at 100 ms RTT.

Describe how QUIC survives a Wi-Fi to cellular handover and why TCP cannot.

Interview-style: Under what measured conditions would you advise a team against enabling HTTP/3? (Hint: think about their users' networks, their observability tooling, and where their CPU budget is already tight.)

WebSockets and Server-Sent Events

Theory

HTTP's request-response shape has one structural gap: the server cannot speak first. Every workaround discussed earlier — polling, long polling — is an attempt to simulate push over a protocol that does not have it. WebSockets and Server-Sent Events close the gap properly, in two different ways.

Server-Sent Events (SSE) is the smaller idea and often the right one. It is still HTTP: the client issues an ordinary GET with Accept: text/event-stream, and the server responds with Content-Type: text/event-stream and simply never finishes the body. Events are pushed as they occur, as plain UTF-8 text in a trivially simple format:

event: update
data: {"price": 42.5}
id: 1042

Each event ends with a blank line, and data: may repeat for multiline payloads. Because SSE is ordinary HTTP, it inherits everything: proxies, compression, authentication headers, HTTP/2 multiplexing, and — importantly — automatic reconnection with resumption. The browser's EventSource reconnects on failure and sends Last-Event-ID, so the server can replay what was missed. That reconnection logic is free, which is more valuable in practice than it sounds.

Its limitations: one direction only (server to client; the client uses ordinary requests to send), text only (binary must be base64-encoded, costing 33%), and under HTTP/1.1 each open stream consumes one of the browser's six connections per origin — a limit that disappears under HTTP/2.

WebSocket (RFC 6455) is the full-duplex answer. It begins as an HTTP request with Upgrade: websocket, and on 101 Switching Protocols the TCP connection stops being HTTP entirely and becomes a bidirectional, message-framed channel. The Sec-WebSocket-Key/Sec-WebSocket-Accept exchange is not security — it is a handshake proving both sides genuinely intend WebSocket, so that a confused cache or proxy cannot be tricked into forwarding raw frames.

WebSocket frames are binary, with a small header (2–14 bytes), an opcode (text, binary, close, ping, pong), and — for client-to-server frames only — a mandatory 32-bit XOR mask. The mask exists for a specific reason: without it, an attacker could craft JavaScript that produces bytes resembling an HTTP request, poisoning intermediary caches. Masking makes the client's bytes unpredictable to the attacker.

What WebSocket gives up is everything HTTP provided: no caching, no automatic reconnection, no standard status codes, no automatic compression (it needs the permessage-deflate extension), and the connection is pinned to one server, which complicates load balancing.

	SSE	WebSocket
Direction	Server → client	Bidirectional
Protocol	HTTP (never leaves it)	Upgrades away from HTTP
Data	UTF-8 text only	Text or binary
Reconnect	Automatic, with `Last-Event-ID`	Manual
Proxy/infra friendliness	High	Lower
Per-message overhead	~10 bytes text	2–14 bytes
Browser API	`EventSource`	`WebSocket`

The engineering rule is to default to SSE for one-way feeds — notifications, live scores, progress bars, log tails — and reach for WebSocket only when the client genuinely needs to send at high rate or low latency: chat, collaborative editing, multiplayer games, terminal sessions.

Examples

Three ways to get data from server to client

 LONG POLLING            SSE                        WEBSOCKET
 C          S            C          S               C          S
 |--GET---->|            |--GET---->|               |--Upgrade->|
 |          |(hold)      |<--200----|               |<--101-----|
 |<--event--|            |   text/event-stream      |  (no longer HTTP)
 |--GET---->|(new req)   |<--event--|               |<--frame---|
 |          |(hold)      |<--event--|               |--frame--->|
 |<--event--|            |<--event--|               |<--frame---|
                         (one request, forever)     (full duplex)

 one request PER event   one request TOTAL          one upgrade TOTAL
 still HTTP              still HTTP                 no longer HTTP
 no server->client       server->client only        BOTH directions
   channel


 THE SSE WIRE FORMAT (literally this simple)

   event: price\n          <- optional event name; default is "message"
   data: {"v":42.5}\n      <- the payload; may repeat for multiline
   id: 1042\n              <- enables resumption after a reconnect
   retry: 5000\n           <- tells the client how long to wait before retrying
   \n                      <- BLANK LINE terminates the event


 THE WEBSOCKET FRAME

    0                   1                   2                   3
   +-+-+-+-+-------+-+-------------+-------------------------------+
   |F|R|R|R| opcode|M| Payload len |    Extended payload length    |
   |I|S|S|S|  (4)  |A|     (7)     |            (16/64)            |
   |N|V|V|V|       |S|             |                               |
   +-+-+-+-+-------+-+-------------+-------------------------------+
   |     Masking-key (4 bytes, CLIENT->SERVER ONLY)                |
   +---------------------------------------------------------------+
   |                      Payload Data                             |
   +---------------------------------------------------------------+
     ^ masking is MANDATORY from client to server: it stops crafted
       payloads from looking like HTTP requests to a confused proxy

# SSE: the server side is almost trivially simple.

def sse_event(data, event=None, event_id=None, retry=None):
    """Format one SSE event. Note the terminating BLANK line."""
    lines = []
    if event:    lines.append(f"event: {event}")
    if event_id: lines.append(f"id: {event_id}")
    if retry:    lines.append(f"retry: {retry}")
    for line in str(data).split("\n"):        # data: repeats for multiline
        lines.append(f"data: {line}")
    return "\n".join(lines) + "\n\n"          # blank line = end of event

print(sse_event('{"price": 42.5}', event="update", event_id=1042), end="")
# event: update
# id: 1042
# data: {"price": 42.5}
#
print(repr(sse_event("line one\nline two")))
# 'data: line one\ndata: line two\n\n'

# Resumption for free: the client sends back the last id it saw.
history = {1040: "a", 1041: "b", 1042: "c", 1043: "d"}
def replay(last_event_id):
    missed = {k: v for k, v in history.items() if k > int(last_event_id)}
    return [sse_event(v, event_id=k) for k, v in sorted(missed.items())]

print(f"client reconnects with Last-Event-ID: 1041")
print(f"server replays {len(replay('1041'))} missed events")
# client reconnects with Last-Event-ID: 1041
# server replays 2 missed events
#   EventSource does the reconnect AND sends the header automatically.

# WebSocket handshake: the Accept value is a fixed, verifiable computation.
import hashlib, base64
GUID = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"        # RFC 6455, fixed constant

def ws_accept(client_key):
    digest = hashlib.sha1((client_key + GUID).encode()).digest()
    return base64.b64encode(digest).decode()

key = "dGhlIHNhbXBsZSBub25jZQ=="
print(f"\nSec-WebSocket-Key:    {key}")
print(f"Sec-WebSocket-Accept: {ws_accept(key)}")
# Sec-WebSocket-Key:    dGhlIHNhbXBsZSBub25jZQ==
# Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=
#   Not encryption -- proof that the peer speaks WebSocket and is not a
#   confused proxy replaying a cached response.

# Client-to-server masking, and why it exists.
def mask(payload: bytes, key: bytes) -> bytes:
    return bytes(b ^ key[i % 4] for i, b in enumerate(payload))

evil = b"GET /admin HTTP/1.1\r\nHost: internal\r\n\r\n"   # attacker-chosen bytes
masking_key = b"\x37\xfa\x21\x3d"
masked = mask(evil, masking_key)
print(f"\nattacker wanted on the wire: {evil[:20]}")
print(f"what actually appears:       {masked[:20]}")
print(f"unmasks correctly:           {mask(masked, masking_key) == evil}")
# attacker wanted on the wire: b'GET /admin HTTP/1.1\r'
# what actually appears:       b'p\x9f\x05\x19\x18\xd4A\x14...'
# unmasks correctly:           True
#   A proxy scanning the stream cannot be tricked into seeing a request.

# Choosing between them, by the numbers.
def overhead(messages, msg_bytes, pattern):
    if pattern == "poll":     return messages * (msg_bytes + 800)   # full headers
    if pattern == "sse":      return messages * (msg_bytes + 10)
    if pattern == "websocket": return messages * (msg_bytes + 6)

print(f"\n{'pattern':<12}{'10k msgs x 50 B':>20}")
for p in ("poll", "sse", "websocket"):
    print(f"{p:<12}{overhead(10_000, 50, p):>18,} B")
# pattern      10k msgs x 50 B
# poll                8,500,000 B
# sse                   600,000 B
# websocket             560,000 B
#   SSE and WebSocket are close on overhead. Choose on DIRECTION,
#   not on bytes: only WebSocket lets the client push.

# SSE: watch a live stream with plain curl -- it is just HTTP.
$ curl -N -H 'Accept: text/event-stream' https://api.example.com/events
# event: build
# data: {"id":91,"state":"running"}
#
# event: build
# data: {"id":91,"state":"passed"}
#
#   -N disables buffering. The blank lines are the event delimiters.

$ curl -sI -H 'Accept: text/event-stream' https://api.example.com/events
# content-type: text/event-stream
# cache-control: no-cache          <- essential: a cached stream is useless
# connection: keep-alive

# Resume after a disconnect -- the server replays from the id:
$ curl -N -H 'Last-Event-ID: 1041' -H 'Accept: text/event-stream' \
       https://api.example.com/events | head -4
# id: 1042
# data: c
#

# WebSocket: the upgrade handshake, and the 101 that ends HTTP:
$ curl -i -N -H 'Connection: Upgrade' -H 'Upgrade: websocket' \
    -H 'Sec-WebSocket-Version: 13' \
    -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' \
    https://ws.example.com/socket
# HTTP/1.1 101 Switching Protocols
# Upgrade: websocket
# Connection: Upgrade
# Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=
#   After this line, the bytes are WebSocket frames, not HTTP.

# A real WebSocket client for testing:
$ websocat wss://ws.example.com/socket
# > {"type":"subscribe","channel":"prices"}
# < {"type":"tick","price":42.5}
# < {"type":"tick","price":42.7}

# Enable per-message compression (not automatic, unlike HTTP):
$ websocat --ws-c-uri=wss://ws.example.com/socket -H='Sec-WebSocket-Extensions: \
permessage-deflate' -
# Sec-WebSocket-Extensions: permessage-deflate

# Count long-lived streams -- the resource both patterns consume:
$ ss -tn state established | grep -c ':443'
# 12043      <- each is server memory that cannot be released

Key takeaways

SSE stays inside HTTP: a GET whose response body never ends, with events delimited by blank lines and a trivially simple event:/data:/id: format.
SSE gives automatic reconnection with Last-Event-ID resumption for free, and inherits HTTP's proxies, auth, and HTTP/2 multiplexing.
SSE is one-way and text-only; binary payloads must be base64-encoded at a 33% cost.
WebSocket upgrades away from HTTP via 101 Switching Protocols, becoming a full-duplex, message-framed, binary-capable channel.
The Sec-WebSocket-Key/Accept exchange is a handshake proof, not encryption; client-to-server masking is mandatory to prevent cache-poisoning via crafted payloads.
WebSocket forfeits caching, automatic reconnection, standard status codes, and easy load balancing.
Default to SSE for one-way feeds; choose WebSocket when the client must send frequently or with low latency.

🧪 Practice

Write the exact SSE bytes for an event named alert with id 77 and a two-line data payload.

Explain what Sec-WebSocket-Accept proves and why the value is computed from a fixed GUID rather than a secret.

For each of these, choose SSE or WebSocket with a one-line reason: a build-log tail, a collaborative document editor, a stock ticker, a browser-based SSH terminal.

Interview-style: Why is client-to-server masking mandatory in WebSocket when the connection is already TLS-protected? (Hint: the threat is not an eavesdropper — it is an intermediary that thinks it is still reading HTTP.)

8.4 Messaging and File Transfer

Email and file transfer are the internet's oldest application protocols, and their designs — store-and-forward delivery, text encodings, retrofitted authentication — still shape how these systems behave today.

SMTP and Mail Delivery Path

Theory

Email predates the web by more than a decade, and its architecture reflects a world where machines were frequently offline. The defining property is store-and-forward: a message is handed from server to server, each one taking full responsibility for it before the previous one lets go. If the next hop is unreachable, the message is queued and retried for days rather than failing.

This is fundamentally different from HTTP. An HTTP request either succeeds or fails now. An SMTP message is accepted for later delivery, which is why "sent" and "delivered" are genuinely different states and why bounce messages exist at all.

The cast of components, with their standard abbreviations:

Component	Role
MUA (Mail User Agent)	The client — Thunderbird, Outlook, a webmail UI
MSA (Mail Submission Agent)	Accepts authenticated mail from users, port 587
MTA (Mail Transfer Agent)	Relays between servers, port 25
MDA (Mail Delivery Agent)	Writes into the recipient's mailbox
MX record	DNS pointer naming a domain's receiving MTA

The path of one message:

  alice@a.com                                       bob@b.com
     MUA --submit(587, auth, TLS)--> MSA/MTA(a.com)
                                        |
                              DNS: MX for b.com?
                                        |
                                   relay(25, STARTTLS)
                                        v
                                     MTA(b.com) --> MDA --> mailbox
                                                              ^
                                             MUA <--IMAP/POP--+

The port split matters and is a common configuration error. Port 25 is for server-to-server relay and is blocked outbound by nearly every residential ISP to limit spam. Port 587 is for authenticated submission by users — it requires credentials. Port 465 is implicit-TLS submission, deprecated, then un-deprecated by RFC 8314, and is now recommended again. A user configuring an outbound server on port 25 is the classic "my mail will not send from home" problem.

SMTP itself is a line-oriented text dialogue with three-digit reply codes, structurally similar to HTTP status codes: 2xx success, 3xx more input needed, 4xx temporary failure (retry — this is what greylisting exploits), 5xx permanent failure (bounce).

The subtlety that causes the most confusion is the two sets of addresses. The envelope (MAIL FROM, RCPT TO) is what SMTP actually routes on; the headers (From:, To:, Cc:) are what the recipient's mail client displays. They need not match at all. That is how BCC works and how mailing lists operate — and it is exactly the gap that made sender forgery trivial, motivating SPF, DKIM, and DMARC.

Chapter 7's reliability machinery has an analogue here: SMTP servers implement exponential retry over hours or days, and issue a delayed delivery notification before finally bouncing. Reliability at this layer is measured in days, not milliseconds.

Examples

The full delivery path, with ports and protocols

  [alice's MUA]
       |  SMTP submission, port 587, AUTH required, STARTTLS
       v
  [MSA / MTA  a.com]  ---- queue ----+
       |                              |  retry for up to 5 days
       |  DNS: MX b.com?              |  if the next hop is down
       |  -> 10 mail.b.com            |
       |                              |
       |  SMTP relay, port 25, STARTTLS (opportunistic)
       v
  [MTA  b.com]
       |
       v
  [MDA] --> /var/mail/bob  or  Maildir/
                  ^
                  |  IMAP (143/993) or POP3 (110/995)
            [bob's MUA]

  PORTS -- the distinction that breaks home setups:
    25  server-to-server relay      blocked outbound by most ISPs
    587 authenticated submission    what a USER should configure
    465 implicit-TLS submission     deprecated, then recommended again


  ENVELOPE vs HEADERS -- two different address sets

  ENVELOPE (what SMTP routes on)      HEADERS (what the user sees)
    MAIL FROM:<bounce@lists.org>        From: alice@a.com
    RCPT TO:<bob@b.com>                 To: everyone@lists.org
    RCPT TO:<carol@c.com>               (carol is not listed: this is BCC)

  Nothing in base SMTP requires these to agree. That gap is why
  sender forgery was trivial for thirty years.

# An SMTP conversation, line by line. Note the reply-code classes.

def smtp_session():
    return [
        ("S", "220 mail.b.com ESMTP Postfix"),          # greeting
        ("C", "EHLO mail.a.com"),                       # extended HELO
        ("S", "250-mail.b.com"),                        # capability list
        ("S", "250-STARTTLS"),
        ("S", "250-SIZE 52428800"),
        ("S", "250 8BITMIME"),
        ("C", "STARTTLS"),
        ("S", "220 Ready to start TLS"),                # everything after: encrypted
        ("C", "MAIL FROM:<alice@a.com>"),               # ENVELOPE sender
        ("S", "250 2.1.0 Ok"),
        ("C", "RCPT TO:<bob@b.com>"),                   # ENVELOPE recipient
        ("S", "250 2.1.5 Ok"),
        ("C", "DATA"),
        ("S", "354 End data with <CR><LF>.<CR><LF>"),   # 3xx: send more
        ("C", "From: Alice <alice@a.com>"),             # HEADER sender (may differ)
        ("C", "To: Bob <bob@b.com>"),
        ("C", "Subject: Lunch"),
        ("C", ""),                                      # blank line ends headers
        ("C", "Are you free at 1pm?"),
        ("C", "."),                                     # lone dot terminates DATA
        ("S", "250 2.0.0 Ok: queued as 4Wx2k13Qq"),     # ACCEPTED, not delivered
        ("C", "QUIT"),
        ("S", "221 2.0.0 Bye"),
    ]

for who, line in smtp_session():
    print(f"{'-->' if who == 'C' else '<--'} {line}")

# Reply-code semantics decide what a sender does next.
def interpret(code):
    c = int(str(code)[0])
    return {2: "success -- proceed",
            3: "send more data",
            4: "TEMPORARY failure -- QUEUE AND RETRY",
            5: "PERMANENT failure -- bounce to sender"}[c]

for code in (250, 354, 421, 450, 550, 552):
    print(f"\n{code}: {interpret(code)}")
# 250: success -- proceed
# 354: send more data
# 421: TEMPORARY failure -- QUEUE AND RETRY
# 450: TEMPORARY failure -- QUEUE AND RETRY
# 550: PERMANENT failure -- bounce to sender
# 552: PERMANENT failure -- bounce to sender
#   Greylisting exploits 4xx: reject once, and real senders retry while
#   most spam software does not.

# The retry schedule: reliability measured in DAYS.
def retry_schedule(max_days=5):
    delay, elapsed, attempts = 60, 0, []
    while elapsed < max_days * 86400:
        attempts.append(elapsed)
        elapsed += delay
        delay = min(delay * 2, 4 * 3600)      # exponential, capped at 4 hours
    return attempts

s = retry_schedule()
print(f"\n{len(s)} delivery attempts over 5 days")
print("first few at:", [f"{t//60}m" for t in s[:6]])
print("last few at: ", [f"{t//3600}h" for t in s[-3:]])
# 33 delivery attempts over 5 days
# first few at: ['0m', '1m', '3m', '7m', '15m', '31m']
# last few at:  ['112h', '116h', '120h']
#   A warning is typically sent at 4 hours; a bounce only at the end.

# Envelope vs header: how BCC actually works.
envelope_rcpts = ["bob@b.com", "carol@c.com"]     # SMTP routes to BOTH
header_to      = ["bob@b.com"]                     # carol sees no trace of herself
print(f"\nSMTP delivers to : {envelope_rcpts}")
print(f"Recipients see   : To: {header_to}")
print("carol@c.com received it; nothing in the message says so.")

# Find where a domain's mail actually goes:
$ dig +short example.com MX
# 10 mail.example.com.
# 20 backup.example.com.
#   Lower number preferred. The backup accepts mail when the primary is down.

# Have a full SMTP conversation by hand:
$ openssl s_client -starttls smtp -connect mail.example.com:587 -quiet
# 250-mail.example.com
# 250-STARTTLS
# 250 8BITMIME
# EHLO test.local
# MAIL FROM:<alice@a.com>
# 250 2.1.0 Ok
# RCPT TO:<bob@b.com>
# 250 2.1.5 Ok

# Check which submission ports a provider offers:
$ for p in 25 465 587; do
>   echo -n "port $p: "; timeout 3 bash -c "</dev/tcp/mail.example.com/$p" \
>     && echo open || echo blocked
> done
# port 25: blocked        <- your ISP blocks outbound 25
# port 465: open
# port 587: open          <- configure THIS one in your mail client

# Trace a message's actual path -- read Received headers BOTTOM to TOP:
$ formail -X Received: < message.eml | tac
# Received: from alice-laptop by mail.a.com; Mon, 23 Sep 2026 09:01:02 +0000
# Received: from mail.a.com by mail.b.com; Mon, 23 Sep 2026 09:01:09 +0000
# Received: from mail.b.com by mda.b.com; Mon, 23 Sep 2026 09:01:10 +0000
#   Each hop PREPENDS its own line, so the oldest is at the bottom.

# Inspect a mail queue and why messages are stuck:
$ postqueue -p | tail -4
# 4Wx2k13Qq  2841 Mon Sep 23 09:01:02  alice@a.com
#   (connect to mail.b.com[203.0.113.9]:25: Connection timed out)
#                                             ^ 4xx-equivalent: still retrying

# Check whether relay is encrypted in transit:
$ grep 'status=sent' /var/log/mail.log | tail -1
# to=<bob@b.com>, relay=mail.b.com[203.0.113.9]:25, status=sent (250 2.0.0 Ok)

Key takeaways

Email is store-and-forward: each server takes responsibility and queues with retries over days, so "sent" and "delivered" are genuinely different states.
The components are MUA → MSA → MTA → MTA → MDA → mailbox, with MX records naming a domain's receiving server.
Port 25 is server relay (usually blocked outbound by ISPs), 587 is authenticated submission, 465 is implicit-TLS submission — misconfiguring this is the classic "cannot send from home" fault.
SMTP reply codes mirror HTTP classes: 4xx means retry later (the basis of greylisting), 5xx means bounce.
The envelope (MAIL FROM/RCPT TO) routes the message; the headers (From:/To:) are only what the client displays — they need not match.
That envelope/header gap is how BCC works and why sender forgery was trivial, motivating SPF, DKIM, and DMARC.

🧪 Practice

Write the full SMTP command sequence to send a one-line message, including where the headers begin and how DATA is terminated.

Explain why a home user cannot usually send mail through port 25 and which port they should configure instead.

Given a message with envelope recipients x@a.com, y@b.com and header To: x@a.com, explain what each recipient sees and why.

Interview-style: A user insists an email was "sent" but the recipient never received it and there was no bounce. (Hint: which state does a 250 queued as ... actually confirm, and where would you look to find out what happened next?)

POP3 and IMAP

Theory

SMTP delivers mail to a server. Something else must let a user read it, and the two protocols that do differ in a way that goes far deeper than feature lists: they disagree about where the mailbox lives.

POP3 (Post Office Protocol version 3, ports 110 / 995 with TLS) assumes the mailbox lives on your device. The server is a temporary holding area — a post office box. The client connects, downloads everything, deletes it from the server, and disconnects. Simple, minimal server storage, and the mail is yours offline.

IMAP (Internet Message Access Protocol, ports 143 / 993 with TLS) assumes the mailbox lives on the server. The client is a window onto it. Messages stay where they are; the client synchronises state — folders, read/unread flags, deletions — so every device sees the same mailbox.

That single architectural difference generates everything else:

	POP3	IMAP
Mailbox of record	The client	The server
Multiple devices	Broken — first device takes the mail	Designed for it
Server storage	Minimal	Full mailbox, grows forever
Folders	None (inbox only)	Full hierarchy
Flags (read, flagged)	None	Synchronised
Partial fetch	No — whole message or nothing	Headers, one MIME part, a range
Server-side search	No	Yes
Offline	Native	Needs explicit caching
Push notification	No	IDLE

Two IMAP capabilities deserve emphasis because they are what make modern mail clients feel instant.

Partial fetch. IMAP can retrieve just the headers of 500 messages to draw a list, then fetch one body when it is opened, then fetch a 10 MB attachment only if the user asks. POP3 must download the entire message to show its subject line. On a mobile connection this is the difference between a usable client and an unusable one.

IDLE (RFC 2177) turns polling into push. The client issues IDLE and the connection stays open; when mail arrives the server sends an untagged update immediately. This is the same request-response-versus-streaming trade-off from 8.1, solved the same way — at the cost of a held connection per client, which is why mail servers care so much about per-connection memory.

POP3 is not obsolete so much as specialised. It remains the right answer when there is exactly one device, when server storage is constrained or expensive, or when a local archive is the point. Everywhere else — which is nearly everywhere — IMAP won because people have more than one device.

JMAP (RFC 8620/8621) is the modern successor: a JSON-over-HTTP protocol that keeps IMAP's server-of-record model while replacing its stateful, connection-oriented design with something that fits contemporary mobile and web clients.

Examples

The architectural difference, drawn

 POP3: the mailbox is on YOUR DEVICE
                                             the server after download:
   [server] --download all--> [laptop]       +-----------+
             delete                          |  (empty)  |
                                             +-----------+
   [phone] connects later --> nothing there. The laptop has it all.
                              ^ this is not a bug; it is the model


 IMAP: the mailbox is on the SERVER
                                             the server always:
   [server] <--sync--> [laptop]              +--------------+
       ^                                     | INBOX (412)  |
       +------sync-----> [phone]             | Sent  (208)  |
       ^                                     | Archive(9k)  |
       +------sync-----> [webmail]           +--------------+
   Read it on the phone -> it is read on the laptop too.


 PARTIAL FETCH: why IMAP feels fast on mobile

   Drawing a 500-message list:
     POP3: RETR every message  = 500 x ~80 KB = 40 MB
     IMAP: FETCH 1:500 (ENVELOPE) = ~50 KB       <- 800x less

   Opening one message with a 10 MB attachment:
     IMAP: FETCH 42 BODY[1]      = just the text part
           FETCH 42 BODY[2]      = only if the user taps the attachment

# POP3 and IMAP sessions side by side -- the commands reveal the model.

pop3 = [
    ("C", "USER bob"),           ("S", "+OK"),
    ("C", "PASS secret"),        ("S", "+OK mailbox ready"),
    ("C", "STAT"),               ("S", "+OK 3 14200"),      # 3 messages, 14200 B
    ("C", "RETR 1"),             ("S", "+OK 4821 octets"),  # WHOLE message
    ("C", "DELE 1"),             ("S", "+OK marked deleted"),
    ("C", "QUIT"),               ("S", "+OK gone"),         # deletions commit here
]
imap = [
    ("C", "a1 LOGIN bob secret"),          ("S", "a1 OK LOGIN completed"),
    ("C", "a2 LIST \"\" *"),               ("S", "* LIST (\\HasNoChildren) \"/\" INBOX"),
                                           ("S", "* LIST (\\HasNoChildren) \"/\" Archive"),
    ("C", "a3 SELECT INBOX"),              ("S", "* 412 EXISTS"),
                                           ("S", "* 3 RECENT"),
    ("C", "a4 FETCH 1:500 (ENVELOPE)"),    ("S", "* 1 FETCH (ENVELOPE (...))"),
    ("C", "a5 FETCH 42 BODY[1]"),          ("S", "* 42 FETCH (BODY[1] {182}...)"),
    ("C", "a6 STORE 42 +FLAGS (\\Seen)"),  ("S", "a6 OK STORE completed"),
    ("C", "a7 SEARCH SINCE 1-Sep-2026 FROM alice"), ("S", "* SEARCH 91 104 233"),
    ("C", "a8 IDLE"),                      ("S", "+ idling"),
                                           ("S", "* 413 EXISTS"),   # PUSH: new mail
]

for name, session in (("POP3", pop3), ("IMAP", imap)):
    print(f"\n=== {name} ===")
    for who, line in session:
        print(f"{'-->' if who == 'C' else '<--'} {line}")
#   Note what only IMAP has: LIST (folders), FETCH BODY[n] (partial),
#   STORE FLAGS (synchronised state), SEARCH (server-side), IDLE (push).

# The bandwidth consequence of partial fetch.
def list_view_bytes(messages, avg_msg_kb=80, envelope_bytes=100):
    return {"POP3 (whole messages)": messages * avg_msg_kb * 1024,
            "IMAP (envelopes only)": messages * envelope_bytes}

for k, v in list_view_bytes(500).items():
    print(f"\n{k:<26} {v:>12,} bytes")
# POP3 (whole messages)        40,960,000 bytes
# IMAP (envelopes only)            50,000 bytes
#   819x. On a mobile link this is the difference between 3 s and 40 minutes.

# IDLE versus polling: the same ratio argument as request-response vs streaming.
def mail_check_cost(hours, poll_interval_s, req_bytes=250):
    polls = hours * 3600 / poll_interval_s
    return polls, polls * req_bytes

for interval in (60, 300):
    polls, b = mail_check_cost(24, interval)
    print(f"\npolling every {interval:>3}s: {polls:>6,.0f} checks/day, {b:>9,.0f} bytes")
print(f"IMAP IDLE       :      0 checks, 1 held connection, instant delivery")
# polling every  60s:  1,440 checks/day,   360,000 bytes
# polling every 300s:    288 checks/day,    72,000 bytes
# IMAP IDLE       :      0 checks, 1 held connection, instant delivery

# IMAP over TLS, by hand -- note the tagged command/response structure:
$ openssl s_client -crlf -connect imap.example.com:993 -quiet
# * OK [CAPABILITY IMAP4rev1 IDLE MOVE SPECIAL-USE] Dovecot ready.
# a1 LOGIN bob secret
# a1 OK Logged in
# a2 LIST "" *
# * LIST (\HasNoChildren) "/" INBOX
# * LIST (\HasNoChildren \Sent) "/" Sent
# a2 OK List completed
# a3 SELECT INBOX
# * 412 EXISTS
# * 3 RECENT
# a3 OK [READ-WRITE] Select completed

# Fetch only headers -- the capability POP3 lacks entirely:
# a4 FETCH 1:5 (BODY.PEEK[HEADER.FIELDS (FROM SUBJECT DATE)])
#   BODY.PEEK, not BODY: PEEK avoids setting the \Seen flag.

# Server-side search -- no messages are downloaded at all:
# a5 SEARCH SINCE 1-Sep-2026 LARGER 1000000 FROM "alice"
# * SEARCH 91 104 233

# POP3 for comparison -- observe how little it can do:
$ openssl s_client -crlf -connect pop.example.com:995 -quiet
# +OK POP3 ready
# USER bob
# +OK
# PASS secret
# +OK maildrop has 3 messages (14200 octets)
# LIST
# +OK 3 messages
# 1 4821
# 2 5102
# .
#   No folders, no flags, no search, no partial fetch. By design.

# Check which capabilities a server advertises before choosing a client config:
$ openssl s_client -connect imap.example.com:993 -quiet 2>/dev/null <<< 'a CAPABILITY
a LOGOUT' | head -2
# * CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE MOVE
#                                                              ^^^^ push available

# Compare server storage between the two models:
$ du -sh /var/mail/bob            # POP3-style: drained after each fetch
# 12K   /var/mail/bob
$ du -sh /var/vmail/bob/Maildir   # IMAP: the mailbox of record, grows forever
# 8.4G  /var/vmail/bob/Maildir

Key takeaways

The difference is architectural: POP3 puts the mailbox on the client, IMAP puts it on the server — every feature difference follows from that.
POP3 downloads and deletes, which breaks multi-device use by design; IMAP synchronises folders, flags, and deletions across every client.
IMAP's partial fetch (envelopes, one MIME part, byte ranges) is what makes mail clients usable on slow links — POP3 must fetch whole messages.
IMAP supports server-side search and IDLE for push, at the cost of a held connection per client.
POP3 remains reasonable for a single device, constrained server storage, or a deliberate local archive.
JMAP is the modern successor: IMAP's server-of-record model over JSON/HTTP, better suited to mobile and web clients.

🧪 Practice

A user reads mail on a phone and a laptop and finds messages missing on one of them. Identify the likely protocol and explain the mechanism.

Compute the bytes needed to display a 1,000-message list under POP3 (80 KB average) and IMAP (100-byte envelopes).

Explain what BODY.PEEK does differently from BODY and why a client would use it.

Interview-style: Why does a mail provider care much more about concurrent connection count with IMAP than with POP3? (Hint: consider what IDLE asks the server to hold open, and multiply by the user base.)

MIME and Message Formats

Theory

SMTP was specified for 7-bit US-ASCII text, with lines no longer than 998 characters. That was a reasonable assumption in 1982 and a severe limitation by 1992: no accented characters, no non-Latin scripts, no images, no attachments, no way to send a message in both plain text and HTML.

Rewriting SMTP was impossible — too many deployed servers. MIME (Multipurpose Internet Mail Extensions, RFC 2045–2049) is instead a layer above it: a set of header conventions and encodings that express arbitrary content within the 7-bit constraint. Every existing server kept working unchanged. This is one of the cleanest examples of extending a protocol you cannot modify.

MIME contributes four things:

1. MIME-Version: 1.0 — a marker saying the rest is MIME-structured.

2. Content-Type — a type/subtype pair with parameters: text/plain; charset=UTF-8, image/png, application/pdf. This is the same media-type registry HTTP uses, and it originated here.

3. Content-Transfer-Encoding — how binary was squeezed through a 7-bit pipe:

Encoding	Mechanism	Overhead	Suits
7bit	None; already safe	0%	ASCII text
8bit	None; needs `8BITMIME`	0%	UTF-8 where supported
quoted-printable	Non-ASCII bytes become `=XX`	~3% on mostly-ASCII	Text with a few accents
base64	3 bytes → 4 ASCII characters	+33%	Images, PDFs, any binary

The 33% base64 overhead is why a 10 MB attachment consumes about 13.7 MB of transfer and why mail size limits are stated in odd numbers.

4. Multipart structures — a message becomes a tree, with parts separated by a boundary string that must not occur in any part's content:

multipart/alternative — the same content in several formats; the client displays the best one it can, and by convention the richest version goes last.
multipart/mixed — different content: a body plus attachments.
multipart/related — parts that reference each other, such as HTML with inline images addressed by Content-ID.

Non-ASCII in headers needed its own solution, since headers are not MIME bodies: RFC 2047 encoded-words, of the form =?UTF-8?B?<base64>?= or =?UTF-8?Q?<quoted-printable>?=. This is why a raw email source shows subject lines like =?UTF-8?B?w4ViZW5kZXNzZW4=?=.

MIME's influence extends well beyond mail. HTTP's Content-Type, the media-type registry, multipart/form-data file uploads, and Content-Disposition are all MIME, reused. Learning it once pays off in both protocols.

Examples

A complete MIME message: a tree, flattened with boundaries

  From: Alice <alice@a.com>
  To: Bob <bob@b.com>
  Subject: =?UTF-8?B?w4ViZW5kZXNzZW4=?=      <- RFC 2047 encoded header
  MIME-Version: 1.0
  Content-Type: multipart/mixed; boundary="OUTER"
                                 ^^^^^^^^^^^^^^^^ must not appear in any part

  --OUTER
  Content-Type: multipart/alternative; boundary="INNER"

  --INNER
  Content-Type: text/plain; charset=UTF-8
  Content-Transfer-Encoding: quoted-printable

  Caf=C3=A9 at 1pm?                          <- "é" as =C3=A9

  --INNER
  Content-Type: text/html; charset=UTF-8
  Content-Transfer-Encoding: quoted-printable

  <p>Caf=C3=A9 at 1pm?</p>                   <- RICHEST version LAST
  --INNER--
  --OUTER
  Content-Type: image/png; name="map.png"
  Content-Transfer-Encoding: base64
  Content-Disposition: attachment; filename="map.png"

  iVBORw0KGgoAAAANSUhEUgAAAAEAAAAB...        <- +33% over the raw bytes
  --OUTER--


  The tree this represents:

     multipart/mixed
       |-- multipart/alternative        "pick ONE of these"
       |     |-- text/plain
       |     +-- text/html              <- preferred (listed last)
       +-- image/png                    "and ALSO this"

# Building and inspecting MIME with the standard library.
from email.message import EmailMessage
from email.header import Header, decode_header
import base64, quopri

msg = EmailMessage()
msg["From"] = "Alice <alice@a.com>"
msg["To"] = "Bob <bob@b.com>"
msg["Subject"] = "Äbendessen"                  # non-ASCII: will be encoded-word
msg.set_content("Café at 1pm?")                 # text/plain part
msg.add_alternative("<p>Café at 1pm?</p>", subtype="html")   # -> multipart/alternative
msg.add_attachment(b"\x89PNG\r\n\x1a\n" + b"\x00" * 200,
                   maintype="image", subtype="png", filename="map.png")

print("Structure:")
for part in msg.walk():
    depth = "  " * len(part.get_content_type().split("/"))
    print(f"{depth}{part.get_content_type():<28}"
          f"cte={part.get('Content-Transfer-Encoding', '-')}")
# Structure:
#     multipart/mixed             cte=-
#     multipart/alternative       cte=-
#     text/plain                  cte=base64
#     text/html                   cte=base64
#     image/png                   cte=base64

print(f"\nSubject on the wire: {msg['Subject']}")
# Subject on the wire: =?utf-8?q?=C3=84bendessen?=
print(f"Decoded            : {decode_header(msg['Subject'])[0][0].decode('utf-8')}")
# Decoded            : Äbendessen

# The base64 tax, quantified -- this is why attachment limits look arbitrary.
def encoded_size(raw_bytes, encoding):
    if encoding == "base64":
        return (raw_bytes + 2) // 3 * 4 * 1.0138    # +33%, plus line breaks
    if encoding == "quoted-printable":
        return raw_bytes * 1.03                      # ~3% for mostly-ASCII text
    return raw_bytes

for mb in (1, 10, 25):
    raw = mb * 1024 * 1024
    b64 = encoded_size(raw, "base64")
    print(f"{mb:>3} MB attachment -> {b64/1024/1024:>6.2f} MB on the wire "
          f"(+{(b64/raw - 1):.0%})")
#   1 MB attachment ->   1.35 MB on the wire (+35%)
#  10 MB attachment ->  13.52 MB on the wire (+35%)
#  25 MB attachment ->  33.79 MB on the wire (+35%)
#   A "25 MB limit" usually means ~18 MB of actual file.

# Choosing an encoding: it depends on how much of the content is ASCII.
def best_encoding(data: bytes):
    non_ascii = sum(1 for b in data if b > 127 or b in (0, 13, 10))
    ratio = non_ascii / max(len(data), 1)
    return "7bit" if ratio == 0 else ("quoted-printable" if ratio < 0.2 else "base64")

samples = [("plain ASCII", b"Hello, world"),
           ("mostly ASCII", "Café at 1pm?".encode()),
           ("binary PNG", b"\x89PNG\r\n\x1a\n\xff\xd8\xff")]
for name, data in samples:
    enc = best_encoding(data)
    print(f"\n{name:<14} -> {enc:<18} "
          f"({encoded_size(len(data), enc)/len(data):.2f}x size)")
# plain ASCII    -> 7bit               (1.00x size)
# mostly ASCII   -> quoted-printable   (1.03x size)
# binary PNG     -> base64             (1.42x size)

# Boundary selection: it MUST NOT occur in any part's content.
import secrets
def safe_boundary(parts):
    while True:
        b = "=_" + secrets.token_hex(16)
        if not any(b.encode() in p for p in parts):
            return b
print(f"\nboundary: {safe_boundary([b'some content', b'more content'])}")
# boundary: =_9f2a8c1d4b7e3a5f0c2e6d8b1a4f7c93
#   A collision would truncate the message at the wrong place.

# See the raw MIME structure of a message:
$ python3 -c "
import email, sys
m = email.message_from_file(open('message.eml'))
for p in m.walk():
    print(f\"{p.get_content_type():<30}{p.get('Content-Transfer-Encoding','-')}\")"
# multipart/mixed               -
# multipart/alternative         -
# text/plain                    quoted-printable
# text/html                     quoted-printable
# application/pdf               base64

# Extract just the attachments:
$ munpack -f message.eml
# report.pdf (application/pdf)

# Decode a quoted-printable body by hand:
$ echo 'Caf=C3=A9 at 1pm?' | python3 -c \
  "import sys,quopri; sys.stdout.buffer.write(quopri.decodestring(sys.stdin.read()))"
# Café at 1pm?

# Decode an RFC 2047 encoded-word subject:
$ echo '=?UTF-8?B?w4ViZW5kZXNzZW4=?=' | python3 -c \
  "import sys; from email.header import decode_header; \
   print(decode_header(sys.stdin.read().strip())[0][0].decode())"
# Äbendessen

# Measure the base64 tax on a real file:
$ ls -l photo.jpg && base64 photo.jpg | wc -c
# -rw-r--r-- 1 user user 2097152 photo.jpg
# 2834775
#   2.0 MB -> 2.7 MB. The 33% is unavoidable with base64.

# The same MIME machinery in HTTP -- multipart/form-data uploads:
$ curl -v -F 'comment=hello' -F 'file=@photo.jpg' https://example.com/upload 2>&1 \
  | grep -iE 'content-type|boundary' | head -2
# > Content-Type: multipart/form-data; boundary=------------------------abc123
#   Identical structure, different protocol. MIME is reused wholesale.

Key takeaways

MIME extends SMTP without changing it: a header-and-encoding layer that expresses arbitrary content within a 7-bit, ASCII-only transport.
Content-Type names the media type, Content-Transfer-Encoding says how it was made 7-bit-safe, and MIME-Version marks the message as MIME.
base64 costs +33%; quoted-printable costs ~3% on mostly-ASCII text — which is why stated attachment limits do not match real file sizes.
multipart/alternative carries the same content in several formats (richest last); multipart/mixed carries different content; multipart/related links inline resources.
The boundary string must not appear in any part, or the message is truncated at the wrong point.
Non-ASCII headers use RFC 2047 encoded-words (=?UTF-8?B?...?=), a separate mechanism from body encoding.
MIME is reused wholesale by HTTP: media types, multipart/form-data, and Content-Disposition all come from here.

🧪 Practice

Write a complete multipart/alternative message with plain-text and HTML versions, including boundaries and the correct part ordering.

Compute the on-the-wire size of a 15 MB video attachment and explain why a 20 MB server limit rejects it.

Decode =?UTF-8?Q?Caf=C3=A9?= and state which encoding it uses and why that one was chosen.

Interview-style: MIME added images and international text to email without changing a single SMTP server. Explain how, and name one other protocol that reuses the result. (Hint: the constraint was 7-bit ASCII — ask what layer the fix was placed at.)

SPF, DKIM, and DMARC

Theory

Base SMTP has no authentication whatsoever. Any server may claim any MAIL FROM, and the visible From: header is entirely independent of it. For three decades, forging mail from any address required no skill at all. SPF, DKIM, and DMARC are the retrofit, and they work in layers because each fixes a different half of the problem.

SPF (Sender Policy Framework) authorises servers. A domain publishes a TXT record listing which IP addresses may send mail bearing its name:

v=spf1 ip4:203.0.113.0/24 include:_spf.provider.com -all

The receiver checks the connecting server's IP against that list. The qualifiers matter: -all fails anything not listed, ~all softfails (accept but mark), ?all is neutral. SPF's fatal flaw is that it validates the envelope sender and is evaluated against the connecting IP — so it breaks on forwarding. When a mailing list or a .forward rule relays your message, the connecting IP is now the forwarder's, which your SPF record does not list.

DKIM (DomainKeys Identified Mail) authenticates content. The sending server signs selected headers and the body with a private key and adds a DKIM-Signature header; the public key is published in DNS at <selector>._domainkey.<domain>. The receiver fetches it and verifies. Because the signature travels with the message, DKIM survives forwarding — as long as nothing modifies the signed headers or body, which mailing lists that append footers routinely do.

DMARC (Domain-based Message Authentication, Reporting and Conformance) ties the two to the header the user actually sees. Neither SPF nor DKIM checks From: — SPF checks the envelope, DKIM checks whatever domain signed. DMARC adds two things:

Alignment: the SPF or DKIM domain must match the From: header domain. This is the piece that actually stops user-visible forgery.
Policy and reporting: what a receiver should do on failure (p=none, p=quarantine, p=reject), plus aggregate reports (rua=) telling the domain owner who is sending as them.

A message passes DMARC if SPF passes and aligns, OR DKIM passes and aligns. Either suffices, which is exactly what makes the pair robust: forwarding breaks SPF but preserves DKIM; a mailing list that rewrites the body breaks DKIM but may preserve SPF if it also rewrites the envelope sender.

The deployment path is universally p=none first — collecting reports to discover every legitimate sender, including the marketing platform and the ticketing system nobody remembered — then p=quarantine, then p=reject. Going straight to p=reject reliably blackholes some department's legitimate mail.

Examples

What each layer authenticates, and where each one breaks

              ENVELOPE                    HEADERS                  BODY
              MAIL FROM:<x@a.com>         From: y@b.com            ...
                   ^                           ^                    ^
                   |                           |                    |
   SPF -----------+                            |                    |
   checks the connecting IP                    |                    |
   against a.com's TXT record                  |                    |
   BREAKS ON FORWARDING (IP changes)           |                    |
                                               |                    |
   DKIM ---------------------------------------+--------------------+
   signature over selected headers + body, key in DNS
   SURVIVES FORWARDING; breaks if a list appends a footer
                                               |
   DMARC --------------------------------------+
   requires SPF or DKIM to ALIGN with the From: domain
   -> this is the layer that stops USER-VISIBLE forgery


 THE ALIGNMENT GAP THAT DMARC CLOSES

   MAIL FROM: <bounce@attacker.com>     SPF: PASS (attacker.com authorises it!)
   From:      "Your Bank" <bank.com>    user sees: bank.com
                                        DMARC: FAIL -- domains do not align


 DEPLOYMENT LADDER (never skip steps)

   p=none        collect rua reports, find every legitimate sender
       |         (marketing platform? ticketing system? the CI server?)
   p=quarantine  failures go to spam; watch reports for false positives
       |
   p=reject      failures are refused outright

# Evaluating SPF, DKIM, and DMARC alignment -- the logic a receiver runs.
import ipaddress

SPF_RECORDS = {"a.com": "v=spf1 ip4:203.0.113.0/24 include:_spf.provider.com -all"}
INCLUDES    = {"_spf.provider.com": ["198.51.100.0/24"]}

def check_spf(domain, connecting_ip):
    record = SPF_RECORDS.get(domain)
    if not record:
        return "none"
    nets = []
    qualifier = "neutral"
    for term in record.split()[1:]:
        if term.startswith("ip4:"):
            nets.append(term[4:])
        elif term.startswith("include:"):
            nets += INCLUDES.get(term[8:], [])
        elif term.endswith("all"):
            qualifier = {"-": "fail", "~": "softfail", "?": "neutral",
                         "+": "pass"}[term[0] if term[0] in "-~?+" else "+"]
    ip = ipaddress.ip_address(connecting_ip)
    if any(ip in ipaddress.ip_network(n) for n in nets):
        return "pass"
    return qualifier

for ip in ("203.0.113.9", "198.51.100.4", "6.6.6.6"):
    print(f"SPF a.com from {ip:<14} -> {check_spf('a.com', ip)}")
# SPF a.com from 203.0.113.9    -> pass
# SPF a.com from 198.51.100.4   -> pass      (via include:)
# SPF a.com from 6.6.6.6        -> fail      (-all)

def dmarc_evaluate(from_domain, spf_domain, spf_result,
                   dkim_domain, dkim_result, policy="reject", strict=False):
    """DMARC passes if SPF **or** DKIM passes AND aligns with the From: domain."""
    def aligned(d):
        if d is None:
            return False
        return d == from_domain if strict else (
            d == from_domain or d.endswith("." + from_domain))

    spf_ok  = spf_result == "pass" and aligned(spf_domain)
    dkim_ok = dkim_result == "pass" and aligned(dkim_domain)
    passed = spf_ok or dkim_ok
    action = "deliver" if passed else {"none": "deliver (report only)",
                                       "quarantine": "spam folder",
                                       "reject": "refuse at SMTP"}[policy]
    return {"spf_aligned": spf_ok, "dkim_aligned": dkim_ok,
            "dmarc": "pass" if passed else "fail", "action": action}

scenarios = [
    ("legitimate direct send",
     dict(from_domain="a.com", spf_domain="a.com", spf_result="pass",
          dkim_domain="a.com", dkim_result="pass")),
    ("forwarded by a mailing list (SPF breaks, DKIM survives)",
     dict(from_domain="a.com", spf_domain="list.org", spf_result="pass",
          dkim_domain="a.com", dkim_result="pass")),
    ("list rewrote the body (DKIM breaks, SPF misaligned)",
     dict(from_domain="a.com", spf_domain="list.org", spf_result="pass",
          dkim_domain="a.com", dkim_result="fail")),
    ("outright forgery -- SPF passes for the ATTACKER's domain",
     dict(from_domain="bank.com", spf_domain="attacker.com", spf_result="pass",
          dkim_domain=None, dkim_result="none")),
]
for name, kw in scenarios:
    r = dmarc_evaluate(**kw)
    print(f"\n{name}")
    print(f"  spf_aligned={r['spf_aligned']}  dkim_aligned={r['dkim_aligned']}"
          f"  -> DMARC {r['dmarc'].upper()}, action: {r['action']}")
# legitimate direct send
#   spf_aligned=True  dkim_aligned=True  -> DMARC PASS, action: deliver
# forwarded by a mailing list (SPF breaks, DKIM survives)
#   spf_aligned=False dkim_aligned=True  -> DMARC PASS, action: deliver
#     ^ THIS is why you need both: DKIM rescued a forwarded message
# list rewrote the body (DKIM breaks, SPF misaligned)
#   spf_aligned=False dkim_aligned=False -> DMARC FAIL, action: refuse at SMTP
#     ^ the known mailing-list problem; ARC exists to address it
# outright forgery -- SPF passes for the ATTACKER's domain
#   spf_aligned=False dkim_aligned=False -> DMARC FAIL, action: refuse at SMTP
#     ^ SPF alone said "pass". Only ALIGNMENT caught this.

# SPF has a hard limit that silently breaks large deployments:
def count_dns_lookups(record, includes, depth=0):
    n = 0
    for term in record.split():
        if term.startswith(("include:", "a:", "mx:", "exists:", "redirect=")):
            n += 1
            if term.startswith("include:"):
                sub = includes.get(term[8:])
                if sub:
                    n += count_dns_lookups(sub, includes, depth + 1)
    return n

big = ("v=spf1 include:_spf.google.com include:sendgrid.net include:mailgun.org "
       "include:_spf.salesforce.com a mx -all")
print(f"\nSPF DNS lookups: {count_dns_lookups(big, {})} (limit is 10)")
# SPF DNS lookups: 6 (limit is 10)
#   Exceed 10 and the result is PERMERROR -- treated as a failure.

# Read a domain's three policies:
$ dig +short example.com TXT | grep spf1
# "v=spf1 ip4:203.0.113.0/24 include:_spf.provider.com -all"

$ dig +short selector1._domainkey.example.com TXT
# "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC..."
#  ^ the PUBLIC key; the selector lets you rotate keys without downtime

$ dig +short _dmarc.example.com TXT
# "v=DMARC1; p=reject; rua=mailto:dmarc@example.com; pct=100; adkim=r; aspf=r"
#             ^policy   ^aggregate reports          ^100%  ^relaxed alignment

# Read the verdict a receiver recorded in the message itself:
$ formail -X Authentication-Results: < message.eml
# Authentication-Results: mx.google.com;
#        spf=pass (google.com: domain of bounce@a.com designates 203.0.113.9)
#        dkim=pass header.i=@a.com header.s=selector1
#        dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=a.com
#                                                 ^^^^^^^^^^^^^^^ alignment target

# Check your SPF lookup count before it hits the limit of 10:
$ python3 -m spf_check example.com 2>/dev/null || \
  dig +short example.com TXT | grep -o 'include:[^ ]*' | wc -l
# 4

# Verify a DKIM signature on a stored message:
$ dkimverify < message.eml
# signature ok
$ formail -X DKIM-Signature: < message.eml | head -3
# DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=a.com; s=selector1;
#         h=from:to:subject:date:message-id;
#           ^ exactly which headers are covered -- anything else can be altered

# Start safe: report-only policy while you discover your real senders.
# _dmarc.example.com TXT "v=DMARC1; p=none; rua=mailto:dmarc@example.com"
#   Read the reports for a few weeks BEFORE moving to quarantine.

Key takeaways

Base SMTP authenticates nothing; SPF, DKIM, and DMARC are a three-layer retrofit, each fixing a different part.
SPF authorises sending IPs for the envelope domain and breaks on forwarding, because the connecting IP changes.
DKIM signs headers and body with a key published in DNS and survives forwarding, but breaks when a list modifies signed content.
DMARC adds alignment with the visible From: header — the piece that actually prevents user-visible forgery — plus policy (none/quarantine/reject) and aggregate reports.
DMARC passes if SPF or DKIM passes and aligns, which is why deploying both is far more robust than either alone.
Deploy p=none first and read the reports; jumping straight to p=reject reliably blocks legitimate internal senders.
SPF has a hard 10 DNS lookup limit; exceeding it yields PERMERROR, which receivers treat as a failure.

🧪 Practice

Write SPF, DKIM, and DMARC records for a domain that sends from its own 203.0.113.0/24 and from a marketing provider, starting in report-only mode.

Explain why SPF fails for forwarded mail but DKIM usually does not, and what DMARC does with that combination.

Show a forged message where SPF passes yet DMARC fails, and identify precisely which check caught it.

Interview-style: A company deploys p=reject on Monday and by Wednesday its invoices are being refused. (Hint: what does the aggregate report show that nobody looked at, and which systems send mail that IT does not operate?)

FTP and FTPS

Theory

FTP (RFC 959, 1985) is worth studying not because you should deploy it — you should not — but because its design decisions are a catalogue of what the modern internet made unworkable, and understanding why explains a great deal about NAT, firewalls, and protocol design.

FTP's distinguishing feature is that it uses two separate TCP connections:

The control connection (port 21) carries commands and replies, and stays open for the whole session.
A data connection (port 20 or a negotiated port) is opened per transfer — one for each file, and one for each directory listing.

Splitting them was reasonable in 1985: commands remain responsive while a large transfer runs, and the data channel can be a clean byte stream with no framing. It became a liability the moment NAT and stateful firewalls appeared, because something has to decide which side opens the data connection, and on which port.

Active mode was the original design. The client sends PORT h1,h2,h3,h4,p1,p2, announcing an address and port, and the server connects back to the client. Behind NAT this fails twice over: the address the client announces is its private address, and inbound connections to a client are exactly what NAT and firewalls block.

Passive mode (PASV) inverts it: the server announces a port and the client connects out to it. This works through client-side NAT, and is why passive mode became the default. It moves the problem to the server side — the server must expose a range of high ports, and its own firewall must allow them.

Two further problems are fatal for modern use:

Everything is plaintext. Credentials, commands, and file contents all travel in the clear. FTPS adds TLS (AUTH TLS, RFC 4217) and fixes confidentiality, but it does not fix the port problem — worse, it makes it harder, because a firewall that could previously read PASV responses to open the right ports can no longer do so once the control channel is encrypted.

It is firewall-hostile by construction. Stateful firewalls need an FTP ALG (application-layer gateway) that parses the control connection to learn which data ports to permit. This is a layering violation, a maintenance burden, and — once FTPS encrypts the control channel — impossible.

The practical conclusion is straightforward: use SFTP (next topic) for interactive transfer and HTTPS for downloads. FTP survives mostly in legacy industrial systems and anonymous public mirrors. Where it must be kept, FTPS with passive mode and a narrow, explicitly configured port range is the least-bad configuration.

Examples

Two connections, and who dials whom

 ACTIVE MODE (the original -- broken by NAT)

   [client :51000]                         [server :21]
        |------- control: PORT 192,168,1,20,199,72 ------>|
        |                     ^ MY private address!       |
        |<====== data: server CONNECTS BACK to :51000 ====|
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                 blocked by the client's NAT/firewall,
                 and the announced address is unroutable


 PASSIVE MODE (the workaround -- now the default)

   [client]                                [server :21]
        |------- control: PASV -------------------------->|
        |<------ 227 Entering Passive Mode (203,0,113,9,195,80)
        |                   ^ server's address, port = 195*256+80 = 50000
        |======= data: client connects OUT to :50000 =====>|
                 ^^^^^^^^^^^^^^^^^^^^^^^^ works through client NAT

   Cost: the SERVER must expose a whole range of high ports.


 WHY FTPS MAKES FIREWALLS WORSE, NOT BETTER

   plain FTP:  firewall READS "227 ... (203,0,113,9,195,80)"
               -> opens port 50000 automatically (the FTP ALG)

   FTPS:       firewall sees encrypted bytes
               -> cannot learn the port
               -> you must pre-open the entire passive range by hand

# Decoding the PORT/PASV address encoding, and why it breaks behind NAT.

def encode_port_command(ip, port):
    """PORT h1,h2,h3,h4,p1,p2 -- port split into two bytes."""
    return f"PORT {ip.replace('.', ',')},{port >> 8},{port & 0xFF}"

def decode_pasv_reply(reply):
    """227 Entering Passive Mode (h1,h2,h3,h4,p1,p2)"""
    nums = [int(n) for n in reply[reply.index("(") + 1: reply.index(")")].split(",")]
    ip = ".".join(str(n) for n in nums[:4])
    port = nums[4] * 256 + nums[5]
    return ip, port

print(encode_port_command("192.168.1.20", 51000))
# PORT 192,168,1,20,199,56
#   ^ a PRIVATE address announced to a server on the public internet.
#     The server will try to connect to it and fail.

print(decode_pasv_reply("227 Entering Passive Mode (203,0,113,9,195,80)"))
# ('203.0.113.9', 50000)
#   195 * 256 + 80 = 50000

# Why active mode fails: walk through what each party attempts.
import ipaddress
def active_mode_outcome(client_ip):
    private = ipaddress.ip_address(client_ip).is_private
    return ("FAIL: announced a private address; server cannot route to it"
            if private else
            "FAIL: server's inbound connection blocked by client firewall")

for ip in ("192.168.1.20", "10.0.0.5", "203.0.113.55"):
    print(f"active mode, client {ip:<14} -> {active_mode_outcome(ip)}")
# active mode, client 192.168.1.20   -> FAIL: announced a private address; ...
# active mode, client 10.0.0.5       -> FAIL: announced a private address; ...
# active mode, client 203.0.113.55   -> FAIL: server's inbound connection blocked ...
#   Active mode works essentially nowhere on the modern internet.

# The connection count: one data connection PER operation.
def connections_for(operations):
    total = 1                                  # the control connection
    detail = ["control (port 21, stays open)"]
    for op in operations:
        total += 1
        detail.append(f"data connection for: {op}")
    return total, detail

ops = ["LIST /", "RETR a.txt", "RETR b.txt", "LIST /sub", "RETR c.txt"]
n, detail = connections_for(ops)
print(f"\n{n} TCP connections for {len(ops)} operations:")
for d in detail:
    print(f"  {d}")
# 6 TCP connections for 5 operations:
#   control (port 21, stays open)
#   data connection for: LIST /
#   ... one per transfer AND per listing
#   Compare: SFTP does all of this over ONE connection.

# Firewall port range sizing for a passive FTPS server.
def passive_range(concurrent_transfers, safety=3):
    return concurrent_transfers * safety

for users in (10, 100, 1000):
    print(f"\n{users:>4} concurrent transfers -> open {passive_range(users):>4} "
          f"high ports on the firewall")
#   10 concurrent transfers -> open   30 high ports on the firewall
#  100 concurrent transfers -> open  300 high ports on the firewall
# 1000 concurrent transfers -> open 3000 high ports on the firewall
#   Every one of these is inbound-permitted. SFTP needs exactly ONE: 22.

# A plaintext FTP session -- observe the credentials in the clear:
$ ftp -n ftp.example.com <<'EOF'
> user anonymous test@example.com
> passive
> ls
> get README
> quit
> EOF
# 230 Login successful.
# 227 Entering Passive Mode (203,0,113,9,195,80).
# 150 Opening BINARY mode data connection for README (1024 bytes).
# 226 Transfer complete.

# What an observer sees on the control connection:
$ sudo tcpdump -i any -n -A 'port 21' -c 20 2>/dev/null | grep -E 'USER|PASS'
# USER alice
# PASS hunter2
#   Credentials in plaintext. This alone disqualifies FTP.

# FTPS: upgrade the control channel with AUTH TLS:
$ curl --ssl-reqd -u alice:secret ftp://ftp.example.com/README
# (control channel encrypted; the firewall can no longer read PASV replies)

$ openssl s_client -starttls ftp -connect ftp.example.com:21 -quiet
# 234 AUTH TLS successful
# 

# Count the connections one transfer actually opens:
$ ss -tn | grep -E ':21|:5[0-9]{4}' 
# ESTAB  192.168.1.20:44122  203.0.113.9:21      <- control
# ESTAB  192.168.1.20:44123  203.0.113.9:50000   <- data, this transfer only

# Configure a bounded passive range (vsftpd) so the firewall rule is finite:
$ grep -E 'pasv_(min|max)_port|ssl_enable' /etc/vsftpd.conf
# pasv_min_port=50000
# pasv_max_port=50100
# ssl_enable=YES

# The FTP ALG that plain FTP relies on -- and FTPS defeats:
$ lsmod | grep nf_conntrack_ftp
# nf_conntrack_ftp       16384  0
#   This module parses the control channel to open data ports.
#   With FTPS it sees only ciphertext and can do nothing.

Key takeaways

FTP uses two connections: a persistent control connection (port 21) and a new data connection per transfer or listing.
Active mode has the server connect back to the client — broken by NAT both because the announced address is private and because inbound connections are blocked.
Passive mode has the client connect out to a server-announced port, which works through client NAT but forces the server to expose a wide port range.
FTP is entirely plaintext, including credentials; FTPS adds TLS but does not solve the port problem.
FTPS actually makes firewalls harder, because encrypting the control channel blinds the FTP ALG that used to open data ports automatically.
Prefer SFTP or HTTPS; keep FTPS only for legacy systems, with passive mode and a narrow configured port range.

🧪 Practice

Decode 227 Entering Passive Mode (198,51,100,7,200,10) into an address and port, showing the arithmetic.

Explain the two independent reasons active mode fails for a client behind NAT.

Count the TCP connections opened by a session that lists two directories and downloads three files, and compare with SFTP.

Interview-style: Adding TLS to FTP made a firewall team's job harder rather than easier. Explain. (Hint: ask what the firewall used to read from the control channel, and what it can read now.)

SFTP and SCP over SSH

Theory

The previous topic's problems — plaintext credentials, two connections, dynamic ports, firewall hostility — are all solved by the same move: run file transfer inside an SSH session. SSH already provides an authenticated, encrypted, multiplexed channel over one connection on one port, so file transfer becomes a matter of defining what to send through it.

The naming is genuinely confusing and worth fixing first. SFTP is not "FTP over SSH". It shares no code, no commands, and no design with FTP. It is the SSH File Transfer Protocol, an independent binary protocol that runs as a subsystem inside SSH. FTPS is FTP with TLS; SFTP is something else entirely.

SCP is the older and simpler of the two. It runs scp remotely over SSH and streams files through the session. It copies a file or a tree and nothing else — no listing, no resume, no partial reads, no renames. It is also effectively frozen: OpenSSH 8.8 (2021) began defaulting scp to use the SFTP protocol underneath, after the original SCP protocol was found to allow a malicious server to write files the client never requested (CVE-2020-15778 and related). The modern advice is to treat scp as a convenience front-end and reach for SFTP or rsync for anything substantial.

SFTP is a full remote filesystem protocol: open, read, write, close, stat, setstat, opendir, readdir, rename, remove, mkdir, symlink, plus resumable transfers via explicit offsets. All of it is packet-framed inside one SSH channel, so a transfer and a directory listing can be in flight simultaneously without a second connection.

The comparison that matters in practice:

	FTP/FTPS	SCP	SFTP	rsync over SSH
Connections	2+	1	1	1
Ports to open	21 + wide range	22	22	22
Encryption	Optional (FTPS)	Always	Always	Always
Key-based auth	Rare	Yes	Yes	Yes
Directory listing	Yes	No	Yes	Yes
Resume	Sometimes	No	Yes	Yes
Delta transfer	No	No	No	Yes
NAT/firewall friendly	No	Yes	Yes	Yes

For synchronising directories that mostly have not changed, rsync over SSH beats both: its rolling-checksum algorithm transfers only the differing blocks, which on a large tree with small edits is often a 100x reduction.

One operational note that follows from running over SSH: access control is SSH's, not the protocol's. Restricting a user to file transfer means configuring ForceCommand internal-sftp with ChrootDirectory in sshd_config, which is a server configuration question rather than anything SFTP itself provides.

Examples

One connection, one port, everything inside it

  FTPS                                SFTP
  +-----------------------+           +-----------------------+
  | control :21 (TLS)     |           |   SSH connection :22  |
  +-----------------------+           |  +------------------+ |
  | data :50000 (TLS)     |           |  | channel: sftp    | |
  +-----------------------+           |  +------------------+ |
  | data :50001 (TLS)     |           |  | channel: shell   | |
  +-----------------------+           |  +------------------+ |
   3 connections,                     |  | channel: forward | |
   ~100 firewall ports                |  +------------------+ |
                                      +-----------------------+
                                       1 connection, 1 port


  SFTP is a REMOTE FILESYSTEM protocol, not a copy command

    SSH_FXP_OPEN    path, flags        -> handle
    SSH_FXP_READ    handle, offset, n  -> data      <- arbitrary offset
    SSH_FXP_WRITE   handle, offset, d  -> status       = RESUMABLE
    SSH_FXP_STAT    path               -> attrs
    SSH_FXP_READDIR handle             -> names
    SSH_FXP_RENAME  old, new           -> status
    SSH_FXP_CLOSE   handle             -> status


  RESUME, which SCP cannot do:

    transfer fails at 700 MB of a 1 GB file
      scp  : start again from 0        -> 1000 MB transferred in total
      sftp : SSH_FXP_READ offset=700M  ->  300 MB transferred

# Why SFTP can resume and SCP cannot: explicit offsets in the protocol.

class SFTPTransfer:
    """Reads carry an OFFSET, so an interrupted transfer continues in place."""
    def __init__(self, total_bytes, chunk=32768):
        self.total, self.chunk, self.offset = total_bytes, chunk, 0

    def resume_from(self, local_size):
        self.offset = local_size            # SSH_FXP_READ(handle, offset, len)
        return self.total - self.offset

file_size = 1024 * 1024 * 1024              # 1 GB
interrupted_at = 700 * 1024 * 1024

t = SFTPTransfer(file_size)
remaining = t.resume_from(interrupted_at)
print(f"SFTP resume : {remaining/1024**2:>7,.0f} MB remaining")
print(f"SCP restart : {file_size/1024**2:>7,.0f} MB remaining")
# SFTP resume :     324 MB remaining
# SCP restart :   1,024 MB remaining

# The rsync delta algorithm: send only what differs.
import hashlib

def rolling_blocks(data, block=4096):
    return {i: hashlib.md5(data[i:i+block]).digest()
            for i in range(0, len(data), block)}

original = bytearray(b"A" * 100 * 4096)          # ~400 KB
modified = bytearray(original)
modified[50*4096:51*4096] = b"B" * 4096          # change ONE block

old_sums, new_sums = rolling_blocks(original), rolling_blocks(modified)
changed = [i for i in new_sums if old_sums.get(i) != new_sums[i]]

print(f"\nfile size          : {len(modified):>9,} bytes")
print(f"blocks changed     : {len(changed)} of {len(new_sums)}")
print(f"scp/sftp transfers : {len(modified):>9,} bytes (the whole file)")
print(f"rsync transfers    : {len(changed)*4096:>9,} bytes (+ checksums)")
print(f"saving             : {1 - len(changed)*4096/len(modified):>9.1%}")
# file size          :   409,600 bytes
# blocks changed     : 1 of 100
# scp/sftp transfers :   409,600 bytes (the whole file)
# rsync transfers    :     4,096 bytes (+ checksums)
# saving             :     99.0%

# Firewall surface: the number that decides what security will approve.
configs = {
    "FTP (active)":  {"ports": "21 + inbound to client", "inbound_to_client": True},
    "FTPS (passive)": {"ports": "21 + 50000-50100", "inbound_to_client": False},
    "SFTP":          {"ports": "22", "inbound_to_client": False},
}
print()
for name, c in configs.items():
    risk = "REQUIRES INBOUND TO CLIENT" if c["inbound_to_client"] else "outbound only"
    print(f"{name:<16} open: {c['ports']:<26} {risk}")
# FTP (active)     open: 21 + inbound to client    REQUIRES INBOUND TO CLIENT
# FTPS (passive)   open: 21 + 50000-50100          outbound only
# SFTP             open: 22                        outbound only

# SFTP: interactive, and genuinely a filesystem interface:
$ sftp alice@server.example.com
# Connected to server.example.com.
# sftp> pwd
# Remote working directory: /home/alice
# sftp> ls -l
# -rw-r--r--  1 alice alice  1048576 Sep 23 09:14 report.pdf
# sftp> get -a report.pdf          # -a = APPEND: resume a partial download
# Resuming download of /home/alice/report.pdf from 700000
# sftp> put -r ./local-dir
# sftp> rename old.txt new.txt     # operations scp simply does not have

# Non-interactive, for scripts:
$ sftp -b - alice@server <<'EOF'
> cd /var/data
> get -a bigfile.tar.gz
> quit
> EOF

# scp: simple, one purpose, and now SFTP underneath:
$ scp -v report.pdf alice@server:/home/alice/ 2>&1 | grep -i 'sftp\|protocol'
# debug1: Using SFTP protocol for transfer     <- OpenSSH >= 8.8 default
$ scp -O report.pdf alice@server:/home/alice/  # -O forces the LEGACY protocol

# rsync over SSH: transfer only the differences:
$ rsync -avz --partial --progress -e ssh ./site/ alice@server:/var/www/site/
# sending incremental file list
# index.html
#       4,096 100%    3.9MB/s    0:00:00 (xfr#1, to-chk=98/100)
# sent 5,102 bytes  received 118 bytes
# total size is 409,600  speedup is 78.47
#                        ^^^^^^^^^^^^^^^ only the changed block moved

# Restrict a user to file transfer only -- this is SSH config, not SFTP:
$ sudo tee -a /etc/ssh/sshd_config <<'EOF'
> Match User sftpuser
>     ForceCommand internal-sftp
>     ChrootDirectory /srv/sftp/%u
>     PermitTTY no
>     X11Forwarding no
>     AllowTcpForwarding no
> EOF
$ sudo sshd -t && sudo systemctl reload sshd     # validate BEFORE reloading

# Verify: one connection, one port, for the entire session:
$ ss -tn | grep ':22'
# ESTAB  192.168.1.20:48122  203.0.113.9:22
#   That is the complete firewall surface.

Key takeaways

SFTP is not FTP over SSH — it is an independent binary protocol running as an SSH subsystem; FTPS is the one that is actually FTP with TLS.
Everything runs over one connection on port 22, eliminating FTP's dynamic data ports and its firewall hostility.
SCP copies and nothing else — no listing, no resume, no rename — and modern OpenSSH uses the SFTP protocol underneath by default after security flaws in the original.
SFTP is a remote filesystem protocol with explicit offsets, which is exactly what makes resume possible.
rsync over SSH adds delta transfer, often a 100x saving when synchronising trees that mostly have not changed.
Restricting a user to transfers is SSH server configuration (ForceCommand internal-sftp, ChrootDirectory), not a protocol feature.

🧪 Practice

State the firewall rules required for FTPS passive mode and for SFTP, and explain why the difference matters to a security team.

Explain the protocol mechanism that lets SFTP resume an interrupted transfer and why SCP cannot.

A 2 GB archive changes by 8 KB daily. Compare daily bytes transferred using scp, sftp, and rsync.

Interview-style: Someone says "we use SFTP, which is just FTP with encryption". Correct them precisely. (Hint: name what SFTP shares with FTP — the answer is nothing but four letters — and say which protocol they were actually describing.)

8.5 Remote Access and Management

Networks must be operated, not just used: this subchapter covers the protocols for logging into devices, collecting their state, reading their logs, agreeing on time, and configuring them programmatically.

SSH Architecture and Key Exchange

Theory

SSH exists because its predecessors were catastrophically insecure. telnet, rlogin, and rsh sent passwords and every keystroke in plaintext, and rsh's trust model was "this host says it is who it claims" — forgeable by anyone who could spoof an address. SSH replaced all of them with one protocol that is authenticated, encrypted, integrity-protected, and multiplexed.

SSH is best understood as three layers stacked inside one TCP connection on port 22:

1. The transport layer performs key exchange, establishes encryption and integrity, and authenticates the server. Note the direction: before you send a password, SSH proves the server is the one you meant.

2. The user authentication layer proves who you are — by public key, password, keyboard-interactive, GSSAPI, or certificate.

3. The connection layer multiplexes channels over the encrypted session: an interactive shell, an SFTP subsystem, several port forwards, and an agent-forwarding channel, all simultaneously.

The key exchange is the part worth understanding properly, because it explains the warnings users see. SSH uses Diffie-Hellman (or its elliptic-curve variant, typically Curve25519): both sides derive a shared secret over a public channel without ever transmitting it, giving forward secrecy — recording the traffic and later stealing the server's private key still does not decrypt it, because the session key was ephemeral and never sent.

The server then signs the exchange hash with its host key, proving it holds the private key matching the public host key the client has. The client checks that against ~/.ssh/known_hosts, which is why:

A first connection prompts with a fingerprint — TOFU, trust on first use. Nothing yet proves the key is right; the user is expected to verify out of band, and almost never does.
A changed host key produces the large warning banner. It usually means a rebuilt server, and occasionally means a machine-in-the-middle.

Public-key authentication is the strong alternative to passwords: the client signs a challenge with its private key, and the server verifies against ~/.ssh/authorized_keys. The private key never leaves the client, so nothing reusable is transmitted at all — phishing and password reuse simply do not apply.

At scale, authorized_keys files become unmanageable, and SSH certificates solve it. A trusted CA signs a user's public key with a validity window and principals, and servers are configured to trust the CA rather than individual keys. Adding a person means issuing a certificate; removing them means letting it expire. No file on any server changes.

Port forwarding is the connection layer's most-used feature and is worth stating precisely because the direction confuses everyone:

Local (-L): a port on your machine tunnels to a host reachable from the server. Used to reach something behind a firewall.
Remote (-R): a port on the server tunnels back to a host reachable from you. Used to expose a local service outward.
Dynamic (-D): a local SOCKS proxy routing arbitrary traffic through the server.

Examples

Three layers inside one TCP connection

  +---------------------------------------------------------+
  |  CONNECTION LAYER   channels multiplexed over one link   |
  |   [shell] [sftp] [-L 8080] [-R 9000] [agent forwarding]  |
  +---------------------------------------------------------+
  |  USER AUTH LAYER    publickey / password / certificate   |
  +---------------------------------------------------------+
  |  TRANSPORT LAYER    KEX, encryption, MAC, SERVER auth    |
  +---------------------------------------------------------+
  |                     TCP port 22                          |
  +---------------------------------------------------------+
     Note the order: the SERVER is authenticated BEFORE you
     ever send a credential.


 KEY EXCHANGE (Curve25519 + host key signature)

   client                                        server
     |--- ClientHello: kex algs, ciphers ------->|
     |<-- ServerHello: chosen algorithms --------|
     |--- ephemeral public key Qc -------------->|
     |<-- ephemeral public key Qs               -|
     |    + host public key                      |
     |    + SIGNATURE over the exchange hash     |
     |                                           |
   both compute shared secret  S = f(qc, Qs) = f(qs, Qc)
                                    ^ never transmitted -> FORWARD SECRECY
   client verifies the signature against known_hosts
                                    ^ this is what the fingerprint prompt is for


 PORT FORWARDING -- which direction is which

  -L 8080:db.internal:5432    "bring THERE to HERE"
     [you:8080] ==ssh==> [server] ----> [db.internal:5432]

  -R 9000:localhost:3000      "expose HERE to THERE"
     [server:9000] <==ssh== [you] ----> [localhost:3000]

  -D 1080                     "route anything through THERE"
     [you:1080 SOCKS] ==ssh==> [server] ----> the internet

# Diffie-Hellman by hand: the shared secret is never transmitted.
import hashlib, secrets

p = 0xFFFFFFFFFFFFFFFFC90FDAA22168C234C4C6628B80DC1CD1  # small prime, for illustration
g = 2

a = secrets.randbelow(p - 2) + 1       # client's EPHEMERAL private value
A = pow(g, a, p)                        # client's public value -> sent
b = secrets.randbelow(p - 2) + 1       # server's EPHEMERAL private value
B = pow(g, b, p)                        # server's public value -> sent

client_secret = pow(B, a, p)            # computed locally from B and a
server_secret = pow(A, b, p)            # computed locally from A and b

print(f"on the wire : A={hex(A)[:20]}... B={hex(B)[:20]}...")
print(f"shared secret matches: {client_secret == server_secret}")
print(f"secret      : {hex(client_secret)[:20]}...  <- NEVER transmitted")
# on the wire : A=0x9c2f1a... B=0x4b8e07...
# shared secret matches: True
# secret      : 0x7d3f91...  <- NEVER transmitted
#   Forward secrecy: a and b are discarded after the session. Stealing the
#   server's HOST key later does not recover this session's traffic.

# Host key verification: what the fingerprint prompt is actually checking.
import base64
def fingerprint(pubkey_b64):
    raw = base64.b64decode(pubkey_b64)
    return "SHA256:" + base64.b64encode(hashlib.sha256(raw).digest()).decode().rstrip("=")

known_hosts = {"server.example.com": "SHA256:47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU"}

def verify_host(host, presented_key_b64):
    fp = fingerprint(presented_key_b64)
    known = known_hosts.get(host)
    if known is None:
        return "TOFU", f"unknown host, fingerprint {fp} -- verify OUT OF BAND"
    if known == fp:
        return "OK", "host key matches known_hosts"
    return "ALERT", ("REMOTE HOST IDENTIFICATION HAS CHANGED -- "
                     "rebuilt server, or machine-in-the-middle")

print()
for host, key in [("new.example.com", ""), ("server.example.com", ""),
                  ("server.example.com", "AAAA")]:
    status, msg = verify_host(host, key)
    print(f"  {status:<6} {msg}")
#   TOFU   unknown host, fingerprint SHA256:47DEQ... -- verify OUT OF BAND
#   OK     host key matches known_hosts
#   ALERT  REMOTE HOST IDENTIFICATION HAS CHANGED -- rebuilt server, or MITM

# Why SSH certificates scale and authorized_keys does not.
def key_management_effort(users, servers):
    return {"authorized_keys entries": users * servers,
            "files to edit when someone leaves": servers,
            "certificate approach": "1 CA public key per server, issue/expire per user"}

for u, s in ((10, 5), (200, 400)):
    r = key_management_effort(u, s)
    print(f"\n{u} users x {s} servers:")
    print(f"  authorized_keys entries to maintain : {r['authorized_keys entries']:,}")
    print(f"  files to edit on offboarding        : {r['files to edit when someone leaves']:,}")
    print(f"  with certificates                   : {r['certificate approach']}")
# 10 users x 5 servers:
#   authorized_keys entries to maintain : 50
#   files to edit on offboarding        : 5
#   with certificates                   : 1 CA public key per server, issue/expire per user
# 200 users x 400 servers:
#   authorized_keys entries to maintain : 80,000
#   files to edit on offboarding        : 400
#   with certificates                   : 1 CA public key per server, issue/expire per user

# Watch the three layers negotiate:
$ ssh -vv alice@server.example.com 2>&1 | grep -E 'kex:|Server host key|Authenticat'
# debug1: kex: algorithm: curve25519-sha256
# debug1: kex: host key algorithm: ssh-ed25519
# debug1: kex: server->client cipher: chacha20-poly1305@openssh.com
# debug1: Server host key: ssh-ed25519 SHA256:47DEQpj8HBSa+/TImW+5JCeuQeRkm5NM
# debug1: Authenticating to server.example.com:22 as 'alice'
# debug1: Authentication succeeded (publickey).

# Verify a host key out of band BEFORE the first connection:
$ ssh-keyscan -t ed25519 server.example.com 2>/dev/null | ssh-keygen -lf -
# 256 SHA256:47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU server.example.com (ED25519)
#   Compare with what the server's operator published. This is the step
#   that turns TOFU into actual verification.

# Generate a modern key pair:
$ ssh-keygen -t ed25519 -C 'alice@laptop' -f ~/.ssh/id_ed25519
# ed25519: small, fast, and no parameter choices to get wrong.

# Issue an SSH certificate instead of appending to authorized_keys:
$ ssh-keygen -s ca_key -I 'alice@corp' -n alice,deploy -V +8h user_key.pub
# Signed user key user_key-cert.pub: id "alice@corp" serial 0 
#   valid from 2026-09-23T09:00:00 to 2026-09-23T17:00:00
#                                    ^ expires by itself: no offboarding step

$ ssh-keygen -L -f user_key-cert.pub | grep -E 'Principals|Valid'
#         Valid: from 2026-09-23T09:00:00 to 2026-09-23T17:00:00
#         Principals: alice deploy

# Port forwarding, all three directions:
$ ssh -L 8080:db.internal:5432 alice@bastion    # reach a DB behind a firewall
$ ssh -R 9000:localhost:3000 alice@public       # expose your local dev server
$ ssh -D 1080 alice@bastion                     # SOCKS proxy through the bastion

# Multiplexing: reuse one connection for every later session:
$ cat >> ~/.ssh/config <<'EOF'
> Host *
>     ControlMaster auto
>     ControlPath ~/.ssh/cm-%r@%h:%p
>     ControlPersist 10m
> EOF
$ time ssh server true      # first: full handshake
# real    0m0.412s
$ time ssh server true      # second: rides the existing connection
# real    0m0.038s

# Audit a server's accepted algorithms:
$ ssh -Q kex | head -3
# curve25519-sha256
# curve25519-sha256@libssh.org
# diffie-hellman-group-exchange-sha256

Key takeaways

SSH replaced telnet/rlogin/rsh, and is structured as three layers in one TCP connection: transport (KEX, encryption, server authentication), user authentication, and connection (channels).
The server is authenticated before the client sends any credential, which is what the host-key fingerprint prompt is protecting.
Key exchange is ephemeral Diffie-Hellman, giving forward secrecy: stealing the host key later does not decrypt recorded sessions.
known_hosts is trust-on-first-use; a changed host key is usually a rebuild and occasionally a machine-in-the-middle — verify out of band with ssh-keyscan.
Public-key authentication transmits nothing reusable; SSH certificates with validity windows and principals replace unmanageable authorized_keys files at scale.
The connection layer multiplexes shells, SFTP, and port forwards: -L brings a remote service local, -R exposes a local service remotely, -D opens a SOCKS proxy.

🧪 Practice

Explain what forward secrecy means for SSH and which part of the handshake provides it.

You connect to a familiar server and get the changed-host-key warning. List the two plausible causes and how you would distinguish them.

Write the ssh command to reach a database on 10.0.5.9:5432 that is only reachable from bastion.example.com.

Interview-style: Why do SSH certificates scale better than authorized_keys for 200 engineers across 400 servers? (Hint: count the files that must change when one person leaves the company, under each scheme.)

Telnet and Its Deprecation

Theory

Telnet (RFC 854, 1983) was the internet's first remote terminal protocol, and for a decade it was how people used remote computers. It is a strikingly simple design: a TCP connection on port 23 carrying a bidirectional stream of characters, with a small in-band negotiation mechanism for terminal options.

Its fatal flaw is equally simple: everything is plaintext. Username, password, every command, every response, every keystroke. Anyone with access to any link on the path — a switch port, a compromised router, a Wi-Fi network — reads the entire session, credentials first.

In 1983 this was defensible. The ARPANET was a few hundred mutually trusting research machines on physically controlled links. The assumption stopped being true the moment networks became shared, commercial, and wireless, and telnet became one of the most reliable ways to harvest credentials in existence.

What telnet has no answer for:

Requirement	Telnet
Confidentiality	None — plaintext
Integrity	None — injectable
Server authentication	None — you cannot tell what you connected to
Credential protection	None — password in the clear
Key-based auth	None

SSH (previous topic) provides all five over the same kind of terminal session, which is why telnet was displaced essentially completely for interactive login by the early 2000s.

Two things keep it alive and are worth being precise about.

Legacy devices. Industrial control systems, older network hardware, and embedded devices sometimes support nothing else. Where such a device genuinely cannot be replaced, the mitigation is to remove it from routable reach — a management VLAN, a jump host, or a serial console — rather than to accept plaintext on a shared network.

Telnet the client as a diagnostic tool. This is the legitimate modern use, and it is not really telnet at all: telnet host 80 is a convenient way to open a raw TCP connection and type a protocol by hand. It is genuinely useful for checking whether a port accepts connections and for speaking HTTP or SMTP manually. Even here, nc (netcat) is the better tool, because it does not perform telnet's option negotiation, which can corrupt binary protocols by interpreting 0xFF bytes as commands.

The lesson generalises beyond telnet, and it is the reason this topic is worth its space: protocols designed for a trusted network do not become safe on an untrusted one just because they still work. The same reasoning retired FTP, plain SNMP v1/v2c, and unauthenticated NTP.

Examples

The same login, two protocols, one observer

  TELNET (port 23)                    SSH (port 22)
  tcpdump on any hop sees:            tcpdump on any hop sees:

    login: a                            \x9f\x2a\x8c\x1d\x4b\x7e...
    login: al                           \x3f\x81\x0c\x55\xe2\x99...
    login: ali                          \x71\xd4\x2b\x60\x8a\x13...
    Password: h                         (indistinguishable noise)
    Password: hu
    Password: hun
    ...
    $ cat /etc/shadow
    root:$6$xyz...

  ^ one character per packet, because of                ^ nothing readable,
    character-at-a-time mode: the password                including the fact
    is reconstructible keystroke by keystroke             that it IS a password


 WHY IT SURVIVED SO LONG, AND WHERE IT STILL LIVES

   1983  trusted research network, physically controlled links  -> fine
   1995  commercial ISPs, shared Ethernet segments              -> dangerous
   2000  SSH is ubiquitous and free                             -> obsolete
   2026  legacy ICS gear, old switches, embedded devices        -> isolate it

 THE ONLY GOOD MODERN USE: a raw TCP connection you type into

   $ telnet example.com 80    <- not "using telnet", but "using a TCP socket"
   (and `nc` is better: no option negotiation to corrupt binary data)

# What a passive observer reconstructs from a telnet session.

captured_packets = [
    (b"\xff\xfd\x18", "IAC DO TERMINAL-TYPE"),    # option negotiation
    (b"login: ", "server prompt"),
    (b"a", "keystroke"), (b"l", "keystroke"), (b"i", "keystroke"),
    (b"c", "keystroke"), (b"e", "keystroke"), (b"\r\n", "enter"),
    (b"Password: ", "server prompt"),
    (b"h", "keystroke"), (b"u", "keystroke"), (b"n", "keystroke"),
    (b"t", "keystroke"), (b"e", "keystroke"), (b"r", "keystroke"),
    (b"2", "keystroke"), (b"\r\n", "enter"),
    (b"$ ", "shell prompt"),
]

keystrokes = b"".join(p for p, kind in captured_packets if kind == "keystroke")
print("reconstructed from the capture:", keystrokes.decode())
# reconstructed from the capture: alicehunter2
#   Username and password, recovered with no cryptanalysis whatsoever.

# Why one keystroke per packet: telnet's character-at-a-time mode.
def packets_for(text, mode):
    return len(text) if mode == "character" else 1

for mode in ("character", "line"):
    print(f"\n{mode}-at-a-time: 'hunter2' -> {packets_for('hunter2', mode)} packet(s)")
# character-at-a-time: 'hunter2' -> 7 packet(s)
# line-at-a-time: 'hunter2' -> 1 packet(s)
#   Character mode also leaks TIMING, from which keystroke-interval analysis
#   can recover content even when a protocol IS encrypted (a known SSH concern,
#   which is why SSH pads and batches).

# The security property comparison, made explicit.
props = ["confidentiality", "integrity", "server auth", "client key auth",
         "forward secrecy"]
telnet = dict.fromkeys(props, False)
ssh    = dict.fromkeys(props, True)
print(f"\n{'property':<20}{'telnet':>9}{'ssh':>7}")
for p in props:
    print(f"{p:<20}{str(telnet[p]):>9}{str(ssh[p]):>7}")
# property              telnet    ssh
# confidentiality        False   True
# integrity              False   True
# server auth            False   True
# client key auth        False   True
# forward secrecy        False   True

# The telnet option negotiation that makes `nc` the better diagnostic tool:
IAC, DO, WILL = 0xFF, 0xFD, 0xFB
binary_payload = bytes([0x89, 0x50, 0x4E, 0xFF, 0x47])   # contains 0xFF
print(f"\nraw bytes        : {binary_payload.hex()}")
print(f"telnet sees 0xFF as IAC -> interprets the next byte as a COMMAND")
print(f"corrupted stream : {binary_payload.replace(bytes([IAC]), b'').hex()}")
# raw bytes        : 89504effff47   (conceptually)
# telnet sees 0xFF as IAC -> interprets the next byte as a COMMAND
#   This is why binary protocols must not be probed with telnet.

# Prove the exposure (in a lab you control, never on someone else's network):
$ sudo tcpdump -i any -n -A 'port 23' -c 50 2>/dev/null | grep -A1 'login\|Password'
# login: alice
# Password: hunter2
#   Nothing to decrypt. The credentials are simply present in the capture.

# The same capture against SSH:
$ sudo tcpdump -i any -n -A 'port 22' -c 50 2>/dev/null | head -3
# ..A.].=.4..#..s...9...v.N....|.Kf.....
#   Not even the protocol phase is distinguishable.

# Find telnet still listening anywhere in your estate:
$ nmap -p 23 --open 10.0.0.0/24
# Nmap scan report for 10.0.0.41
# PORT   STATE SERVICE
# 23/tcp open  telnet        <- a switch or embedded device; isolate it

# Disable it where you find it (Linux):
$ sudo systemctl disable --now telnet.socket
$ sudo ss -tlnp | grep ':23' || echo 'telnet no longer listening'
# telnet no longer listening

# Cisco-style: remove telnet from the VTY lines:
# line vty 0 15
#   transport input ssh        <- ssh ONLY; drop telnet entirely
#   exec-timeout 10 0

# The legitimate modern use -- and why nc is better:
$ telnet example.com 80
# Trying 93.184.216.34...
# Connected to example.com.
# GET / HTTP/1.1
# Host: example.com
#
# HTTP/1.1 200 OK

$ nc example.com 80 <<'EOF'
> GET / HTTP/1.1
> Host: example.com
> Connection: close
>
> EOF
# HTTP/1.1 200 OK
#   No option negotiation, so binary protocols survive intact.

# Check a port without sending anything at all:
$ nc -zv example.com 443
# Connection to example.com 443 port [tcp/https] succeeded!

Key takeaways

Telnet is a plaintext terminal protocol on port 23: credentials, commands, and output are all readable by anyone on the path.
It provides no confidentiality, no integrity, no server authentication, and no key-based authentication — SSH provides all of them.
Its design was reasonable for a small, trusted, physically controlled network and became indefensible once networks were shared and wireless.
Where legacy devices support nothing else, the fix is isolation (management VLAN, jump host, serial console), not acceptance.
telnet host port as a manual TCP client is still useful, but nc is better because telnet's option negotiation corrupts binary protocols.
The general lesson applies beyond telnet: a protocol designed for a trusted network does not become safe on an untrusted one just because it still works.

🧪 Practice

List the five security properties SSH provides that telnet does not, and state which one protects you before you type a password.

A factory has a controller that supports only telnet and cannot be replaced. Describe a defensible deployment.

Explain why nc is preferable to telnet when manually probing a binary protocol.

Interview-style: Telnet was a perfectly reasonable design in 1983. What changed? (Hint: the protocol did not change — describe what happened to the assumptions it was built on, and name two other protocols retired for the same reason.)

SNMP Agents, Managers, and MIBs

Theory

Once a network has more than a handful of devices, you cannot log into each one to ask how it is. You need a way to read state from thousands of devices uniformly, regardless of vendor. SNMP (Simple Network Management Protocol) is that mechanism, and despite its age it still carries a large share of network monitoring.

The model has three parts:

The agent runs on the managed device (switch, router, server, printer) and exposes its state.
The manager is the monitoring system that polls agents and collects data.
The MIB (Management Information Base) is the schema: a hierarchical, standardised namespace of everything that can be read or written.

The MIB is a tree, and each node is identified by an OID (Object Identifier) — a dotted sequence of integers, exactly analogous to a DNS name but numeric and read left to right:

  1.3.6.1.2.1.2.2.1.10.3
  | | | | | | | | | |  |
  | | | | | | | | | |  +-- interface index 3
  | | | | | | | | | +----- ifInOctets (bytes received)
  | | | | | | | | +------- ifEntry
  | | | | | | | +--------- ifTable
  | | | | | | +----------- interfaces
  | | | | | +------------- mib-2
  | | | | +--------------- mgmt
  | | | +----------------- internet
  | | +------------------- dod
  | +--------------------- org
  +----------------------- iso

The standard subtree 1.3.6.1.2.1 (mib-2) is what makes SNMP vendor-neutral: ifInOctets means the same thing on every device that implements it. Vendors add their own extensions under 1.3.6.1.4.1.<enterprise-number>.

Operations are deliberately few: GET (read one OID), GETNEXT (read the next OID in tree order, used to walk), GETBULK (v2c+, fetch many at once — vastly more efficient than repeated GETNEXTs), SET (write), and TRAP/INFORM (the agent notifies the manager without being asked; INFORM is acknowledged, TRAP is not).

The interaction model is polling plus traps: managers poll periodically for metrics, and agents send traps for events that cannot wait for the next poll. This is the same request-response-versus-push trade-off from 8.1, resolved by using both.

The version history is where the operational security lives, and it matters:

Version	Authentication	Encryption	Verdict
v1	Community string (plaintext)	None	Obsolete
v2c	Community string (plaintext)	None	Still widespread, still insecure
v3	User-based, HMAC	Yes (DES/AES)	The only acceptable choice

The "community string" in v1/v2c is effectively a password sent in cleartext in every packet, and public (read) and private (write) are the defaults that shipped on an enormous number of devices. A readable SNMP agent discloses the entire network topology; a writable one lets an attacker reconfigure it. SNMPv3 adds real user authentication and encryption and should be the only version exposed.

There is also an amplification concern: a small GETBULK request can generate a very large response, which makes an open SNMP agent usable as a reflector in volumetric attacks — another reason to restrict agents to a management network.

Examples

The SNMP model

   +-----------------+   GET / GETNEXT / GETBULK / SET   +----------------+
   |    MANAGER      | --------------------------------> |     AGENT      |
   |  (monitoring    | <-------------------------------- | (switch,router,|
   |   system)       |            responses              |  server)       |
   |                 | <-------------------------------- |                |
   +-----------------+          TRAP / INFORM            +----------------+
        polls every 60 s          asynchronous event
        for metrics               (link down, fan fail)
                                  INFORM is acked; TRAP is fire-and-forget


  THE MIB TREE (the OID namespace)

                      1 iso
                      |
                      3 org
                      |
                      6 dod
                      |
                      1 internet
                   /     \
              2 mgmt      4 private
                 |            |
              1 mib-2      1 enterprises
             /   |   \         |
      1 system 2 interfaces  9 cisco / 2636 juniper / 8072 net-snmp
                    |
                 2 ifTable
                    |
                 1 ifEntry
              /     |      \
      1 ifIndex  2 ifDescr  10 ifInOctets
                                 |
                          .3 = interface 3

  Full OID: 1.3.6.1.2.1.2.2.1.10.3   ("bytes in, on interface 3")
  Standard subtree = same meaning on EVERY vendor's device.

# Walking a MIB and turning counters into rates -- what a poller actually does.

MIB = {   # a toy agent's view of an interface table
    "1.3.6.1.2.1.1.5.0":        ("sysName",     "core-sw-01"),
    "1.3.6.1.2.1.1.3.0":        ("sysUpTime",   184_523_400),   # centiseconds
    "1.3.6.1.2.1.2.2.1.2.1":    ("ifDescr.1",   "GigabitEthernet1/0/1"),
    "1.3.6.1.2.1.2.2.1.8.1":    ("ifOperStatus.1", 1),           # 1=up 2=down
    "1.3.6.1.2.1.2.2.1.10.1":   ("ifInOctets.1", 4_102_884_213),
    "1.3.6.1.2.1.2.2.1.2.3":    ("ifDescr.3",   "GigabitEthernet1/0/3"),
    "1.3.6.1.2.1.2.2.1.8.3":    ("ifOperStatus.3", 2),
    "1.3.6.1.2.1.2.2.1.10.3":   ("ifInOctets.3", 812_004),
}

def snmp_get(oid):
    return MIB.get(oid, (None, "noSuchObject"))

def snmp_walk(prefix):
    """GETNEXT repeated: return every OID under a subtree, in tree order."""
    return sorted((o, v) for o, v in MIB.items() if o.startswith(prefix))

print("GET sysName:", snmp_get("1.3.6.1.2.1.1.5.0")[1])
# GET sysName: core-sw-01

print("\nWALK the interface table:")
for oid, (name, value) in snmp_walk("1.3.6.1.2.1.2.2.1"):
    print(f"  {oid:<28} {name:<18} {value}")
#   1.3.6.1.2.1.2.2.1.10.1       ifInOctets.1       4102884213
#   1.3.6.1.2.1.2.2.1.2.1        ifDescr.1          GigabitEthernet1/0/1
#   1.3.6.1.2.1.2.2.1.8.1        ifOperStatus.1     1
#   ... (interface 3 shows ifOperStatus=2, i.e. DOWN)

# SNMP returns COUNTERS, not rates. The manager computes the rate --
# and must handle counter wrap, which is the classic SNMP bug.
def rate_bps(prev, curr, seconds, counter_bits=32):
    if curr >= prev:
        delta = curr - prev
    else:
        delta = (2 ** counter_bits - prev) + curr     # the counter WRAPPED
    return delta * 8 / seconds

print(f"\nnormal   : {rate_bps(4_102_884_213, 4_110_884_213, 60)/1e6:>8.2f} Mb/s")
print(f"wrapped  : {rate_bps(4_290_000_000, 12_000_000, 60)/1e6:>8.2f} Mb/s")
print(f"if wrap ignored: {(12_000_000-4_290_000_000)*8/60/1e6:>8.2f} Mb/s  <- negative!")
# normal   :     1.07 Mb/s
# wrapped  :     2.29 Mb/s
# if wrap ignored:  -570.40 Mb/s  <- negative!
#   A 32-bit octet counter wraps in ~34 seconds at 1 Gb/s. Use the 64-bit
#   ifHCInOctets counters (1.3.6.1.2.1.31.1.1.1.6) on anything above 100 Mb/s.

def wrap_time(speed_bps, bits=32):
    return (2 ** bits) / (speed_bps / 8)

for name, speed in (("100 Mb/s", 1e8), ("1 Gb/s", 1e9), ("10 Gb/s", 1e10)):
    print(f"\n32-bit counter on {name:<9} wraps every {wrap_time(speed):>8.1f} s"
          f"  (64-bit: {wrap_time(speed, 64)/86400/365:>,.0f} years)")
# 32-bit counter on 100 Mb/s  wraps every    343.6 s  (64-bit: 46,796 years)
# 32-bit counter on 1 Gb/s    wraps every     34.4 s  (64-bit: 4,680 years)
# 32-bit counter on 10 Gb/s   wraps every      3.4 s  (64-bit: 468 years)
#   At 10 Gb/s, a 60-second poll interval misses ~17 wraps. The data is garbage.

# GETBULK vs repeated GETNEXT: why v2c+ matters for large tables.
def walk_cost(rows, columns, mode, rtt_ms=5, bulk=25):
    ops = rows * columns
    requests = ops if mode == "getnext" else -(-ops // bulk)
    return requests, requests * rtt_ms

for mode in ("getnext", "getbulk"):
    reqs, ms = walk_cost(48, 12, mode)
    print(f"\n{mode:<8}: {reqs:>4} requests, {ms:>6,} ms to walk a 48-port switch")
# getnext :  576 requests,  2,880 ms to walk a 48-port switch
# getbulk :   24 requests,    120 ms to walk a 48-port switch

# Read one value (v3 -- authenticated and encrypted):
$ snmpget -v3 -l authPriv -u monitor -a SHA -A 'authpass123' \
          -x AES -X 'privpass456' core-sw-01 sysName.0
# SNMPv2-MIB::sysName.0 = STRING: core-sw-01

# Walk the interface table:
$ snmpwalk -v3 -l authPriv -u monitor -a SHA -A 'authpass123' \
           -x AES -X 'privpass456' core-sw-01 ifDescr
# IF-MIB::ifDescr.1 = STRING: GigabitEthernet1/0/1
# IF-MIB::ifDescr.3 = STRING: GigabitEthernet1/0/3

# Use the 64-bit counters on anything faster than 100 Mb/s:
$ snmpbulkwalk -v3 ... core-sw-01 ifHCInOctets
# IF-MIB::ifHCInOctets.1 = Counter64: 41028842130912
#                          ^^^^^^^^^ no practical wrap

# Translate between names and numeric OIDs:
$ snmptranslate -On IF-MIB::ifInOctets
# .1.3.6.1.2.1.2.2.1.10
$ snmptranslate -Of .1.3.6.1.2.1.2.2.1.10.3
# .iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInOctets.3

# The v2c problem, demonstrated -- the "password" is in every packet:
$ snmpwalk -v2c -c public core-sw-01 system
$ sudo tcpdump -i any -n -A 'port 161' -c 3 2>/dev/null | grep -o 'public'
# public
#   Plaintext, in every single request. And `public` is a shipped default.

# Find devices still answering v2c with default community strings:
$ nmap -sU -p 161 --script snmp-info --script-args creds.snmp=:public 10.0.0.0/24
# | snmp-info:
# |   enterprise: net-snmp
# |_  snmpEngineTime: 21h13m52s
#   Anything that answers here discloses your topology to the internet.

# Receive traps (asynchronous events, not polled):
$ snmptrapd -f -Lo
# 2026-09-23 09:14:22 core-sw-01 [UDP: [10.0.0.41]:161]:
# IF-MIB::linkDown  ifIndex.3 = INTEGER: 3
#   The manager learns in seconds rather than at the next poll.

# Restrict the agent to the management network (net-snmp):
$ grep -E '^(rouser|agentaddress)' /etc/snmp/snmpd.conf
# rouser monitor authPriv
# agentaddress 10.99.0.15:161        <- management VLAN only, not 0.0.0.0

Key takeaways

SNMP's three parts are the agent (on the device), the manager (polling system), and the MIB (the standardised OID namespace).
OIDs are dotted integers read left to right; the 1.3.6.1.2.1 (mib-2) subtree is what makes SNMP vendor-neutral, with vendor extensions under 1.3.6.1.4.1.
Operations are GET, GETNEXT, GETBULK, SET, and TRAP/INFORM — GETBULK is dramatically more efficient for walking tables, and INFORM is acknowledged where TRAP is not.
The model combines periodic polling for metrics with asynchronous traps for events.
v1 and v2c send a cleartext community string (public/private by default) and must not be exposed; v3 provides real authentication and encryption.
SNMP returns counters, not rates: managers must compute deltas and handle wrap — use 64-bit ifHC* counters above 100 Mb/s, since a 32-bit counter wraps in ~34 seconds at 1 Gb/s.
Open agents leak topology and can act as amplification reflectors, so bind them to a management network.

🧪 Practice

Decode the OID 1.3.6.1.2.1.2.2.1.8.5 into its named path and state what it reports.

A 10 Gb/s interface is polled every 5 minutes with 32-bit counters. Explain why the resulting graph is meaningless and give the fix.

Compare request counts for walking a 48-port switch's 12-column interface table with GETNEXT versus GETBULK.

Interview-style: Why is a read-only SNMPv2c agent reachable from the internet a serious finding, even though it cannot change anything? (Hint: consider what a full MIB walk reveals about a network, and what a small request with a large response can be used for.)

Syslog and Centralized Logging

Theory

A device that logs only to itself is useless in an incident. The disk fills, the device reboots, the attacker deletes the evidence, and correlating events across fifty machines means fifty SSH sessions. Centralised logging solves all of that by shipping log records off the device as they are produced.

Syslog is the protocol that has done this since the 1980s. Its message format is deliberately trivial: a priority, a timestamp, a hostname, a tag, and free-form text.

The priority value (PRI) is the one piece with real structure, and it is a small arithmetic trick worth knowing:

  PRI = facility x 8 + severity

Facility says which subsystem produced the message (kernel=0, mail=2, auth=4, daemon=3, local0–7=16–23 for applications). Severity says how bad it is, and the numbering is inverted from intuition — lower is worse:

Value	Severity	Meaning
0	Emergency	System unusable
1	Alert	Immediate action required
2	Critical	Critical condition
3	Error	Error condition
4	Warning	Warning condition
5	Notice	Normal but significant
6	Informational	Informational
7	Debug	Debug-level

So <34> decodes as facility 4 (auth), severity 2 (critical). Filters expressed as "severity <= 4" mean "warning and worse", which reads backwards until you have seen it a few times.

There are two format generations. BSD syslog (RFC 3164) is the legacy one: no year in the timestamp, no time zone, no sub-second precision, and a structure that was documented after the fact by observing what implementations did. RFC 5424 fixed all of it: ISO 8601 timestamps with time zone and fractional seconds, explicit version, and structured data elements carrying typed key-value pairs.

The transport choice is the operationally significant decision:

Transport	Port	Reliable	Encrypted	Notes
UDP	514	No	No	The default; silently drops under load
TCP	514	Yes	No	Delivery confirmed
TLS	6514	Yes	Yes	RFC 5425; the correct modern choice
RELP	2514	Yes, with acks	Optional	Survives collector restarts

UDP syslog loses messages, and it loses them exactly when you most need them — during the traffic burst that accompanies an incident. It also offers no authentication, so anyone who can reach the collector can inject fabricated log entries, which is a genuine anti-forensics technique. TLS transport fixes both.

The other operational essentials follow from what logs are for:

Time synchronisation is a prerequisite. Correlating events across machines requires their clocks to agree, which is the next topic's subject.
Log integrity matters if logs are evidence: append-only storage, off-host replication, or signed records.
Retention and volume must be planned, because a busy estate produces log volumes that are expensive to keep and slow to search.

Modern stacks (Elasticsearch, Loki, Splunk, journald with remote forwarding) add indexing, structured fields, and query languages, but the shipping layer underneath is usually still syslog.

Examples

The PRI arithmetic, and why filters read backwards

  PRI = facility x 8 + severity

  <34> -> 34 / 8 = 4 remainder 2
          facility 4 = auth,  severity 2 = critical

  SEVERITY IS INVERTED: lower number = more serious

     0 emergency  <- worst
     1 alert
     2 critical
     3 error
     4 warning
     5 notice
     6 informational
     7 debug      <- least serious

  so "*.warning" means "severity <= 4" = warning AND WORSE
     "*.debug"   means "severity <= 7" = everything


 TWO FORMAT GENERATIONS

 RFC 3164 (legacy BSD) -- note what is MISSING
   <34>Oct 11 22:14:15 mymachine su: 'su root' failed for alice
        ^^^^^^^^^^^^^^ no year, no timezone, no sub-second precision
                       -> ambiguous across new year and across regions

 RFC 5424 (modern)
   <34>1 2026-09-23T09:14:15.003Z core-sw-01 su 1234 ID47
       [auth@32473 user="alice" result="fail" src="10.0.0.9"] 'su root' failed
    ^  ^ ^                        ^          ^   ^    ^      ^
   PRI ver  ISO 8601 + TZ + ms   host      app  pid  msgid   STRUCTURED DATA
                                                            (typed key-value)


 CENTRALISATION, and why it is not optional

   [router] --+
   [switch] --+--> [collector] --> index --> search/alert/retain
   [server] --+       ^
   [firewall]--+      |
                      +-- survives the device being wiped, rebooted,
                          filled, or compromised

# Encoding and decoding syslog priorities.

FACILITIES = {0:"kern",1:"user",2:"mail",3:"daemon",4:"auth",5:"syslog",
              6:"lpr",7:"news",8:"uucp",9:"cron",10:"authpriv",11:"ftp",
              16:"local0",17:"local1",18:"local2",19:"local3",
              20:"local4",21:"local5",22:"local6",23:"local7"}
SEVERITIES = {0:"emergency",1:"alert",2:"critical",3:"error",
              4:"warning",5:"notice",6:"informational",7:"debug"}

def decode_pri(pri):
    return FACILITIES.get(pri // 8, "?"), SEVERITIES.get(pri % 8, "?")

def encode_pri(facility_num, severity_num):
    return facility_num * 8 + severity_num

for pri in (0, 34, 86, 165, 191):
    f, s = decode_pri(pri)
    print(f"<{pri:>3}> -> facility={f:<10} severity={s}")
# <  0> -> facility=kern       severity=emergency
# < 34> -> facility=auth       severity=critical
# < 86> -> facility=authpriv   severity=notice
# <165> -> facility=local4     severity=notice
# <191> -> facility=local7     severity=debug

print(f"\nauth.critical encodes to <{encode_pri(4, 2)}>")
# auth.critical encodes to <34>

# Building an RFC 5424 message with structured data.
from datetime import datetime, timezone

def rfc5424(facility, severity, host, app, pid, msgid, sd, message):
    pri = encode_pri(facility, severity)
    ts = datetime.now(timezone.utc).isoformat(timespec="milliseconds").replace("+00:00", "Z")
    sd_str = "".join(
        f"[{sid} " + " ".join(f'{k}="{v}"' for k, v in params.items()) + "]"
        for sid, params in sd.items()) or "-"
    return f"<{pri}>1 {ts} {host} {app} {pid} {msgid} {sd_str} {message}"

print("\n" + rfc5424(4, 2, "core-sw-01", "su", 1234, "ID47",
                     {"auth@32473": {"user": "alice", "result": "fail",
                                     "src": "10.0.0.9"}},
                     "'su root' failed for alice"))
# <34>1 2026-09-23T09:14:15.003Z core-sw-01 su 1234 ID47 [auth@32473 user="alice"
#   result="fail" src="10.0.0.9"] 'su root' failed for alice
#   Structured data is machine-parseable WITHOUT regex over free text.

# Why UDP syslog loses exactly the messages you need.
def delivery(messages_per_sec, collector_capacity, transport, duration_s=60):
    sent = messages_per_sec * duration_s
    if transport == "udp":
        delivered = min(sent, collector_capacity * duration_s)   # silently dropped
        return sent, delivered, sent - delivered
    return sent, sent, 0                     # TCP/TLS: backpressure, none lost

print(f"\n{'scenario':<28}{'sent':>10}{'delivered':>11}{'LOST':>9}")
for name, rate in (("normal operation", 500), ("incident burst", 40_000)):
    for transport in ("udp", "tcp"):
        s, d, l = delivery(rate, 5_000, transport)
        print(f"{name + ' / ' + transport:<28}{s:>10,}{d:>11,}{l:>9,}")
# scenario                          sent  delivered     LOST
# normal operation / udp          30,000     30,000        0
# normal operation / tcp          30,000     30,000        0
# incident burst / udp         2,400,000    300,000 2,100,000
# incident burst / tcp         2,400,000  2,400,000        0
#   UDP drops 87% of the messages generated DURING the incident.

# Log injection: what unauthenticated syslog permits.
forged = rfc5424(4, 6, "core-sw-01", "sshd", 9999, "ID01", {},
                 "Accepted publickey for root from 10.0.0.99")
print(f"\nanyone who can reach udp/514 can send:\n  {forged[:80]}...")
print("  -> indistinguishable from a genuine record without TLS + client certs")

# Volume and retention planning.
def storage_gb(devices, msgs_per_device_per_sec, avg_bytes, days, compression=0.12):
    raw = devices * msgs_per_device_per_sec * 86400 * days * avg_bytes
    return raw / 1e9, raw * compression / 1e9

for days in (7, 90, 365):
    raw, comp = storage_gb(500, 20, 220, days)
    print(f"\n{days:>3} days: {raw:>9,.0f} GB raw, {comp:>8,.0f} GB compressed")
#   7 days:     1,330 GB raw,      160 GB compressed
#  90 days:    17,107 GB raw,    2,053 GB compressed
# 365 days:    69,379 GB raw,    8,325 GB compressed

# Send a test message and watch it arrive:
$ logger -p auth.crit -t su "'su root' failed for alice"
$ journalctl -t su -n 1 --output=short-iso
# 2026-09-23T09:14:15+0000 core-sw-01 su[1234]: 'su root' failed for alice

# Forward everything to a collector over TLS (rsyslog):
$ sudo tee /etc/rsyslog.d/50-remote.conf <<'EOF'
> global(DefaultNetstreamDriver="gtls"
>        DefaultNetstreamDriverCAFile="/etc/ssl/certs/ca.pem")
> action(type="omfwd" target="logs.example.com" port="6514"
>        protocol="tcp" StreamDriverMode="1" StreamDriverAuthMode="x509/name"
>        queue.type="LinkedList" queue.filename="fwdq"
>        action.resumeRetryCount="-1" queue.saveOnShutdown="on")
> EOF
$ sudo rsyslogd -N1 && sudo systemctl restart rsyslog    # validate first
#   The disk-backed queue means a collector outage does not lose messages.

# Verify the transport is actually encrypted:
$ sudo ss -tnp | grep 6514
# ESTAB 0 0 10.0.0.41:41022 203.0.113.50:6514 users:(("rsyslogd",pid=712,fd=8))
$ sudo tcpdump -i any -n -A 'port 6514' -c 2 2>/dev/null | grep -c 'su root'
# 0        <- nothing readable on the wire

# Compare with plain UDP syslog:
$ sudo tcpdump -i any -n -A 'port 514' -c 2 2>/dev/null | grep -o 'su root'
# su root  <- readable, and forgeable by anyone who can reach the port

# Filter by severity -- remember lower is worse:
$ journalctl -p warning -n 5       # warning AND everything more severe
$ journalctl -p 4 --since '1 hour ago' | wc -l
# 37

# Cisco-style device configuration:
# logging host 203.0.113.50 transport tls port 6514
# logging trap warnings             ! severity <= 4
# logging source-interface Loopback0
# service timestamps log datetime msec show-timezone   ! never omit this

# Check the forwarding queue is not silently backing up:
$ sudo rsyslogd -o /dev/null 2>/dev/null; ls -la /var/spool/rsyslog/
# -rw------- 1 root root 0 Sep 23 09:14 fwdq.00000001
#   Non-zero and growing = the collector is unreachable.

Key takeaways

Centralised logging survives the device being wiped, rebooted, filled, or compromised, and makes cross-device correlation possible at all.
PRI = facility × 8 + severity; severity is inverted — 0 is emergency, 7 is debug — so a "warning" filter means warning and worse.
RFC 3164 lacks year, time zone, and sub-second precision; RFC 5424 adds ISO 8601 timestamps and structured data that is parseable without regex.
UDP syslog silently drops messages under load — precisely during an incident — and permits log injection by anyone who can reach the port.
TLS on port 6514 (RFC 5425) is the correct modern transport; disk-backed queues protect against collector outages.
Logs are only correlatable if clocks agree, which makes NTP a prerequisite rather than a nicety.
Retention and volume must be planned: a 500-device estate generates terabytes per year before compression.

🧪 Practice

Decode <134> and <11> into facility and severity, and state which is more urgent.

Explain why a syslog filter set to "warning" also captures errors and emergencies but not informational messages.

Compute compressed storage for 200 devices at 15 messages per second, 200 bytes each, retained for 180 days.

Interview-style: An investigation finds that logs from the hour of the incident are missing, although the collector was healthy throughout. (Hint: what does the transport do when the message rate exceeds what the collector can absorb, and does it tell anyone?)

NTP and Clock Synchronization

Theory

Computer clocks are bad. A typical crystal oscillator drifts by tens of parts per million, which is a few seconds per day — enough that within a week machines across an estate disagree by minutes. Almost nothing in a distributed system tolerates that:

Log correlation becomes impossible; events appear in the wrong order across machines.
TLS certificates fail validation, since "valid from / valid until" is checked against the local clock.
Kerberos rejects authentication outside a 5-minute window by design.
Distributed databases that order writes by timestamp corrupt their ordering.
Scheduled jobs fire at the wrong moment, or twice, or not at all.

NTP (Network Time Protocol) synchronises clocks across a network to within milliseconds over the internet and microseconds on a LAN. Its central problem is subtle and worth appreciating: you cannot simply ask another machine for the time, because the answer is stale by however long it took to arrive, and you do not directly know that.

NTP's solution is four timestamps per exchange:

  t1 = client sends request         (by client's clock)
  t2 = server receives it           (by server's clock)
  t3 = server sends reply           (by server's clock)
  t4 = client receives reply        (by client's clock)

  round-trip delay  d = (t4 - t1) - (t3 - t2)
                        \_______/   \_______/
                        total time   server's processing time

  clock offset      o = ((t2 - t1) + (t3 - t4)) / 2

The offset formula assumes the path is symmetric — that the request and reply each took d/2. That assumption is NTP's main accuracy limit, and it is why asymmetric routing degrades time quality: if the outbound path takes 40 ms and the return takes 10 ms, the computed offset is off by 15 ms and nothing in the protocol can detect it.

NTP is organised in strata, a hierarchy like DNS's:

Stratum 0: the reference clocks themselves — GPS receivers, caesium and rubidium standards. Not network-reachable.
Stratum 1: servers directly attached to a stratum 0 device.
Stratum 2: synchronised to stratum 1, and so on to stratum 15. Stratum 16 means unsynchronised.

Each level adds error, so lower strata are better, but a well-connected stratum 3 on a short path often beats a distant stratum 1.

A client should use at least four sources, because NTP's selection algorithm does more than average: it identifies falsetickers — servers whose time is inconsistent with the majority — and discards them. Three sources cannot break a tie reliably; four can.

Two behaviours are worth knowing operationally. Slewing versus stepping: for small offsets, NTP gently adjusts the clock rate so time never jumps or runs backwards, which matters because monotonic assumptions in software break badly on a backwards step. For large offsets it steps, usually only at start-up. And leap seconds: occasionally inserted to keep UTC aligned with the earth's rotation, historically a recurring source of outages, which is why large operators use leap smearing — spreading the extra second across many hours.

Security matters here too. Unauthenticated NTP can be spoofed, and shifting a target's clock can expire valid certificates or revive revoked ones. NTS (Network Time Security, RFC 8915) adds authentication. Older NTP also had a serious amplification problem via the monlist command, long since disabled.

For sub-microsecond requirements — financial trading, telecom, industrial control — PTP (IEEE 1588) uses hardware timestamping in switches to remove software-stack jitter entirely.

Examples

The four-timestamp exchange

  CLIENT                                            SERVER
    |                                                  |
  t1 ---------- request (contains t1) --------------> t2
    |                                                  |  (processing)
  t4 <--------- reply (contains t1,t2,t3) ----------- t3
    |                                                  |

   delay  d = (t4 - t1) - (t3 - t2)     total elapsed minus server's time
   offset o = ((t2 - t1) + (t3 - t4)) / 2

   ASSUMPTION: the path is symmetric, each direction took d/2.
   If outbound = 40 ms and return = 10 ms, the offset is wrong by 15 ms
   and NOTHING in the protocol can detect it.


 THE STRATUM HIERARCHY

   stratum 0   [GPS] [caesium] [rubidium]     reference clocks, not networked
                 |      |         |
   stratum 1   [ntp1] [ntp2]  [ntp3]          directly attached
                 |      |         |
   stratum 2   [org server]  [ISP server]     sync'd to stratum 1
                 |
   stratum 3   [site server]
                 |
   stratum 4   [your laptop]

   stratum 16 = UNSYNCHRONISED (the value that means "do not trust me")


 WHY FOUR SOURCES

   2 sources: they disagree -- which is wrong? No way to tell.
   3 sources: one disagrees -- probably that one, but a tie is possible.
   4 sources: a clear majority; the outlier ("falseticker") is discarded.

      A: +2 ms   B: +3 ms   C: +2 ms   D: +4200 ms
                                        ^^^^^^^^^ falseticker, rejected

# The NTP offset/delay calculation, including the symmetry assumption.

def ntp_offset_delay(t1, t2, t3, t4):
    delay  = (t4 - t1) - (t3 - t2)
    offset = ((t2 - t1) + (t3 - t4)) / 2
    return offset, delay

# Symmetric path: 20 ms each way, client is 5 s slow.
t1, t2, t3, t4 = 1000.000, 1005.020, 1005.021, 1000.041
o, d = ntp_offset_delay(t1, t2, t3, t4)
print(f"symmetric  : offset={o:+.4f}s delay={d*1000:.1f}ms")
# symmetric  : offset=+5.0000s delay=40.0ms
#   Correct: the client is 5 s behind and should advance by 5 s.

# Asymmetric path: 35 ms out, 5 ms back. Same true offset of 5 s.
t1, t2, t3, t4 = 1000.000, 1005.035, 1005.036, 1000.041
o, d = ntp_offset_delay(t1, t2, t3, t4)
print(f"asymmetric : offset={o:+.4f}s delay={d*1000:.1f}ms  <- {(o-5)*1000:+.1f}ms error")
# asymmetric : offset=+5.0150s delay=40.0ms  <- +15.0ms error
#   The delay looks identical. The error is invisible to the protocol.

# Falseticker rejection: why four sources, not two.
def select_sources(offsets_ms, tolerance=50):
    """Simplified: keep sources clustered with the majority."""
    import statistics
    med = statistics.median(offsets_ms.values())
    good = {k: v for k, v in offsets_ms.items() if abs(v - med) <= tolerance}
    bad  = {k: v for k, v in offsets_ms.items() if abs(v - med) > tolerance}
    return good, bad, (statistics.mean(good.values()) if good else None)

sources = {"ntp1": 2.1, "ntp2": 3.4, "ntp3": 2.8, "ntp4": 4200.0}
good, bad, chosen = select_sources(sources)
print(f"\nsurvivors  : {good}")
print(f"falseticker: {bad}")
print(f"applied    : {chosen:+.2f} ms")
# survivors  : {'ntp1': 2.1, 'ntp2': 3.4, 'ntp3': 2.8}
# falseticker: {'ntp4': 4200.0}
# applied    : +2.77 ms
#   With only ntp1 and ntp4 configured, there is no majority to appeal to.

# Drift: why this must run continuously, not once at boot.
def drift_seconds(ppm, days):
    return ppm * 1e-6 * days * 86400

print(f"\n{'crystal quality':<26}{'drift/day':>12}{'drift/month':>14}")
for name, ppm in (("cheap PC crystal", 50), ("good server oscillator", 5),
                  ("TCXO", 0.5), ("OCXO", 0.01)):
    print(f"{name:<26}{drift_seconds(ppm,1):>10.2f} s{drift_seconds(ppm,30):>12.2f} s")
# crystal quality                drift/day   drift/month
# cheap PC crystal                    4.32 s       129.60 s
# good server oscillator              0.43 s        12.96 s
# TCXO                                0.04 s         1.30 s
# OCXO                                0.00 s         0.03 s
#   A cheap crystal breaks Kerberos (5-min window) in about 3.5 days.

# What breaks, and at what magnitude of error.
thresholds = [
    ("TLS certificate validity",   "hours to days"),
    ("Kerberos authentication",    "5 minutes"),
    ("Log correlation across hosts","seconds"),
    ("Distributed DB write ordering","milliseconds"),
    ("Financial trade sequencing",  "microseconds"),
]
print()
for what, tol in thresholds:
    print(f"  {what:<32} breaks beyond ~{tol}")

# Slew vs step: why backwards jumps are dangerous.
def adjustment_mode(offset_s, step_threshold=0.128):
    if abs(offset_s) < step_threshold:
        return "SLEW (adjust the clock RATE; time never jumps or goes backwards)"
    return "STEP (jump the clock -- only safe at start-up)"

for off in (0.003, 0.100, 2.5, -45.0):
    print(f"\noffset {off:>+7.3f}s -> {adjustment_mode(off)}")
# offset  +0.003s -> SLEW (adjust the clock RATE; time never jumps or goes backwards)
# offset  +0.100s -> SLEW (...)
# offset  +2.500s -> STEP (jump the clock -- only safe at start-up)
# offset -45.000s -> STEP (...)
#   A backwards step breaks anything assuming monotonic time: expiry logic,
#   rate limiters, and duration measurements all misbehave.

# Current synchronisation state -- check this first, always:
$ timedatectl status
#                Local time: Wed 2026-09-23 09:14:15 UTC
#               NTP service: active
#           System clock synchronized: yes
#   "synchronized: no" explains an astonishing range of unrelated-looking faults.

# Which sources are in use, and which were rejected:
$ chronyc sources -v
# MS Name/IP address        Stratum Poll Reach LastRx Last sample
# ^* time1.example.com            2    6   377    21  -0.000412s[-0.000398s]
# ^+ time2.example.com            2    6   377    19  +0.000891s[+0.000891s]
# ^+ time3.example.com            3    6   377    23  -0.001204s[-0.001204s]
# ^x time4.example.com            2    6   377    18  +4.201033s[+4.201033s]
#  ^ ^* = selected, ^+ = acceptable, ^x = FALSETICKER (rejected)
#    Reach 377 (octal) = the last 8 polls all succeeded.

# Measured offset, delay, and the symmetry problem in one view:
$ chronyc tracking
# Reference ID    : C0A80101 (time1.example.com)
# Stratum         : 3
# System time     : 0.000038 seconds slow of NTP time
# Root delay      : 0.021 seconds
# Root dispersion : 0.001 seconds
# Skew            : 0.412 ppm
# Frequency       : 12.043 ppm slow of UTC
#                   ^ the crystal's measured drift, corrected continuously

# Query a server directly and see all four timestamps:
$ ntpdate -q time.example.com
# server 203.0.113.9, stratum 2, offset -0.000412, delay 0.02134

# Enable authenticated time (NTS) -- unauthenticated NTP is spoofable:
$ sudo tee -a /etc/chrony/chrony.conf <<'EOF'
> server time.cloudflare.com iburst nts
> server ptbtime1.ptb.de iburst nts
> server nts.netnod.se iburst nts
> server time.system76.com iburst nts
> EOF
$ sudo systemctl restart chronyd && chronyc -N authdata
# Name/IP address       Mode KeyID Type KLen Last Atmp NAK Cook CLen
# time.cloudflare.com    NTS     1   15  256  23m    0   0    8  100
#                        ^^^ authenticated: spoofing now requires the key

# Verify at least four sources are actually reachable:
$ chronyc sources | grep -c '^\^'
# 4

# Diagnose the classic symptom -- TLS failing for no apparent reason:
$ curl https://example.com/ 2>&1 | head -2
# curl: (60) SSL certificate problem: certificate is not yet valid
$ timedatectl | grep -i synchronized
# System clock synchronized: no      <- there it is

# Sub-microsecond requirements need hardware timestamping (PTP):
$ sudo ptp4l -i eth0 -m -S 2>&1 | head -3
# ptp4l[12.345]: selected /dev/ptp0 as PTP clock
# ptp4l[13.456]: master offset  -47 s2 freq  +1204 path delay  8241
#                               ^^^ nanoseconds, not milliseconds

Key takeaways

Crystal drift of tens of ppm means seconds per day, which breaks TLS validation, Kerberos, log correlation, and distributed write ordering.
NTP uses four timestamps to compute offset and round-trip delay, because a bare answer is already stale on arrival.
The offset formula assumes a symmetric path; asymmetric routing introduces error that the protocol cannot detect.
Strata run from 0 (reference clocks) to 15, with 16 meaning unsynchronised; each level adds error but path quality can matter more than stratum number.
Configure at least four sources so the selection algorithm can identify and discard falsetickers.
Slewing adjusts the clock rate for small offsets so time never runs backwards; stepping is reserved for large offsets, normally at start-up.
NTS (RFC 8915) authenticates time, closing a spoofing vector that can expire valid certificates; PTP (IEEE 1588) provides sub-microsecond accuracy via hardware timestamping.

🧪 Practice

Given t1=100.000, t2=100.512, t3=100.513, t4=100.031, compute the offset and delay and say which way the client's clock must move.

Explain why asymmetric routing degrades NTP accuracy and why the reported delay does not reveal the problem.

A machine with a 50 ppm crystal last synchronised 4 days ago. Compute its drift and state which services have already broken.

Interview-style: Why does NTP prefer to slew rather than step, and what breaks when a clock jumps backwards? (Hint: think about every piece of code that measures a duration by subtracting two readings of the clock.)

NETCONF, RESTCONF, and YANG

Theory

Network devices were configured for decades by typing into a CLI. That works for one device and fails completely at scale, for reasons that are structural rather than merely inconvenient:

Output is formatted for humans, so automation resorts to screen-scraping, which breaks whenever a vendor adjusts a column.
Syntax is vendor-specific, so every vendor needs separate tooling.
There are no transactions. A configuration is applied line by line; a failure halfway leaves the device in a state that is neither the old one nor the new one.
There is no schema, so nothing can validate a change before it is applied.

NETCONF (RFC 6241), RESTCONF (RFC 8040), and YANG (RFC 7950) are the standards-based answer, and they divide the work cleanly: YANG is the data model, while NETCONF and RESTCONF are two protocols for moving that data.

YANG is a modelling language that defines the structure of configuration and state: hierarchical containers and lists, typed leaves with ranges and patterns, mandatory and optional nodes, and constraints (must, when). Because the model is machine-readable, tooling can validate a configuration before sending it and generate APIs and documentation from the same source. The IETF and OpenConfig publish vendor-neutral models, so interfaces/interface/config/mtu means the same thing across implementations.

NETCONF carries YANG data as XML over SSH (port 830), using RPCs: get-config, edit-config, copy-config, lock, unlock, commit, validate. Its decisive feature is datastores — separate running, candidate, and startup configurations. You edit the candidate, validate it, then commit it atomically. Add confirmed-commit and the device automatically rolls back if you do not confirm within a timeout, which means a change that cuts off your own management access undoes itself rather than requiring a site visit.

RESTCONF exposes the same YANG models over HTTPS with REST conventions and JSON or XML: GET to read, PUT/PATCH to modify, DELETE to remove, with resource paths derived mechanically from the YANG tree. It is far easier to consume from ordinary web tooling, at the cost of NETCONF's transactional datastores.

	CLI scraping	NETCONF	RESTCONF
Transport	SSH (interactive)	SSH (830)	HTTPS (443)
Encoding	Human text	XML	JSON or XML
Schema	None	YANG	YANG
Transactions	No	Yes (candidate + commit)	No
Rollback	Manual	Automatic (confirmed-commit)	No
Validation before apply	No	Yes	Partial
Tooling	Fragile	Specialised	Ordinary HTTP tools

gNMI (gRPC Network Management Interface) is the newer alternative, using protobuf over gRPC/HTTP2 and adding efficient streaming telemetry — a push model that replaces SNMP polling for high-frequency metrics, with the same request-response-versus-streaming trade-off seen throughout this chapter.

The practical guidance: use NETCONF where transactional safety matters (core routers, changes that can lock you out), RESTCONF where integration simplicity matters, and gNMI for telemetry at scale.

Examples

The layering: one model, several protocols

   +--------------------------------------------------+
   |            YANG DATA MODELS                       |
   |  ietf-interfaces, openconfig-bgp, vendor modules  |
   |  typed, constrained, machine-readable             |
   +--------------------------------------------------+
        |               |                 |
   +---------+    +------------+    +-----------+
   | NETCONF |    | RESTCONF   |    |   gNMI    |
   | XML/SSH |    | JSON/HTTPS |    | protobuf  |
   | port 830|    | port 443   |    | gRPC      |
   +---------+    +------------+    +-----------+
   transactional   easy to consume   streaming telemetry


 NETCONF DATASTORES -- why this is safer than a CLI

   +-----------+  edit-config   +-----------+   commit   +-----------+
   | CANDIDATE | <------------- |  client   | --------> |  RUNNING  |
   |  (draft)  | --validate-->  |           |  ATOMIC   | (in force)|
   +-----------+                +-----------+            +-----------+
                                                               |
                                                          copy-config
                                                               v
                                                         +-----------+
                                                         |  STARTUP  |
                                                         | (on boot) |
                                                         +-----------+

   CLI:      line 1 ok, line 2 ok, line 3 FAILS -> half-configured device
   NETCONF:  validate -> commit -> all of it, or none of it


 CONFIRMED COMMIT -- the feature that saves the engineer

   t=0    commit confirmed 300      change applies
   t=1    ...your SSH session dies because the change broke management access
   t=300  no confirmation received -> DEVICE ROLLS ITSELF BACK
                                      no site visit required

# A YANG model, and the validation it makes possible BEFORE sending anything.

yang_model = """
module example-interfaces {
  namespace "urn:example:interfaces";
  prefix exif;

  container interfaces {
    list interface {
      key "name";                          // uniqueness enforced by the model
      leaf name   { type string; }
      leaf enabled{ type boolean; default "true"; }
      leaf mtu    { type uint16 { range "68..9216"; } }   // CONSTRAINT
      leaf description { type string { length "0..255"; } }
    }
  }
}
"""

SCHEMA = {
    "mtu":         {"type": "uint16", "range": (68, 9216)},
    "enabled":     {"type": "boolean"},
    "name":        {"type": "string", "mandatory": True},
    "description": {"type": "string", "length": (0, 255)},
}

def validate(config):
    """The point of a schema: reject bad config BEFORE touching the device."""
    errors = []
    for iface in config.get("interface", []):
        for key, value in iface.items():
            spec = SCHEMA.get(key)
            if not spec:
                errors.append(f"{iface.get('name','?')}: unknown leaf '{key}'")
                continue
            if spec["type"] == "uint16":
                lo, hi = spec["range"]
                if not isinstance(value, int) or not lo <= value <= hi:
                    errors.append(f"{iface['name']}: mtu {value} outside {lo}..{hi}")
            if spec["type"] == "boolean" and not isinstance(value, bool):
                errors.append(f"{iface['name']}: enabled must be boolean")
        if "name" not in iface:
            errors.append("interface missing mandatory key 'name'")
    names = [i.get("name") for i in config.get("interface", [])]
    if len(names) != len(set(names)):
        errors.append("duplicate interface key")
    return errors

good = {"interface": [{"name": "eth0", "mtu": 9000, "enabled": True}]}
bad  = {"interface": [{"name": "eth0", "mtu": 65000, "enabled": "yes"},
                      {"name": "eth0", "mtu": 1500},
                      {"mtu": 1500}]}

print("valid config errors  :", validate(good))
print("invalid config errors:")
for e in validate(bad):
    print("   -", e)
# valid config errors  : []
# invalid config errors:
#    - eth0: mtu 65000 outside 68..9216
#    - eth0: enabled must be boolean
#    - duplicate interface key
#    - interface missing mandatory key 'name'
#   None of this reached the device. A CLI would have applied line 1 first.

# The transaction difference, simulated.
def apply_cli(device, lines):
    """Line by line: a failure leaves the device half-configured."""
    for i, (key, value, ok) in enumerate(lines):
        if not ok:
            return f"FAILED at line {i+1}; lines 1..{i} ARE STILL APPLIED", device
        device[key] = value
    return "ok", device

def apply_netconf(device, lines):
    """Candidate + validate + commit: all or nothing."""
    candidate = dict(device)
    for key, value, ok in lines:
        if not ok:
            return "validate FAILED; running config untouched", device
        candidate[key] = value
    return "committed atomically", candidate

change = [("mtu", 9000, True), ("vlan", 20, True), ("bgp-asn", "invalid", False)]
start = {"mtu": 1500, "vlan": 10}

status, result = apply_cli(dict(start), change)
print(f"\nCLI     : {status}\n          device now: {result}")
status, result = apply_netconf(dict(start), change)
print(f"NETCONF : {status}\n          device now: {result}")
# CLI     : FAILED at line 3; lines 1..2 ARE STILL APPLIED
#           device now: {'mtu': 9000, 'vlan': 20}        <- neither old nor new
# NETCONF : validate FAILED; running config untouched
#           device now: {'mtu': 1500, 'vlan': 10}        <- safely unchanged

# RESTCONF paths are derived mechanically from the YANG tree.
def restconf_path(module, container, list_name, key, leaf=None):
    p = f"/restconf/data/{module}:{container}/{list_name}={key}"
    return p + (f"/{leaf}" if leaf else "")

print(f"\n{restconf_path('ietf-interfaces','interfaces','interface','eth0')}")
print(f"{restconf_path('ietf-interfaces','interfaces','interface','eth0','mtu')}")
# /restconf/data/ietf-interfaces:interfaces/interface=eth0
# /restconf/data/ietf-interfaces:interfaces/interface=eth0/mtu
#   No guesswork: the URL is a direct transcription of the model.

# Discover what a device supports (capabilities include every YANG module):
$ ssh -s admin@router.example.com netconf <<'EOF'
> <?xml version="1.0"?>
> <hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
>   <capabilities><capability>urn:ietf:params:netconf:base:1.1</capability></capabilities>
> </hello>
> ]]>]]>
> EOF
# <capability>urn:ietf:params:netconf:capability:candidate:1.0</capability>
# <capability>urn:ietf:params:netconf:capability:confirmed-commit:1.1</capability>
# <capability>urn:ietf:params:xml:ns:yang:ietf-interfaces?revision=2018-02-20</capability>

# The safe change sequence: edit candidate, validate, commit with confirmation.
# <edit-config><target><candidate/></target><config>...</config></edit-config>
# <validate><source><candidate/></source></validate>
# <commit><confirmed/><confirm-timeout>300</confirm-timeout></commit>
#   If you lose access, the device reverts in 5 minutes by itself.
# <commit/>     <- only now is it permanent

# RESTCONF is ordinary HTTP, which is the point:
$ curl -sk -u admin:secret \
  -H 'Accept: application/yang-data+json' \
  https://router.example.com/restconf/data/ietf-interfaces:interfaces | jq .
# {
#   "ietf-interfaces:interfaces": {
#     "interface": [
#       { "name": "GigabitEthernet1", "type": "iana-if-type:ethernetCsmacd",
#         "enabled": true, "ietf-ip:ipv4": { "address": [...] } }
#     ]
#   }
# }

# Change a single leaf:
$ curl -sk -u admin:secret -X PATCH \
  -H 'Content-Type: application/yang-data+json' \
  -d '{"ietf-interfaces:mtu": 9000}' \
  https://router.example.com/restconf/data/ietf-interfaces:interfaces/interface=Gi1/mtu
# HTTP/1.1 204 No Content

# Validate a config against its YANG model offline, before any device sees it:
$ yanglint -s ietf-interfaces.yang candidate-config.xml
# (silence means valid)
$ yanglint -s ietf-interfaces.yang broken-config.xml
# err : Value "65000" is out of type uint16 min/max bounds. (/ietf-interfaces:...)

# Inspect a model's structure as a tree:
$ pyang -f tree ietf-interfaces.yang | head -8
# module: ietf-interfaces
#   +--rw interfaces
#      +--rw interface* [name]
#         +--rw name           string
#         +--rw enabled?       boolean
#         +--rw type           identityref

# gNMI streaming telemetry: push instead of poll:
$ gnmic -a router:57400 -u admin -p secret subscribe \
    --path '/interfaces/interface[name=eth0]/state/counters' --stream-mode sample \
    --sample-interval 10s
# {"source":"router:57400","timestamp":...,"updates":[{"Path":"...in-octets","values":
#   {"in-octets":4102884213}}]}
#   Continuous push; no polling loop, and far higher frequency than SNMP.

Key takeaways

CLI automation fails structurally: human-formatted output, vendor-specific syntax, no transactions, no schema.
YANG is the data model — typed, constrained, machine-readable — while NETCONF and RESTCONF are transports for it.
NETCONF uses XML over SSH (port 830) and provides candidate/running/startup datastores with validate and atomic commit.
confirmed-commit automatically rolls back if the change is not confirmed — protection against locking yourself out of the device.
RESTCONF exposes the same models over HTTPS with JSON, trading transactions for ease of integration with ordinary web tooling.
Vendor-neutral models (IETF, OpenConfig) make one configuration idea portable across implementations.
gNMI adds streaming telemetry over gRPC, replacing SNMP polling for high-frequency metrics.

🧪 Practice

Explain the role of each NETCONF datastore and the sequence of operations for a safe configuration change.

Write the RESTCONF URL to read the MTU of interface eth1 from the ietf-interfaces module.

Describe what happens under CLI configuration versus NETCONF when the third of five configuration lines is invalid.

Interview-style: What specific failure does confirmed-commit protect against, and why is that failure so expensive without it? (Hint: imagine the change is a firewall rule applied to the interface you are connected through, on a device in another country.)

8.6 Content Delivery and Streaming

Delivering bulk content and real-time media are the two hardest things applications ask of a network: this subchapter covers how distribution is scaled geographically and how audio and video survive a best-effort internet.

Content Delivery Network Architecture

Theory

Chapter 1 established that the speed of light sets a floor on latency, and no amount of bandwidth removes it. A round trip from Sydney to Frankfurt is roughly 250 ms in fibre, and that is physics. If the content lives in Frankfurt, an Australian user pays that on every request, and a TCP-plus-TLS handshake multiplies it by three before a single byte of content arrives.

The only way to reduce the distance is to move the content closer. That is all a CDN is: a large, geographically distributed set of caching servers — edge PoPs (points of presence) — placed near users, in front of a small number of origin servers that hold the authoritative content.

The benefits compound in a way that is worth enumerating, because latency is only the most obvious one:

Latency: 250 ms becomes 10 ms because the bytes come from the same city.
Origin offload: a 95% edge hit rate means the origin serves 5% of requests, so it can be far smaller.
Bandwidth cost: traffic is served from peering-rich locations rather than from expensive origin transit.
Availability: an origin outage is invisible for cached content, and CDNs can serve stale content deliberately during one.
Attack absorption: a volumetric DDoS hits hundreds of well-provisioned PoPs rather than one origin.
TLS termination at the edge: the expensive handshake happens 10 ms away instead of 250 ms away, which is often the single largest win.

The architecture is tiered, because a flat edge layer would each independently miss to the origin:

  user -> edge PoP (city) -> mid-tier/shield (region) -> origin

The shield layer collapses many edge misses into one origin request, protecting the origin from a cache stampede when a popular object expires everywhere at once.

Two concepts are the source of most CDN operational work.

Cache key. What makes two requests "the same object"? By default the URL, but it may include the query string, selected headers, device class, or country. Too broad a key and users get each other's content; too narrow and the hit rate collapses because every variant is a separate object.

Invalidation. Getting content into a CDN is trivial; getting it out is not. Purging is slow and rate-limited at scale, so the strong pattern is the immutable content-hashed asset from 8.3: never invalidate, just change the URL. Where purging is unavoidable, cache tags (surrogate keys) let one operation invalidate a related group.

Modern CDNs go further than caching: they run code at the edge (edge functions), terminate TLS, apply WAF rules, and perform image optimisation, which turns the CDN into part of the application rather than a transparent accelerator.

Examples

Why distance is the dominant cost

  WITHOUT A CDN                       WITH A CDN
  user (Sydney)                       user (Sydney)
       |  250 ms RTT                       |  8 ms RTT
       |  x3 for TCP+TLS+HTTP              |  x3
       v                                   v
  origin (Frankfurt)                  edge PoP (Sydney)
                                           |  cache MISS only (~5%)
  first byte: ~750 ms                      v
                                      shield (Singapore)
                                           |
                                           v
                                      origin (Frankfurt)

                                      first byte: ~24 ms (hit)


 THE TIERED HIERARCHY, and why the shield exists

   400 edge PoPs each missing on the same object at the same moment:
     without a shield:  400 simultaneous origin requests  <- stampede
     with a shield:       1 origin request, 399 served from the shield


 CACHE KEY: the setting that causes both outages and poor hit rates

   key = URL only
     /page?utm_source=twitter  and  /page?utm_source=email
     -> ONE object. Good hit rate. Correct, if the query does not change output.

   key = URL + full query string
     -> a separate object per tracking parameter. Hit rate collapses.

   key = URL + Cookie
     -> a separate object PER USER. The cache is now useless.
        (and if the response is marked public, you have an incident)

# What a CDN is worth, quantified: latency, origin load, and bandwidth.

def first_byte_ms(rtt_ms, tls=True, cache_hit=True, origin_rtt_ms=0):
    handshakes = 3 if tls else 2                  # TCP + TLS + request
    t = rtt_ms * handshakes
    if not cache_hit:
        t += origin_rtt_ms * 2                    # edge must fetch from origin
    return t

print(f"{'scenario':<34}{'first byte':>12}")
print(f"{'direct to origin (250 ms away)':<34}{first_byte_ms(250):>10} ms")
print(f"{'CDN hit (8 ms away)':<34}{first_byte_ms(8):>10} ms")
print(f"{'CDN miss (8 ms + 240 to origin)':<34}"
      f"{first_byte_ms(8, cache_hit=False, origin_rtt_ms=240):>10} ms")
# scenario                            first byte
# direct to origin (250 ms away)           750 ms
# CDN hit (8 ms away)                       24 ms
# CDN miss (8 ms + 240 to origin)          504 ms
#   Even a MISS beats going direct, because TLS terminated at the edge.

# Origin offload: the number that decides how big the origin must be.
def origin_load(total_rps, edge_hit_rate, shield_hit_rate=0.0):
    edge_misses = total_rps * (1 - edge_hit_rate)
    return edge_misses * (1 - shield_hit_rate)

print()
for hit in (0.0, 0.80, 0.95, 0.99):
    plain  = origin_load(500_000, hit)
    shield = origin_load(500_000, hit, shield_hit_rate=0.9)
    print(f"edge hit {hit:>5.0%} -> origin {plain:>10,.0f} rps "
          f"| with shield {shield:>9,.0f} rps")
# edge hit    0% -> origin    500,000 rps | with shield    50,000 rps
# edge hit   80% -> origin    100,000 rps | with shield    10,000 rps
# edge hit   95% -> origin     25,000 rps | with shield     2,500 rps
# edge hit   99% -> origin      5,000 rps | with shield       500 rps
#   At 99% + shield, an origin that would have needed hundreds of machines
#   runs comfortably on two.

# Cache key design: the direct trade-off between correctness and hit rate.
def hit_rate(requests, key_fn):
    seen, hits = set(), 0
    for r in requests:
        k = key_fn(r)
        if k in seen:
            hits += 1
        else:
            seen.add(k)
    return hits / len(requests)

requests = [
    {"url": "/article", "query": "utm_source=twitter", "cookie": "sid=a", "country": "AU"},
    {"url": "/article", "query": "utm_source=email",   "cookie": "sid=b", "country": "AU"},
    {"url": "/article", "query": "",                   "cookie": "sid=c", "country": "DE"},
    {"url": "/article", "query": "utm_source=twitter", "cookie": "sid=d", "country": "AU"},
    {"url": "/article", "query": "",                   "cookie": "sid=e", "country": "AU"},
]
keys = {
    "url only":            lambda r: r["url"],
    "url + query":         lambda r: (r["url"], r["query"]),
    "url + country":       lambda r: (r["url"], r["country"]),
    "url + cookie":        lambda r: (r["url"], r["cookie"]),
}
print()
for name, fn in keys.items():
    print(f"cache key: {name:<18} hit rate {hit_rate(requests, fn):>6.0%}")
# cache key: url only           hit rate    80%
# cache key: url + query        hit rate    40%
# cache key: url + country      hit rate    40%
# cache key: url + cookie       hit rate     0%
#   Keying on Cookie gives a 0% hit rate AND risks serving one user's
#   content to another if the response is marked public.

# Cache stampede: why the shield tier is not optional.
def stampede(edge_pops, object_popularity_rps, ttl_s, shielded):
    """When a popular object expires, every PoP misses simultaneously."""
    simultaneous = edge_pops if not shielded else 1
    return simultaneous, simultaneous * object_popularity_rps / edge_pops

for shielded in (False, True):
    n, rps = stampede(400, 500_000, 60, shielded)
    print(f"\nshield={'yes' if shielded else 'no ':<3} -> {n:>3} simultaneous "
          f"origin requests at TTL expiry")
# shield=no  -> 400 simultaneous origin requests at TTL expiry
# shield=yes ->   1 simultaneous origin requests at TTL expiry

# Which PoP served you, and was it a hit?
$ curl -sI https://cdn.example.com/static/app.4f9a2c.js | \
    grep -iE 'x-cache|x-served-by|age|cf-ray|server-timing'
# x-cache: HIT
# x-served-by: cache-syd1234-SYD
#                        ^^^ Sydney PoP
# age: 84213
#   age = seconds this object has been in the cache.

# Compare edge and origin latency from the same machine:
$ curl -s -o /dev/null -w 'edge   %{time_appconnect} tls, %{time_starttransfer} ttfb\n' \
       https://cdn.example.com/static/app.js
# edge   0.024 tls, 0.031 ttfb
$ curl -s -o /dev/null -w 'origin %{time_appconnect} tls, %{time_starttransfer} ttfb\n' \
       https://origin.example.com/static/app.js
# origin 0.742 tls, 0.981 ttfb
#   30x. Most of it is the TLS handshake happening nearby.

# Measure the hit rate over a sample of requests:
$ for i in $(seq 1 20); do
>   curl -sI "https://cdn.example.com/static/app.js?v=$((RANDOM % 3))" | \
>     awk '/x-cache/{print $2}'
> done | sort | uniq -c
#      17 HIT
#       3 MISS
#   85% on a 3-variant key. Randomising the query would give 0%.

# Inspect what the CDN uses as its cache key:
$ curl -sI https://cdn.example.com/page | grep -iE 'vary|cache-control'
# vary: Accept-Encoding, Accept-Language
# cache-control: public, max-age=300, s-maxage=3600, stale-while-revalidate=86400
#                                     ^^^^^^^^^^^^ shared caches keep it 1 h
#                                                  ^ serve stale for a day
#                                                    while refreshing in background

# Purge by tag rather than by URL -- one call, many objects:
$ curl -sX POST -H "Authorization: Bearer $TOKEN" \
  -d '{"tags":["product-1234"]}' https://api.cdn.example.com/purge
# {"queued": true, "objects_affected": 3184}

# Confirm an origin is genuinely shielded (only the shield should reach it):
$ awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -3
#  892134 203.0.113.10     <- the shield PoP
#      12 198.51.100.7     <- health checks
#   If you see hundreds of distinct source addresses, shielding is not working.

Key takeaways

A CDN exists because distance is physics: the only way to cut RTT is to move content closer to users.
Beyond latency it provides origin offload, cheaper bandwidth, availability during origin outages, DDoS absorption, and edge TLS termination — the last of which is often the biggest single win.
The hierarchy is edge PoP → shield/mid-tier → origin; the shield collapses simultaneous edge misses and prevents cache stampedes.
The cache key determines what counts as the same object: too broad risks serving the wrong content, too narrow destroys the hit rate — keying on Cookie gives 0%.
Invalidation is the hard part: prefer immutable content-hashed URLs so nothing ever needs purging, and use cache tags when purging is unavoidable.
Even a cache miss usually beats going direct, because the TLS handshake still completes at the nearby edge.

🧪 Practice

Compute time-to-first-byte for a 200 ms origin and a 10 ms edge, with and without a cache hit, assuming TLS.

A service handles 1M rps with a 97% edge hit rate and a shield absorbing 90% of misses. Compute the origin's request rate.

Explain why including the full query string in the cache key can destroy hit rate, and give a safer alternative.

Interview-style: Your CDN reports a 99% hit rate but the origin is still overloaded every hour on the hour. (Hint: 1% of a very large number, arriving all at once — what happens to a popular object at the moment its TTL expires everywhere?)

DNS-Based and Anycast Server Selection

Theory

A CDN with 400 PoPs raises an immediate question: when a user requests cdn.example.com, which PoP should answer? Getting this wrong wastes the entire investment — routing a Sydney user to Frankfurt makes the CDN slower than no CDN at all.

There are two principal mechanisms, and they solve the problem at different layers.

DNS-based selection decides during name resolution. The authoritative DNS server looks at who is asking and returns the address of a nearby PoP. Each user gets a different answer for the same name.

Its strengths are control and granularity: the CDN can weight by PoP capacity, steer around an outage, perform gradual rollouts, and apply per-customer policy — all as data, changed in seconds.

Its weaknesses are structural:

The resolver's location is not the user's location. The authoritative server sees the recursive resolver's address, not the client's. A user in Sydney using a resolver in Singapore gets a Singapore PoP. EDNS Client Subnet (ECS) partially fixes this by forwarding a truncated client prefix, at a real privacy cost, and public resolvers differ in whether they send it.
TTL granularity limits failover. A cached answer persists for its TTL, so failover cannot be faster than that, and resolvers that enforce TTL minimums make it slower still.

Anycast decides in the routing layer. The same IP address is announced by BGP from many locations, and the internet's routing system delivers each packet to whichever announcement is closest in BGP terms. There is one DNS answer for everyone; the network does the steering.

Anycast's strengths are that failover is automatic — withdraw the route and traffic reroutes in seconds without touching DNS — and that it is inherently DDoS-resistant, because an attack is split across every site announcing the prefix. It is what root DNS servers use, and what makes "13 root servers" mean 1,500 machines.

Its weaknesses are equally structural:

"Closest in BGP" is not "closest in geography or latency." BGP chooses on AS path length and local policy, not distance or performance, so a user can be routed past a nearer PoP.
Route changes can move a connection mid-flight. For UDP/DNS this is harmless. For a long-lived TCP connection, arriving at a different PoP means a reset — which is why anycast suits DNS and short HTTP exchanges better than long downloads, and why QUIC's connection IDs help.
Granular control is limited: you steer with BGP attributes, which is coarse compared with a DNS answer per user.

	DNS-based	Anycast
Decision layer	Application (DNS)	Routing (BGP)
Granularity	Per resolver / per subnet	Per route
Failover speed	TTL-limited	Seconds, automatic
Client location accuracy	Resolver's, unless ECS	True network topology
Long-lived TCP	Stable	May break on reroute
DDoS resistance	Limited	Excellent — attack is split

In practice large CDNs use both: anycast for the coarse regional split and resilience, DNS for finer steering within a region. A third mechanism, HTTP redirect, is precise but costs an extra round trip and is now used mainly for video manifest steering.

Examples

Two mechanisms, two layers

 DNS-BASED: a different ANSWER per user

   [Sydney user] -> resolver -> [authoritative DNS]
                                    "who is asking? 1.2.3.0/24 -> AU"
                   <- 203.0.113.9 (Sydney PoP)
   [Berlin user] -> resolver -> [authoritative DNS]
                   <- 198.51.100.4 (Frankfurt PoP)

   ONE name, MANY answers. Control lives in the DNS response.

   THE RESOLVER PROBLEM:
     [Sydney user] -> [resolver in Singapore] -> authoritative
                                                 sees SINGAPORE
                      <- Singapore PoP  (user is 6,000 km away)
     ECS fixes this by forwarding 1.2.3.0/24 with the query.


 ANYCAST: the SAME address, announced from everywhere

                     203.0.113.9 announced by BGP from:
        [Sydney PoP]     [Frankfurt PoP]     [Sao Paulo PoP]
              \                 |                   /
               \                |                  /
            ---------- the internet's routing table ----------
                              |
                    each packet goes to the
                    topologically nearest announcement

   ONE name, ONE answer, and the NETWORK steers.

   FAILOVER:  Sydney PoP withdraws its BGP announcement
              -> traffic reroutes to the next-nearest in SECONDS
              -> no DNS change, no TTL to wait for


 THE ANYCAST TCP HAZARD

   t=0   client establishes TCP to 203.0.113.9  -> reaches Sydney
   t=5   a BGP route change occurs
   t=6   client's next packet -> reaches MELBOURNE
         Melbourne has no state for this connection -> RST
   (harmless for a single DNS query; fatal for a 2 GB download)

# DNS-based selection, including the resolver-location problem.

POPS = {
    "syd": {"ip": "203.0.113.9",  "lat": -33.9, "lon": 151.2, "capacity": 100},
    "fra": {"ip": "198.51.100.4", "lat":  50.1, "lon":   8.7, "capacity": 200},
    "sin": {"ip": "192.0.2.15",   "lat":   1.3, "lon": 103.8, "capacity": 150},
}
import math
def km(a_lat, a_lon, b_lat, b_lon):
    R = 6371
    dlat, dlon = math.radians(b_lat - a_lat), math.radians(b_lon - a_lon)
    h = (math.sin(dlat/2)**2 + math.cos(math.radians(a_lat)) *
         math.cos(math.radians(b_lat)) * math.sin(dlon/2)**2)
    return 2 * R * math.asin(math.sqrt(h))

def select_pop(lat, lon, healthy=None):
    healthy = healthy or set(POPS)
    best = min((p for p in POPS if p in healthy),
               key=lambda p: km(lat, lon, POPS[p]["lat"], POPS[p]["lon"]))
    return best, km(lat, lon, POPS[best]["lat"], POPS[best]["lon"])

user = (-33.87, 151.21)          # Sydney
resolver_sg = (1.35, 103.82)     # a resolver in Singapore

pop, d = select_pop(*user)
print(f"if the CDN sees the USER     : {pop} ({d:>6,.0f} km, ~{d/200:>4.0f} ms RTT)")
pop, d = select_pop(*resolver_sg)
true_d = km(user[0], user[1], POPS[pop]["lat"], POPS[pop]["lon"])
print(f"if the CDN sees the RESOLVER : {pop} ({true_d:>6,.0f} km, ~{true_d/200:>4.0f} ms RTT)")
# if the CDN sees the USER     : syd (     4 km, ~   0 ms RTT)
# if the CDN sees the RESOLVER : sin ( 6,304 km, ~  32 ms RTT)
#   This is exactly what EDNS Client Subnet exists to fix.

# Failover speed: DNS is bounded by TTL; anycast is bounded by BGP convergence.
def failover_seconds(mechanism, ttl=60, resolver_min_ttl=30, bgp_converge=15):
    if mechanism == "dns":
        return max(ttl, resolver_min_ttl)      # cached answers persist
    return bgp_converge

print()
for ttl in (30, 60, 300):
    print(f"DNS failover (ttl={ttl:>3}s) : up to {failover_seconds('dns', ttl):>3} s of errors")
print(f"Anycast failover           : ~{failover_seconds('anycast'):>3} s, automatic")
# DNS failover (ttl= 30s) : up to  30 s of errors
# DNS failover (ttl= 60s) : up to  60 s of errors
# DNS failover (ttl=300s) : up to 300 s of errors
# Anycast failover           : ~ 15 s, automatic

# DDoS absorption: anycast splits the attack across every site.
def attack_per_site(total_gbps, sites, mechanism):
    return total_gbps if mechanism == "unicast" else total_gbps / sites

for mech, sites in (("unicast", 1), ("anycast", 120)):
    per = attack_per_site(2400, sites, mech)
    status = "OVERWHELMED" if per > 100 else "absorbed"
    print(f"\n2.4 Tb/s attack, {mech:<8} ({sites:>3} sites): "
          f"{per:>8,.1f} Gb/s per site -> {status}")
# 2.4 Tb/s attack, unicast  (  1 sites):  2,400.0 Gb/s per site -> OVERWHELMED
# 2.4 Tb/s attack, anycast  (120 sites):     20.0 Gb/s per site -> absorbed

# The anycast TCP hazard, and why QUIC is more tolerant.
def reroute_effect(protocol, bytes_transferred, total_bytes):
    if protocol == "tcp":
        return f"RST -- restart from 0 ({bytes_transferred:,} bytes wasted)"
    return "connection ID is recognised OR a fast reconnect; loss is minimal"

print()
for proto in ("tcp", "quic"):
    print(f"reroute mid-transfer, {proto:<5}: {reroute_effect(proto, 1_400_000_000, 2e9)}")
# reroute mid-transfer, tcp  : RST -- restart from 0 (1,400,000,000 bytes wasted)
# reroute mid-transfer, quic : connection ID is recognised OR a fast reconnect; ...

# DNS-based selection: the same name, different answers by location.
$ dig +short @1.1.1.1 cdn.example.com
# 203.0.113.9
$ dig +short @8.8.8.8 cdn.example.com
# 198.51.100.4
#   Different resolvers, different answers -> DNS-based steering is in use.

# Test what a specific client subnet would be given (ECS):
$ dig +subnet=1.2.3.0/24 @8.8.8.8 cdn.example.com +short
# 203.0.113.9
$ dig +subnet=80.90.0.0/24 @8.8.8.8 cdn.example.com +short
# 198.51.100.4
#   If the answer does not change with +subnet, ECS is not being honoured.

# Anycast: the same address everywhere, but a different machine answers.
$ dig +short CH TXT id.server @9.9.9.9
# "res100.sin.rrdns.pch.net"
#   From another continent, the SAME query to the SAME address returns a
#   different server name. That is anycast.

# Distinguish anycast from DNS steering -- check the TTL and the address:
$ for r in 1.1.1.1 8.8.8.8 9.9.9.9; do
>   echo -n "$r -> "; dig +short @$r cdn.example.com | head -1
> done
# 1.1.1.1 -> 203.0.113.9
# 8.8.8.8 -> 203.0.113.9
# 9.9.9.9 -> 203.0.113.9
#   Identical answers everywhere = anycast is doing the steering.

# Which PoP actually served the request:
$ curl -sI https://cdn.example.com/ | grep -iE 'x-served-by|cf-ray|x-amz-cf-pop'
# x-served-by: cache-syd1234-SYD

# Confirm you reached a nearby PoP -- the hop count tells you:
$ traceroute -n -m 12 203.0.113.9 | tail -3
#  6  103.x.x.x   4.1 ms
#  7  203.0.113.9 4.8 ms
#   7 hops and 4.8 ms: local. 18 hops and 240 ms would mean misrouting.

# See the BGP announcements behind an anycast prefix:
$ whois -h whois.radb.net '!g203.0.113.0/24' 2>/dev/null | head -3
# (or use a looking glass: the same prefix originated from one ASN,
#  visible from many geographically separate vantage points)

Key takeaways

DNS-based selection returns a different address per requester, giving fine control over capacity weighting, rollouts, and policy.
Its weaknesses are that the authoritative server sees the resolver's location, not the user's (partially fixed by EDNS Client Subnet, at a privacy cost) and that failover is bounded by TTL.
Anycast announces one address from many locations and lets BGP steer, giving automatic second-scale failover and excellent DDoS absorption by splitting attacks across sites.
Anycast's limits: "closest in BGP" is not closest in latency, steering control is coarse, and a route change can reset a long-lived TCP connection.
Anycast suits DNS and short exchanges; DNS-based steering suits long transfers; large CDNs use both together.
QUIC's connection IDs make anycast reroutes far less damaging than they are for TCP.

🧪 Practice

Explain why a Sydney user might be sent to a Singapore PoP under DNS-based selection, and how ECS addresses it.

Compare failover time for a DNS answer with a 300-second TTL against withdrawing a BGP announcement.

Describe what happens to a 2 GB download over anycast TCP when routing changes mid-transfer, and why QUIC is more resilient.

Interview-style: Root DNS servers use anycast; large video CDNs mostly use DNS-based steering for the media itself. Explain the difference in requirements. (Hint: compare how long a single exchange lasts in each case, and what a mid-flight reroute costs.)

Adaptive Bitrate Streaming (DASH, HLS)

Theory

Streaming video over the internet has a problem that file download does not: the network's capacity varies during playback, and there is no way to know in advance what it will be. Pick a bitrate that is too high and the video stalls; too low and the picture is needlessly poor. Early streaming protocols (RTMP and similar) used a dedicated stateful connection and negotiated a fixed rate, which failed both ways and needed special servers and open ports.

Adaptive bitrate streaming solves it with two ideas that, combined, are almost embarrassingly simple:

1. Split the video into short segments. Typically 2–10 seconds each, every one independently decodable because it begins with a keyframe.

2. Encode the whole video several times at different bitrates, with segment boundaries aligned across all versions. The player fetches segment 1 at 5 Mb/s, measures how quickly it arrived, and if throughput dropped it fetches segment 2 at 2 Mb/s. Because the boundaries align, the switch is seamless.

The consequence that made this the universal approach is that it is ordinary HTTP. Segments are just files. That means every CDN caches them with no modification, every firewall passes them, TLS works, range requests work, and no special server is needed. Streaming stopped being a separate infrastructure problem and became a caching problem — which the previous topic had already solved.

The two standards are structurally identical and differ mainly in lineage:

	HLS (Apple)	DASH (MPEG/ISO)
Manifest	`.m3u8` (text playlist)	`.mpd` (XML)
Segments	`.ts` or fragmented MP4	Fragmented MP4
Codec constraints	Historically stricter	Codec-agnostic
Native browser support	Safari; elsewhere via JS	Via JS (Media Source Extensions)
DRM	FairPlay	Common Encryption (multi-DRM)

CMAF largely ends the duplication: a single set of fragmented-MP4 segments can be referenced by both an HLS and a DASH manifest, so a provider stores one copy instead of two.

The player's adaptation logic is where quality is won or lost, and the naive approach fails. A purely throughput-based algorithm reacts to measured download speed and oscillates badly, because every quality change alters the measurement. A buffer-based algorithm uses buffer occupancy as the control signal — a full buffer means it is safe to go up, a draining buffer means go down now — and is far more stable. Production players (BOLA, MPC, and similar) combine both.

The two costs to remember:

Latency. The player must buffer several segments before starting, so live streams over ABR run 15–45 seconds behind reality — a genuine problem when the neighbours cheer a goal before you see it. Low-Latency HLS/DASH use partial segments and chunked transfer to reach 2–5 seconds.

Storage and encoding. A video encoded at 8 renditions costs roughly 8 times the storage and 8 times the encoding CPU. Per-title encoding — choosing a rendition ladder based on the content's complexity, since an animation needs far fewer bits than a sports match — is the standard optimisation.

Examples

The rendition ladder and aligned segments

  bitrate
    ^
  8M|  [1080p] [1080p] [1080p] [1080p] [1080p]
  5M|  [ 720p] [ 720p] [ 720p] [ 720p] [ 720p]
  2M|  [ 480p] [ 480p] [ 480p] [ 480p] [ 480p]
  0.5M| [ 240p] [ 240p] [ 240p] [ 240p] [ 240p]
    +-------------------------------------------> time
       seg 1    seg 2    seg 3    seg 4    seg 5
       (4 s)    (4 s)    (4 s)    (4 s)    (4 s)

  Segment boundaries ALIGN across renditions, and each begins with a
  keyframe -- so the player can switch rows between any two segments.

  An actual playback session:
       seg1: 8M   (good network)
       seg2: 8M
       seg3: 2M   <- throughput dropped; step down rather than stall
       seg4: 5M   <- recovering
       seg5: 8M


 IT IS JUST HTTP -- which is why it won

    GET /video/1080p/seg_00003.m4s   -> an ordinary cacheable file
    GET /video/manifest.mpd          -> an ordinary cacheable file

    every CDN, every proxy, every firewall already handles this.
    No RTMP server, no special ports, no new infrastructure.


 WHY LIVE ABR IS 15-45 SECONDS BEHIND

   encode (1 seg) -> package -> CDN -> player buffers 3 segments -> play
      4 s            1 s       1 s          12 s                  = 18 s
   Low-Latency HLS/DASH: partial segments + chunked transfer -> 2-5 s

# An HLS master playlist and a DASH manifest describe the same thing.

def hls_master(renditions):
    out = ["#EXTM3U", "#EXT-X-VERSION:7"]
    for r in renditions:
        out.append(f'#EXT-X-STREAM-INF:BANDWIDTH={r["bps"]},'
                   f'RESOLUTION={r["res"]},CODECS="avc1.640028,mp4a.40.2"')
        out.append(r["uri"])
    return "\n".join(out)

renditions = [
    {"bps": 8_000_000, "res": "1920x1080", "uri": "1080p/index.m3u8"},
    {"bps": 5_000_000, "res": "1280x720",  "uri": "720p/index.m3u8"},
    {"bps": 2_000_000, "res": "854x480",   "uri": "480p/index.m3u8"},
    {"bps":   500_000, "res": "426x240",   "uri": "240p/index.m3u8"},
]
print(hls_master(renditions))
# #EXTM3U
# #EXT-X-VERSION:7
# #EXT-X-STREAM-INF:BANDWIDTH=8000000,RESOLUTION=1920x1080,CODECS="avc1.640028,mp4a.40.2"
# 1080p/index.m3u8
# ...

# A media playlist: the actual segment list.
def hls_media(segments, duration=4):
    out = ["#EXTM3U", "#EXT-X-VERSION:7", f"#EXT-X-TARGETDURATION:{duration}",
           "#EXT-X-MEDIA-SEQUENCE:0"]
    for i in range(segments):
        out += [f"#EXTINF:{duration}.0,", f"seg_{i:05d}.m4s"]
    out.append("#EXT-X-ENDLIST")          # absent on a LIVE playlist
    return "\n".join(out)

print("\n" + hls_media(3))
# #EXTM3U
# #EXT-X-TARGETDURATION:4
# #EXTINF:4.0,
# seg_00000.m4s
# ... (#EXT-X-ENDLIST present = VOD; absent = live, keep re-fetching)

# Two adaptation algorithms, compared on the same network trace.
def throughput_based(bandwidth_trace, ladder, safety=0.9):
    chosen, buffer_s = [], 0
    for bw in bandwidth_trace:
        best = max([b for b in ladder if b <= bw * safety], default=min(ladder))
        chosen.append(best)
    return chosen

def buffer_based(bandwidth_trace, ladder, seg_s=4, target=20):
    chosen, buffer_s = [], 0.0
    for bw in bandwidth_trace:
        # Pick based on how much buffer we have, not on raw throughput.
        if buffer_s < seg_s * 2:      level = min(ladder)          # emergency
        elif buffer_s < target * 0.5: level = sorted(ladder)[1]
        elif buffer_s < target:       level = sorted(ladder)[-2]
        else:                          level = max(ladder)
        download_s = (level * seg_s) / bw                 # time to fetch it
        buffer_s = max(0.0, buffer_s + seg_s - download_s)
        chosen.append(level)
    return chosen

ladder = [500_000, 2_000_000, 5_000_000, 8_000_000]
trace  = [9e6, 9e6, 1.5e6, 1.4e6, 9e6, 9e6, 2e6, 9e6, 9e6, 9e6]

tb = throughput_based(trace, ladder)
bb = buffer_based(trace, ladder)
print(f"\n{'seg':>4}{'bandwidth':>12}{'throughput-based':>19}{'buffer-based':>15}")
for i, bw in enumerate(trace):
    print(f"{i:>4}{bw/1e6:>10.1f}M{tb[i]/1e6:>17.1f}M{bb[i]/1e6:>14.1f}M")
switches_tb = sum(1 for i in range(1, len(tb)) if tb[i] != tb[i-1])
switches_bb = sum(1 for i in range(1, len(bb)) if bb[i] != bb[i-1])
print(f"\nquality switches: throughput-based={switches_tb}, buffer-based={switches_bb}")
# quality switches: throughput-based=5, buffer-based=3
#   Fewer switches means a visibly more stable picture; viewers notice
#   oscillation more than they notice a slightly lower average bitrate.

# The latency budget for live ABR.
def live_latency(segment_s, buffer_segments, encode_s=1, package_s=1, cdn_s=1):
    return segment_s + encode_s + package_s + cdn_s + segment_s * buffer_segments

print()
for seg, buf in ((6, 3), (4, 3), (2, 2), (1, 2)):
    print(f"segment={seg}s buffer={buf} segs -> ~{live_latency(seg, buf):>4.0f} s behind live")
# segment=6s buffer=3 segs -> ~  27 s behind live
# segment=4s buffer=3 segs -> ~  19 s behind live
# segment=2s buffer=2 segs -> ~   9 s behind live
# segment=1s buffer=2 segs -> ~   6 s behind live
#   Shorter segments cut latency but increase request count and overhead.
#   LL-HLS/LL-DASH go further with PARTIAL segments delivered chunked.

# Storage cost of the ladder.
def ladder_cost(duration_min, ladder_bps):
    total = sum(b * duration_min * 60 / 8 for b in ladder_bps)
    single = max(ladder_bps) * duration_min * 60 / 8
    return single / 1e9, total / 1e9

single, total = ladder_cost(120, ladder)
print(f"\n2-hour film: {single:.1f} GB at top rendition only, "
      f"{total:.1f} GB for the full ladder ({total/single:.1f}x)")
# 2-hour film: 7.2 GB at top rendition only, 13.9 GB for the full ladder (1.9x)
#   Plus the encoding CPU for every rendition -- which per-title encoding
#   reduces by tailoring the ladder to the content's complexity.

# Fetch and read a master playlist -- it is plain text:
$ curl -s https://stream.example.com/master.m3u8
# #EXTM3U
# #EXT-X-STREAM-INF:BANDWIDTH=8000000,RESOLUTION=1920x1080
# 1080p/index.m3u8
# #EXT-X-STREAM-INF:BANDWIDTH=2000000,RESOLUTION=854x480
# 480p/index.m3u8

# The media playlist for one rendition:
$ curl -s https://stream.example.com/480p/index.m3u8 | head -8
# #EXT-X-TARGETDURATION:4
# #EXT-X-MEDIA-SEQUENCE:1204
#                       ^^^^ increments on a LIVE stream; the player re-fetches

# Live or VOD? Check for the endlist tag:
$ curl -s https://stream.example.com/480p/index.m3u8 | grep -c ENDLIST
# 0        <- live: keep polling the playlist for new segments

# Segments are ordinary cacheable files -- this is the whole trick:
$ curl -sI https://stream.example.com/480p/seg_01204.m4s | \
    grep -iE 'content-type|content-length|cache-control|x-cache'
# content-type: video/iso.segment
# content-length: 1048576
# cache-control: public, max-age=31536000
# x-cache: HIT
#   Immutable, long TTL, CDN-cached. No streaming server involved.

# Inspect a DASH manifest:
$ curl -s https://stream.example.com/manifest.mpd | \
    xmllint --format - | grep -E 'Representation|bandwidth' | head -4
# <Representation id="1" bandwidth="8000000" width="1920" height="1080"/>
# <Representation id="2" bandwidth="5000000" width="1280" height="720"/>

# Verify segments are keyframe-aligned across renditions (they must be):
$ ffprobe -v error -show_frames -select_streams v \
    -show_entries frame=key_frame,pkt_pts_time 480p/seg_00001.m4s | head -4
# [FRAME] key_frame=1 pkt_pts_time=4.000000
#          ^ every segment must START with a keyframe, or switching breaks

# Build a rendition ladder:
$ ffmpeg -i source.mp4 \
    -filter_complex "[0:v]split=3[v1][v2][v3]; \
      [v1]scale=1920:1080[1080p];[v2]scale=1280:720[720p];[v3]scale=854:480[480p]" \
    -map '[1080p]' -b:v:0 8M -map '[720p]' -b:v:1 5M -map '[480p]' -b:v:2 2M \
    -g 96 -keyint_min 96 -sc_threshold 0 \
    -f hls -hls_time 4 -hls_playlist_type vod \
    -master_pl_name master.m3u8 -var_stream_map "v:0 v:1 v:2" 'stream_%v/index.m3u8'
#   -g 96 with 24 fps = a keyframe every 4 s, matching -hls_time 4.
#   -sc_threshold 0 disables scene-change keyframes, which would misalign segments.

# Watch a player adapt in real time:
$ ffplay -v info https://stream.example.com/master.m3u8 2>&1 | grep -i 'variant'

Key takeaways

ABR splits video into short, independently decodable segments encoded at several bitrates with aligned boundaries, so the player can switch between segments seamlessly.
Because segments are ordinary HTTP files, every CDN, proxy, and firewall handles them unchanged — streaming became a caching problem rather than a separate infrastructure.
HLS (.m3u8) and DASH (.mpd) are structurally equivalent; CMAF lets one set of segments serve both.
Throughput-based adaptation oscillates; buffer-based adaptation is far more stable, and production players combine the two.
Viewers notice quality oscillation more than a slightly lower average bitrate, so stability is the right optimisation target.
Live ABR runs 15–45 seconds behind; Low-Latency HLS/DASH reach 2–5 seconds using partial segments and chunked transfer.
A full rendition ladder multiplies storage and encoding cost, which per-title encoding reduces by matching the ladder to content complexity.

🧪 Practice

Explain why segments must begin with a keyframe and what breaks if a scene-change keyframe misaligns them across renditions.

Compute live latency for 2-second segments with a 3-segment buffer, and state what Low-Latency HLS changes.

Given bandwidth samples of 9, 9, 1.5, 1.4, 9 Mb/s and a ladder of 0.5/2/5/8 Mb/s, choose renditions under a throughput-based and a buffer-based policy.

Interview-style: Why did HTTP-based streaming displace purpose-built streaming protocols such as RTMP? (Hint: list what a segment is from the network's point of view, and what infrastructure already existed to handle exactly that.)

VoIP and RTP/RTCP

Theory

Voice and video calls invert every assumption behind TCP. Section 8.1's framework makes the reason precise: a call has a hard deadline — roughly 150 ms one way — and a packet that arrives after its playout moment is worthless no matter how correct it is. Retransmission costs at least one RTT, which typically exceeds the entire remaining budget. So real-time media discards loss recovery in exchange for timeliness, which means UDP.

But raw UDP is not enough. UDP provides no sequence numbers, no timing information, and no way to tell what the payload is. RTP (Real-time Transport Protocol, RFC 3550) adds exactly what media needs and nothing more:

Sequence number (16 bits) — detect loss and reordering, without requesting retransmission.
Timestamp (32 bits) — the sampling instant, in media clock units (8 kHz for narrowband audio, 90 kHz for video). This drives playout timing and is not a wall-clock time.
SSRC (32 bits) — a synchronisation source identifier, so several streams can share a session.
Payload type (7 bits) — which codec this is.
Marker bit — significant events, such as the last packet of a video frame or the start of a talkspurt.

RTP travels with a companion protocol, RTCP, on the adjacent port. RTCP carries no media — it carries quality reports: receiver reports with loss fraction, cumulative loss, highest sequence number, interarrival jitter, and round-trip estimates; sender reports mapping the RTP timestamp to a wall clock so audio and video can be lip-synchronised. RTCP traffic is kept to about 5% of session bandwidth, so it scales to large conferences.

The mechanism that makes a jittery network sound smooth is the jitter buffer. Packets are held briefly and released at a constant rate, converting variable arrival into steady playout. The buffer's size is the central trade-off: deeper absorbs more jitter but adds directly to end-to-end latency, so modern implementations size it adaptively from measured jitter. Packets arriving after their playout deadline are discarded — they count as loss even though the network delivered them, which is why "late" and "lost" are the same thing here.

Losses are concealed rather than repaired. Packet loss concealment synthesises a plausible replacement from surrounding audio; FEC sends redundant data proactively so some losses need no recovery at all; redundant encoding (RED) piggybacks a low-bitrate copy of the previous packet on the next one. All three trade bandwidth or quality for the thing retransmission cannot provide: an answer that arrives in time.

SRTP (RFC 3711) adds encryption and authentication with low overhead, and is mandatory in WebRTC.

Examples

The RTP header: exactly what media needs

   0                   1                   2                   3
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |V=2|P|X|  CC   |M|     PT      |       sequence number         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                           timestamp                           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           synchronization source (SSRC) identifier            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     ^ 12 bytes. Note what is ABSENT: no acknowledgements, no windows,
       no retransmission. Detecting loss is useful; fixing it is not.


 THE JITTER BUFFER: variable arrival -> constant playout

   network arrivals (ms):  0   18   45   52   80   85  120
                           |    |    |    |    |    |    |
                           v    v    v    v    v    v    v
                        +--------------------------------------+
                        |          JITTER BUFFER (60 ms)       |
                        +--------------------------------------+
                           |    |    |    |    |    |    |
   playout (every 20ms):  60   80  100  120  140  160  180
                           ^ smooth, regardless of arrival pattern

   TRADE-OFF:
     buffer too SMALL -> late packets discarded  -> audible dropouts
     buffer too LARGE -> added latency           -> people talk over each other

   A packet arriving at 190 ms for a 180 ms slot is DISCARDED.
   The network delivered it. It is still a loss.


 WHY NOT JUST RETRANSMIT?

   RTT = 60 ms, remaining budget after network+encode = 40 ms
   retransmit costs 60 ms > 40 ms  ->  the replacement arrives too late
   CONCEAL instead: synthesise something plausible, on time.

# RTP packet construction and receiver-side loss/jitter measurement.
import struct

def rtp_header(seq, timestamp, ssrc, payload_type=0, marker=0):
    b0 = (2 << 6)                                  # version 2, no padding/ext/CC
    b1 = (marker << 7) | payload_type              # M bit + PT
    return struct.pack("!BBHII", b0, b1, seq, timestamp, ssrc)

# G.711 narrowband: 8 kHz clock, 20 ms packets -> 160 timestamp units each.
packets = [rtp_header(1000 + i, 160 * i, 0xDEADBEEF) for i in range(5)]
for i, p in enumerate(packets[:3]):
    v, m_pt, seq, ts, ssrc = struct.unpack("!BBHII", p)
    print(f"seq={seq} ts={ts:>4} ssrc={ssrc:#x}  ({len(p)} byte header)")
# seq=1000 ts=   0 ssrc=0xdeadbeef  (12 byte header)
# seq=1001 ts= 160 ssrc=0xdeadbeef  (12 byte header)
# seq=1002 ts= 320 ssrc=0xdeadbeef  (12 byte header)
#   The timestamp is in MEDIA CLOCK units, not milliseconds.

# RFC 3550 interarrival jitter -- the figure RTCP reports.
def interarrival_jitter(arrivals_ms, rtp_timestamps, clock_hz=8000):
    J = 0.0
    for i in range(1, len(arrivals_ms)):
        rtp_delta = (rtp_timestamps[i] - rtp_timestamps[i-1]) * 1000 / clock_hz
        arr_delta = arrivals_ms[i] - arrivals_ms[i-1]
        D = abs(arr_delta - rtp_delta)             # deviation from expected spacing
        J += (D - J) / 16                          # the RFC's smoothing filter
    return J

smooth  = [0, 20, 40, 60, 80, 100]
jittery = [0, 12, 48, 55, 96, 101]
ts = [160 * i for i in range(6)]
print(f"\nsmooth  arrivals -> jitter {interarrival_jitter(smooth, ts):>6.2f} ms")
print(f"jittery arrivals -> jitter {interarrival_jitter(jittery, ts):>6.2f} ms")
# smooth  arrivals -> jitter   0.00 ms
# jittery arrivals -> jitter   3.23 ms

# The jitter buffer trade-off, measured.
def buffer_performance(arrivals_ms, buffer_ms, frame_ms=20):
    first_playout = arrivals_ms[0] + buffer_ms
    late = sum(1 for i, a in enumerate(arrivals_ms)
               if a > first_playout + i * frame_ms)
    return late, buffer_ms

import random
random.seed(11)
arrivals = []
t = 0
for i in range(200):
    t += 20 + random.choice([-12, -4, 0, 3, 8, 45])     # occasional big spike
    arrivals.append(t)

print(f"\n{'buffer':>8}{'late/discarded':>17}{'added latency':>16}")
for buf in (0, 20, 60, 120, 200):
    late, _ = buffer_performance(arrivals, buf)
    print(f"{buf:>6} ms{late:>14} pkts{buf:>13} ms")
#   buffer   late/discarded   added latency
#      0 ms           137 pkts            0 ms
#     20 ms           118 pkts           20 ms
#     60 ms            79 pkts           60 ms
#    120 ms            34 pkts          120 ms
#    200 ms             3 pkts          200 ms
#   Every millisecond of buffer buys fewer dropouts and costs conversational
#   quality. Adaptive sizing tracks measured jitter instead of guessing.

# Is retransmission ever worth it? Compare against the remaining budget.
def recovery_choice(rtt_ms, budget_ms, elapsed_ms):
    remaining = budget_ms - elapsed_ms
    return ("RETRANSMIT (it will arrive in time)" if rtt_ms < remaining
            else f"CONCEAL (retransmit needs {rtt_ms} ms, only {remaining} ms left)")

for rtt, elapsed in ((20, 40), (60, 40), (60, 120)):
    print(f"\nrtt={rtt:>3}ms elapsed={elapsed:>3}ms of 150ms -> {recovery_choice(rtt,150,elapsed)}")
# rtt= 20ms elapsed= 40ms of 150ms -> RETRANSMIT (it will arrive in time)
# rtt= 60ms elapsed= 40ms of 150ms -> RETRANSMIT (it will arrive in time)
# rtt= 60ms elapsed=120ms of 150ms -> CONCEAL (retransmit needs 60 ms, only 30 ms left)

# FEC: pay bandwidth up front so some losses need no recovery at all.
def fec_overhead(protected_packets, parity_packets):
    return parity_packets / protected_packets, parity_packets   # recoverable losses

for k, r in ((10, 1), (5, 1), (4, 2)):
    ov, rec = fec_overhead(k, r)
    print(f"\n{k}:{r} FEC -> {ov:>5.0%} bandwidth overhead, "
          f"recovers up to {rec} loss(es) per block with NO round trip")
# 10:1 FEC ->   10% bandwidth overhead, recovers up to 1 loss(es) per block with NO round trip
#  5:1 FEC ->   20% bandwidth overhead, recovers up to 1 loss(es) per block ...
#  4:2 FEC ->   50% bandwidth overhead, recovers up to 2 loss(es) per block ...

# Capture RTP and let Wireshark decode it:
$ sudo tshark -i any -f 'udp portrange 16384-32768' -d udp.port==16400,rtp \
    -T fields -e rtp.seq -e rtp.timestamp -e rtp.ssrc -c 5
# 1000  0     0xdeadbeef
# 1001  160   0xdeadbeef
# 1003  480   0xdeadbeef     <- 1002 is missing: loss, visible in the sequence

# Wireshark's RTP stream analysis is the fastest quality diagnosis available:
$ tshark -r call.pcap -q -z rtp,streams
# Src IP  Port  SSRC       Payload  Pkts  Lost      Max Delta  Max Jitter  Mean Jitter
# 10.0.0.5 16400 0xdeadbeef g711U   3021  18 (0.6%)   118.2ms     41.20ms      8.14ms
#                                        ^^^^^^^^^^   ^^^^^^^^^   ^^^^^^^^^^
#                                        tolerable    a 118 ms gap = audible

# RTCP receiver reports carry the quality figures the sender reacts to:
$ tshark -r call.pcap -Y rtcp -T fields \
    -e rtcp.ssrc.fraction -e rtcp.ssrc.cumulative -e rtcp.ssrc.jitter | head -3
# 3   18   651
#  ^ fraction lost, out of 256 (3/256 = 1.2%)
#       ^ cumulative packets lost
#            ^ jitter in RTP timestamp units (651/8000 = 81 ms)

# Check the ports: RTP on even, RTCP on the next odd port by convention:
$ ss -uan | grep -E ':1640[0-9]'
# UNCONN 0 0 10.0.0.5:16400    <- RTP
# UNCONN 0 0 10.0.0.5:16401    <- RTCP

# Measure the path's suitability BEFORE blaming the application:
$ ping -c 200 -i 0.02 -q peer.example.com | tail -2
# 200 packets transmitted, 198 received, 1% packet loss
# rtt min/avg/max/mdev = 18.1/24.7/186.3/14.2 ms
#                                  ^^^^^ a 186 ms spike; mdev 14.2 ms is the jitter
#   1% loss is fine for voice. 14 ms jitter with 186 ms spikes is not.

# Confirm media is encrypted (SRTP), as WebRTC requires:
$ tshark -r call.pcap -Y 'rtp' -T fields -e rtp.payload -c 1 | head -c 40
# 9f2a8c1d4b7e3a5f0c2e6d8b1a4f7c93...   <- ciphertext, not decodable audio

Key takeaways

Real-time media uses UDP because retransmission cannot beat the deadline — a late packet is worthless regardless of correctness.
RTP adds sequence numbers, media timestamps, SSRC, and payload type to UDP: enough to detect loss and schedule playout, deliberately without recovery.
RTP timestamps are in media clock units (8 kHz audio, 90 kHz video), not milliseconds.
RTCP carries no media — it reports loss fraction, cumulative loss, and interarrival jitter, and maps RTP time to wall clock for lip sync, capped near 5% of session bandwidth.
The jitter buffer converts variable arrival into constant playout; deeper absorbs more jitter but adds latency, so it is sized adaptively.
A packet that arrives after its playout deadline is discarded — late and lost are the same thing.
Losses are handled by concealment, FEC, and redundant encoding, all of which trade bandwidth or quality for timeliness; SRTP provides encryption.

🧪 Practice

Explain why RTP includes a sequence number when it never requests retransmission.

Given 20 ms packets at an 8 kHz clock, state the RTP timestamp increment per packet and compute the timestamp of the tenth packet.

A call has 60 ms RTT and 110 ms already consumed of a 150 ms budget. Decide between retransmission and concealment, showing the arithmetic.

Interview-style: A user reports choppy audio; monitoring shows 0.2% packet loss and 45 ms jitter. (Hint: ask which of those two numbers the jitter buffer is failing to absorb, and what raising it would cost the conversation.)

SIP Signaling

Theory

RTP carries media, but it cannot start a call. Something must find the person, ring them, negotiate which codecs both sides support, agree where to send the packets, and later hang up. That is signalling, and SIP (Session Initiation Protocol, RFC 3261) is the standard for it.

The essential separation, which mirrors the control-plane/data-plane split from 8.1: SIP establishes sessions; RTP carries the media; the two are independent. SIP messages typically pass through proxies, while media usually flows directly between endpoints. Killing the SIP path does not stop an established call's audio.

SIP is deliberately modelled on HTTP: text-based, request-response, with methods and three-digit status codes. If you can read an HTTP exchange you can read a SIP one.

SIP method	Purpose
INVITE	Start a session
ACK	Confirm a final response to an INVITE
BYE	End a session
CANCEL	Abandon a pending INVITE
REGISTER	Bind an address-of-record to a current location
OPTIONS	Query capabilities
REFER	Transfer a call
SUBSCRIBE/NOTIFY	Event notification (presence, message-waiting)

Status codes reuse HTTP's classes with one addition: 1xx provisional responses (100 Trying, 180 Ringing, 183 Session Progress) are essential here because setup takes human time. 200 OK, 404 Not Found, 486 Busy Here, and 487 Request Terminated behave as you would expect.

Addressing uses a SIP URI — sip:alice@example.com — which is an address-of-record, a stable identity rather than a location. REGISTER binds it to wherever the user currently is (sip:alice@192.168.1.20:5060), which is exactly the DNS-like indirection that makes mobility work.

The INVITE/200/ACK three-way handshake is unusual and worth noting: unlike every other SIP transaction, INVITE uses three messages rather than two, because the time between ringing and answering is unbounded and the ACK confirms the final answer separately.

Media parameters are not carried by SIP itself but by SDP (Session Description Protocol) in the message bodies, using an offer/answer model (RFC 3264): the caller offers its supported codecs and receive addresses, the callee answers with the intersection it accepts.

SIP's chronic operational problem is NAT, and it is instructive. SDP carries IP addresses inside the message body, so an endpoint behind NAT advertises its private address and media goes nowhere. The workarounds are a catalogue of NAT traversal: rport and Via received parameters, STUN, symmetric RTP, session border controllers, and media relays. It is the same problem FTP had in 8.4, from the same cause — addresses embedded in a payload that NAT does not rewrite.

Examples

Signalling and media take different paths

  [alice]                 [proxy]                 [bob]
     |----INVITE (SDP offer)-->|                    |
     |<---100 Trying-----------|----INVITE--------->|
     |                         |<---180 Ringing-----|
     |<---180 Ringing----------|                    |   (bob's phone rings)
     |                         |<---200 OK (answer)-|   (bob picks up)
     |<---200 OK---------------|                    |
     |----ACK----------------->|----ACK------------>|
     |                         |                    |
     |<=========== RTP MEDIA (direct) ============>|   <- NOT via the proxy
     |                         |                    |
     |----BYE----------------->|----BYE------------>|
     |<---200 OK---------------|<---200 OK----------|

  SIGNALLING: small, via proxies, authoritative
  MEDIA:      large, direct, peer-to-peer
  Kill the proxy mid-call and the audio KEEPS FLOWING.


 INVITE IS A THREE-WAY HANDSHAKE (unlike every other SIP method)

   INVITE ---->        because the gap between 180 Ringing and 200 OK
   200 OK <----        is human-scale and unbounded, the final response
   ACK    ---->        needs its own confirmation


 ADDRESS-OF-RECORD vs CONTACT (the DNS-like indirection)

   sip:alice@example.com          <- stable identity, printed on a card
            |  REGISTER binds it to:
            v
   sip:alice@192.168.1.20:5060    <- current location, changes constantly
   sip:alice@10.4.2.9:5060        <- another device, also registered

   One identity, several contacts. Forking rings all of them.


 THE NAT PROBLEM (identical in cause to FTP's)

   SDP body says: c=IN IP4 192.168.1.20      <- a PRIVATE address
                             ^^^^^^^^^^^^ NAT rewrote the IP HEADER
                                          but not the message BODY
   -> the far end sends RTP into a private address space: one-way audio

# A SIP INVITE with an SDP offer, and the answer it produces.

def sip_invite(from_uri, to_uri, call_id, local_ip, local_port, codecs):
    sdp = "\r\n".join([
        "v=0",
        f"o=alice 2890844526 2890844526 IN IP4 {local_ip}",
        "s=Call",
        f"c=IN IP4 {local_ip}",                      # where to SEND media
        "t=0 0",
        f"m=audio {local_port} RTP/AVP " + " ".join(str(c['pt']) for c in codecs),
        *[f"a=rtpmap:{c['pt']} {c['name']}/{c['rate']}" for c in codecs],
        "a=sendrecv",
    ])
    headers = "\r\n".join([
        f"INVITE {to_uri} SIP/2.0",
        f"Via: SIP/2.0/UDP {local_ip}:5060;branch=z9hG4bK776asdhds;rport",
        f"From: <{from_uri}>;tag=1928301774",
        f"To: <{to_uri}>",
        f"Call-ID: {call_id}",
        "CSeq: 314159 INVITE",
        f"Contact: <sip:alice@{local_ip}:5060>",
        "Content-Type: application/sdp",
        f"Content-Length: {len(sdp)}",
    ])
    return headers + "\r\n\r\n" + sdp

offer_codecs = [{"pt": 111, "name": "opus", "rate": 48000},
                {"pt": 0,   "name": "PCMU", "rate": 8000},
                {"pt": 8,   "name": "PCMA", "rate": 8000}]
print(sip_invite("sip:alice@example.com", "sip:bob@example.com",
                 "a84b4c76e66710", "192.168.1.20", 16400, offer_codecs))
# INVITE sip:bob@example.com SIP/2.0
# Via: SIP/2.0/UDP 192.168.1.20:5060;branch=z9hG4bK776asdhds;rport
# ...
# c=IN IP4 192.168.1.20        <- PRIVATE address in the BODY: the NAT problem
# m=audio 16400 RTP/AVP 111 0 8

# Offer/answer: the answer is the INTERSECTION, in the answerer's preference.
def sdp_answer(offered, supported):
    common = [c for c in offered if c["name"] in {s["name"] for s in supported}]
    return common[:1]                                  # pick one: the best match

bob_supports = [{"pt": 0, "name": "PCMU", "rate": 8000},
                {"pt": 8, "name": "PCMA", "rate": 8000}]
chosen = sdp_answer(offer_codecs, bob_supports)
print(f"\nalice offered : {[c['name'] for c in offer_codecs]}")
print(f"bob supports  : {[c['name'] for c in bob_supports]}")
print(f"negotiated    : {[c['name'] for c in chosen]}")
# alice offered : ['opus', 'PCMU', 'PCMA']
# bob supports  : ['PCMU', 'PCMA']
# negotiated    : ['PCMU']
#   No opus: bob cannot decode it. The intersection is what actually runs.

# Registration: binding a stable identity to a current location.
registrar = {}
def register(aor, contact, expires=3600):
    registrar.setdefault(aor, []).append({"contact": contact, "expires": expires})
    return f"200 OK (expires={expires})"

print()
print(register("sip:alice@example.com", "sip:alice@192.168.1.20:5060"))
print(register("sip:alice@example.com", "sip:alice@10.4.2.9:5060"))
print(f"bindings for alice: {[b['contact'] for b in registrar['sip:alice@example.com']]}")
# 200 OK (expires=3600)
# 200 OK (expires=3600)
# bindings for alice: ['sip:alice@192.168.1.20:5060', 'sip:alice@10.4.2.9:5060']
#   An INVITE now FORKS to both: desk phone and mobile ring together.

# Call state machine: what each response means for the caller.
def call_state(responses):
    state = "calling"
    for code in responses:
        if 100 <= code < 200:   state = {100: "trying", 180: "ringing",
                                         183: "early media"}.get(code, "provisional")
        elif code == 200:        state = "answered -> send ACK, media starts"
        elif code == 486:        state = "busy"
        elif code == 408:        state = "timeout, no answer"
        elif code == 404:        state = "user not found"
        elif 400 <= code:        state = f"failed ({code})"
    return state

for seq in ([100, 180, 200], [100, 180, 486], [100, 404], [100, 180, 408]):
    print(f"\nresponses {str(seq):<18} -> {call_state(seq)}")
# responses [100, 180, 200]    -> answered -> send ACK, media starts
# responses [100, 180, 486]    -> busy
# responses [100, 404]         -> user not found
# responses [100, 180, 408]    -> timeout, no answer

# The NAT symptom, made explicit.
def media_outcome(sdp_ip, behind_nat, fix=None):
    private = sdp_ip.startswith(("10.", "192.168.", "172.16."))
    if behind_nat and private and not fix:
        return "ONE-WAY AUDIO: far end sends RTP into a private address"
    if fix:
        return f"works ({fix} rewrote or discovered the public address)"
    return "works"

print()
for fix in (None, "STUN", "symmetric RTP", "SBC media relay"):
    print(f"  fix={str(fix):<16} -> {media_outcome('192.168.1.20', True, fix)}")
#   fix=None             -> ONE-WAY AUDIO: far end sends RTP into a private address
#   fix=STUN             -> works (STUN rewrote or discovered the public address)
#   fix=symmetric RTP    -> works (symmetric RTP ...)
#   fix=SBC media relay  -> works (SBC media relay ...)

# Watch a call set up -- SIP is text, so the trace is readable:
$ sudo tshark -i any -f 'port 5060' -Y sip -T fields \
    -e sip.Method -e sip.Status-Code -e sip.Call-ID
# INVITE      a84b4c76e66710
#        100  a84b4c76e66710
#        180  a84b4c76e66710
#        200  a84b4c76e66710
# ACK         a84b4c76e66710
# BYE         a84b4c76e66710
#        200  a84b4c76e66710

# Extract the SDP and check which address is advertised -- the NAT check:
$ tshark -r call.pcap -Y 'sip.CSeq.method == "INVITE"' -T fields -e sdp.connection_info
# IN IP4 192.168.1.20
#          ^^^^^^^^^^^^ private: expect one-way audio unless something fixes it

# Which codec was actually negotiated:
$ tshark -r call.pcap -Y sdp -T fields -e sdp.media.format | tail -1
# ITU-T G.711 PCMU

# Probe a SIP endpoint's capabilities without placing a call:
$ sipsak -vv -s sip:bob@example.com -M 2>&1 | grep -E 'SIP/2.0|Allow'
# SIP/2.0 200 OK
# Allow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, NOTIFY, SUBSCRIBE

# Check current registrations on the registrar (Asterisk):
$ asterisk -rx 'pjsip show contacts'
# Contact: alice/sip:alice@192.168.1.20:5060  Avail  21.043
# Contact: alice/sip:alice@10.4.2.9:5060      Avail  18.221
#   Two contacts for one AOR: an INVITE forks to both.

# Diagnose one-way audio -- signalling succeeded but media did not:
$ tshark -r call.pcap -q -z rtp,streams
# Src IP        Port   Pkts   Lost
# 10.0.0.5      16400  3021   0        <- alice -> bob: flowing
# (no reverse stream)                  <- bob -> alice: NOTHING
#   SIP said 200 OK. Media is unidirectional. Look at the SDP address.

# Confirm rport is in use (it tells the far end the real source port):
$ tshark -r call.pcap -Y sip -T fields -e sip.Via | head -1
# SIP/2.0/UDP 192.168.1.20:5060;branch=z9hG4bK776asdhds;rport=41022;received=81.2.44.9
#                                                        ^^^^^^^^^^^^ ^^^^^^^^^^^^^^^
#                                        the proxy recorded the PUBLIC address/port

Key takeaways

SIP establishes sessions; RTP carries media — they are independent, and an established call's audio survives the signalling path failing.
SIP is modelled on HTTP: text-based, request-response, with methods (INVITE, ACK, BYE, REGISTER, REFER) and HTTP-style status codes plus essential 1xx provisional responses.
A SIP URI is an address-of-record; REGISTER binds it to current contacts, and multiple bindings cause an INVITE to fork to every device.
INVITE uses a three-way handshake (INVITE/200/ACK) because the wait for a human to answer is unbounded.
Media parameters travel in SDP using offer/answer: the answer is the intersection of supported codecs.
NAT is SIP's chronic problem because SDP carries addresses inside the body, which NAT does not rewrite — the same failure as FTP, fixed with rport, STUN, symmetric RTP, or an SBC.
The classic symptom is one-way audio: signalling succeeds, media flows in one direction only.

🧪 Practice

List the SIP messages exchanged for a successful call from INVITE to hang-up, and mark which ones carry SDP.

Explain why INVITE needs an ACK when other SIP methods do not.

Alice offers opus, G.722, and PCMU; Bob supports PCMU and PCMA. State the negotiated codec and why.

Interview-style: A call connects and rings normally, but only one party can hear the other. (Hint: signalling and media take different paths — ask which address the SDP body advertised and whether anything on the path could have rewritten it.)

WebRTC Peer Connections

Theory

WebRTC is the culmination of this chapter: it takes every mechanism covered — NAT traversal, RTP and RTCP, SDP offer/answer, jitter buffers, adaptive bitrate, DTLS — and packages them so a web page can open a direct, encrypted, real-time connection to another browser with no plugin and no installed software.

Its defining architectural decision is that WebRTC specifies no signalling protocol. This looks like an omission and is deliberate. Peers must exchange SDP offers and answers and ICE candidates before any connection exists, but how they exchange them is left entirely to the application — WebSocket, HTTP polling, SIP, or a message written on paper would all work. This is the control-plane/data-plane split from 8.1 stated as a specification: the standard defines the data plane and refuses to constrain the control plane.

Connection establishment is where the NAT problem of 8.1 and 8.6 is finally solved properly, using ICE (Interactive Connectivity Establishment):

Each peer gathers candidates — every address it might be reachable at:
- host — local interface addresses.
- server-reflexive (srflx) — the public address as seen by a STUN server, discovered by asking "what address did this packet come from?".
- relay — an address on a TURN server, which forwards media if nothing else works.
Candidates are exchanged through the application's signalling channel.
Peers test every candidate pair simultaneously with STUN connectivity checks, and select the best pair that works.

The ordering is economic as much as technical: host is free, srflx is nearly free, and relay costs the operator the full media bitrate in both directions. Roughly 10–20% of connections require TURN, and driving that number down is a budget line.

Security is not optional. WebRTC mandates DTLS for key exchange and SRTP for media; there is no unencrypted mode. Fingerprints of the DTLS certificates are carried in the SDP, so the signalling channel authenticates the media channel — which is why the signalling channel must itself be secured.

Three API surfaces cover everything:

RTCPeerConnection — the connection, ICE, DTLS, and media transport.
MediaStream (via getUserMedia) — camera and microphone capture.
RTCDataChannel — arbitrary application data over SCTP-over-DTLS, offering both reliable-ordered and unreliable-unordered modes. That second mode is what games use, and it is exactly the UDP-like service browsers otherwise cannot get.

For multi-party calls, mesh connections scale as O(n²) and become impractical beyond four or five participants, so production systems use an SFU (Selective Forwarding Unit) that receives each participant's stream once and forwards it, or an MCU that mixes streams into one at higher server CPU cost.

Examples

The ICE candidate ladder: try cheapest first

  1. HOST          192.168.1.20:54321     same LAN         cost: 0
        | if unreachable
  2. SRFLX (STUN)  81.2.44.9:54321        NAT hole punched cost: ~0
        | if symmetric NAT on both sides
  3. RELAY (TURN)  203.0.113.50:49152     server forwards  cost: FULL BITRATE x2
                                          ~10-20% of calls


 SIGNALLING IS OUT OF SCOPE -- and that is deliberate

   [browser A]                              [browser B]
        |                                        |
        |  SDP offer / answer, ICE candidates    |
        +----> [ YOUR signalling server ] <------+
               (WebSocket? HTTP? SIP? carrier pigeon?
                WebRTC does not care and does not say)
        |                                        |
        |<====== DTLS + SRTP media, DIRECT =====>|
                 (this part IS specified, and is mandatory)


 MULTI-PARTY: why mesh does not scale

  MESH (n=4): each peer sends to every other      SFU (n=4)
     A---B     connections = n(n-1)/2 = 6           A  B
     |\ /|     uploads per peer = n-1 = 3            \ /
     | X |                                          [SFU]
     |/ \|     n=10 -> 45 connections,               / \
     C---D            9 uploads each                C   D
                      -> unusable               1 upload per peer


 ENCRYPTION IS MANDATORY

   SDP carries: a=fingerprint:sha-256 4A:AD:B9:B1:3F:...
                               ^ hash of the DTLS certificate
   -> the signalling channel authenticates the media channel
   -> which is why the signalling channel MUST be secure (wss://, not ws://)

# ICE candidate gathering, prioritisation, and pair selection.

def candidate(typ, ip, port, component=1, local_pref=65535):
    """RFC 8445 priority: type preference dominates, so host beats srflx beats relay."""
    type_pref = {"host": 126, "srflx": 100, "prflx": 110, "relay": 0}[typ]
    priority = (type_pref << 24) | (local_pref << 8) | (256 - component)
    return {"type": typ, "ip": ip, "port": port, "priority": priority}

local = [candidate("host",  "192.168.1.20", 54321),
         candidate("srflx", "81.2.44.9",    54321),
         candidate("relay", "203.0.113.50", 49152)]
remote = [candidate("host",  "10.4.2.9",     61003),
          candidate("srflx", "24.5.113.9",   61003),
          candidate("relay", "203.0.113.50", 49153)]

for c in sorted(local, key=lambda x: -x["priority"]):
    print(f"  {c['type']:<6} {c['ip']:<15}:{c['port']:<6} priority={c['priority']:>12,}")
#   host   192.168.1.20   :54321  priority=2,113,929,471
#   srflx  81.2.44.9      :54321  priority=1,677,721,855
#   relay  203.0.113.50   :49152  priority=      16,776,447
#   Ordering is by cost as much as by quality.

def check_pair(l, r, same_lan=False, nat_type="cone"):
    """Simplified connectivity check outcome."""
    if l["type"] == "host" and r["type"] == "host":
        return same_lan
    if "relay" in (l["type"], r["type"]):
        return True                                    # a relay always works
    return nat_type != "symmetric"                     # hole punching needs a cone NAT

def select_pair(local, remote, **kw):
    pairs = sorted(((l, r) for l in local for r in remote),
                   key=lambda p: -(p[0]["priority"] + p[1]["priority"]))
    for l, r in pairs:
        if check_pair(l, r, **kw):
            return l, r
    return None, None

print()
for name, kw in (("same LAN",          dict(same_lan=True,  nat_type="cone")),
                 ("both behind cone NAT", dict(same_lan=False, nat_type="cone")),
                 ("symmetric NAT",     dict(same_lan=False, nat_type="symmetric"))):
    l, r = select_pair(local, remote, **kw)
    cost = "FREE" if l["type"] != "relay" and r["type"] != "relay" else "RELAYED (costs money)"
    print(f"{name:<22} -> {l['type']:<6} <-> {r['type']:<6}  {cost}")
# same LAN               -> host   <-> host    FREE
# both behind cone NAT   -> srflx  <-> srflx   FREE
# symmetric NAT          -> relay  <-> relay   RELAYED (costs money)

# Mesh vs SFU: the O(n^2) wall.
def topology_load(n, bitrate_mbps=1.5):
    mesh_conns = n * (n - 1) // 2
    mesh_up_per_peer = (n - 1) * bitrate_mbps
    sfu_up_per_peer = bitrate_mbps
    sfu_server = n * bitrate_mbps * (n - 1)          # SFU forwards to everyone else
    return mesh_conns, mesh_up_per_peer, sfu_up_per_peer, sfu_server

print(f"\n{'n':>4}{'mesh conns':>12}{'mesh up/peer':>15}{'SFU up/peer':>13}{'SFU server':>13}")
for n in (2, 4, 6, 10, 50):
    c, mu, su, ss = topology_load(n)
    print(f"{n:>4}{c:>12}{mu:>13.1f}M{su:>12.1f}M{ss:>12.1f}M")
#    n  mesh conns   mesh up/peer  SFU up/peer   SFU server
#    2           1          1.5M         1.5M         1.5M
#    4           6          4.5M         1.5M        18.0M
#    6          15          7.5M         1.5M        45.0M
#   10          45         13.5M         1.5M       135.0M
#   50        1225         73.5M         1.5M      3675.0M
#   13.5 Mb/s uplink for a 10-person mesh call exceeds most home connections.
#   The SFU keeps each peer at 1.5 Mb/s regardless of participant count.

# Data channel modes: the UDP-like service browsers cannot otherwise get.
modes = [
    ("reliable ordered",        dict(ordered=True,  maxRetransmits=None), "file transfer, chat"),
    ("unreliable unordered",    dict(ordered=False, maxRetransmits=0),    "game state, position updates"),
    ("partially reliable",      dict(ordered=False, maxRetransmits=2),    "sensor data, telemetry"),
]
print()
for name, cfg, use in modes:
    print(f"  {name:<22} {str(cfg):<48} {use}")
#   reliable ordered       {'ordered': True, 'maxRetransmits': None}    file transfer, chat
#   unreliable unordered   {'ordered': False, 'maxRetransmits': 0}      game state, position updates
#   partially reliable     {'ordered': False, 'maxRetransmits': 2}      sensor data, telemetry

# TURN cost, the metric that drives engineering effort.
def turn_cost(sessions, minutes, mbps, relay_fraction, usd_per_gb=0.09):
    gb = sessions * relay_fraction * minutes * 60 * mbps * 2 / 8 / 1000
    return gb, gb * usd_per_gb

print()
for frac in (0.20, 0.12, 0.06):
    gb, usd = turn_cost(2_000_000, 25, 1.5, frac)
    print(f"relay {frac:>4.0%} -> {gb:>12,.0f} GB  ${usd:>10,.0f}/month")
# relay  20% ->    2,250,000 GB  $   202,500/month
# relay  12% ->    1,350,000 GB  $   121,500/month
# relay   6% ->      675,000 GB  $    60,750/month

// The minimal WebRTC flow. Note that signalling is YOUR code, not the API's.

const pc = new RTCPeerConnection({
  iceServers: [
    { urls: "stun:stun.example.com:3478" },                  // free: discovers srflx
    { urls: "turn:turn.example.com:3478",                    // costs money: last resort
      username: "user", credential: "pass" }
  ],
  iceTransportPolicy: "all"        // "relay" forces TURN -- useful for testing only
});

// 1. Capture local media.
const stream = await navigator.mediaDevices.getUserMedia({ audio: true, video: true });
stream.getTracks().forEach(t => pc.addTrack(t, stream));

// 2. A data channel, created BEFORE the offer so it is negotiated in it.
const chan = pc.createDataChannel("game", {
  ordered: false,          // do not wait for lost packets
  maxRetransmits: 0        // never retransmit -> UDP-like, for position updates
});

// 3. Candidates are discovered asynchronously -- send each one as it appears
//    (trickle ICE; waiting for them all would add seconds to setup).
pc.onicecandidate = ({ candidate }) => {
  if (candidate) signaling.send({ type: "candidate", candidate });
};

// 4. Offer/answer, exchanged over YOUR signalling channel.
const offer = await pc.createOffer();
await pc.setLocalDescription(offer);
signaling.send({ type: "offer", sdp: offer.sdp });

signaling.onmessage = async ({ type, sdp, candidate }) => {
  if (type === "answer")    await pc.setRemoteDescription({ type, sdp });
  if (type === "candidate") await pc.addIceCandidate(candidate);
};

// 5. Watch which candidate pair actually won -- this is the cost question.
pc.oniceconnectionstatechange = async () => {
  if (pc.iceConnectionState === "connected") {
    const stats = await pc.getStats();
    stats.forEach(r => {
      if (r.type === "candidate-pair" && r.state === "succeeded") {
        console.log("selected pair, rtt:", r.currentRoundTripTime);
        // Look up r.localCandidateId to find whether it was host/srflx/RELAY.
      }
    });
  }
};

# Verify a STUN server responds (this is the srflx discovery step):
$ stunclient stun.example.com 3478
# Binding test: success
# Local address: 192.168.1.20:54321
# Mapped address: 81.2.44.9:54321
#                 ^^^^^^^^^^^^^^^^ the server-reflexive candidate

# Determine the NAT type -- this predicts whether TURN will be needed:
$ stunclient --mode full stun.example.com 3478 | grep -E 'Nat behavior|filtering'
# Nat behavior: Endpoint Independent Mapping     <- cone NAT: hole punching works
# Nat filtering: Address Dependent Filtering
#   "Address and Port Dependent Mapping" = symmetric NAT -> TURN required

# Test a TURN server's credentials before users discover they are wrong:
$ turnutils_uclient -u user -w pass -p 3478 turn.example.com
# 0: Total transmit time is 1 sec; Total lost packets 0

# Measure what fraction of your sessions are relayed (the cost metric):
$ jq -r '.candidatePairs[] | select(.state=="succeeded") | .localCandidateType' \
    sessions/*.json | sort | uniq -c
#   4102 host
#  11893 srflx
#   2205 relay        <- 12% relayed; each point is real money

# Confirm media is encrypted -- WebRTC has no unencrypted mode:
$ tshark -r webrtc.pcap -Y 'dtls.handshake.type == 1' -T fields -e dtls.handshake.type
# 1
$ tshark -r webrtc.pcap -Y 'rtp' -T fields -e rtp.payload -c 1 | head -c 32
# 8f3a2b91c4d5e6f7a0b1c2d3e4f5a6b7      <- SRTP ciphertext

# Force relay-only to test the TURN path works before you need it:
#   iceTransportPolicy: "relay"  in the RTCPeerConnection config

Key takeaways

WebRTC bundles ICE, STUN/TURN, DTLS, SRTP, RTP/RTCP, and SDP offer/answer into a browser API requiring no plugin.
Signalling is deliberately unspecified: the application chooses how to exchange SDP and candidates, which is the control-plane/data-plane split written into a standard.
ICE tries candidates cheapest-first — host, then server-reflexive via STUN, then relay via TURN — and TURN is the only one that costs the operator bandwidth.
Roughly 10–20% of sessions require TURN, so NAT-traversal success rate is a direct cost metric.
Encryption is mandatory: DTLS for key exchange, SRTP for media, with certificate fingerprints in the SDP — so the signalling channel must itself be secure.
RTCDataChannel offers unreliable, unordered delivery over SCTP-over-DTLS, the UDP-like service browsers cannot otherwise obtain.
Mesh topologies scale as O(n²) and break past four or five participants; an SFU keeps each peer's upload at one stream regardless of participant count.

🧪 Practice

List the three ICE candidate types in priority order and state which one costs the operator bandwidth.

Explain why WebRTC deliberately does not specify a signalling protocol, and name two protocols an application could use.

Compute per-peer upload for a 7-participant call at 2 Mb/s under mesh and under an SFU, and say where mesh becomes impractical.

Interview-style: Why is an unreliable, unordered RTCDataChannel the right choice for multiplayer game state? (Hint: ask what a retransmitted position update describes by the time it arrives, and compare with the packet that superseded it.)