2. Computing and Networking Fundamentals

Every architecture diagram eventually runs on real CPUs, real disks, and real cables with real physics, and the properties of that hardware decide which designs are fast, which are cheap, and which are impossible. This chapter covers the machine-level resource model, the network protocols that carry every request, the communication patterns built on them, and the infrastructure that routes traffic in production.

<a id="21-hardware-and-resource-model"></a>

2.1 Hardware and Resource Model

Every system is built from four resources with wildly different speeds and costs; knowing their relative magnitudes is what turns architecture from guesswork into arithmetic.

CPU, Memory, Disk, and Network as Resources

Every request your system serves consumes a mix of four things: CPU cycles to compute, memory to hold working data, disk to persist it, and network to move it. A system's capacity is set by whichever of the four saturates first, so knowing what each one costs is the foundation of every capacity estimate and every performance investigation.

Think of a restaurant kitchen. The chefs are CPU — they do the actual work, and you can only hire so many. The counter space is memory — fast to reach, but small and expensive. The walk-in pantry is disk — huge and cheap, but every trip costs you time. And the delivery van is the network — it brings ingredients from far away, and no amount of kitchen efficiency makes the van arrive faster.

THE RESOURCE HIERARCHY (typical single server, order of magnitude)

  Resource   Capacity          Speed              Cost per unit    Volatile?
  --------------------------------------------------------------------------
  CPU        8-128 cores       ~1 ns / cache hit  high             n/a
  Memory     64 GB - 2 TB      ~100 ns            ~$5/GB/month     yes
  SSD        1-30 TB           ~100 us            ~$0.10/GB/month  no
  HDD        1-20 TB           ~10 ms (seek)      ~$0.02/GB/month  no
  Network    10-100 Gbps       ~0.5 ms in-DC      egress billed    n/a

  Rule of thumb: each step down is ~1000x slower and ~10x cheaper per byte.

The practical skill is classifying a workload by which resource it leans on, because the fix is completely different in each case.

# Same "work", three very different resource profiles.

def cpu_bound(image_bytes):
    # Pure computation: scales with cores, benefits from vectorization,
    # gains nothing from more RAM or a faster disk.
    return resize_and_compress(image_bytes)

def memory_bound(user_ids):
    # Holds a large working set; the fix is a bigger cache or smaller records,
    # not more cores. Exceeding RAM triggers swapping and falls off a cliff.
    return {uid: SESSION_CACHE[uid] for uid in user_ids}

def io_bound(order_id):
    # Spends nearly all its wall-clock time waiting. More cores do nothing;
    # concurrency (async, connection pools) is what raises throughput.
    row = db.query("SELECT * FROM orders WHERE id = %s", order_id)   # ~1-10 ms
    resp = http.get(f"https://psp.example.com/charges/{row.charge}")  # ~50 ms
    return combine(row, resp)

A crucial asymmetry: CPU-bound work is limited by how many workers you have, while I/O-bound work is limited by how many you can have waiting at once. That is why a Node.js or async Python service handles thousands of concurrent slow requests on one core, and why adding cores to an I/O-bound service changes nothing.

Symptom	Likely bound	First lever
CPU at 90%+, latency rises with load	CPU	Profile hot paths, add cores/instances
Steady swapping, GC pauses, OOM kills	Memory	Shrink working set, add RAM, cache less
High iowait, disk queue depth > 1	Disk	Batch writes, add IOPS, use SSD
Link near capacity, retransmits rising	Network	Compress, cache at edge, move data closer
All four idle but latency is high	Waiting	Downstream dependency or lock contention

Key Takeaways

• Every request consumes CPU, memory, disk, and network; capacity is set by whichever saturates first.
• Each step down the hierarchy is roughly 1000x slower and 10x cheaper per byte.
• CPU-bound work needs more workers; I/O-bound work needs more concurrency.
• Identify which resource a workload is bound by before optimizing anything.

🧪 Practice

1. Classify each as CPU-, memory-, disk-, or network-bound: video transcoding, an in-memory session store, a nightly log compaction job, serving 4K video to viewers.
2. A service is at 20% CPU with 300 ms p99 latency. List three explanations and the metric that would confirm each.
3. Interview: You must serve a 50 GB dataset with sub-millisecond lookups. Walk through your options. (Hint: compare the cost of keeping it entirely in RAM against the latency of an SSD, and consider whether the access distribution is uniform.)

Latency Numbers Every Engineer Should Know

The most useful thing an engineer can carry in their head is a rough sense of how long operations take relative to one another. Not because the exact figures matter — they change with every hardware generation — but because the ratios barely change, and the ratios are what decide designs. If you know a network round trip costs 100,000 times a memory access, you never need to be told to cache.

LATENCY LADDER (orders of magnitude; scaled so 1 ns = 1 second)

 Operation                              Real time      Human scale
 -----------------------------------------------------------------------
 L1 cache reference                     1 ns           1 second
 Branch mispredict                      3 ns           3 seconds
 L2 cache reference                     4 ns           4 seconds
 Mutex lock/unlock (uncontended)        17 ns          17 seconds
 Main memory reference                  100 ns         1.7 minutes
 Compress 1 KB                          2 us           33 minutes
 Read 1 MB sequentially from memory     3 us           50 minutes
 SSD random read                        16 us          4.4 hours
 Read 1 MB sequentially from SSD        0.5 ms         5.8 days
 Round trip within the same datacenter  0.5 ms         5.8 days
 HDD seek                               10 ms          3.9 months
 Round trip across a continent          60 ms          1.9 years
 Round trip Europe <-> Australia        250 ms         7.9 years

Reading the ladder as human time makes the design consequences obvious. A memory access is a coffee break; a cross-ocean round trip is nearly a decade. No amount of clever code closes that gap — only moving the data closer does.

# Latency budgets are built by adding up the ladder, not by guessing.
def timeline_request_budget():
    budget = {
        "tls + tcp handshake (cached session)":   5,   # ms, client to edge
        "edge -> origin round trip":             40,
        "auth check (cached in memory)":          0.1,
        "db query (indexed, same AZ)":            2,
        "3 sequential internal calls @ 1 ms":     3,    # sequential is the trap
        "serialize 200 KB json":                  4,
        "network transfer to client":            25,
    }
    total = sum(budget.values())
    print(f"{total:.1f} ms budget consumed")             # 79.1 ms
    # Making the 3 internal calls concurrent saves 2 ms. Moving the origin
    # closer to the edge saves 40. Always attack the biggest term.
    return total

timeline_request_budget()

Three consequences worth internalizing:

• Anything crossing a network is at least ~10,000x a memory access. Batch, cache, and parallelize network calls; sequential chains of them are the most common cause of slow APIs.
• Physics sets a floor. Light in fibre travels ~200,000 km/s, so a 20,000 km round trip cannot beat ~100 ms no matter what you build. Global low latency means geographic distribution, not optimization.
• Random disk access on spinning media is catastrophic at ~10 ms; this single number explains why databases are built around sequential writes.

Key Takeaways

• Memorize the ratios, not the exact numbers; the ratios drive design.
• Memory ~100 ns, SSD ~16 us, in-datacenter round trip ~0.5 ms, cross-ocean ~250 ms.
• Network calls dominate latency budgets — batch and parallelize them.
• The speed of light imposes a hard floor that only geography can lower.

🧪 Practice

1. Estimate the total latency of a request that makes 5 sequential database queries in the same datacenter, then the same 5 run concurrently.
2. A page issues 30 sequential API calls to a server 80 ms away. How long does it take? Propose two fixes and estimate each one's saving.
3. Interview: Why is reading 1 MB from memory faster than reading 1 MB from SSD, even though both are "just reading bytes"? (Hint: think about what physically has to happen per request, and about per-operation overhead versus per-byte cost.)

Sequential vs. Random I/O

Two programs can read the same number of bytes from the same disk and differ in speed by a factor of a hundred. The difference is access pattern: reading bytes that sit next to each other (sequential) versus jumping around (random). This single property shapes the internals of every database and file format you will ever use.

On a spinning disk the reason is mechanical: a read head must physically move to the right track and wait for the platter to rotate under it. That is the ~10 ms seek. Once positioned, streaming the next megabyte costs almost nothing. The analogy is a record player — dropping the needle takes a second, but playing continuously is free.

SSDs have no moving parts, so the gap narrows dramatically, but it does not close. Each request carries fixed overhead (command submission, flash translation lookup), reads happen in pages of 4-16 KB, and erases happen in much larger blocks. Sequential access amortizes the per-operation cost across many bytes and lets the device and the OS read ahead.

THROUGHPUT BY ACCESS PATTERN (order of magnitude)

                    Sequential        Random 4 KB       Ratio
  HDD               ~200 MB/s         ~1 MB/s           ~200x
  SATA SSD          ~550 MB/s         ~90 MB/s          ~6x
  NVMe SSD          ~5,000 MB/s       ~600 MB/s         ~8x

  Disk layout:
    sequential  [####][####][####][####]   one seek, then stream
    random      [##]......[##]....[##]     a seek per operation

# The design consequence: turn random writes into sequential ones.

# Anti-pattern: update rows one at a time -> random writes + a round trip each
for order in orders:                                    # 10,000 orders
    db.execute("UPDATE orders SET status=%s WHERE id=%s",
               (order.status, order.id))                # ~10,000 random I/Os

# Better: one batched, sequentially written statement
db.execute_many("UPDATE orders SET status=%s WHERE id=%s",
                [(o.status, o.id) for o in orders])     # 1 round trip

# Best for append-heavy workloads: write an ordered log, sort/merge later.
# This is exactly what an LSM tree does — buffer writes in memory, flush them
# as one sorted, sequentially written file, and merge files in the background.
with open("events.log", "ab") as f:                     # append-only = sequential
    for e in events:
        f.write(serialize(e))
    f.flush()
    os.fsync(f.fileno())                # one durability barrier for the batch

This is why so many storage systems look the way they do: write-ahead logs are append-only, LSM trees convert random writes into sequential flushes, columnar formats store each column contiguously so scans stream, and clustered indexes physically order rows so that range queries read consecutive pages.

Pattern	Cost driver	Design response
Random point reads	One I/O per lookup	Cache in memory; use an index
Random writes	Seek/erase amplification	Buffer and flush sequentially (LSM)
Full scans	Bytes moved	Columnar layout, compression
Range queries	Locality	Clustered index, sorted files

Key Takeaways

• Sequential I/O beats random I/O by ~200x on HDD and still ~6-8x on SSD.
• The gap comes from seek time (HDD) and per-operation overhead plus page granularity (SSD).
• Storage engines are designed to convert random writes into sequential ones.
• Batching is the simplest way to turn random access into sequential access.

🧪 Practice

1. Estimate the time to read 1 GB sequentially versus as 250,000 random 4 KB reads, on both HDD and NVMe SSD.
2. Explain why appending to a log file and periodically merging is faster than updating records in place, despite writing more total bytes.
3. Interview: Why do LSM trees favour write-heavy workloads while B-trees favour read-heavy ones? (Hint: think about which one turns writes into sequential I/O and what it pays for that at read time.)

Bare Metal vs. Virtual Machines vs. Containers

Before you can run code you must decide what it runs on, and the options form a spectrum trading isolation and control against density and startup speed. The choice affects performance predictability, cost, deployment speed, and blast radius — so it is an architectural decision, not just an ops detail.

Bare metal is a physical server dedicated to you: maximum performance, no hypervisor tax, complete control, but slow to provision and impossible to subdivide efficiently. Virtual machines run a hypervisor that emulates hardware, letting many guest operating systems share one physical host with strong isolation. Containers skip the guest OS entirely — they are processes on a shared kernel, isolated by namespaces (what a process can see) and cgroups (what it can consume).

The analogy: bare metal is owning a house, a VM is renting an apartment with your own plumbing and walls, and a container is renting a room in a shared flat — cheap and instant, but you share the kitchen, and a badly behaved flatmate affects you.

  BARE METAL              VIRTUAL MACHINES           CONTAINERS

 +-------------+        +-----------+-----------+   +-----+-----+-----+
 |    App      |        |   App     |   App     |   | App | App | App |
 +-------------+        +-----------+-----------+   +-----+-----+-----+
 |     OS      |        | Guest OS  | Guest OS  |   |   Shared libs   |
 +-------------+        +-----------+-----------+   +-----------------+
 |  Hardware   |        |      Hypervisor       |   |   Host kernel   |
 +-------------+        +-----------------------+   +-----------------+
                        |      Hardware         |   |    Hardware     |
                        +-----------------------+   +-----------------+

  isolation:  strongest ------------------------------------> weakest
  density:    lowest    ------------------------------------> highest
  boot time:  minutes           ~30 seconds            ~100 milliseconds

# A container image is a filesystem plus metadata — no kernel inside it.
FROM python:3.12-slim              # base layer: userland only, shares host kernel
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt   # cached layer
COPY . .
# Limits are enforced by cgroups at run time, not by the image:
#   docker run --cpus=2 --memory=1g myapp
# Exceeding the memory limit gets the process OOM-killed, not swapped.
CMD ["python", "-m", "app"]

Dimension	Bare metal	Virtual machine	Container
Isolation boundary	Physical	Hypervisor	Kernel namespaces
Startup time	Minutes to hours	Tens of seconds	Sub-second
Overhead	None	2-10%	~1%
Density per host	1	Tens	Hundreds
Kernel	Yours	Yours (guest)	Shared with host
Blast radius of escape	n/a	Very contained	Host-wide risk
Best for	Predictable high performance, licensing	Mixed OSes, strong tenancy isolation	Microservices, fast scaling and CI

Two practical warnings. Noisy neighbours: on shared hosts another tenant's load can steal CPU and I/O from you — watch the "steal time" metric on VMs. Shared kernel: containers of mutually untrusted tenants on one kernel are a security decision; that is why lightweight VM runtimes (Firecracker, gVisor) exist to give container-like startup with VM-like isolation.

Key Takeaways

• Bare metal, VMs, and containers trade isolation and control against density and startup speed.
• Containers share the host kernel — near-zero overhead, weaker isolation.
• Container limits come from cgroups at run time, not from the image.
• Choose based on isolation requirements, startup speed needs, and tenancy model.

🧪 Practice

1. For each, pick a platform and justify it: a latency-sensitive trading engine, a multi-tenant SaaS running untrusted customer code, a CI runner fleet, a legacy Windows application.
2. Explain why a container starts in milliseconds while a VM takes tens of seconds.
3. Interview: Your containerized service is randomly slow but its CPU usage looks low. What do you check? (Hint: the metric you want describes time the hypervisor gave to someone else, and there is a second one about cgroup throttling.)

Vertical Resource Limits

Scaling up feels like the simplest answer to load — buy a bigger machine — and it works until it does not. Knowing precisely where the ceilings are tells you when to stop scaling vertically and start scaling horizontally, and it prevents a whole class of outages caused by exhausting a limit nobody was watching.

The ceilings are not just "cores and RAM". The ones that actually cause production incidents are the small, unglamorous ones.

LIMITS THAT BITE, IN ROUGH ORDER OF HOW OFTEN THEY CAUSE INCIDENTS

 1. File descriptors      default ulimit often 1024; every socket and file is one
 2. Ephemeral ports       ~28,000 per source IP:dest pair -> outbound conn limit
 3. Thread count          each thread costs ~1 MB of stack; 10k threads = 10 GB
 4. Connection pool size  smaller than you think; blocks under load
 5. Memory bandwidth      cores starve waiting for RAM before CPU saturates
 6. NUMA locality         cross-socket memory access is ~2x slower
 7. Single-core ceiling   one hot lock or one single-threaded loop caps it all
 8. Network interface     10-100 Gbps per NIC, shared by everything on the host
 9. Instance size         the largest cloud instance is a hard wall

# Ephemeral port exhaustion: the classic "we scaled up and it broke" bug.
EPHEMERAL_RANGE = 32_768, 60_999          # typical Linux default
ports_available = EPHEMERAL_RANGE[1] - EPHEMERAL_RANGE[0] + 1
print(ports_available)                     # 28232

# A TCP connection is identified by (src_ip, src_port, dst_ip, dst_port), so
# the limit applies PER destination pair, not globally.
TIME_WAIT_SECONDS = 60                     # sockets linger after close
max_new_conns_per_second = ports_available / TIME_WAIT_SECONDS
print(round(max_new_conns_per_second))      # 470

# 470 new connections/second to a single backend, per source IP. A service
# opening a fresh connection per request hits this wall long before CPU does.
# Fixes: connection pooling and keep-alive (reuse), more destination IPs,
# or widening the port range.

# Checking the ceilings on a running host
ulimit -n                                   # file descriptor soft limit
cat /proc/sys/net/ipv4/ip_local_port_range  # ephemeral port range
ss -s                                       # socket counts by state
nproc                                       # visible cores
numactl --hardware                          # NUMA topology and node distances

Limit	Typical default	Symptom when hit
File descriptors	1024 (soft)	"Too many open files"; accept() fails
Ephemeral ports	~28k	"Cannot assign requested address"
Thread stacks	1 MB each	OOM at a few thousand threads
Connection pool	10-20	Latency spikes, queueing at the pool
Largest cloud instance	~few TB RAM	Nowhere left to scale up

The strategic point: vertical scaling buys time, and time is genuinely valuable — it is far cheaper than a distributed rewrite. But its ceiling is fixed and its cost curve turns superlinear at the top end, while a single machine remains a single failure domain. Plan the horizontal path before you need it.

Key Takeaways

• Vertical limits are rarely CPU or RAM; they are descriptors, ports, threads, and pools.
• Ephemeral port exhaustion caps new outbound connections at a few hundred per second per destination — pool and keep alive.
• The largest instance is a hard wall, and one machine is one failure domain.
• Scale up to buy time, but design the horizontal path before you hit the ceiling.

🧪 Practice

1. A service opens a new database connection per request at 2,000 RPS. Which limit does it hit first, and what is the fix?
2. Calculate how much memory 5,000 threads consume at the default stack size, and propose an architecture that avoids needing them.
3. Interview: Your service works fine at 500 RPS and fails at 1,500 RPS with connection errors, while CPU sits at 40%. Diagnose it. (Hint: enumerate the per-process and per-host limits a connection consumes.)

<a id="22-network-protocols"></a>

2.2 Network Protocols

Every request in a distributed system is carried by a stack of protocols, and each layer contributes latency, failure modes, and constraints you will eventually have to reason about.

OSI and TCP/IP Models

Networking is hard because it solves many problems at once: turning bits into signals, finding a machine across the planet, recovering lost data, and agreeing on what the bytes mean. Layering is how that complexity is made tractable — each layer solves one problem and offers a clean service to the layer above, so TCP does not care whether it runs over fibre or Wi-Fi, and HTTP does not care how packets were retransmitted.

The OSI model is the seven-layer teaching model. The TCP/IP model is the four-layer description of what is actually implemented. You will hear both, often mixed ("a layer 7 load balancer" is OSI language about an HTTP proxy).

OSI layer	TCP/IP layer	Job	Examples
7 Application	Application	Meaning of the data	HTTP, gRPC, DNS, SMTP
6 Presentation	Application	Encoding, encryption, compression	TLS, JSON, Protobuf
5 Session	Application	Dialogue and session state	TLS sessions, RPC sessions
4 Transport	Transport	Process-to-process delivery	TCP, UDP, QUIC
3 Network	Internet	Host-to-host routing across networks	IP, ICMP, BGP
2 Data link	Link	Frames on one physical hop	Ethernet, Wi-Fi, ARP
1 Physical	Link	Bits on the wire	Copper, fibre, radio

The mechanism that makes layering work is encapsulation: each layer wraps the payload from above in its own header, and the receiving side unwraps in reverse. It is a set of nested envelopes — the postal service reads only the outer address and never opens the letter.

ENCAPSULATION OF ONE HTTP REQUEST

  Application   [ GET /orders HTTP/1.1  Host: api.example.com ]
                              |
  TLS           [ TLS record | encrypted application data     ]
                              |
  Transport     [ TCP hdr | ...payload...                     ]   src/dst port,
                              |                                   seq, ack, flags
  Network       [ IP hdr  | TCP segment                       ]   src/dst IP, TTL
                              |
  Link          [ Eth hdr | IP packet | Eth trailer           ]   MAC addresses
                              |
  Physical       101101000111010101110...

  Overhead: ~40 bytes of TCP+IP headers per packet, plus ~20-60 for TLS
  records. On a 1500-byte MTU that is ~3-7% pure protocol tax.

Why this matters architecturally: it tells you what a device can see. A layer 4 load balancer sees IPs and ports, so it can route fast but cannot read a URL path. A layer 7 load balancer terminates the connection and parses HTTP, so it can route by path or header — at the cost of doing more work and breaking the end-to-end connection.

Key Takeaways

• Layering isolates concerns so each protocol can evolve independently.
• OSI has seven layers (teaching model); TCP/IP has four (what is built).
• Encapsulation wraps each layer's payload in headers, adding a few percent of overhead.
• The layer a device operates at determines what it can see and therefore what routing decisions it can make.

🧪 Practice

1. For each, name the OSI layer: an Ethernet MAC address, a TCP port, a TLS certificate, an HTTP Host header, a BGP route.
2. Compute the protocol overhead percentage of sending a 100-byte payload over TCP/IP on Ethernet, and compare it with a 1400-byte payload.
3. Interview: What can a layer 4 load balancer do that a layer 7 one cannot, and vice versa? (Hint: think about what each must decrypt or parse, and what that costs.)

TCP vs. UDP

IP gets a packet from one host to another, but it makes no promises: packets may be lost, duplicated, reordered, or delayed. The transport layer decides what to do about that, and there are exactly two mainstream answers. TCP turns the unreliable packet service into a reliable, ordered byte stream. UDP does almost nothing and hands you the raw unreliability.

TCP is a phone call: you dial, the other side picks up, you both know the call is live, and words arrive in order. UDP is shouting across a crowded room: fast, no setup, and no guarantee anyone heard.

TCP THREE-WAY HANDSHAKE (one full round trip before any data)

  Client                                Server
    |  ---------- SYN (seq=x) --------->  |
    |  <----- SYN-ACK (seq=y, ack=x+1) -- |
    |  ---------- ACK (ack=y+1) -------->  |
    |  ========== data flows ==========>  |

  Cost: 1 RTT before the first byte. Add TLS 1.3 (1 more RTT) and a
  cross-continent link (60 ms RTT) and you have spent 120 ms before the
  server has even seen the request path.

TCP's guarantees are not free. It gives you: reliable delivery (lost segments are retransmitted), ordering (a sequence number per byte), flow control (the receiver advertises a window so a fast sender cannot swamp it), and congestion control (the sender backs off when the network shows loss). The price is connection setup, per-connection state on both ends, and head-of-line blocking — one lost segment stalls delivery of everything behind it, even data that already arrived.

Property	TCP	UDP
Connection	Handshake required (1 RTT)	Connectionless, send immediately
Reliability	Retransmits lost data	None; the app must handle loss
Ordering	Guaranteed	None
Flow/congestion control	Yes	None (app must implement)
Header size	20+ bytes	8 bytes
Head-of-line blocking	Yes	No
Typical uses	HTTP/1-2, databases, RPC, email	DNS, QUIC/HTTP3, VoIP, gaming, metrics

import socket

# TCP: connect, then treat it as a stream. recv() may return partial data,
# so real code must frame messages itself (length prefix or delimiter).
tcp = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
tcp.settimeout(5)                       # ALWAYS set a timeout; the default is forever
tcp.connect(("example.com", 80))        # costs one round trip
tcp.sendall(b"GET / HTTP/1.1\r\nHost: example.com\r\nConnection: close\r\n\r\n")
data = tcp.recv(4096)                   # a "stream": boundaries are not preserved

# UDP: no connect, no guarantee. Each sendto() is one independent datagram
# that may vanish, arrive twice, or arrive out of order.
udp = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
udp.settimeout(2)
udp.sendto(b"metric:latency:42", ("stats.internal", 8125))   # fire and forget
# Chosen deliberately here: losing one metric sample is cheaper than making
# the application wait on a handshake and retransmissions.

Choose UDP when loss is cheaper than delay (live audio, telemetry, game state) or when you intend to implement your own reliability — which is exactly what QUIC does, rebuilding ordering and retransmission in userspace on top of UDP to escape TCP's head-of-line blocking and ossified middleboxes.

Key Takeaways

• TCP provides reliability, ordering, flow and congestion control at the cost of a handshake and head-of-line blocking.
• UDP provides nothing beyond addressing — which makes it fast and fully controllable by the application.
• A TCP handshake costs one RTT before any data; TLS adds more.
• Use UDP when timeliness beats completeness, or when building custom reliability such as QUIC.

🧪 Practice

1. Choose TCP or UDP for each and justify: file upload, live video call, DNS query, database replication, real-time multiplayer position updates.
2. Explain why a single lost packet can stall an entire HTTP/2 connection but not an HTTP/3 one.
3. Interview: Your API's p99 latency is dominated by connection setup. What do you change? (Hint: count the round trips before the first byte, and consider which of them can be reused or eliminated.)

IP Addressing and Subnets

Every machine on a network needs an address, and every design that involves a private network, a VPC, a firewall rule, or a peering connection eventually becomes a conversation about address ranges. Getting the plan wrong is expensive: overlapping ranges cannot be peered, and re-addressing a live network is one of the least pleasant migrations there is.

An IPv4 address is 32 bits, written as four bytes (10.0.12.5). CIDR notation appends a prefix length: 10.0.0.0/16 means "the first 16 bits are the network, the remaining 16 identify hosts". The smaller the number after the slash, the bigger the block. Think of it as a postcode: /8 is a country, /16 a city, /24 a street, /32 a single house.

CIDR SIZING CHEAT SHEET

  Prefix   Addresses   Typical use
  ------------------------------------------------------
  /8       16,777,216  A whole private range (10.0.0.0/8)
  /16      65,536      One VPC
  /20      4,096       A large subnet
  /24      256         One subnet per AZ (a common default)
  /28      16          A small isolated subnet
  /32      1           A single host (used in firewall rules)

  Private ranges (RFC 1918) — never routed on the public internet:
    10.0.0.0/8        172.16.0.0/12        192.168.0.0/16

import ipaddress

vpc = ipaddress.ip_network("10.0.0.0/16")
print(vpc.num_addresses)                       # 65536

# Carve one /24 subnet per availability zone
subnets = list(vpc.subnets(new_prefix=24))     # 256 possible /24s
for name, net in zip(("az-a", "az-b", "az-c"), subnets):
    print(name, net, net.num_addresses)
# az-a 10.0.0.0/24 256
# az-b 10.0.1.0/24 256
# az-c 10.0.2.0/24 256

# Cloud providers reserve addresses in every subnet (AWS reserves 5:
# network, VPC router, DNS, future use, broadcast), so usable hosts are fewer.
print(256 - 5)                                 # 251 usable per /24

# Overlap is the cardinal sin: these two VPCs can never be peered.
a = ipaddress.ip_network("10.0.0.0/16")
b = ipaddress.ip_network("10.0.128.0/17")
print(a.overlaps(b))                           # True -> re-address one of them

IPv6 solves exhaustion with 128-bit addresses (2001:db8::1), enough that NAT becomes unnecessary and every device can have a globally routable address. It is widely deployed at the edge (mobile networks, CDNs) and increasingly inside cloud VPCs, but dual-stack operation remains the norm.

Three planning rules that prevent most pain:

• Allocate generously. Address space is free; a /16 per VPC costs nothing and leaves room to grow.
• Never overlap across environments, accounts, or partners you might ever connect — reserve a distinct block per region and per environment up front.
• Leave gaps between allocations so a subnet can be expanded without fragmenting the plan.

Key Takeaways

• CIDR prefix length sets block size: smaller prefix means more addresses.
• RFC 1918 ranges (10/8, 172.16/12, 192.168/16) are for private networks.
• Cloud subnets reserve several addresses, so usable capacity is less than the arithmetic suggests.
• Overlapping CIDRs cannot be peered; plan non-overlapping ranges from day one and allocate generously.

🧪 Practice

1. How many usable addresses are in a /26? How many /26 subnets fit in a /22?
2. Design a VPC address plan for three regions, three AZs each, with separate public and private subnets and no overlaps.
3. Interview: Two companies merge and both use 10.0.0.0/16 internally. What are your options? (Hint: one option rewrites addresses, one hides them behind a translation layer, and one avoids routing between them entirely.)

DNS Resolution and Record Types

Humans use names and the network uses addresses, so something has to translate between them. DNS is that system: a globally distributed, hierarchical, heavily cached database that turns api.example.com into an IP address. It is also, in practice, the control plane for traffic management — failover, geo-routing, and blue-green cutovers are frequently DNS changes.

Resolution walks down the hierarchy, from the root, to the top-level domain, to the authoritative server for the domain. Almost every step is cached, which is why the average lookup is fast and why changes take time to spread.

RECURSIVE RESOLUTION OF api.example.com

  Browser --> OS cache --> Recursive resolver (ISP / 8.8.8.8 / VPC resolver)
                                  |
                                  |-- root servers:  "ask .com at 192.5.6.30"
                                  |-- .com TLD:      "ask ns1.example.com"
                                  |-- authoritative: "api.example.com = 93.184.x.x"
                                  v
                            answer cached for TTL seconds at every hop

  Cold path: 4 round trips. Warm path (cached): ~0 ms. This is why a low TTL
  costs latency and a high TTL costs agility.

Record	Purpose	Example
A	Name to IPv4 address	api.example.com -> 93.184.216.34
AAAA	Name to IPv6 address	api.example.com -> 2606:2800::1
CNAME	Alias to another name	www -> api.example.com
ALIAS/ANAME	CNAME-like, but legal at the zone apex	example.com -> lb.provider.net
MX	Mail servers, with priority	10 mail.example.com
TXT	Arbitrary text: SPF, DKIM, domain proofs	v=spf1 include:_spf.google.com ~all
NS	Delegates a zone to nameservers	example.com -> ns1.provider.net
SRV	Service location: host, port, priority	_sip._tcp -> 10 60 5060 sip.example.com
PTR	Reverse lookup, IP to name	Used for mail reputation, logging

# TTL is the single most consequential DNS setting in an incident.
TTL_SECONDS = 300          # 5 minutes

# Failover time is bounded below by the TTL, because resolvers and clients
# keep serving the cached answer until it expires.
print(f"worst-case client failover: {TTL_SECONDS}s")     # 300s

# Before a planned migration, lower the TTL well in advance:
#   T-48h: set TTL to 60      (old TTL must expire first for this to take effect)
#   T-0:   change the record  (clients converge within ~60s)
#   T+24h: raise TTL back to 3600 to reduce lookup latency and query cost
#
# Caveat: some clients and JVMs cache DNS beyond the TTL — historically
# forever. Never rely on DNS alone for fast failover; pair it with a load
# balancer or a client that re-resolves.

Two architectural consequences. First, DNS-based load balancing is coarse: you can return different answers per region or weight them for a canary, but you cannot see request paths, and stale caches mean some clients keep using the old answer. Second, DNS is a dependency: if your resolver is unavailable, healthy services become unreachable, which is why large outages so often turn out to be DNS.

Key Takeaways

• DNS resolves names hierarchically (root, TLD, authoritative) with caching at every hop.
• TTL trades failover speed against lookup latency and query cost.
• CNAME cannot exist at a zone apex; use ALIAS/ANAME records there.
• DNS is a control plane and a dependency — never the only failover mechanism.

🧪 Practice

1. Trace the records needed to serve shop.example.com from a load balancer, plus email for the domain.
2. Plan a zero-downtime migration to a new load balancer using TTL changes, with a timeline.
3. Interview: You changed a DNS record 10 minutes ago and some users still hit the old server. Explain why and how to design around it. (Hint: count every layer that could be holding a cached answer, including the application runtime.)

TLS Handshake and Certificates

Plain HTTP sends everything readable to anyone on the path, and offers no way to know the server you reached is the one you asked for. TLS fixes three things at once: confidentiality (nobody can read the traffic), integrity (nobody can modify it undetected), and authentication (the server proves it owns the domain). Every one of those costs round trips, which is why TLS shows up in latency budgets.

Authentication works through a chain of trust. A certificate authority (CA) that your operating system already trusts signs a certificate binding a public key to a domain name. The server presents that certificate; the client verifies the signature chain up to a trusted root, checks the name matches, and checks it has not expired or been revoked.

TLS 1.3 HANDSHAKE (1 RTT) vs TLS 1.2 (2 RTT)

  TLS 1.2                            TLS 1.3
  Client        Server               Client        Server
    |-- ClientHello ---->|             |-- ClientHello + key share -->|
    |<-- ServerHello ----|             |<-- ServerHello + key share --|
    |    Certificate     |             |    Certificate, Finished     |
    |<-- ServerHelloDone-|             |-- Finished ----------------->|
    |-- KeyExchange ---->|             |== application data =========>|
    |-- Finished ------->|
    |<-- Finished -------|             1 RTT (plus TCP's 1 RTT)
    |== data ===========>|
    2 RTT (plus TCP's 1 RTT)

  Session resumption: 0 RTT for repeat visitors (with replay caveats).
  Certificate chain: leaf (api.example.com) -> intermediate CA -> root CA

import ssl, socket, datetime

ctx = ssl.create_default_context()          # verifies chain AND hostname by default
with socket.create_connection(("example.com", 443), timeout=5) as sock:
    with ctx.wrap_socket(sock, server_hostname="example.com") as tls:
        #                       ^ SNI: tells the server which cert to present,
        #                         which is how one IP hosts many TLS domains
        cert = tls.getpeercert()
        print(tls.version())                                    # TLSv1.3
        expires = datetime.datetime.strptime(cert["notAfter"],
                                             "%b %d %H:%M:%S %Y %Z")
        print("days until expiry:", (expires - datetime.datetime.utcnow()).days)

# Never do this in production — it disables the entire authentication guarantee
# and turns TLS into unauthenticated encryption:
#   ctx.check_hostname = False
#   ctx.verify_mode = ssl.CERT_NONE

Design decisions that follow from all this:

• Where to terminate. Terminating TLS at the load balancer or CDN is cheapest and keeps certificates in one place; terminating at each service gives end-to-end encryption. Many designs do both: terminate at the edge, then re-encrypt internally.
• Certificate expiry is an outage waiting to happen. Automate issuance and renewal (ACME/Let's Encrypt or a cloud CA) and alert well before expiry.
• mTLS (mutual TLS) makes the client present a certificate too, which is the standard way services authenticate each other inside a mesh.
• Session resumption and connection reuse matter more than cipher choice for latency: the fastest handshake is the one you do not perform.

Key Takeaways

• TLS provides confidentiality, integrity, and server authentication via a CA-signed certificate chain.
• TLS 1.3 handshakes in 1 RTT (0 RTT on resumption); TLS 1.2 needs 2.
• SNI lets one IP serve many domains; hostname verification is what makes the guarantee real.
• Automate certificate renewal and decide deliberately where TLS terminates.

🧪 Practice

1. Count the round trips for a first-time HTTPS request over TLS 1.3 to a server 60 ms away, then for a resumed connection.
2. Explain what breaks if a client skips hostname verification but still validates the certificate chain.
3. Interview: How would you design certificate management for 200 microservices doing mTLS? (Hint: think about issuance, rotation frequency, and what happens if the issuing service is down.)

HTTP/1.1, HTTP/2, and HTTP/3

HTTP is the protocol almost all application traffic speaks, and its three live versions differ not in semantics — the same methods, headers, and status codes throughout — but in how efficiently they move those semantics over a network. Each version exists to remove the previous one's dominant bottleneck.

HTTP/1.1 sends one request at a time per connection. Keep-alive lets a connection be reused sequentially, but a slow response blocks everything behind it. Browsers worked around this by opening ~6 connections per origin, and developers worked around that by concatenating files and sprite-sheeting images.

HTTP/2 multiplexes many concurrent streams over a single TCP connection and compresses headers with HPACK, eliminating application-level head-of-line blocking. But all those streams still ride one TCP connection, so a single lost packet stalls every stream — TCP-level head-of-line blocking.

HTTP/3 solves that by moving to QUIC over UDP, where each stream has its own loss recovery. It also folds the transport and TLS handshakes together (1 RTT, or 0 on resumption) and identifies connections by an ID rather than the 4-tuple, so a phone switching from Wi-Fi to cellular keeps its connection alive.

SIX REQUESTS OVER EACH VERSION

 HTTP/1.1 (6 connections)      HTTP/2 (1 connection)     HTTP/3 (1 QUIC conn)
  conn1 [req1--------]          [1|2|3|4|5|6 interleaved]  [1|2|3|4|5|6]
  conn2 [req2------]             one TCP connection         independent streams
  conn3 [req3---------]          packet loss stalls ALL     loss stalls ONE
  ...                            streams                    stream
  6x handshakes, 6x congestion   1 handshake, header        1 handshake,
  windows, header repeated       compression                connection migration

// The semantics are identical across versions — this code is unchanged.
const res = await fetch("https://api.example.com/orders", {
  headers: { "authorization": `Bearer ${token}` },
});

// What changes is the transport underneath, and therefore the guidance:
//  - HTTP/1.1: bundle assets, minimize request count, use few connections
//  - HTTP/2+:  many small requests are fine; bundling can HURT caching
//              because one changed byte invalidates the whole bundle
//  - Server push existed in HTTP/2 and is deprecated; use `103 Early Hints`
//    or <link rel=preload> instead.

Feature	HTTP/1.1	HTTP/2	HTTP/3
Transport	TCP	TCP	QUIC over UDP
Concurrency	1 per connection	Multiplexed streams	Multiplexed streams
Head-of-line blocking	Application	TCP level	Per stream only
Header compression	None	HPACK	QPACK
Handshake RTTs (new)	1 (TCP) + 2 (TLS 1.2)	1 + 1 (TLS 1.3)	1 combined
Connection migration	No	No	Yes
Best for	Simple/internal	Most web traffic	Mobile, lossy networks

For internal service-to-service traffic the same reasoning leads to gRPC, which is built on HTTP/2 precisely for its multiplexing and header compression over long-lived connections.

Key Takeaways

• All three versions share HTTP semantics; they differ in transport efficiency.
• HTTP/2 removes application head-of-line blocking but inherits TCP's.
• HTTP/3 (QUIC over UDP) gives per-stream loss recovery, a combined handshake, and connection migration.
• Optimization advice inverts by version: bundle for HTTP/1.1, keep requests granular for HTTP/2 and later.

🧪 Practice

1. Explain why concatenating 50 JavaScript files helps on HTTP/1.1 but can hurt on HTTP/2.
2. A mobile client loses 2% of packets. Predict how HTTP/2 and HTTP/3 behave differently and why.
3. Interview: Your internal RPC traffic is slow despite a fast network. Which HTTP version and connection strategy would you choose? (Hint: think about how many handshakes long-lived internal connections should ever perform.)

<a id="23-communication-patterns"></a>

2.3 Communication Patterns

Protocols determine how bytes move; communication patterns determine who initiates, who waits, and what happens when the other side is slow or gone.

Client-Server Model

Almost every system you will design is a variation on one idea: some processes ask for things and some processes provide them. Centralizing the provision gives you a single place to enforce authorization, validate data, hold the authoritative copy, and deploy fixes — which is why this model dominates despite putting all the load on one side.

The client initiates; the server listens. The asymmetry is the whole point: the server has a stable, discoverable address, while clients come and go, may be untrusted, and may be behind NAT with no reachable address at all.

  [Browser]  \
  [Mobile]    >---- request ---->  [Load Balancer] ---> [Server] ---> [Database]
  [CLI tool] /  <--- response ---

  Clients: many, untrusted, ephemeral, unaddressable
  Servers: few, trusted, long-lived, discoverable
  The server holds the authoritative state; the client holds a cached view.

The property that makes this model scale is statelessness. If a server keeps no per-client state between requests, any server can answer any request, so you can add servers freely and lose one without losing sessions. State does not disappear — it moves to a shared store (a database, a cache, or a signed token carried by the client).

# Stateful: this server cannot be load balanced without sticky sessions,
# and a restart logs everyone out.
SESSIONS = {}                                  # lives in one process's memory

def handle_login_stateful(user, password):
    token = new_token()
    SESSIONS[token] = user                     # only THIS instance knows it
    return token

# Stateless: any instance can serve any request; scale and restart freely.
def handle_login_stateless(user, password, redis, signing_key):
    token = new_token()
    redis.setex(f"sess:{token}", 3600, user.id)   # shared state, explicit TTL
    return token
    # Alternative: a signed JWT carries the state in the token itself,
    # removing the lookup entirely at the cost of not being revocable
    # until it expires.

Property	Client	Server
Initiates	Yes	No (responds)
Addressable	Usually not (NAT, mobile)	Yes (DNS name, stable IP)
Trust	Untrusted; validate everything	Trusted execution environment
Holds truth	A cached view	The authoritative copy
Scaling concern	Bandwidth, battery, cache size	Concurrency, statelessness

The model's limitation is that servers cannot initiate. Any "push" to a client — a notification, a live update — requires one of the workarounds covered later in this subchapter, because the client is the only party that can open the connection.

Key Takeaways

• Clients initiate, servers respond; only servers have stable, reachable addresses.
• Centralization buys authorization, consistency, and deployability.
• Stateless servers are what make horizontal scaling and restarts painless — push state to a shared store or a signed token.
• Servers cannot initiate contact, which is why push requires special patterns.

🧪 Practice

1. List three pieces of state a web server might hold and, for each, say where it should live to keep the server stateless.
2. Explain why sticky sessions reduce fault tolerance and elasticity.
3. Interview: A server stores uploaded files on local disk. What breaks when you scale to three instances behind a load balancer? (Hint: consider both the write path and a later read that lands elsewhere.)

Peer-to-Peer Model

The client-server model concentrates cost at the server: every byte a user downloads is a byte the operator pays to send. Peer-to-peer inverts that. Each participant is both client and server, so capacity grows with the number of participants instead of being consumed by them — the property that makes P2P attractive for content distribution and unavoidable for systems that must survive without central infrastructure.

The analogy is a potluck dinner versus a restaurant. The restaurant scales by hiring more cooks; the potluck scales because every new guest brings a dish.

 CLIENT-SERVER                      PEER-TO-PEER

     [Server]                         [P1]-----[P2]
    /   |    \                         | \     / |
 [C1] [C2] [C3]                        |  \   /  |
                                       |   [P3]  |
 Capacity: fixed, paid by operator     [P4]-----[P5]
 Failure: server down = all down
                                    Capacity: grows with peers
                                    Failure: peers leave constantly (churn)

Pure P2P is rare because discovery is hard: a new peer must find others, and peers behind NAT cannot accept inbound connections. Most real systems are hybrid — a small central service handles discovery, coordination, and NAT traversal, and the bulk data flows peer to peer.

# Hybrid P2P: the central service brokers, the peers carry the payload.
# 1. Peers register with a tracker/signaling server (small, cheap, central)
tracker.announce(peer_id="p7", content_hash="a3f9...", ip="203.0.113.7", port=51413)

# 2. A peer asks who else has the content
peers = tracker.get_peers(content_hash="a3f9...")     # [p2, p4, p9]

# 3. NAT traversal: both peers punch outbound holes toward each other,
#    coordinated through the server, because neither can accept inbound.
#    If that fails, traffic falls back to a relay (TURN) — which costs the
#    operator bandwidth again.
for p in peers:
    conn = nat_traverse(p) or relay_via(TURN_SERVER, p)
    fetch_chunks(conn, missing_chunk_ids)             # bulk data, no server cost

# 4. Integrity cannot be assumed: peers are untrusted, so every chunk is
#    verified against a hash from a trusted source.
assert sha256(chunk) == expected_hashes[chunk_id]

Aspect	Client-server	Peer-to-peer
Capacity	Fixed; operator pays	Grows with peer count
Latency	Predictable	Variable; depends on peer proximity
Trust	Server is authoritative	Peers untrusted; verify everything
Discovery	DNS	Tracker, DHT, or gossip
NAT	Not a problem	A central problem
Consistency	Straightforward	Hard; no single source of truth
Real examples	Nearly all web APIs	BitTorrent, WebRTC media, blockchains, gossip membership

P2P ideas appear inside server-side systems too: gossip protocols spread membership and failure information between nodes of a cluster without a coordinator, which is exactly the P2P trade — no single point of failure, at the cost of eventual rather than immediate consistency.

Key Takeaways

• P2P capacity grows with participants instead of being drained by them.
• Discovery and NAT traversal are the hard parts, which is why hybrid designs dominate.
• Peers are untrusted: verify every chunk cryptographically.
• Gossip inside clusters is P2P thinking applied server-side.

🧪 Practice

1. Explain why a live sports stream to 10 million viewers is a good P2P candidate while a bank ledger is not.
2. Describe how two peers behind separate NATs establish a direct connection, and what happens when it fails.
3. Interview: Design content distribution for a game patch of 50 GB to 5 million players in one hour. (Hint: compute the aggregate bandwidth first, then decide which parts must stay centralized.)

Synchronous vs. Asynchronous Communication

When service A needs something from service B, it can either wait for the answer or not. That single choice determines how failures propagate, how the two services scale, and how hard the system is to reason about — which makes it one of the highest-leverage decisions in any architecture.

Synchronous communication is a phone call: the caller blocks until the callee answers. It is simple, immediate, and easy to debug, because the whole operation has one stack trace and one obvious result. Asynchronous communication is email: the sender hands off a message and continues. The callee processes it whenever it can, and the sender learns the outcome later, if at all.

The critical property is temporal coupling. In a synchronous call, both services must be healthy at the same instant. Availabilities multiply, and slowness propagates upstream: if B takes 5 seconds, A's threads are held for 5 seconds, and A's callers queue behind them. This is how one slow dependency takes down an entire request path.

 SYNCHRONOUS                          ASYNCHRONOUS

  A --request--> B                     A --publish--> [Queue] --deliver--> B
  A  (blocked)                         A continues immediately
  A <-response-- B                     B processes when able; may retry

  Availability: 0.999 x 0.999 = 0.998  A's availability is independent of B
  Latency:      A pays B's latency     A pays only the enqueue (~1 ms)
  Failure:      propagates upstream    absorbed by the queue (until it fills)
  Result:       immediate              eventual; needs a callback/poll/event

# Synchronous: simple, but A's latency and availability now include B's.
def place_order_sync(order, inventory, payments, email):
    inventory.reserve(order)          # 40 ms — must succeed now (correctness)
    payments.charge(order)            # 300 ms — must succeed now (correctness)
    email.send_confirmation(order)    # 900 ms — does NOT need to be now
    return "created"                  # total ~1240 ms; email outage = failed order

# Asynchronous where it is safe: keep only what correctness requires inline.
def place_order_mixed(order, inventory, payments, bus):
    inventory.reserve(order)                    # sync: prevents overselling
    payments.charge(order)                      # sync: user must know it worked
    bus.publish("order.created", order.to_event())   # async: ~1 ms, fire and forget
    return "created"                            # total ~341 ms
    # The email service consumes order.created later. If it is down for an
    # hour, orders still succeed and confirmations arrive when it recovers.

The decision rule is simple to state: make it synchronous when the caller cannot proceed correctly without the result, and asynchronous otherwise. Everything the user does not need before seeing a response — emails, analytics, search indexing, recommendations, webhooks — belongs off the critical path.

Question	Synchronous	Asynchronous
Does the caller need the result to proceed?	Yes	No
Is the operation slow or bursty?	Bad fit	Good fit
Must failures be visible to the user now?	Yes	No
Can work be retried safely later?	Harder	Natural
Debugging difficulty	Low	Higher (distributed traces needed)

Asynchrony is not free: it buys resilience and pays in complexity — eventual consistency, duplicate delivery, message ordering, dead letters, and the need for a way to report failures that nobody is waiting for.

Key Takeaways

• Synchronous calls create temporal coupling: availabilities multiply and slowness propagates upstream.
• Asynchronous messaging decouples availability and absorbs bursts, at the cost of eventual consistency.
• Keep only correctness-critical work synchronous; move the rest off the critical path.
• Async systems need idempotent consumers, dead-letter handling, and tracing.

🧪 Practice

1. For a checkout flow, classify each step as sync or async: inventory reservation, payment, receipt email, loyalty points, fraud scoring, warehouse notification.
2. Compute the end-to-end availability of a synchronous chain of six services at 99.9% each, then explain what a queue between steps 3 and 4 changes.
3. Interview: Making a call async removed a failure mode but introduced a bug where users see stale data. What is happening and how do you handle it? (Hint: the write path and the read path no longer share a clock.)

Request-Response vs. Event-Driven

Two services can be connected in two fundamentally different ways. In request-response, the caller knows who it needs and asks them directly for something specific. In event-driven communication, a producer announces that something happened and does not know or care who reacts.

The difference is not the transport — both can run over HTTP or a queue — it is the direction of knowledge. Request-response is calling a specific colleague to ask a question. Event-driven is posting on a noticeboard: whoever cares subscribes, and adding a new reader requires no change from you.

 REQUEST-RESPONSE (caller knows the callee)

   [Order svc] --"charge this"--> [Payments]
   [Order svc] --"reserve this"-> [Inventory]
   [Order svc] --"send email"---> [Email]
   Adding a consumer = changing Order svc. Order svc knows all 3.

 EVENT-DRIVEN (producer knows nobody)

   [Order svc] --"OrderCreated"--> [ topic ] --+--> [Payments]
                                               +--> [Inventory]
                                               +--> [Email]
                                               +--> [Analytics]  (added later,
                                                                 no producer change)

# Request-response: an imperative command aimed at a known service.
response = payments_client.charge(order_id=42, amount=Decimal("19.99"))
if response.status != "captured":
    raise PaymentFailed(response.reason)     # the caller owns the outcome

# Event-driven: a factual statement about the past, addressed to no one.
bus.publish("order.created", {
    "event_id":   "9f1c...",        # for consumer-side deduplication
    "occurred_at": "2026-03-11T10:04:22Z",
    "order_id":   42,
    "amount":     "19.99",
    "currency":   "EUR",
})
# Note the naming: events are past tense facts ("order.created"), never
# commands ("send_email"). If the producer names the reaction, it has just
# recreated coupling with extra steps.

Dimension	Request-response	Event-driven
Coupling	Caller knows the callee	Producer knows nobody
Adding a consumer	Change the caller	Subscribe; no producer change
Result	Immediate and known	Eventual; unknown to the producer
Error handling	Caller sees and handles it	Consumer retries; producer unaware
Data flow	Pull what you need	Push what happened
Debugging	One trace, one stack	Follow the chain across consumers
Best for	Queries, user-facing commands	Fan-out, integration, audit trails

Most real architectures use both, and the boundary tends to fall in a predictable place: queries and user-facing commands are request-response; notifications of state changes are events. A useful discipline is to keep events as statements of fact that a consumer can interpret independently — the moment an event's payload is shaped for one specific consumer, the coupling has come back.

Key Takeaways

• Request-response couples caller to callee; event-driven decouples producer from consumers entirely.
• Adding a consumer in an event-driven system requires no producer change — that is its main benefit.
• Name events as past-tense facts, never as commands.
• Use request-response for queries and user-facing commands; events for state-change fan-out.

🧪 Practice

1. Rewrite three imperative calls ("send welcome email", "update search index", "grant loyalty points") as a single event plus consumers.
2. Explain how a badly named event (order.send_email) reintroduces coupling.
3. Interview: How do you debug a user report of "I never got my confirmation email" in an event-driven system? (Hint: name the artifacts you need at each hop and what identifier ties them together.)

Short Polling and Long Polling

Servers cannot initiate contact with clients, so any "live" feature has to be faked by the client asking repeatedly. Polling is the simplest way to do that, and understanding its cost is what motivates every push technique that follows.

Short polling asks on a fixed timer: "anything new?" Usually the answer is no, and you have paid for a request, a response, and possibly a TLS handshake to learn nothing. Long polling improves on it by asking the server to hold the request open until there is something to say (or a timeout fires), then immediately reconnecting. Data arrives as soon as it exists, with no wasted round trips in between.

 SHORT POLLING (interval 5 s)          LONG POLLING

 C -->|? no          t=0                C -->|?            t=0
 C -->|? no          t=5                     |   (held open, server waits)
 C -->|? no          t=10                    |
 C -->|YES (data)    t=15  <- up to 5 s      |<-- data      t=7  <- immediate
     wasted: 3 of 4 requests            C -->|?            t=7 (reconnect)
     latency: 0-5 s                          |   (held open)
                                        wasted: none
                                        latency: ~0
                                        cost: one held connection per client

// Short polling: simple, predictable load, but wasteful and laggy.
setInterval(async () => {
  const res = await fetch(`/api/messages?since=${lastSeenId}`);
  const msgs = await res.json();
  if (msgs.length) render(msgs);           // most calls return an empty array
}, 5000);                                   // worst-case staleness: 5 seconds

// Long polling: the server holds the request until data exists or it times out.
async function longPoll() {
  while (true) {
    try {
      // Timeout must be shorter than any proxy/load-balancer idle timeout,
      // otherwise the connection is cut without a response.
      const res = await fetch(`/api/messages?since=${lastSeenId}&wait=25`);
      if (res.status === 200) {
        const msgs = await res.json();
        lastSeenId = msgs.at(-1).id;       // cursor prevents gaps and dupes
        render(msgs);
      }
      // 204 No Content = the wait expired with nothing new; just reconnect.
    } catch (e) {
      await sleep(backoff());               // back off on errors, do not hammer
    }
  }
}

Aspect	Short polling	Long polling
Latency to new data	Up to the interval	Near zero
Wasted requests	Many (usually empty)	Almost none
Server connections	Brief, one per poll	One held open per client
Server memory	Low	Higher; needs async handlers
Infrastructure issues	None	Proxy timeouts, thread exhaustion
Complexity	Trivial	Moderate

Long polling only scales if the server can hold connections cheaply — a thread-per-request server exhausts its pool at a few thousand clients, which is why async runtimes matter here. Both techniques remain useful: short polling for low-frequency status checks (a background job's progress), long polling as a robust fallback where WebSockets are blocked by corporate proxies.

Key Takeaways

• Polling exists because clients must initiate; servers cannot push unprompted.
• Short polling wastes requests and adds up to one interval of staleness.
• Long polling holds the request open for near-zero latency and near-zero waste, but ties up a connection per client.
• Long polling needs async servers and timeouts shorter than every proxy in the path.

🧪 Practice

1. Calculate daily requests for 100,000 clients short polling every 3 seconds, then estimate the fraction that return no data.
2. Explain why a long-poll timeout must be shorter than the load balancer's idle timeout, and what the user sees when it is not.
3. Interview: When would you still choose short polling over WebSockets in 2026? (Hint: think about update frequency, client count, and what infrastructure sits between you and the user.)

Server-Sent Events

Many "live" features only need data to flow one way: stock tickers, live scores, build logs, progress bars, notification feeds, and streamed AI responses. Opening a full bidirectional socket for those is more machinery than the problem requires. Server-Sent Events (SSE) is the minimal answer — a single long-lived HTTP response that the server keeps writing to.

Because it is ordinary HTTP, SSE inherits everything HTTP already has: compression, authentication headers, cookies, proxies, HTTP/2 multiplexing, and observability. The browser API also handles reconnection automatically and replays missed events using an ID the server supplies — a feature you would otherwise have to build yourself.

 SSE: ONE REQUEST, MANY RESPONSES

  Client                                Server
    |--- GET /events  Accept: text/event-stream --->|
    |<-- 200  Content-Type: text/event-stream ------|  (response never ends)
    |<-- data: {"price": 42.10}                     |
    |<-- data: {"price": 42.35}                     |
    |    ...connection drops...                     |
    |--- GET /events  Last-Event-ID: 1043 --------->|  (browser reconnects
    |<-- data: {...}   (resumes from 1044)          |   automatically)

// Client: three lines, with reconnection and resumption built in.
const es = new EventSource("/api/prices");        // GET only; cookies are sent
es.onmessage = (e) => render(JSON.parse(e.data));
es.addEventListener("alert", (e) => toast(e.data));  // named event types
es.onerror = () => { /* browser retries automatically; do not reconnect here */ };

# Server: the wire format is plain text, one field per line, blank line = end.
def price_stream(request):
    def generate():
        last_id = int(request.headers.get("Last-Event-ID", 0))
        for event in price_feed.since(last_id):     # resume where the client left
            yield f"id: {event.id}\n"               # enables Last-Event-ID resume
            yield f"event: price\n"                 # optional named type
            yield f"data: {json.dumps(event.data)}\n\n"   # blank line ends the event
            # Heartbeats keep proxies from closing an idle connection:
            # yield ": keep-alive\n\n"   (a comment line, ignored by the client)
    return Response(generate(), mimetype="text/event-stream",
                    headers={"Cache-Control": "no-cache",
                             "X-Accel-Buffering": "no"})   # disable proxy buffering

Aspect	SSE	WebSockets
Direction	Server to client only	Full duplex
Protocol	Plain HTTP	Upgraded protocol (ws://)
Data format	UTF-8 text only	Text or binary
Reconnection	Automatic, with Last-Event-ID	Manual, you implement it
Proxy/firewall friendliness	High (it is just HTTP)	Sometimes blocked
Connection limit	6 per origin on HTTP/1.1 (fine on HTTP/2)	Not affected
Best for	Feeds, logs, progress, token streaming	Chat, games, collaborative editing

The rule of thumb: if the client only needs to listen, use SSE. It is simpler, survives proxies better, and gives you reconnection for free. Reach for WebSockets when the client also needs to send frequently.

Key Takeaways

• SSE is a single long-lived HTTP response streaming text events server to client.
• Automatic reconnection with Last-Event-ID replay is built into the browser API.
• Being plain HTTP means auth, compression, and proxies just work.
• Choose SSE for one-way streams; choose WebSockets when the client must also send.

🧪 Practice

1. Write the SSE wire format for three events, one with a custom event type and one with an ID.
2. Explain why heartbeats and disabled proxy buffering are necessary in production SSE.
3. Interview: Design the transport for streaming an AI model's tokens to a browser. Justify SSE or WebSockets. (Hint: ask which direction carries the high-frequency traffic.)

WebSockets

Some interactions are genuinely conversational: chat, collaborative editing, multiplayer games, live trading. Both sides send frequently and unprompted, and neither polling nor a one-way stream fits. WebSockets provide a persistent, full-duplex connection over a single TCP connection, where either side can send a message at any time with almost no per-message overhead.

The connection begins as an ordinary HTTP request carrying an Upgrade header. The server agrees with 101 Switching Protocols, and from that point the same TCP connection speaks the WebSocket framing protocol instead of HTTP. Starting as HTTP is what lets WebSockets traverse the existing web infrastructure — ports 80 and 443, proxies, and load balancers.

 UPGRADE HANDSHAKE

  Client                                  Server
    |-- GET /ws HTTP/1.1                     |
    |   Upgrade: websocket                   |
    |   Connection: Upgrade                  |
    |   Sec-WebSocket-Key: dGhlIHNhbXBs...   |
    |--------------------------------------->|
    |<-- HTTP/1.1 101 Switching Protocols ---|
    |    Sec-WebSocket-Accept: s3pPLMBi...   |
    |<========= full duplex frames =========>|
       either side sends, ~2-14 bytes of framing overhead per message

const ws = new WebSocket("wss://api.example.com/ws");   // wss = TLS; always use it

ws.onopen = () => ws.send(JSON.stringify({ type: "subscribe", room: "eng" }));
ws.onmessage = (e) => handle(JSON.parse(e.data));

// The browser does NOT reconnect for you — unlike SSE, this is your job.
ws.onclose = (e) => {
  if (e.code !== 1000) scheduleReconnect();     // 1000 = normal closure
};

// Production checklist the naive example always misses:
//  - exponential backoff WITH jitter, or every client reconnects in lockstep
//    after a deploy and stampedes the server
//  - application-level ping/pong to detect dead connections that TCP has not
//    noticed yet (a silently dropped mobile connection can look "open" for
//    minutes)
//  - re-subscribe and re-authenticate after each reconnect; the server kept
//    no memory of you
//  - bound the outbound buffer: a slow client must be dropped, not allowed to
//    consume unbounded server memory

The architectural cost of WebSockets is statefulness. Each connection is pinned to one server instance for its entire life, which changes several things at once:

Concern	Consequence
Load balancing	Connections are long-lived, so load spreads unevenly after scaling
Deployments	Every deploy disconnects every client; drain and stagger rollouts
Fan-out	A message for a user on another instance needs a pub/sub backplane (Redis, Kafka)
Capacity	Memory per connection (buffers, subscriptions) sets the ceiling — typically tens of thousands per node
Autoscaling	New instances get no traffic until clients reconnect

 SCALING WEBSOCKETS: THE BACKPLANE

  [client A] --- [ws node 1] --\                     /-- [ws node 1] --- [client A]
                                >--- [Redis pub/sub] <
  [client B] --- [ws node 2] --/                     \-- [ws node 2] --- [client B]

  Node 2 receives B's message, publishes to the channel; node 1 delivers to A.
  Without a backplane, users are only reachable if they share an instance.

Key Takeaways

• WebSockets give full-duplex, low-overhead messaging after an HTTP upgrade handshake.
• Connections are stateful and pinned to one instance, complicating balancing, deploys, and autoscaling.
• Reconnection, heartbeats, and re-subscription are your responsibility, not the browser's.
• Cross-instance delivery requires a pub/sub backplane.

🧪 Practice

1. Estimate how many WebSocket connections one server holds given 40 KB of memory per connection and 16 GB of usable RAM.
2. Describe what happens to 500,000 connected clients during a rolling deploy and how to make it non-disruptive.
3. Interview: Design a chat system for 1 million concurrent users. (Hint: start from connections per node, then explain how a message reaches a recipient connected to a different node.)

<a id="24-network-infrastructure"></a>

2.4 Network Infrastructure

Between a client and your application code sits a stack of network machinery that shapes latency, security, and cost; designing it deliberately is as much part of system design as choosing a database.

Proxies and Reverse Proxies

A proxy is a server that sits in the middle of a connection and speaks on someone's behalf. The value is that it gives you a single place to apply policy — caching, authentication, rate limiting, logging, TLS termination — without touching either endpoint. The direction it faces determines whose behalf it acts on, and that single distinction is the whole concept.

A forward proxy sits in front of clients and represents them to the internet. The client knows about it; the destination does not know who the real client was. A reverse proxy sits in front of servers and represents them to the world. The client thinks it is talking to the server; it has no idea how many machines are behind it.

The analogy: a forward proxy is a personal assistant making calls for you. A reverse proxy is a company switchboard — callers dial one number and never learn which desk answered.

 FORWARD PROXY (acts for clients)      REVERSE PROXY (acts for servers)

 [Employee]--\                             /--[App server 1]
 [Employee]---> [Proxy] --> Internet      /
 [Employee]--/                  [Client]--> [Reverse proxy] --[App server 2]
                                                     \
  Use: corporate egress control,                      \--[App server 3]
       content filtering, shared cache,      Use: load balancing, TLS
       hiding client identity                    termination, caching,
                                                 rate limiting, routing

# A reverse proxy configuration doing five jobs at once.
upstream api_backend {
    least_conn;                          # send to the instance with fewest conns
    server 10.0.1.10:8080 max_fails=3 fail_timeout=30s;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=30s;
    keepalive 32;                        # reuse upstream connections (see 2.1)
}

server {
    listen 443 ssl http2;
    ssl_certificate     /etc/ssl/api.crt;   # 1. TLS terminates here
    ssl_certificate_key /etc/ssl/api.key;

    location /static/ {
        proxy_cache static_cache;           # 2. cache: origin never sees these
        proxy_cache_valid 200 10m;
        proxy_pass http://api_backend;
    }

    location /api/ {
        proxy_pass http://api_backend;      # 3. load balance across upstreams
        proxy_set_header X-Real-IP        $remote_addr;
        proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;   # 4. preserve client context
        proxy_read_timeout 30s;             # 5. bound slow upstreams
    }
}

That X-Forwarded-For header matters more than it looks: once a proxy is in the path, the application sees the proxy's IP as the client address. Rate limiting, geolocation, and audit logging all break silently unless the app reads the forwarded header — and trusts it only from known proxies, since clients can forge it.

Capability	Forward proxy	Reverse proxy
Sits in front of	Clients	Servers
Who configures it	The client/network	The service operator
Hides	The client's identity	The server topology
Typical duties	Egress policy, filtering, shared cache	TLS, load balancing, caching, WAF, routing
Client aware of it	Yes	No

Reverse proxies are the foundation of most edge infrastructure: load balancers, API gateways, CDNs, and service mesh sidecars are all specialized reverse proxies.

Key Takeaways

• Forward proxies represent clients; reverse proxies represent servers.
• A reverse proxy centralizes TLS, caching, load balancing, and rate limiting in one place.
• Proxies replace the observed client IP — use X-Forwarded-For, and trust it only from known hops.
• Load balancers, API gateways, CDNs, and mesh sidecars are all reverse proxies.

🧪 Practice

1. Classify each as forward or reverse proxy usage: a corporate web filter, a CDN edge node, a service mesh sidecar handling outbound calls, an API gateway.
2. Explain what breaks in rate limiting and geolocation when the app reads the socket's peer IP behind a proxy.
3. Interview: Why is trusting X-Forwarded-For from any source a security problem, and how do you do it safely? (Hint: think about who can set a header and how many trusted hops are actually in your path.)

Network Address Translation

There are about 4.3 billion IPv4 addresses and far more devices, so the internet's stopgap was to let many private machines share a few public ones. NAT is the mechanism: a device rewrites the source address of outbound packets to its own public address, remembers the mapping, and reverses the rewrite on the way back.

The analogy is an office building with one street address and an internal mail room. Outbound letters all carry the building's address; the mail room keeps a ledger so replies find the right desk.

 SOURCE NAT (outbound)

  [10.0.1.7:52431] --> [NAT gateway] --> [203.0.113.9:41002] --> [server:443]

  Translation table (connection tracking):
   +-------------------+-----------------------+-------------------+
   | internal          | external              | destination       |
   +-------------------+-----------------------+-------------------+
   | 10.0.1.7:52431    | 203.0.113.9:41002     | 93.184.216.34:443 |
   | 10.0.2.4:51002    | 203.0.113.9:41003     | 93.184.216.34:443 |
   +-------------------+-----------------------+-------------------+

  Return traffic to 203.0.113.9:41002 is rewritten back to 10.0.1.7:52431.
  Nothing on the internet can initiate a connection inward: there is no
  table entry until the inside starts one. That is an accidental firewall.

Two directions matter. SNAT (source NAT) is the outbound case above — the default for private subnets reaching the internet. DNAT (destination NAT), also called port forwarding, rewrites the destination so an external address reaches a specific internal host; it is how a load balancer's public IP maps to private backends.

# NAT's hidden capacity limit: it is a connection-tracking table, and the
# port space per (external IP, destination IP, destination port) is finite.
PORTS_PER_EXTERNAL_IP = 65_535 - 1_024        # ~64,511 usable
print(PORTS_PER_EXTERNAL_IP)                   # 64511

# A fleet behind one NAT gateway, all calling the SAME external API endpoint,
# shares that space. (AWS NAT Gateway: ~55,000 concurrent connections per
# unique destination; add more IPs or endpoints to scale past it.)
concurrent_per_instance = 200
instances = 300
print(concurrent_per_instance * instances)     # 60000 -> exceeds the budget

# Symptom: "connection timed out" to one specific third party while everything
# else works. Fixes: connection pooling and keep-alive (fewer, longer-lived
# connections), multiple NAT gateway IPs, or a VPC endpoint that bypasses NAT.

Three consequences worth carrying into designs:

• NAT breaks inbound connections, which is why P2P needs hole punching and why servers live on public subnets or behind load balancers.
• NAT is stateful, so it is a bottleneck and a failure domain: its table has a size limit, its entries have idle timeouts (a long-lived idle connection can be silently dropped), and failing over loses all mappings.
• NAT costs money in the cloud, billed per hour and per gigabyte processed. Routing traffic to cloud services through VPC endpoints instead of a NAT gateway is a common and substantial cost saving.

Key Takeaways

• NAT lets many private hosts share few public addresses by rewriting addresses and ports.
• SNAT handles outbound traffic; DNAT (port forwarding) directs inbound traffic to internal hosts.
• NAT is stateful: table size, idle timeouts, and port exhaustion are real production limits.
• NAT prevents unsolicited inbound connections, and in cloud environments it is a metered, billable component.

🧪 Practice

1. Explain why two devices on the same home network can both browse the web despite sharing one public IP.
2. Your service intermittently fails to reach one third-party API but reaches others fine. Give two NAT-related explanations.
3. Interview: Why can't you initiate a connection to a laptop behind a home router, and how do video-call applications work around it? (Hint: both sides must send outbound first, and something must coordinate the timing.)

Firewalls and Security Groups

Every open port is an invitation. A firewall enforces the principle that nothing should be reachable unless there is a reason for it — default deny, allow by exception. This is the cheapest security control that exists: it costs nothing at run time and eliminates entire attack classes by making vulnerable services unreachable in the first place.

Firewalls filter on the properties they can see at their layer: source and destination address, port, and protocol (layer 3/4), or HTTP semantics like paths, headers, and payloads (layer 7, a web application firewall).

The critical distinction is stateful versus stateless. A stateful firewall remembers connections: if it allowed your outbound request, it automatically allows the reply. A stateless one evaluates every packet in isolation, so you must write rules for both directions — and forgetting the return rule is the classic reason "my rules look right but nothing works".

 LAYERED FILTERING IN A CLOUD NETWORK

  Internet
     |
  [ Network ACL ]   stateless, subnet-wide, allow AND deny rules, ordered
     |              -> broad strokes: block a hostile CIDR entirely
  [ Security group ] stateful, per-instance, ALLOW rules only
     |              -> "web tier may receive 443 from anywhere"
  [ Host firewall ]  stateful, per-OS (iptables/nftables)
     |              -> defense in depth on the machine itself
  [ Application ]    authn/authz — the last and most important gate

# Stateful (iptables): one rule for established traffic covers all replies.
iptables -P INPUT DROP                                    # default deny
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT  # replies OK
iptables -A INPUT -p tcp --dport 443 -j ACCEPT            # public HTTPS
iptables -A INPUT -p tcp --dport 22 -s 10.0.0.0/8 -j ACCEPT  # SSH from VPC only

# Cloud security groups: reference other groups, not IP ranges. The app tier
# is reachable only from the web tier, whatever its addresses become.
resource "aws_security_group" "app" {
  name = "app-tier"

  ingress {
    from_port       = 8080
    to_port         = 8080
    protocol        = "tcp"
    security_groups = [aws_security_group.web.id]   # identity, not addresses
  }

  egress {
    from_port       = 5432
    to_port         = 5432
    protocol        = "tcp"
    security_groups = [aws_security_group.db.id]    # restrict egress too:
  }                                                 # limits data exfiltration
}

Control	Layer	Stateful	Scope	Rules
Network ACL	3/4	No	Subnet	Allow and deny, ordered
Security group	3/4	Yes	Instance/ENI	Allow only
Host firewall	3/4	Yes	One machine	Allow and deny
Web application firewall	7	Yes	Application	Pattern-based (SQLi, XSS, bots)

Referencing security groups by identity rather than by IP range is the single most valuable habit here: it survives autoscaling, instance replacement, and address changes, and it expresses the intent ("the app tier trusts the web tier") rather than an accident of addressing.

Key Takeaways

• Default deny, allow by exception; unreachable services cannot be exploited.
• Stateful firewalls auto-allow replies; stateless ones need explicit rules in both directions.
• Layer controls: subnet ACLs, instance security groups, host firewalls, then application authorization.
• Reference security groups by identity rather than IP so rules survive scaling.

🧪 Practice

1. Write the security group rules for a three-tier app (web, app, database) where each tier only accepts traffic from the one above it.
2. Explain why a stateless ACL that allows inbound 443 still breaks HTTPS unless you add another rule, and what that rule is.
3. Interview: Why restrict outbound traffic when the threat is inbound? (Hint: think about what an attacker does after gaining a foothold.)

Virtual Private Cloud Design

A VPC is your own logically isolated network inside a shared cloud. It exists so that "which machines can talk to which" is a decision you make explicitly, rather than an accident of who happens to know an IP address. Nearly every security and connectivity property of a cloud system traces back to the VPC layout.

The core structure is simple: a VPC owns a CIDR block, and it is divided into subnets, each living in exactly one availability zone. A subnet's route table determines whether it is public (a route to an internet gateway) or private (no such route; outbound traffic goes through a NAT gateway instead).

 STANDARD THREE-TIER VPC (10.0.0.0/16), TWO AZs

  +----------------------- VPC 10.0.0.0/16 ------------------------+
  |                                                                |
  |   AZ-a                            AZ-b                         |
  |  +---------------------+        +---------------------+        |
  |  | public  10.0.0.0/24 |        | public  10.0.1.0/24 |        |
  |  |  [ALB] [NAT gw]     |        |  [ALB] [NAT gw]     | <- IGW route
  |  +---------------------+        +---------------------+        |
  |  | private 10.0.10.0/24|        | private 10.0.11.0/24|        |
  |  |  [app] [app]        |        |  [app] [app]        | <- NAT route
  |  +---------------------+        +---------------------+        |
  |  | data    10.0.20.0/24|        | data    10.0.21.0/24|        |
  |  |  [db primary]       |        |  [db standby]       | <- no internet
  |  +---------------------+        +---------------------+        |
  +----------------------------------------------------------------+

  Only the public subnets are reachable from the internet. Everything that
  holds data sits two layers deep with no route out at all.

# Deriving a VPC plan from capacity estimates, not from habit.
import ipaddress

vpc = ipaddress.ip_network("10.20.0.0/16")     # one /16 per region+environment
tiers = {"public": 24, "private": 22, "data": 24}   # private is largest: most pods
azs = ("a", "b", "c")

# Reserve a distinct /20 slice per AZ so tiers can grow without collisions.
for az, block in zip(azs, vpc.subnets(new_prefix=20)):
    print(az, block)
# a 10.20.0.0/20   b 10.20.16.0/20   c 10.20.32.0/20
# Half the /16 is deliberately left unallocated for future tiers and
# for workloads (like Kubernetes pods) that consume addresses fast.

The decisions that matter most, in order of how expensive they are to reverse:

Decision	Why it is hard to change later
VPC CIDR range	Overlaps block all future peering; re-addressing is an outage-level migration
Subnet sizing per tier	Growing a subnet is not possible; you add new ones and rebalance
Public vs. private placement	Moving a database to a private subnet changes every connection path
Cross-VPC connectivity	Peering does not transit; a hub-and-spoke (transit gateway) retrofit touches every route table

Two patterns are worth knowing by name. VPC endpoints let instances in a private subnet reach cloud services (object storage, queues) over the provider's internal network — no NAT gateway, no internet exposure, lower cost. And hub-and-spoke connectivity (a transit gateway) replaces the mesh of pairwise peerings that becomes unmanageable past a handful of VPCs, because peering connections are not transitive.

Key Takeaways

• A VPC is an isolated network; subnets are AZ-scoped and public or private by route table, not by name.
• Put anything holding data in private subnets with no route to the internet.
• CIDR planning is effectively irreversible — allocate generously and never overlap.
• VPC endpoints avoid NAT costs and exposure; transit gateways replace non-transitive peering meshes.

🧪 Practice

1. Design a VPC for a two-region, three-AZ deployment with public, private, and data tiers, and state every CIDR.
2. Explain the traffic path for an instance in a private subnet calling an external payment API, naming every component it traverses.
3. Interview: Your database is in a public subnet with a security group allowing only the app tier. Is that acceptable? (Hint: consider defense in depth and what a single misconfigured rule would then expose.)

Regions and Availability Zones

Cloud providers give you two units of geography, and confusing them is a common and expensive mistake. An availability zone (AZ) is one or more datacenters with independent power, cooling, and networking, close enough to its siblings for sub-millisecond links. A region is a cluster of AZs in one geographic area, far from other regions.

The distinction exists because two different problems need two different solutions. AZs protect against facility failure — a fire, a flood, a power loss — while keeping latency low enough that synchronous replication is practical. Regions protect against regional disaster and put data close to users, but they are far enough apart that synchronous coordination is painful.

 GEOGRAPHY AND LATENCY

  REGION eu-west-1                        REGION us-east-1
  +---------------------------+           +---------------------------+
  | AZ-a  <--0.5-2 ms-->  AZ-b|           | AZ-a   AZ-b   AZ-c        |
  |   \                    /  |           |                           |
  |    \--- AZ-c ---------/   |           |                           |
  +---------------------------+           +---------------------------+
              \_________________ ~70-80 ms ________________/

  Within an AZ:      ~0.1-0.5 ms   -> anything goes
  Across AZs:        ~0.5-2 ms     -> synchronous replication is fine
  Across regions:    ~30-250 ms    -> synchronous writes are painful
                                      (every commit pays a full round trip)

# The multi-AZ vs multi-region decision expressed as consequences.
TOPOLOGIES = {
    "single-az":     {"survives": "instance failure",
                      "availability": 0.99,   "extra_cost": 1.0, "write_ms": 2},
    "multi-az":      {"survives": "datacenter failure",
                      "availability": 0.9995, "extra_cost": 2.0, "write_ms": 4},
    "multi-region-active-passive": {"survives": "regional disaster",
                      "availability": 0.9999, "extra_cost": 2.4, "write_ms": 4},
    "multi-region-active-active":  {"survives": "regional disaster + serves locally",
                      "availability": 0.99999,"extra_cost": 3.0, "write_ms": 80},
}
for name, t in TOPOLOGIES.items():
    print(f"{name:28} {t['availability']:<8} {t['extra_cost']}x  writes {t['write_ms']}ms")

# The pattern: multi-AZ is cheap insurance almost everyone should buy.
# Multi-region is expensive and forces a consistency decision — either
# asynchronous replication (accept data loss on failover, an RPO > 0) or
# synchronous writes (accept cross-region latency on every commit).

Concern	Multi-AZ	Multi-region
Protects against	Datacenter loss	Regional disaster, geographic latency
Replication	Synchronous is practical	Usually asynchronous (RPO > 0)
Failover complexity	Low; often automatic	High; DNS/global routing, data reconciliation
Data transfer cost	Small per-GB charge	Significantly higher
Compliance angle	Neutral	Enables data residency requirements
When to adopt	Almost always	When downtime cost or user geography justifies it

Two practical notes. Spread across at least three AZs when quorum systems are involved, because a quorum of three survives losing one while two cannot. And cross-AZ traffic is billed per gigabyte — a chatty service accidentally talking across AZs on every request produces a surprising bill and extra latency, which is why AZ-aware routing exists.

Key Takeaways

• AZs are independent facilities within a region, ~0.5-2 ms apart; cross-region is tens to hundreds of ms.
• Multi-AZ is affordable protection against datacenter failure and supports synchronous replication.
• Multi-region protects against regional disaster and serves users locally, at high complexity and cost.
• Use three AZs for quorum systems, and watch cross-AZ data transfer charges.

🧪 Practice

1. Explain why a quorum-based database needs three AZs rather than two.
2. A synchronous write requires acknowledgement from a replica 80 ms away. What is the maximum sequential write throughput per connection?
3. Interview: When is multi-region worth its cost? (Hint: quantify downtime cost per hour and user-perceived latency, and compare with the added engineering and infrastructure spend.)

Edge Networks

Physics is the one constraint no code can optimize away: a round trip to a server 10,000 km distant cannot beat roughly 100 ms. If your users are global and your origin is in one region, most of their latency is spent in transit. Edge networks solve this by putting servers close to users — hundreds of points of presence (PoPs) worldwide that terminate connections, serve cached content, and forward what they cannot answer.

The gain is larger than "one round trip saved", because connection setup is multi-round-trip. Terminating TCP and TLS at a PoP 10 ms away instead of an origin 150 ms away turns a ~450 ms setup into ~30 ms, and the PoP holds a warm, pre-established connection to the origin.

 WITHOUT EDGE                        WITH EDGE

 [User in Sydney]                    [User in Sydney]
     |  150 ms RTT                       |  10 ms RTT to Sydney PoP
     |  x3 (TCP + TLS + request)         |  x3 = 30 ms
     v                                   v
 [Origin in Virginia]               [Edge PoP] --warm, pooled connection-->
                                              [Origin in Virginia]
  ~450 ms to first byte              cache hit:  ~10 ms
                                     cache miss: ~180 ms (still faster: the
                                     handshakes are already done)

Anycast is what makes this work without client-side logic: the same IP address is announced from every PoP, and internet routing delivers each user to the topologically nearest one automatically.

# What the edge caches is controlled entirely by origin response headers.
HTTP/1.1 200 OK
Cache-Control: public, max-age=60, s-maxage=86400, stale-while-revalidate=600
#              ^public  ^browser  ^shared cache   ^serve stale while refreshing
#                        60s       24h at the edge
Vary: Accept-Encoding
#     ^ cache separate variants; a careless Vary (on User-Agent) destroys hit rate
ETag: "a3f9c2"
#     ^ enables cheap revalidation: the origin can answer 304 Not Modified

// Edge functions run the same request-handling code at every PoP, which lets
// personalization and routing happen without a trip to the origin.
export default async function handler(request) {
  const country = request.headers.get("cf-ipcountry") ?? "US";   // set by the PoP

  // Redirect at the edge: ~10 ms instead of a 150 ms round trip to origin.
  if (country === "DE" && !request.url.includes("/de/")) {
    return Response.redirect(new URL("/de/", request.url), 302);
  }

  // A/B assignment at the edge keeps the cached page identical for everyone
  // in a bucket, preserving the cache hit rate.
  const bucket = hash(request.headers.get("cookie")) % 2 ? "a" : "b";
  return fetch(originURL, { headers: { "x-bucket": bucket } });
}

Edge capability	What it removes
Static caching	Origin requests entirely for hot assets
TLS termination	Multiple long round trips from the setup path
Connection pooling to origin	Handshake cost on every cache miss
DDoS absorption	Attack traffic, filtered before it reaches origin
Edge compute	Round trips for redirects, auth checks, A/B splits
Image/video optimization	Bytes transferred to the client

The main design constraint is cache correctness: anything user-specific must either not be cached or be varied on the right key. The typical pattern is to cache the shell aggressively at the edge and fetch personalized fragments separately — which keeps hit rates high without ever serving one user's data to another.

Key Takeaways

• Edge PoPs cut latency by terminating connections physically near users.
• The saving is multiplied because handshakes cost several round trips.
• Anycast routes users to the nearest PoP with no client-side logic.
• Cache behaviour is driven by origin headers; a careless Vary or an uncached personalized response destroys the benefit.

🧪 Practice

1. Compute time-to-first-byte for a user 150 ms from origin versus 10 ms from a PoP, counting TCP and TLS 1.3 handshakes in both cases.
2. Write Cache-Control headers for: a hashed JS bundle, a news homepage, a logged-in user's dashboard.
3. Interview: How would you serve personalized content without destroying the edge cache hit rate? (Hint: split what varies from what does not, and consider what can be decided at the edge itself.)

Chapter Summary

Concept	The one-line version
Four resources	CPU, memory, disk, network; whichever saturates first is your capacity
Latency ladder	Memory ~100 ns, SSD ~16 us, in-DC RTT ~0.5 ms, cross-ocean ~250 ms
Sequential vs. random	Sequential wins by ~200x on HDD, ~6-8x on SSD; storage engines exploit it
Platforms	Bare metal, VM, container trade isolation against density and boot speed
Vertical limits	Descriptors, ephemeral ports, threads, and pools bite before CPU does
Layering	Each layer wraps the one above; a device sees only its own layer
TCP vs. UDP	Reliability and ordering with head-of-line blocking, versus raw speed
CIDR	Prefix length sets block size; overlapping ranges can never be peered
DNS	Hierarchical and cached; TTL trades failover speed against latency
TLS	Confidentiality, integrity, authentication; 1 RTT in TLS 1.3
HTTP versions	Same semantics; H2 removes app-level and H3 removes transport-level HOL blocking
Sync vs. async	Sync multiplies availabilities and propagates latency; async decouples both
Push techniques	Short poll, long poll, SSE (one-way), WebSockets (duplex, stateful)
Proxies	Forward proxies act for clients; reverse proxies act for servers
NAT	Address sharing with a stateful table — a real capacity and cost limit
Firewalls	Default deny; stateful groups referenced by identity, layered in depth
VPC	Subnets are public or private by route table; CIDR choices are permanent
AZs and regions	AZs are ms apart (sync replication); regions are tens to hundreds of ms
Edge	Terminate near the user; the win is several round trips, not just one

---

<a id="3-apis-and-service-interfaces"></a>