13. Modern Network Infrastructure
The networks of the previous chapters were built from boxes that each made their own decisions, configured by hand, carrying traffic between physical machines whose addresses matched their location. Virtualisation, cloud computing, and container orchestration broke all three assumptions at once: endpoints now appear and vanish in seconds, tenants share one physical fabric while requiring complete isolation, and no human can keep pace with the rate of change. This chapter covers the architectural responses — separating control from forwarding (SDN), decoupling logical topology from physical wiring (overlays and network function virtualisation), the fabric designs that serve machine-to-machine traffic (leaf-spine, ECMP, RDMA), the cloud and wide-area constructs built on top (virtual private clouds, SD-WAN, edge), and the automation without which none of it is operable.
<a id="131-software-defined-networking"></a>
13.1 Software-Defined Networking
Software-defined networking begins from one observation: a router does two completely different jobs at rates that differ by a factor of a billion, and bundling them into one closed appliance is a choice rather than a necessity — this subchapter covers what happens when you separate them, who takes over the thinking, and how far the programmability now extends.
Separation of Control and Data Planes
Theory
Open any traditional router and you find two subsystems sharing a chassis. One runs routing protocols, processes topology changes, and computes where traffic should go. The other takes a packet, looks up a destination, and sends it out a port. They are bundled together because that is how networking equipment evolved, not because the jobs resemble each other. In fact they resemble each other almost not at all.
The control plane thinks. It needs global knowledge (the whole topology), general-purpose computation (Dijkstra, policy evaluation, BGP path selection), and it runs when something changes — a link fails, an operator edits a policy, a peer withdraws a route. A stable network might see ten such events an hour. Milliseconds of computation per event are entirely acceptable.
The data plane does not think. It needs no knowledge beyond a lookup table, performs one fixed operation, and runs on every packet. A single 100 Gb/s port must decide 8.3 million times per second, giving a budget of about 120 nanoseconds per packet. Nothing general-purpose can do that; it requires fixed-function silicon with a table in fast memory.
That frequency gap is roughly a factor of three billion, and it is the entire argument for separation. Once you notice that the fast part needs no intelligence and the intelligent part needs no speed, the question becomes: why does every box need its own copy of the intelligence?
The analogy is a taxi fleet. Traditionally every driver navigates independently, each with their own map, each deciding routes from their own partial view, and the fleet's behaviour emerges from thousands of local decisions that mostly agree. A dispatcher with a live view of the whole city can instead compute better routes centrally and simply tell each driver which way to turn. The drivers still do the driving — that part cannot be centralised — but the thinking moves to where the complete picture is.
SDN applies exactly this move: extract the control plane from every device, run it as software on servers with a global view, and leave the devices as programmable forwarding engines that take instructions. Three properties follow, and they are the actual benefits:
| Property | What it means | Why it was impossible before |
|---|---|---|
| Global view | one entity knows the entire topology and all policy | each device saw only its neighbours' advertisements |
| Direct programmability | forwarding state is written by an external program | state was a side effect of distributed protocol convergence |
| Decoupled evolution | new behaviour ships as software, not hardware | behaviour was fixed in the vendor's firmware |
The global view is the one that changes what is possible. A distributed routing protocol converges toward a consistent answer, but no participant can express a goal like "place these flows so that no link exceeds 70%", because that is a global optimisation and each router sees only locally. A controller with the full topology can solve it directly — which is why the traffic engineering of Section 12.3 looks so much more natural in an SDN framing than as a distributed protocol.
It is important to be precise about what is centralised. The decision is centralised; the forwarding is not and cannot be, because packets must physically traverse devices. It is also a mistake to think SDN means "one controller somewhere". It means the control plane is logically centralised, which in practice is a replicated cluster with a consistent state store, because a genuinely single controller would be both a bottleneck and a single point of failure.
The trade-offs are real and worth naming up front: the controller becomes a critical dependency, the control channel becomes a network that itself needs to work, and a network that was resilient through distributed autonomy now has a coordination layer that can fail in new and more global ways.
Examples
The same two jobs, before and after the split
TRADITIONAL: every box carries both, and they must agree by protocol
+---------------------+ +---------------------+ +---------------------+
| control: OSPF/BGP |<->| control: OSPF/BGP |<->| control: OSPF/BGP |
| local view only | | local view only | | local view only |
+---------------------+ +---------------------+ +---------------------+
| data: FIB lookup | | data: FIB lookup | | data: FIB lookup |
+---------------------+ +---------------------+ +---------------------+
^ N copies of the intelligence, converging by gossip
SDN: one logically central control plane, N dumb-but-fast forwarders
+-------------------------------+
| CONTROLLER (replicated) |
| global topology + policy |
+-------------------------------+
| | | southbound
v v v (OpenFlow, gNMI, P4Runtime)
+-----------+ +-----------+ +-----------+
| data plane| | data plane| | data plane|
| flow table| | flow table| | flow table|
+-----------+ +-----------+ +-----------+
What did NOT move: packets still physically cross each device.
What DID move: the decision about where they go.
import time, random, heapq, ipaddress
# ---------- CONTROL PLANE: global knowledge, runs on EVENTS, may be slow ----------
random.seed(1)
NODES = [f"R{i}" for i in range(200)]
TOPO = {n: {} for n in NODES}
for i, n in enumerate(NODES): # a connected random graph
for m in random.sample(NODES, 4) + [NODES[(i + 1) % len(NODES)]]:
if m != n:
c = random.randint(1, 20)
TOPO[n][m] = c; TOPO[m][n] = c
def compute_fib(topo, me):
"""Dijkstra + next-hop extraction. Essentially all a routing protocol does."""
dist, prev, pq = {me: 0}, {}, [(0, me)]
while pq:
d, u = heapq.heappop(pq)
if d > dist.get(u, 1e18):
continue
for v, c in topo[u].items():
if d + c < dist.get(v, 1e18):
dist[v], prev[v] = d + c, u
heapq.heappush(pq, (d + c, v))
fib = {}
for dst in topo:
if dst == me or dst not in dist:
continue
hop = dst
while prev.get(hop) != me: # walk back to my first hop
hop = prev[hop]
fib[dst] = hop
return fib
# ---------- DATA PLANE: local only, runs on EVERY PACKET, must be fast ----------
ROUTES = sorted(((ipaddress.ip_network(p), nh) for p, nh in
[("10.1.0.0/16", "R2"), ("10.1.5.0/24", "R3"),
("10.0.0.0/8", "R2"), ("0.0.0.0/0", "R4")]),
key=lambda e: -e[0].prefixlen) # longest prefix first
def forward(dst_ip):
"""No topology knowledge, no negotiation, no memory of previous packets."""
a = ipaddress.ip_address(dst_ip)
for net, nh in ROUTES:
if a in net:
return nh
return None
fib = compute_fib(TOPO, "R0")
print(f"control plane: computed {len(fib)} next hops for R0 "
f"from a {len(NODES)}-node topology")
for ip in ("10.1.5.9", "10.1.9.9", "10.9.0.1", "8.8.8.8"):
print(f" data plane: {ip:12s} -> {forward(ip)}")
# The asymmetry is about FREQUENCY, not cost per call.
t0 = time.perf_counter(); compute_fib(TOPO, "R0")
control_ms = (time.perf_counter() - t0) * 1000
TOPOLOGY_EVENTS_PER_HOUR = 10 # generous for a stable network
PPS_PER_100G_PORT = 100e9 / (1500 * 8)
print(f"\ncontrol plane: {control_ms:7.2f} ms per run, ~{TOPOLOGY_EVENTS_PER_HOUR}/hour"
f" -> {TOPOLOGY_EVENTS_PER_HOUR/3600:.4f} invocations/s")
print(f"data plane : must decide {PPS_PER_100G_PORT/1e6:7.2f} M times/s per port"
f" -> {1e9/PPS_PER_100G_PORT:.0f} ns budget per packet")
print(f"frequency ratio: {PPS_PER_100G_PORT/(TOPOLOGY_EVENTS_PER_HOUR/3600):.2e}")
# control plane: computed 199 next hops for R0 from a 200-node topology
# data plane: 10.1.5.9 -> R3
# data plane: 10.1.9.9 -> R2
# data plane: 10.9.0.1 -> R2
# data plane: 8.8.8.8 -> R4
#
# control plane: 0.53 ms per run, ~10/hour -> 0.0028 invocations/s
# data plane : must decide 8.33 M times/s per port -> 120 ns budget per packet
# frequency ratio: 3.00e+09
#
# Two workloads in one chassis, three BILLION times apart in rate. One wants a
# CPU and libraries; the other wants an ASIC and a fixed-size table. That gap is
# the whole argument for separating them.
# Seeing the two planes separately on a Linux router.
# Control plane: the routing daemon's view, computed from protocol state.
vtysh -c 'show ip route ospf'
# O>* 10.2.0.0/24 [110/20] via 10.0.0.2, eth1, weight 1, 00:14:22
# ^ RIB: what the control plane DECIDED, with protocol and metric
# Data plane: what is actually programmed for forwarding.
ip route show table main
# 10.2.0.0/24 via 10.0.0.2 dev eth1
# ^ FIB: no metric, no protocol, no history -- just the lookup result
# The hardware FIB is a third copy on a real switch, and the three can diverge.
# A classic outage: control plane correct, hardware table stale or full.
ethtool -S swp1 | grep -i 'drop\|discard'
vtysh -c 'show ip route summary' # compare route COUNT against ASIC capacity
Key takeaways
- A router performs two jobs whose invocation rates differ by roughly a factor of three billion: the control plane runs on topology events, the data plane on every packet.
- The control plane thinks and needs global knowledge and general computation; the data plane needs only a table and 120 ns per packet at 100 Gb/s.
- SDN extracts the control plane into external software, giving a global view, direct programmability, and independent evolution of software from hardware.
- The global view enables goals no distributed protocol can express, such as "no link above 70%", because that is a global optimisation.
- Only the decision is centralised; forwarding necessarily stays distributed across devices.
- "Centralised" means logically centralised: in practice a replicated cluster, since a single controller is both a bottleneck and a single point of failure.
- The costs are a new critical dependency, a control channel that must itself work, and failures that can be more global than in an autonomous design.
🧪 Practice
- Classify each as control plane or data plane: decrementing TTL, running the BGP best-path algorithm, matching a destination against a longest-prefix table, processing an OSPF link-state advertisement, incrementing an interface counter.
- Compute the per-packet time budget for a 400 Gb/s port at 1500-byte packets and again at 64-byte packets, and explain which case constrains hardware design.
- Describe two failure modes that exist in an SDN network but not in a traditional one, and one that exists in a traditional network but not in an SDN one.
- Interview-style: A colleague says SDN means replacing all routers with one big central router. (Hint: separate the words "decision" and "forwarding", then ask what physically has to happen to a packet travelling between two cities.)
SDN Controller Architecture
Theory
Once the control plane leaves the devices, something has to hold it, and that something turns out to be a distributed system with all the usual distributed-system problems. A controller is not "a program that programs switches"; it is a platform with a specific internal structure.
Working upward from the wire, a controller has four layers:
- Southbound drivers speak the device protocols: OpenFlow, NETCONF, gNMI, P4Runtime, or vendor APIs. Each is a different dialect for "install this state, report that counter".
- A device and topology store maintains the authoritative model of what exists: switches, ports, links (usually discovered by injecting LLDP-style probes), hosts, and their current status.
- Core services provide the functions applications need: path computation, flow-rule installation with conflict resolution, statistics collection, and event notification.
- Northbound APIs expose all of the above to applications, usually as REST or gRPC. The applications — a firewall, a load balancer, a tenant-isolation service — are where the actual network behaviour is written.
The first genuine design decision is reactive versus proactive flow programming. In the reactive model, a switch with no matching rule sends the packet to the controller, which decides and installs a rule. It is elegant and gives the controller perfect visibility, and it has two serious costs: every new flow pays a controller round trip before its first packet moves, and the controller's load scales with the flow arrival rate of the entire network. In the proactive model, rules are installed before traffic arrives, so the data path is never blocked and the controller's load scales with the rate of policy change instead. Real designs are overwhelmingly proactive, with reactive handling reserved for the small set of events that genuinely need a decision.
The second decision is what happens when the controller is unreachable, and it must be decided explicitly because the defaults differ and all of them are wrong for somebody:
| Mode | Behaviour on controller loss | Trade |
|---|---|---|
| fail-secure | keep existing flows, drop table-misses | policy holds; new flows fail |
| fail-standalone | fall back to legacy L2/L3 learning | network keeps working; policy drifts |
| fail-open | forward everything | availability preserved; policy gone |
The third is state consistency across the cluster. A controller cluster needs a replicated store, and the choice between strong and eventual consistency has direct operational consequences. Strong consistency (Raft, as used by ONOS and OpenDaylight for critical state) means all instances agree before acting, requires a quorum, and stops accepting writes when the quorum is lost. Eventual consistency scales better and allows two instances to briefly install contradictory rules. The usual resolution is to split state by criticality: strongly consistent for forwarding and policy, eventually consistent for statistics and topology liveness.
A useful way to hold all this: the controller is the network's operating system. It abstracts heterogeneous hardware behind a uniform interface, arbitrates between applications that want conflicting things, and exposes a programming model. And as with an operating system, most of the engineering difficulty is not the abstraction itself but the arbitration — two applications both installing rules for the same traffic is the SDN equivalent of two processes writing the same file.
Examples
Controller internals, and the two directions of traffic through them
APPLICATIONS [ tenant isolation ] [ firewall ] [ TE / path opt ]
| | |
northbound API ======+=================+==============+====== REST / gRPC
v
+-----------------------------------------------------------+
| CORE SERVICES |
| path computation | flow-rule manager (conflict resolve) |
| statistics | event bus | intent framework |
+-----------------------------------------------------------+
| DEVICE + TOPOLOGY STORE (replicated: Raft / gossip) |
+-----------------------------------------------------------+
| SOUTHBOUND DRIVERS OpenFlow | NETCONF | gNMI | P4Runtime|
+-----------------------------------------------------------+
|
southbound ==========+============================== to devices
v
[ switch ] [ switch ] [ switch ]
Reactive vs proactive, on the wire:
REACTIVE PROACTIVE
pkt -> switch (no match) controller pre-installs rules
-> PACKET_IN to controller pkt -> switch (match) -> out
-> controller decides ^ zero controller involvement
-> FLOW_MOD installs rule
-> PACKET_OUT / resend
first packet delayed ~2.5 ms first packet delayed 0 ms
load scales with FLOW arrivals load scales with POLICY changes
# Reactive vs proactive flow programming: the decision that sizes a controller.
RTT_TO_CONTROLLER_MS = 2.0
CONTROLLER_DECISION_MS = 0.5
NEW_FLOWS_PER_SEC = 50_000 # a mid-size data centre
CONTROLLER_CAPACITY_PPS = 30_000 # decisions per second per controller instance
def reactive(new_flows_ps, capacity):
setup_ms = RTT_TO_CONTROLLER_MS + CONTROLLER_DECISION_MS
instances = -(-new_flows_ps // capacity) # ceiling division
return setup_ms, new_flows_ps, instances
def proactive(new_flows_ps, capacity):
return 0.0, 0, 1 # rules pre-installed: no per-flow work
for name, fn in (("reactive (table-miss -> controller)", reactive),
("proactive (pre-installed rules)", proactive)):
setup, load, inst = fn(NEW_FLOWS_PER_SEC, CONTROLLER_CAPACITY_PPS)
print(f"{name:36s} first-packet delay {setup:4.1f} ms "
f"controller load {load:>6,} dec/s instances {inst}")
# reactive (table-miss -> controller) first-packet delay 2.5 ms controller load 50,000 dec/s instances 2
# proactive (pre-installed rules) first-packet delay 0.0 ms controller load 0 dec/s instances 1
#
# 2.5 ms on the first packet is most of a 3-packet DNS exchange and noise for a
# 1 GB transfer. Hence: proactive for the common case, reactive only for what
# genuinely needs a decision.
# Failure modes: what a switch does when the controller is unreachable.
MODES = {
"fail-standalone": "fall back to legacy L2/L3 learning; network works, policy drifts",
"fail-secure": "keep existing flows, drop table-misses; policy holds, new flows fail",
"fail-open": "forward everything; availability preserved, policy GONE",
}
for m, d in MODES.items():
print(f" {m:16s} {d}")
# Cluster sizing: a strongly consistent store needs a quorum to accept writes.
def quorum(n):
return n // 2 + 1
print("\nRaft-style controller cluster:")
for n in (1, 3, 5, 7):
print(f" {n} nodes -> quorum {quorum(n)}, tolerates {n - quorum(n)} failures"
f"{' <- no redundancy at all' if n == 1 else ''}")
# 1 nodes -> quorum 1, tolerates 0 failures <- no redundancy at all
# 3 nodes -> quorum 2, tolerates 1 failures
# 5 nodes -> quorum 3, tolerates 2 failures
# 7 nodes -> quorum 4, tolerates 3 failures
# Even node counts buy nothing: 4 nodes also tolerate only 1 failure, so
# controller clusters are always odd-sized.
# Talking to a controller through its northbound API (ONOS-style REST).
# What does the controller believe exists? Start every investigation here.
curl -su onos:rocks http://ctrl:8181/onos/v1/devices | jq '.devices[] |
{id, available, role}'
# { "id": "of:0000000000000001", "available": true, "role": "MASTER" }
# { "id": "of:0000000000000002", "available": false, "role": "NONE" }
# ^ device down, or control channel down?
# Links are DISCOVERED, not configured -- a missing link is usually a probe problem.
curl -su onos:rocks http://ctrl:8181/onos/v1/links | jq '.links | length'
# Which application installed a given rule? The usual first question in an incident.
curl -su onos:rocks http://ctrl:8181/onos/v1/flows/of:0000000000000001 \
| jq '.flows[] | {appId, priority, selector: .selector.criteria, treatment:
.treatment.instructions}'
# Cluster health: without a quorum the controller cannot accept writes.
curl -su onos:rocks http://ctrl:8181/onos/v1/cluster | jq '.nodes[] | {id, status}'
# On the switch side, verify the control channel and the configured failure mode.
ovs-vsctl show | grep -A3 Controller
# Controller "tcp:10.0.0.1:6653"
# is_connected: true
ovs-vsctl get bridge br0 fail_mode
# secure <- fail-secure: no legacy forwarding if the controller is lost
# standalone <- fall back to normal L2 learning
Key takeaways
- A controller has four layers: southbound drivers, a device and topology store, core services, and a northbound API that applications program against.
- Reactive programming punts table-misses to the controller, adding a round trip to every new flow and scaling controller load with flow arrivals. Proactive pre-installs rules and scales with policy change instead. Real designs are proactive by default.
- Choose the controller-loss behaviour explicitly: fail-secure (policy holds, new flows fail), fail-standalone (network works, policy drifts), or fail-open (policy gone).
- Cluster state needs a replicated store; use strong consistency for forwarding and policy, eventual consistency for statistics and liveness.
- Quorum arithmetic makes controller clusters odd-sized: 3 nodes tolerate 1 failure, 5 tolerate 2, and 4 tolerate no more than 3 does.
- Think of the controller as the network's operating system; its hardest problem is arbitration between applications that want conflicting forwarding state.
🧪 Practice
- A network sees 120,000 new flows per second and each controller instance can decide 40,000 per second. Compute the instance count for a fully reactive design, and state what the proactive equivalent requires.
- For a hospital network and for a public Wi-Fi network, choose fail-secure or fail-standalone and justify each choice in one sentence.
- Explain why a 4-node controller cluster provides no more availability than a 3-node one, and what it does cost.
- Interview-style: Two SDN applications both install rules matching the same traffic with the same priority, and behaviour becomes non-deterministic. (Hint: this is an arbitration problem rather than a protocol problem — think about what the controller layer between applications and devices ought to be doing, and what an operating system does with two processes opening the same file.)
OpenFlow and Flow Tables
Theory
OpenFlow was the protocol that made SDN concrete, and its central abstraction is worth understanding even though its dominance has faded: it replaced the specialised tables of a traditional switch with one general table shape that can express all of them.
A traditional switch has a MAC table, an IP routing table, an ACL table, and a VLAN table, each with fixed semantics baked into the silicon and firmware. OpenFlow's insight is that all four are the same thing — match some header fields, take some actions, count what you did — and that a single generalised table can therefore subsume them. A flow entry has five parts:
| Part | Purpose |
|---|---|
| Match fields | which packets this entry applies to; unspecified fields are wildcards |
| Priority | which entry wins when several match |
| Actions (instructions) | forward to a port, drop, modify a field, push a tag, send to the controller, go to another table |
| Counters | packets and bytes matched; the source of SDN telemetry |
| Timeouts | idle and hard expiry, so state cleans itself up |
Because unspecified fields are wildcards and priority resolves overlaps, one table expresses a MAC table (match destination MAC), a router (match destination IP prefix), a firewall (match 5-tuple, action drop), and a VLAN-aware switch (match and push tags) simultaneously. This generality is the whole point, and it is also the source of the main practical problem: matching on arbitrary combinations of wildcarded fields with priorities requires TCAM, which is expensive, power-hungry, and small — typically a few thousand to a few tens of thousands of entries. A design that installs one rule per flow exhausts it quickly, which is why aggregate rules matter so much in practice.
Later versions added multiple tables in a pipeline, so a packet can pass through a sequence of stages (say, L2 lookup, then ACL, then routing, then output) with metadata carried between them. This drastically reduces rule count, because it replaces a cross product with a sum: separate tables of 100 and 100 entries handle what a single table would need 10,000 entries to express.
Two mechanisms deserve specific mention. Table-miss behaviour is itself a flow entry (priority zero, all wildcards), and what it does — punt to controller, drop, or fall through — is what makes a switch reactive or proactive. Group tables provide indirection: a flow points at a group, and the group contains several buckets that can be selected (for ECMP hashing), used as fast-failover (use the first live port), or applied together (for multicast). Indirection matters operationally because changing one group entry can redirect thousands of flows without rewriting any of them.
OpenFlow's decline is instructive. Its match fields are a fixed list defined by the specification, so supporting a new protocol required a new version of the standard and new hardware — exactly the coupling SDN set out to remove. Programmable data planes (two topics on) fixed that by letting the program define the fields. Today OpenFlow persists in software switches, campus deployments, and research, while production SDN more often uses gNMI/OpenConfig for configuration, BGP for distribution, and P4Runtime for programmable pipelines.
Examples
One table shape, four traditional tables
match fields | priority | actions | counters | timeouts
------------------------+----------+----------------+----------+---------
eth_dst=00:11:.. | 100 | output:4 | 8.2M pkt | idle 300
^ this row IS a MAC table entry
ipv4_dst=10.1.0.0/16 | 100 | output:2 | 1.1G pkt | none
^ this row IS a routing table entry
ip_proto=6,tcp_dst=25 | 50 | drop | 14k pkt | none
^ this row IS a firewall rule
in_port=3 | 200 | push_vlan:20 | 91M pkt | none
^ this row IS a VLAN access-port config
Multi-table pipeline: a SUM of rules instead of a PRODUCT
SINGLE TABLE PIPELINE
must enumerate every table 0: L2 (100 entries)
combination: table 1: ACL (100 entries)
100 x 100 = 10,000 entries -------------------------------
total 200 entries
TCAM: exhausted TCAM: comfortable
Group table indirection:
flow: ipv4_dst=10.0.0.0/8 -> group 7
flow: ipv4_dst=10.1.0.0/16 -> group 7 thousands of flows
flow: ipv4_dst=172.16.0.0/12 -> group 7 point at ONE group
|
group 7 (type SELECT)
bucket 1 -> output:1 \ hashed per flow
bucket 2 -> output:2 / = ECMP
group 7 (type FAST_FAILOVER)
bucket 1 -> output:1 if live \ sub-millisecond
bucket 2 -> output:2 if live / local repair
# An OpenFlow-style flow table: priority ordering, wildcards, actions, counters.
ANY = None
class FlowEntry:
def __init__(self, priority, match, actions, idle_timeout=0):
self.priority, self.match, self.actions = priority, match, actions
self.idle_timeout = idle_timeout
self.packets = self.bytes = 0
def matches(self, pkt):
# A field absent from `match` is a WILDCARD: it is never compared.
return all(pkt.get(f) == v for f, v in self.match.items() if v is not ANY)
def __repr__(self):
m = ",".join(f"{k}=" + (f"0x{v:04x}" if isinstance(v, int) and k == "eth_type"
else str(v))
for k, v in self.match.items() if v is not ANY) or "*"
return f"prio={self.priority:<5} {m:44s} -> {self.actions}"
class FlowTable:
def __init__(self, table_miss=("CONTROLLER",)):
self.entries, self.table_miss, self.miss_count = [], table_miss, 0
def add(self, entry):
self.entries.append(entry)
# THE invariant: highest priority wins, so keep the list sorted by it.
self.entries.sort(key=lambda e: -e.priority)
return self
def lookup(self, pkt, size=1500):
for e in self.entries:
if e.matches(pkt):
e.packets += 1; e.bytes += size # per-entry counters = telemetry
return e.actions, e
self.miss_count += 1
return self.table_miss, None # table-miss: the reactive hook
t = FlowTable()
t.add(FlowEntry(65535, {"eth_type": 0x0806}, ("FLOOD",))) # ARP
t.add(FlowEntry(200, {"eth_type": 0x0800, "ipv4_dst": "10.0.0.5",
"ip_proto": 6, "tcp_dst": 443}, ("output:3",))) # exact
t.add(FlowEntry(100, {"eth_type": 0x0800, "ipv4_dst": "10.0.0.5"},
("output:2",))) # less specific
t.add(FlowEntry(50, {"eth_type": 0x0800, "ip_proto": 6, "tcp_dst": 25},
("DROP",))) # firewall
t.add(FlowEntry(10, {"eth_type": 0x0800}, ("NORMAL",))) # catch-all
print("Flow table, in match order:")
for e in t.entries:
print(" ", e)
# Flow table, in match order:
# prio=65535 eth_type=0x0806 -> ('FLOOD',)
# prio=200 eth_type=0x0800,ipv4_dst=10.0.0.5,ip_proto=6,tcp_dst=443 -> ('output:3',)
# prio=100 eth_type=0x0800,ipv4_dst=10.0.0.5 -> ('output:2',)
# prio=50 eth_type=0x0800,ip_proto=6,tcp_dst=25 -> ('DROP',)
# prio=10 eth_type=0x0800 -> ('NORMAL',)
PACKETS = [
("HTTPS to .5", {"eth_type": 0x0800, "ipv4_dst": "10.0.0.5", "ip_proto": 6,
"tcp_dst": 443}),
("SSH to .5", {"eth_type": 0x0800, "ipv4_dst": "10.0.0.5", "ip_proto": 6,
"tcp_dst": 22}),
("SMTP to .9", {"eth_type": 0x0800, "ipv4_dst": "10.0.0.9", "ip_proto": 6,
"tcp_dst": 25}),
("ARP request", {"eth_type": 0x0806}),
("IPv6 packet", {"eth_type": 0x86DD}),
]
print("\nLookups:")
for label, pkt in PACKETS:
actions, hit = t.lookup(pkt)
why = "matched a more specific entry" if hit and hit.priority >= 100 else (
"matched" if hit else "TABLE MISS -> punt to controller")
print(f" {label:14s} -> {str(actions):16s} {why}")
print(f"\ntable misses: {t.miss_count}")
# Lookups:
# HTTPS to .5 -> ('output:3',) matched a more specific entry
# SSH to .5 -> ('output:2',) matched a more specific entry
# SMTP to .9 -> ('DROP',) matched
# ARP request -> ('FLOOD',) matched a more specific entry
# IPv6 packet -> ('CONTROLLER',) TABLE MISS -> punt to controller
#
# Note SSH to .5: it does NOT hit the priority-200 entry (wrong port) and falls
# through to the priority-100 subnet entry. Priority + wildcards together are
# what let one table express "specific exceptions over a general rule".
# Why one rule per flow does not work: TCAM is small.
TCAM_ENTRIES = 4_000
for style, rules_per_host in (("one rule per FLOW", 40), ("one rule per HOST", 1),
("one rule per SUBNET", 0.02)):
hosts = TCAM_ENTRIES / rules_per_host
print(f" {style:22s} supports {hosts:>9,.0f} hosts in {TCAM_ENTRIES:,} TCAM entries")
# one rule per FLOW supports 100 hosts in 4,000 TCAM entries
# one rule per HOST supports 4,000 hosts in 4,000 TCAM entries
# one rule per SUBNET supports 200,000 hosts in 4,000 TCAM entries
# Open vSwitch: the flow-table abstraction you can actually poke at.
# Install rules, highest priority first in effect (not in file order).
ovs-ofctl add-flow br0 "table=0,priority=200,ip,nw_dst=10.0.0.5,tcp,tp_dst=443,\
actions=output:3"
ovs-ofctl add-flow br0 "table=0,priority=100,ip,nw_dst=10.0.0.5,actions=output:2"
ovs-ofctl add-flow br0 "table=0,priority=50,ip,tcp,tp_dst=25,actions=drop"
ovs-ofctl add-flow br0 "table=0,priority=0,actions=CONTROLLER:65535" # table-miss
# Read the counters: this is the telemetry, per rule, for free.
ovs-ofctl dump-flows br0
# priority=200,tcp,nw_dst=10.0.0.5,tp_dst=443 n_packets=81234 n_bytes=98M actions=output:3
# priority=50,tcp,tp_dst=25 n_packets=17 n_bytes=1122 actions=drop
# ^ 17 blocked SMTP attempts, no extra tooling
# THE essential debugging tool: ask which rule a hypothetical packet would hit.
ovs-appctl ofproto/trace br0 \
'in_port=1,dl_type=0x0800,nw_src=10.0.0.1,nw_dst=10.0.0.5,nw_proto=6,tp_dst=22'
# Rule: table=0 cookie=0 priority=100,ip,nw_dst=10.0.0.5
# Datapath actions: 2
# ^ confirms the fall-through to the less specific rule
# Group table for ECMP across two uplinks, then point flows at it.
ovs-ofctl -O OpenFlow13 add-group br0 \
"group_id=7,type=select,bucket=output:1,bucket=output:2"
ovs-ofctl -O OpenFlow13 add-flow br0 "priority=90,ip,actions=group:7"
Key takeaways
- OpenFlow replaces the MAC, routing, ACL, and VLAN tables with one generalised table: match fields, priority, actions, counters, timeouts.
- Unspecified match fields are wildcards and priority resolves overlaps, which is what lets one table express specific exceptions over general rules.
- Wildcard-plus-priority matching requires TCAM, which is small and expensive — a few thousand entries — so aggregate rather than per-flow rules are essential.
- A multi-table pipeline turns a cross product of rules into a sum, which is the main tool for staying inside TCAM limits.
- The table-miss entry determines reactive versus proactive behaviour; group tables add indirection for ECMP, fast failover, and multicast.
- Per-entry counters make telemetry a free side effect of forwarding.
- OpenFlow's fixed match-field list recoupled software to hardware and is the reason programmable data planes superseded it.
🧪 Practice
- Write the flow entries that make a switch forward HTTPS to 10.0.0.9 out port 5, drop all other traffic to that host, and punt everything else to the controller. Give the priorities and explain the ordering.
- A design needs to match 200 source prefixes against 50 destination ports. Compute the entry count as one table and as a two-table pipeline.
- Explain what happens to existing flows when a fast-failover group's first bucket goes down, and why this is faster than a controller-driven reroute.
- Interview-style: A team installs one exact-match flow rule per TCP connection on a top-of-rack switch and the network fails under load. (Hint: look up the size of the table that supports wildcard matching with priorities, then divide by the connection rate.)
Northbound and Southbound APIs
Theory
The controller sits between two very different audiences, and it needs a different kind of interface for each. Naming them by direction is a convention worth internalising because it tells you immediately what a given API is for.
Southbound faces the devices. It is about mechanism: install this entry, read that counter, set this interface's MTU. It is necessarily specific, high-volume, and stateful, and it must cope with hardware heterogeneity. The main dialects:
| Protocol | Model | Typical use |
|---|---|---|
| OpenFlow | flow tables, imperative | flow programming in software switches, campus, research |
| NETCONF | configuration data, YANG models, XML | structured configuration with transactions |
| gNMI | YANG over gRPC, with streaming | modern configuration plus telemetry subscriptions |
| P4Runtime | tables defined by a P4 program | programmable pipelines |
| BGP / BGP-LS | route distribution, topology export | large-scale distribution, controller topology learning |
| SNMP, CLI, SSH | legacy | what you actually find in brownfield networks |
Northbound faces applications and humans. It is about intent: isolate these tenants, give this customer 50 Mb/s, keep no link above 70%. It should be small, declarative, and stable — because it is a contract that applications are written against, and it should not change when the hardware does. Northbound APIs are usually REST or gRPC with a resource model, sometimes a domain-specific intent language.
The asymmetry between them is the point. The southbound side is wide, imperative, and churns as hardware evolves. The northbound side should be narrow, declarative, and durable. The controller's job is to absorb the difference. This is precisely the value the abstraction delivers: an application says "isolate tenant A from tenant B" once, and the controller expands it into hundreds of device-specific entries on whatever mix of hardware happens to be deployed, reissuing them as devices come and go.
That expansion ratio is worth internalising as a number. A handful of human statements becomes hundreds or thousands of device entries, and the ratio grows with the size of the network. This is simultaneously the benefit (humans manage intent, not entries) and the risk (a one-line change can rewrite a great deal of state, and there is no way to review the result by reading it).
Two consequences follow for anyone operating such a system.
First, the northbound API is where policy conflicts must be resolved, because by the time intent has become flow entries the original reasoning is gone. A controller that lets two applications install contradictory rules without arbitration has pushed a decidable problem down to where it is undecidable.
Second, translation must be verifiable. Since nobody can read thousands of generated entries, the controller needs to answer questions in the original vocabulary: "can tenant A reach tenant B on port 5432?" This is the reachability-verification idea that matured into the intent-based networking of Section 13.5, and it is the difference between an abstraction you can trust and one you merely hope about.
A last note on OpenConfig and YANG, which matter more than any single protocol. YANG is a language for describing configuration and state models; OpenConfig is a set of vendor-neutral YANG models. Together they mean a single request can configure equipment from different vendors, which removes the need for the controller to hold a per-vendor template library. gNMI is the transport that carries them and additionally supports streaming telemetry — subscribe once and receive updates on change — replacing the polling model of SNMP.
Examples
Two interfaces, opposite design goals
HUMANS / APPLICATIONS
|
"isolate web from db,
except Postgres" <- INTENT: declarative,
| small, stable, vendor-free
================ NORTHBOUND (REST / gRPC / intent) ================
|
+-------------+
| CONTROLLER | absorbs the difference:
| translate | 1 intent -> N device entries
| arbitrate | resolves app conflicts
| verify | answers questions in intent terms
+-------------+
|
================ SOUTHBOUND (OpenFlow / gNMI / P4Runtime) =========
|
+-----------------+-----------------+
v v v
[ vendor A ] [ vendor B ] [ white box ] <- MECHANISM:
NETCONF OpenFlow P4Runtime imperative, wide,
wants XML wants flow_mod wants table_entry hardware-specific
Design rule of thumb:
northbound narrow, declarative, changes rarely (it is a CONTRACT)
southbound wide, imperative, changes often (it tracks HARDWARE)
# Northbound: INTENT, expressed once. Southbound: the rules it compiles into.
TENANTS = {
"web": {"subnet": "10.1.0.0/24", "ports": [1, 2]},
"db": {"subnet": "10.2.0.0/24", "ports": [3]},
"mgmt": {"subnet": "10.9.0.0/24", "ports": [4]},
}
INTENTS = [
{"kind": "isolate", "tenants": ["web", "db"]}, # default deny
{"kind": "allow", "src": "web", "dst": "db", "proto": 6, "port": 5432},
{"kind": "allow-any", "src": "mgmt"}, # jump host
]
def compile_intents(intents, tenants):
"""One high-level statement expands into many device rules. That expansion is
the value of the abstraction -- and the reason it must be verifiable."""
rules = []
for i in intents:
if i["kind"] == "isolate":
for a in i["tenants"]:
for b in i["tenants"]:
if a != b:
rules.append(dict(priority=100, src=tenants[a]["subnet"],
dst=tenants[b]["subnet"], action="DROP"))
elif i["kind"] == "allow":
rules.append(dict(priority=200, src=tenants[i["src"]]["subnet"],
dst=tenants[i["dst"]]["subnet"], proto=i["proto"],
port=i["port"], action="FORWARD"))
elif i["kind"] == "allow-any":
rules.append(dict(priority=300, src=tenants[i["src"]]["subnet"],
dst="0.0.0.0/0", action="FORWARD"))
return sorted(rules, key=lambda r: -r["priority"]) # priority order matters
rules = compile_intents(INTENTS, TENANTS)
print(f"{len(INTENTS)} intents compiled into {len(rules)} flow rules:\n")
for r in rules:
extra = f" proto={r.get('proto')} port={r.get('port')}" if "proto" in r else ""
print(f" prio {r['priority']:3d} {r['src']:14s} -> {r['dst']:14s}"
f"{extra:22s} {r['action']}")
# 3 intents compiled into 4 flow rules:
#
# prio 300 10.9.0.0/24 -> 0.0.0.0/0 FORWARD
# prio 200 10.1.0.0/24 -> 10.2.0.0/24 proto=6 port=5432 FORWARD
# prio 100 10.1.0.0/24 -> 10.2.0.0/24 DROP
# prio 100 10.2.0.0/24 -> 10.1.0.0/24 DROP
SWITCHES = ["leaf-1", "leaf-2", "leaf-3"]
print(f"\npushed to {len(SWITCHES)} switches -> {len(rules) * len(SWITCHES)} "
f"device entries from {len(INTENTS)} human statements "
f"(1 : {len(rules) * len(SWITCHES) / len(INTENTS):.1f})")
# pushed to 3 switches -> 12 device entries from 3 human statements (1 : 4.0)
# At 200 switches and 30 intents this ratio reaches thousands of entries that no
# human will ever read -- hence the next function.
def evaluate(rules, src, dst, proto=None, port=None):
"""Verification in the vocabulary of the INTENT, not of the rules."""
for r in rules: # already priority-ordered
if not src.startswith(r["src"][:5]):
continue
if r["dst"] != "0.0.0.0/0" and not dst.startswith(r["dst"][:5]):
continue
if "proto" in r and (proto != r["proto"] or port != r["port"]):
continue
return r["action"]
return "DROP (implicit)"
print("\nPolicy verification -- does the compiled output match what was asked?")
for label, args in (("web -> db :5432", ("10.1.0.7", "10.2.0.3", 6, 5432)),
("web -> db :22", ("10.1.0.7", "10.2.0.3", 6, 22)),
("db -> web:80", ("10.2.0.3", "10.1.0.7", 6, 80)),
("mgmt-> db :22", ("10.9.0.2", "10.2.0.3", 6, 22))):
print(f" {label:18s} -> {evaluate(rules, *args)}")
# web -> db :5432 -> FORWARD
# web -> db :22 -> DROP
# db -> web:80 -> DROP
# mgmt-> db :22 -> FORWARD
# This is the only honest way to review a compiled policy: ask questions in the
# language the human used, not by reading generated entries.
# --- Southbound, the modern way: gNMI with OpenConfig models ---
# Read state with a vendor-neutral path (same path works across vendors).
gnmic -a switch1:57400 -u admin -p admin --skip-verify get \
--path "/interfaces/interface[name=Ethernet1]/state/counters"
# Configure with the same model, as a transaction.
gnmic -a switch1:57400 -u admin -p admin --skip-verify set \
--update-path "/interfaces/interface[name=Ethernet1]/config/mtu" \
--update-value 9216
# STREAMING telemetry: subscribe once, receive on change. This replaces polling.
gnmic -a switch1:57400 -u admin -p admin --skip-verify subscribe \
--path "/interfaces/interface/state/counters/out-discards" \
--stream-mode sample --sample-interval 10s
# {"source":"switch1","path":"...Ethernet3...out-discards","value":41822}
# ^ pushed by the device; no poll loop, no SNMP timeout tuning
# --- NETCONF: transactional configuration, with a real rollback ---
ssh -s admin@switch1 netconf <<'XML'
<rpc message-id="1"><edit-config>
<target><candidate/></target> <!-- stage into a candidate datastore -->
<config><interfaces xmlns="http://openconfig.net/yang/interfaces">
<interface><name>Ethernet1</name><config><mtu>9216</mtu></config></interface>
</interfaces></config>
</edit-config></rpc>
XML
# then <commit/> to apply atomically, or <discard-changes/> to abandon.
# This is what the CLI never had: a change you can assemble, review, and reject.
# --- Northbound: intent in, expansion hidden ---
curl -X POST http://ctrl:8181/onos/v1/intents -H 'Content-Type: application/json' \
-d '{"type":"HostToHostIntent","appId":"org.onosproject.cli",
"one":"00:00:00:00:00:01/None","two":"00:00:00:00:00:02/None"}'
# One POST. The controller computes the path and installs rules on every switch
# along it, and reinstalls them if the topology changes.
Key takeaways
- Southbound faces devices and is about mechanism: wide, imperative, high-volume, hardware-specific, and it churns as hardware evolves.
- Northbound faces applications and is about intent: narrow, declarative, and stable, because it is a contract applications are written against.
- The controller exists to absorb the difference between them, expanding a few human statements into thousands of device entries.
- That expansion ratio is the benefit and the risk: nobody can review generated state by reading it, so the northbound API must support verification in intent terms.
- Policy conflicts must be resolved at the northbound layer, because the original reasoning is gone once intent becomes flow entries.
- YANG models plus OpenConfig give vendor-neutral configuration; gNMI carries them and adds streaming telemetry, replacing SNMP polling.
- NETCONF offers candidate datastores and atomic commits, which the CLI never provided.
🧪 Practice
- Label each as northbound or southbound: a REST call creating a tenant network, a
flow_modmessage, a gNMI subscription to interface counters, a request asking whether two subnets can communicate.- Explain why a northbound API should not expose flow priorities, and what it should expose instead.
- Write the three intents needed for a three-tier application (web, app, db) permitting only adjacent-tier communication, then count the directional rules they compile into.
- Interview-style: An operator wants to replace SNMP polling of 4,000 interfaces every 30 seconds. (Hint: name the protocol and the subscription model, then compare what crosses the network under polling versus on-change updates, and what happens to freshness during a burst.)
P4 and Programmable Data Planes
Theory
OpenFlow separated the control plane from the data plane but left one coupling intact: the data plane's behaviour remained fixed. Its match fields were a list in the specification, so a switch could match on IPv4 and TCP because the spec said so, and supporting a new encapsulation meant a new spec version and new silicon. Adding VXLAN support to the industry took years. The abstraction had moved the thinking out of the box while leaving the box's vocabulary frozen.
P4 ("Programming Protocol-independent Packet Processors") removes that coupling by making the pipeline itself programmable. A P4 program specifies:
- Headers: the formats the device understands, including ones nobody has standardised.
- A parser: a state machine that walks the byte stream and decides which headers are present.
- Tables: what fields to match, what kind of match (exact, ternary, longest-prefix), and which actions are permitted.
- Actions: the operations that modify headers and metadata.
- Control flow: the order in which tables are applied, including conditionals.
- A deparser: which headers to write back out, in what order.
The consequence is a genuine inversion. Traditionally the chip defines what the network can do and protocol designers wait for silicon. With P4 the program defines it, and the chip is a substrate. A new encapsulation becomes a program change and a recompile, and the control plane API is generated from the program — which is what P4Runtime is: a control interface whose tables and fields come from the P4 source rather than from a fixed standard.
A precision worth holding onto: P4 is not a general-purpose language and deliberately so. It has no loops, no dynamic memory allocation, and no recursion, because the target is a pipeline that must finish every packet in a fixed number of stages at line rate. Programs that cannot be laid out in the available stages simply fail to compile. This restriction is the reason P4 can run at terabit rates while being programmable — it is a language shaped by the constraint that all programs terminate in bounded time.
What it enables in practice:
| Capability | What it does | Why it needed programmability |
|---|---|---|
| In-band network telemetry (INT) | each hop appends its queue depth and timestamp to the packet | no standard header existed; the format is program-defined |
| Custom encapsulations | new tunnel or source-routing formats | fixed-function chips parse only known headers |
| In-network computing | aggregate values or cache key-value lookups in the switch | requires stateful registers and arbitrary field arithmetic |
| Load balancing in the fabric | consistent-hash a VIP to backends at line rate | replaces a fleet of software load balancers |
| Protocol offload | terminate or translate a protocol in the pipeline | behaviour must be expressible, not just matchable |
INT deserves emphasis because it changes what is observable. Every hop stamps its own queue occupancy into the packet, so the receiver reconstructs the exact per-hop delay experienced by that packet — the measurement that Chapter 12's budgets need and that no amount of polling can produce, because polling samples the queue at moments unrelated to when your packet crossed it.
The costs are real. P4-capable hardware is a narrower market than commodity switching silicon. A programmable pipeline is a program you now own, with its own bugs, tests, and upgrade path, and a compilation failure late in development can force a redesign. Most production networks therefore use P4 for a specific capability rather than as a wholesale replacement, and the common deployment is a small number of programmable devices at points where custom behaviour pays for itself.
Examples
// A minimal P4-16 program: parse Ethernet/IPv4 plus a CUSTOM tunnel header
// that no vendor's fixed-function chip has ever heard of.
header ethernet_t { bit<48> dst; bit<48> src; bit<16> ether_type; }
header mytunnel_t { bit<16> proto_id; bit<16> dst_id; } // entirely our invention
header ipv4_t { bit<8> version_ihl; bit<8> diffserv; bit<16> total_len;
bit<32> src; bit<32> dst; bit<8> ttl; }
struct headers_t { ethernet_t ethernet; mytunnel_t mytunnel; ipv4_t ipv4; }
parser MyParser(packet_in pkt, out headers_t hdr) {
state start { pkt.extract(hdr.ethernet);
// The parser is a STATE MACHINE. Adding a protocol = adding a transition.
transition select(hdr.ethernet.ether_type) {
0x1212: parse_mytunnel; // our custom EtherType
0x0800: parse_ipv4;
default: accept;
} }
state parse_mytunnel { pkt.extract(hdr.mytunnel);
transition select(hdr.mytunnel.proto_id) { 0x0800: parse_ipv4;
default: accept; } }
state parse_ipv4 { pkt.extract(hdr.ipv4); transition accept; }
}
control MyIngress(inout headers_t hdr, inout standard_metadata_t meta) {
action drop() { mark_to_drop(meta); }
action ipv4_forward(bit<9> port) {
meta.egress_spec = port;
hdr.ipv4.ttl = hdr.ipv4.ttl - 1; // an action mutating a header field
}
action set_egress(bit<9> port) { meta.egress_spec = port; }
table ipv4_lpm { // the CONTROL plane fills this at runtime
key = { hdr.ipv4.dst: lpm; } // match kind is part of the program
actions = { ipv4_forward; drop; }
size = 1024;
default_action = drop();
}
table mytunnel_exact {
key = { hdr.mytunnel.dst_id: exact; }
actions = { set_egress; drop; }
size = 1024;
default_action = drop();
}
apply { // CONTROL FLOW is program-defined:
if (hdr.mytunnel.isValid()) { // custom header wins when present
mytunnel_exact.apply();
} else if (hdr.ipv4.isValid()) {
ipv4_lpm.apply();
}
}
}
// No loops, no malloc, no recursion: every program must finish in a fixed
// number of pipeline stages. That restriction is why this runs at terabit rates.
# A model of the same pipeline: PARSE -> MATCH-ACTION -> DEPARSE.
ETH_TYPE_IPV4, ETH_TYPE_MYTUNNEL = 0x0800, 0x1212
def parser(raw):
"""A P4 parser is a state machine over the byte stream. This one accepts a
header the silicon vendor never heard of -- that is the whole innovation."""
hdr, off = {}, 0
hdr["ethernet"] = dict(dst=raw[0], src=raw[1], ether_type=raw[2]); off = 3
if hdr["ethernet"]["ether_type"] == ETH_TYPE_MYTUNNEL:
hdr["mytunnel"] = dict(proto_id=raw[off], dst_id=raw[off + 1]); off += 2
nxt = hdr["mytunnel"]["proto_id"]
else:
nxt = hdr["ethernet"]["ether_type"]
if nxt == ETH_TYPE_IPV4:
hdr["ipv4"] = dict(src=raw[off], dst=raw[off + 1], ttl=raw[off + 2])
return hdr
# Tables are DECLARED in P4 and POPULATED at runtime by the control plane.
ipv4_lpm = {"10.0.1.1": ("ipv4_forward", 1), "10.0.2.2": ("ipv4_forward", 2)}
mytunnel_ex = {1: ("set_egress", 1), 2: ("set_egress", 2)}
def ingress(hdr):
"""Match-action. Which tables are applied, and in what order, is program-defined."""
meta = {"egress_port": None, "drop": False, "path": []}
if "mytunnel" in hdr: # custom header takes precedence
meta["path"].append("mytunnel_exact")
hit = mytunnel_ex.get(hdr["mytunnel"]["dst_id"])
if hit:
meta["egress_port"] = hit[1]
else:
meta["drop"] = True
elif "ipv4" in hdr:
meta["path"].append("ipv4_lpm")
hit = ipv4_lpm.get(hdr["ipv4"]["dst"])
if hit:
meta["egress_port"] = hit[1]
hdr["ipv4"]["ttl"] -= 1 # action mutates the header
if hdr["ipv4"]["ttl"] == 0:
meta["drop"] = True
else:
meta["drop"] = True
else:
meta["drop"] = True
return meta
def deparse(hdr):
"""Emit headers back onto the wire, in the order the program specifies."""
out = [hdr["ethernet"]["dst"], hdr["ethernet"]["src"], hdr["ethernet"]["ether_type"]]
if "mytunnel" in hdr:
out += [hdr["mytunnel"]["proto_id"], hdr["mytunnel"]["dst_id"]]
if "ipv4" in hdr:
out += [hdr["ipv4"]["src"], hdr["ipv4"]["dst"], hdr["ipv4"]["ttl"]]
return out
TESTS = [
("plain IPv4 to .2.2",
["aa:bb", "cc:dd", ETH_TYPE_IPV4, "10.0.1.1", "10.0.2.2", 64]),
("IPv4, unknown dst",
["aa:bb", "cc:dd", ETH_TYPE_IPV4, "10.0.1.1", "10.0.9.9", 64]),
("IPv4, TTL about to expire",
["aa:bb", "cc:dd", ETH_TYPE_IPV4, "10.0.1.1", "10.0.2.2", 1]),
("custom myTunnel dst_id=1",
["aa:bb", "cc:dd", ETH_TYPE_MYTUNNEL, ETH_TYPE_IPV4, 1, "10.0.1.1",
"10.0.2.2", 64]),
]
for label, raw in TESTS:
hdr = parser(raw)
meta = ingress(hdr)
verdict = "DROP" if meta["drop"] else f"port {meta['egress_port']}"
print(f"{label:28s} headers={list(hdr)} table={meta['path']} -> {verdict}")
if not meta["drop"]:
print(f"{'':28s} on wire: {deparse(hdr)}")
# plain IPv4 to .2.2 headers=['ethernet', 'ipv4'] table=['ipv4_lpm'] -> port 2
# on wire: ['aa:bb', 'cc:dd', 2048, '10.0.1.1', '10.0.2.2', 63]
# IPv4, unknown dst headers=['ethernet', 'ipv4'] table=['ipv4_lpm'] -> DROP
# IPv4, TTL about to expire headers=['ethernet', 'ipv4'] table=['ipv4_lpm'] -> DROP
# custom myTunnel dst_id=1 headers=['ethernet', 'mytunnel', 'ipv4'] table=['mytunnel_exact'] -> port 1
# on wire: ['aa:bb', 'cc:dd', 4626, 2048, 1, '10.0.1.1', '10.0.2.2', 64]
#
# Note the TTL decrement appearing in the emitted bytes (64 -> 63), and that the
# custom-header packet never touched ipv4_lpm. Both behaviours came from the
# PROGRAM, not from the chip.
In-band network telemetry: why programmability changes what is measurable
WITHOUT INT (polling) WITH INT (per-packet, in-band)
controller asks each switch each hop APPENDS its own state
"what is your queue depth?" to the packet as it passes
every 10 s +-------------------------------+
-> a sample from a moment | payload | h1 | h2 | h3 | h4 |
unrelated to your packet +-------------------------------+
-> tells you nothing about ^ ^ ^
the packet that was slow hop id, ingress/egress
timestamp, queue depth
Receiver reconstructs the EXACT experience of THAT packet:
hop 1 queue 2 pkt dwell 12 us
hop 2 queue 84 pkt dwell 1010 us <- the culprit, identified per packet
hop 3 queue 1 pkt dwell 9 us
total in-network delay 1031 us
This is the measurement Chapter 12's delay budgets actually need, and no
polling interval can produce it.
Key takeaways
- OpenFlow freed the control plane but left the data plane's vocabulary fixed in the specification and the silicon; adding a protocol took years.
- A P4 program defines headers, a parser state machine, tables and their match kinds, actions, control flow, and a deparser. The program defines behaviour; the chip is a substrate.
- P4Runtime generates the control API from the program, so the control interface is no longer a fixed standard.
- P4 deliberately has no loops, no recursion, and no dynamic allocation, because every packet must finish in a bounded number of stages. That restriction is what allows line-rate programmability.
- It enables in-band network telemetry, custom encapsulations, in-network computing, fabric load balancing, and protocol offload.
- INT gives per-packet, per-hop queue and timing data — the measurement polling structurally cannot provide.
- The costs are narrower hardware availability and owning a program with its own bugs and upgrade path, so P4 is usually deployed for specific capabilities rather than wholesale.
🧪 Practice
- List the six components of a P4 program and state, for each, whether it is fixed at compile time or populated at runtime.
- Explain why P4 forbids loops, and describe what happens to a program whose pipeline does not fit the target's stage count.
- Design the header format for an INT metadata stack carrying hop ID, egress timestamp, and queue depth for up to eight hops. Give the field widths and total overhead.
- Interview-style: A team wants to know which hop delayed a specific slow request, and their monitoring polls every switch's queue depth every 10 seconds. (Hint: compare the timescale of a queueing event with the polling interval, then consider what would have to travel with the packet itself.)
<a id="132-virtualization-and-overlays"></a>
13.2 Virtualization and Overlays
Virtual machines and containers appear and vanish faster than any physical network can be rewired, and they need addresses that do not depend on where they happen to be running: this subchapter covers the techniques that decouple the network a workload sees from the network it actually runs on, from software-implemented network functions through the tunnels that carry logical segments over a physical fabric, to the container and service-mesh layers built on top.
Network Function Virtualization
Theory
For most of networking's history, a network function meant a box. A firewall was a firewall appliance; a load balancer was a load balancer appliance. Each was purpose-built hardware running vendor software, and adding a function to a network meant a purchase order, a rack unit, a cabling change, and a maintenance window. The coupling was total: the function you wanted determined the hardware you bought and where you physically put it.
NFV breaks that coupling by observing that most of these functions are just software processing packets, and that general-purpose servers have become fast enough to do the job. A VNF (virtualised network function) is that software running in a VM or container on commodity hardware. The gains are the ones you would expect from turning hardware into software: you can instantiate a firewall in minutes rather than weeks, scale out under load, place a function wherever capacity exists, and run different vendors' functions on the same server.
The framework has three parts, and the names appear throughout vendor documentation:
| Component | Role |
|---|---|
| VNF | the function itself: a virtual firewall, router, load balancer, or session border controller |
| NFVI (infrastructure) | the compute, storage, and network substrate the VNFs run on |
| MANO (management and orchestration) | places, scales, heals, and chains VNFs; the part that actually determines whether NFV works |
The composition idea is service function chaining: a packet passes through an ordered sequence of functions — say firewall, then deep packet inspection, then NAT — and because the functions are software, the chain is a configuration rather than a cabling diagram. Different traffic can take different chains, which is genuinely impossible with physical appliances short of building every path in copper.
The hard part is performance, and understanding why requires looking at the per-packet budget. A 10 Gb/s stream of minimum-size packets is 19.5 million packets per second, leaving about 154 cycles per packet on a 3 GHz core. The conventional Linux network path spends several thousand cycles per packet on interrupt handling, socket buffer allocation, and copies between kernel and user space. The arithmetic does not work: the general-purpose path is roughly thirty times too slow for a single 10 Gb/s port.
This is why NFV brought a family of data-path techniques into mainstream networking:
- DPDK moves the driver into user space, polls the NIC instead of taking interrupts, uses huge pages to reduce TLB misses, and processes packets in batches. It reaches line rate but dedicates cores to spinning at 100% CPU whether traffic is flowing or not.
- XDP / eBPF runs a verified program in the driver, before the kernel allocates a socket buffer, which removes the most expensive per-packet allocation while staying inside the kernel.
- SR-IOV lets the NIC present virtual functions directly to VMs, bypassing the hypervisor's switch entirely. It costs no host CPU at all, and the trade is severe: the virtual switch no longer sees the traffic, so overlay encapsulation, security policy, and per-VM telemetry must move into the NIC or be given up.
- SmartNICs and DPUs are the current answer to that trade: put a programmable processor on the NIC and run the virtual switch there, recovering both the CPU savings and the visibility.
A realistic assessment: NFV delivered enormous operational flexibility and rarely matched the performance-per-watt of purpose-built silicon. Its clearest wins are at the network edge, where flexibility matters more than raw throughput — customer premises equipment, mobile packet cores, and enterprise branch functions — while the highest-volume core forwarding stayed in hardware.
Examples
From appliances to functions
BEFORE: the chain is the cabling
[ internet ]--[ firewall ]--[ IDS ]--[ load balancer ]--[ servers ]
^ a box ^ a box ^ a box
adding a function = purchase + rack + recable + maintenance window
different chain per customer = physically impossible
AFTER: the chain is a configuration
+--------------------------------------------------------------+
| COMMODITY SERVERS (NFVI) |
| +--------+ +--------+ +--------+ +--------+ |
| | vFW | | vIDS | | vLB | | vRouter| <- VNFs |
| +--------+ +--------+ +--------+ +--------+ |
| virtual switch / SmartNIC data path |
+--------------------------------------------------------------+
^ MANO places, scales, heals, chains
chain A (paying customer) : vFW -> vIDS -> vLB
chain B (guest traffic) : vFW -> vLB
chain C (IoT segment) : vFW -> vIDS
-- three topologies, one physical wiring
# Why software packet processing needs special plumbing: the per-packet budget.
GHZ = 3.0
CYCLES_PER_SEC = GHZ * 1e9
def budget(line_rate_gbps, pkt_bytes):
pps = line_rate_gbps * 1e9 / (pkt_bytes * 8)
return pps, CYCLES_PER_SEC / pps, 1e9 / pps
print("Per-packet budget on one 3.0 GHz core:")
for rate, size in ((1, 1500), (10, 1500), (10, 64), (100, 1500), (100, 64)):
pps, cycles, ns = budget(rate, size)
print(f" {rate:3d} Gb/s, {size:4d} B -> {pps/1e6:7.2f} Mpps "
f"{cycles:7.0f} cycles {ns:7.1f} ns per packet")
# Per-packet budget on one 3.0 GHz core:
# 1 Gb/s, 1500 B -> 0.08 Mpps 36000 cycles 12000.0 ns per packet
# 10 Gb/s, 1500 B -> 0.83 Mpps 3600 cycles 1200.0 ns per packet
# 10 Gb/s, 64 B -> 19.53 Mpps 154 cycles 51.2 ns per packet
# 100 Gb/s, 1500 B -> 8.33 Mpps 360 cycles 120.0 ns per packet
# 100 Gb/s, 64 B -> 195.31 Mpps 15 cycles 5.1 ns per packet
# ^ small packets, not bits, are what break software forwarding
_, REF_CYCLES, _ = budget(10, 64)
print(f"\nReference case: 10 Gb/s at 64 B = {REF_CYCLES:.0f} cycles per packet.\n")
COSTS = {"interrupt + softirq": 1_000, "sk_buff alloc/free": 500,
"kernel/user copy": 800, "context switch (amortised)": 2_000,
"actual forwarding logic": 300}
total = sum(COSTS.values())
print("Cost of the traditional kernel network path:")
for k, v in COSTS.items():
print(f" {k:30s} {v:6,} cycles")
print(f" {'TOTAL':30s} {total:6,} cycles -> "
f"{total/REF_CYCLES:.0f} cores needed for one 10 Gb/s port\n")
# Cost of the traditional kernel network path:
# interrupt + softirq 1,000 cycles
# sk_buff alloc/free 500 cycles
# kernel/user copy 800 cycles
# context switch (amortised) 2,000 cycles
# actual forwarding logic 300 cycles
# TOTAL 4,600 cycles -> 30 cores needed for one 10 Gb/s port
#
# Note that only 300 of 4,600 cycles is the ACTUAL WORK. Everything else is
# the general-purpose machinery that a packet forwarder does not need.
print("What each bypass technique removes, and what it leaves:")
for name, removed, note in (
("kernel (baseline)", [], "sk_buff per packet, interrupt driven"),
("XDP / eBPF", ["sk_buff alloc/free", "kernel/user copy"],
"runs in the driver, before sk_buff allocation"),
("DPDK poll mode", ["interrupt + softirq", "sk_buff alloc/free",
"kernel/user copy", "context switch (amortised)"],
"userspace driver, busy-poll, hugepages, no interrupts"),
("SR-IOV", ["interrupt + softirq", "sk_buff alloc/free", "kernel/user copy",
"context switch (amortised)", "actual forwarding logic"],
"NIC switches straight into the VM; the vSwitch is bypassed"),
):
left = total - sum(COSTS[r] for r in removed)
cores = left / REF_CYCLES
verdict = "line rate on one core" if 0 < cores <= 1 else (
"no host CPU at all" if cores == 0 else f"{cores:.1f} cores")
print(f" {name:20s} {left:5,} cycles -> {verdict:22s} {note}")
# kernel (baseline) 4,600 cycles -> 29.9 cores sk_buff per packet, interrupt driven
# XDP / eBPF 3,300 cycles -> 21.5 cores runs in the driver, before sk_buff allocation
# DPDK poll mode 300 cycles -> 2.0 cores userspace driver, busy-poll, hugepages, no interrupts
# SR-IOV 0 cycles -> no host CPU at all NIC switches straight into the VM; the vSwitch is bypassed
#
# The trade SR-IOV makes: zero CPU cost, but the hypervisor's virtual switch no
# longer sees the traffic, so overlay encapsulation, ACLs and per-VM telemetry
# must move into the NIC (a SmartNIC/DPU) or be given up entirely.
# Inspecting an NFV data path on a Linux host.
# Is this NIC doing SR-IOV, and how many virtual functions are configured?
cat /sys/class/net/ens1f0/device/sriov_numvfs # 0 = not in use
lspci | grep -i 'virtual function'
# 3b:10.1 Ethernet controller: Intel X710 Virtual Function
# ^ each VF is handed straight to a VM; the host vSwitch never sees it
# DPDK: which interfaces has the kernel given up to a userspace driver?
dpdk-devbind.py --status
# Network devices using DPDK-compatible driver
# 0000:3b:00.0 'Ethernet X710' drv=vfio-pci unused=i40e
# ^ no kernel interface exists for this NIC any more; ip link will not show it
# Confirm the cost of poll mode: a DPDK core is pinned at 100% by design.
top -H -p "$(pgrep -f testpmd)" -n1 | head -12
# PID %CPU COMMAND
# 4412 99.9 lcore-worker-1 <- spinning whether or not traffic is arriving
# XDP: attach an eBPF program at the driver, before sk_buff allocation.
ip link set dev ens1f0 xdpgeneric obj filter.o sec xdp
ip -d link show dev ens1f0 | grep -i xdp
# prog/xdp id 217 tag a1b2c3d4 jited
# Hugepages, which DPDK requires to keep TLB misses off the per-packet path.
grep Huge /proc/meminfo
# HugePages_Total: 2048
# Hugepagesize: 2048 kB
Key takeaways
- NFV replaces purpose-built appliances with software functions on commodity servers, turning weeks of procurement and cabling into minutes of orchestration.
- The three framework pieces are the VNF (the function), the NFVI (the substrate), and MANO (orchestration, which determines whether it works in practice).
- Service function chaining makes the order of functions a configuration, so different traffic can take different chains — impossible with physical appliances.
- The hard constraint is the per-packet budget: 10 Gb/s of 64-byte packets leaves about 154 cycles per packet, and the ordinary kernel path spends thousands.
- Only a small fraction of the kernel path's cost is the actual forwarding work; the rest is general-purpose machinery a forwarder does not need.
- DPDK (userspace poll mode), XDP (eBPF in the driver), and SR-IOV (direct hardware to VM) each remove a different part of that overhead.
- SR-IOV costs no host CPU but blinds the virtual switch, which is why SmartNICs and DPUs exist: run the switch on the NIC and keep both benefits.
- NFV's wins are at the edge, where flexibility outweighs throughput; core forwarding largely stayed in hardware.
🧪 Practice
- Compute the packets per second and per-packet cycle budget for a 25 Gb/s port at 64-byte and 1500-byte packets on a 2.5 GHz core.
- Explain why SR-IOV breaks overlay networking, and name the hardware category that resolves the conflict.
- Define two service function chains for a branch office, one for corporate traffic and one for guest Wi-Fi, and state what would be needed to build the same thing from physical appliances.
- Interview-style: A virtual firewall benchmarks at 9.4 Gb/s with 1500-byte packets and 0.6 Gb/s with 64-byte packets, and the team concludes the NIC is faulty. (Hint: convert both figures to packets per second and compare against a per-packet cycle budget rather than a bit rate.)
Virtual Switches and Bridges
Theory
A hypervisor hosting forty virtual machines has a problem the physical network cannot solve for it: those forty machines need to communicate with each other and with the outside world, but they share one or two physical NICs. Something inside the host must perform switching, and that something is a virtual switch.
At its simplest this is straightforward. Each VM or container gets a virtual interface, typically one end of a veth pair (a software cable: two interfaces where anything entering one emerges from the other) or a tap device for a VM. The other ends attach to a software bridge that learns MAC addresses and forwards frames exactly as a physical switch does. A Linux bridge does precisely this and little else, and for many purposes it is entirely sufficient.
The reason virtual switching became a substantial discipline is that the virtual switch is the one place in the infrastructure that sits between every workload and the network. That position makes it the natural home for a great deal more than forwarding: tenant isolation, security policy, overlay encapsulation, QoS, and per-workload telemetry all belong where every packet must pass and where the workload's identity is still known. Open vSwitch (OVS) is the canonical implementation — a programmable switch supporting OpenFlow, VXLAN and GENEVE encapsulation, and per-port policy.
Understanding OVS requires understanding its two-tier architecture, because it explains most of its performance characteristics. The slow path lives in user space (ovs-vswitchd) and evaluates the full OpenFlow pipeline: all tables, all priorities, all wildcards. This is flexible and costs tens of microseconds per packet. The fast path is a kernel module (or a DPDK userspace datapath) holding a cache of exact-match and "megaflow" entries. The first packet of a flow misses the cache, takes an upcall to user space for a full evaluation, and the resulting decision is installed in the cache so that every subsequent packet of that flow is handled in about a microsecond.
The operational consequence is important and frequently surprises people: a virtual switch's cost depends on flow arrival rate, not on bandwidth. A hundred long-lived connections moving gigabytes cost almost nothing, because the cache hit rate approaches 100%. A hundred thousand short connections moving the same bytes are dramatically more expensive, because every one pays an upcall. Port scans, short RPC bursts, and DNS-heavy workloads are the pathological cases, and cache hit rate is the metric to watch.
The modern alternative is worth knowing because it is displacing OVS in container environments. eBPF-based datapaths (as used by Cilium) attach programs directly at the driver or socket layer and implement forwarding, policy, and load balancing as verified kernel programs with hash-map lookups. There is no separate userspace pipeline to miss into, so the flow-arrival sensitivity largely disappears, and for container-to-container traffic the path can bypass the bridge and the network stack almost entirely.
| Linux bridge | Open vSwitch | eBPF datapath | |
|---|---|---|---|
| Forwarding model | MAC learning | OpenFlow tables | verified programs + maps |
| Overlay support | none (needs separate device) | VXLAN, GENEVE, GRE built in | VXLAN, GENEVE |
| Policy | iptables alongside | flow rules, per-port | programs, identity-based |
| Cost driver | bandwidth | flow arrival rate | mostly bandwidth |
| Typical use | simple KVM hosts, containers | OpenStack, NSX, SDN fabrics | Kubernetes (Cilium) |
Examples
Inside one hypervisor host
VM A VM B container C
+------+ +------+ +------------+
| eth0 | | eth0 | | eth0 |
+--+---+ +--+---+ +-----+------+
| tap0 | tap1 | veth pair (one end here,
| | | other end in the netns)
+--+---------------+------------------+-----------+
| VIRTUAL SWITCH (OVS / bridge) |
| learns MACs, applies policy, encapsulates |
+---------------------+---------------------------+
| patch / uplink port
+----+----+
| ens1f0 | physical NIC
+----+----+
|
[ physical fabric ]
Why so much lives here: this is the ONLY point that sees every packet
while the workload's identity is still known. Past the NIC it is just
an IP address.
OVS two-tier lookup:
packet -> [ kernel datapath cache ] HIT -> action ~1.5 us
| MISS (upcall)
v
[ ovs-vswitchd, full OpenFlow pipeline ] ~40 us
| installs a cache entry
+-> every later packet of this flow is a HIT
# Why a virtual switch needs a CACHE: the first packet is expensive, the rest must not be.
SLOW_PATH_US = 40.0 # full OpenFlow pipeline evaluation in userspace
FAST_PATH_US = 1.5 # exact-match cache hit in the kernel datapath
class Datapath:
"""Two-tier lookup: an exact-match cache in front of the real pipeline."""
def __init__(self):
self.cache, self.slow, self.fast = {}, 0, 0
def key(self, pkt):
return (pkt["src"], pkt["dst"], pkt["proto"], pkt["dport"])
def process(self, pkt):
k = self.key(pkt)
if k in self.cache:
self.fast += 1
return self.cache[k], FAST_PATH_US
self.slow += 1 # UPCALL to userspace
action = "output:2" if pkt["dport"] != 25 else "drop"
self.cache[k] = action # install: later packets are fast
return action, SLOW_PATH_US
dp = Datapath()
total_us = 0.0
for flow in range(200): # 200 flows, 100 packets each
for _ in range(100):
pkt = dict(src=f"10.0.0.{flow % 250}", dst="10.0.1.5", proto=6,
dport=443 if flow % 10 else 25)
_, us = dp.process(pkt)
total_us += us
packets = dp.slow + dp.fast
print(f"packets {packets:,} cache misses {dp.slow} hits {dp.fast:,} "
f"hit rate {100*dp.fast/packets:.1f}%")
print(f"total time {total_us/1000:.2f} ms mean {total_us/packets:.2f} us/packet")
all_slow = packets * SLOW_PATH_US
print(f"if every packet took the slow path: {all_slow/1000:.2f} ms "
f"({all_slow/total_us:.1f}x worse)")
# packets 20,000 cache misses 200 hits 19,800 hit rate 99.0%
# total time 37.70 ms mean 1.89 us/packet
# if every packet took the slow path: 800.00 ms (21.2x worse)
print("\nCost is driven by FLOW ARRIVALS, not by bandwidth:")
for hit_rate in (0.99, 0.90, 0.50, 0.0):
mean = hit_rate * FAST_PATH_US + (1 - hit_rate) * SLOW_PATH_US
print(f" cache hit rate {hit_rate:5.0%} -> {mean:5.2f} us/packet, "
f"{1500*8/(mean*1e-6)/1e9:5.2f} Gb/s per core")
# cache hit rate 99% -> 1.89 us/packet, 6.37 Gb/s per core
# cache hit rate 90% -> 5.35 us/packet, 2.24 Gb/s per core
# cache hit rate 50% -> 20.75 us/packet, 0.58 Gb/s per core
# cache hit rate 0% -> 40.00 us/packet, 0.30 Gb/s per core
#
# The same bytes in a few long flows vs many short ones differ by 20x. This is
# why a port scan can hurt a hypervisor far more than a bulk transfer.
# --- The simple case: a Linux bridge and a veth pair ---
ip link add br-test type bridge && ip link set br-test up
ip netns add app1
ip link add veth-host type veth peer name veth-app # a software cable, two ends
ip link set veth-app netns app1 # move one end into the namespace
ip link set veth-host master br-test up # attach the other to the bridge
ip netns exec app1 ip addr add 10.10.0.2/24 dev veth-app
ip netns exec app1 ip link set veth-app up
bridge link show # which interfaces are enslaved
bridge fdb show br br-test # the learned MAC table -- a real switch's table
# 02:42:ac:11:00:02 dev veth-host master br-test
# --- Open vSwitch: the programmable case ---
ovs-vsctl add-br br-int
ovs-vsctl add-port br-int vnet0 # a VM's tap device
ovs-vsctl add-port br-int vxlan0 -- set interface vxlan0 type=vxlan \
options:remote_ip=10.255.0.2 options:key=10100 # overlay, built into the switch
ovs-vsctl show
# THE performance metric for OVS: cache hit rate.
ovs-dpctl show
# lookups: hit:48213991 missed:118402 lost:0
# ^ hit/(hit+missed) = 99.75%. Falling hit rate = rising CPU, and the cause
# is usually a short-flow workload, not a bandwidth increase.
ovs-dpctl dump-flows | head -3 # the CACHED megaflows, not the OpenFlow rules
ovs-ofctl dump-flows br-int # the OpenFlow rules, evaluated only on a miss
# Per-port QoS and mirroring: things a physical switch would charge for.
ovs-vsctl set interface vnet0 ingress_policing_rate=100000 # kb/s
ovs-vsctl -- set bridge br-int mirrors=@m \
-- --id=@m create mirror name=mymirror select-all=true output-port=@snooper
Key takeaways
- Every hypervisor needs internal switching because many workloads share few physical NICs; veth pairs and tap devices connect them to a software bridge.
- The virtual switch is the only point that sees every packet while the workload's identity is still known, which is why isolation, policy, encapsulation, QoS, and telemetry all live there.
- Open vSwitch has a slow path (full OpenFlow pipeline in user space, tens of microseconds) and a fast path (kernel cache, about a microsecond), linked by an upcall on the first packet of each flow.
- Consequently a virtual switch's cost tracks flow arrival rate, not bandwidth — the same bytes can cost 20x more as many short flows than as a few long ones.
- Cache hit rate is the metric to monitor; a falling hit rate with steady bandwidth means the workload's flow profile changed.
- eBPF datapaths attach verified programs at the driver or socket layer, removing the userspace-miss penalty, and now dominate container networking.
🧪 Practice
- Create a bridge, a network namespace, and a veth pair connecting them, then show the commands that verify the MAC table learned the namespace's address.
- An OVS host reports
hit:12000000 missed:4000000. Compute the hit rate, estimate the mean per-packet cost using 1.5 and 40 microseconds, and say what workload change would explain it.- Explain why a virtual switch is the correct place to enforce tenant isolation rather than the first physical switch.
- Interview-style: A hypervisor's CPU usage doubles overnight while its network throughput is unchanged. (Hint: the relevant counter is not bytes or packets per second — think about what the switch does differently for the first packet of a connection, and what kind of workload maximises that.)
Tunneling: GRE, VXLAN, GENEVE
Theory
A tunnel is the mechanism that makes an overlay possible, and the idea is simple enough to state in one sentence: take a complete packet and treat it as the payload of another packet. The original packet becomes opaque cargo; the network forwards the outer packet using ordinary rules and delivers it to a device that strips the outer headers and releases the inner packet, which then continues as though it had been transmitted locally.
The reason this is powerful is that it decouples addressing from location. The inner packet can use any addresses the tenant likes, including addresses that duplicate another tenant's, because the physical network never examines them. The analogy is shipping containers: the contents can be anything, in any arrangement, and the shipping line routes only the container by its own identifier. It neither knows nor cares what is inside.
The three encapsulations in common use differ in what metadata they carry and how well they work with the network underneath:
GRE (Generic Routing Encapsulation) is the oldest and simplest: an IP header plus a 4-byte GRE header, optionally with a key field for demultiplexing. Its weakness is that it is its own IP protocol number (47), so the underlay sees no ports. Every packet between the same two endpoints looks identical to a load-balancing hash, so all of it pins to one path, and middleboxes frequently mishandle it.
VXLAN (RFC 7348) fixed the practical problems by encapsulating in UDP. That single choice brings three benefits. It carries a 24-bit VNI (VXLAN network identifier), giving about 16.7 million segments instead of the 4,096 that a 12-bit VLAN tag allows — the constraint that made VLANs unusable for multi-tenant clouds. It traverses existing equipment, because everything handles UDP. And critically, the encapsulator sets the outer UDP source port from a hash of the inner flow, which gives the underlay's ECMP hash the entropy it needs to spread traffic across parallel paths. Without that, a leaf-spine fabric with four spines would carry all traffic between a pair of hosts on one spine.
GENEVE (RFC 8926) is the current direction. It keeps VXLAN's UDP framing and 24-bit VNI and adds extensible TLV options, so metadata beyond the segment identifier can travel with the packet — security labels, source group identity, telemetry. VXLAN's header is fixed, so extending it required new protocols each time; GENEVE was designed on the assumption that requirements would keep changing.
| GRE | VXLAN | GENEVE | |
|---|---|---|---|
| Outer transport | IP proto 47 | UDP 4789 | UDP 6081 |
| Segment ID | 32-bit key (optional) | 24-bit VNI | 24-bit VNI |
| Extensible metadata | no | no | yes, TLV options |
| Underlay ECMP entropy | poor | good (hashed UDP source port) | good |
| Typical overhead | 24 B | 50 B | 50 B and up |
Now the part that causes more production incidents than everything else in this chapter combined: MTU. Encapsulation adds bytes. A VXLAN-encapsulated Ethernet frame carries 50 extra bytes, so a 1500-byte inner packet becomes a 1550-byte outer packet. If the underlay MTU is also 1500, that packet cannot be transmitted. The resulting symptom is diagnostic poison: small packets work perfectly, so ping succeeds, DNS resolves, and SSH connects — and any bulk transfer stalls, because full-size segments are the ones that fail. Teams routinely spend days on this.
The fixes, in order of preference:
- Raise the underlay MTU to 9000 (jumbo frames) and leave workloads at 1500. This is the right answer in any network you control, and it is why data centre fabrics are built with jumbo frames.
- Lower the workload MTU to 1450 so the encapsulated packet fits. Correct, but requires configuring every workload.
- Clamp TCP MSS at the tunnel endpoint. Helps TCP only; UDP applications still break.
Examples
Encapsulation: the same bytes, wrapped
what the VM sent (inner):
+----------------+----------+--------------------------+
| eth dst/src | IP 10.1.0.5 -> 10.1.0.9 | payload |
+----------------+----------+--------------------------+
what crosses the physical fabric (outer):
+-------------+---------------------+-----------+--------+---------------+
| outer eth | outer IP | outer UDP | VXLAN | ENTIRE inner |
| | 10.255.0.1 -> | sport= | VNI= | frame above, |
| | 10.255.0.2 | hash(inner)| 10100| untouched |
+-------------+---------------------+-----------+--------+---------------+
^ VTEP addresses, routed normally ^ the ECMP entropy source
The fabric never looks past this.
Header layouts:
GRE [ IP(20) ][ GRE(4) ][ payload ] 24 B
VXLAN [ IP(20) ][ UDP(8) ][ VXLAN(8) ][ inner eth(14) ][...] 50 B
GENEVE [ IP(20) ][ UDP(8) ][ GENEVE(8+opts) ][ inner eth(14) ] 50 B+
The MTU trap, drawn:
inner 1500 B + VXLAN 50 B = 1550 B outer
|
underlay MTU = 1500 B
v
DOES NOT FIT -> fragment or drop
ping (84 B) -> 134 B -> fits -> "the network is fine"
ssh (small) -> fits -> "the network is fine"
scp (1500 B) -> 1550 B -> DROPS -> "the network is broken"
^ this asymmetry is why the bug survives so long
# Encapsulation overhead and the MTU arithmetic that breaks real deployments.
INNER_MTU_DEFAULT, UNDERLAY_MTU, JUMBO_MTU = 1500, 1500, 9000
INNER_ETH = 14 # L2 overlays carry the inner Ethernet header too
ENCAPS = {
# name: (outer_l3, outer_l4, tunnel_hdr, description)
"GRE": (20, 0, 4, "IP + GRE"),
"GRE+key": (20, 0, 8, "IP + GRE with key field"),
"VXLAN": (20, 8, 8, "IP + UDP + VXLAN, 24-bit VNI"),
"GENEVE": (20, 8, 8, "IP + UDP + GENEVE base header, no options"),
"GENEVE+opt": (20, 8, 24, "GENEVE with 16 B of TLV options"),
"IPsec ESP": (20, 0, 38, "IP + ESP + IV + pad + ICV (AES-GCM, tunnel mode)"),
"VXLAN+IPsec": (40, 8, 46, "VXLAN inside ESP: the overhead trap"),
}
print(f"{'encapsulation':14s} {'overhead':>9s} {'usable inner MTU':>17s} "
f"{'jumbo underlay':>15s} notes")
for name, (l3, l4, tun, desc) in ENCAPS.items():
oh = l3 + l4 + tun + (INNER_ETH if ("VXLAN" in name or "GENEVE" in name) else 0)
print(f"{name:14s} {oh:6d} B {UNDERLAY_MTU - oh:11d} B "
f"{JUMBO_MTU - oh:11d} B {desc}")
# encapsulation overhead usable inner MTU jumbo underlay notes
# GRE 24 B 1476 B 8976 B IP + GRE
# GRE+key 28 B 1472 B 8972 B IP + GRE with key field
# VXLAN 50 B 1450 B 8950 B IP + UDP + VXLAN, 24-bit VNI
# GENEVE 50 B 1450 B 8950 B IP + UDP + GENEVE base header, no options
# GENEVE+opt 66 B 1434 B 8934 B GENEVE with 16 B of TLV options
# IPsec ESP 58 B 1442 B 8942 B IP + ESP + IV + pad + ICV (AES-GCM, tunnel mode)
# VXLAN+IPsec 108 B 1392 B 8892 B VXLAN inside ESP: the overhead trap
vxlan_oh = 20 + 8 + 8 + INNER_ETH
print(f"\nThe failure mode: a VM keeps the default {INNER_MTU_DEFAULT} B MTU on a"
f" VXLAN overlay whose underlay is also {UNDERLAY_MTU} B.")
print(f" outer frame = {INNER_MTU_DEFAULT} + {vxlan_oh} = "
f"{INNER_MTU_DEFAULT + vxlan_oh} B > {UNDERLAY_MTU} B -> fragment or drop")
print(" small packets fit, so ping and SSH succeed while bulk transfer hangs.")
print("\nThe three fixes, in order of preference:")
print(f" 1. underlay jumbo frames: underlay MTU {JUMBO_MTU}, workload stays 1500")
print(f" 2. lower the inner MTU: set workload MTU to {UNDERLAY_MTU - vxlan_oh}")
print(f" 3. clamp TCP MSS: MSS = {UNDERLAY_MTU - vxlan_oh - 40} "
f"(TCP only; UDP still breaks)")
# 1. underlay jumbo frames: underlay MTU 9000, workload stays 1500
# 2. lower the inner MTU: set workload MTU to 1450
# 3. clamp TCP MSS: MSS = 1410 (TCP only; UDP still breaks)
print(f"\nTenant identifier space -- the constraint that retired VLANs:")
for name, bits in (("802.1Q VLAN", 12), ("VXLAN VNI", 24), ("GENEVE VNI", 24)):
print(f" {name:14s} {bits:2d} bits -> {2**bits:>10,} segments")
# 802.1Q VLAN 12 bits -> 4,096 segments
# VXLAN VNI 24 bits -> 16,777,216 segments
# GENEVE VNI 24 bits -> 16,777,216 segments
# --- Create tunnels of each kind on Linux ---
# GRE: simple, but no ports for the underlay to hash on.
ip link add gre1 type gre local 192.0.2.1 remote 192.0.2.2 ttl 255
ip link set gre1 mtu 1476 up # 1500 - 24 B of overhead
# VXLAN: UDP-encapsulated, 24-bit VNI, learns remote VTEPs or is programmed.
ip link add vxlan100 type vxlan id 100 local 10.255.0.1 dstport 4789 \
dev ens1f0 nolearning
ip link set vxlan100 mtu 1450 up # 1500 - 50 B of overhead
bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 10.255.0.2 # remote VTEP
# GENEVE: same framing, extensible options.
ip link add gnv0 type geneve id 100 remote 10.255.0.2 dstport 6081
ip link set gnv0 mtu 1450 up
# --- THE diagnostic for the MTU bug: bisect with DF set ---
# Find the largest packet that survives the path, without fragmentation.
ping -M do -s 1472 -c 2 10.1.0.9 # 1472 + 28 = 1500 total
# PING ... 1472(1500) bytes of data.
# ping: local error: message too long, mtu=1450
# ^ the path MTU is 1450: this is the overlay overhead, confirmed
ping -M do -s 1422 -c 2 10.1.0.9 # 1422 + 28 = 1450 -> succeeds
tracepath 10.1.0.9 # reports where the MTU drops
# --- The fixes ---
ip link set dev ens1f0 mtu 9000 # 1: jumbo underlay (preferred)
ip link set dev eth0 mtu 1450 # 2: lower the workload MTU
iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
-j TCPMSS --clamp-mss-to-pmtu # 3: clamp MSS (TCP only)
# Verify the overhead is really what you think, by capturing the OUTER packet.
tcpdump -ni ens1f0 -v 'udp port 4789' -c 2
# IP 10.255.0.1.51423 > 10.255.0.2.4789: VXLAN, flags [I] (0x08), vni 100
# ^ note the ephemeral source port: that is the ECMP entropy, hashed per inner flow
# IP 10.1.0.5 > 10.1.0.9: ICMP echo request <- the inner packet, intact
Key takeaways
- A tunnel makes a whole packet the payload of another packet, which decouples addressing from location and lets tenants reuse addresses freely.
- GRE is simplest but uses its own IP protocol number, so it gives the underlay no port entropy and pins traffic to one path.
- VXLAN encapsulates in UDP, carries a 24-bit VNI (16.7 million segments versus a VLAN's 4,096), and sets the outer UDP source port from a hash of the inner flow so ECMP can spread traffic.
- GENEVE keeps that design and adds extensible TLV options, so metadata beyond the segment ID can travel with the packet.
- Encapsulation costs bytes: about 50 for VXLAN or GENEVE, and over 100 when stacked inside IPsec.
- The MTU mismatch is the most common overlay failure, and its signature is that small packets work while bulk transfers stall.
- Fix it by raising the underlay MTU to 9000 if you can, otherwise lowering the workload MTU, with MSS clamping as a TCP-only fallback.
- Diagnose it with
ping -M doat descending sizes to find the real path MTU.
🧪 Practice
- Compute the usable inner MTU for GENEVE with 32 bytes of options over a 1500-byte underlay, and again over a 9000-byte underlay.
- Explain why GRE between two hosts cannot use more than one path in a four-way ECMP fabric, and what VXLAN does differently.
- Given a report that "SSH works but file copies hang" on a new overlay, write the exact sequence of commands you would run to confirm the cause.
- Interview-style: A team runs VXLAN inside an IPsec tunnel and reports intermittent failures that correlate with large requests. (Hint: add up both layers' overheads, compare against the path MTU, and consider what each layer does when an oversized packet arrives with the don't-fragment bit set.)
Overlay vs Underlay Networks
Theory
Once tunnels exist, a network has two distinct planes stacked on top of one another, and keeping them conceptually separate is the single most useful habit for operating or debugging such an environment.
The underlay is the physical network: real switches, real cables, real IP addresses assigned to real interfaces. Its only job is to carry packets between tunnel endpoints. It should be simple, stable, and boring — usually a routed leaf-spine fabric with a single address family, ECMP, and no tenant awareness whatsoever. It changes when hardware changes, which is to say rarely.
The overlay is the logical network the workloads see: their addresses, their segments, their security policy. It is created and destroyed constantly as workloads come and go, and it is entirely software.
The relationship is worth stating precisely, because it is what makes the whole design work: the underlay knows nothing about tenants, and the overlay knows nothing about topology. The underlay sees only tunnel endpoint addresses and has no idea how many tenants or segments exist. The overlay sees a flat logical segment and has no idea which rack anything is in. Each is free to change without disturbing the other, which is exactly the decoupling that lets a workload migrate between racks without renumbering.
The analogy is a postal system and a company's internal mail. The postal service routes between building addresses and knows nothing about departments. Internal mail is addressed to departments and knows nothing about streets. A department can move to a different building without changing anyone's internal addressing, and the postal service needs no knowledge of the reorganisation.
The overlay's central problem is mapping: given a logical destination, which tunnel endpoint currently holds it? Filling that table is the job of the overlay control plane, and there are three approaches:
| Approach | How the map is filled | Trade-off |
|---|---|---|
| Flood and learn | multicast or unicast-replicate unknown traffic, learn from replies | no control plane needed; floods badly, scales poorly |
| Controller-programmed | a controller pushes the map to every endpoint | precise and fast; requires a controller |
| EVPN (BGP) | endpoints advertise their local MAC and IP bindings via BGP | standards-based, scales, multi-vendor; today's default |
EVPN has become the standard answer because it reuses BGP, which is already proven at scale, and because advertising bindings explicitly eliminates flooding: a switch that learns a MAC locally advertises it, so every other switch knows the mapping without ever having to flood to discover it. It also handles ARP suppression, which removes a surprising amount of broadcast traffic from large segments.
Two operational consequences deserve emphasis.
First, debugging becomes two-layer, and you must know which layer you are in. A workload-to-workload failure could be an overlay mapping problem or an underlay reachability problem, and the diagnosis differs completely. The discipline is to test the underlay first — can the tunnel endpoints reach each other, at full MTU? — and only then the overlay. Tools that work at one layer are blind at the other, which is why a traceroute between two VMs across a VXLAN fabric shows a single hop: the entire physical path is invisible by design.
Second, the overlay inherits every underlay problem while hiding its cause. Congestion on one spine link appears as unexplained latency for a subset of overlay flows, with nothing in the overlay to indicate why. An underlay MTU shortfall appears as stalled transfers. This is the price of abstraction, and it is why underlay monitoring must stay first-class even though no workload traffic is addressed to it.
Examples
Two planes, deliberately ignorant of each other
OVERLAY (what workloads see)
VM-A 10.1.0.5 <------------ one logical segment, VNI 10100 ------> VM-C 10.1.0.9
^ same subnet, "adjacent", traceroute shows ONE hop
| neither VM knows a fabric exists
....................................................................
UNDERLAY (what actually happens)
VTEP 10.255.0.1 VTEP 10.255.0.2
[ leaf-1 ]---\ /---[ leaf-2 ]
>--[ spine-1 ]--[ spine-2 ]--<
/ \
^ routed, ECMP, jumbo MTU, ZERO tenant awareness
Who knows what:
underlay knows: 10.255.0.1 and 10.255.0.2 exist, and 4 paths between them
underlay does NOT know: tenants, VNIs, VM addresses, policy
overlay knows: VM-A and VM-C are in VNI 10100, and which VTEP holds each
overlay does NOT know: racks, spines, link utilisation, physical paths
Debugging order (get this wrong and you waste hours):
1. underlay reachability at FULL MTU: ping -M do -s 8972 10.255.0.2
2. underlay path health: which of the 4 paths, any drops?
3. overlay mapping: is VM-C's MAC/IP known to leaf-1?
4. overlay policy: is the traffic permitted?
# An overlay is a MAPPING problem: logical identity -> physical location.
import hashlib
from collections import Counter
class Fabric:
"""VTEPs hold the map. The underlay only ever routes VTEP-to-VTEP."""
def __init__(self):
self.mac_to_vtep = {} # the overlay control plane's entire job
self.underlay = {} # VTEP IP -> which rack it sits in
def learn(self, vni, mac, vtep, rack):
self.mac_to_vtep[(vni, mac)] = vtep # NOTE: the key includes the VNI
self.underlay[vtep] = rack
def forward(self, vni, src_mac, dst_mac):
src_vtep = self.mac_to_vtep.get((vni, src_mac))
dst_vtep = self.mac_to_vtep.get((vni, dst_mac))
if dst_vtep is None:
return ("BUM", f"unknown MAC in VNI {vni}: replicate to all VTEPs in VNI")
if src_vtep == dst_vtep:
return ("LOCAL", f"both on {src_vtep}: never touches the underlay")
return ("ENCAP", f"{src_vtep} -> {dst_vtep} "
f"(rack {self.underlay[src_vtep]} -> {self.underlay[dst_vtep]}), "
f"outer dst IP = {dst_vtep}, VNI {vni}")
f = Fabric()
f.learn(10100, "aa:00:00:00:00:01", "10.255.0.1", "rack-1")
f.learn(10100, "aa:00:00:00:00:02", "10.255.0.1", "rack-1")
f.learn(10100, "aa:00:00:00:00:03", "10.255.0.2", "rack-2")
f.learn(10200, "bb:00:00:00:00:01", "10.255.0.3", "rack-3")
for label, vni, s, d in (
("same VTEP", 10100, "aa:00:00:00:00:01", "aa:00:00:00:00:02"),
("across the fabric", 10100, "aa:00:00:00:00:01", "aa:00:00:00:00:03"),
("unknown MAC", 10100, "aa:00:00:00:00:01", "aa:00:00:00:00:99"),
("other tenant", 10200, "bb:00:00:00:00:01", "aa:00:00:00:00:03")):
kind, why = f.forward(vni, s, d)
print(f"{label:19s} {kind:6s} {why}")
# same VTEP LOCAL both on 10.255.0.1: never touches the underlay
# across the fabric ENCAP 10.255.0.1 -> 10.255.0.2 (rack rack-1 -> rack-2), outer dst IP = 10.255.0.2, VNI 10100
# unknown MAC BUM unknown MAC in VNI 10100: replicate to all VTEPs in VNI
# other tenant BUM unknown MAC in VNI 10200: replicate to all VTEPs in VNI
#
# The last line is the isolation property: tenant 10200 cannot reach tenant
# 10100's MAC at all, because the lookup key INCLUDES the VNI. Two tenants may
# use identical MAC and IP addresses without colliding.
# Why the underlay needs entropy, and where VXLAN gets it.
def ecmp_hash(outer_src, outer_dst, udp_src, n_paths):
key = f"{outer_src}|{outer_dst}|{udp_src}".encode()
return int(hashlib.md5(key).hexdigest(), 16) % n_paths
fixed = Counter(ecmp_hash("10.255.0.1", "10.255.0.2", 4789, 4) for _ in range(1000))
varied = Counter(ecmp_hash("10.255.0.1", "10.255.0.2", 30000 + i % 997, 4)
for i in range(1000))
print(f"\nouter UDP sport FIXED -> path distribution {dict(sorted(fixed.items()))}")
print(f"outer UDP sport HASHED -> path distribution {dict(sorted(varied.items()))}")
# outer UDP sport FIXED -> path distribution {1: 1000}
# outer UDP sport HASHED -> path distribution {0: 266, 1: 273, 2: 228, 3: 233}
#
# With a fixed source port, every flow between two VTEPs pins to ONE underlay
# path: a 4-way spine collapses to a single link. This is exactly why the
# encapsulator hashes the INNER flow into the OUTER source port.
# --- Underlay first, always. Test endpoint-to-endpoint at FULL MTU ---
ping -M do -s 8972 -c 3 10.255.0.2 # 8972 + 28 = 9000; must succeed on jumbo
ip route get 10.255.0.2 # which underlay path, which interface
vtysh -c 'show bgp ipv4 unicast 10.255.0.2/32' # is the VTEP route even present?
# --- Then the overlay map ---
bridge fdb show | grep vxlan100 # MAC -> remote VTEP bindings
# aa:00:00:00:00:03 dev vxlan100 dst 10.255.0.2 self <- learned remotely
# 00:00:00:00:00:00 dev vxlan100 dst 10.255.0.2 self <- BUM replication entry
ip neigh show dev vxlan100 # ARP suppression cache
# EVPN: the modern control plane. Who advertised this MAC?
vtysh -c 'show bgp l2vpn evpn route type macip' | head
# *> [2]:[0]:[48]:[aa:00:00:00:00:03] 10.255.0.2 <- MAC advertised BY leaf-2
vtysh -c 'show evpn mac vni 10100'
vtysh -c 'show evpn vni detail'
# --- Why traceroute lies in an overlay ---
ip netns exec vm-a traceroute -n 10.1.0.9
# 1 10.1.0.9 0.412 ms <- ONE hop: the whole fabric is invisible by design
# The physical path must be traced in the underlay instead:
traceroute -n 10.255.0.2
# 1 10.0.1.1 <- leaf
# 2 10.0.255.1 <- spine
# 3 10.255.0.2 <- remote VTEP
# --- Correlating a layer: which underlay link is a given overlay flow using? ---
# Capture the outer packet and read the hashed source port, then check that path.
tcpdump -ni ens1f0 -c 1 'udp port 4789 and host 10.255.0.2' 2>/dev/null
ethtool -S swp1 | grep -iE 'discard|drop|pause' # per-underlay-link health
Key takeaways
- The underlay is the physical network carrying packets between tunnel endpoints; keep it simple, routed, jumbo-enabled, and tenant-unaware.
- The overlay is the logical network workloads see; it changes constantly and is entirely software.
- The decoupling is mutual: the underlay knows nothing about tenants, and the overlay knows nothing about topology. That is what lets workloads migrate without renumbering.
- The overlay's core problem is mapping logical identity to tunnel endpoint. EVPN over BGP is the current standard, replacing flood-and-learn and eliminating unnecessary flooding.
- Including the VNI in the lookup key is what provides tenant isolation and lets tenants reuse identical addresses.
- Debug the underlay first, at full MTU, then the overlay map, then policy. Tools at one layer are blind to the other.
- A
traceroutebetween overlay endpoints shows one hop: the physical path is hidden by design, so underlay monitoring must stay first-class. - The overlay inherits underlay problems while concealing their cause, which is the real cost of the abstraction.
🧪 Practice
- State which plane each belongs to: a VNI, a spine switch's loopback address, a tenant's security group, an ECMP hash, a VTEP's MAC-to-endpoint table.
- Two VMs on the same overlay segment cannot communicate. Write the ordered diagnostic steps, naming the command and the expected result at each step.
- Explain how EVPN removes the need to flood unknown unicast, and what it replaces the flooding with.
- Interview-style: Overlay traffic between two racks shows 3% loss while every overlay counter looks clean and the underlay reports no link failures. (Hint: there are several parallel underlay paths and the overlay flow uses exactly one of them — work out how to discover which, and what per-link counter would reveal a problem the aggregate hides.)
Container Networking and CNI
Theory
Containers changed the network's requirements more than virtual machines did, for reasons of scale and lifetime. A VM lives for months and is provisioned deliberately; a container may live for seconds, and a cluster may create and destroy thousands per minute. A networking model that requires any human action, or any change to physical configuration, is simply not viable.
The original Docker model was port mapping: containers shared the host's network namespace or sat behind NAT, and reaching a container meant reaching a port on its host. This works for a handful of containers on one machine and collapses as a general model, because ports become a globally contended resource, containers cannot use their natural ports, and addresses depend on placement.
Kubernetes replaced it with a model defined by three invariants, and almost everything about container networking follows from them:
- Every pod gets its own IP address, routable within the cluster.
- Pods can communicate without NAT.
- A node can reach all pods on that node without NAT.
These are strong requirements. They mean the cluster needs a flat address space large enough for every pod, and that the network must place and withdraw addresses at pod lifecycle speed. Kubernetes deliberately does not implement this. It defines the contract and delegates to a plugin, which is CNI (Container Network Interface).
CNI is notable for its minimalism. A plugin is an executable that the runtime invokes with a verb (ADD, DEL, CHECK), passing configuration as JSON on standard input and the target network namespace as an environment variable, and reading the result as JSON from standard output. That is the whole interface. On ADD the plugin must create the pod's interface, allocate an address, configure routes, and return what it did.
The work itself is the sequence you would perform by hand: create a network namespace, create a veth pair, move one end in and rename it eth0, allocate an IP, set the address and default route, and attach the host end to whatever datapath the plugin uses.
IPAM is where a design decision with real consequences hides. The usual approach gives each node a slice of the cluster CIDR — a /24 out of a /16, say — so that a node can allocate addresses locally without coordinating with anything. That is what makes allocation fast enough for container lifecycles. The cost is arithmetic that must be checked at design time: a /16 split into /24 blocks supports 256 nodes and about 253 pods per node, and neither number can be changed later without renumbering the cluster.
The plugins differ in how pods on different nodes reach each other:
| Plugin | Cross-node forwarding | Policy | Notes |
|---|---|---|---|
| Flannel | VXLAN overlay or host routes | none natively | simplest; minimal features |
| Calico | native routing with BGP, or overlay | rich, iptables or eBPF | no encapsulation needed if the underlay can route pod CIDRs |
| Cilium | eBPF, with VXLAN or native routing | identity-based, eBPF | also replaces kube-proxy; L7 aware |
| cloud plugins | the cloud's own VPC routing | cloud security groups | pods get real VPC addresses |
The distinction to internalise is overlay versus native routing. An overlay works anywhere because it only needs IP connectivity between nodes, and it costs encapsulation overhead and MTU complexity. Native routing avoids both by advertising pod CIDRs into the physical network with BGP, which is faster and more debuggable, and requires an underlay willing to carry those routes. That is a conversation with the network team, which is often the actual deciding factor.
One more piece completes the picture. Pod addresses are ephemeral, so clients cannot use them directly; a Service provides a stable virtual address that is load-balanced across the pods currently backing it. Implemented traditionally by kube-proxy writing iptables or IPVS rules on every node, and increasingly by eBPF programs, it is a per-node distributed load balancer rather than a central one, which is why there is no bottleneck and no single point of failure.
Examples
What a pod's network actually looks like
NODE (10.0.3.7) pod CIDR for this node: 10.244.3.0/24
+---------------------------------------------------------------+
| pod "web" netns pod "db" netns |
| +---------------------+ +---------------------+ |
| | eth0 10.244.3.2/24 | | eth0 10.244.3.3/24 | |
| | default via .3.1 | | default via .3.1 | |
| +----------+----------+ +----------+----------+ |
| | veth pair | veth pair |
| +----------+-------------------------------+----------+ |
| | bridge / eBPF datapath 10.244.3.1 (gateway) | |
| +--------------------------+--------------------------+ |
| | eth0 10.0.3.7 |
+-----------------------------+---------------------------------+
|
[ cluster network ]
|
reaching 10.244.5.0/24 (another node) happens either by
OVERLAY: encapsulate to that node's IP (VXLAN)
NATIVE ROUTING: the fabric knows 10.244.5.0/24 via BGP
The CNI contract, in full:
runtime --exec--> /opt/cni/bin/<plugin>
env: CNI_COMMAND=ADD, CNI_NETNS=/var/run/netns/..., CNI_IFNAME=eth0
stdin: { network configuration as JSON }
stdout: { interfaces, ips, routes, dns }
That is the entire API. No daemon, no gRPC, no library.
# What a CNI plugin actually does, as a sequence of concrete steps.
import ipaddress, json
class NodeIPAM:
"""Each node owns a slice of the cluster CIDR, so allocation needs NO coordination.
That is what makes it fast enough for container lifecycles."""
def __init__(self, cluster_cidr, node_prefix_len, node_index):
cluster = ipaddress.ip_network(cluster_cidr)
self.pod_cidr = list(cluster.subnets(new_prefix=node_prefix_len))[node_index]
hosts = list(self.pod_cidr.hosts())
self.gateway = hosts[0] # the node's bridge acts as gateway
self.free, self.leased = hosts[1:], {}
def allocate(self, pod):
ip = self.free.pop(0)
self.leased[pod] = ip
return ip
def capacity(self):
return len(self.free) + len(self.leased)
ipam = NodeIPAM("10.244.0.0/16", 24, 3) # node index 3 of a /16 cluster
print(f"cluster 10.244.0.0/16, /24 per node -> this node owns {ipam.pod_cidr}")
print(f" gateway {ipam.gateway}, pods per node {ipam.capacity()}, "
f"max nodes {2**(24-16)}")
# cluster 10.244.0.0/16, /24 per node -> this node owns 10.244.3.0/24
# gateway 10.244.3.1, pods per node 253, max nodes 256
#
# CHECK THIS AT DESIGN TIME: 256 nodes and 253 pods per node are both hard
# ceilings, and neither can change without renumbering the whole cluster.
print("\nCNI ADD, step by step:")
for i, s in enumerate([
"create a network namespace for the pod",
"create a veth pair: one end for the pod, one for the host",
"move the pod end into the netns and rename it eth0",
"allocate an IP from this node's pod CIDR (IPAM)",
"assign the IP and set the default route via the node gateway",
"attach the host end to the bridge / program the eBPF datapath",
"return the result as JSON on stdout"], 1):
print(f" {i}. {s}")
for pod in ("web-7f8c", "db-2a1b", "cache-9d3e"):
print(f" allocated {ipam.allocate(pod)} to {pod}")
# allocated 10.244.3.2 to web-7f8c
# allocated 10.244.3.3 to db-2a1b
# allocated 10.244.3.4 to cache-9d3e
result = {"cniVersion": "1.0.0",
"interfaces": [{"name": "eth0", "sandbox": "/var/run/netns/cni-web-7f8c"}],
"ips": [{"address": f"{ipam.leased['web-7f8c']}/24",
"gateway": str(ipam.gateway), "interface": 0}],
"routes": [{"dst": "0.0.0.0/0", "gw": str(ipam.gateway)}]}
print("\nCNI ADD result returned to the runtime:")
print(json.dumps(result, indent=2))
# {
# "cniVersion": "1.0.0",
# "interfaces": [ { "name": "eth0", "sandbox": "/var/run/netns/cni-web-7f8c" } ],
# "ips": [ { "address": "10.244.3.2/24", "gateway": "10.244.3.1", "interface": 0 } ],
# "routes": [ { "dst": "0.0.0.0/0", "gw": "10.244.3.1" } ]
# }
print("\nThe Kubernetes network model, stated as three invariants:")
for inv in ("every pod gets its own routable IP (no port mapping)",
"pods reach each other directly, without NAT",
"a node reaches all pods on that node without NAT"):
print(f" - {inv}")
print("Everything a CNI plugin does serves those three sentences.")
# --- Do by hand what a CNI plugin does, to see there is no magic ---
ip netns add demo
ip link add veth-h type veth peer name veth-c
ip link set veth-c netns demo
ip netns exec demo ip link set veth-c name eth0
ip netns exec demo ip addr add 10.244.3.99/24 dev eth0
ip netns exec demo ip link set eth0 up
ip netns exec demo ip route add default via 10.244.3.1
ip link set veth-h master cni0 up
ip netns exec demo ip addr show eth0 # exactly what a pod sees
# --- Inspect a real pod's network from the node ---
PID=$(crictl inspect --output go-template --template '{{.info.pid}}' "$CONTAINER_ID")
nsenter -t "$PID" -n ip addr show eth0
nsenter -t "$PID" -n ip route show
nsenter -t "$PID" -n ss -tlnp
# --- Which CNI is installed, and how is it configured? ---
cat /etc/cni/net.d/*.conflist | jq '.plugins[].type'
# "calico"
# "portmap" <- plugins CHAIN: each runs in order
ls /opt/cni/bin/
# --- Pod CIDR allocation: verify your ceilings before they bite ---
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDR}{"\n"}{end}'
kubectl cluster-info dump | grep -m1 cluster-cidr
# --cluster-cidr=10.244.0.0/16 -> 256 nodes at /24 per node
# --- Native routing vs overlay: which is in use? ---
ip route | grep 10.244
# 10.244.5.0/24 via 10.0.3.9 dev eth0 <- NATIVE routing (no encapsulation)
# 10.244.5.0/24 dev flannel.1 onlink <- OVERLAY (VXLAN device)
calicoctl node status # BGP peerings, if Calico is routing natively
# --- Services are per-node load balancers, not a central one ---
iptables -t nat -L KUBE-SERVICES -n | head
ipvsadm -Ln | head # if kube-proxy runs in IPVS mode
cilium service list # if eBPF replaced kube-proxy
Key takeaways
- Containers made port mapping untenable: ports became contended, and addresses depended on placement.
- The Kubernetes model is three invariants: a routable IP per pod, pod-to-pod without NAT, and node-to-local-pod without NAT.
- Kubernetes defines the contract and delegates to CNI, whose entire interface is an executable taking JSON on stdin and returning JSON on stdout.
- A plugin's
ADDcreates a netns and veth pair, allocates an address, sets routes, and attaches to a datapath. - IPAM gives each node a slice of the cluster CIDR so allocation needs no coordination; the resulting node and pod ceilings are fixed at design time.
- Overlay plugins work anywhere but cost encapsulation and MTU complexity; native routing with BGP is faster and more debuggable but requires an underlay that will carry pod routes.
- A Service supplies the stable virtual address that ephemeral pod IPs cannot, implemented as a per-node distributed load balancer via iptables, IPVS, or eBPF.
🧪 Practice
- A cluster uses
10.100.0.0/14with a/25per node. Compute the maximum node count and the pods per node, and state which limit binds first for 400 nodes of 60 pods.- Recreate a pod's network manually with
ip netnsandveth, then give the two commands that prove the namespace has an address and a default route.- Compare overlay and native-routing CNI modes on MTU, debuggability, and what each requires from the physical network.
- Interview-style: Pods on the same node communicate fine, but cross-node pod traffic fails for large payloads only. (Hint: identify what is added to a packet only when it leaves the node, then compare the totals against the node interface's MTU.)
Service Mesh and Sidecar Proxies
Theory
A microservice architecture needs a set of capabilities on every call between services: retries with sensible backoff, timeouts, circuit breaking, mutual authentication, authorisation, load balancing across instances, traffic splitting for canary releases, and metrics and traces for every request. The question is where that logic lives.
The first answer was libraries: each service imports a resilience and discovery library. It works, and it has a decisive flaw in a polyglot organisation — the library must be implemented and maintained for every language, and a policy change requires recompiling and redeploying every service. Upgrading a timeout default becomes a months-long migration across dozens of teams.
A service mesh moves the logic out of the process and into a proxy beside it. The sidecar pattern runs that proxy as a separate container in the same pod, and traffic is transparently redirected through it, usually by iptables rules or eBPF. The application makes an ordinary, unmodified call to a plain hostname; the proxy intercepts it, applies policy, establishes mutual TLS to the destination's proxy, load balances, retries, records metrics, and forwards.
The properties follow directly: the behaviour is language-agnostic because it operates on network traffic rather than inside the runtime, and it is uniform and centrally controllable because policy is configured once in the control plane and distributed to every proxy. This is the clean separation the mesh delivers: the data plane is the fleet of proxies carrying traffic, and the control plane configures them, holds the service identity and certificate authority, and distributes policy.
The costs are equally direct, and they are the reason meshes are not universal.
Latency. Every call crosses two proxies, one outbound and one inbound. Each adds a fraction of a millisecond at the median and more at the tail. On a single hop this is negligible; in a chain eight services deep, sixteen proxy traversals compound, and the tail latency added can exceed the actual work. Deep call chains are where meshes hurt most, and the right response is usually to shorten the chain rather than tune the proxy.
Resources. A sidecar per pod costs memory and CPU per pod. At a few hundred pods this is a rounding error. At several thousand, the sidecars alone can consume tens of gigabytes of RAM and hundreds of cores before any application code runs.
Operational complexity. The mesh becomes a critical dependency in the request path, with its own configuration language, failure modes, and upgrade process. A misconfigured retry policy can turn one slow service into a cluster-wide amplification event.
The industry response to the resource cost is the sidecar-less or ambient model: run a shared per-node component handling transport-level concerns such as mutual TLS and L4 routing, and insert a full L7 proxy only for the workloads that need L7 features. This trades some isolation for a substantial reduction in footprint.
The honest recommendation: a mesh earns its cost when you have many services in multiple languages and genuinely need uniform mutual TLS, fine-grained authorisation, and per-call observability. With a handful of services, or a single language where a library will do, it is usually more machinery than the problem requires.
Examples
Where the logic lives -- three answers
1. IN THE APPLICATION (library)
[ service A + retry/LB/mTLS library ] ---> [ service B + same library ]
+ no extra hop - per-language, redeploy to change policy
2. SIDECAR MESH
pod A pod B
+-------------------------+ +-------------------------+
| app ---> sidecar proxy |=mTLS=> | sidecar proxy ---> app |
+-------------------------+ +-------------------------+
^ iptables/eBPF redirect ^ policy applied here too
+ language agnostic, central policy - 2 proxies per hop, RAM per pod
3. AMBIENT / SIDECAR-LESS
node 1 node 2
+------------------------+ +------------------------+
| app A -> node proxy |==mTLS=> | node proxy -> app B |
+------------------------+ +------------------------+
+ one proxy per NODE, not per pod - weaker per-pod isolation
Control plane vs data plane in a mesh:
+---------------------------------------------+
| CONTROL PLANE: identity + CA, policy, |
| service discovery, config distribution |
+---------------------------------------------+
| xDS config pushed to every proxy
v v v
[ proxy ] [ proxy ] [ proxy ] <- DATA PLANE: carries the traffic
The control plane never touches a request.
# The sidecar tax: every hop through a proxy adds latency and consumes memory.
SIDECAR_P50_MS, SIDECAR_P99_MS = 0.5, 2.5 # per traversal, one direction
SIDECAR_MEM_MB, SIDECAR_CPU_MC = 50, 100 # per pod
def call_chain(hops, backend_ms):
"""A request crossing `hops` services passes 2 proxies per hop (out and in)."""
traversals = hops * 2
return (backend_ms + traversals * SIDECAR_P50_MS,
backend_ms + traversals * SIDECAR_P99_MS,
traversals)
print(f"{'hops':>5s} {'proxies':>8s} {'backend':>9s} {'p50':>9s} {'p99':>9s} "
f"{'p99 overhead':>13s}")
for hops in (1, 2, 3, 5, 8):
p50, p99, tr = call_chain(hops, 20.0)
print(f"{hops:5d} {tr:8d} {20.0:8.1f}ms {p50:8.1f}ms {p99:8.1f}ms "
f"{100*(p99-20.0)/20.0:11.1f}%")
# hops proxies backend p50 p99 p99 overhead
# 1 2 20.0ms 21.0ms 25.0ms 25.0%
# 2 4 20.0ms 22.0ms 30.0ms 50.0%
# 3 6 20.0ms 23.0ms 35.0ms 75.0%
# 5 10 20.0ms 25.0ms 45.0ms 125.0%
# 8 16 20.0ms 28.0ms 60.0ms 200.0%
#
# At 8 hops the proxies TRIPLE the p99. The fix is to shorten the call chain,
# not to tune the proxy.
for pods in (100, 1_000, 5_000):
print(f"{pods:>5,} pods -> {pods*SIDECAR_MEM_MB/1024:7.1f} GB RAM and "
f"{pods*SIDECAR_CPU_MC/1000:6.1f} CPU cores of sidecars, "
f"before any application code")
# 100 pods -> 4.9 GB RAM and 10.0 CPU cores of sidecars, before any application code
# 1,000 pods -> 48.8 GB RAM and 100.0 CPU cores of sidecars, before any application code
# 5,000 pods -> 244.1 GB RAM and 500.0 CPU cores of sidecars, before any application code
print("\nWhat that cost buys, and what you would otherwise build:")
for f, alt in {
"mutual TLS between all services": "per-service certs + rotation in every app",
"retries, timeouts, circuit breaking": "a resilience library per language",
"per-call metrics and tracing": "instrumentation in every codebase",
"traffic splitting / canary": "custom routing logic in every client",
"authorization by workload identity": "auth checks in every service"}.items():
print(f" {f:38s} otherwise: {alt}")
NODE_PROXY_MEM_MB = 200
print("\nAmbient (per-node proxy) instead of per-pod sidecars:")
for pods, nodes in ((1_000, 50), (5_000, 200)):
side, amb = pods*SIDECAR_MEM_MB/1024, nodes*NODE_PROXY_MEM_MB/1024
print(f" {pods:>5,} pods on {nodes:>3} nodes: sidecar {side:6.1f} GB vs "
f"ambient {amb:5.1f} GB ({side/amb:.0f}x less)")
# 1,000 pods on 50 nodes: sidecar 48.8 GB vs ambient 9.8 GB (5x less)
# 5,000 pods on 200 nodes: sidecar 244.1 GB vs ambient 39.1 GB (6x less)
# Traffic splitting for a canary release: the capability that justifies a mesh.
# The application code is unchanged and unaware.
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: reviews
spec:
hosts: ["reviews"] # the plain hostname the app already calls
http:
- match: # internal testers get v2 deterministically
- headers:
x-canary: {exact: "true"}
route:
- destination: {host: reviews, subset: v2}
- route: # everyone else: 95/5 weighted split
- destination: {host: reviews, subset: v1}
weight: 95
- destination: {host: reviews, subset: v2}
weight: 5
retries:
attempts: 3
perTryTimeout: 2s
retryOn: 5xx,reset,connect-failure
timeout: 10s # an OUTER bound: 3 x 2s tries must fit inside
---
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication # mutual TLS, enforced without touching any app
metadata:
name: default
spec:
mtls: {mode: STRICT}
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy # identity-based, not IP-based, authorisation
metadata:
name: db-access
spec:
selector:
matchLabels: {app: postgres}
rules:
- from:
- source:
principals: ["cluster.local/ns/prod/sa/api-server"]
to:
- operation: {ports: ["5432"]}
# Note what the identity is: a service account, not an address. Pod IPs change
# constantly, so IP-based policy cannot express this at all.
# --- How interception actually works: look at the pod's iptables ---
kubectl exec -it web-7f8c -c istio-proxy -- iptables -t nat -L ISTIO_OUTPUT -n
# REDIRECT tcp -- anywhere anywhere redir ports 15001
# ^ all outbound TCP is bent into the sidecar; the app never knows
# What configuration did this proxy actually receive from the control plane?
istioctl proxy-config routes web-7f8c --name 80 -o json | jq '.[].virtualHosts[0]'
istioctl proxy-config clusters web-7f8c | head
istioctl proxy-config endpoints web-7f8c | grep reviews
# Is the config in sync across the fleet? A stale proxy is a classic mesh bug.
istioctl proxy-status
# NAME CDS LDS EDS RDS
# web-7f8c SYNCED SYNCED SYNCED STALE <- this pod has old routes
# Verify mTLS is genuinely in use rather than merely configured.
istioctl authn tls-check web-7f8c reviews.prod.svc.cluster.local
# HOST STATUS SERVER CLIENT
# reviews.prod.svc.cluster.local OK mTLS mTLS
# Measure the sidecar tax on your own traffic before adopting it widely.
kubectl exec -it loadgen -- \
fortio load -qps 500 -t 30s -c 16 http://reviews:9080/
# Run it with the sidecar injected and with injection disabled, and compare p50
# and p99. The p99 difference is the number that matters, not the mean.
Key takeaways
- Cross-service concerns (retries, timeouts, mutual TLS, load balancing, metrics) must live somewhere; a library couples them to each language and requires redeploying every service to change policy.
- A service mesh moves them into a proxy beside the application, intercepting traffic transparently, which makes the behaviour language-agnostic and centrally configurable.
- The data plane is the proxies; the control plane distributes config, holds workload identity, and never touches a request.
- Latency is the main cost: two proxy traversals per hop, compounding in deep call chains — at eight hops the added tail can exceed the real work.
- Resources are the second cost: a sidecar per pod means tens of gigabytes and hundreds of cores at a few thousand pods.
- Ambient / sidecar-less designs use one proxy per node for L4 concerns and an L7 proxy only where needed, cutting footprint several-fold.
- Mesh authorisation uses workload identity, not IP addresses, which is the only thing that works when pod addresses change constantly.
- A mesh earns its cost with many services in multiple languages; below that, a library is usually the right answer.
🧪 Practice
- Compute the p50 and p99 added latency for a 6-hop call chain with 0.4 ms and 3 ms per proxy traversal, and express the p99 addition as a percentage of a 15 ms backend.
- Explain why mesh authorisation policies reference service accounts rather than IP addresses.
- Write the mesh configuration for a 90/10 canary split with 2 retries, a 1-second per-try timeout, and a 5-second overall timeout. Explain why the outer timeout must exceed the product of retries and per-try timeout.
- Interview-style: After a mesh rollout, a single slow downstream service causes a cluster-wide outage that did not occur before. (Hint: consider what the proxies were configured to do when a call fails, and what that does to the total request volume arriving at an already struggling service.)
<a id="133-data-center-networking"></a>
13.3 Data Center Networking
A data centre network serves a traffic pattern almost opposite to the one classical enterprise designs assumed: most bytes move between machines inside the building rather than to users outside it, and the flows are simultaneously enormous and microsecond-sensitive — this subchapter covers the topology that resulted, how traffic is spread across it, where load balancing belongs, and what it takes to carry storage and memory traffic over Ethernet.
Three-Tier vs Leaf-Spine Topology
Theory
The classical enterprise topology has three tiers: access switches where machines plug in, aggregation switches collecting groups of access switches, and a core connecting aggregation blocks. It was designed for a world in which most traffic went from a user's machine to somewhere outside the building, so the natural shape was a tree: traffic flows up toward the root and out.
That design has two properties which became fatal as traffic patterns changed.
First, hop count depends on placement. Two machines on the same access switch are one hop apart. Two machines under the same aggregation pair are three hops apart. Two machines in different aggregation blocks are five hops apart. If your application's performance depends on communication between components, it now depends on where those components physically landed, which is exactly the coupling that virtualisation and orchestration were supposed to eliminate. A scheduler placing a container cannot reason about a network whose latency varies by placement.
Second, the tree structure has redundant links that cannot be used. Ethernet forwarding requires a loop-free topology, and the classical solution was spanning tree, which achieves loop freedom by disabling links. Half the uplinks a data centre paid for sat idle waiting for a failure, and worse, a spanning tree reconvergence is a network-wide event measured in seconds.
Leaf-spine (a folded Clos network) restructures this. Every leaf switch connects to every spine switch, and leaves never connect to each other. The consequences follow mechanically from that wiring rule:
- Uniform hop count. Any leaf reaches any other leaf in exactly two hops through one spine. Within a rack it is one hop; between any two racks it is three switches. There is no "far" rack, so placement stops affecting latency.
- All links active. Since forwarding is routed at layer 3 rather than bridged, ECMP uses every spine simultaneously. There is no spanning tree and nothing is disabled.
- Predictable, incremental scaling. More servers means more leaves. More bandwidth between racks means more spines. Each is an independent axis.
- Small failure domains. A leaf failure affects one rack. A spine failure removes 1/N of the inter-rack capacity and nothing else — with four spines, losing one costs 25% of bandwidth and zero connectivity.
The analogy is the difference between a road network built as a hierarchy of feeder roads onto one motorway and a grid of equal streets. In the hierarchy, your journey time depends on which feeder you start from, and a motorway closure is catastrophic. In the grid, any point reaches any other by a similar route, and one closure diverts traffic without isolating anyone.
The design's cost is cabling and port count: every leaf needs a link to every spine, so cabling grows as leaves times spines, and a substantial share of every switch's ports are consumed by fabric links rather than servers. That is the price of non-blocking any-to-any capacity, and it is why fabric designs are often quoted in terms of how many server ports remain after uplinks.
For scale beyond one fabric, the pattern nests: pods of leaf-spine fabrics interconnected by a super-spine layer, which is a three-stage Clos, and the same reasoning applies one level up.
Examples
THREE-TIER: hop count depends on where you landed
[ core ]---[ core ]
/ \ / \
[ agg ] [ agg ] ... (another block)
/ \ / \
[acc] [acc] [acc] [acc]
||| ||| ||| |||
servers x---x = link disabled by
spanning tree
same access switch : 1 hop
same agg block : 3 hops
different agg block : 5 hops <- performance depends on PLACEMENT
half the uplinks : idle <- paid for, not usable
LEAF-SPINE: every leaf to every spine, never leaf to leaf
[spine-1] [spine-2] [spine-3] [spine-4]
| | \ / | | \ / | | \ / | |
| | \/ | | \/ | | \/ | |
| | /\ | | /\ | | /\ | |
[leaf-1] [leaf-2] [leaf-3] ... [leaf-N]
||| ||| ||| |||
servers servers servers servers
within a rack : 1 hop
ANY rack to ANY rack: 3 hops (leaf -> spine -> leaf), always
all 4 spine links : active, via ECMP
lose one spine : -25% bandwidth, 0 loss of connectivity
# Three-tier vs leaf-spine: hop count, path count, and failure domains.
def three_tier(servers_per_access=48, access=100):
return dict(name="three-tier", servers=servers_per_access * access,
worst_hops=5, # access-agg-core-agg-access
paths_between_racks=2, # active/standby, or 2 with MLAG
failure_domain="aggregation pair = many racks")
def leaf_spine(ports_per_leaf=52, uplinks=4, leaves=64, spines=4):
downlinks = ports_per_leaf - uplinks
return dict(name="leaf-spine", servers=downlinks * leaves,
worst_hops=3, # leaf-spine-leaf, ALWAYS
paths_between_racks=spines,
failure_domain="one leaf = one rack")
for t in (three_tier(), leaf_spine()):
print(f"{t['name']:12s} servers {t['servers']:>6,} worst-case hops "
f"{t['worst_hops']} parallel paths {t['paths_between_racks']} "
f"failure domain: {t['failure_domain']}")
# three-tier servers 4,800 worst-case hops 5 parallel paths 2 failure domain: aggregation pair = many racks
# leaf-spine servers 3,072 worst-case hops 3 parallel paths 4 failure domain: one leaf = one rack
print("\nThe property that matters most is PREDICTABILITY of hop count:")
print(" three-tier: 1 hop in a rack, 3 within a pod, 5 across pods")
print(" leaf-spine: 1 hop in a rack, 3 otherwise -- for EVERY pair of racks")
print(" -> latency becomes a property of the fabric, not of placement,")
print(" which is what lets a scheduler ignore the network entirely.\n")
def fabric_math(ports_per_leaf, uplinks, leaf_port_gbps, uplink_gbps, leaves):
downlinks = ports_per_leaf - uplinks
server_bw = downlinks * leaf_port_gbps # capacity facing the servers
uplink_bw = uplinks * uplink_gbps # capacity facing the fabric
return dict(servers_per_leaf=downlinks, oversub=server_bw / uplink_bw,
total_servers=downlinks * leaves,
bisection=leaves / 2 * uplink_bw)
print(f"{'leaf configuration':34s} {'srv/leaf':>9s} {'oversub':>9s} "
f"{'total srv':>10s} {'bisection':>12s}")
for label, args in (
("48x25G down, 4x100G up", (52, 4, 25, 100, 32)),
("48x25G down, 6x100G up", (54, 6, 25, 100, 32)),
("48x25G down, 12x100G up", (60, 12, 25, 100, 32)),
("32x100G down, 8x400G up", (40, 8, 100, 400, 32))):
m = fabric_math(*args)
print(f"{label:34s} {m['servers_per_leaf']:9d} {m['oversub']:8.2f}:1 "
f"{m['total_servers']:10,} {m['bisection']:9,.0f} Gb/s")
# leaf configuration srv/leaf oversub total srv bisection
# 48x25G down, 4x100G up 48 3.00:1 1,536 6,400 Gb/s
# 48x25G down, 6x100G up 48 2.00:1 1,536 9,600 Gb/s
# 48x25G down, 12x100G up 48 1.00:1 1,536 19,200 Gb/s
# 32x100G down, 8x400G up 32 1.00:1 1,024 51,200 Gb/s
#
# Note the trade in the first three rows: identical server count, and the only
# way to reduce oversubscription is to spend leaf ports on uplinks instead of
# servers. Non-blocking (1:1) means HALF the switch faces the fabric.
# Confirming a leaf-spine fabric is behaving as designed.
# 1. Uniform hop count: every remote leaf should be the same distance away.
for leaf in 10.0.1.1 10.0.2.1 10.0.9.1 10.0.31.1; do
echo -n "$leaf: "; traceroute -n -q1 "$leaf" 2>/dev/null | wc -l
done
# every value should be identical; a different one means an unintended path
# 2. All spines in use: the route to any remote prefix should list every spine.
vtysh -c 'show ip route 10.244.5.0/24'
# B>* 10.244.5.0/24 [20/0] via 10.0.255.1, swp49 <- spine-1
# via 10.0.255.2, swp50 <- spine-2
# via 10.0.255.3, swp51 <- spine-3
# via 10.0.255.4, swp52 <- spine-4
# Fewer next hops than spines = a fabric link or BGP session is down.
# 3. No spanning tree should be doing anything in a routed fabric.
mstpctl showbridge 2>/dev/null || echo "no STP: correct for a routed fabric"
# 4. Even loading across uplinks -- the check that catches ECMP problems.
for p in swp49 swp50 swp51 swp52; do
printf "%s %s\n" "$p" "$(ethtool -S $p | awk '/tx_bytes:/{print $2}')"
done
# values should be within a few percent of each other over a sampling interval
# 5. BGP sessions: one per spine, on every leaf. Unnumbered keeps it simple.
vtysh -c 'show bgp summary' | tail -8
# Neighbor V AS MsgRcvd MsgSent Up/Down State/PfxRcd
# swp49 4 65100 18442 18401 02:14:52 412
# swp50 4 65100 18440 18399 02:14:52 412 <- equal prefix counts
Key takeaways
- Three-tier designs assumed traffic leaves the building, so hop count depends on placement (1, 3, or 5 hops) and spanning tree disables redundant links.
- Leaf-spine (folded Clos) connects every leaf to every spine and no leaf to another leaf.
- The result is uniform hop count — one hop inside a rack, three between any two racks — so placement stops affecting latency and schedulers can ignore the network.
- All links are active because forwarding is routed with ECMP; there is no spanning tree to disable anything.
- Scaling has two independent axes: more leaves for servers, more spines for inter-rack bandwidth.
- Failure domains shrink: a leaf outage affects one rack, and losing one of four spines costs 25% of bandwidth and no connectivity.
- The cost is cabling and ports spent on uplinks instead of servers; a non-blocking fabric dedicates roughly half of each leaf to the fabric.
- Beyond one fabric the pattern nests with a super-spine layer.
🧪 Practice
- A leaf has 48 server ports at 25 Gb/s and 6 uplinks at 100 Gb/s. Compute the oversubscription ratio and the number of uplinks needed for a non-blocking design.
- Explain why a leaf-spine fabric does not run spanning tree, and what replaces its loop-prevention role.
- For a fabric of 32 leaves and 8 spines with 100 Gb/s fabric links, compute the bisection bandwidth and the capacity lost when one spine fails.
- Interview-style: A team reports that two services perform well most days and badly after a redeployment, with no code changes. (Hint: in which topology does the physical distance between two workloads change when a scheduler moves them, and what does that do to a request that makes many internal calls?)
East-West vs North-South Traffic
Theory
The terms come from how network diagrams are conventionally drawn, with users at the top and servers at the bottom. North-south traffic moves vertically: between the outside world and the data centre, or between a client and a server. East-west traffic moves horizontally: between machines inside the data centre.
The reason this distinction organises data centre design is that the ratio between them inverted, and the inversion was caused by how software is built rather than by anything in networking. A monolithic application served a page from one machine, possibly consulting one database: nearly all traffic was north-south. A modern application decomposes the same page view into dozens or hundreds of internal calls — authentication, user profile, recommendations, inventory, feature lookups, model inference, caching, logging, tracing — each crossing the network. The bytes the user receives may be under a megabyte while the internal traffic generated to produce them is several times that.
Three design consequences follow, and each explains something about the rest of this chapter.
Capacity belongs between racks, not only at the edge. If most traffic never leaves the building, then the expensive, carefully engineered link is the one between two racks rather than the one to the internet. This is the direct justification for leaf-spine: any-to-any capacity matters more than any-to-edge capacity.
Latency multiplies. A single user request that triggers a deep chain of internal calls pays the fabric round trip once per call. At a 200 microsecond fabric round trip, 176 serial calls would be 35 milliseconds of pure network time before any work is done. Parallelism helps, and the tail latency of the slowest call dominates, which is why data centre fabrics are engineered for predictable low latency rather than for peak throughput. It is also why the fan-out pattern in the example below is where architects look first when a service is slow.
The perimeter stops being where security happens. A traditional firewall sits at the north-south boundary and inspects everything entering. If 80% of bytes never cross that boundary, then 80% of traffic is uninspected, and an attacker who reaches any internal workload moves laterally without passing a single control. This is the entire motivation for microsegmentation and zero trust inside the data centre: policy must be enforced between workloads, which is why it ends up in the virtual switch, in eBPF, or in a service mesh rather than in a perimeter appliance. It is also why internal traffic being unencrypted by default became unacceptable, and therefore why mutual TLS everywhere became a goal.
Examples
Where the bytes go for ONE page view
USER
| north-south: 850 kB (HTML, CSS, JS, images)
v
===================== perimeter firewall ======================
| ^ inspects only the north-south traffic
v
[ api-gateway ]
|----> [ auth ] east-west
|----> [ user-profile ] east-west
|----> [ recommendations ] east-west
| |----> [ feature-store ] x40 <-- FAN-OUT: the real cost
| |----> [ model-server ] x4
|----> [ inventory ]
| |----> [ cache ] x24
| |----> [ database ]
+----> [ logging / metrics ] x90
east-west total: 3,368 kB in 176 calls = 4x the north-south bytes
What that means for each design question:
capacity -> between racks (leaf-spine), not just at the edge
latency -> 176 calls, so the fabric RTT is multiplied 176 times
security -> 80% of bytes never reach the firewall
encryption -> internal traffic is the majority and is plaintext by default
# One user request, decomposed: where do the bytes actually go?
NORTH_SOUTH_KB = 850 # HTML, CSS, JS and images the browser receives
INTERNAL = [ # (call, number of calls, kB each)
("api-gateway -> auth", 2, 4),
("api-gateway -> user-profile", 3, 12),
("api-gateway -> recommendations", 1, 180),
("recommendations -> feature-store", 40, 26), # fan-out: the real cost
("recommendations -> model-server", 4, 340),
("api-gateway -> inventory", 6, 18),
("inventory -> cache", 24, 3),
("inventory -> database", 6, 64),
("all services -> logging/metrics", 90, 2),
]
east_west_kb = sum(calls * kb for _, calls, kb in INTERNAL)
calls = sum(c for _, c, _ in INTERNAL)
print(f"{'internal call':34s} {'calls':>6s} {'kB each':>8s} {'total kB':>9s}")
for name, c, kb in INTERNAL:
print(f"{name:34s} {c:6d} {kb:8d} {c*kb:9d}")
print(f"{'':34s} {calls:6d} {'':8s} {east_west_kb:9d}")
print(f"\nnorth-south (user <-> data centre): {NORTH_SOUTH_KB:7,} kB")
print(f"east-west (service <-> service) : {east_west_kb:7,} kB in {calls} calls")
print(f"ratio: {east_west_kb/NORTH_SOUTH_KB:.1f} : 1 east-west to north-south")
# north-south (user <-> data centre): 850 kB
# east-west (service <-> service) : 3,368 kB in 176 calls
# ratio: 4.0 : 1 east-west to north-south
total_kb = NORTH_SOUTH_KB + east_west_kb
print("\nArchitectural consequences, each following directly from that ratio:")
for c in (
"capacity must be between racks, not just at the edge -> leaf-spine",
"any-to-any matters more than any-to-edge -> uniform hop count",
f"latency multiplies: {calls} calls x 0.2 ms = {calls*0.2:.0f} ms of pure network",
f"a perimeter firewall sees only {100*NORTH_SOUTH_KB/total_kb:.0f}% of the bytes, "
f"so {100*east_west_kb/total_kb:.0f}% is never inspected -> east-west needs its own policy",
"internal traffic is unencrypted by default -> the case for a service mesh",
):
print(f" - {c}")
# - capacity must be between racks, not just at the edge -> leaf-spine
# - any-to-any matters more than any-to-edge -> uniform hop count
# - latency multiplies: 176 calls x 0.2 ms = 35 ms of pure network
# - a perimeter firewall sees only 20% of the bytes, so 80% is never inspected -> east-west needs its own policy
# - internal traffic is unencrypted by default -> the case for a service mesh
print("\nThe latency multiplication, made explicit:")
for rtt_us in (100, 200, 500, 2000):
print(f" fabric RTT {rtt_us:5d} us -> {calls} serial calls cost "
f"{calls*rtt_us/1000:7.1f} ms; with 8-way parallelism "
f"{calls*rtt_us/1000/8:6.1f} ms")
# fabric RTT 100 us -> 176 serial calls cost 17.6 ms; with 8-way parallelism 2.2 ms
# fabric RTT 200 us -> 176 serial calls cost 35.2 ms; with 8-way parallelism 4.4 ms
# fabric RTT 500 us -> 176 serial calls cost 88.0 ms; with 8-way parallelism 11.0 ms
# fabric RTT 2000 us -> 176 serial calls cost 352.0 ms; with 8-way parallelism 44.0 ms
#
# A 2 ms fabric would add a third of a second to one page view. This is why data
# centre networks optimise for predictable LOW latency while wide-area links
# optimise for cost per bit.
# Measuring the ratio in your own environment rather than assuming it.
# 1. At the border router: what actually crosses the perimeter?
# Compare with the sum of intra-fabric traffic from leaf uplink counters.
ip -s link show dev border0 | awk '/RX:|TX:/{getline; print}'
# 2. Per-rack east-west volume: sum the leaf uplink counters.
for leaf in leaf-{1..8}; do
ssh $leaf "cat /sys/class/net/swp49/statistics/tx_bytes"
done | paste -sd+ | bc
# 3. Flow-level view: conntrack shows internal versus external destinations.
conntrack -L 2>/dev/null | awk '{print $5}' | grep -oP 'dst=\K[0-9.]+' \
| awk -F. '$1==10 {i++} $1!=10 {e++} END {printf "internal %d external %d ratio %.1f:1\n", i, e, i/e}'
# 4. Kubernetes: count the service-to-service calls behind one user request.
# A trace makes the fan-out visible in a way no counter can.
kubectl exec -it deploy/api-gateway -- \
curl -s -H 'x-b3-sampled: 1' localhost:8080/checkout -o /dev/null -w '%{time_total}\n'
# then read the trace: span count = internal calls, critical path = what to fix
# 5. Is east-west policy actually being enforced? If this succeeds, it is not.
kubectl run probe --rm -it --image=busybox --restart=Never -- \
sh -c 'nc -zv postgres.prod.svc.cluster.local 5432'
# open <- an arbitrary pod reached the production database: no microsegmentation
Key takeaways
- North-south is client-to-data-centre traffic; east-west is machine-to-machine inside it.
- Microservice decomposition inverted the ratio: one page view can generate several times more internal bytes than it delivers to the user, across hundreds of internal calls.
- Therefore capacity belongs between racks, which is the direct justification for leaf-spine fabrics.
- Latency multiplies by the call count, so fabrics are engineered for predictable low latency rather than peak throughput, and fan-out patterns are where slowness usually originates.
- A perimeter firewall inspects only the north-south fraction, leaving the majority of traffic uninspected — the motivation for microsegmentation and zero trust.
- Because internal traffic dominates and is plaintext by default, encrypting east-west traffic becomes a first-order concern rather than an optional hardening step.
- Measure the ratio in your own environment; the fan-out is usually larger than anyone expects.
🧪 Practice
- Classify each as east-west or north-south: a browser fetching an image, a replica synchronising with its primary, a batch job reading from object storage in the same region, a DNS query to a public resolver.
- A request triggers 60 internal calls with a fabric round trip of 250 microseconds and 6-way parallelism. Compute the network time contributed, and the value if the round trip were 1 millisecond.
- Explain why a perimeter firewall provides weak protection in a microservice data centre, and name two mechanisms that address the gap.
- Interview-style: A page that was fast as a monolith is slow after being split into services, though every service reports sub-millisecond processing time. (Hint: the per-service numbers can all be excellent while the total is poor — count what was added between them, and consider whether the calls are serial or parallel.)
ECMP and Load Distribution
Theory
A leaf-spine fabric offers several equal-cost paths between any two racks, and ECMP (equal-cost multi-path) is how they get used. The mechanism is simple: when the routing table holds several next hops of equal cost, the switch computes a hash over the packet's flow identifiers — typically source and destination IP, protocol, and source and destination ports — and uses it to select one.
Hashing rather than round-robin is a deliberate choice, and the reason is packet ordering. Every packet of a given flow hashes identically, so all packets of a flow take the same path and arrive in order. Round-robin across paths of slightly different latency would reorder packets, which classic TCP interprets as loss, triggering spurious retransmissions and window reductions. ECMP therefore keeps flows intact by construction.
That design choice creates the mechanism's central limitation, and it is worth stating exactly: ECMP balances flows, not bytes. The hash is uniform over flow identifiers, which guarantees an even spread of flow count and says nothing about an even spread of traffic volume. Two distinct failure modes follow.
Too few flows. If only four large flows cross a four-path fabric, the hash assigns each independently, so collisions are likely — this is the birthday problem. The chance that four flows land on four distinct paths is only about 9%. On average two share a path while another sits idle, meaning one link carries twice its share while capacity goes unused.
Unequal flow sizes. Data centre traffic is famously bimodal: a very large number of small, short "mice" flows and a small number of enormous long-lived "elephant" flows, with the elephants carrying most of the bytes. Balancing flow count perfectly is nearly irrelevant if one elephant is a thousand times larger than a mouse. One badly placed elephant congests a link while its neighbours idle.
The mitigations differ in where they intervene:
| Mitigation | Mechanism | Limitation |
|---|---|---|
| More hash entropy | include L4 ports; hash the inner flow into the VXLAN outer source port | helps only when flows are numerous |
| Flowlet switching | re-hash at natural burst boundaries (gaps within a flow) | needs hardware support; small reordering risk |
| Per-packet spraying | spread every packet independently | near-perfect balance, but reorders — needs a tolerant transport |
| Adaptive / dynamic load balancing | choose the least-loaded path using local queue occupancy | local view only; can oscillate |
| Application-level splitting | open many connections instead of one | requires application change |
| Central scheduling | a controller places large flows explicitly | requires detecting elephants and a controller |
Two practical notes. First, the hash must be consistent along the path but ideally different at each hop, or all switches make correlated decisions and a collision at one hop repeats at the next; vendors use a per-device seed for this reason. Second, ECMP interacts with the overlay: because the underlay sees only outer headers, all overlay traffic between two tunnel endpoints looks like one flow unless the encapsulator injects entropy — which is precisely why VXLAN derives the outer UDP source port from a hash of the inner flow.
Examples
Why hashing, and what hashing costs
ROUND ROBIN HASH ON THE FLOW
pkt1 -> spine-1 all packets of flow A -> spine-2
pkt2 -> spine-2 all packets of flow B -> spine-4
pkt3 -> spine-3 ...
pkt4 -> spine-4
perfect BYTE balance perfect FLOW-COUNT balance
but REORDERING -> TCP sees loss order preserved per flow
but byte balance is left to chance
The elephant problem, drawn
4 spines, 4 elephant flows, hash assigns each independently:
spine-1 [####################] 2 elephants -> 200% of its share
spine-2 [ ] 0 elephants -> idle
spine-3 [##########] 1 elephant
spine-4 [##########] 1 elephant
Probability of a perfect 1-1-1-1 spread: ~9%
The fabric has ample total capacity and one link is congested anyway.
# ECMP spreads FLOWS, not BYTES. That distinction is the whole problem.
import hashlib, statistics
def ecmp(five_tuple, n_paths):
h = hashlib.md5(str(five_tuple).encode()).digest()
return int.from_bytes(h[:4], "big") % n_paths # per-flow, deterministic
def path_loads(flows, n_paths):
load = [0.0] * n_paths
for ft, bytes_ in flows:
load[ecmp(ft, n_paths)] += bytes_
return load
# Case 1: many flows of equal size. A hash handles this very well.
N = 4
mice = [((f"10.0.0.{i%250}", "10.0.1.5", 6, 30000 + i, 443), 50_000)
for i in range(2000)]
load = path_loads(mice, N)
print("2000 equal-size flows over 4 paths: " +
" ".join(f"{l/sum(load):6.1%}" for l in load) +
f" hot link {max(load)/(sum(load)/N):.2f}x ideal")
# 2000 equal-size flows over 4 paths: 26.2% 24.3% 23.7% 25.8% hot link 1.05x ideal
# Case 2: few LARGE flows. Now what matters is how many share one path, because
# each flow is a link's worth of traffic. A single example proves nothing, so
# sample many placements.
def max_flows_per_path(n_flows, n_paths, trials=5000):
maxes = []
for t in range(trials):
counts = [0] * n_paths
for i in range(n_flows):
counts[ecmp((f"10.{t//250}.{t%250}.{i}", "10.0.1.5", 6,
40000 + i, 443), n_paths)] += 1
maxes.append(max(counts))
return maxes
print("\nFew large flows: how many share the BUSIEST path (5000 placements)")
print(f"{'flows':>6s} {'paths':>6s} {'ideal':>6s} {'mean max':>9s} {'p95 max':>8s} "
f"{'perfect':>8s}")
for n_el, paths in ((4, 4), (8, 4), (4, 8), (8, 8), (16, 8), (32, 8)):
m = sorted(max_flows_per_path(n_el, paths))
perfect = sum(1 for x in m if x <= -(-n_el // paths)) / len(m)
print(f"{n_el:6d} {paths:6d} {n_el/paths:6.1f} {statistics.mean(m):9.2f} "
f"{m[int(0.95*len(m))]:8d} {perfect:8.1%}")
# flows paths ideal mean max p95 max perfect
# 4 4 1.0 2.11 3 9.7%
# 8 4 2.0 3.52 5 4.2%
# 4 8 0.5 1.63 3 41.9%
# 8 8 1.0 2.59 4 0.2%
# 16 8 2.0 4.19 6 0.0%
# 32 8 4.0 7.04 9 0.0%
#
# Read the first row: 4 flows over 4 paths SHOULD put 1 on each. On average 2
# share one path while another idles, and only ~10% of placements are perfect.
# Note also that the RATIO of mean-max to ideal improves as flows per path rises:
# ECMP works well precisely when there are many flows per path.
def p_all_distinct(flows, paths):
p = 1.0
for i in range(flows):
p *= (paths - i) / paths
if p <= 0:
return 0.0
return p
print("\nThe combinatorics behind it (the birthday problem):")
print(" probability that 4 large flows land on 4 distinct paths")
for paths in (4, 8, 16, 32):
print(f" over {paths:2d} paths: {p_all_distinct(4, paths):6.1%}")
# over 4 paths: 9.4%
# over 8 paths: 41.0%
# over 16 paths: 66.7%
# over 32 paths: 82.3%
# The simulation's 41.9% for 4 flows over 8 paths matches this 41.0% -- the same
# quantity computed two ways.
print("\nMitigations, and what each actually changes:")
for m, note in [
("more hash entropy (L4 ports, VXLAN sport)", "helps only when flows are numerous"),
("flowlet / adaptive load balancing", "re-hash at burst boundaries, not per flow"),
("per-packet spraying", "perfect balance, but reorders: needs a tolerant transport"),
("split elephants in the application", "many connections instead of one"),
("central scheduling of large flows", "a controller places them explicitly"),
]:
print(f" {m:42s} {note}")
# --- Is ECMP installed, and across how many next hops? ---
ip route show 10.244.5.0/24
# 10.244.5.0/24 proto bgp metric 20
# nexthop via 10.0.255.1 dev swp49 weight 1
# nexthop via 10.0.255.2 dev swp50 weight 1
# nexthop via 10.0.255.3 dev swp51 weight 1
# nexthop via 10.0.255.4 dev swp52 weight 1
# Kernel ECMP must be told to hash on L4 ports, or it uses L3 only.
sysctl net.ipv4.fib_multipath_hash_policy
# 0 = L3 only (src/dst IP) <- poor entropy: all flows between two hosts collide
# 1 = L3 + L4 (adds ports) <- what you want
sysctl -w net.ipv4.fib_multipath_hash_policy=1
# --- Which path will a SPECIFIC flow take? Ask, do not guess. ---
ip route get 10.244.5.10 from 10.244.3.2 ipproto tcp sport 43210 dport 443
# 10.244.5.10 from 10.244.3.2 via 10.0.255.3 dev swp51
# ^ change sport and the answer changes: that is the hash at work
# --- Detect imbalance: compare uplink counters over an interval ---
for p in swp49 swp50 swp51 swp52; do
a=$(cat /sys/class/net/$p/statistics/tx_bytes); sleep 10
b=$(cat /sys/class/net/$p/statistics/tx_bytes)
echo "$p $(( (b-a)*8/10/1000000 )) Mb/s"
done
# swp49 8420 Mb/s
# swp50 310 Mb/s <- a 27x spread across supposedly equal paths
# swp51 9110 Mb/s means elephant flows, not a broken fabric
# swp52 280 Mb/s
# --- Find the elephants responsible ---
ss -tin | awk '/bytes_sent/{print}' | sort -t: -k2 -rn | head
conntrack -L -o extended 2>/dev/null | head
# --- Mitigations as configuration ---
# Give the overlay entropy (this is usually already the default, so verify).
ip -d link show vxlan100 | grep -o 'srcport [0-9 ]*'
# srcport 0 0 <- means "hash the inner flow", which is what you want
Key takeaways
- ECMP selects among equal-cost next hops by hashing the flow's identifiers, which keeps every packet of a flow on one path and preserves ordering.
- Order preservation is the reason for hashing rather than round-robin: reordering looks like loss to classic TCP.
- The consequence is that ECMP balances flow count, not bytes. Uniformity over flows guarantees nothing about traffic volume.
- Too few flows causes collisions by the birthday problem: four flows over four paths spread perfectly only about 9% of the time.
- Unequal flow sizes break it further, because data centre traffic is bimodal and a few elephants carry most of the bytes.
- ECMP works well when there are many flows per path; imbalance shrinks as that ratio rises.
- Mitigations include more hash entropy, flowlet switching, per-packet spraying (needs a reorder-tolerant transport), application-level splitting, and central placement of large flows.
- Overlays need the encapsulator to inject entropy — VXLAN hashes the inner flow into the outer UDP source port — or the underlay sees one flow per endpoint pair.
- A large spread across supposedly equal uplinks usually means elephant flows, not a broken fabric.
🧪 Practice
- Compute the probability that three large flows land on three distinct paths in a fabric with 4 spines, and again with 16 spines.
- Explain why ECMP hashes rather than round-robins, and what would have to change in the transport for per-packet spraying to be safe.
- Given uplink counters of 9.1, 8.4, 0.3, and 0.28 Gb/s, state your diagnosis and the next two commands you would run.
- Interview-style: A backup job between two racks reaches only a quarter of the fabric's inter-rack capacity, and the fabric shows no errors or drops. (Hint: ask how many TCP connections the backup tool opens, then consider what a per-flow hash can possibly do with that number.)
Load Balancing: L4 vs L7
Theory
A load balancer distributes requests across a pool of backends, and the decisive design question is how much of the request it must understand in order to decide. That single question determines its cost, its capabilities, and where it belongs.
An L4 load balancer works with transport-layer information only: addresses, protocol, and ports. It picks a backend by hashing the connection's 5-tuple and then forwards packets, never examining the payload. It is very fast, because the per-packet work is a hash and a rewrite, and it can run at tens of gigabits per core or entirely in switch hardware. It also passes TLS through untouched, which means it needs no certificates and cannot see plaintext.
An L7 load balancer terminates the connection, parses the application protocol, and decides based on the content: URL path, HTTP method, headers, cookies. This enables things L4 cannot express at all — route /api/v2 to a different pool than /api, send requests with a canary header to a new version, retry a failed request against another backend, rewrite paths, compress responses, cache. The costs are equally structural: it must terminate TLS (so it holds the certificates and sees plaintext), it maintains two connections per request instead of forwarding one flow, and per-request cost rises from microseconds to hundreds of microseconds.
| L4 | L7 | |
|---|---|---|
| Decision input | 5-tuple | path, headers, cookies, method, body |
| TLS | passes through | must terminate |
| Connections | one flow forwarded | two, one per side |
| Throughput per core | tens of Gb/s | roughly 1-2 Gb/s |
| Retry a failed request | impossible | natural |
| Health checking | TCP probe | HTTP status codes |
| Session affinity | by 5-tuple hash | by cookie or header |
There is a subtlety inside L4 that matters more than the L4/L7 distinction in practice: how the backend is chosen. The naive approach takes the hash modulo the pool size, and it has a severe flaw — changing the pool size changes almost every mapping. Adding one backend to a pool of four remaps about 80% of existing connections, and each remapped connection arrives at a backend with no state for it and is reset. Scaling the pool would therefore break most live connections.
Consistent hashing fixes this by mapping both backends and flows onto a ring, so adding a backend only steals the flows in its own arc. Adding a fifth backend to four moves about 20% of flows, which is the theoretical minimum. This is why consistent hashing appears in every serious L4 load balancer, and the same reasoning applies to cache sharding and partitioned databases.
Finally, the topology question: where does the return traffic go? Three models, and the differences are operationally significant:
- Proxy mode: the balancer sits in both directions. Simple and universal, and it must handle the full bidirectional bandwidth, hiding the client address unless a forwarding header is added.
- DSR (direct server return): requests go through the balancer, responses go straight from the backend to the client. Since responses are usually far larger than requests, this removes most of the load from the balancer, at the cost of trickier configuration.
- Anycast with ECMP: many balancer instances advertise the same address and the network spreads traffic across them. This is how large-scale front ends are built, and it works because consistent hashing makes every instance reach the same decision independently.
The modern pattern combines layers: an L4 stage at the edge for volume and denial-of-service absorption, then an L7 stage for routing intelligence, and increasingly an eBPF-based L4 stage that runs on every node rather than as a separate tier.
Examples
Same request, two levels of understanding
L4: sees only this
+-----------------------------+
| src 203.0.113.7:51234 |
| dst 198.51.100.1:443 | -> hash -> backend 3
| proto TCP | cannot know what is being asked for
+-----------------------------+
L7: terminates, then sees this
+-----------------------------------------------+
| GET /api/v2/orders?id=42 HTTP/2 |
| host: shop.example.com | -> route by PATH
| cookie: session=abc123 | -> affinity by COOKIE
| x-canary: true | -> send to canary pool
+-----------------------------------------------+
Return path models:
PROXY DSR ANYCAST + ECMP
client client client
| ^ | ^ |
v | v \ v
[ LB ] [ LB ] \ [ LB1 ][ LB2 ][ LB3 ]
| ^ | \ (same VIP advertised)
v | v \ |
backend backend --+ backends
both directions request in, network spreads by ECMP;
through the LB response direct consistent hashing makes
(bandwidth x2) (LB carries ~10%) every LB agree
# L4 vs L7: what each can decide, and what it costs to decide it.
import hashlib, bisect
def l4_pick(five_tuple, backends):
"""Modulo hashing: simple, and it remaps almost everything when the pool changes."""
h = int(hashlib.md5(str(five_tuple).encode()).hexdigest(), 16)
return backends[h % len(backends)]
class ConsistentHash:
"""Backends and flows both land on a ring, so a new backend steals only its arc."""
def __init__(self, backends, vnodes=160):
self.ring, self.vnodes = [], vnodes
for b in backends:
self._add(b)
self.ring.sort()
def _add(self, b):
for v in range(self.vnodes): # virtual nodes even out the arcs
self.ring.append(
(int(hashlib.md5(f"{b}#{v}".encode()).hexdigest(), 16), b))
def add(self, b):
self._add(b); self.ring.sort()
def pick(self, key):
h = int(hashlib.md5(str(key).encode()).hexdigest(), 16)
i = bisect.bisect([r[0] for r in self.ring], h) % len(self.ring)
return self.ring[i][1]
BACKENDS = [f"srv{i}" for i in range(4)]
keys = [(f"10.0.0.{i%250}", "vip", 6, 30000 + i, 443) for i in range(10_000)]
plain_before = [l4_pick(k, BACKENDS) for k in keys]
plain_after = [l4_pick(k, BACKENDS + ["srv4"]) for k in keys]
moved_plain = sum(1 for a, b in zip(plain_before, plain_after) if a != b)
ch = ConsistentHash(BACKENDS)
ch_before = [ch.pick(k) for k in keys]
ch.add("srv4")
moved_ch = sum(1 for a, b in zip(ch_before, [ch.pick(k) for k in keys]) if a != b)
print("Adding a 5th backend to a pool of 4 -- how many flows are remapped?")
print(f" modulo hashing : {moved_plain:6,} of {len(keys):,} "
f"({moved_plain/len(keys):6.1%}) -> each remapped connection is RESET")
print(f" consistent hashing: {moved_ch:6,} of {len(keys):,} "
f"({moved_ch/len(keys):6.1%}) -> near the ideal 1/5 = 20.0%")
# Adding a 5th backend to a pool of 4 -- how many flows are remapped?
# modulo hashing : 7,994 of 10,000 ( 79.9%) -> each remapped connection is RESET
# consistent hashing: 2,135 of 10,000 ( 21.3%) -> near the ideal 1/5 = 20.0%
#
# This is why every serious L4 balancer uses consistent hashing: without it,
# scaling the pool is an outage.
def l7_pick(request, rules, default="pool-web"):
"""L7 decides on CONTENT, which requires terminating and parsing the request."""
for r in rules:
if request["path"].startswith(r["prefix"]) and \
all(request["headers"].get(k) == v for k, v in r.get("headers", {}).items()):
return r["pool"]
return default
RULES = [
{"prefix": "/api/v2", "pool": "pool-api-v2"},
{"prefix": "/api", "pool": "pool-api-v1"}, # order matters: v2 first
{"prefix": "/static", "pool": "pool-cdn"},
{"prefix": "/", "headers": {"x-canary": "true"}, "pool": "pool-canary"},
]
print("\nRouting decisions L4 cannot make at all:")
for req in ({"path": "/api/v2/orders", "headers": {}},
{"path": "/api/users", "headers": {}},
{"path": "/static/logo.png", "headers": {}},
{"path": "/home", "headers": {"x-canary": "true"}},
{"path": "/home", "headers": {}}):
print(f" {req['path']:20s} headers={str(req['headers']):28s} -> "
f"{l7_pick(req, RULES)}")
# /api/v2/orders headers={} -> pool-api-v2
# /api/users headers={} -> pool-api-v1
# /static/logo.png headers={} -> pool-cdn
# /home headers={'x-canary': 'true'} -> pool-canary
# /home headers={} -> pool-web
print("\nCost comparison:")
print(f" {'aspect':20s} {'L4':28s} L7")
for a, l4, l7 in [
("decision input", "5-tuple only", "path, headers, cookies, method"),
("TLS", "passes through untouched", "must terminate (or re-encrypt)"),
("connection model", "one flow, forwarded", "two connections, one per side"),
("per-request cost", "~microseconds", "~hundreds of microseconds"),
("throughput per core", "tens of Gb/s", "1-2 Gb/s"),
("retries / rewrite", "impossible", "natural"),
("health checking", "TCP-level probe", "HTTP status codes"),
]:
print(f" {a:20s} {l4:28s} {l7}")
# --- L4 with IPVS: kernel-level, consistent hashing, very fast ---
ipvsadm -A -t 198.51.100.1:443 -s sh # sh = source hashing
ipvsadm -a -t 198.51.100.1:443 -r 10.0.1.11:443 -g # -g = DSR (gatewaying)
ipvsadm -a -t 198.51.100.1:443 -r 10.0.1.12:443 -g
ipvsadm -Ln --stats
# -> InActConn/ActiveConn per backend; check for an even spread
# --- L4 with eBPF (Cilium), replacing kube-proxy ---
cilium service list
# ID Frontend Service Type Backend
# 12 10.96.0.10:53 ClusterIP 10.244.1.4:53, 10.244.2.7:53
cilium bpf lb list # the actual maps the datapath consults
# --- L7 with Envoy/nginx: routing on content ---
cat <<'CONF'
upstream pool_api_v2 { server 10.0.2.11:8080; server 10.0.2.12:8080; }
upstream pool_api_v1 { server 10.0.1.11:8080; server 10.0.1.12:8080; }
upstream pool_canary { server 10.0.3.11:8080; }
server {
listen 443 ssl http2;
ssl_certificate /etc/ssl/shop.crt; # L7 MUST hold the certificate
ssl_certificate_key /etc/ssl/shop.key;
location /api/v2 { proxy_pass http://pool_api_v2; } # longest prefix first
location /api { proxy_pass http://pool_api_v1; }
location / {
if ($http_x_canary = "true") { proxy_pass http://pool_canary; }
proxy_pass http://pool_api_v1;
proxy_next_upstream error timeout http_503; # RETRY: an L7-only ability
}
}
CONF
# --- Verify the difference in what each can see ---
# L4: the balancer never sees the path, so all of these hash identically.
for p in /api/v2/x /api/y /static/z; do
curl -s -o /dev/null -w "%{http_code} %{remote_ip}\n" "https://vip$p"
done
# --- Anycast front end: the same VIP from many instances ---
vtysh -c 'show bgp ipv4 unicast 198.51.100.1/32'
# multiple paths, one per LB instance -> the network spreads clients by ECMP,
# and consistent hashing makes every instance pick the same backend
Key takeaways
- The design question is how much of the request must be understood: L4 uses the 5-tuple, L7 parses the application protocol.
- L4 is fast (tens of Gb/s per core), passes TLS through untouched, and cannot route on content or retry a request.
- L7 routes on path, headers, and cookies, can retry, rewrite, and cache, and must terminate TLS, hold certificates, and maintain two connections per request.
- Inside L4, consistent hashing versus modulo hashing matters more than the L4/L7 choice: modulo remaps about 80% of flows when a backend is added, consistent hashing about 20%, the theoretical minimum.
- Virtual nodes on the hash ring are what make the arcs even.
- Return-path topology matters: proxy mode carries both directions, DSR offloads the large response path, and anycast with ECMP scales the front end horizontally.
- The modern pattern layers them: L4 at the edge for volume and attack absorption, L7 for routing intelligence, with eBPF L4 increasingly on every node.
🧪 Practice
- For a pool growing from 8 to 9 backends, compute the fraction of flows remapped under modulo hashing and under ideal consistent hashing.
- List three routing decisions that require L7 and explain what each would cost in TLS handling.
- Explain why direct server return substantially reduces load balancer bandwidth, and name one operational drawback.
- Interview-style: Every time the team scales their L4 load balancer pool, users report being logged out. (Hint: work out what fraction of connections change backend when the pool size changes under a modulo hash, and what the receiving backend knows about a connection it never established.)
RDMA and Lossless Fabrics
Theory
Some workloads treat the network as an extension of memory rather than as a communication channel. Distributed training reads parameters from another machine's memory; a storage array serves blocks that must arrive as fast as a local disk; an in-memory database replicates writes synchronously. For these, the ordinary networking stack is the bottleneck — not the wire, but the software above it. A kernel TCP round trip inside a rack is tens of microseconds, most of it spent in system calls, copies, and interrupt handling, while the wire time is under a microsecond.
RDMA (remote direct memory access) removes that software. An application registers a memory region, and the NIC reads from or writes to that region directly, with no involvement from the remote CPU and no copies. Three properties follow:
- Zero copy: data moves between application memory and the NIC without intermediate buffers.
- Kernel bypass: the application posts work requests directly to hardware queues, so there is no system call on the data path.
- One-sided operations: a read or write completes without the remote CPU doing anything, which is what makes remote memory behave like memory rather than like a request-response protocol.
The result is single-digit microsecond latency and near-zero CPU cost per byte. The catch is that the transport is implemented in the NIC, and RDMA's original transport (from InfiniBand) assumes a lossless network. Its recovery from a dropped packet is far cruder than TCP's: some implementations retransmit from the start of the whole operation. Loss that costs TCP a small throughput dip can cost RDMA a collapse.
Running RDMA over Ethernet (RoCEv2, which is RDMA over UDP/IP and therefore routable) consequently requires the fabric to essentially not drop packets, and that requirement is what makes data centre QoS different from Chapter 12's.
The first mechanism is PFC (priority flow control, 802.1Qbb). When a switch's buffer for a given traffic class fills, it sends a pause frame upstream telling the sender to stop transmitting that class. Nothing is dropped. The dangers are serious and worth stating explicitly:
- Head-of-line blocking: the pause stops a whole class, including flows not responsible for the congestion.
- Congestion spreading: pauses propagate upstream hop by hop, so congestion at one leaf can stall senders several hops away.
- Deadlock: with a cyclic buffer dependency, a set of switches can pause each other permanently.
The second mechanism is ECN with DCQCN. Switches mark packets when a queue exceeds a threshold, and the sender reduces its rate in proportion to the fraction marked. This is the same mechanism as Section 12.2's ECN, applied with a data-centre-tuned control law. DCTCP is the TCP equivalent, and its key difference from classic TCP is that the reduction is proportional to the marked fraction rather than an unconditional halving, so it settles at a small standing queue instead of oscillating.
The correct combination is ECN as the primary control and PFC as an emergency backstop tuned so it almost never fires. A fabric where PFC fires routinely is misconfigured and at risk of congestion spreading.
The other phenomenon that shapes these fabrics is incast. Many-to-one communication is the natural shape of sharded queries and map-reduce: one node requests data from hundreds of others, and all respond at once. The responses converge on a single switch port whose buffer is measured in hundreds of kilobytes, so the buffer overflows almost immediately. For TCP the consequence is severe: a lost segment at the end of a short response has no subsequent packets to trigger fast retransmit, so recovery waits for a retransmission timeout whose minimum is typically 200 milliseconds in a network whose round trip is 100 microseconds. One drop can add a thousandfold to a query's latency. The response is not bigger buffers but preventing the queue from filling: ECN marking, smaller initial windows, reduced minimum timeouts, and application-level staggering of requests.
Examples
What RDMA removes
TCP path (each arrow is software work)
app -> syscall -> socket buffer -> COPY -> kernel TCP -> driver -> NIC
|
app <- syscall <- socket buffer <- COPY <- kernel TCP <- IRQ <------+
~20-50 us round trip inside a rack, most of it CPU
RDMA path
app --(post work request to a hardware queue)--> NIC ==> wire
|
remote NIC writes DIRECTLY into registered memory <------+
remote CPU: NOT INVOLVED
~2-5 us round trip, near-zero CPU per byte
Incast: why buffers cannot be the answer
400 servers each send a 64 kB response at the same instant
\ | /
\ | /
[ ToR switch ] egress port buffer: ~200 kB
|
one client 25.6 MB arriving, 0.2 MB of buffer
-> ~99% must be dropped or paused
# Incast: why data centre fabrics need something other than bigger buffers.
SWITCH_PORT_GBPS, BUFFER_PER_PORT_KB = 25, 200
def incast(senders, response_kb):
"""Synchronised many-to-one: treat the burst as arriving faster than it drains."""
arriving_kb = senders * response_kb
overflow_kb = max(0.0, arriving_kb - BUFFER_PER_PORT_KB)
return arriving_kb, overflow_kb, (overflow_kb / arriving_kb) if arriving_kb else 0
print(f"ToR egress port: {SWITCH_PORT_GBPS} Gb/s with {BUFFER_PER_PORT_KB} kB of buffer")
print(f"{'senders':>8s} {'each':>8s} {'arrives':>10s} {'overflows':>11s} {'loss':>7s}")
for senders in (2, 10, 40, 100, 400):
a, o, frac = incast(senders, 64)
print(f"{senders:8d} {64:7d}kB {a:9.0f}kB {o:10.0f}kB {frac:7.1%}")
# ToR egress port: 25 Gb/s with 200 kB of buffer
# senders each arrives overflows loss
# 2 64kB 128kB 0kB 0.0%
# 10 64kB 640kB 440kB 68.8%
# 40 64kB 2560kB 2360kB 92.2%
# 100 64kB 6400kB 6200kB 96.9%
# 400 64kB 25600kB 25400kB 99.2%
print("\nWhy a single drop is catastrophic here:")
DC_RTT_US, MIN_RTO_MS = 100, 200
print(f" fabric RTT : {DC_RTT_US} us")
print(f" TCP minimum RTO : {MIN_RTO_MS} ms")
print(f" ratio : {MIN_RTO_MS*1000/DC_RTT_US:,.0f}x")
print(" A short response has no later packets to trigger fast retransmit, so")
print(" recovery waits for the timeout. One drop turns a 100 us query into 200 ms.")
print(" Hence: PREVENT the queue from filling rather than buffer more.\n")
# fabric RTT : 100 us
# TCP minimum RTO : 200 ms
# ratio : 2,000x
def dctcp_alpha(alpha, marked_fraction, g=1/16):
"""DCTCP: track the marked fraction with an EWMA, react in PROPORTION to it."""
return (1 - g) * alpha + g * marked_fraction
alpha = 0.0
print("DCTCP: reduction proportional to the marked fraction, not a blind halving")
print(f"{'round':>6s} {'marked':>8s} {'alpha':>7s} {'cwnd factor':>12s}")
for rnd, marked in enumerate([0.0, 0.1, 0.3, 0.5, 0.5, 0.2, 0.0, 0.0], 1):
alpha = dctcp_alpha(alpha, marked)
print(f"{rnd:6d} {marked:8.1%} {alpha:7.4f} {1 - alpha/2:12.4f}")
# round marked alpha cwnd factor
# 1 0.0% 0.0000 1.0000
# 2 10.0% 0.0063 0.9969
# 3 30.0% 0.0246 0.9877
# 4 50.0% 0.0543 0.9728
# 5 50.0% 0.0822 0.9589
# 6 20.0% 0.0895 0.9552
# 7 0.0% 0.0839 0.9580
# 8 0.0% 0.0787 0.9607
#
# Classic TCP would apply a factor of 0.5 on the FIRST mark and then sawtooth.
# DCTCP trims by a few percent and converges to a small standing queue, which is
# exactly what a latency-sensitive fabric needs.
print("\nPFC vs ECN on a lossless class:")
for name, pro, con in (
("PFC (pause frames)", "truly lossless, which RDMA's transport requires",
"HEAD-OF-LINE blocking; pauses propagate upstream and can DEADLOCK"),
("ECN / DCQCN", "end-to-end rate control, nothing is paused",
"reacts in about one RTT, so microbursts can still overflow"),
):
print(f" {name:20s} + {pro}")
print(f" {'':20s} - {con}")
print("\nUse ECN as the primary control and PFC as a backstop tuned so it almost")
print("never fires. A fabric where PFC fires routinely is misconfigured.")
# --- Is RDMA present and which transport? ---
ibv_devices && ibv_devinfo | grep -E 'transport|state'
rdma link show
# link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev ens1f0
# Measure the latency RDMA actually delivers, versus TCP on the same hosts.
ib_write_lat -d mlx5_0 -a 10.0.1.12 # RDMA: expect single-digit microseconds
qperf 10.0.1.12 tcp_lat # TCP: expect tens of microseconds
# --- PFC: configured, and is it FIRING? The second question is the important one ---
mlnx_qos -i ens1f0
# PFC configuration: priority 3 enabled <- the lossless class
ethtool -S ens1f0 | grep -iE 'pause|pfc'
# tx_pause_ctrl_phy: 0
# rx_pause_ctrl_phy: 4182931 <- WE are being paused, a lot
# Rising PFC counters mean congestion is being pushed upstream instead of
# being controlled end to end. Tune ECN so this counter stays near zero.
# --- ECN marking on the switch: the primary control ---
# (vendor CLI shown for clarity; the principle is a threshold in BYTES per queue)
# interface Ethernet1/1
# random-detect ecn minimum-threshold 150 kbytes maximum-threshold 1500 kbytes
ethtool -S swp1 | grep -i ecn
# ecn_marked_packets: 89214 <- healthy: senders are being signalled
# --- Host side: enable DCTCP for TCP workloads sharing the fabric ---
sysctl -w net.ipv4.tcp_congestion_control=dctcp
sysctl -w net.ipv4.tcp_ecn=1
# And shorten the pathological RTO floor for incast-heavy workloads:
ip route change 10.0.0.0/8 dev ens1f0 rto_min 5ms
# --- Confirm incast is what you are seeing ---
ss -ti | grep -c 'retrans'
nstat -az | grep -iE 'TcpExtTCPTimeouts|TcpRetransSegs'
# TcpExtTCPTimeouts 4821 <- timeouts, not fast retransmits: the incast signature
Key takeaways
- For memory-like workloads the bottleneck is the software stack, not the wire: a kernel TCP round trip in a rack is tens of microseconds against under a microsecond of wire time.
- RDMA gives zero copy, kernel bypass, and one-sided operations, reaching single-digit microsecond latency with near-zero CPU per byte.
- Its transport lives in the NIC and assumes a lossless network; its loss recovery is far cruder than TCP's, so drops can be catastrophic.
- PFC makes a class lossless with pause frames, at the risk of head-of-line blocking, congestion spreading, and deadlock.
- ECN with DCQCN (or DCTCP for TCP) controls rate end to end, reducing in proportion to the marked fraction rather than halving blindly, which settles at a small standing queue.
- Use ECN as the primary control and PFC as a rarely-firing backstop; routine PFC activity indicates misconfiguration.
- Incast overwhelms a shallow port buffer with synchronised many-to-one responses, and a single drop on a short response waits for a retransmission timeout about 2000x the fabric round trip.
- The fix for incast is preventing queue buildup — ECN, smaller initial windows, a lower RTO floor, request staggering — not bigger buffers.
🧪 Practice
- Compute the bytes arriving at one port when 150 servers each send a 32 kB response simultaneously, and the fraction exceeding a 300 kB buffer.
- Explain the mechanism by which PFC can deadlock, and why ECN cannot.
- Given a fabric round trip of 80 microseconds and a minimum retransmission timeout of 200 milliseconds, compute the latency multiplier of a single timeout and name two host settings that reduce it.
- Interview-style: An RDMA storage cluster performs well in isolation and degrades badly once a batch analytics job shares the fabric. (Hint: consider which traffic class the batch job lands in, what the switch does when that class fills a shared buffer, and where pause frames travel.)
Oversubscription and Bisection Bandwidth
Theory
Every fabric design answers one question with money: how much capacity is there between groups of machines, relative to what those machines could generate? Two related metrics express the answer, and they are the vocabulary in which fabric designs are specified and compared.
Oversubscription ratio is a local, per-switch measure: the ratio of downlink capacity facing servers to uplink capacity facing the fabric. A leaf with 48 server ports at 25 Gb/s has 1200 Gb/s facing the servers; with four 100 Gb/s uplinks it has 400 Gb/s facing the fabric, giving 3:1. The interpretation is direct: if every server on that leaf transmitted at line rate to a server on another leaf, only a third of the offered traffic would fit.
Bisection bandwidth is a global measure: cut the network into two equal halves and sum the capacity of the links crossing the cut. It bounds the worst-case all-to-all throughput and is the right metric for workloads such as shuffles, distributed training, and large joins, where every node communicates with every other node.
A fabric is non-blocking when the oversubscription ratio is 1:1, meaning every server could transmit at line rate simultaneously to any destination. The arithmetic of achieving that is the fabric designer's central trade-off: a leaf must dedicate half its ports and half its capacity to uplinks. A 64-port switch becomes a 32-server switch. You are paying for two switch ports for every server port, and the bill grows further because those uplinks usually run at higher speed and need optics.
Whether that is worth it depends entirely on the traffic pattern, and the honest answer is a spectrum:
| Workload | Communication pattern | Sensible ratio |
|---|---|---|
| Web/application servers | mostly north-south, modest east-west | 4:1 to 6:1 |
| General virtualisation | mixed, unpredictable placement | 3:1 |
| Databases, analytics | heavy shuffles between nodes | 2:1 or 1:1 |
| Distributed training, HPC | all-to-all, synchronised collectives | 1:1, sometimes with a dedicated fabric |
| Storage / RDMA | large sustained flows, loss-intolerant | 1:1 |
Three practical points shape real designs.
Oversubscription is a statistical bet. A 3:1 ratio is fine if servers rarely transmit simultaneously at line rate, which is usually true for general workloads and emphatically false for synchronised collectives. The bet fails precisely when traffic is correlated, which is the defining characteristic of distributed training: all nodes finish computing and start exchanging at the same instant. This is why AI fabrics are built non-blocking while general-purpose clouds are not.
Locality can substitute for bandwidth. If a scheduler places communicating workloads on the same leaf, their traffic never crosses the fabric, and a 3:1 fabric behaves like a much richer one. Topology-aware placement is therefore a genuine alternative to buying uplinks, and it is why orchestrators expose rack and zone labels.
Measured utilisation is not the whole story. Average uplink utilisation of 20% can coexist with microbursts that fill buffers, and it is the bursts that cause the drops. Oversubscription should be judged against peak and burst behaviour, using short sampling intervals, not against a five-minute average.
Examples
Where the capacity goes
LEAF with 48x25G servers and 4x100G uplinks
servers: 48 x 25 = 1200 Gb/s <-- what the servers can offer
uplinks: 4 x 100 = 400 Gb/s <-- what can leave the rack
oversubscription = 1200/400 = 3:1
meaning: all servers at line rate to other racks -> 1/3 gets through
NON-BLOCKING leaf, same switch
servers: 24 x 25 = 600 Gb/s
uplinks: 24 x 25 = 600 Gb/s (or 6 x 100G)
oversubscription = 1:1
cost: HALF the ports carry no servers
BISECTION of a 32-leaf, 4-spine fabric with 100G fabric links
16 leaves | 16 leaves
................ cut ................
each leaf has 4 x 100G to the spines, and half of each leaf's uplink
capacity crosses the cut in the worst case:
16 leaves x 4 x 100 Gb/s = 6400 Gb/s across the bisection
Reading the number: a 6.4 Tb/s bisection means an all-to-all shuffle of
100 TB takes at least 100e12 x 8 / 6.4e12 = 125 seconds, however fast
the servers are.
# Oversubscription, bisection, and what non-blocking actually costs.
def leaf(ports, port_gbps, uplinks, uplink_gbps):
servers = ports - uplinks
down = servers * port_gbps
up = uplinks * uplink_gbps
return dict(servers=servers, down=down, up=up, oversub=down / up)
print(f"{'leaf design':32s} {'servers':>8s} {'down':>9s} {'up':>8s} {'oversub':>9s}")
for label, args in (
("64x25G: 4 uplinks", (64, 25, 4, 100)),
("64x25G: 8 uplinks", (64, 25, 8, 100)),
("64x25G: 12 uplinks", (64, 25, 12, 100)),
("64x25G: 16 uplinks", (64, 25, 16, 100))):
m = leaf(*args)
print(f"{label:32s} {m['servers']:8d} {m['down']:7.0f}G {m['up']:6.0f}G "
f"{m['oversub']:8.2f}:1")
# leaf design servers down up oversub
# 64x25G: 4 uplinks 60 1500G 400G 3.75:1
# 64x25G: 8 uplinks 56 1400G 800G 1.75:1
# 64x25G: 12 uplinks 52 1300G 1200G 1.08:1
# 64x25G: 16 uplinks 48 1200G 1600G 0.75:1
#
# Going from 3.75:1 to roughly 1:1 costs 8 server ports of the 60 -- a 13%
# reduction in servers per rack, plus 8 more optics per leaf.
def fabric(leaves, uplinks_per_leaf, uplink_gbps, servers_per_leaf):
bisection = (leaves / 2) * uplinks_per_leaf * uplink_gbps
return dict(servers=leaves * servers_per_leaf, bisection_gbps=bisection,
per_server_gbps=bisection / (leaves * servers_per_leaf / 2))
print(f"\n{'fabric':34s} {'servers':>8s} {'bisection':>12s} {'per server':>12s}")
for label, args in (
("32 leaves, 4x100G up, 60 srv", (32, 4, 100, 60)),
("32 leaves, 8x100G up, 56 srv", (32, 8, 100, 56)),
("32 leaves, 12x100G up, 52 srv", (32, 12, 100, 52)),
("64 leaves, 8x400G up, 56 srv", (64, 8, 400, 56))):
m = fabric(*args)
print(f"{label:34s} {m['servers']:8,} {m['bisection_gbps']:9,.0f}G "
f"{m['per_server_gbps']:9.2f}G")
# fabric servers bisection per server
# 32 leaves, 4x100G up, 60 srv 1,920 6,400G 6.67G
# 32 leaves, 8x100G up, 56 srv 1,792 12,800G 14.29G
# 32 leaves, 12x100G up, 52 srv 1,664 19,200G 23.08G
# 64 leaves, 8x400G up, 56 srv 3,584 102,400G 57.14G
print("\nWhat bisection bandwidth means for a real job: an all-to-all shuffle")
for tb, bisec_gbps in ((100, 6_400), (100, 19_200), (1_000, 102_400)):
seconds = tb * 1e12 * 8 / (bisec_gbps * 1e9)
print(f" shuffle {tb:5,} TB across a {bisec_gbps:7,} Gb/s bisection: "
f"at least {seconds:8.1f} s ({seconds/60:5.1f} min)")
# shuffle 100 TB across a 6,400 Gb/s bisection: at least 125.0 s ( 2.1 min)
# shuffle 100 TB across a 19,200 Gb/s bisection: at least 41.7 s ( 0.7 min)
# shuffle 1,000 TB across a 102,400 Gb/s bisection: at least 78.1 s ( 1.3 min)
# This is a FLOOR set by the topology. No amount of server or storage speed
# improves it; only more fabric capacity does.
print("\nThe statistical bet: oversubscription assumes traffic is UNCORRELATED.")
for pattern, simultaneity in (("web servers", 0.15), ("virtualisation", 0.30),
("analytics shuffle", 0.80),
("synchronised collective", 1.00)):
for ratio in (3.0,):
effective = simultaneity * ratio
verdict = "fits" if effective <= 1.0 else f"OVERSUBSCRIBED {effective:.1f}x"
print(f" {pattern:26s} {simultaneity:4.0%} of servers active at once "
f"on a {ratio:.0f}:1 fabric -> {verdict}")
# web servers 15% of servers active at once on a 3:1 fabric -> fits
# virtualisation 30% of servers active at once on a 3:1 fabric -> fits
# analytics shuffle 80% of servers active at once on a 3:1 fabric -> OVERSUBSCRIBED 2.4x
# synchronised collective 100% of servers active at once on a 3:1 fabric -> OVERSUBSCRIBED 3.0x
#
# Distributed training is the worst case by construction: every node finishes
# computing and starts exchanging at the SAME INSTANT. That is why AI fabrics
# are built non-blocking while general-purpose clouds are not.
# --- Derive the real oversubscription ratio from the running device ---
# Sum server-facing and fabric-facing port speeds rather than trusting the design doc.
for p in /sys/class/net/swp*; do
n=$(basename "$p"); s=$(cat "$p/speed" 2>/dev/null || echo 0)
case "$n" in swp4[9-9]|swp5[0-2]) role=uplink;; *) role=server;; esac
echo "$role $s"
done | awk '{t[$1]+=$2} END {printf "server %d Mb/s, uplink %d Mb/s, oversub %.2f:1\n",
t["server"], t["uplink"], t["server"]/t["uplink"]}'
# --- Judge it against BURSTS, not averages ---
# A 5-minute average of 20% can hide buffer-filling microbursts.
for i in $(seq 60); do
cat /sys/class/net/swp49/statistics/tx_bytes; sleep 1
done | awk 'NR>1{printf "%.2f Gb/s\n", ($1-p)*8/1e9} {p=$1}' | sort -rn | head -5
# the top samples matter, not the mean
# The definitive burst evidence: are packets being discarded on the uplink?
ethtool -S swp49 | grep -iE 'discard|drop|no_buffer'
# tx_discards: 118422 <- the fabric is too narrow for the bursts,
# whatever the average utilisation says
# --- Is locality substituting for bandwidth? Check placement awareness ---
kubectl get nodes --show-labels | grep -o 'topology.kubernetes.io/zone=[^,]*' | sort | uniq -c
kubectl get pods -o wide -l app=shuffle-heavy | awk '{print $7}' | sort | uniq -c
# if all replicas sit on one rack's nodes, fabric capacity is barely used
Key takeaways
- Oversubscription ratio is local: downlink capacity facing servers divided by uplink capacity facing the fabric. 3:1 means a third of fully offered cross-rack traffic fits.
- Bisection bandwidth is global: the capacity crossing a cut into two equal halves, and it bounds worst-case all-to-all throughput.
- Non-blocking (1:1) requires a leaf to dedicate about half its ports to uplinks, so you buy two switch ports per server port plus the optics.
- Choose the ratio from the workload: 4:1 to 6:1 for web tiers, 3:1 for general virtualisation, 1:1 or 2:1 for analytics, storage, and training.
- Oversubscription is a statistical bet that traffic is uncorrelated, and it fails exactly when traffic is synchronised — which is the defining property of distributed collectives.
- Bisection bandwidth sets a hard floor on shuffle time that no server or storage speed can improve.
- Locality can substitute for bandwidth: topology-aware placement keeps traffic off the fabric entirely.
- Judge a fabric against bursts and discards, not five-minute averages; rising uplink discards prove the fabric is too narrow regardless of mean utilisation.
🧪 Practice
- A leaf has 32 ports of 100 Gb/s. Compute the oversubscription ratio with 4, 8, and 16 uplinks, and state the server count in each case.
- For a fabric of 48 leaves with 6 uplinks of 100 Gb/s each, compute the bisection bandwidth and the minimum time to shuffle 250 TB across it.
- Explain why a 4:1 ratio may be fine for a web tier and unacceptable for distributed training, referring to traffic correlation.
- Interview-style: A fabric averages 18% uplink utilisation and the monitoring team insists there is ample headroom, yet an analytics job reports steady packet loss. (Hint: consider the timescale on which a switch buffer fills versus the timescale of the utilisation graph, and name the counter that settles the argument.)
<a id="134-cloud-and-wide-area-evolution"></a>
13.4 Cloud and Wide-Area Evolution
Running workloads in someone else's data centre changes both what a network is and where its boundaries lie: addressing becomes a software construct you define, the link to your own premises becomes a purchasing decision, and serving users worldwide becomes a question of where computation and data are allowed to live — this subchapter covers the virtual network constructs clouds provide, the ways to reach them, the wide-area architecture that replaced fixed circuits, and how traffic and state are distributed across regions.
Virtual Private Clouds and Subnets
Theory
A VPC is a logically isolated network you define inside a provider's physical infrastructure. You choose its address range, divide it into subnets, write its routing tables, and attach gateways. Underneath, it is the overlay machinery of Section 13.2 — encapsulation over a shared fabric, with a mapping service and distributed policy enforcement — but the interface you are given is that of a network you own.
The value is that the network becomes a software artefact. A subnet is an API call rather than a VLAN configured on a switch, so a network can be created, versioned, reviewed in a pull request, and destroyed. This is what makes infrastructure-as-code possible for networking, and it is the foundation for everything in Section 13.5.
The constructs are few and worth knowing precisely, because their names differ between providers while the concepts do not:
| Construct | What it is | Notes |
|---|---|---|
| VPC / VNet | the isolated address space | one region, one primary CIDR |
| Subnet | a slice of the CIDR, in one availability zone | zone-scoped: this is why you need one per zone |
| Route table | destinations to targets, per subnet | what makes a subnet "public" or "private" |
| Internet gateway | bidirectional internet reachability | a subnet with a default route to it is public |
| NAT gateway | outbound-only internet access | how private subnets reach the internet |
| Security group | stateful instance-level firewall | allow-only; return traffic is automatic |
| Network ACL | stateless subnet-level filter | needs explicit rules in both directions |
| Peering / transit gateway | VPC-to-VPC or hub connectivity | requires non-overlapping CIDRs |
| Private endpoint | reach a provider service without internet | keeps traffic on the provider's network |
Two points cause most real problems.
Address planning is effectively irreversible. The VPC CIDR cannot be meaningfully changed once workloads exist, and it constrains everything you might later want to connect: your on-premises network, partner VPCs, other regions, and Kubernetes pod ranges. If any of those overlap, peering and VPN are impossible without NAT, which destroys the address transparency that made the design comprehensible. The discipline is to allocate from a single organisation-wide plan, leave generous headroom, and never reuse a range — a habit that costs nothing at design time and is extremely expensive to retrofit.
Subnets also need more room than people expect. Providers reserve several addresses per subnet, so a /28 yields only eleven usable addresses. A load balancer subnet sized /28 is a well-known outage: the balancer needs spare addresses to scale out, cannot get them, and stops scaling during exactly the traffic surge that required it.
The distinction between a public and a private subnet is a route, not a property. A subnet is public because its route table sends 0.0.0.0/0 to an internet gateway. There is no "public" attribute. This is worth internalising because it means the entire exposure of a subnet is determined by one routing entry, and reviewing route tables is therefore a security control, not merely a connectivity check.
The stateful versus stateless difference between security groups and network ACLs is the other reliable source of confusion. A security group is stateful: permit an outbound connection and its replies are automatically allowed. A network ACL is stateless, evaluating each packet independently, so permitting outbound traffic without also permitting the inbound replies on the ephemeral port range silently breaks every outbound connection. The symptom is connections that hang rather than fail, which sends people looking at applications instead of at the ACL.
Examples
A conventional three-tier VPC across three zones
VPC 10.20.0.0/16
+---------------------------------------------------------------------+
| zone a zone b zone c |
| +---------------+ +---------------+ +---------------+ |
| | public /24 | | public /24 | | public /24 | |
| | ALB, NAT GW | | ALB, NAT GW | | ALB, NAT GW | |
| +-------+-------+ +-------+-------+ +-------+-------+ |
| | route 0.0.0.0/0 -> internet gateway <-- THIS makes it |
| | "public" |
| +---------------+ +---------------+ +---------------+ |
| | private /22 | | private /22 | | private /22 | |
| | app servers | | app servers | | app servers | |
| +-------+-------+ +-------+-------+ +-------+-------+ |
| | route 0.0.0.0/0 -> NAT gateway (outbound only) |
| +---------------+ +---------------+ +---------------+ |
| | data /24 | | data /24 | | data /24 | |
| | databases | | databases | | databases | |
| +---------------+ +---------------+ +---------------+ |
| ^ NO default route at all: cannot reach the internet |
+---------------------------------------------------------------------+
| |
[ internet gateway ] [ private endpoint ]
reach object storage without
traversing the internet
Subnets are ZONE-scoped, which is why each tier appears three times.
A subnet cannot span zones, so zone redundancy means subnet multiplication.
import ipaddress
# VPC address planning: the one decision you cannot undo later.
VPC = ipaddress.ip_network("10.20.0.0/16")
AZS = ["a", "b", "c"]
TIERS = [("public", 24), ("private", 22), ("data", 24)] # (name, prefix to USE)
RESERVED_BY_PROVIDER = 5 # network, gateway, DNS, future use, broadcast
print(f"VPC {VPC}: {VPC.num_addresses:,} addresses")
print("Allocate a /22 BLOCK per subnet, then use only what is needed inside it,")
print("so every subnet can grow later without renumbering its neighbours.\n")
print(f"{'subnet':14s} {'reserved block':>16s} {'in use':>16s} {'usable':>8s} purpose")
blocks = list(VPC.subnets(new_prefix=22))
i = 0
for az in AZS:
for tier, plen in TIERS:
block = blocks[i]; i += 1
used = list(block.subnets(new_prefix=plen))[0] if plen > 22 else block
print(f"{tier + '-' + az:14s} {str(block):>16s} {str(used):>16s} "
f"{used.num_addresses - RESERVED_BY_PROVIDER:8,} "
f"{'internet-facing' if tier == 'public' else 'no direct egress'}")
print(f"\nblocks consumed: {i} of {len(blocks)} available /22s "
f"-> {len(blocks) - i} left for growth")
# subnet reserved block in use usable purpose
# public-a 10.20.0.0/22 10.20.0.0/24 251 internet-facing
# private-a 10.20.4.0/22 10.20.4.0/22 1,019 no direct egress
# data-a 10.20.8.0/22 10.20.8.0/24 251 no direct egress
# public-b 10.20.12.0/22 10.20.12.0/24 251 internet-facing
# private-b 10.20.16.0/22 10.20.16.0/22 1,019 no direct egress
# data-b 10.20.20.0/22 10.20.20.0/24 251 no direct egress
# public-c 10.20.24.0/22 10.20.24.0/24 251 internet-facing
# private-c 10.20.28.0/22 10.20.28.0/22 1,019 no direct egress
# data-c 10.20.32.0/22 10.20.32.0/24 251 no direct egress
#
# blocks consumed: 9 of 64 available /22s -> 55 left for growth
print(f"\nThe provider reserves {RESERVED_BY_PROVIDER} addresses per subnet:")
for plen in (28, 27, 24, 22):
n = ipaddress.ip_network(f"10.0.0.0/{plen}")
print(f" /{plen}: {n.num_addresses:5,} addresses -> "
f"{n.num_addresses - RESERVED_BY_PROVIDER:5,} usable")
# /28: 16 addresses -> 11 usable <- a load balancer subnet this size
# /27: 32 addresses -> 27 usable cannot scale out: a known outage
# /24: 256 addresses -> 251 usable
# /22: 1,024 addresses -> 1,019 usable
print("\nOverlap check against networks you may later need to connect:")
for name, cidr in {"on-premises": "10.20.128.0/17", "partner VPC": "10.30.0.0/16",
"k8s pod CIDR": "10.20.0.0/16",
"another region": "10.21.0.0/16"}.items():
ov = VPC.overlaps(ipaddress.ip_network(cidr))
print(f" {name:16s} {cidr:18s} overlaps: {str(ov):5s}"
f"{' -> peering/VPN IMPOSSIBLE without NAT' if ov else ' -> can be peered'}")
# on-premises 10.20.128.0/17 overlaps: True -> peering/VPN IMPOSSIBLE without NAT
# partner VPC 10.30.0.0/16 overlaps: False -> can be peered
# k8s pod CIDR 10.20.0.0/16 overlaps: True -> peering/VPN IMPOSSIBLE without NAT
# another region 10.21.0.0/16 overlaps: False -> can be peered
#
# Run this check BEFORE creating the VPC. Afterwards the only remedies are NAT
# (which destroys address transparency) or renumbering (which means downtime).
print("\nSecurity groups vs network ACLs -- the distinction that trips everyone up:")
def sg_allows(rules, direction, port, established=False):
if established: # STATEFUL: the reply needs no rule at all
return True
return any(r["dir"] == direction and r["port"] == port for r in rules)
def nacl_allows(rules, direction, port):
for r in sorted(rules, key=lambda x: x["num"]): # lowest rule number wins
if r["dir"] == direction and r["port"] in (port, "all"):
return r["action"] == "allow"
return False # implicit deny at the end
SG = [{"dir": "out", "port": 443}]
NACL = [{"num": 100, "dir": "out", "port": 443, "action": "allow"},
{"num": 200, "dir": "in", "port": "all", "action": "deny"}]
print(f" outbound 443 : SG {sg_allows(SG,'out',443)} "
f"NACL {nacl_allows(NACL,'out',443)}")
print(f" the REPLY : SG {sg_allows(SG,'in',54321,established=True)} (stateful) "
f"NACL {nacl_allows(NACL,'in',54321)} (stateless -> BLOCKED)")
# outbound 443 : SG True NACL True
# the REPLY : SG True (stateful) NACL False (stateless -> BLOCKED)
#
# A NACL needs an explicit inbound allow for the EPHEMERAL range 1024-65535.
# Omitting it makes connections HANG rather than fail cleanly, which sends
# people to debug the application instead of the ACL.
# --- Address planning: verify before you build, not after ---
aws ec2 describe-vpcs --query 'Vpcs[].[VpcId,CidrBlock,Tags[?Key==`Name`].Value|[0]]' \
--output table
# List every CIDR your organisation already uses, then check the new one against
# all of them. This is a five-minute check that prevents a multi-month migration.
# --- Is this subnet public? Read the ROUTE TABLE, not the name ---
SUBNET=subnet-0a1b2c3d
RT=$(aws ec2 describe-route-tables \
--filters "Name=association.subnet-id,Values=$SUBNET" \
--query 'RouteTables[0].RouteTableId' --output text)
aws ec2 describe-route-tables --route-table-ids "$RT" \
--query 'RouteTables[0].Routes[].[DestinationCidrBlock,GatewayId,NatGatewayId]' \
--output table
# 0.0.0.0/0 igw-0abc... None <- PUBLIC, whatever the subnet is called
# 0.0.0.0/0 None nat-0def... <- private with outbound egress
# (no default route) <- fully isolated
# --- How many addresses are actually left? The pre-incident check ---
aws ec2 describe-subnets --query \
'Subnets[].[SubnetId,CidrBlock,AvailableIpAddressCount,AvailabilityZone]' \
--output table
# Alert when AvailableIpAddressCount drops below your scale-out headroom.
# --- Security group vs NACL: which one is blocking you? ---
aws ec2 describe-security-groups --group-ids sg-0abc \
--query 'SecurityGroups[0].[IpPermissions,IpPermissionsEgress]'
aws ec2 describe-network-acls --filters "Name=association.subnet-id,Values=$SUBNET" \
--query 'NetworkAcls[0].Entries[].[RuleNumber,Egress,Protocol,PortRange,RuleAction]' \
--output table
# The signature: if outbound is allowed and connections HANG, suspect the NACL's
# missing inbound ephemeral-range rule, not the security group.
# --- Reachability analysis: let the provider prove the path for you ---
aws ec2 create-network-insights-path --source i-0aaa --destination i-0bbb \
--protocol tcp --destination-port 5432
aws ec2 start-network-insights-analysis --network-insights-path-id nip-0abc
# It names the exact component that blocks the path, which beats guessing.
Key takeaways
- A VPC is a software-defined isolated network built on the overlay machinery of Section 13.2, and its real value is that the network becomes a versionable artefact.
- A subnet is zone-scoped, which is why zone redundancy multiplies subnet count.
- Address planning is irreversible: overlapping CIDRs make peering and VPN impossible without NAT, so allocate from one organisation-wide plan with generous headroom.
- Providers reserve about five addresses per subnet, so a
/28gives eleven usable — too few for anything that scales. - Public versus private is a route, not a property: a default route to an internet gateway is what makes a subnet public, which makes route-table review a security control.
- Security groups are stateful (replies allowed automatically); network ACLs are stateless and need explicit rules for the inbound ephemeral range.
- A missing NACL ephemeral rule makes connections hang rather than fail, which misdirects debugging toward the application.
- Use private endpoints to reach provider services without traversing the internet.
🧪 Practice
- Plan a VPC for two regions, three zones each, with public, private, and data tiers, from
10.40.0.0/14. State the CIDR of every subnet and the remaining headroom.- Explain why a
/28subnet for an auto-scaling load balancer is dangerous, and compute the usable addresses.- A service in a private subnet can open connections that then hang. Give the two most likely causes and the command that distinguishes them.
- Interview-style: Two years after launch, a company must connect its VPC to a newly acquired subsidiary whose network uses the same address range. (Hint: name the property that makes routing impossible here, then list the remedies and what each one costs in transparency or downtime.)
Cloud Interconnect and Direct Connect
Theory
Most organisations run a hybrid: some workloads in a cloud region, some on their own premises, and traffic between them. The question is what carries that traffic, and there are two families of answer.
Over the internet, encrypted. An IPsec VPN between an on-premises device and a cloud VPN gateway is available in minutes, costs nothing beyond the gateway, and works from anywhere. Its limitations follow from the internet being best-effort: latency and jitter vary, throughput per tunnel is capped by encryption and by the provider's per-tunnel limit (commonly around 1.25 Gb/s), and there is no service guarantee. For many workloads this is entirely adequate.
Over a dedicated circuit. A private link between your equipment and the provider's network — Direct Connect, ExpressRoute, Cloud Interconnect — gives predictable latency and jitter, a real bandwidth guarantee, a contractual SLA, and traffic that never touches the public internet. It takes weeks to provision, involves a cross-connect in a colocation facility or a partner, and carries a monthly port charge.
The decision is usually made by a factor people do not expect: egress pricing. Providers charge for data leaving their network, and the rate over the internet is several times the rate over a dedicated interconnect. Once monthly egress is substantial, the dedicated circuit pays for its own port fee purely on the per-gigabyte difference, before any consideration of latency or reliability. Below that crossover, the internet is cheaper. Computing the break-even point is the single most useful thing to do when this question arises, and it is usually in the region of tens of terabytes per month.
Beyond cost, a dedicated interconnect buys things money cannot buy on the internet: a latency SLA, freedom from competing with internet traffic, private addressing end to end with BGP route exchange, and — often decisive — the ability to satisfy a regulatory requirement that traffic not traverse the public internet.
The essential caveat is redundancy. A single circuit is a single point of failure with a lead time of weeks, which is the worst possible combination. Two circuits give a dramatic availability improvement, but only if they are genuinely independent: different provider routers, different colocation facilities, different physical paths, ideally different carriers. Two circuits through the same conduit share a failure that the availability arithmetic does not capture, and the standard recommendation is to keep a VPN configured as a backup path even when dedicated circuits exist, precisely because it fails differently.
Architecturally, connections are increasingly aggregated through a transit gateway (or virtual hub) rather than attached to individual VPCs. Instead of a mesh of peerings and one interconnect per VPC, everything attaches to a hub that routes between them. This turns N-squared peering relationships into N attachments, and centralises the route policy — the same hub-and-spoke reasoning that applies to any interconnection problem.
Examples
Three ways to reach a cloud, and where each fits
1. IPsec VPN over the internet
[ on-prem router ]===encrypted===[ internet ]===[ cloud VPN gateway ]
minutes to set up | best-effort quality | ~1.25 Gb/s per tunnel | high egress rate
2. Dedicated interconnect
[ on-prem router ]---[ colo cross-connect ]---[ provider router ]---[ VPC ]
weeks to provision | SLA-backed | 1-100 Gb/s | low egress rate | port fee
3. Hub and spoke via a transit gateway
[ on-premises ]
|
[ transit gateway ]
/ | \
[ VPC A ] [ VPC B ] [ VPC C ]
N attachments instead of N(N-1)/2 peerings, and ONE place for route policy
Peering mesh growth, the reason hubs win:
3 VPCs -> 3 peerings 6 VPCs -> 15 peerings
4 VPCs -> 6 peerings 10 VPCs -> 45 peerings
5 VPCs -> 10 peerings 20 VPCs -> 190 peerings
# Connecting on-premises to a cloud: the options, and what actually decides it.
OPTIONS = {
# name: (setup weeks, Mb/s, jitter, per-GB egress, monthly port fee)
"internet + IPsec VPN": (0.2, 1_250, "variable", 0.09, 0),
"internet + SD-WAN": (0.5, 2_500, "managed", 0.09, 200),
"dedicated 1G link": (8, 1_000, "low", 0.02, 1_800),
"dedicated 10G link": (10, 10_000, "low", 0.02, 6_000),
"partner/exchange 1G": (2, 1_000, "low", 0.02, 2_400),
}
print(f"{'option':24s} {'setup':>7s} {'Mb/s':>7s} {'jitter':>9s} "
f"{'egress/GB':>10s} {'port/mo':>9s}")
for name, (wk, mbps, jit, gb, port) in OPTIONS.items():
print(f"{name:24s} {wk:5.1f}wk {mbps:7,} {jit:>9s} {gb:9.2f} {port:8,}")
# option setup Mb/s jitter egress/GB port/mo
# internet + IPsec VPN 0.2wk 1,250 variable 0.09 0
# internet + SD-WAN 0.5wk 2,500 managed 0.09 200
# dedicated 1G link 8.0wk 1,000 low 0.02 1,800
# dedicated 10G link 10.0wk 10,000 low 0.02 6,000
# partner/exchange 1G 2.0wk 1,000 low 0.02 2,400
print("\nThe decision is usually EGRESS COST, not bandwidth or latency:")
print(f"{'monthly egress':>15s} " + " ".join(f"{n[:16]:>17s}" for n in OPTIONS))
for tb in (1, 10, 50, 200, 1000):
row = [f"{tb*1000*per_gb + port:17,.0f}"
for _, _, _, per_gb, port in OPTIONS.values()]
print(f"{tb:12,} TB " + " ".join(row))
# monthly egress internet + IPsec internet + SD-WA dedicated 1G lin dedicated 10G li partner/exchange
# 1 TB 90 290 1,820 6,020 2,420
# 10 TB 900 1,100 2,000 6,200 2,600
# 50 TB 4,500 4,700 2,800 7,000 3,400
# 200 TB 18,000 18,200 5,800 10,000 6,400
# 1,000 TB 90,000 90,200 21,800 26,000 22,400
def monthly(name, tb):
_, _, _, per_gb, port = OPTIONS[name]
return tb * 1000 * per_gb + port
lo, hi = 0.0, 2000.0 # bisect for the crossover point
for _ in range(60):
mid = (lo + hi) / 2
if monthly("internet + IPsec VPN", mid) < monthly("dedicated 1G link", mid):
lo = mid
else:
hi = mid
print(f"\nVPN vs dedicated 1G break-even: {lo:.1f} TB/month "
f"({monthly('internet + IPsec VPN', lo):,.0f} vs "
f"{monthly('dedicated 1G link', lo):,.0f} per month)")
# VPN vs dedicated 1G break-even: 25.7 TB/month (2,314 vs 2,314 per month)
#
# Below it, use the internet. Above it, the dedicated link pays for its own port
# fee on the per-GB difference alone, before counting any quality benefit.
print("\nRedundancy arithmetic (0.5% monthly unavailability per circuit):")
for n in (1, 2, 3):
a = 1 - 0.005 ** n
print(f" {n} circuit(s) -> {100*a:9.5f}% available "
f"({(1-a)*30*24*60:8.1f} minutes/month expected downtime)")
# 1 circuit(s) -> 99.50000% available ( 216.0 minutes/month expected downtime)
# 2 circuit(s) -> 99.99750% available ( 1.1 minutes/month expected downtime)
# 3 circuit(s) -> 99.99999% available ( 0.0 minutes/month expected downtime)
#
# This arithmetic assumes INDEPENDENCE. Two circuits in the same conduit, or
# landing on the same provider router, share a failure it does not model -- so
# keep a VPN configured as a backup, because it fails differently.
print("\nPeering mesh vs hub: why transit gateways exist")
for n in (3, 5, 10, 20, 50):
print(f" {n:2d} VPCs -> {n*(n-1)//2:4,} peerings (full mesh) "
f"vs {n:2d} attachments (hub)")
# 3 VPCs -> 3 peerings (full mesh) vs 3 attachments (hub)
# 5 VPCs -> 10 peerings (full mesh) vs 5 attachments (hub)
# 10 VPCs -> 45 peerings (full mesh) vs 10 attachments (hub)
# 20 VPCs -> 190 peerings (full mesh) vs 20 attachments (hub)
# 50 VPCs -> 1,225 peerings (full mesh) vs 50 attachments (hub)
# --- Is the interconnect up, and is BGP actually exchanging routes? ---
aws directconnect describe-connections \
--query 'connections[].[connectionId,connectionState,bandwidth,location]' --output table
aws directconnect describe-virtual-interfaces \
--query 'virtualInterfaces[].[virtualInterfaceId,virtualInterfaceState,bgpPeers[0].bgpStatus]' \
--output table
# dxvif-0abc available up <- both must be up; "available" alone is not enough
# --- On the customer router: what are we sending and receiving? ---
vtysh -c 'show bgp summary'
vtysh -c 'show bgp neighbors 169.254.255.1 advertised-routes' | head
vtysh -c 'show bgp neighbors 169.254.255.1 routes' | head
# A classic hybrid failure: the cloud side advertises correctly, the on-premises
# side never advertises its own prefixes, so return traffic has nowhere to go.
# --- Is traffic ACTUALLY using the interconnect, or silently on the VPN? ---
ip route get 10.20.4.10
# 10.20.4.10 via 169.254.255.1 dev dx-vif <- interconnect
# 10.20.4.10 via 169.254.21.1 dev vti0 <- VPN backup: you are paying
# interconnect fees for nothing
mtr -rwc 20 10.20.4.10 # latency and jitter tell you which path you are on
# --- Verify redundancy is real, not nominal ---
aws directconnect describe-connections \
--query 'connections[].[connectionId,location,awsDevice]' --output table
# If both circuits show the same `location` or the same `awsDevice`, they share
# a failure domain and the availability arithmetic above does not apply.
# --- Measure egress to decide the crossover honestly ---
aws cloudwatch get-metric-statistics --namespace AWS/EC2 \
--metric-name NetworkOut --start-time "$(date -u -d '30 days ago' +%FT%TZ)" \
--end-time "$(date -u +%FT%TZ)" --period 86400 --statistics Sum \
--query 'Datapoints[].Sum' --output text | tr '\t' '\n' | paste -sd+ | bc
# Divide by 1e12 for TB/month and compare against the break-even figure.
Key takeaways
- Hybrid connectivity is either encrypted over the internet (minutes to deploy, best-effort, roughly 1.25 Gb/s per tunnel) or a dedicated circuit (weeks to provision, SLA-backed, low egress rate, monthly port fee).
- The deciding factor is usually egress pricing, not bandwidth or latency: the internet rate is several times the interconnect rate.
- Compute the break-even volume explicitly; it typically lands in the tens of terabytes per month.
- Dedicated circuits also buy a latency SLA, isolation from internet traffic, private addressing with BGP, and regulatory compliance.
- A single circuit is a single point of failure with a weeks-long lead time. Two circuits help only if they are genuinely independent in router, facility, path, and ideally carrier.
- Keep a VPN configured as backup even with dedicated circuits, because it fails differently.
- Use a transit gateway hub to turn N-squared peerings into N attachments with one place for route policy.
- Verify that traffic is actually on the interconnect; silently falling back to the VPN means paying for both.
🧪 Practice
- Compute the monthly cost of 80 TB of egress over a VPN at 0.09 per GB versus a dedicated link at 0.02 per GB with a 2,000 port fee, and state which is cheaper.
- Explain why two dedicated circuits terminating on the same provider router do not deliver the availability the arithmetic suggests.
- Compare a full mesh of peerings against a transit gateway for 12 VPCs, giving the relationship counts and one operational advantage of each.
- Interview-style: A company provisions a redundant pair of 10 Gb/s interconnects, and a fibre cut takes both down simultaneously. (Hint: the availability model assumes one specific property of the two circuits — identify it, then list the four dimensions along which you would verify it before signing.)
SD-WAN Architecture
Theory
The traditional branch office network was built from private circuits, usually MPLS, with one or two per site and a routing configuration that decided paths by static metrics. It worked, and by the 2010s it had three problems that compounded.
It was expensive. A private circuit costs an order of magnitude more per megabit than consumer broadband. It was slow to change. Adding a site, or bandwidth to one, meant a carrier order with weeks of lead time. And it routed traffic the wrong way. When applications moved to the cloud, branch traffic was still backhauled to a central data centre to exit through the corporate firewall, so a user in a branch reaching a cloud application sent packets to headquarters and back out again — the "trombone" or hairpin, adding tens of milliseconds and consuming expensive circuit capacity to reach a service on the public internet.
SD-WAN applies the chapter's central pattern to the wide area: separate the control plane from the forwarding devices, put policy in a controller, and let branch devices use any available transport — MPLS, broadband, LTE or 5G, satellite — as an undifferentiated pool. The pieces:
- Edge devices at each site, forming encrypted tunnels over every available link to other sites and to cloud gateways.
- A controller distributing policy and topology, so a new site is provisioned from a template rather than configured by hand.
- Continuous measurement on every tunnel: loss, latency, jitter, sampled every few seconds.
- Application-aware policy that states requirements per application class, not per destination prefix.
- Local internet breakout, so cloud and internet traffic exits directly at the branch instead of being backhauled.
The property that distinguishes it from traditional routing is worth stating precisely: path selection is driven by measured quality against a per-application requirement, and it changes continuously without any configuration change. A traditional WAN's routing reflects the topology as configured; an SD-WAN's reflects the network as currently measured. When broadband degrades in the afternoon, voice moves to MPLS by itself, and when it recovers, voice moves back.
There is a further capability that is often overlooked and is genuinely distinctive. When no single path meets an application's requirement, SD-WAN can synthesise an acceptable path through remediation: forward error correction to repair loss, or duplicating packets across two paths and using whichever copy arrives first. Duplication multiplies the effective loss rates together, so two mediocre links can deliver a better-than-either result. It costs double the bytes, so it is applied only to the class that needs it and only while no single path qualifies. This is why SD-WAN is described as remediation rather than merely path selection.
The trade-offs are real. The controller is a new critical dependency. Encrypted overlays over consumer broadband inherit its variability — measurement and remediation manage that variability rather than removing it. And security moves with the traffic: local internet breakout means each branch is now an internet edge, so the firewall function must be distributed or delivered as a cloud service. That last point is why SD-WAN and SASE (cloud-delivered security combined with SD-WAN) converged into one conversation: once you stop backhauling, you must put the security controls where the traffic now exits.
Examples
The trombone, and its removal
BEFORE: backhaul everything to the data centre
[ branch ] --MPLS--> [ HQ data centre ] --firewall--> [ internet ] -> [ SaaS ]
^ ^ ^
20 ms +35 ms +25 ms
total ~80 ms for a service that is 25 ms away, using the most
expensive circuit in the network to carry internet traffic
AFTER: local breakout, policy enforced at the branch or in a cloud PoP
[ branch ] --broadband--> [ internet ] --> [ SaaS ] ~25 ms
| ^
| cloud security PoP (SASE) inspects here
+--MPLS/overlay--> [ HQ ] only for traffic that belongs at HQ
Policy is per APPLICATION, not per prefix:
application requirement eligible now
----------- ------------------------------ --------------------
voice loss<1% lat<150ms jitter<30ms mpls
video-conf loss<0.5% lat<150ms jitter<30ms mpls
erp loss<1% lat<200ms mpls, broadband
backup cost-optimised broadband
web cost-optimised, loss<2% broadband
...and when broadband degrades, the RIGHT-HAND COLUMN changes by itself.
# SD-WAN: choose the path per application from MEASURED quality, not a static route.
LINKS = {
# name: (loss %, latency ms, jitter ms, cost per GB, capacity Mb/s)
"mpls": (0.01, 18, 1.2, 0.85, 50),
"broadband": (0.60, 26, 9.0, 0.02, 500),
"lte": (1.40, 48, 22.0, 4.00, 100),
}
APPS = { # the SLA each class needs in order to be usable
"voice": dict(max_loss=1.0, max_latency=150, max_jitter=30, prefer="quality"),
"video-conf": dict(max_loss=0.5, max_latency=150, max_jitter=30, prefer="quality"),
"erp": dict(max_loss=1.0, max_latency=200, max_jitter=999, prefer="quality"),
"backup": dict(max_loss=5.0, max_latency=999, max_jitter=999, prefer="cost"),
"web-browsing": dict(max_loss=2.0, max_latency=300, max_jitter=999, prefer="cost"),
}
def eligible(link, req):
loss, lat, jit, _, _ = LINKS[link]
return loss <= req["max_loss"] and lat <= req["max_latency"] \
and jit <= req["max_jitter"]
def choose(app, req):
ok = [l for l in LINKS if eligible(l, req)]
if not ok:
return None, "no single path meets the SLA"
if req["prefer"] == "cost":
pick = min(ok, key=lambda l: LINKS[l][3])
return pick, f"cheapest eligible ({LINKS[pick][3]:.2f}/GB)"
# a simple quality score: latency + jitter + a heavy weight on loss
pick = min(ok, key=lambda l: LINKS[l][1] + LINKS[l][2] + LINKS[l][0] * 50)
return pick, f"best quality among {ok}"
print("Steady state -- all three links healthy:")
for app, req in APPS.items():
p, why = choose(app, req)
print(f" {app:14s} {str(p):12s} {why}")
# voice mpls best quality among ['mpls', 'broadband']
# video-conf mpls best quality among ['mpls']
# erp mpls best quality among ['mpls', 'broadband']
# backup broadband cheapest eligible (0.02/GB)
# web-browsing broadband cheapest eligible (0.02/GB)
print("\nBroadband degrades (a common afternoon pattern):")
LINKS["broadband"] = (4.20, 71, 38.0, 0.02, 500)
for app, req in APPS.items():
p, why = choose(app, req)
print(f" {app:14s} {str(p):12s} {why}")
# voice mpls best quality among ['mpls']
# video-conf mpls best quality among ['mpls']
# erp mpls best quality among ['mpls']
# backup broadband cheapest eligible (0.02/GB)
# web-browsing mpls cheapest eligible (0.85/GB)
# ^ quality forced an expensive choice: visible in the bill
print("\nAnd the MPLS circuit fails entirely:")
saved = LINKS.pop("mpls")
for app, req in APPS.items():
p, why = choose(app, req)
print(f" {app:14s} {str(p):12s} {why}")
# voice None no single path meets the SLA
# video-conf None no single path meets the SLA
# erp None no single path meets the SLA
# backup broadband cheapest eligible (0.02/GB)
# web-browsing lte cheapest eligible (4.00/GB)
#
# THREE different decisions, and not one route was reconfigured.
print("\nWhen no single path qualifies, SD-WAN can SYNTHESISE one by duplicating")
print("packets across two paths and taking the first copy to arrive:")
def duplicate(a, b):
la, ta, ja, ca, _ = LINKS[a]
lb, tb, jb, cb, _ = LINKS[b]
return dict(loss=la/100 * lb/100 * 100, # BOTH copies must be lost
latency=min(ta, tb), # first arrival wins
jitter=min(ja, jb),
cost=ca + cb) # you pay for both copies
d = duplicate("broadband", "lte")
req = APPS["voice"]
ok = (d["loss"] <= req["max_loss"] and d["latency"] <= req["max_latency"]
and d["jitter"] <= req["max_jitter"])
print(f" broadband alone : loss {LINKS['broadband'][0]:5.2f}% "
f"latency {LINKS['broadband'][1]:3.0f} ms jitter {LINKS['broadband'][2]:4.1f} ms")
print(f" lte alone : loss {LINKS['lte'][0]:5.2f}% "
f"latency {LINKS['lte'][1]:3.0f} ms jitter {LINKS['lte'][2]:4.1f} ms")
print(f" DUPLICATED : loss {d['loss']:5.3f}% latency {d['latency']:3.0f} ms "
f"jitter {d['jitter']:4.1f} ms cost {d['cost']:.2f}/GB")
print(f" meets the voice SLA: {ok}")
# broadband alone : loss 4.20% latency 71 ms jitter 38.0 ms
# lte alone : loss 1.40% latency 48 ms jitter 22.0 ms
# DUPLICATED : loss 0.059% latency 48 ms jitter 22.0 ms cost 4.02/GB
# meets the voice SLA: True
#
# Two unusable links became one usable path, because independent loss rates
# MULTIPLY. The cost is double the bytes, so it is applied only to the class
# that needs it and only while no single path qualifies.
LINKS["mpls"] = saved
MONTHLY_GB = 4_000
print(f"\nThe economics that drove adoption ({MONTHLY_GB:,} GB/month per branch):")
for label, mix in (("all MPLS", {"mpls": 1.0}),
("SD-WAN: 20% MPLS, 80% broadband", {"mpls": 0.2, "broadband": 0.8}),
("all broadband", {"broadband": 1.0})):
cost = sum(MONTHLY_GB * frac * LINKS[l][3] for l, frac in mix.items())
print(f" {label:34s} {cost:9,.0f} per month")
# all MPLS 3,400 per month
# SD-WAN: 20% MPLS, 80% broadband 744 per month
# all broadband 80 per month
# Across 200 branches the middle option saves about 6.4 M per year while keeping
# a quality-guaranteed path available for the classes that need one.
# --- What does the edge device currently measure on each transport? ---
# (vendor CLIs differ; every SD-WAN exposes the same four numbers per tunnel)
show sdwan app-route stats | head -20
# TUNNEL LOSS% LATENCY JITTER STATE
# mpls-to-hub 0.01 18ms 1.2ms up
# biz-internet-hub 4.20 71ms 38.0ms up <- degraded, not down
# lte-to-hub 1.40 48ms 22.0ms up
# "up but not usable" is the state traditional routing cannot express, and it is
# exactly the state SD-WAN exists to handle.
# --- Which policy is steering which application right now? ---
show sdwan policy from-vsmart
show sdwan app-route sla-class
show sdwan tunnel statistics fec # is FEC or duplication currently active?
# --- Verify local breakout is actually happening (no trombone) ---
traceroute -n outlook.office365.com
# 1 192.168.10.1 <- branch edge
# 2 81.x.x.x <- local ISP: CORRECT, breaking out locally
# versus the trombone signature:
# 2 10.255.0.1 <- the overlay to HQ: traffic is being backhauled
# --- Prove the saving, per application, with a real measurement ---
for host in outlook.office365.com teams.microsoft.com; do
echo -n "$host "; ping -c 20 -q "$host" | awk -F/ '/rtt/{print $5" ms avg, "$7" ms mdev"}'
done
# --- The security consequence of breakout: each branch is now an internet edge ---
# Confirm traffic is still being inspected somewhere.
curl -s https://ipinfo.io/org
# "AS13335 Cloudflare" <- egressing via a cloud security PoP (SASE): inspected
# "AS3320 Deutsche Telekom" <- egressing raw to the local ISP: NOT inspected
Key takeaways
- Traditional branch WANs were expensive per megabit, slow to change, and backhauled cloud traffic through headquarters, creating the trombone.
- SD-WAN separates control from forwarding at the wide-area edge and treats every transport as one pool: MPLS, broadband, LTE, satellite.
- Its distinguishing property is that path selection follows measured quality against per-application requirements and changes continuously with no configuration change.
- Local internet breakout removes the trombone, cutting both latency and private-circuit consumption.
- When no single path qualifies, remediation synthesises one: forward error correction, or packet duplication whose independent loss rates multiply. It costs double the bytes, so it is applied selectively.
- A traditional WAN cannot express the state "up but not usable", which is precisely the state SD-WAN handles.
- Costs: the controller is a new critical dependency, and consumer transports remain variable.
- Security moves with the traffic: breakout makes every branch an internet edge, which is why SD-WAN and cloud-delivered security (SASE) converged.
🧪 Practice
- Given links measuring 0.2% loss / 22 ms / 2 ms jitter and 3.5% loss / 40 ms / 25 ms jitter, decide which carries voice (loss under 1%, latency under 150 ms, jitter under 30 ms) and which carries backups.
- Compute the effective loss of duplicating packets across links with 3% and 2% loss, and state the bandwidth cost.
- Explain the trombone effect with a concrete latency calculation, then state what local breakout changes and what new obligation it creates.
- Interview-style: After an SD-WAN rollout, a branch's monthly circuit bill rises sharply even though usage is unchanged. (Hint: policy chooses by measured quality, and quality on the cheap transport may have degraded — work out what the policy would do next and what that does to the expensive link's volume.)
Edge and Fog Computing
Theory
Cloud computing centralised: a handful of very large regions, enormous economies of scale, and one operational model. Edge computing pushes computation back outward, and the reason is not fashion but two constraints that centralisation cannot satisfy.
The first is the speed of light. A round trip to a region 1,500 km away costs about 22 milliseconds in fibre propagation alone, before any processing. For a web page that is invisible. For a control loop that must close in one millisecond, or a headset that must render a frame within twenty milliseconds of head movement, it is disqualifying. No amount of engineering removes it; only shortening the distance does.
The second is data volume. A factory with hundreds of high-rate sensors generates more data than is economical to ship anywhere. Processing locally and transmitting only aggregates or anomalies can reduce backhaul by three orders of magnitude, and this argument has nothing to do with latency at all. It is frequently the larger of the two motivations in industrial deployments.
There is a spectrum of placements, not a binary choice:
| Tier | Typical distance | Compute available | Fits |
|---|---|---|---|
| Device / on-premises | metres | very limited | control loops, safety interlocks |
| Access edge (cell site, cabinet) | kilometres | limited | augmented reality, local analytics |
| Metro edge (city PoP, CDN node) | tens of km | moderate | cloud gaming, content, inference |
| Regional edge | hundreds of km | substantial | regional services, aggregation |
| Cloud region | hundreds to thousands of km | effectively unlimited | training, analytics, storage |
An important correction to the usual intuition: below roughly a hundred kilometres, propagation stops being the bottleneck. The radio access network, the access technology, and processing dominate. Moving compute from a metro edge to a cell site may save half a millisecond of propagation while changing nothing else, so "closer" is not automatically "better" — the question is always which term of the budget you are actually attacking.
The counter-pressure is operational and severe, and it is why the edge has not swallowed the cloud. Three cloud regions can be operated by a small team. Twenty thousand access-edge sites cannot. Every deployment must reach every site, every site needs monitoring and physical security, capacity cannot be pooled across sites, and failures are local and numerous. The cost is not primarily the hardware; it is that operational complexity scales with site count.
Hence the design rule: push to the edge only what the latency or data-volume budget forces, and keep everything else central, where it is cheap to operate and easy to update. In practice this means a split architecture — inference at the edge with training in the cloud, filtering at the edge with analytics in the cloud, control loops at the edge with coordination in the cloud.
A note on vocabulary. Fog computing originally named an intermediate layer between devices and cloud, emphasising a hierarchy of processing tiers, and MEC (multi-access edge computing) names compute placed inside a mobile operator's network at or near base stations. The distinctions matter less than the tiering idea itself, which is what the table above captures.
Examples
The latency ladder, and where the budget actually goes
tier distance propagation RTT access+processing total
--------------------- -------- --------------- ----------------- -----
device / on-premises 50 m 0.00 ms 0.5 ms 0.5 ms
access edge (cell site) 5 km 0.07 ms 8.0 ms 8.1 ms
metro edge 40 km 0.58 ms 10.0 ms 10.6 ms
regional edge 300 km 4.35 ms 12.0 ms 16.4 ms
cloud region 1,500 km 21.75 ms 14.0 ms 35.8 ms
^ only here does distance dominate
Read it carefully: between the access edge and the metro edge, propagation
differs by 0.5 ms and the totals differ by 2.5 ms. Placement decisions at
short range are NOT propagation decisions.
The split architecture that results:
[ device ] control loop, safety <- must be local, 1 ms
|
[ access/metro edge ] inference, filtering <- 10 ms class
| aggregate 100 Hz -> 1 Hz
|
[ cloud region ] training, analytics, storage <- no latency requirement
receives 1/1000 of the raw data
# Edge computing: distance sets a FLOOR, but access technology often dominates.
FIBER_MS_PER_KM = 0.005 * 1.45 # ~5 us/km in glass, x1.45 for real routes
TIERS = [
# name, km, access+processing RTT ms, compute capability
("device / on-premises", 0.05, 0.5, "very limited"),
("access edge (cell site)", 5, 8.0, "limited"), # 5G radio dominates here
("metro edge", 40, 10.0, "moderate"),
("regional edge", 300, 12.0, "substantial"),
("cloud region", 1500, 14.0, "effectively unlimited"),
]
print(f"{'tier':26s} {'km':>7s} {'prop RTT':>10s} {'access+proc':>12s} "
f"{'total RTT':>10s} compute")
totals = {}
for name, km, overhead, compute in TIERS:
prop = 2 * km * FIBER_MS_PER_KM
totals[name] = prop + overhead
print(f"{name:26s} {km:7.0f} {prop:9.2f}ms {overhead:11.1f}ms "
f"{prop + overhead:9.2f}ms {compute}")
# tier km prop RTT access+proc total RTT compute
# device / on-premises 0 0.00ms 0.5ms 0.50ms very limited
# access edge (cell site) 5 0.07ms 8.0ms 8.07ms limited
# metro edge 40 0.58ms 10.0ms 10.58ms moderate
# regional edge 300 4.35ms 12.0ms 16.35ms substantial
# cloud region 1500 21.75ms 14.0ms 35.75ms effectively unlimited
print("\nWhere each workload can actually live, against its end-to-end RTT budget:")
for app, budget in (("industrial motion control", 1),
("AR/VR motion-to-photon", 20),
("cloud gaming", 40),
("interactive web", 100),
("batch analytics", 10_000)):
ok = [n for n, t in totals.items() if t <= budget]
print(f" {app:26s} budget {budget:6,} ms -> "
f"{', '.join(ok) if ok else 'NO tier qualifies: redesign the interaction'}")
# industrial motion control budget 1 ms -> device / on-premises
# AR/VR motion-to-photon budget 20 ms -> device / on-premises, access edge (cell site), metro edge, regional edge
# cloud gaming budget 40 ms -> all five tiers qualify
# interactive web budget 100 ms -> all five tiers qualify
# batch analytics budget 10,000 ms -> all five tiers qualify
#
# Only the 1 ms case genuinely forces the device. Note that cloud gaming fits a
# cloud region on NETWORK latency alone -- what pushes it to the metro edge in
# practice is the rest of the budget: encode, decode and display.
print("\nThe economic counter-pressure, which is why edge is not used everywhere:")
COST_PER_SITE_MONTHLY = 4_000
for tier, n in {"cloud region": 3, "regional edge": 20, "metro edge": 300,
"access edge": 20_000}.items():
print(f" {tier:16s} {n:7,} sites -> "
f"{n*COST_PER_SITE_MONTHLY/1e6:6.2f} M/month of footprint; every "
f"deploy must reach {n:,} places")
# cloud region 3 sites -> 0.01 M/month of footprint; every deploy must reach 3 places
# regional edge 20 sites -> 0.08 M/month of footprint; every deploy must reach 20 places
# metro edge 300 sites -> 1.20 M/month of footprint; every deploy must reach 300 places
# access edge 20,000 sites -> 80.00 M/month of footprint; every deploy must reach 20,000 places
#
# The hardware is not the problem. Operational complexity scaling with site
# count is the problem: 20,000 places to deploy, monitor, secure and repair.
print("\nData reduction -- often the larger reason to compute at the edge:")
SENSORS, HZ, BYTES = 500, 100, 24
raw_gb_per_day = SENSORS * HZ * BYTES * 86400 / 1e9
for label, factor in (("ship every raw sample", 1.0),
("aggregate to 1 Hz at the edge", 1/100),
("ship only anomalies (0.1%)", 0.001)):
print(f" {label:34s} {raw_gb_per_day*factor:8.2f} GB/day "
f"({raw_gb_per_day*factor*30:9.1f} GB/month)")
# ship every raw sample 103.68 GB/day ( 3110.4 GB/month)
# aggregate to 1 Hz at the edge 1.04 GB/day ( 31.1 GB/month)
# ship only anomalies (0.1%) 0.10 GB/day ( 3.1 GB/month)
#
# A 1000x reduction in backhaul is a capacity and cost argument with nothing to
# do with latency, and in industrial deployments it is usually the bigger one.
# --- Establish the latency floor before designing anything ---
# Measure, do not assume: the access network usually dominates at short range.
ping -c 50 -q <metro-edge-ip> | awk -F/ '/rtt/{print "metro "$5" ms avg"}'
ping -c 50 -q <regional-ip> | awk -F/ '/rtt/{print "regional "$5" ms avg"}'
ping -c 50 -q <cloud-region-ip> | awk -F/ '/rtt/{print "cloud "$5" ms avg"}'
# If metro and regional differ by 2 ms, moving compute between them buys 2 ms.
# Decide whether 2 ms justifies operating 15x more sites.
# --- Decompose the budget: which term is actually large? ---
mtr -rwc 50 <cloud-region-ip> # per-hop: where latency and jitter accumulate
curl -w 'dns %{time_namelookup}s tcp %{time_connect}s tls %{time_appconnect}s \
ttfb %{time_starttransfer}s total %{time_total}s\n' -o /dev/null -s https://api.example.com
# If TLS setup dominates, session resumption at the edge beats moving compute.
# --- Edge fleet reality: the operational cost, made visible ---
kubectl get nodes -l node-role.kubernetes.io/edge --no-headers | wc -l
kubectl get nodes -l node-role.kubernetes.io/edge \
-o custom-columns='NAME:.metadata.name,READY:.status.conditions[-1].type' | grep -vc Ready
# Every number above one is a site someone must visit or remotely recover.
# --- Verify data reduction is happening where you think ---
# Compare ingest at the edge with what leaves for the cloud.
curl -s localhost:9090/api/v1/query --data-urlencode \
'query=rate(edge_samples_ingested_total[5m])'
curl -s localhost:9090/api/v1/query --data-urlencode \
'query=rate(edge_samples_forwarded_total[5m])'
# The ratio is your actual reduction factor. If it is near 1, the edge is acting
# as a relay rather than a filter, and you are paying for sites that do nothing.
Key takeaways
- Edge computing answers two constraints centralisation cannot: propagation delay (about 22 ms round trip at 1,500 km) and data volume that is uneconomical to ship.
- Placement is a spectrum from device through access, metro, and regional edge to cloud region, not a binary choice.
- Below roughly 100 km, propagation is not the bottleneck — access technology and processing are — so "closer" is not automatically better.
- Decompose the budget before moving compute: if TLS setup or encoding dominates, shortening distance buys little.
- The counter-pressure is that operational complexity scales with site count: 3 regions are manageable, 20,000 edge sites are not, and hardware is not the main cost.
- Push to the edge only what the budget forces; keep the rest central where it is cheap to operate and easy to update.
- The resulting split is typically inference and filtering at the edge, training and analytics in the cloud.
- Data reduction of a thousandfold is a capacity and cost argument independent of latency, and often the larger motivation.
- Verify the reduction is real; an edge that relays rather than filters is paying for nothing.
🧪 Practice
- Compute the propagation round trip to a region 2,400 km away, and state which of a 5 ms, 50 ms, and 500 ms budget it can satisfy.
- A sensor fleet produces 200 sensors at 50 Hz with 16-byte samples. Compute daily raw volume and the volume after 10:1 and 1000:1 reduction.
- Explain why moving a workload from a metro edge to a cell site may not improve latency measurably, and what you would measure to decide.
- Interview-style: A team proposes deploying their inference service to 5,000 access-edge sites to cut latency from 25 ms to 22 ms. (Hint: put the 3 ms gain beside the change in number of places to deploy, monitor and repair, and ask which term of the 25 ms is actually being reduced.)
Multi-Region and Global Load Balancing
Theory
Serving users worldwide from one region means most users are far away, and losing that region means losing everything. Multi-region architecture addresses both, and it introduces a problem that no amount of network engineering solves: state.
The traffic-steering part is comparatively easy, with two mechanisms.
DNS-based global load balancing answers a name differently depending on who asks and which regions are healthy. It is simple, works with any protocol, and supports rich policies — latency-based, geographic, weighted, failover. Its weakness is failover time, and the arithmetic is unforgiving. Total time is detection (several health-check intervals) plus the record's TTL plus the time for resolvers and clients that ignore your TTL. That last term is the one you cannot control and it usually dominates: browsers and libraries cache aggressively, and some resolvers enforce their own minimums. A configuration that looks like one minute of exposure routinely behaves like several.
Lowering the TTL is not a free fix. Query load is inversely proportional to TTL, so a one-second TTL from a large client population turns authoritative DNS into a bottleneck and adds a resolution round trip to a large share of requests. Typical practice settles at 30 to 60 seconds as a compromise.
Anycast advertises the same address from many locations and lets BGP deliver each client to a topologically near one. Failover happens by route withdrawal, measured in seconds rather than minutes, and there is no client-side caching to defeat it. The trade is loss of control: you cannot choose which site a given client reaches, the choice follows BGP policy rather than latency, and a client can move between sites mid-session if routing changes. That makes anycast excellent for stateless, short-lived request-response traffic (DNS, CDNs, DDoS absorption) and awkward for long-lived stateful connections, though QUIC's connection migration reduces this problem.
The steering policies and their characteristic failure modes:
| Policy | Optimises | Fails when |
|---|---|---|
| Latency-based | lowest measured round trip | measurement is per-resolver, not per-client |
| Geolocation | regulatory or content requirements | users are on VPNs or roaming |
| Weighted | gradual rollout, canaries | ignores health and distance |
| Failover | simplicity | the secondary is idle and untested |
| Anycast | fast failover, attack absorption | no control over which site a client reaches |
Now the part that actually decides multi-region designs. Routing a user to a nearby region helps only if that region can serve the request, which means the data must be there. And distributing data means confronting a physical limit: a synchronous write quorum spanning regions cannot complete faster than the round trip between them. US-east to Europe is roughly 80 milliseconds, so every synchronous cross-region write costs at least that. Asynchronous replication avoids the latency and creates a loss window: a failover loses whatever had not replicated.
This is the real decision, and it is a data decision rather than a network one:
- Active-passive: one region serves, another stands ready. Simple, and failover risks data loss proportional to replication lag.
- Active-active with partitioned data: each region owns a subset (by user or tenant), so writes stay local. This works well and requires the application to be partitionable.
- Active-active with global consistency: synchronous quorum across regions. Correct, and every write pays the inter-region round trip.
- Active-active, eventually consistent: fast everywhere, and the application must tolerate conflicts and resolve them.
A last practical point: health checks define correctness. A check that only verifies a TCP port will happily keep sending traffic to a region whose database is unreachable. A check should exercise the dependency path that matters, and it should be evaluated from multiple vantage points, because a region that is unreachable from one probe location may be perfectly healthy from where users actually are.
Examples
Two steering mechanisms, and why their failover times differ so much
DNS-BASED
client -> resolver -> authoritative DNS ("which region?") -> region A
^ health checks decide the answer
failover = detect (3 x 10 s) + TTL (60 s) + clients ignoring TTL (~120 s)
= about 210 s of exposure
ANYCAST
client -> [ nearest site advertising 198.51.100.1, per BGP ]
failover = BGP withdraw (0.5 s) + reconvergence (2 s)
= about 2.5 s, with no client cache to defeat
The multi-region data problem, which steering cannot solve
region A (US-east) region B (EU-west)
+-----------------+ 80 ms RTT +-----------------+
| app + db |<-------------------->| app + db |
+-----------------+ +-----------------+
synchronous quorum write : >= 80 ms, ALWAYS, by physics
asynchronous replication : ~0 ms, and a failover loses the lag window
partitioned ownership : writes stay local; needs a partitionable app
Routing a user to a nearby region is useless if the write still has to
cross the ocean. The architecture is decided by the DATA, not the DNS.
# Global load balancing: DNS vs anycast, and why failover time differs so much.
def dns_failover(ttl, check_interval, checks_to_fail, propagation_slop):
detect = check_interval * checks_to_fail
return dict(detect=detect, ttl=ttl, slop=propagation_slop,
total=detect + ttl + propagation_slop)
def anycast_failover(bgp_withdraw_ms, reconverge_ms):
return dict(detect=bgp_withdraw_ms/1000, ttl=0, slop=reconverge_ms/1000,
total=(bgp_withdraw_ms + reconverge_ms)/1000)
d = dns_failover(ttl=60, check_interval=10, checks_to_fail=3, propagation_slop=120)
a = anycast_failover(bgp_withdraw_ms=500, reconverge_ms=2_000)
print(f"{'method':10s} {'detect':>9s} {'TTL wait':>9s} {'slop':>9s} {'TOTAL':>9s}")
print(f"{'DNS GLB':10s} {d['detect']:8.0f}s {d['ttl']:8.0f}s {d['slop']:8.0f}s "
f"{d['total']:8.0f}s")
print(f"{'anycast':10s} {a['detect']:8.1f}s {a['ttl']:8.0f}s {a['slop']:8.1f}s "
f"{a['total']:8.1f}s")
print(f"ratio: anycast recovers {d['total']/a['total']:.0f}x faster")
# method detect TTL wait slop TOTAL
# DNS GLB 30s 60s 120s 210s
# anycast 0.5s 0s 2.0s 2.5s
# ratio: anycast recovers 84x faster
#
# The "slop" term is resolvers and clients that ignore your TTL. It is the term
# you cannot control, and it dominates the total.
CLIENTS = 2_000_000
print("\nWhy not simply set the TTL to 1 second? Look at the query load:")
for ttl in (1, 10, 60, 300, 3600):
print(f" TTL {ttl:5,}s -> {CLIENTS/ttl:12,.0f} DNS queries/s "
f"from {CLIENTS:,} clients")
# TTL 1s -> 2,000,000 DNS queries/s from 2,000,000 clients
# TTL 10s -> 200,000 DNS queries/s from 2,000,000 clients
# TTL 60s -> 33,333 DNS queries/s from 2,000,000 clients
# TTL 300s -> 6,667 DNS queries/s from 2,000,000 clients
# TTL 3,600s -> 556 DNS queries/s from 2,000,000 clients
# A 1-second TTL makes authoritative DNS the bottleneck and adds a resolution
# round trip to a large share of requests. Practice settles at 30-60 s.
print("\nSteering policies, what each optimises, and how each fails:")
for p, (opt, fail) in {
"latency-based": ("send to the lowest-RTT healthy region",
"measurement is per-resolver, not per-client"),
"geolocation": ("send by client country or region",
"wrong for VPN users and mobile roaming"),
"weighted": ("split by fixed proportion",
"ignores health and distance; good for canaries"),
"failover": ("primary until unhealthy, then secondary",
"secondary capacity sits idle and untested"),
"anycast": ("the network chooses, by BGP",
"no control over WHICH site a client reaches")}.items():
print(f" {p:15s} {opt:44s} {fail}")
print("\nMulti-region write latency: the physics you cannot optimise away.")
for name, rtt in {"same region": 1, "US-east to US-west": 60,
"US-east to EU": 80, "US-east to Asia": 180}.items():
print(f" {name:20s} RTT {rtt:4d} ms -> synchronous quorum write "
f">= {rtt:4d} ms; async replication ~0 ms but a failover loses the lag")
# same region RTT 1 ms -> synchronous quorum write >= 1 ms; ...
# US-east to US-west RTT 60 ms -> synchronous quorum write >= 60 ms; ...
# US-east to EU RTT 80 ms -> synchronous quorum write >= 80 ms; ...
# US-east to Asia RTT 180 ms -> synchronous quorum write >= 180 ms; ...
#
# This is the actual multi-region decision: not how to route traffic, but
# whether the data layer can tolerate the round trip or the loss window.
# --- What does the world actually resolve your name to? ---
for r in 1.1.1.1 8.8.8.8 9.9.9.9 208.67.222.222; do
echo -n "$r -> "; dig +short @$r api.example.com | head -1
done
# Different answers mean the policy is working. Identical answers from resolvers
# on different continents mean it is NOT, and latency steering is a no-op.
# --- Is the TTL what you think, and is anyone honouring it? ---
dig api.example.com | grep -A1 'ANSWER SECTION'
# api.example.com. 60 IN A 198.51.100.10
# ^^ your intent; clients and resolvers may ignore it
dig +norecurse @ns1.example.com api.example.com | awk '/IN\s+A/{print $2}'
# --- Anycast or unicast? Same address from many places is the signature ---
for probe in lon fra nyc sin; do
echo -n "$probe: "; ssh probe-$probe "dig +short api.example.com; \
traceroute -n -m5 198.51.100.10 | tail -1"
done
# Same IP everywhere with SHORT, DIFFERENT paths = anycast.
# Same IP everywhere with one long path = unicast, and distant users suffer.
# --- Health checks: are they exercising the DEPENDENCY, or only the port? ---
curl -s https://api.example.com/healthz | jq .
# {"status":"ok"} <- proves nothing
# {"status":"ok","db":"ok","cache":"ok","replica_lag_s":0.4}
# ^ this one can actually fail when the region cannot serve requests
# --- Measure replication lag: it IS your data-loss window ---
psql -h replica.eu-west -c \
"SELECT now() - pg_last_xact_replay_timestamp() AS lag;"
# lag: 00:00:02.41 <- a failover right now loses ~2.4 s of writes.
# Decide in advance whether that is acceptable, and write it down.
# --- Rehearse the failover, or you do not have one ---
aws route53 get-health-check-status --health-check-id abcd1234
# Then actually fail the primary in a game day and measure the REAL total,
# including the clients that ignored the TTL. The measured number is always
# larger than the configured one.
Key takeaways
- DNS-based steering is flexible and protocol-agnostic, but failover is detection plus TTL plus clients that ignore the TTL — the uncontrollable term that usually dominates.
- Lowering the TTL is not free: query load is inversely proportional to TTL, so practice settles around 30 to 60 seconds.
- Anycast fails over by route withdrawal in seconds with no client cache to defeat, at the cost of no control over which site a client reaches.
- Anycast suits stateless short request-response traffic; long-lived stateful sessions are awkward, though QUIC connection migration helps.
- Each steering policy has a characteristic failure: latency steering measures resolvers not clients, geolocation breaks on VPNs, failover leaves the secondary untested.
- The decisive constraint is data, not routing: a synchronous cross-region quorum write cannot beat the inter-region round trip, and asynchronous replication trades that latency for a loss window.
- Choose deliberately among active-passive, partitioned active-active, globally consistent active-active, and eventually consistent active-active.
- Health checks define correctness: check the dependency path, not just a port, and evaluate from multiple vantage points.
- Rehearse failover; the measured total always exceeds the configured one.
🧪 Practice
- Compute total DNS failover exposure for a 30-second TTL, checks every 5 seconds requiring 3 failures, and 90 seconds of client caching slop. Compare with anycast at 1 second withdraw plus 3 seconds reconvergence.
- For 500,000 clients, compute authoritative query load at TTLs of 5, 30, and 300 seconds, and recommend one with justification.
- Design the health check for a region serving reads from a replica, listing each condition that should mark it unhealthy and why a port check is insufficient.
- Interview-style: A service is deployed in three regions with latency-based DNS steering, and users in one country consistently reach the furthest region. (Hint: the steering decision is made from the address of something that is not the client — identify what, then consider where that component might physically sit relative to the user.)
<a id="135-network-automation"></a>
13.5 Network Automation
Every technology in this chapter multiplies the number of things a network engineer is responsible for, from a few dozen switches to thousands of virtual networks, policies, and endpoints that change by the minute: this subchapter covers the practices that make that manageable — describing networks as code, converging devices toward a declared state, collecting telemetry at the rate events actually happen, verifying outcomes rather than configurations, and shipping changes through a pipeline instead of a maintenance window.
Infrastructure as Code for Networks
Theory
The traditional way to configure a network was to log into a device and type. It has properties that are easy to overlook until you compare them with software practice: there is no record of why a line exists, no way to review a change before it happens, no way to reproduce a device's configuration from scratch with confidence, and no way to tell whether two supposedly identical devices actually are. The running configuration is the only source of truth, and it is editable by anyone with access.
Infrastructure as code inverts this. A file in version control describes the desired state, and a tool makes reality match it. What the network is, is what the repository says. The benefits are the benefits of software engineering applied to a domain that lacked them: history with authorship and reasons, review before rather than after, reproducibility, testing, and the ability to build identical environments.
The key conceptual distinction is imperative versus declarative.
An imperative description is a sequence of actions: create this VLAN, add that ACL line. It is easy to write and has a serious flaw — it depends on the starting state. Running it twice may create duplicates or fail; running it on a device that has drifted may produce something different again. There is no expression of what the end state should be, only of steps.
A declarative description states the desired end state and lets the tool compute the actions. This gives two properties that matter enormously in practice:
- Idempotence: applying the same description repeatedly converges and then does nothing. Re-running is safe, which means the description can be applied on a schedule as a correction mechanism.
- Drift detection: because the desired state is written down, the tool can compare it against reality at any time and report differences. Manual changes stop being invisible.
The mechanism that makes declarative tools work is the state file, and it is worth understanding because it is the source of most confusion. Given only a desired state and a device's actual state, a tool cannot tell whether an object that exists but is not in the desired state should be deleted (we created it, and it has been removed from the description) or left alone (someone else owns it). The state file records what this tool created and therefore owns. Without it, a declarative tool either deletes resources it never created or can never delete anything.
This leads to the workflow that makes network changes reviewable in a way the CLI never allowed: plan, then apply. The plan is a computed diff — what will be created, changed, and destroyed — produced without touching anything. It is an artefact you can read, attach to a change request, and reject. The single most valuable habit in this area is reviewing plans, because an unexpected line in a plan is a defect caught for free, and the same line discovered after apply is an incident.
The practical caution is that network devices are not cloud APIs. A cloud provider's API is transactional, versioned, and designed for this model. A switch may apply changes line by line with no rollback, may reorder ACL entries, and may need a commit. Tools bridge this with candidate configurations, commit-confirm, and config replacement rather than merge, but the impedance mismatch is real and is why network IaC is harder than cloud IaC.
Examples
Two ways to describe the same outcome
IMPERATIVE (a script): depends on where you started
vlan 30
name guest
interface Gi1/0/5
switchport access vlan 30
-> run twice: maybe duplicates, maybe an error
-> run on a drifted device: unknown result
-> nothing states what SHOULD be true afterwards
DECLARATIVE (a description): independent of where you started
vlans:
10: {name: users}
20: {name: voice}
30: {name: guest}
-> run twice: the second run changes nothing (IDEMPOTENT)
-> run on a drifted device: it converges back (SELF-HEALING)
-> the file IS the statement of what should be true
The workflow that follows:
[ git repo ]--- desired state
|
v
[ plan ] <--- reads ACTUAL state from devices
| and OWNED state from the state file
|
+--> a reviewable diff: + create ~ change - destroy
|
[ review / approve ] <-- the step the CLI never had
|
v
[ apply ] ---> devices, then the state file is updated
# Imperative vs declarative, and the role of STATE.
DESIRED = { # what the configuration file says
"vlan-10": {"name": "users", "mtu": 1500},
"vlan-20": {"name": "voice", "mtu": 1500},
"vlan-30": {"name": "guest", "mtu": 1500},
}
ACTUAL = { # what the device currently has
"vlan-10": {"name": "users", "mtu": 1500}, # matches
"vlan-20": {"name": "voice", "mtu": 9000}, # drifted
"vlan-99": {"name": "legacy", "mtu": 1500}, # not in the desired state
}
STATE = {"vlan-10", "vlan-20"} # what WE created, and therefore own
def plan(desired, actual, owned):
"""The whole idea: diff desired against actual, restricted to what we own."""
create = [k for k in desired if k not in actual]
update = [k for k in desired if k in actual and desired[k] != actual[k]]
destroy = [k for k in actual if k not in desired and k in owned]
ignore = [k for k in actual if k not in desired and k not in owned]
return create, update, destroy, ignore
create, update, destroy, ignore = plan(DESIRED, ACTUAL, STATE)
print("PLAN (nothing changed yet -- this is the reviewable artefact):")
for k in create:
print(f" + create {k} {DESIRED[k]}")
for k in update:
print(f" ~ update {k} {ACTUAL[k]} -> {DESIRED[k]}")
for k in destroy:
print(f" - destroy {k} {ACTUAL[k]}")
for k in ignore:
print(f" ignore {k} (exists but not in our state file: NOT ours to delete)")
print(f"\nsummary: {len(create)} to add, {len(update)} to change, "
f"{len(destroy)} to destroy, {len(ignore)} untracked")
# PLAN (nothing changed yet -- this is the reviewable artefact):
# + create vlan-30 {'name': 'guest', 'mtu': 1500}
# ~ update vlan-20 {'name': 'voice', 'mtu': 9000} -> {'name': 'voice', 'mtu': 1500}
# ignore vlan-99 (exists but not in our state file: NOT ours to delete)
#
# summary: 1 to add, 1 to change, 0 to destroy, 1 untracked
#
# The state file is what distinguishes "we should delete this" from "someone
# else owns this". Without it, a declarative tool either deletes resources it
# never created, or can never delete anything at all.
def apply_once(actual, desired):
changed = 0
for k, v in desired.items():
if actual.get(k) != v:
actual[k] = dict(v); changed += 1
return changed
work = {k: dict(v) for k, v in ACTUAL.items()}
print("\nIdempotence -- applying the same desired state repeatedly:")
for run in range(1, 4):
n = apply_once(work, DESIRED)
print(f" run {run}: {n} change(s) made"
f"{' <- converged; further runs are no-ops' if n == 0 else ''}")
# run 1: 2 change(s) made
# run 2: 0 change(s) made <- converged; further runs are no-ops
# run 3: 0 change(s) made <- converged; further runs are no-ops
# An imperative script appending the same VLAN would have done it three times.
print("\nDrift detection is the same diff, run on a schedule with no apply:")
work["vlan-30"]["mtu"] = 1400 # someone edited the device by hand
c, u, d, i = plan(DESIRED, work, STATE | {"vlan-30"})
print(f" drift found in: {u} -> desired {DESIRED['vlan-30']}, "
f"actual {work['vlan-30']}")
# drift found in: ['vlan-30'] -> desired {'name': 'guest', 'mtu': 1500}, actual {'name': 'guest', 'mtu': 1400}
# Report it, or auto-remediate. Either way the file remains the source of truth,
# and a manual change stops being invisible.
# Declarative network infrastructure: the file IS the statement of intent.
variable "vpc_cidr" { default = "10.20.0.0/16" }
variable "azs" { default = ["eu-west-1a", "eu-west-1b", "eu-west-1c"] }
resource "aws_vpc" "main" {
cidr_block = var.vpc_cidr
enable_dns_hostnames = true
tags = { Name = "prod", ManagedBy = "terraform" } # provenance, in the resource
}
# One block describes THREE subnets: the loop is what makes zones consistent,
# so a fourth zone is a one-line change rather than three copied blocks.
resource "aws_subnet" "private" {
for_each = { for i, az in var.azs : az => i }
vpc_id = aws_vpc.main.id
availability_zone = each.key
# Derive the CIDR rather than hardcoding: no arithmetic mistakes, no overlaps.
cidr_block = cidrsubnet(var.vpc_cidr, 6, each.value + 10)
tags = { Name = "private-${each.key}", Tier = "private" }
}
resource "aws_security_group" "db" {
name = "db"
vpc_id = aws_vpc.main.id
ingress {
from_port = 5432
to_port = 5432
protocol = "tcp"
security_groups = [aws_security_group.app.id] # reference the SG, not a CIDR:
} # it keeps working as the app scales
egress { # explicit: the default is "all"
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
# A guard rail expressed as code, so it is enforced rather than remembered.
check "no_public_db" {
assert {
condition = !contains(aws_security_group.db.ingress[*].cidr_blocks, ["0.0.0.0/0"])
error_message = "The database security group must never allow 0.0.0.0/0."
}
}
# --- The plan/apply workflow, which is the point of all of this ---
terraform init
terraform plan -out=tfplan # reads real state; changes NOTHING
terraform show -json tfplan | jq -r '
.resource_changes[] | select(.change.actions != ["no-op"]) |
"\(.change.actions[0])\t\(.address)"'
# update aws_subnet.private["eu-west-1b"]
# delete aws_security_group.legacy <- ALWAYS question a delete
terraform apply tfplan # applies exactly the reviewed plan, nothing else
# --- Drift detection on a schedule: run the plan and alert if it is non-empty ---
terraform plan -detailed-exitcode >/dev/null 2>&1
case $? in
0) echo "no drift" ;;
2) echo "DRIFT DETECTED"; terraform plan -no-color | head -40 ;;
1) echo "plan failed" ;;
esac
# --- Network devices: the same model, with device-specific safety ---
# Config REPLACE rather than merge, so the file is genuinely the source of truth.
napalm --user admin --password "$PW" --vendor eos 10.0.1.1 \
configure --replace configs/leaf-1.cfg --dry-run
# Prints the diff the device itself computes. Review it, then drop --dry-run.
# Commit-confirm: the network equivalent of a safety net. If you lose your
# session because of the change, the device reverts itself.
ssh admin@10.0.1.1 'configure; load replace leaf-1.conf; commit confirmed 5'
# You then have 5 minutes to run `commit` to keep it. Do nothing and it rolls
# back. This is the correct way to change anything on the management path.
Key takeaways
- CLI configuration leaves no record of intent, no review before change, and no way to prove two devices are identical. The running config is the only truth and anyone can edit it.
- Infrastructure as code makes a version-controlled file the source of truth, bringing history, review, reproducibility, and testing.
- Imperative descriptions are sequences of actions and depend on the starting state; declarative descriptions state the end state.
- Declarative gives idempotence (re-running converges and then does nothing) and drift detection (manual changes become visible).
- The state file records what the tool owns, which is what makes safe deletion possible at all.
- Plan then apply: the plan is a computed diff you can review and reject. Reviewing plans is the highest-value habit here, and an unexpected delete line deserves suspicion every time.
- Prefer config replace over merge so the file really is the source of truth.
- Network devices are not transactional cloud APIs; use candidate configs and commit-confirm, especially for anything touching the management path.
🧪 Practice
- Rewrite this imperative sequence declaratively and state what changes when it runs twice:
vlan 40,name iot,interface Gi1/0/9,switchport access vlan 40.- Given a desired state of three ACL rules, an actual state of four, and a state file listing two, determine which rules are created, changed, destroyed, and ignored.
- Explain what a tool cannot decide without a state file, and give a concrete example of the wrong outcome in each direction.
- Interview-style: An engineer runs an apply and a plan line they did not read deletes a production security group. (Hint: name the workflow step that existed specifically to prevent this, and the one device-level mechanism that would have limited the damage even after apply.)
Configuration Management Tools
Theory
Infrastructure as code describes what should exist. Configuration management is the machinery that makes a device match that description, and the tools in this space differ along a few axes worth knowing before choosing one.
Push versus pull. A push model has a central system connect to devices and make changes: simple to reason about, requires the central system to reach every device, and changes happen when you run it. A pull model has an agent on each device fetch its desired state periodically: scales better, works through firewalls and for intermittently-connected devices, and applies changes on the agent's schedule rather than yours. Network devices historically could not run agents, so network tooling is overwhelmingly push-based, while server tooling spans both.
Agentless versus agent-based. Agentless tools use SSH or an API and require nothing installed, which is why Ansible dominates network automation: a switch that speaks SSH or NETCONF is manageable with no software added. The cost is that every operation pays connection setup, and the tool must understand each platform's syntax.
Templating plus inventory is the pattern that does the real work, and it is worth internalising because it is what turns a configuration problem into a data problem. Configuration becomes a template (the structure, identical across devices) plus inventory data (the per-device values). One template and four hundred inventory entries replace four hundred hand-written configs. Consistency becomes automatic, because a structural change is made once, and the inventory becomes the thing you review — which is a much smaller and more meaningful artefact than four hundred config files.
Idempotence deserves restating in this context because it is a property of each individual operation, not just of the whole run. A well-written module checks the current value before writing, so "ensure VLAN 30 exists" is safe to run a thousand times. This is what allows configuration management to run continuously as a correction mechanism rather than only at change time. A module that blindly appends is not idempotent and breaks that guarantee, which is why writing raw CLI commands into a task is discouraged: command and raw modules cannot check anything.
A distinction that causes real confusion: configuration management converges the device's configuration; it does not verify the network's behaviour. A run can report success on every device while the network is broken, because the configuration is exactly as described and the description is wrong. That gap is what the next two topics exist to close.
Finally, secrets and provenance. Device credentials must not sit in the repository, so tooling integrates with vaults and secret stores. And every generated configuration should carry a marker saying it is generated, by what, from which commit — because the alternative is an engineer editing a generated file by hand, whose change is silently erased on the next run, which erodes trust in the automation faster than any outage.
| Tool | Model | Language | Typical network use |
|---|---|---|---|
| Ansible | push, agentless | YAML + Jinja2 | the default for network devices |
| Terraform | push, API-driven | HCL, declarative + state | cloud networks, VPCs, some devices |
| Nornir | push, Python | Python | when you need real programming, not YAML |
| Salt | push or pull, agent optional | YAML + Jinja2 | large fleets, event-driven reactions |
| Puppet / Chef | pull, agent | Ruby DSL | servers; rare on network devices |
| NAPALM | library, not a tool | Python | vendor-neutral config replace and getters |
Examples
Template plus inventory: 400 configs from one file
TEMPLATE (structure, written once) INVENTORY (data, per device)
+--------------------------------+ +---------------------------------+
| hostname {{ name }} | | leaf-1: {id: 1, asn: 65001, |
| router bgp {{ asn }} | + | loopback: 10.255.0.1} |
| router-id {{ loopback }} | | leaf-2: {id: 2, asn: 65002, |
| {% for u in uplinks %} | | loopback: 10.255.0.2} |
| interface {{ u.port }} | | ... |
| {% endfor %} | | leaf-400: {...} |
+--------------------------------+ +---------------------------------+
| |
+----------------+-------------------+
v
400 device configurations, consistent by construction
What this changes about review:
before: review 400 config files (nobody does this properly)
after : review 1 template + a diff of the inventory (feasible and meaningful)
What configuration management does NOT do:
[ device config matches the description ] <- CM reports SUCCESS here
|
v
[ is the network actually working? ] <- NOT answered
^ this is 13.5's intent verification, next topic
# Ansible: idempotent, agentless, template-driven network configuration.
- name: Configure leaf switches
hosts: leaves
gather_facts: false
connection: network_cli
vars:
ansible_network_os: arista.eos.eos
tasks:
# IDEMPOTENT: the module reads current state and writes only what differs.
- name: Ensure VLANs exist
arista.eos.eos_vlans:
config:
- {vlan_id: 10, name: users}
- {vlan_id: 20, name: voice}
- {vlan_id: 30, name: guest}
state: replaced # `replaced` makes the list authoritative:
# VLANs absent from it are REMOVED. `merged`
# would leave stray VLANs in place for ever.
- name: Ensure BGP is configured
arista.eos.eos_bgp_global:
config:
as_number: "{{ bgp_asn }}" # from inventory, per device
router_id: "{{ loopback_ip }}"
neighbor:
- {neighbor_address: "{{ item }}", peer_group: SPINES}
state: merged
loop: "{{ spine_ips }}"
# Render a full candidate config and let the DEVICE compute the diff.
- name: Render candidate configuration
ansible.builtin.template:
src: leaf.j2
dest: "build/{{ inventory_hostname }}.cfg"
delegate_to: localhost
changed_when: false
- name: Show what would change (check mode: no changes applied)
arista.eos.eos_config:
src: "build/{{ inventory_hostname }}.cfg"
replace: config # REPLACE, not merge: the file is the truth
diff_against: running
check_mode: true
register: cfg_diff
- name: Print the diff for review
ansible.builtin.debug:
var: cfg_diff.diff
when: cfg_diff.changed
# NOT idempotent, and therefore a last resort: `command` cannot check state.
# - name: Bad practice
# arista.eos.eos_command:
# commands: ["configure", "vlan 30", "name guest"]
The template, with the marker that protects it from hand editing
{# leaf.j2 #}
! ============================================================
! GENERATED FILE -- DO NOT EDIT
! source : git@example.com:net/config.git
! commit : {{ git_commit }}
! built : {{ build_time }}
! by : {{ pipeline_url }}
! Hand edits are erased on the next run. Change the template or inventory.
! ============================================================
hostname {{ inventory_hostname }}
!
router bgp {{ bgp_asn }}
router-id {{ loopback_ip }}
{% for spine in spine_ips %}
neighbor {{ spine }} peer group SPINES
{% endfor %}
!
{% for vlan in vlans %}
vlan {{ vlan.id }}
name {{ vlan.name }}
{% endfor %}
Why the header matters more than it looks: without it, an engineer fixes
something by hand at 3 a.m., the next run reverts it, and the team stops
trusting the automation. Provenance prevents that conversation.
# --- Always run check mode first: it is this tool's "plan" ---
ansible-playbook leaves.yml --check --diff --limit leaf-1
# --check : make no changes
# --diff : show exactly what would change
# --limit : one device before four hundred
# --- Verify idempotence, which is the property you depend on ---
ansible-playbook leaves.yml --limit leaf-1
ansible-playbook leaves.yml --limit leaf-1
# The SECOND run must report "changed=0". If it does not, a task is not
# idempotent, and continuous enforcement will flap the device for ever.
# --- Inventory is the reviewable artefact: keep it structured and diffable ---
ansible-inventory --graph
ansible-inventory --host leaf-1 | jq '{bgp_asn, loopback_ip, spine_ips}'
git diff inventory/ # this diff is the change; the 400 configs are a consequence
# --- Secrets never live in the repository ---
ansible-vault encrypt_string --stdin-name device_password
export ANSIBLE_NET_AUTH=$(vault kv get -field=password secret/net/admin)
# --- Config replace with a vendor-neutral library, when Ansible is too coarse ---
python3 - <<'PY'
from napalm import get_network_driver
driver = get_network_driver("eos")
with driver("10.0.1.1", "admin", "secret") as dev:
dev.load_replace_candidate(filename="build/leaf-1.cfg")
print(dev.compare_config()) # the DEVICE computes the diff
# dev.commit_config() # or dev.discard_config()
PY
# --- Scheduled enforcement: the payoff of idempotence ---
# A cron entry that runs check mode and alerts on any non-zero change count
# turns configuration management into continuous drift correction.
ansible-playbook leaves.yml --check 2>&1 | grep -E 'changed=[1-9]' && echo DRIFT
Key takeaways
- Push versus pull and agentless versus agent-based are the main axes; network tooling is overwhelmingly push and agentless, because devices could not run agents.
- The core pattern is template plus inventory: structure written once, data per device. It turns a configuration problem into a data problem.
- This also transforms review: instead of 400 config files, you review one template and a diff of the inventory.
- Idempotence is a property of each operation. Modules that check before writing allow continuous enforcement;
commandandrawmodules cannot check and are a last resort. - Prefer
replacedovermergedsemantics so the description is authoritative and stray objects are removed. - Configuration management converges configuration, not behaviour: every device can report success while the network is broken.
- Keep secrets in a vault, never in the repository.
- Put a generated-file marker with provenance in every rendered config; without it, hand edits get silently reverted and trust in the automation collapses.
🧪 Practice
- Write the Jinja2 template and inventory entries that generate interface descriptions and BGP neighbours for three leaves with different ASNs and uplink counts.
- Explain the difference between
mergedandreplacedstate, and give a scenario wheremergedleaves the network in an incorrect condition.- A playbook reports
changed=3on every run with no configuration edits between runs. Explain the likely cause and how you would confirm it.- Interview-style: An engineer fixes an outage by editing a switch config by hand, and the fix disappears an hour later. (Hint: something is running on a schedule, and something was missing from the file they edited — name both, and say what the file should have told them.)
Telemetry and Streaming Analytics
Theory
You cannot operate what you cannot see, and how you collect visibility determines which problems you are capable of detecting at all. This is not a tooling preference; it is a mathematical constraint.
The traditional approach is polling: a collector asks each device for each counter on an interval, classically via SNMP. It has three problems that compound at scale.
Staleness is bounded below by the interval. A five-minute poll means data can be five minutes old, which is an eternity during an incident. Load scales with the product of devices, objects, and frequency — and because SNMP requires a round trip per variable, the bottleneck is usually request rate at the collector rather than bandwidth. And most importantly, events shorter than the interval are invisible. This is the same sampling argument as Chapter 12's bufferbloat discussion: a microburst that fills a buffer in two milliseconds cannot be recovered from a data set sampled every five minutes, however long you retain it. Retention does not fix resolution.
Streaming telemetry inverts the relationship. The collector subscribes once, and the device pushes updates — either on a timer (sample mode) or when a value changes (on-change mode). The gains are structural: no per-request overhead, an efficient binary encoding (protobuf over gRPC in gNMI), sub-second granularity where it matters, and on-change semantics that report state transitions when they happen rather than when someone next asks.
Be precise about what streaming does and does not fix, because the marketing overstates it. On-change telemetry catches state transitions of any duration, so a four-second routing adjacency flap is reported even though a ten-second poll would miss it. But a two-millisecond buffer microburst is invisible to every sampling interval and to on-change counter updates alike. Catching that requires the device itself to record it: a high-watermark queue-depth counter maintained in the ASIC, or in-band telemetry stamping per-hop queue occupancy into packets as Section 13.1 described. The general rule is that the measurement must happen where and when the event happens; transport speed cannot compensate for resolution.
The data has three distinct shapes, and using the wrong one is the usual cause of an unusable observability stack:
| Shape | What it is | Good for | Bad for |
|---|---|---|---|
| Metrics | numbers over time, with labels | trends, alerting, dashboards | anything per-flow or per-request |
| Flow records | per-flow summaries (NetFlow, IPFIX, sFlow) | who talked to whom, volumes | sub-second timing |
| Logs and events | discrete occurrences with context | state changes, audit, causality | aggregate trends |
The most common and most expensive mistake is cardinality explosion: adding high-variety labels to metrics. Per-device and per-interface labels are fine. Per-VLAN multiplies the series count, per-queue multiplies again, and per-flow labels multiply it by a factor of thousands, turning a metrics system into a very poor database. Flow-level detail belongs in flow records, which are designed for it. Every label is a multiplier, and the arithmetic should be done before deployment rather than discovered afterwards.
Finally, the analytics half. Streaming telemetry at sub-second granularity produces far more data than humans can look at, so the value comes from processing it in flight: computing rates and derivatives, detecting anomalies against a baseline, correlating across devices to distinguish one failing link from a fabric-wide problem, and triggering automated responses. This is where telemetry connects to closed-loop automation — a stream that nothing reacts to is just an expensive archive.
Examples
Polling versus streaming, and the resolution wall
POLLING (pull) STREAMING (push)
collector --request--> device collector --subscribe once--> device
collector <--response-- device collector <--update-- device
collector --request--> device collector <--update-- device
(repeat per variable, (device pushes; no requests)
per device, per interval)
cost: requests x devices x objects cost: updates only
freshness: >= the interval freshness: sample interval, or on change
bottleneck: collector request rate bottleneck: ingest and storage
THE RESOLUTION WALL -- what sampling cannot recover
actual queue depth over 10 seconds:
| .:|:. .:|:. .:|:.
| .: | :. .: | :. .: | :.
+--------------------------------------------------------- time
^ 2 ms bursts, filling the buffer and causing drops
sampled every 10 s: * * *
^ each sample lands between bursts: queue looks EMPTY
sampled every 1 s: * * * * * * * * * * * * * * * * * * * *
^ still almost certainly misses a 2 ms event
What DOES catch it:
- an ASIC high-watermark counter (max depth since last read)
- in-band telemetry stamping per-hop queue depth into the packet (13.1)
The measurement must happen WHERE and WHEN the event happens.
# Polling vs streaming telemetry: freshness, load, and what each can detect.
DEVICES, INTERFACES_PER_DEVICE, COUNTERS_PER_INTERFACE = 400, 48, 12
total_series = DEVICES * INTERFACES_PER_DEVICE * COUNTERS_PER_INTERFACE
print(f"{DEVICES} devices x {INTERFACES_PER_DEVICE} interfaces x "
f"{COUNTERS_PER_INTERFACE} counters = {total_series:,} time series\n")
SNMP_BYTES_PER_VAR = 180 # request + response, per variable
GNMI_BYTES_PER_UPDATE = 60 # protobuf, no per-request overhead
print(f"{'approach':28s} {'interval':>9s} {'samples/s':>11s} {'bytes/s':>11s} "
f"{'worst staleness':>16s}")
for label, interval, per_sample in (
("SNMP poll every 300 s", 300, SNMP_BYTES_PER_VAR),
("SNMP poll every 60 s", 60, SNMP_BYTES_PER_VAR),
("SNMP poll every 10 s", 10, SNMP_BYTES_PER_VAR),
("gNMI stream, 10 s", 10, GNMI_BYTES_PER_UPDATE),
("gNMI stream, 1 s", 1, GNMI_BYTES_PER_UPDATE)):
sps = total_series / interval
print(f"{label:28s} {interval:8d}s {sps:11,.0f} {sps*per_sample:11,.0f} "
f"{interval:15d}s")
# approach interval samples/s bytes/s worst staleness
# SNMP poll every 300 s 300s 768 138,240 300s
# SNMP poll every 60 s 60s 3,840 691,200 60s
# SNMP poll every 10 s 10s 23,040 4,147,200 10s
# gNMI stream, 10 s 10s 23,040 1,382,400 10s
# gNMI stream, 1 s 1s 230,400 13,824,000 1s
# ^ same interval, 3x fewer bytes: protobuf and no request overhead
print("\nThe polling problem is not only bytes -- it is REQUESTS:")
for interval in (300, 60, 10):
rps = total_series / interval
print(f" poll every {interval:3d}s -> {rps:10,.0f} SNMP requests/s "
f"({'feasible' if rps < 20_000 else 'NOT feasible on one collector'})")
# poll every 300s -> 768 SNMP requests/s (feasible)
# poll every 60s -> 3,840 SNMP requests/s (feasible)
# poll every 10s -> 23,040 SNMP requests/s (NOT feasible on one collector)
print("\nWhat each approach can DETECT. A sampled method needs an event to last")
print("at least about two intervals to be seen reliably:")
print(f" {'event':32s} {'duration':>12s} {'300s':>7s} {'10s':>7s} {'1s':>7s} "
f"{'event-driven':>13s}")
for name, dur_s in (("microburst filling a buffer", 0.002),
("BGP session flap", 4),
("link down and back up", 12),
("queue building", 45),
("gradual memory leak", 86_400)):
row = ["yes" if dur_s >= 2 * interval else "MISSED" for interval in (300, 10, 1)]
ed = "yes" if dur_s >= 0.1 else "needs ASIC" # on-change catches transitions
print(f" {name:32s} {dur_s:11,.3f}s {row[0]:>7s} {row[1]:>7s} {row[2]:>7s} "
f"{ed:>13s}")
# event duration 300s 10s 1s event-driven
# microburst filling a buffer 0.002s MISSED MISSED MISSED needs ASIC
# BGP session flap 4.000s MISSED MISSED yes yes
# link down and back up 12.000s MISSED MISSED yes yes
# queue building 45.000s MISSED yes yes yes
# gradual memory leak 86,400.000s yes yes yes yes
#
# Read the first row honestly: a 2 ms microburst is invisible to EVERY sampling
# interval and to on-change counter updates too. Only a device-side high-water
# mark or in-band telemetry records it. Longer retention does not add resolution.
print("\nCardinality: labels, not devices, are what blow up a metrics system")
BYTES_PER_SAMPLE, SAMPLES_PER_DAY = 16, 86_400
for label_set, mult in (("no extra labels", 1),
("+ per-VLAN (x50)", 50),
("+ per-VLAN + per-queue (x8)", 50 * 8),
("+ per-flow 5-tuple (x10,000)", 50 * 8 * 10_000)):
n = total_series * mult
print(f" {label_set:30s} {n:>15,} series -> "
f"{n*BYTES_PER_SAMPLE*SAMPLES_PER_DAY/1e9:>13,.0f} GB/day at 1 Hz")
# no extra labels 230,400 series -> 319 GB/day at 1 Hz
# + per-VLAN (x50) 11,520,000 series -> 15,925 GB/day at 1 Hz
# + per-VLAN + per-queue (x8) 92,160,000 series -> 127,402 GB/day at 1 Hz
# + per-flow 5-tuple (x10,000) 921,600,000,000 series -> 1,274,019,840 GB/day at 1 Hz
#
# Every label is a MULTIPLIER. Do this arithmetic before deployment. Per-flow
# detail belongs in flow records, which are designed for exactly that shape.
# --- Subscribe once, receive continuously: the streaming model ---
gnmic -a leaf-1:57400 -u admin -p "$PW" --skip-verify subscribe \
--path "/interfaces/interface/state/counters" \
--stream-mode sample --sample-interval 10s --encoding proto
# {"source":"leaf-1","path":".../out-discards","value":41822,"timestamp":...}
# ON-CHANGE for state, which is where streaming genuinely beats polling:
gnmic -a leaf-1:57400 -u admin -p "$PW" --skip-verify subscribe \
--path "/network-instances/network-instance/protocols/protocol/bgp/neighbors/neighbor/state/session-state" \
--stream-mode on-change
# A 4-second adjacency flap is reported as it happens. No polling interval
# short enough to catch it would be affordable across 400 devices.
# --- The counter that survives the resolution wall: a high-water mark ---
# The ASIC records the maximum since last read, so a 2 ms peak is not lost.
ethtool -S swp1 | grep -iE 'watermark|max_queue|peak'
# tc_0_max_queue_depth_bytes: 1842176
# Read-and-clear semantics: each poll returns the PEAK since the last poll,
# not an instantaneous sample. This is how you see microbursts at 10 s polling.
# --- Flow records for the "who talked to whom" question metrics cannot answer ---
nfdump -R /var/flows -s ip/bytes -n 10
# Use flows for per-conversation detail; never put a 5-tuple in a metric label.
# --- Check your own cardinality BEFORE it becomes an incident ---
curl -s localhost:9090/api/v1/status/tsdb | jq '.data.seriesCountByMetricName[:5]'
curl -s localhost:9090/api/v1/query --data-urlencode 'query=count({__name__=~".+"})' \
| jq -r '.data.result[0].value[1]'
# Alert on total series growth. A new label on a busy metric can multiply this
# overnight, and the failure mode is the monitoring system dying during the
# incident it was meant to help you diagnose.
# --- Close the loop: a stream nothing reacts to is an expensive archive ---
gnmic -a leaf-1:57400 --skip-verify subscribe --path "/interfaces/interface/state/oper-status" \
--stream-mode on-change | while read -r line; do
echo "$line" | grep -q '"DOWN"' && ./remediate.sh "$line"
done
Key takeaways
- Polling bounds freshness by the interval, scales load with devices times objects times frequency, and — because SNMP needs a round trip per variable — usually hits a collector request-rate limit first.
- Events shorter than the sampling interval are invisible, and longer retention cannot recover resolution.
- Streaming telemetry subscribes once and lets the device push, with binary encoding, sub-second granularity, and on-change reporting of state transitions.
- On-change catches transitions of any duration, but a two-millisecond microburst is invisible to streaming too. Catching it requires an ASIC high-watermark counter or in-band telemetry.
- The measurement must happen where and when the event happens; transport speed cannot compensate for resolution.
- Use the right data shape: metrics for trends, flow records for per-conversation detail, logs and events for state changes and causality.
- Cardinality explosion is the classic failure: every label is a multiplier, and per-flow labels turn a metrics system into a bad database. Do the arithmetic first.
- Telemetry earns its cost only when something reacts to it; otherwise it is an expensive archive.
🧪 Practice
- For 250 devices with 32 interfaces and 10 counters each, compute time series, samples per second, and bytes per second at 60-second polling and at 5-second streaming with 60-byte updates.
- Explain why a 3-second link flap is invisible to 30-second polling but visible to on-change telemetry, and name one event invisible to both.
- A team wants per-source-IP byte counters as a metric label for 40,000 clients. Compute the series count, state the problem, and propose the correct data shape.
- Interview-style: An operator raises polling from 5 minutes to 30 seconds to catch packet loss, and the loss still appears without explanation. (Hint: relate the duration of a buffer-filling event to any feasible sampling interval, then name the kind of counter whose semantics survive that gap.)
Intent-Based Networking
Theory
Automation as described so far closes half a loop. Infrastructure as code says what configuration should exist; configuration management makes devices match it. Neither answers the question the business actually asks, which is not "does the configuration match the template?" but "is the network doing what we need?"
Those are genuinely different questions, and the gap between them is where outages live. A configuration can be exactly as specified and the network still wrong, because the specification was wrong, because a link failed and the remaining topology no longer satisfies a requirement, or because the interaction of several correct configurations produces an incorrect outcome. Configuration compliance cannot detect any of these, since the configuration is compliant in every case.
Intent-based networking closes the loop by making the outcome the thing that is declared and verified. It has four parts, and the fourth is what distinguishes it from ordinary automation:
- Capture intent in outcome terms: "the application tier can reach the database on 5432", "guest traffic can never reach the database", "every site has at least two paths", "latency between these tiers stays under 2 milliseconds".
- Translate intent into device configuration, as the northbound-to-southbound compilation of Section 13.1 described.
- Activate it across the fleet.
- Assure it continuously: observe the network's actual behaviour, verify that each intent still holds, and remediate or alert when one does not.
Step four is the whole point. Intent is not a configuration format; it is a standing assertion that is re-checked. A link failure does not change any configuration, so compliance checking stays green while the redundancy intent is now violated. An operator adding a temporary rule during an incident and forgetting to remove it leaves an isolation intent violated with no configuration error anywhere. Continuous verification catches both.
The verification techniques divide into two families, and both have their place:
- Formal verification builds a model of the forwarding behaviour from the collected configurations and topology, then proves properties about it — that a packet from A to B on port 5432 is permitted, that no forwarding loop exists, that no rule is unreachable. It is exhaustive over the modelled space and can prove a property holds for all packets, which testing cannot. Its limit is that it verifies the model, so a modelling gap is a verification gap.
- Active probing actually sends traffic and observes the result. It tests the real system including everything the model omits, and covers only what it probes.
Mature systems use both: formal verification pre-change (it can run against a proposed configuration before it is applied, which is enormously valuable) and probing post-change.
Two cautions are worth stating plainly. First, intent must be expressed in verifiable terms. "The network should be fast" is not an intent; "p99 latency between these two tiers stays under 2 milliseconds, measured every 30 seconds" is. If you cannot write the check, you have written an aspiration. Second, automatic remediation is a decision, not a default. A system that automatically reverts a change may be undoing the fix an engineer is applying during an incident. The usual progression is to alert first, build confidence in the verification, then automate remediation for narrow well-understood classes of violation.
Examples
Closing the loop: the fourth step is what makes it intent-based
+-------------------- 1. CAPTURE INTENT ---------------------+
| app -> db:5432 reachable |
| guest -> db never |
| dc1 has >= 2 paths |
| app -> db latency <= 2 ms |
+------------------------------+-----------------------------+
v
2. TRANSLATE to device configuration
v
3. ACTIVATE across the fleet
v
+------------------- 4. ASSURE, CONTINUOUSLY ----------------+
| observe ACTUAL behaviour, not configuration |
| verify every intent still holds |
| alert / remediate on violation |
+------------------------------+-----------------------------+
|
+--> back to 1 when intent changes
The two questions, and why only one of them is useful:
compliance : "does the config match the template?" -> GREEN
intent : "can guest still reach the database?" -> RED
^ a link failed, or someone added a temporary rule.
No configuration error exists in either case.
# Intent-based networking: the closed loop. Intent -> translate -> VERIFY -> remediate.
INTENTS = [
{"id": "I1", "kind": "reachable", "src": "app", "dst": "db", "port": 5432},
{"id": "I2", "kind": "isolated", "src": "guest", "dst": "db"},
{"id": "I3", "kind": "redundant", "site": "dc1", "min_paths": 2},
{"id": "I4", "kind": "max_latency", "src": "app", "dst": "db", "ms": 2.0},
]
# The network as OBSERVED. Verification reads this, not the intended config.
OBSERVED = {
"acl": {("app", "db", 5432): "permit", ("guest", "db", 5432): "permit"},
"paths": {"dc1": 2},
"latency": {("app", "db"): 1.4},
}
def verify(intent, obs):
if intent["kind"] == "reachable":
key = (intent["src"], intent["dst"], intent["port"])
return obs["acl"].get(key) == "permit", \
f"ACL for {key} is {obs['acl'].get(key, 'absent')}"
if intent["kind"] == "isolated":
leaks = [k for k in obs["acl"]
if k[0] == intent["src"] and k[1] == intent["dst"]
and obs["acl"][k] == "permit"]
return not leaks, (f"VIOLATED: {intent['src']} can reach {intent['dst']} "
f"on {[k[2] for k in leaks]}" if leaks else "no permit found")
if intent["kind"] == "redundant":
n = obs["paths"].get(intent["site"], 0)
return n >= intent["min_paths"], f"{n} path(s), need {intent['min_paths']}"
if intent["kind"] == "max_latency":
m = obs["latency"].get((intent["src"], intent["dst"]))
return (m is not None and m <= intent["ms"],
f"measured {m} ms, limit {intent['ms']} ms")
return False, "unknown intent kind"
def report(obs, label):
print(f"--- verification: {label} ---")
violations = 0
for i in INTENTS:
ok, why = verify(i, obs)
violations += 0 if ok else 1
print(f" {i['id']} {i['kind']:12s} {'PASS' if ok else 'FAIL'} {why}")
print(f" {violations} violation(s)\n")
return violations
report(OBSERVED, "as deployed")
# --- verification: as deployed ---
# I1 reachable PASS ACL for ('app', 'db', 5432) is permit
# I2 isolated FAIL VIOLATED: guest can reach db on [5432]
# I3 redundant PASS 2 path(s), need 2
# I4 max_latency PASS measured 1.4 ms, limit 2.0 ms
# 1 violation(s)
print("Remediating I2 by removing the offending permit...")
del OBSERVED["acl"][("guest", "db", 5432)]
report(OBSERVED, "after remediation")
# --- verification: after remediation ---
# I1 reachable PASS ACL for ('app', 'db', 5432) is permit
# I2 isolated PASS no permit found
# I3 redundant PASS 2 path(s), need 2
# I4 max_latency PASS measured 1.4 ms, limit 2.0 ms
# 0 violation(s)
# The real value is CONTINUOUS verification, because the network keeps changing.
print("Someone adds a 'temporary' rule during an incident, and a link fails:")
OBSERVED["acl"][("guest", "db", 5432)] = "permit"
OBSERVED["paths"]["dc1"] = 1
report(OBSERVED, "continuous check, 3 a.m.")
# --- verification: continuous check, 3 a.m. ---
# I1 reachable PASS ACL for ('app', 'db', 5432) is permit
# I2 isolated FAIL VIOLATED: guest can reach db on [5432]
# I3 redundant FAIL 1 path(s), need 2
# I4 max_latency PASS measured 1.4 ms, limit 2.0 ms
# 2 violation(s)
#
# Note what happened to I3: a LINK FAILED. No configuration changed, so every
# compliance check in the previous topic still reports success. Only an intent
# check that looks at observed behaviour catches it.
# Intent expressed as verifiable assertions, kept in version control beside the
# configuration it constrains. Each one must be mechanically checkable.
apiVersion: intent.example.com/v1
kind: NetworkIntent
metadata:
name: prod-three-tier
spec:
reachability:
- name: app-to-db
from: {tier: app}
to: {tier: db}
ports: [5432]
expect: permitted
verify: {method: formal, plus: active-probe, interval: 60s}
- name: guest-isolation
from: {tier: guest}
to: {tier: db}
ports: [any]
expect: denied # a DENIAL is as important to verify as access
verify: {method: formal, interval: 60s}
onViolation: {action: alert, severity: critical, page: true}
redundancy:
- name: dc1-paths
site: dc1
minDisjointPaths: 2 # holds only while the topology holds:
verify: {method: topology, interval: 30s} # config checks cannot see this
performance:
- name: app-db-latency
from: {tier: app}
to: {tier: db}
metric: p99_latency_ms
max: 2.0 # a NUMBER, not "should be fast"
verify: {method: active-probe, interval: 30s}
onViolation: {action: alert}
# Remediation is a deliberate choice, not a default.
remediation:
mode: alert-only # start here
# mode: auto # only after the verification has earned trust
exclusions:
- duringIncident: true # never auto-revert while an incident is open:
# you may be undoing the fix in progress
# --- Formal verification BEFORE applying: the highest-value check available ---
# Build a model from configs and topology, then prove properties about it.
batfish-client -c 'init-snapshot configs/proposed' \
-c 'answer reachability \
pathConstraints=(startLocation="app", endLocation="db") \
headers=(dstPorts="5432") actions="SUCCESS"'
# Answers for ALL packets matching the header space, not just probed ones.
batfish-client -c 'answer reachability \
pathConstraints=(startLocation="guest", endLocation="db") actions="SUCCESS"'
# ANY result here is a violation of the isolation intent. Run it on the PROPOSED
# configuration and the change never reaches production.
batfish-client -c 'answer detectLoops' -c 'answer unusedStructures'
# --- Cloud-native equivalent: the provider proves the path for you ---
aws ec2 start-network-insights-analysis --network-insights-path-id nip-0abc
aws ec2 describe-network-insights-analyses --query \
'NetworkInsightsAnalyses[0].[NetworkPathFound,Explanations[0].ExplanationCode]'
# [false, "ACL_RULE"] <- names the exact component blocking the path
# --- Active probing: test the real system, including what the model omits ---
for dscp_pair in "app db 5432 expect-open" "guest db 5432 expect-closed"; do
set -- $dscp_pair
timeout 3 nc -z "$1.prod.svc" "$3" && r=open || r=closed
echo "$1 -> $2:$3 = $r (expected ${4#expect-})"
done
# guest -> db:5432 = open (expected closed) <- the violation, observed
# --- Continuous: run the whole suite on a schedule and alert on any FAIL ---
# The output that matters is not "config matches" but "every intent still holds".
./verify-intents.sh --all --format json | jq -r '
.results[] | select(.status=="FAIL") | "\(.id)\t\(.reason)"'
Key takeaways
- Configuration compliance asks "does the config match the template?"; intent asks "is the network doing what we need?" — and a compliant configuration can produce an incorrect network.
- The loop is capture intent, translate, activate, assure, and the fourth step is what distinguishes intent-based networking from ordinary automation.
- Intent is a standing assertion that is re-checked, so it catches things no configuration check can: a link failure violating redundancy, or a forgotten temporary rule violating isolation.
- Formal verification models forwarding behaviour and proves properties over all packets, and can run against a proposed configuration before it is applied.
- Active probing tests the real system including what the model omits, but only what it probes. Mature practice uses both.
- Verify denials as carefully as permissions; isolation is an intent too.
- Express intent in verifiable terms with numbers; if you cannot write the check, you have written an aspiration.
- Automatic remediation is a decision, not a default — alert first, and never auto-revert during an open incident.
🧪 Practice
- Write four verifiable intents for a three-tier application, covering one reachability, one isolation, one redundancy, and one performance property, each with a check method and interval.
- Explain how a link failure can violate an intent while every configuration compliance check still passes.
- Compare formal verification and active probing on coverage, timing relative to a change, and the kind of defect each misses.
- Interview-style: A team's intent system automatically reverts non-compliant changes, and during an outage it repeatedly undoes the engineer's fix. (Hint: the verification is behaving as designed — the problem is in the remediation policy, so name the condition under which auto-revert must be suspended and who should decide.)
CI/CD for Network Changes
Theory
Networks are the last major piece of infrastructure still commonly changed by hand during a maintenance window, and the reason is understandable: a bad network change can remove the access you need in order to fix it. But the consequence is that network changes remain the leading cause of network outages, and the maintenance-window model actively makes this worse by batching many changes together, so when something breaks there are twenty candidate causes and the only option is to revert all of them.
Continuous integration and delivery applies a different theory: many small, automatically validated, independently revertible changes instead of few large manual ones. Each change is small enough that its blast radius is understood and its rollback is obvious.
The pipeline's structure follows one principle: each stage should catch a class of defect more cheaply than the next. Lint catches syntax in seconds; a dry run catches unintended diffs in minutes; a virtual lab catches convergence failures before any real device is touched; formal verification catches policy violations in the proposed configuration; a canary catches everything the model missed, on one device.
| Stage | Catches | Cost |
|---|---|---|
| Lint / schema | syntax, types, naming | seconds |
| Policy as code | forbidden patterns, missing required fields | seconds |
| Build candidate | template rendering errors | seconds |
| Dry run / diff | unintended changes, on every target | minutes |
| Virtual lab | does the topology converge at all | minutes |
| Intent verification | reachability and isolation violations | minutes |
| Canary apply | everything the models missed, on one device | minutes |
| Staged rollout | correlated failures at scale | hours |
Two elements of this are worth singling out.
The virtual lab is what makes network CI possible at all. Containerised network operating systems let a pipeline build the actual topology, converge the actual routing protocols, and run traffic tests — for every change, automatically. Before this existed, the only way to test a routing change was to apply it, which is why network changes were tested in production.
Staged rollout with automatic abort is where the real safety comes from. Applying to one device, then five percent, then twenty-five percent, with verification between stages and automatic halt on failure, converts a fleet-wide outage into a single-device incident. The arithmetic is worth internalising: a canary of one percent reduces the blast radius by a hundredfold compared with pushing to everything, and that ratio is the entire value proposition.
The hard constraint that has no software analogue is revertibility. Some network changes remove the path you would revert over — changing a management VLAN, renumbering an underlay, modifying the firewall rule that permits your own management access. For these, "just roll back" is not available, and the mechanisms that exist are specific:
- Commit-confirm: the device applies the change and automatically reverts unless you confirm within a timeout. If the change cuts you off, you do nothing and it undoes itself. This is the single most valuable network-specific safety feature.
- Out-of-band access: a console server or separate management path that the change cannot affect.
- Config replace over merge: makes the previous known-good configuration a single artefact that can be restored wholesale.
The cultural change matters as much as the tooling. Small changes, merged often, are safer than large ones batched into a window — which is counterintuitive to a discipline that grew up treating change as the enemy. The reasoning is that risk scales superlinearly with the number of simultaneous changes, because interactions between them are what defeat both testing and diagnosis.
Examples
The pipeline, and where each defect is cheapest to catch
developer commit
|
[ lint / schema ]------ fail --> seconds wasted
|
[ policy as code ]----- fail --> "no 0.0.0.0/0 on a db security group"
|
[ build candidate ]---- fail --> template error, no device touched
|
[ dry run / diff ]----- fail --> "this would delete 40 BGP neighbours"
|
[ virtual lab ]-------- fail --> topology does not converge
|
[ intent verify ]------ fail --> "guest could reach db"
|
[ CANARY: 1 device ]--- fail --> blast radius: 1 device, auto-revert
|
[ 5% ] -> [ 25% ] -> [ 100% ] verification between every stage
^ automatic abort at any step
Cost of catching the same defect at each stage:
lint 1 unit
dry run 10
virtual lab 40
canary 400
full rollout 50,000 <- an outage
^ 50,000x more expensive than the lint that would have caught it
# CI/CD for network changes: staged rollout and the blast radius it buys.
FLEET = 400
STAGES = [("canary", 1), ("stage-1", 5), ("stage-2", 10), ("stage-3", 25),
("stage-4", 59)]
print(f"Fleet: {FLEET} devices")
print(f"{'stage':10s} {'% of fleet':>11s} {'devices':>8s} {'cumulative':>11s} "
f"{'blast radius if it breaks here':>32s}")
cum = 0
for name, pct in STAGES:
n = round(FLEET * pct / 100)
cum += n
print(f"{name:10s} {pct:10d}% {n:8d} {cum:11d} "
f"{f'{n} devices ({100*n/FLEET:.2f}% of fleet)':>32s}")
# stage % of fleet devices cumulative blast radius if it breaks here
# canary 1% 4 4 4 devices (1.00% of fleet)
# stage-1 5% 20 24 20 devices (5.00% of fleet)
# stage-2 10% 40 64 40 devices (10.00% of fleet)
# stage-3 25% 100 164 100 devices (25.00% of fleet)
# stage-4 59% 236 400 236 devices (59.00% of fleet)
canary_n = round(FLEET * STAGES[0][1] / 100)
print(f"\n push to all at once -> blast radius {FLEET} devices (100%)")
print(f" staged -> blast radius {canary_n} devices "
f"({STAGES[0][1]}%) if the canary catches it")
print(f" reduction: {FLEET / max(1, canary_n):.0f}x")
# push to all at once -> blast radius 400 devices (100%)
# staged -> blast radius 4 devices (1%) if the canary catches it
# reduction: 100x
print("\nThe economics of catching a defect early:")
COST = {"lint": 1, "dry run": 10, "virtual lab": 40, "canary": 400,
"full rollout (outage)": 50_000}
for where, c in COST.items():
print(f" caught at {where:24s} costs ~{c:7,} units")
print(f" ratio, lint to outage: {COST['full rollout (outage)']/COST['lint']:,.0f}x")
# caught at lint costs ~ 1 units
# caught at dry run costs ~ 10 units
# caught at virtual lab costs ~ 40 units
# caught at canary costs ~ 400 units
# caught at full rollout (outage) costs ~ 50,000 units
# ratio, lint to outage: 50,000x
print("\nThe rule that makes any of this safe: EVERY change must be revertible.")
def can_revert(change):
return all(change.get(k) for k in ("has_rollback", "tested_rollback",
"no_data_migration", "reachable_after"))
for name, c in (
("add a BGP neighbour", dict(has_rollback=1, tested_rollback=1,
no_data_migration=1, reachable_after=1)),
("change management VLAN", dict(has_rollback=1, tested_rollback=1,
no_data_migration=1, reachable_after=0)),
("renumber the underlay", dict(has_rollback=1, tested_rollback=0,
no_data_migration=0, reachable_after=1))):
ok = can_revert(c)
note = "" if ok else (" <- you may lose the path you would revert OVER"
if not c["reachable_after"] else
" <- rollback is untested or not atomic")
print(f" {name:24s} safely revertible: {str(ok):5s}{note}")
# add a BGP neighbour safely revertible: True
# change management VLAN safely revertible: False <- you may lose the path you would revert OVER
# renumber the underlay safely revertible: False <- rollback is untested or not atomic
#
# The management-VLAN case is the classic: the change removes the very access
# needed to undo it. Such changes need commit-confirm (auto-revert unless
# confirmed) or genuine out-of-band access -- never a bare apply.
# A network change pipeline. Note the ordering: cheapest checks first, and no
# real device is touched until four stages have passed.
name: network-change
on: [pull_request, push]
jobs:
validate: # seconds: run on every push, block the PR
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Lint and schema
run: |
yamllint inventory/ templates/
python3 tools/validate_inventory.py --schema schema/inventory.json
- name: Policy as code
run: |
# Guard rails that are ENFORCED rather than remembered.
opa eval --data policy/ --input build/candidate.json \
'data.network.deny[_]' --fail-defined
# e.g. deny 0.0.0.0/0 ingress on a db group; require a change ticket
# on any BGP change; forbid removing more than 5 neighbours
build-and-diff: # minutes: what WOULD change, on every target
needs: validate
steps:
- name: Render candidate configs
run: ansible-playbook site.yml --check --diff > diff.txt
- name: Fail on suspicious diffs
run: |
# A large deletion count is almost always a template or inventory bug.
test "$(grep -c '^-' diff.txt)" -lt 20 || {
echo "Refusing: more than 20 deletions"; exit 1; }
- uses: actions/upload-artifact@v4
with: {name: diff, path: diff.txt} # attach it to the PR for review
virtual-lab: # minutes: does the topology actually converge?
needs: build-and-diff
steps:
- name: Build the topology in containers
run: containerlab deploy -t lab/fabric.clab.yml
- name: Wait for protocol convergence
run: ./lab/wait-for-bgp.sh --timeout 180
- name: Traffic and reachability tests
run: ./lab/test-reachability.sh
- name: Formal verification of the PROPOSED config
run: ./lab/batfish-verify.sh configs/candidate # before any device
canary: # minutes: one real device, real traffic
needs: virtual-lab
if: github.ref == 'refs/heads/main'
environment: production-canary # requires a human approval
steps:
- name: Apply with commit-confirm
run: |
ansible-playbook site.yml --limit canary-leaf \
--extra-vars "commit_confirm_minutes=5"
# If this change cuts us off, doing nothing reverts it in 5 minutes.
- name: Verify intents before confirming
run: ./verify-intents.sh --device canary-leaf --all
- name: Confirm the commit
run: ansible-playbook confirm.yml --limit canary-leaf
rollout: # hours: progressive, with automatic abort
needs: canary
strategy:
max-parallel: 1
matrix: {stage: [5pct, 10pct, 25pct, 100pct]}
steps:
- run: ansible-playbook site.yml --limit "${{ matrix.stage }}"
- run: ./verify-intents.sh --limit "${{ matrix.stage }}" --all
- name: Abort and revert on failure
if: failure()
run: ansible-playbook rollback.yml --limit "${{ matrix.stage }}"
# --- Build the real topology locally, before the pipeline does ---
containerlab deploy -t lab/fabric.clab.yml
containerlab inspect -t lab/fabric.clab.yml
docker exec -it clab-fabric-leaf1 vtysh -c 'show bgp summary'
containerlab destroy -t lab/fabric.clab.yml --cleanup
# This is what changed network engineering: the actual protocols, converging,
# in a pipeline, for every commit.
# --- Commit-confirm: the one network-specific safety net to internalise ---
# Junos
ssh admin@leaf-1 'configure; load replace leaf-1.conf; commit confirmed 5'
# Reverts automatically in 5 minutes unless you run `commit` again.
# Arista
ssh admin@leaf-1 'configure session chg1; rollback clean-config; \
copy flash:leaf-1.cfg session-config; configure session chg1 commit timer 00:05:00'
# IOS-XE
ssh admin@leaf-1 'configure terminal revert timer 5'
# Use this for ANY change to the management path. It costs nothing and it is
# the difference between a 5-minute self-healing blip and a site visit.
# --- Prove the rollback works BEFORE you need it ---
ansible-playbook rollback.yml --limit canary-leaf --check --diff
# An untested rollback is not a rollback. Exercise it in the virtual lab too.
# --- Measure whether the practice is working ---
git log --oneline --since='30 days ago' -- inventory/ | wc -l # change frequency
# Track alongside: change failure rate, mean time to restore, canary catch rate.
# Rising frequency with flat or falling failure rate is the signal that small
# batches are working. If failure rate rises with frequency, the pipeline's
# earlier stages are too weak, not the batch size too small.
Key takeaways
- Network changes remain a leading cause of outages, and maintenance windows make it worse by batching many changes so that a failure has many candidate causes and the only remedy is reverting all of them.
- CI/CD substitutes many small, automatically validated, independently revertible changes.
- Order pipeline stages so each catches a class of defect more cheaply than the next: lint, policy, build, dry run, virtual lab, intent verification, canary, staged rollout.
- The virtual lab (containerised network operating systems) is what makes network CI possible, letting real protocols converge in a pipeline for every commit.
- Staged rollout with automatic abort is where the safety comes from: a one percent canary cuts the blast radius a hundredfold.
- Catching a defect at lint rather than in production differs in cost by roughly four orders of magnitude.
- Revertibility is the hard constraint. Changes that remove the management path cannot simply be rolled back, so use commit-confirm, out-of-band access, and config replace.
- Commit-confirm is the single most valuable network-specific safety feature: do nothing and the change undoes itself.
- Track change frequency together with change failure rate and time to restore; rising frequency with stable failure rate means small batches are working.
🧪 Practice
- Design a five-stage pipeline for a 200-device fabric, giving for each stage what it validates, its expected duration, and the blast radius if a defect escapes it.
- Classify each as safely revertible or not, and name the mitigation: adding a VLAN, changing an OSPF area, modifying the management interface ACL, adding a BGP peer.
- Explain why a one percent canary on a 500-device fleet is worth the added rollout time, computing the blast radius in both cases.
- Interview-style: A team adopts CI/CD for network changes and their change failure rate goes up, not down. (Hint: smaller batches are not the problem — ask which pipeline stages exist, which classes of defect are therefore reaching production, and whether the virtual lab models the thing that broke.)
<a id="14-network-operations-analysis-and-troubleshooting"></a>