14. Network Operations, Analysis, and Troubleshooting

Every preceding chapter described how networks are supposed to work; this one is about the considerably more common situation in which something does not, and someone has to find out why before the next status update. The skills involved are less about knowing more protocols than about knowing which single measurement would distinguish two competing explanations, and then taking it. This chapter covers the diagnostic tools and what each one actually proves, how to capture and read packets, the methodologies that turn guessing into elimination, and the monitoring and capacity practice that lets you notice problems before a user reports them.

<a id="141-diagnostic-tools"></a>

14.1 Diagnostic Tools

A diagnostic tool is only useful if you know precisely which claim its output supports and which it does not: this subchapter covers the standard toolkit — reachability probes, socket inspection, name resolution, service checks, throughput measurement, and packet capture — with emphasis on the misreadings that each one invites.

ping, traceroute, and mtr

Theory

These three tools answer escalating questions about a path, and most of the trouble they cause comes from reading them as though they answered a different question than they do.

ping sends an ICMP echo request and waits for a reply. A successful ping proves something quite narrow but genuinely useful: a bidirectional path exists to that address, and the host's network stack is responding. It does not prove the application works, and — importantly — a failed ping proves almost nothing, because ICMP is routinely filtered by firewalls and deprioritised by routers. "Ping fails, therefore the host is down" is the single most common wrong conclusion in network troubleshooting.

What ping is genuinely excellent at is the distribution of round-trip times. Send a hundred probes and the spread tells you more than the average: the minimum is the path with empty queues, so it approximates propagation plus serialisation, and the difference between minimum and maximum is queueing variation. A path with minimum 20 ms and maximum 22 ms is a quiet path 20 ms away. A path with minimum 20 ms and maximum 400 ms is a congested path 20 ms away, and the average of the two cases can be identical.

traceroute discovers the intermediate hops by exploiting TTL. It sends probes with TTL 1, 2, 3, and so on; each router that decrements TTL to zero discards the packet and returns an ICMP time-exceeded message, which reveals its address. The mechanism explains all of traceroute's quirks. A hop showing * * * means that router does not generate ICMP, not that the path is broken — traffic passes through it perfectly. Each probe may take a different ECMP path, so consecutive hops can belong to different routes. And the reverse path is invisible, though it is often where the fault lies.

mtr runs traceroute continuously and accumulates per-hop statistics, which makes it far more useful than either — and also the tool most often misread. Two rules make its output trustworthy:

Rule one: loss at an intermediate hop matters only if it persists at every later hop. Routers rate-limit the ICMP they generate, so a hop can show 40% loss while forwarding transit traffic flawlessly. If hop 3 shows 40% loss and hops 4, 5, and 6 show zero, the loss is an artefact of hop 3's control-plane rate limiting. Real loss propagates: a packet actually dropped at hop 3 never reaches hop 6 either.

Rule two: read best, and read deltas. The avg column at an intermediate hop reflects how promptly that router chose to answer, which is unrelated to how quickly it forwards. Latency should increase monotonically along the path; where it jumps is where the distance or the congestion is. A lower average at a later hop does not mean the path got faster.

Examples

How traceroute actually works, and what each output means

  sender                    R1        R2        R3        target
    |-- TTL=1 ------------->X                                       R1 replies:
    |<-- ICMP time exceeded |                                       "hop 1 is me"
    |-- TTL=2 ----------------------->X                             R2 replies
    |<-- ICMP time exceeded -----------
    |-- TTL=3 ------------------------------->X                     R3: silent
    |                             (no reply: * * *)                 -> not broken,
    |-- TTL=4 ------------------------------------------>           just no ICMP
    |<-- ICMP port unreachable / echo reply ------------            destination!

  Three different outputs, three different meanings:
    10.0.0.1  1.2 ms     a router that answers
    * * *                a router that does NOT GENERATE ICMP (traffic is fine)
    !X or !A             administratively prohibited: a firewall said no

# Reading an mtr report correctly: the rule that prevents most false diagnoses.
HOPS = [
    # hop, host,            sent, loss%, best, avg, worst
    (1, "192.168.1.1",      100,  0.0,   0.4,  0.6,  1.2),
    (2, "10.20.0.1",        100,  0.0,   8.1,  9.0, 11.4),
    (3, "core1.isp.net",    100, 42.0,   9.2, 10.1, 14.0),   # scary, and harmless
    (4, "core2.isp.net",    100,  0.0,   9.4, 10.3, 15.1),
    (5, "peer.example.net", 100,  0.0,  22.7, 23.9, 29.8),
    (6, "target.example",   100,  3.0,  23.1, 41.6, 210.4),  # the real problem
]

def analyse(hops):
    findings = []
    for i, (n, host, sent, loss, best, avg, worst) in enumerate(hops):
        downstream = hops[i + 1:]
        # THE RULE: loss at hop N matters only if it persists at every later hop.
        persists = bool(downstream) and all(h[3] > 0 for h in downstream)
        if loss > 0 and not persists and downstream:
            findings.append((n, host, "IGNORE",
                             f"{loss:.0f}% loss here but 0% downstream -> "
                             f"ICMP rate limiting, not packet loss"))
        elif loss > 0:
            findings.append((n, host, "REAL LOSS",
                             f"{loss:.0f}% loss, and it persists to the destination"))
        if worst - best > 50:
            findings.append((n, host, "LATENCY", f"spread {worst-best:.0f} ms "
                             f"(best {best:.0f}, worst {worst:.0f}) -> queueing"))
    return findings

print(f"{'hop':>3s} {'host':22s} {'loss%':>6s} {'best':>7s} {'avg':>7s} {'worst':>7s}")
for n, host, sent, loss, best, avg, worst in HOPS:
    print(f"{n:3d} {host:22s} {loss:6.1f} {best:7.1f} {avg:7.1f} {worst:7.1f}")
print("\nAutomated reading:")
for n, host, kind, why in analyse(HOPS):
    print(f"  hop {n:2d} {host:22s} [{kind:9s}] {why}")
# hop host                    loss%    best     avg   worst
#   1 192.168.1.1               0.0     0.4     0.6     1.2
#   2 10.20.0.1                 0.0     8.1     9.0    11.4
#   3 core1.isp.net            42.0     9.2    10.1    14.0
#   4 core2.isp.net             0.0     9.4    10.3    15.1
#   5 peer.example.net          0.0    22.7    23.9    29.8
#   6 target.example            3.0    23.1    41.6   210.4
#
# Automated reading:
#   hop  3 core1.isp.net          [IGNORE   ] 42% loss here but 0% downstream -> ICMP rate limiting, not packet loss
#   hop  6 target.example         [REAL LOSS] 3% loss, and it persists to the destination
#   hop  6 target.example         [LATENCY  ] spread 187 ms (best 23, worst 210) -> queueing
#
# A router that rate-limits the ICMP it GENERATES still forwards transit traffic
# perfectly. Loss is real only when it appears at a hop AND at every hop after it.

print("\nSecond rule: latency must be MONOTONIC to be meaningful.")
prev = 0.0
for n, host, sent, loss, best, avg, worst in HOPS:
    delta = best - prev
    flag = "  <- where the distance is" if delta > 8 else ""
    print(f"  hop {n}: best {best:6.1f} ms, +{delta:5.1f} ms over hop {n-1}{flag}")
    prev = best
#   hop 1: best    0.4 ms, +  0.4 ms over hop 0
#   hop 2: best    8.1 ms, +  7.7 ms over hop 1
#   hop 3: best    9.2 ms, +  1.1 ms over hop 2
#   hop 4: best    9.4 ms, +  0.2 ms over hop 3
#   hop 5: best   22.7 ms, + 13.3 ms over hop 4  <- where the distance is
#   hop 6: best   23.1 ms, +  0.4 ms over hop 5
#
# The 13.3 ms jump at hop 5 is a long link, not a problem. The 187 ms spread at
# hop 6 is the problem, and it is at the DESTINATION, not in the provider core.

# --- ping: read the DISTRIBUTION, not the average ---
ping -c 100 -i 0.2 api.example.com
#  rtt min/avg/max/mdev = 19.8/24.1/412.6/38.2 ms
#       ^min: the quiet path     ^max: queueing    ^mdev: jitter
#  min 19.8 with max 412.6 is a 20 ms path with a congestion problem, and the
#  average of 24.1 conceals it entirely.

# Test the path MTU, which the earlier chapters showed is a frequent culprit.
ping -M do -s 1472 -c 2 api.example.com     # 1472 + 28 = 1500
#  ping: local error: message too long, mtu=1450   <- the real path MTU

# Do NOT conclude a host is down from a failed ping. Test the actual service.
nc -zv api.example.com 443 && echo "port open: ICMP was merely filtered"

# --- traceroute: choose a probe the path will not silently discard ---
traceroute -n -T -p 443 api.example.com   # TCP SYN to 443: follows the real path
traceroute -n -I api.example.com          # ICMP: often treated differently
traceroute -n -U api.example.com          # UDP high ports: the default, often filtered
#  Firewalls frequently permit 443 and drop everything else, so a TCP traceroute
#  to the service port is the one that reflects your application's path.

traceroute -n -A api.example.com          # annotate each hop with its AS number
#   5  62.115.120.1 [AS1299]   <- shows where your traffic leaves your provider

# --- mtr: the tool to leave running while you reproduce a fault ---
mtr -rwzbc 200 api.example.com > mtr-$(date +%H%M).txt
#  -r report mode  -w wide  -z show AS  -b show both IP and name  -c 200 probes
#  Apply the two rules to the output: is loss persistent, and is `best` monotonic?

# The measurement that most reports omit: the REVERSE path.
ssh api.example.com "mtr -rwzbc 200 $(curl -s ifconfig.me)"
#  Routing is frequently asymmetric, so a one-directional trace can miss the
#  fault completely. Always get both directions before escalating.

Key takeaways

• A successful ping proves a bidirectional path and a responding stack. A failed ping proves almost nothing, because ICMP is widely filtered and deprioritised.
• Read ping's distribution: the minimum approximates propagation plus serialisation, and the min-to-max spread is queueing.
• traceroute works by TTL expiry, so * * * means a router does not generate ICMP, not that the path is broken.
• Each traceroute probe may take a different ECMP path, and the reverse path is invisible even though the fault is often there.
• mtr rule one: intermediate-hop loss matters only if it persists at every later hop; otherwise it is ICMP rate limiting.
• mtr rule two: use best and per-hop deltas, not avg. Latency must be monotonic to be meaningful.
• Use a TCP traceroute to the service port so the probe follows the path your application actually takes.
• Always capture both directions before escalating to a provider.

🧪 Practice

1. An mtr report shows 60% loss at hop 4 and 0% at hops 5 through 9. State your conclusion and justify it in one sentence.
2. A ping reports min/avg/max = 8.2/9.1/9.8 ms to one host and 8.4/95.2/860.1 ms to another. Describe what is different about the second path and what you would measure next.
3. Explain why traceroute -T -p 443 and traceroute -U can produce different hop lists to the same destination, and which you would trust for an HTTPS problem.
4. Interview-style: A monitoring system alerts that a production server is down because ICMP stopped replying, but users report no problems. (Hint: consider what the check actually measures versus what the users depend on, and what property of ICMP makes it a poor availability signal.)

netstat, ss, and lsof

Theory

The tools above look outward along a path. These look inward at one host: which sockets exist, in what state, owned by which process. They answer a different class of question — not "can I reach it?" but "is this machine's own networking in the state I think it is?"

ss is the modern replacement for netstat and should be preferred: it reads kernel socket state directly rather than parsing /proc/net/tcp, so it is dramatically faster on a busy host and exposes far more detail, including per-socket congestion control state. lsof answers the complementary question of which process holds a given file descriptor or port, which matters when the answer is "something is already listening on 8080 and I do not know what".

The skill worth developing is reading socket states diagnostically, because each state is evidence about where a failure sits:

State	Meaning	What a pile-up of it tells you
LISTEN	accepting connections	the service is bound; check the address it bound to
SYN-SENT	we sent SYN, no reply	packets are not arriving, or replies are not returning
SYN-RECV	we replied, no final ACK	possible SYN flood, or asymmetric routing
ESTABLISHED	open and usable	normal, unless the count is near a limit
CLOSE-WAIT	peer closed, we have not	an application bug: it is not closing sockets
FIN-WAIT-2	we closed, peer has not	the peer is not closing
TIME_WAIT	waiting out 2×MSL after close	normal churn, unless it is exhausting ports

Two of those deserve emphasis because they are frequently confused. A pile of CLOSE-WAIT sockets is an application defect, not a network problem — the kernel has told the application the peer closed, and the application has not called close. No amount of network tuning fixes it. A pile of TIME_WAIT sockets, by contrast, is normal and is the side of the connection that closed first doing its job of absorbing stray duplicates.

TIME_WAIT matters when it becomes a limit. An outbound connection consumes an ephemeral port for the duration of TIME_WAIT, and the constraint applies per destination address and port. With roughly 28,000 ephemeral ports and a 60-second TIME_WAIT, a host can sustain only a few hundred new connections per second to a single destination — which is exactly the shape of a reverse proxy talking to one backend address. The fix in order of preference is connection reuse, then more destination addresses, then a wider port range, and only then kernel tuning.

The other limit worth knowing is the accept queue, because it fails silently. When connections arrive faster than the application calls accept(), they queue; when the queue fills, the kernel drops the SYN rather than rejecting it. The client sees no error, just a retry after one second, then three. The symptom is "occasional slow connections" with nothing in any application log, and the evidence is a kernel counter (ListenOverflows), which is why this fault survives so long undiagnosed.

Examples

# Socket states, and the two exhaustion limits that cause real outages.
EPHEMERAL_LOW, EPHEMERAL_HIGH = 32768, 60999
TIME_WAIT_SECONDS = 60          # 2 x MSL on Linux

ports = EPHEMERAL_HIGH - EPHEMERAL_LOW + 1
print(f"ephemeral range {EPHEMERAL_LOW}-{EPHEMERAL_HIGH} = {ports:,} ports")
print(f"TIME_WAIT holds a 4-tuple for {TIME_WAIT_SECONDS}s after close\n")

print("Maximum sustainable NEW connection rate to ONE destination ip:port:")
print(f"{'connections/s':>15s} {'TIME_WAIT sockets':>19s} {'verdict':>28s}")
for cps in (100, 400, 500, 1_000, 5_000):
    tw = cps * TIME_WAIT_SECONDS
    verdict = "fine" if tw < ports * 0.5 else (
              "near exhaustion" if tw < ports else "PORT EXHAUSTION")
    print(f"{cps:15,} {tw:19,} {verdict:>28s}")
print(f"\nhard ceiling = {ports:,} ports / {TIME_WAIT_SECONDS}s = "
      f"{ports/TIME_WAIT_SECONDS:,.0f} connections/s to a single destination")
#   connections/s   TIME_WAIT sockets                      verdict
#             100               6,000                         fine
#             400              24,000              near exhaustion
#             500              30,000              PORT EXHAUSTION
#           1,000              60,000              PORT EXHAUSTION
#           5,000             300,000              PORT EXHAUSTION
#
# hard ceiling = 28,232 ports / 60s = 471 connections/s to a single destination
#
# Note the SCOPE: the limit is per (src ip, dst ip, dst port), so it bites a
# proxy talking to ONE backend address and never appears in a load test that
# spreads across many destinations.

print("\nThe four fixes, in order of preference:")
for fix, effect in (
    ("connection reuse / keep-alive", "removes the churn entirely: best fix by far"),
    ("more destination addresses", "multiplies the tuple space"),
    ("widen the ephemeral range", f"1024-65535 gives {65535-1024+1:,} ports"),
    ("tcp_tw_reuse", "reuse TIME_WAIT for outbound; NOT the removed tcp_tw_recycle"),
):
    print(f"  {fix:32s} {effect}")

print("\nThe other exhaustion limit: the ACCEPT QUEUE, which drops silently.")
print(f"{'backlog':>8s} {'accept rate/s':>14s} {'arrival rate/s':>15s} {'outcome':>32s}")
for backlog, accept_rate, arrival in ((128, 2_000, 1_500), (128, 2_000, 5_000),
                                      (4096, 2_000, 5_000), (4096, 8_000, 5_000)):
    if arrival <= accept_rate:
        out = "queue drains: healthy"
    else:
        out = f"queue fills in {backlog/(arrival-accept_rate)*1000:.0f} ms, SYNs dropped"
    print(f"{backlog:8d} {accept_rate:14,} {arrival:15,} {out:>32s}")
#  backlog  accept rate/s  arrival rate/s                          outcome
#      128          2,000           1,500            queue drains: healthy
#      128          2,000           5,000  queue fills in 43 ms, SYNs dropped
#     4096          2,000           5,000 queue fills in 1365 ms, SYNs dropped
#     4096          8,000           5,000            queue drains: healthy
#
# Read row three carefully: a bigger backlog only BUYS TIME. If the application
# cannot accept as fast as connections arrive, the queue fills regardless of
# size. The backlog absorbs bursts; it does not fix a throughput deficit.

# --- Socket inventory: what is listening, and on which address ---
ss -tlnp
#  State  Recv-Q Send-Q Local Address:Port  Process
#  LISTEN 0      4096   127.0.0.1:8080      users:(("app",pid=812,fd=6))
#                       ^^^^^^^^^ bound to LOOPBACK only: unreachable from
#                       anywhere else. This one line resolves a great many
#                       "the service is running but I cannot connect" tickets.

# Recv-Q on a LISTEN socket is the CURRENT accept-queue depth; Send-Q is its
# maximum. A persistently non-zero Recv-Q means the app is not accepting fast enough.

# --- State distribution: the shape of the problem, in one command ---
ss -tan | awk 'NR>1 {c[$1]++} END {for (s in c) printf "%-12s %6d\n", s, c[s]}' \
  | sort -k2 -rn
#  ESTAB        1842
#  TIME-WAIT   24011     <- approaching the ephemeral-port ceiling
#  CLOSE-WAIT    937     <- APPLICATION BUG: it is not closing sockets
#  SYN-SENT       64     <- outbound connections getting no reply

# --- The silent failure: accept-queue overflow ---
nstat -az | grep -E 'ListenOverflows|ListenDrops'
#  TcpExtListenOverflows  18422    <- every one of these is a dropped SYN and a
#  TcpExtListenDrops      18422       client that waited 1s, then 3s, for a retry
# Nothing in the application log will mention it.

# --- Per-socket TCP internals: the most underused diagnostic on Linux ---
ss -tin dst 10.0.2.15
#  cubic wscale:7,7 rto:204 rtt:3.4/0.9 mss:1448 cwnd:10 ssthresh:7
#  bytes_sent:92847 retrans:0/14 rcv_space:14480 delivery_rate:41.2Mbps
#      ^rtt and its variance   ^cwnd: is the window the limit?
#      ^retrans: 14 total retransmits    ^what the kernel measured, per socket
# cwnd stuck near ssthresh with rising retrans means loss is the constraint,
# not bandwidth -- the same distinction the throughput topic formalises.

# --- Which process holds a port? The question `ss -p` sometimes cannot answer ---
sudo lsof -i :8080 -sTCP:LISTEN
sudo lsof -p 812 -a -i            # every socket this one process holds
sudo lsof -i @10.0.2.15           # every socket talking to that peer
sudo ss -tanp | grep ':8080'      # equivalent, usually faster

# --- Ephemeral range and TIME_WAIT settings, for the arithmetic above ---
sysctl net.ipv4.ip_local_port_range net.ipv4.tcp_tw_reuse
cat /proc/sys/net/core/somaxconn   # ceiling on the backlog an app may request

Key takeaways

• Prefer ss over netstat: it reads kernel state directly and exposes per-socket congestion detail. Use lsof to find which process holds a port.
• Socket states are evidence. Read the distribution, not individual sockets.
• A pile of CLOSE-WAIT is an application bug — it is not calling close. No network tuning fixes it.
• A pile of TIME_WAIT is normal, and becomes a limit only when it exhausts ephemeral ports.
• The port-exhaustion ceiling is roughly ports divided by TIME_WAIT duration, per destination address and port — a few hundred connections per second to a single backend.
• Fix it with connection reuse first, then more destination addresses, then a wider range, then kernel tuning.
• Check that a service is bound to the right address: 127.0.0.1:8080 is unreachable from anywhere else.
• A full accept queue drops SYNs silently; the only evidence is ListenOverflows. A bigger backlog buys time but does not fix an accept-rate deficit.
• ss -ti exposes per-socket RTT, congestion window, and retransmits, which distinguishes a loss problem from a bandwidth problem.

🧪 Practice

1. Given 3,200 sockets in CLOSE-WAIT on a web server, state whether this is a network or application problem and give the reasoning.
2. A proxy opens new connections to one backend at 800 per second with a 60-second TIME_WAIT and a 28,000-port range. Determine whether it will exhaust ports, and give the two best fixes.
3. Write the commands that show the accept-queue depth of a listening socket and the count of overflow events, and explain what a non-zero overflow count means for the client.
4. Interview-style: A service works when tested from the host itself and is unreachable from every other machine, and the process is definitely running and listening. (Hint: one field in ss -tlnp output answers this immediately — think about what a socket bound to a specific address can and cannot receive.)

dig, nslookup, and host

Theory

Name resolution fails more often than almost anything else, and it is diagnosed badly more often than almost anything else, for one structural reason: resolution is a chain of several independent components, and most tools test only part of it. Saying "DNS is broken" is not a diagnosis; identifying which link in the chain is broken is.

The chain, from the application outward:

1. The application calls getaddrinfo(), which consults the system's name-service configuration — /etc/hosts, nsswitch.conf, possibly a local caching resolver such as systemd-resolved.
2. The stub resolver reads /etc/resolv.conf for nameservers, search domains, and options.
3. The recursive resolver (your ISP's, or a public one) does the actual work, and caches the answer. This is where stale answers live.
4. Root, then TLD, then authoritative servers are consulted by the recursive resolver, and the authoritative server is the only source of truth.

The essential discipline is knowing which link each tool tests:

Command	Tests	Notably
getent hosts NAME	the application's path	includes /etc/hosts and nsswitch
dig NAME	stub to recursive resolver	does not read /etc/hosts
dig @8.8.8.8 NAME	one specific recursive resolver	bypasses yours
dig +trace NAME	root, TLD, authoritative	bypasses all caches
dig @ns1.example.com NAME	the authoritative server	the source of truth
dig +norecurse NAME	what a resolver has cached	shows the remaining TTL

That table resolves most incidents. If dig works and the application does not, the difference is in /etc/hosts or nsswitch — and this is why dig not reading /etc/hosts is the most important footnote in DNS troubleshooting. If dig @authoritative gives the right answer and dig gives a wrong one, you have a caching problem and the remaining TTL tells you how long it lasts. Prefer dig over nslookup and host: it shows the full response including flags, TTLs, and the authority section, which is exactly the information the diagnosis needs.

Two specific traps are worth knowing.

TTL and migrations. After changing a record, resolvers keep serving the old value for up to its previous TTL. The consequence is that the TTL must be lowered before a cutover, typically a day ahead; lowering it during the cutover accomplishes nothing, because resolvers already hold the old record with its old long TTL.

The ndots trap, which produces mysterious latency in container environments. A stub resolver appends search domains to names with fewer than ndots dots. Kubernetes sets ndots:5 with three search domains, so resolving a short name can take four queries instead of one. At 20 ms per query that is 60 milliseconds of pure overhead on every cold lookup, and it shows up as inexplicable application latency. Fully qualifying hot names with a trailing dot removes it.

Examples

# The resolution chain, and which link in it a given tool actually tests.
CHAIN = [
    ("application",      "getaddrinfo()", "honours /etc/hosts, nsswitch, systemd-resolved"),
    ("stub resolver",    "/etc/resolv.conf", "search domains, ndots, timeout, rotate"),
    ("recursive resolver", "your ISP or 8.8.8.8", "CACHES: this is where stale answers live"),
    ("root servers",     ".", "referral to the TLD"),
    ("TLD servers",      "com.", "referral to the zone"),
    ("authoritative",    "ns1.example.com", "the ONLY source of truth"),
]
print(f"{'stage':20s} {'what it is':22s} what can go wrong there")
for stage, what, risk in CHAIN:
    print(f"{stage:20s} {what:22s} {risk}")

print("\nWhich tool tests which link:")
for cmd, tests, note in (
    ("getent hosts NAME",      "application path", "includes /etc/hosts and nsswitch"),
    ("dig NAME",               "stub -> recursive", "does NOT read /etc/hosts"),
    ("dig @8.8.8.8 NAME",      "one specific recursive", "bypasses your resolver"),
    ("dig +trace NAME",        "root -> TLD -> authoritative", "bypasses ALL caches"),
    ("dig @ns1.example.com NAME", "authoritative only", "the source of truth"),
    ("dig +norecurse NAME",    "cache contents only", "is it cached, and with what TTL"),
):
    print(f"  {cmd:28s} tests {tests:30s} {note}")

print("\nThe classic diagnosis: compare the authoritative answer with the cached one.")
for name, s in {
        "propagation delay": dict(auth="203.0.113.50", cache="198.51.100.10", ttl=1800),
        "healthy":           dict(auth="203.0.113.50", cache="203.0.113.50", ttl=240),
        "split-horizon":     dict(auth="203.0.113.50", cache="10.4.2.17", ttl=60),
        }.items():
    if s["auth"] == s["cache"]:
        verdict = "consistent: DNS is not your problem"
    elif s["ttl"] > 600:
        verdict = f"stale cache, up to {s['ttl']/60:.0f} min remaining -> wait or flush"
    else:
        verdict = "differs with a short TTL -> split view, or a very recent change"
    print(f"  {name:20s} auth {s['auth']:14s} cached {s['cache']:14s} "
          f"TTL {s['ttl']:5d}s -> {verdict}")
#   propagation delay    auth 203.0.113.50   cached 198.51.100.10  TTL  1800s -> stale cache, up to 30 min remaining -> wait or flush
#   healthy              auth 203.0.113.50   cached 203.0.113.50   TTL   240s -> consistent: DNS is not your problem
#   split-horizon        auth 203.0.113.50   cached 10.4.2.17      TTL    60s -> differs with a short TTL -> split view, or a very recent change

print("\nWhy a low TTL matters BEFORE a migration, not during:")
for ttl in (86400, 3600, 300, 60):
    print(f"  TTL {ttl:6,}s -> stale answers possible for {ttl/3600:6.2f} hours "
          f"after the change")
#   TTL 86,400s -> stale answers possible for  24.00 hours after the change
#   TTL  3,600s -> stale answers possible for   1.00 hours after the change
#   TTL    300s -> stale answers possible for   0.08 hours after the change
#   TTL     60s -> stale answers possible for   0.02 hours after the change
# Lowering the TTL during a cutover does NOTHING: resolvers already hold the old
# record with its old long TTL. Lower it a day ahead.

print("\nThe ndots trap, which produces mysterious latency in containers:")
SEARCH = ["prod.svc.cluster.local", "svc.cluster.local", "cluster.local"]
for name, dots in (("api", 0), ("api.prod", 1), ("example.com", 1),
                   ("api.prod.svc.cluster.local.", 5)):
    if name.endswith("."):
        tries, detail = 1, "fully qualified (trailing dot): ONE query"
    elif dots < 5:                      # ndots:5 is the Kubernetes default
        tries = len(SEARCH) + 1
        detail = f"{len(SEARCH)} search-domain attempts, then the name as given"
    else:
        tries, detail = 1, "enough dots: tried as absolute first"
    print(f"  {name:30s} {dots} dot(s) -> {tries} query(ies)  {detail}")
#   api                            0 dot(s) -> 4 query(ies)  3 search-domain attempts, then the name as given
#   api.prod                       1 dot(s) -> 4 query(ies)  3 search-domain attempts, then the name as given
#   example.com                    1 dot(s) -> 4 query(ies)  3 search-domain attempts, then the name as given
#   api.prod.svc.cluster.local.    5 dot(s) -> 1 query(ies)  fully qualified (trailing dot): ONE query
#
# Note the third row: `example.com` has 1 dot, which is fewer than ndots:5, so
# an EXTERNAL name also pays three failed cluster lookups first. At 20 ms per
# query that is 60 ms of pure overhead on every cold external lookup.

# --- Walk the chain from the outside in. Stop at the first disagreement. ---

# 1. What does the APPLICATION see? (the only answer that matters to users)
getent hosts api.example.com
#  If this differs from dig, the cause is /etc/hosts or nsswitch -- not DNS.

# 2. What does your configured resolver say, and with what TTL left?
dig api.example.com +noall +answer
#  api.example.com.  113  IN  A  198.51.100.10
#                    ^^^ 113s left on a cached record

# 3. What does the AUTHORITATIVE server say? (the source of truth)
AUTH=$(dig +short NS example.com | head -1)
dig @"$AUTH" api.example.com +noall +answer
#  api.example.com.  300  IN  A  203.0.113.50
#  Differs from step 2 -> a stale cache, and step 2's TTL says for how long.

# 4. Bypass every cache and watch the full delegation
dig +trace api.example.com | grep -E 'IN\s+(NS|A|AAAA)' | tail -8

# 5. Is it cached here, without causing it to be fetched?
dig +norecurse @127.0.0.53 api.example.com
#  status: NOERROR with an answer -> cached.  REFUSED/no answer -> not cached.

# --- Distinguish the three failures that all look like "DNS is broken" ---
dig api.example.com | grep -E 'status:|flags:'
#  status: NOERROR   + no ANSWER section -> the name exists, that TYPE does not
#  status: NXDOMAIN                      -> the name does not exist
#  status: SERVFAIL                      -> resolver or DNSSEC failure, RETRY elsewhere
dig +cd api.example.com     # +cd disables DNSSEC validation: if this fixes a
                            # SERVFAIL, the problem is DNSSEC, not reachability

# --- Measure resolution cost, including the ndots penalty ---
dig api.example.com | grep 'Query time'
time getent hosts api          # short name in a container: watch for ~4x the cost
grep -E 'ndots|search|nameserver' /etc/resolv.conf

# --- Which resolver is even in use? Often not the one you assume ---
resolvectl status | grep -A3 'DNS Servers'
cat /etc/resolv.conf     # may just point at 127.0.0.53 (systemd-resolved)

Key takeaways

• Resolution is a chain; "DNS is broken" is not a diagnosis. Identify the failing link.
• Know which link each tool tests. Crucially, dig does not read /etc/hosts, so getent hosts is what reflects the application's view.
• Prefer dig over nslookup and host because it shows flags, TTLs, and the authority section.
• Compare the authoritative answer with the cached one; if they differ, the remaining TTL tells you how long the problem lasts.
• Distinguish NXDOMAIN (no such name), NOERROR with no answer (no such record type), and SERVFAIL (resolver or DNSSEC failure). Test +cd to implicate DNSSEC.
• Lower TTLs a day before a migration; lowering them during the cutover has no effect on already-cached records.
• The ndots trap can turn one lookup into four in containers; fully qualify hot names with a trailing dot.
• Check which resolver is actually in use — often a local stub, not what /etc/resolv.conf appears to say.

🧪 Practice

1. dig api.example.com returns the correct address but the application connects to an old one. Name the two most likely causes and the single command that distinguishes them.
2. Explain the difference between NXDOMAIN, NOERROR with an empty answer, and SERVFAIL, giving a plausible cause for each.
3. A container resolves external names 60 ms slower than the host does. Explain the mechanism and give two fixes.
4. Interview-style: A team lowers a record's TTL from 24 hours to 60 seconds and cuts over immediately, and some users still reach the old server the next day. (Hint: ask when resolvers learned the TTL they are currently honouring, and what value it had at that moment.)

curl and telnet for Service Checks

Theory

Reachability tools prove packets arrive. Service checks prove the application answers, and the gap between those two is where a great many incidents live: the host pings, the port is open, and the service returns 503.

telnet host port and its better replacement nc -zv host port test one narrow thing: can a TCP connection be established? That is genuinely useful as a first step, because it partitions the problem cleanly. If the connection fails, the problem is at or below the transport layer — routing, firewall, or nothing listening. If it succeeds, the network has done its job and the problem is in the application or in TLS. Knowing which half of the problem you are in is worth the two seconds it takes.

curl is the more valuable tool because it exercises the whole request and, with -w, reports the timing of each phase. This turns "the site is slow" from an opinion into a measurement that names a responsible component.

The key thing to understand about curl's timing output is that the values are cumulative timestamps, not durations — this is the most common misreading. Subtracting adjacent values gives the per-phase durations, and each phase points somewhere specific:

Phase	Computed as	Dominant phase implies
DNS	time_namelookup	resolver problem or cold cache
TCP connect	time_connect - time_namelookup	one RTT: distance, or a full accept queue
TLS handshake	time_appconnect - time_connect	one to two RTTs plus crypto: distance, no resumption
Server think time	time_starttransfer - time_appconnect	the application, not the network
Transfer	time_total - time_starttransfer	bandwidth, loss, or a slow producer

That decomposition is the single most useful diagnostic in web operations, because it assigns the problem to a team. A dominant server phase is not a network incident, however much it feels like one.

The second thing worth internalising is the round-trip structure of a cold HTTPS request. DNS costs a round trip on a cache miss, the TCP handshake costs one, TLS 1.3 costs one (TLS 1.2 costs two), and the request itself costs one. That is four round trips before a single byte of content arrives — on a 70 ms path, 280 milliseconds. A warm connection with cached DNS costs one. This is why connection reuse and session resumption matter more than bandwidth for interactive workloads, and it is why the same measurement taken twice in a row can differ by a factor of four for reasons that have nothing to do with a fault.

Examples

# Decomposing an HTTP request into phases, so "the site is slow" becomes actionable.
SAMPLES = {   # curl -w output, seconds, for four different real problems
    "healthy":       dict(dns=0.004, tcp=0.021, tls=0.048, ttfb=0.071, total=0.094),
    "slow DNS":      dict(dns=2.503, tcp=2.520, tls=2.547, ttfb=2.570, total=2.593),
    "far away":      dict(dns=0.005, tcp=0.148, tls=0.437, ttfb=0.584, total=0.610),
    "slow backend":  dict(dns=0.004, tcp=0.021, tls=0.048, ttfb=1.904, total=1.928),
    "slow transfer": dict(dns=0.004, tcp=0.021, tls=0.048, ttfb=0.071, total=3.412),
}

def phases(s):
    """curl reports CUMULATIVE timestamps. Subtracting adjacent values gives the
    per-phase durations. Reading them as durations is the usual mistake."""
    return dict(dns=s["dns"], connect=s["tcp"] - s["dns"],
                tlsh=s["tls"] - s["tcp"], server=s["ttfb"] - s["tls"],
                transfer=s["total"] - s["ttfb"])

print(f"{'case':14s} {'DNS':>8s} {'connect':>8s} {'TLS':>8s} {'server':>8s} "
      f"{'transfer':>9s} {'total':>8s}  dominant phase")
for name, s in SAMPLES.items():
    p = phases(s)
    print(f"{name:14s} " + " ".join(f"{p[k]*1000:7.0f}m" for k in
          ("dns", "connect", "tlsh", "server", "transfer")) +
          f" {s['total']*1000:7.0f}m  {max(p, key=p.get)}")
# case                DNS  connect      TLS   server  transfer    total  dominant phase
# healthy              4m      17m      27m      23m      23m      94m  tlsh
# slow DNS          2503m      17m      27m      23m      23m    2593m  dns
# far away             5m     143m     289m     147m      26m     610m  tlsh
# slow backend         4m      17m      27m    1856m      24m    1928m  server
# slow transfer        4m      17m      27m      23m    3341m    3412m  transfer
#
# Note the "far away" row: every RTT-bound phase scales together, which is the
# signature of distance rather than of any single broken component.

print("\nWhat each dominant phase points at, and who should fix it:")
for phase, meaning in (
    ("dns",      "resolver, or a cold cache -> DNS / resolver config"),
    ("connect",  "1 RTT: distance, or a full accept queue -> network / capacity"),
    ("tlsh",     "1-2 RTT plus crypto: distance, no resumption -> CDN / TLS config"),
    ("server",   "the application's own think time -> application team"),
    ("transfer", "bandwidth, loss, or a slow producer -> network or app streaming"),
):
    print(f"  {phase:9s} {meaning}")

print("\nThe RTT-multiple structure of a COLD HTTPS request:")
RTT_MS = 70
for step, rtts in (("DNS (cache miss)", 1), ("TCP handshake", 1),
                   ("TLS 1.3 handshake", 1), ("request + first byte", 1)):
    print(f"  {step:28s} {rtts} RTT = {rtts*RTT_MS:4d} ms")
print(f"  {'TOTAL before any content':28s}       = {4*RTT_MS:4d} ms")
print(f"  TLS 1.2 adds one more RTT -> {5*RTT_MS} ms")
print(f"  Warm connection + cached DNS -> {RTT_MS} ms")
print(f"  Reuse alone removes {4*RTT_MS - RTT_MS} ms, which no bandwidth upgrade can.")
#   DNS (cache miss)             1 RTT =   70 ms
#   TCP handshake                1 RTT =   70 ms
#   TLS 1.3 handshake            1 RTT =   70 ms
#   request + first byte         1 RTT =   70 ms
#   TOTAL before any content           =  280 ms
#   TLS 1.2 adds one more RTT -> 350 ms
#   Warm connection + cached DNS -> 70 ms
#   Reuse alone removes 210 ms, which no bandwidth upgrade can.

# --- Step 1: is it a network problem or an application problem? ---
nc -zv api.example.com 443
#  Connection to api.example.com 443 port [tcp/https] succeeded!
#  SUCCEEDS -> the network did its job; look at TLS or the application.
#  FAILS    -> routing, firewall, or nothing listening. Stop looking at the app.
timeout 3 bash -c 'cat < /dev/null > /dev/tcp/api.example.com/443' \
  && echo open || echo "closed or filtered"     # no nc installed

# --- Step 2: the phase breakdown. Keep this as a shell function. ---
curlt() {
  curl -sS -o /dev/null -w \
'dns      %{time_namelookup}s
connect  %{time_connect}s
tls      %{time_appconnect}s
ttfb     %{time_starttransfer}s
total    %{time_total}s
code     %{http_code}   size %{size_download}B   speed %{speed_download}B/s
' "$@"
}
curlt https://api.example.com/health
#  These are CUMULATIVE. Subtract adjacent lines to get per-phase durations.

# --- Separate DNS from everything else by pinning the address ---
curlt --resolve api.example.com:443:203.0.113.50 https://api.example.com/health
#  If this is fast and the previous command was slow, the problem is resolution.

# --- Warm vs cold, on the same connection: quantify the reuse benefit ---
curl -sS -o /dev/null -w 'req1 %{time_total}s\n' \
     -o /dev/null -w 'req2 %{time_total}s\n' \
     https://api.example.com/health https://api.example.com/health
#  The second request skips DNS, TCP and TLS. The difference IS the setup cost.

# --- Inspect TLS without a browser ---
curl -vI https://api.example.com 2>&1 | grep -E 'SSL connection|subject:|expire'
openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null \
  2>/dev/null | openssl x509 -noout -subject -dates -ext subjectAltName
#  Verify the SNI name is in the SAN list: a mismatch is a very common
#  "works in curl -k, fails in the browser" cause.

# --- Check a service behind a load balancer, one backend at a time ---
for ip in $(dig +short api.example.com); do
  echo -n "$ip "
  curl -sS -o /dev/null -m 5 -w '%{http_code} %{time_total}s\n' \
    --resolve "api.example.com:443:$ip" https://api.example.com/health \
    || echo FAILED
done
#  200 0.09s
#  200 0.08s
#  503 0.07s     <- one unhealthy backend: the intermittent-failure signature

Key takeaways

• nc -zv (or telnet) partitions the problem: connection failure means transport or below, success means the network is fine and the fault is in TLS or the application.
• curl -w reports cumulative timestamps, not durations. Subtract adjacent values to get per-phase times.
• The phase that dominates names the responsible component: DNS, connect, TLS, server think time, or transfer.
• A dominant server phase is an application problem, not a network incident.
• A cold HTTPS request costs about four round trips before any content; TLS 1.2 costs five. A warm connection costs one.
• Connection reuse and session resumption therefore matter more than bandwidth for interactive traffic.
• Use --resolve to pin an address and separate DNS from the rest, and to test each backend individually behind a load balancer.
• Verify the SNI name appears in the certificate's SAN list; a mismatch explains many browser-only failures.

🧪 Practice

1. Given time_namelookup=0.006, time_connect=0.092, time_appconnect=0.271, time_starttransfer=0.310, time_total=0.345, compute each phase duration and name the dominant one.
2. Compute the expected cold-start time for an HTTPS request on a 45 ms path with TLS 1.3, and the warm-connection time.
3. Write a command that tests one specific backend behind a DNS name without changing /etc/hosts, and explain why that matters during an intermittent failure.
4. Interview-style: Users report a slow API. The network team shows normal ping times and no packet loss, and the application team shows p99 handler latency of 8 ms. (Hint: both teams may be right — name the phases neither of them measured, and the one command that would show all of them at once.)

iperf and Throughput Measurement

Theory

"The network is slow" needs a number, and iperf3 produces one by generating traffic between two endpoints you control. The reason to use it rather than a file copy is that a file copy measures the whole stack — disk, application, protocol, network — while iperf isolates the network path.

The essential thing to understand before reading any result is that a single TCP stream has two distinct ceilings, and which one you hit determines the fix.

The window ceiling. A sender can have at most one window of unacknowledged data in flight, so throughput cannot exceed window / RTT. The bandwidth-delay product is the window size needed to fill a given path. Without window scaling the maximum window is 64 KB, which on an 80 ms transatlantic path caps a stream at about 6.6 Mb/s regardless of a 1 Gb/s link. The symptom is throughput that is flat and reproducible and completely insensitive to test duration.

The loss ceiling. A loss-based congestion control algorithm spends its life in a sawtooth, and the resulting average throughput is approximately proportional to MSS / (RTT × √loss). On an 80 ms path with 0.1% loss, a single stream is capped near 5.6 Mb/s however large the window. The symptom is a non-zero retransmit count, and the distinguishing feature is that increasing the window does not help.

This is why -P (parallel streams) matters so much for interpretation. Each stream gets its own window and its own sawtooth, so N streams achieve roughly N times a single stream's rate until the link itself becomes the constraint. The two measurements answer different questions and you need both:

• -P 1 answers "what will one application connection get?"
• -P 8 answers "is the path capable of more, given concurrency?"

If -P 8 is eight times -P 1, you are per-flow limited by window or loss. If -P 8 equals -P 1, the link is the limit. Reporting only one number is ambiguous, and reporting only -P 8 is how vendors make a lossy path look fine.

TCP and UDP tests measure different things. A TCP test measures goodput an application would see and tells you why via retransmits and congestion window. A UDP test at a rate you specify measures what the path does at that rate, reporting loss and jitter — and if you specify a rate above capacity, loss is the correct result, not a failure. UDP is the right tool for validating a QoS configuration or a real-time path; TCP is the right tool for predicting application behaviour.

Two cautions. Testing to the wrong place is the most common error: a test to a nearby server proves nothing about a distant path, and a test between two hosts on the same switch proves nothing about the WAN. And on high-rate tests the endpoint is frequently the bottleneck — a single CPU core cannot always drive 10 Gb/s — so a low result may be measuring your test machine.

Examples

import math
# Throughput testing: what limits a single stream, and when to use several.
def bdp_bytes(mbps, rtt_ms):
    return mbps * 1e6 / 8 * rtt_ms / 1000

def window_limited_mbps(window_bytes, rtt_ms):
    return window_bytes * 8 / (rtt_ms / 1000) / 1e6

def loss_limited_mbps(mss_bytes, rtt_ms, loss):
    """Mathis approximation: a single loss-based TCP flow's ceiling."""
    if loss <= 0:
        return float("inf")
    return (mss_bytes * 8 / 1e6) * (1.22 / (rtt_ms / 1000 * math.sqrt(loss)))

DEFAULT_WINDOW = 64 * 1024        # the ceiling WITHOUT window scaling
print("BDP = bytes in flight needed to fill the pipe. '64KB win' is the ceiling")
print("WITHOUT window scaling. 'w/ scaling' assumes the window is large enough.\n")
print(f"{'path':20s} {'link':>7s} {'RTT':>7s} {'BDP':>10s} "
      f"{'64KB win':>10s} {'loss cap':>10s} {'w/ scaling':>11s}")
for name, mbps, rtt, loss in [("LAN", 1_000, 0.5, 0.0), ("metro", 1_000, 8.0, 0.0),
                              ("national", 1_000, 35.0, 0.0),
                              ("transatlantic", 1_000, 80.0, 0.0),
                              ("transatlantic+loss", 1_000, 80.0, 0.001),
                              ("satellite", 100, 600.0, 0.0)]:
    lcap = loss_limited_mbps(1460, rtt, loss)
    print(f"{name:20s} {mbps:6,}M {rtt:6.1f}m {bdp_bytes(mbps, rtt)/1024:9,.0f}K "
          f"{window_limited_mbps(DEFAULT_WINDOW, rtt):9.1f}M "
          f"{'inf' if lcap == float('inf') else f'{lcap:8.1f}M':>10s} "
          f"{min(mbps, lcap):10.1f}M")
# path                    link     RTT        BDP   64KB win   loss cap  w/ scaling
# LAN                   1,000M    0.5m        61K    1048.6M        inf     1000.0M
# metro                 1,000M    8.0m       977K      65.5M        inf     1000.0M
# national              1,000M   35.0m     4,272K      15.0M        inf     1000.0M
# transatlantic         1,000M   80.0m     9,766K       6.6M        inf     1000.0M
# transatlantic+loss    1,000M   80.0m     9,766K       6.6M       5.6M        5.6M
# satellite               100M  600.0m     7,324K       0.9M        inf      100.0M
#
# Read the 64KB column: without window scaling, a 1 Gb/s transatlantic link
# delivers 6.6 Mb/s. That is a 150x shortfall caused by one setting.

print("\nWhy parallel streams: each flow gets its own window and its own sawtooth.")
rtt, loss, link = 80.0, 0.001, 1_000
single = loss_limited_mbps(1460, rtt, loss)
for n in (1, 2, 4, 8, 16, 32):
    total = min(link, n * single)
    print(f"  {n:2d} stream(s) -> {total:7.1f} Mb/s "
          f"({100*total/link:5.1f}% of a {link:,} Mb/s link)")
#    1 stream(s) ->     5.6 Mb/s (  0.6% of a 1,000 Mb/s link)
#    2 stream(s) ->    11.3 Mb/s (  1.1% of a 1,000 Mb/s link)
#    4 stream(s) ->    22.5 Mb/s (  2.3% of a 1,000 Mb/s link)
#    8 stream(s) ->    45.1 Mb/s (  4.5% of a 1,000 Mb/s link)
#   16 stream(s) ->    90.1 Mb/s (  9.0% of a 1,000 Mb/s link)
#   32 stream(s) ->   180.2 Mb/s ( 18.0% of a 1,000 Mb/s link)
#
# Even 32 streams reach only 18%. At 0.1% loss on an 80 ms path, LOSS is the
# binding constraint, so concurrency buys a linear gain and never line rate.
print("\n  -P 1 -> 'what will one application connection get?'")
print("  -P 8 -> 'is the path capable of more, given concurrency?'")
print("  8x difference -> per-flow limited (window or loss).")
print("  no difference -> the LINK itself is the limit.")

# --- Server side, on the far end ---
iperf3 -s -p 5201
# Firewalls must permit 5201 (control AND data by default).

# --- The two measurements that together are unambiguous ---
iperf3 -c far-host -p 5201 -t 30 -P 1 -i 5    # one stream: what an app gets
iperf3 -c far-host -p 5201 -t 30 -P 8 -i 5    # eight: what the path can do
#  [SUM]  0.00-30.00  sec  1.54 GBytes   441 Mbits/sec  1842  sender
#                                                       ^^^^ RETRANSMITS:
#  non-zero retransmits with flat throughput = loss-limited, not window-limited.

# --- Which ceiling did you hit? Ask the kernel, per socket, during the test ---
ss -tin dst far-host | grep -E 'cwnd|retrans|rtt'
#  rtt:80.2/1.1 mss:1448 cwnd:512 ssthresh:498 retrans:0/1842 delivery_rate:44Mbps
#  cwnd hovering near ssthresh with rising retrans -> LOSS-limited.
#  cwnd pinned at a constant with retrans:0/0      -> WINDOW-limited.

# Confirm window scaling is actually enabled and the buffers allow a large window.
sysctl net.ipv4.tcp_window_scaling net.ipv4.tcp_rmem net.ipv4.tcp_wmem
iperf3 -c far-host -w 16M -t 20            # force a large window explicitly

# --- UDP: measure what the path does at a rate YOU choose ---
iperf3 -c far-host -u -b 900M -t 20 -l 1200
#  [  5] 0.00-20.00 sec  2.05 GBytes  879 Mbits/sec  0.041 ms  6122/1789341 (0.34%)
#                                                    ^jitter   ^lost/total
#  Loss at a rate above capacity is the MEASUREMENT, not a failure.

# Validate a QoS policy by marking the test traffic (Chapter 12's classes).
iperf3 -c far-host -u -b 50M --dscp 46 -t 20      # EF: should show low loss/jitter
iperf3 -c far-host -u -b 50M --dscp 8  -t 20      # CS1 scavenger: should suffer first

# --- Rule out your own machine before blaming the network ---
iperf3 -c localhost -P 4 -t 10       # loopback: an upper bound for this host
mpstat -P ALL 1 5 | tail -6          # a single core at 100% during the test means
                                     # you are measuring the endpoint, not the path
ethtool -S eth0 | grep -iE 'drop|error|discard'

Key takeaways

• iperf isolates the network path, unlike a file copy which measures the whole stack.
• A single TCP stream has two ceilings. The window ceiling is window / RTT; the bandwidth-delay product is the window needed to fill the path.
• Without window scaling, the 64 KB maximum caps a 1 Gb/s transatlantic path near 6.6 Mb/s.
• The loss ceiling is approximately MSS / (RTT × √loss); a bigger window does not raise it.
• Distinguish them from the kernel: constant cwnd with zero retransmits is window-limited; cwnd near ssthresh with rising retransmits is loss-limited.
• Report both -P 1 and -P 8. An eightfold difference means per-flow limits; no difference means the link is the limit.
• TCP measures goodput and why; UDP measures what the path does at a chosen rate. Loss in a UDP test above capacity is the measurement, not a failure.
• Test to the right place, and rule out your own endpoint — a single core often cannot drive 10 Gb/s.

🧪 Practice

1. Compute the bandwidth-delay product for a 10 Gb/s link with a 40 ms round trip, and the throughput a 64 KB window would allow on it.
2. An iperf3 test shows 220 Mb/s with -P 1 and 1.8 Gb/s with -P 8 on a 10 Gb/s link. State your diagnosis and the next measurement you would take.
3. Design a UDP test plan that validates a three-class QoS policy, giving the rates and DSCP values you would send and what result would prove the policy works.
4. Interview-style: A customer on a new 1 Gb/s transatlantic circuit reports 6 Mb/s file transfers, and the provider's own test shows 940 Mb/s. (Hint: the provider almost certainly used many parallel streams — work out what a single stream can achieve at that round trip with a 64 KB window, and which setting changes it.)

tcpdump Filters and Capture

Theory

Every other tool reports a summary. Packet capture shows what actually crossed the wire, which makes it the tool of last resort and the arbiter when two teams disagree. The cost is volume: a saturated 10 Gb/s link produces 75 GB per minute, so the whole discipline of capture is deciding what not to keep.

BPF capture filters are the primary lever, and the reason they matter is where the work happens. A capture filter is compiled and executed in the kernel, so excluded packets are never copied to userspace at all. A display filter in Wireshark runs after the packet has been copied, written to disk, read back, and parsed. The difference is orders of magnitude, and it is why the filter belongs at capture time whenever you know what you are looking for.

The syntax is worth committing to memory because it is small and it is the same everywhere BPF is used:

Filter	Matches
host 10.0.0.5	either direction to that address
src net 10.1.0.0/16	source in that network
port 443	either source or destination port
tcp portrange 8000-8100	a range
tcp[tcpflags] & tcp-syn != 0	SYN set (connection attempts)
tcp[tcpflags] & tcp-rst != 0	resets: extremely useful
ip[1] & 0xfc == 0xb8	DSCP 46 (EF), masking off ECN
vlan 20 and host 10.0.0.5	inside a VLAN tag
not (port 22 or port 3389)	exclude your own management traffic

The second lever is snaplen. -s 128 keeps only the first 128 bytes of each packet, which holds all the headers and discards the payload. On typical traffic that is roughly a sevenfold reduction, and for the majority of network problems — retransmissions, resets, handshake failures, DSCP marking, MTU issues — headers are all you need. Full payload is needed only when the problem is in the application's bytes.

The third is a ring buffer: -W 10 -C 100 keeps ten files of 100 MB each, overwriting the oldest. This is what makes capture viable for intermittent faults, and the trade is worth stating plainly: for an intermittent problem, history beats detail. A long header-only capture that was running when the fault occurred is infinitely more useful than a full-payload capture you started afterwards.

Finally, and critically: a capture that drops packets is worse than no capture, because missing packets look exactly like network loss and will send you chasing a fault that exists only in your tooling. Always read the drop counters that tcpdump prints on exit, and treat any non-zero value as invalidating the capture for loss analysis.

Examples

# Capture sizing: the arithmetic that decides between a useful capture and a full disk.
def capture_volume(mbps, seconds, snaplen=None, avg_pkt=900, filtered_fraction=1.0):
    bytes_on_wire = mbps * 1e6 / 8 * seconds * filtered_fraction
    if snaplen is None:
        return bytes_on_wire
    # With a snaplen each packet is truncated: only min(snaplen, pkt) is stored.
    return bytes_on_wire * (min(snaplen, avg_pkt) / avg_pkt)

print("Capture volume for 60 seconds, at various link loads:")
print(f"{'load':>8s} {'full capture':>14s} {'snaplen 128':>14s} {'+ filtered 2%':>15s}")
for mbps in (10, 100, 1_000, 10_000):
    print(f"{mbps:6,}M {capture_volume(mbps, 60)/1e9:12.2f}GB "
          f"{capture_volume(mbps, 60, snaplen=128)/1e9:12.2f}GB "
          f"{capture_volume(mbps, 60, snaplen=128, filtered_fraction=0.02)/1e9:13.3f}GB")
#     load   full capture    snaplen 128   + filtered 2%
#     10M         0.07GB         0.01GB         0.000GB
#    100M         0.75GB         0.11GB         0.002GB
#  1,000M         7.50GB         1.07GB         0.021GB
# 10,000M        75.00GB        10.67GB         0.213GB
#
# 75 GB/minute unfiltered on a 10 Gb/s link, versus 213 MB with a filter and a
# snaplen: a 350x reduction, and for most problems no loss of information.

print("\nWhy a BPF filter beats filtering later: WHERE the work happens.")
for stage, note, cost in (
        ("kernel BPF (capture filter)", "packet never leaves the kernel", 1),
        ("tcpdump write to disk", "already copied and written", 40),
        ("Wireshark display filter", "already on disk, already parsed", 400)):
    print(f"  {stage:30s} relative cost {cost:4d}x   {note}")

print("\nSizing a ring buffer to survive until a human looks at it:")
rate_gb_per_min = capture_volume(1_000, 60, snaplen=128) / 1e9
print(f"  1 Gb/s at snaplen 128: {rate_gb_per_min:.2f} GB/min")
for budget_gb in (10, 100, 500):
    minutes = budget_gb / rate_gb_per_min
    print(f"  {budget_gb:4d} GB budget -> {minutes:7.1f} minutes "
          f"({minutes/60:5.1f} hours) of history")
#   1 Gb/s at snaplen 128: 1.07 GB/min
#     10 GB budget ->     9.4 minutes (  0.2 hours) of history
#    100 GB budget ->    93.8 minutes (  1.6 hours) of history
#    500 GB budget ->   468.8 minutes (  7.8 hours) of history
#
# For an intermittent fault, HISTORY beats DETAIL: a long header-only capture
# that was running beats a full-payload one you started afterwards.

print("\nDrop accounting: a capture that drops packets is worse than none,")
print("because the missing packets look exactly like network loss.")
for sent, kernel_drop, iface_drop in ((1_000_000, 0, 0), (1_000_000, 84_221, 0),
                                      (1_000_000, 0, 12_004)):
    total = sent + kernel_drop + iface_drop
    ok = kernel_drop + iface_drop == 0
    print(f"  captured {sent:,} | kernel dropped {kernel_drop:,} | "
          f"iface dropped {iface_drop:,} -> "
          f"{'TRUSTWORTHY' if ok else 'NOT TRUSTWORTHY'} "
          f"({100*(kernel_drop+iface_drop)/total:.1f}% missing)")
#   captured 1,000,000 | kernel dropped 0 | iface dropped 0 -> TRUSTWORTHY (0.0% missing)
#   captured 1,000,000 | kernel dropped 84,221 | iface dropped 0 -> NOT TRUSTWORTHY (7.8% missing)
#   captured 1,000,000 | kernel dropped 0 | iface dropped 12,004 -> NOT TRUSTWORTHY (1.2% missing)

# --- The capture you should default to: filtered, truncated, ring-buffered ---
tcpdump -i eth0 -nn -s 128 -w /var/cap/trace.pcap -W 20 -C 100 \
  'host 10.0.2.15 and tcp port 5432 and not port 22'
#  -nn no name resolution (never let DNS add latency to a capture)
#  -s 128 headers only        -W 20 -C 100 -> 20 files x 100 MB, oldest overwritten
#  the filter runs IN THE KERNEL: excluded packets are never copied

# --- READ THE DROP COUNTERS. Always. ---
# tcpdump prints these on exit (Ctrl-C):
#  1284412 packets captured
#  1284412 packets received by filter
#        0 packets dropped by kernel        <- MUST be 0 for loss analysis
ip -s link show eth0 | grep -A1 RX
ethtool -S eth0 | grep -iE 'rx_dropped|rx_missed|fifo'
#  Non-zero here means the NIC dropped before tcpdump saw anything.

# --- The filters that answer specific questions ---
# Connection attempts only (who is trying to connect, and is it reaching us?)
tcpdump -i eth0 -nn 'tcp[tcpflags] & tcp-syn != 0 and tcp[tcpflags] & tcp-ack == 0'

# Resets: the fastest way to find which side is refusing
tcpdump -i eth0 -nn 'tcp[tcpflags] & tcp-rst != 0'

# Retransmissions cannot be filtered in BPF (it needs state) -- capture, then:
tshark -r trace.pcap -Y 'tcp.analysis.retransmission' -T fields \
  -e frame.number -e ip.src -e ip.dst | head

# QoS verification: DSCP 46 (EF). Mask 0xfc to ignore the ECN bits.
tcpdump -i eth0 -nn -v 'ip[1] & 0xfc == 0xb8'

# ICMP unreachables, including the fragmentation-needed messages that reveal MTU
tcpdump -i eth0 -nn 'icmp[icmptype] == 3'

# Exclude your own session, or the capture fills with itself
tcpdump -i eth0 -nn 'not (port 22 and host 203.0.113.9)'

# --- Capture on both sides simultaneously: the decisive measurement ---
# The single most useful technique in this chapter. If a packet appears in the
# sender's capture and not the receiver's, the network lost it. If it appears in
# both, the network did not. This settles almost every disagreement.
ssh sender   'timeout 60 tcpdump -i eth0 -nn -s 128 -w /tmp/s.pcap host 10.0.2.15' &
ssh receiver 'timeout 60 tcpdump -i eth0 -nn -s 128 -w /tmp/r.pcap host 10.0.1.9'  &
wait
for h in sender receiver; do scp $h:/tmp/*.pcap ./$h.pcap; done
for f in sender receiver; do
  echo -n "$f: "; tshark -r $f.pcap -T fields -e tcp.seq 2>/dev/null | sort -u | wc -l
done
#  Compare the sequence-number sets: what is in one and not the other was lost
#  BETWEEN the two capture points.

# --- Long-running background capture, for intermittent faults ---
# Keep a rolling window running permanently on a suspect host. Cheap insurance.
tcpdump -i eth0 -nn -s 96 -w /var/cap/ring.pcap -W 48 -C 200 -z gzip \
  'not (port 22 or port 161)' &

Key takeaways

• Packet capture is the arbiter when summaries disagree, and the whole discipline is deciding what not to keep.
• BPF capture filters run in the kernel, so excluded packets are never copied. A display filter runs after copying, writing, reading, and parsing.
• Learn the small filter vocabulary: host, net, port, portrange, the tcp[tcpflags] tests, ip[1] & 0xfc for DSCP, and not to exclude your own session.
• Snaplen of about 128 bytes keeps all headers and gives roughly a sevenfold reduction; full payload is needed only for application-byte problems.
• Use a ring buffer so an intermittent fault is captured when it happens: history beats detail.
• Always read the drop counters. A capture with kernel or interface drops cannot be used for loss analysis, because missing packets look like network loss.
• Capture on both sides simultaneously — a packet present in the sender's trace and absent from the receiver's proves the network lost it, and nothing else does.
• Use -nn so name resolution never perturbs the capture.

🧪 Practice

1. Write the BPF filter that captures only TCP connection attempts to port 5432 from the 10.1.0.0/16 network, excluding your own SSH session.
2. Compute the capture volume for 5 minutes on a 40% utilised 1 Gb/s link, with and without a 128-byte snaplen.
3. Explain why retransmissions cannot be expressed as a BPF capture filter, and how you would find them instead.
4. Interview-style: A capture taken on a server shows 4% packet loss, and the network team's device counters show none. (Hint: there is a third possibility besides one of them being wrong — check what tcpdump itself reported on exit, and what the NIC counters say about packets that never reached it.)

<a id="142-packet-analysis"></a>

14.2 Packet Analysis

A packet capture is the only evidence that settles an argument between two teams, and it is also the easiest evidence to misread: this subchapter covers choosing a capture point that sees what you need, filtering the result down to the relevant conversation, reconstructing application data from segments, recognising what retransmissions and resets actually indicate, and what remains diagnosable when the payload is encrypted.

Capture Points and Port Mirroring

Theory

Before filtering or reading anything, you have to decide where to capture, and this choice determines what the capture is capable of proving. A capture taken in the wrong place can be internally consistent, entirely truthful, and useless for the question you are asking.

There are four options, and the differences matter:

On the host, with tcpdump. Cheap, immediate, and it sees exactly what this machine's stack sent or received. Two caveats: it observes packets after NIC offload processing, so hardware features such as large receive offload can coalesce several wire packets into one that tcpdump reports as a single huge segment, and it cannot see anything the network dropped before delivery.

A SPAN or mirror port on a switch. The switch copies traffic from selected ports to a monitor port. It requires no cabling change and can be enabled remotely, which is why it is the usual choice. Its weakness is the one covered below: it can silently drop copies.

A passive TAP. A physical device in line with a cable that copies every bit to a monitor port. It sees everything on that link, cannot drop under load, and cannot be reconfigured remotely. It requires physical access and a cable break to install, and it sees only the one link.

Inline or bridged capture, where the capturing host is in the traffic path. It sees and can modify everything, and its failure breaks the link, which makes it unsuitable for production paths unless that risk is accepted deliberately.

The mistake worth dwelling on is mirror oversubscription, because it produces false conclusions rather than obvious failures. A mirror destination must carry both directions of every mirrored source. Mirroring one fully utilised 10 Gb/s link therefore needs up to 20 Gb/s of destination capacity, and mirroring four busy 1 Gb/s access ports into one 1 Gb/s monitor port cannot work at all. When the destination cannot keep up, the switch drops mirror copies silently — and those missing packets look exactly like network packet loss in your analysis. People then spend days investigating loss that exists only in the measurement apparatus.

The practical rule is to let the question dictate the capture point:

Question	Capture where
Did the server receive this request?	on the server
Did the network deliver this packet?	both ends, simultaneously
What does the wire actually carry?	a TAP, or a mirror you have verified
Which VLAN or DSCP is on the frame?	a mirror or TAP; the host sees post-processing
Is TLS negotiating what I expect?	either end; a midpoint sees only ciphertext

The second row is the most valuable technique in this chapter and is worth stating as a principle: a simultaneous capture at both ends turns an argument into a measurement. If a packet appears in the sender's capture and not the receiver's, the network lost it. If it appears in both, the network did not. Nothing else settles that question.

Examples

Four capture points, and what each one can and cannot see

  [ client ]---(A)---[ switch ]---(B)---[ router ]---[ switch ]---(C)---[ server ]
                        |                                            |
                     SPAN/TAP                                    tcpdump
                        |
                  [ analyser ]

  (A) TAP on the client link  : every bit of THIS link, both directions
  (B) SPAN on the switch      : copies of selected ports -- CAN OVERFLOW
  (C) tcpdump on the server   : what the server's stack saw, post-offload

  Mirror oversubscription, drawn:

    port 1 (10G, 95% used)  ---\
    port 2 (10G, 95% used)  ----+---> monitor port (10G)
                                       ^ needs 2 x 2 x 9.5 = 38 Gb/s
                                         gets 10 Gb/s
                                         -> ~74% of COPIES silently discarded
                                         -> your capture shows "packet loss"
                                            that the network never had

  The decisive technique: capture at BOTH ends at the same time

    sender capture           receiver capture        conclusion
    seq 1000 present         seq 1000 present        network delivered it
    seq 2448 present         seq 2448 ABSENT         NETWORK LOST IT
    seq 2448 absent          seq 2448 absent         sender never sent it

# Capture point selection, and the oversubscription that silently ruins mirrors.
POINTS = {
    # name:            (sees, misses, drops under load, intrusive)
    "host (tcpdump)":  ("everything this host sends/receives, post-offload",
                        "what the NIC coalesced or the switch dropped", "no", "no"),
    "SPAN / mirror":   ("a copy of selected switch ports",
                        "anything dropped BEFORE the mirror, and mirror overflow",
                        "YES", "no"),
    "TAP (passive)":   ("every bit on one physical link",
                        "nothing on that link; sees only that link", "no", "no"),
    "inline (bridge)": ("everything, and can modify", "nothing", "no",
                        "YES: a failure breaks the link"),
}
for name, (sees, misses, drops, intrusive) in POINTS.items():
    print(f"{name:18s} sees   {sees}")
    print(f"{'':18s} misses {misses}")
    print(f"{'':18s} drops under load: {drops:4s}   affects traffic: {intrusive}")

print("\nThe arithmetic nobody does before configuring a SPAN session:")
def span_check(sources_gbps, dest_gbps, utilisation):
    # A mirror carries BOTH directions of every mirrored source: hence the x2.
    offered = sum(s * utilisation for s in sources_gbps) * 2
    return offered, dest_gbps, offered / dest_gbps

print(f"{'mirrored sources':28s} {'dest':>7s} {'util':>6s} {'offered':>9s} "
      f"{'ratio':>7s}  verdict")
for label, srcs, dest, util in (
        ("1 x 1G access port",     [1], 1, 0.30),
        ("4 x 1G access ports",    [1]*4, 1, 0.30),
        ("2 x 10G server ports",   [10]*2, 10, 0.40),
        ("1 x 10G uplink",         [10], 10, 0.60),
        ("1 x 10G uplink (busy)",  [10], 10, 0.95)):
    offered, d, ratio = span_check(srcs, dest, util)
    verdict = "fits" if ratio <= 1 else f"MIRROR DROPS {100*(1-1/ratio):.0f}% of copies"
    print(f"{label:28s} {d:6.0f}G {util:5.0%} {offered:8.1f}G {ratio:6.2f}x  {verdict}")
# mirrored sources                dest   util   offered   ratio  verdict
# 1 x 1G access port                1G   30%      0.6G   0.60x  fits
# 4 x 1G access ports               1G   30%      2.4G   2.40x  MIRROR DROPS 58% of copies
# 2 x 10G server ports             10G   40%     16.0G   1.60x  MIRROR DROPS 38% of copies
# 1 x 10G uplink                   10G   60%     12.0G   1.20x  MIRROR DROPS 17% of copies
# 1 x 10G uplink (busy)            10G   95%     19.0G   1.90x  MIRROR DROPS 47% of copies
#
# Only the first row works. When the destination overflows, the switch drops
# MIRROR COPIES silently, and those look exactly like network packet loss.

print("\nWhere to capture, given the question you are asking:")
for q, where in (
    ("does the server receive this request?", "the server itself"),
    ("did the network deliver this packet?", "BOTH ends, simultaneously"),
    ("what does the wire actually carry?", "a TAP, or a mirror you have verified"),
    ("which VLAN/DSCP is on the frame?", "a mirror or TAP (host sees post-processing)"),
    ("is TLS negotiating what I think?", "either end; a midpoint sees only ciphertext"),
):
    print(f"  {q:40s} -> {where}")

# --- Configure a mirror, then VERIFY it is not dropping ---
# (Cisco-style; the principle is identical everywhere)
#   monitor session 1 source interface Gi1/0/5 both
#   monitor session 1 destination interface Gi1/0/48
show monitor session 1
# Then check the DESTINATION port for drops. This step is almost always skipped.
show interfaces Gi1/0/48 | include drops|discard
#  Output drops: 4182931     <- your capture is missing 4.1 M packets

# Narrow the mirror at the SOURCE so it fits, rather than capturing everything.
#   monitor session 1 filter access-group CAPTURE-ACL
# A filtered mirror of the one conversation you care about cannot overflow.

# --- Verify a capture point before trusting it: send known traffic ---
# From the client, send a countable, identifiable stream:
ping -c 100 -p deadbeef 10.0.2.15
# On the analyser, count what arrived:
tshark -r mirror.pcap -Y 'icmp.type==8' 2>/dev/null | wc -l
#  100 -> the mirror is faithful.  87 -> it is dropping; fix it before analysing.

# --- Host-side: disable offload so tcpdump shows real wire packets ---
ethtool -k eth0 | grep -E 'generic-receive-offload|large-receive-offload|tcp-segmentation'
#  generic-receive-offload: on      <- tcpdump will show merged super-segments
sudo ethtool -K eth0 gro off lro off
#  A 24 kB "packet" in a capture on an Ethernet link is offload, not a real frame.
#  Turn it off for analysis, and remember to turn it back on.

# --- THE decisive measurement: simultaneous capture at both ends ---
CLIENT=10.0.1.9; SERVER=10.0.2.15
ssh $CLIENT "timeout 60 tcpdump -i any -nn -s 96 -w /tmp/c.pcap host $SERVER" &
ssh $SERVER "timeout 60 tcpdump -i any -nn -s 96 -w /tmp/s.pcap host $CLIENT" &
wait
scp $CLIENT:/tmp/c.pcap ./client.pcap; scp $SERVER:/tmp/s.pcap ./server.pcap

# Compare the sequence numbers each side observed.
for f in client server; do
  tshark -r $f.pcap -Y "ip.src==$CLIENT && tcp.len>0" \
    -T fields -e tcp.seq 2>/dev/null | sort -n -u > $f.seq
  echo "$f: $(wc -l < $f.seq) distinct segments"
done
comm -23 client.seq server.seq | head
#  Anything listed here left the client and never reached the server:
#  the network lost it, and now you have proof rather than an opinion.

# --- Clock skew: align the two captures before comparing timestamps ---
# Without this, "the server replied before the client asked" is a common artefact.
tshark -r client.pcap -T fields -e frame.time_epoch -c1
tshark -r server.pcap -T fields -e frame.time_epoch -c1
chronyc tracking | grep -E 'System time|Last offset'   # on both hosts

Key takeaways

• The capture point determines what the capture can prove; a correct capture in the wrong place is useless.
• Host capture is cheap but sees packets after NIC offload, so coalesced super-segments are an artefact, not real frames.
• A SPAN or mirror needs no recabling and can silently drop copies; a TAP cannot drop but needs physical access; inline capture can modify traffic and breaks the link if it fails.
• A mirror must carry both directions of every source, so one busy 10 Gb/s link needs up to 20 Gb/s of destination.
• Dropped mirror copies look exactly like network loss. Always check the destination port's drop counter, and prefer a filtered mirror.
• Verify a capture point with known countable traffic before trusting it.
• Capture at both ends simultaneously to prove whether the network lost a packet; nothing else answers that question.
• Align clocks before comparing timestamps across two captures.

🧪 Practice

1. Compute the mirror destination bandwidth needed to mirror three 10 Gb/s ports at 50% utilisation, and state whether a single 10 Gb/s monitor port suffices.
2. Explain why a capture on a server might show a 40 kB TCP segment on an Ethernet link, and the command that prevents it.
3. Design a procedure that proves a SPAN session is faithful before you rely on it for loss analysis.
4. Interview-style: An analyser attached to a mirror port reports 6% packet loss on a link; the switch's own interface counters on that link show zero errors or discards. (Hint: there is a third device in the measurement path that neither figure describes — work out what it has to carry, and which counter on it nobody checked.)

Wireshark Display Filters

Theory

Once a capture exists, the problem becomes finding the twelve interesting packets among two million. Wireshark and its command-line equivalent tshark provide display filters, and the first thing to be clear about is that these are a completely different language from the BPF capture filters of the previous subchapter. Confusing them is the most common beginner error, and the syntax is deceptively similar:

Purpose	Capture filter (BPF)	Display filter (Wireshark)
One host	host 10.0.0.5	ip.addr == 10.0.0.5
One port	port 443	tcp.port == 443
SYN packets	tcp[tcpflags] & tcp-syn != 0	tcp.flags.syn == 1
Negation	not port 22	!(tcp.port == 22)
Combination	and / or / not	&& / || / !

Beyond syntax, they differ in power and cost. A capture filter runs in the kernel on undecoded bytes, so it is fast and limited to what can be computed from fixed offsets. A display filter runs after full protocol dissection, so it can reference any field of any protocol — and, critically, it can reference Wireshark's own analysis, which is where its real value lies. tcp.analysis.retransmission is not a field in any packet; it is a conclusion Wireshark reached by tracking the connection's state. That class of filter is impossible in BPF and is what makes display filters worth learning.

The filters that earn their keep in practice:

Filter	Finds
tcp.analysis.flags	every TCP problem Wireshark detected, in one expression
tcp.analysis.retransmission	retransmissions
tcp.analysis.zero_window	a receiver that has stopped accepting data
tcp.flags.reset == 1	resets
tcp.time_delta > 1	gaps of over a second within a conversation
http.response.code >= 500	server errors
http.time > 1	slow responses, with the request matched to them
dns.flags.rcode != 0	failed DNS answers
tls.handshake.type == 1	ClientHello, for SNI and offered versions
tls.alert_message.desc	TLS failures and their reason
ip.ttl < 10	possibly forged packets, or a long path
tcp.len > 0	data packets only, excluding bare ACKs

Two habits make filtering far more effective. First, tcp.stream == N isolates one conversation, and almost all analysis should begin by identifying the relevant stream number and restricting to it — everything else is noise. Second, tcp.analysis.flags is the right first filter on any capture you have not looked at yet, because it surfaces every anomaly the dissector noticed without you having to guess which one to look for.

A caution about tcp.time_delta and similar time fields: they measure time between packets in the capture, which is the capture point's view. On a mirror that buffers, or with clock adjustments, these can be misleading. They are indicative, not authoritative.

Examples

Two filter languages, one common confusion

  CAPTURE FILTER (BPF)                 DISPLAY FILTER (Wireshark)
  ---------------------                --------------------------
  runs: in the KERNEL                  runs: after full DISSECTION
  sees: raw bytes at fixed offsets     sees: every named field, plus ANALYSIS
  cost: ~free (never copied)           cost: must already have the packet
  power: limited                       power: very high

  host 10.0.0.5 and port 443           ip.addr==10.0.0.5 && tcp.port==443
  tcp[tcpflags] & tcp-syn != 0         tcp.flags.syn==1 && tcp.flags.ack==0
  (cannot express)                     tcp.analysis.retransmission
                                       ^ a CONCLUSION, not a field: Wireshark
                                         tracked the connection to decide this

  The workflow that actually finds problems:

   1. tcp.analysis.flags          <- what is wrong ANYWHERE in this capture?
            |
   2. note the tcp.stream number of an interesting packet
            |
   3. tcp.stream == 42            <- isolate that ONE conversation
            |
   4. tcp.stream == 42 && tcp.len > 0   <- drop bare ACKs, see the data flow
            |
   5. Follow Stream               <- read the application exchange

# --- Step 1: what is wrong, anywhere? Always start here. ---
tshark -r trace.pcap -Y 'tcp.analysis.flags' -T fields \
  -e frame.number -e ip.src -e ip.dst -e tcp.stream -e _ws.expert.message \
  2>/dev/null | head -20
#  1842  10.0.1.9  10.0.2.15  42  This frame is a (suspected) retransmission
#  1907  10.0.2.15 10.0.1.9   42  TCP window specified by the receiver is now zero
#                                 ^ stream 42 has both loss AND a stalled receiver

# --- Step 2: which conversations exist, and which are the interesting ones? ---
tshark -r trace.pcap -q -z conv,tcp | head -15
#  <-> pairs with packet counts, bytes and duration; sort by whatever matters
tshark -r trace.pcap -q -z expert,note      # every anomaly, grouped and counted

# --- Step 3: isolate ONE conversation. Most analysis should begin here. ---
tshark -r trace.pcap -Y 'tcp.stream == 42' -T fields \
  -e frame.time_relative -e ip.src -e tcp.seq -e tcp.ack -e tcp.len \
  -e tcp.flags.str -e tcp.analysis.flags

# --- The filters worth memorising ---
tshark -r trace.pcap -Y 'tcp.flags.reset == 1' -T fields \
  -e frame.number -e ip.src -e ip.ttl -e tcp.port
#  Include ip.ttl: a RST whose TTL differs from the peer's other packets was
#  forged by a middlebox, not sent by the server.

tshark -r trace.pcap -Y 'tcp.analysis.zero_window'      # receiver stopped reading
tshark -r trace.pcap -Y 'tcp.time_delta > 1 && tcp.len > 0'   # 1s+ stalls
tshark -r trace.pcap -Y 'http.time > 1' -T fields \
  -e http.request.uri -e http.time -e http.response.code
#  http.time is the REQUEST-to-RESPONSE duration, matched by the dissector:
#  this one filter finds slow endpoints without any application instrumentation.

tshark -r trace.pcap -Y 'dns.flags.rcode != 0' -T fields -e dns.qry.name -e dns.flags.rcode
tshark -r trace.pcap -Y 'tls.handshake.type == 1' -T fields \
  -e tls.handshake.extensions_server_name -e tls.handshake.version
tshark -r trace.pcap -Y 'tls.alert_message.desc' -T fields \
  -e ip.src -e tls.alert_message.desc

# --- Statistics that answer questions faster than reading packets ---
tshark -r trace.pcap -q -z io,stat,1,'COUNT(tcp.analysis.retransmission)tcp.analysis.retransmission'
#  Retransmissions per second: is the loss continuous or bursty? The shape
#  distinguishes a saturated link from a brief event.
tshark -r trace.pcap -q -z endpoints,tcp | sort -k3 -rn | head
tshark -r trace.pcap -q -z http,tree | head -30

# --- Build a capture filter from what you learned, then re-capture narrowly ---
# Having identified the conversation, capture it properly next time:
#   tcpdump -i eth0 -nn -s0 -w focused.pcap 'host 10.0.2.15 and tcp port 5432'

Key takeaways

• Capture filters (BPF) and display filters (Wireshark) are different languages. Confusing them is the most common error.
• A capture filter runs in the kernel on raw bytes; a display filter runs after dissection and can reference any field.
• Display filters can reference Wireshark's own analysis, such as tcp.analysis.retransmission, which is a conclusion rather than a packet field and cannot be expressed in BPF.
• Start with tcp.analysis.flags on any unfamiliar capture: it surfaces every anomaly the dissector found.
• Isolate one conversation with tcp.stream == N before detailed analysis; everything else is noise.
• Include ip.ttl when examining resets — an anomalous TTL means a middlebox forged it.
• http.time gives per-request duration with the response matched to the request, which finds slow endpoints without application instrumentation.
• Time-delta fields reflect the capture point's view and are indicative rather than authoritative.
• Use statistics views (-z conv,tcp, -z io,stat, -z expert) to characterise a capture before reading individual packets.

🧪 Practice

1. Translate each into both filter languages: traffic to or from 192.168.5.20 on port 8080; TCP packets with only the SYN flag; everything except SSH.
2. Write the tshark command that lists every HTTP request taking over 500 ms, showing the URI and response code.
3. Explain why tcp.analysis.retransmission cannot be a capture filter, and what Wireshark must do to evaluate it.
4. Interview-style: An engineer applies the filter tcp.port == 443 as a capture filter and gets a syntax error, then applies port 443 as a display filter and gets no results. (Hint: name the two languages and where each is evaluated, then give the correct expression for each context.)

Following Streams and Reassembly

Theory

TCP presents a byte stream to applications, but the network carries segments. A capture shows segments; the application saw a stream. Reconstructing one from the other is what "Follow TCP Stream" does, and understanding the reconstruction is necessary because the places where it is difficult are exactly the places where bugs live.

The receiver's job is to deliver bytes in order and exactly once, and it must handle three complications that all occur routinely in real captures:

Out-of-order arrival. Segments can take different ECMP paths and arrive in the wrong order. The receiver holds later data in a buffer until the gap fills, then delivers everything at once. The application never observes the disorder.

Duplicates. A retransmission of data already received arrives again. The receiver discards it. A duplicate in a capture is not an error; it is a retransmission doing its job.

Overlaps. A retransmission may partially overlap already-received data, because the sender repacketised. The receiver trims the overlap and keeps only the new tail.

The consequence that matters most for reading captures: if a gap never fills, everything after it is invisible to the application, no matter how clearly your capture shows it arriving. This produces one of the most confusing situations in troubleshooting, where "the packets are in the capture" and "the server never received the request" are both completely true. The application read the bytes before the gap and then blocked forever, while the bytes after the gap sat in a kernel buffer it could not reach.

A second consequence is that stream byte offsets do not correspond to packet boundaries. A single HTTP header can span two segments; two HTTP requests can share one. Any analysis that assumes one message per packet is wrong, and this is why Wireshark's reassembly must be enabled for application-layer dissectors to work at all.

Practical notes on using stream following. The stream index (tcp.stream) is Wireshark's own numbering of conversations in the capture and is the handle for all subsequent work. Following a stream shows both directions, conventionally colour-coded, and can export the payload in either direction — which is how you extract a transferred file or the exact bytes of a malformed request. For UDP there is no reassembly to do, since each datagram is independent, so following a UDP stream simply concatenates datagrams and any apparent ordering is the capture's, not a protocol guarantee.

Examples

# A minimal TCP reassembler: what "Follow Stream" is actually doing.
# Real captures contain out-of-order, duplicate, and OVERLAPPING segments.
SEGMENTS = [
    # (seq, payload)  -- the stream is a 67-byte HTTP request
    (1,  b"GET /api/v1/ord"),          # 1..15
    (16, b"ers HTTP/1.1\r\n"),         # 16..29
    (53, b"Accept: */*\r\n\r\n"),      # arrives EARLY: 53..67, gap at 30
    (16, b"ers HTTP/1.1\r\nHost: "),   # retransmission that OVERLAPS by 14 bytes
    (36, b"api.example.com\r\n"),      # 36..52, fills the gap
    (53, b"Accept: */*\r\n\r\n"),      # a pure DUPLICATE
]

def reassemble(segments, isn=1):
    """Hold out-of-order data until the gap fills, exactly as a receiver does."""
    buf, next_seq, pending, log = bytearray(), isn, {}, []
    for seq, data in segments:
        if seq < next_seq:                          # starts before what we need
            overlap = next_seq - seq
            if overlap >= len(data):
                log.append(f"seq {seq:3d} len {len(data):2d}  DUPLICATE, discarded")
                continue
            log.append(f"seq {seq:3d} len {len(data):2d}  OVERLAP {overlap}B, trimmed "
                       f"to seq {next_seq}")
            seq, data = next_seq, data[overlap:]    # keep only the new tail
        if seq > next_seq:                          # a gap: cannot deliver yet
            pending[seq] = data
            log.append(f"seq {seq:3d} len {len(data):2d}  OUT OF ORDER, held "
                       f"(waiting for {next_seq})")
            continue
        buf += data
        next_seq = seq + len(data)
        log.append(f"seq {seq:3d} len {len(data):2d}  delivered, next={next_seq}")
        while next_seq in pending:                  # the gap just filled
            d = pending.pop(next_seq)
            buf += d
            log.append(f"seq {next_seq:3d} len {len(d):2d}  released from hold, "
                       f"next={next_seq + len(d)}")
            next_seq += len(d)
    return bytes(buf), next_seq, pending, log

data, nxt, held, log = reassemble(SEGMENTS)
for line in log:
    print("  " + line)
print(f"\nreassembled {len(data)} bytes, next expected seq {nxt}, "
      f"{len(held)} segment(s) still held")
#   seq   1 len 15  delivered, next=16
#   seq  16 len 14  delivered, next=30
#   seq  53 len 15  OUT OF ORDER, held (waiting for 30)
#   seq  16 len 20  OVERLAP 14B, trimmed to seq 30
#   seq  30 len  6  delivered, next=36
#   seq  36 len 17  delivered, next=53
#   seq  53 len 15  released from hold, next=68
#   seq  53 len 15  DUPLICATE, discarded
#
# reassembled 67 bytes, next expected seq 68, 0 segment(s) still held
#
# Note that 8 observed segments produced 67 bytes of a 67-byte stream: two were
# redundant. Counting packets tells you nothing about how much data moved.

print("stream contents:")
for line in data.decode().split("\r\n"):
    print(f"  | {line}")
# stream contents:
#   | GET /api/v1/orders HTTP/1.1
#   | Host: api.example.com
#   | Accept: */*
#   |
#   |

print("\nWhat this means when reading a capture:")
for point in (
    "stream byte offsets do NOT correspond to packet boundaries",
    "a duplicate is not an error: it is a retransmission doing its job",
    "an out-of-order segment is held, so the RECEIVER never sees a gap",
    "overlapping retransmissions are trimmed, not counted twice",
    "if a gap never fills, everything after it is invisible to the application",
):
    print(f"  - {point}")

print("\nThe last point produces 'the server never got my request':")
BROKEN = [(1, b"GET /api/v1/ord"),          # the segment at 16 is lost for good
          (53, b"Accept: */*\r\n\r\n"),
          (36, b"api.example.com\r\n")]
d2, n2, h2, _ = reassemble(BROKEN)
print(f"  delivered to the application: {len(d2)} bytes -> {d2!r}")
print(f"  held, undeliverable         : {len(h2)} segment(s) at {sorted(h2)}")
#   delivered to the application: 15 bytes -> b'GET /api/v1/ord'
#   held, undeliverable         : 2 segment(s) at [36, 53]
#
# The application sees a truncated request and times out, while the capture
# clearly shows the later bytes arriving. That is why "the packets are there"
# and "the server never got it" can BOTH be true at the same time.

# --- Find the conversation, then follow it ---
tshark -r trace.pcap -q -z conv,tcp | sort -k7 -rn | head -5
#  Identify the stream by address pair and volume, then note its index:
tshark -r trace.pcap -Y 'ip.addr==10.0.2.15 && tcp.port==5432' \
  -T fields -e tcp.stream | sort -u
#  42

# --- Follow it: the reassembled application exchange, both directions ---
tshark -r trace.pcap -q -z follow,tcp,ascii,42
#  ===================================================================
#  Follow: tcp,ascii
#  Node 0: 10.0.1.9:51234        <- client bytes
#  Node 1: 10.0.2.15:5432        <- server bytes, indented in the output
#  ...

# Extract ONE direction as raw bytes (how you recover a transferred file)
tshark -r trace.pcap -q -z follow,tcp,raw,42 | tail -n +7 | xxd -r -p > payload.bin
file payload.bin

# --- See the segment-level truth behind the stream ---
tshark -r trace.pcap -Y 'tcp.stream==42' -T fields \
  -e frame.number -e frame.time_relative -e ip.src -e tcp.seq -e tcp.nxtseq \
  -e tcp.len -e tcp.analysis.flags | head -20
#  Watch for: tcp.nxtseq of one packet not matching tcp.seq of the next from the
#  same sender. That discontinuity IS the gap, and it is what stalls the receiver.

# --- Find a gap that never filled: the stalled-application signature ---
tshark -r trace.pcap -Y 'tcp.analysis.lost_segment' -T fields \
  -e frame.number -e tcp.stream -e ip.src
tshark -r trace.pcap -Y 'tcp.analysis.zero_window || tcp.analysis.window_full' \
  -T fields -e frame.number -e tcp.stream -e ip.src
#  A lost segment followed by dup ACKs that never stop, and no further data,
#  is the pattern behind "it hung with no error".

# --- Reassembly must be ENABLED for application dissectors to work ---
tshark -r trace.pcap -o 'tcp.desegment_tcp_streams:TRUE' -Y 'http.request' \
  -T fields -e http.request.uri
#  With desegmentation off, an HTTP request split across two segments is not
#  recognised as HTTP at all, and the filter silently returns nothing.

# --- UDP has no reassembly: each datagram is independent ---
tshark -r trace.pcap -q -z follow,udp,ascii,3
#  Any apparent ordering here is the CAPTURE's, not a protocol guarantee.

Key takeaways

• TCP carries segments and presents a byte stream; following a stream is a reconstruction, and its hard cases are where bugs live.
• A receiver must handle out-of-order arrival (hold until the gap fills), duplicates (discard), and overlaps (trim and keep the new tail).
• A duplicate in a capture is not an error — it is a retransmission working.
• If a gap never fills, everything after it is invisible to the application, which is why "the packets are in the capture" and "the server never got it" can both be true.
• Stream byte offsets do not match packet boundaries: one message can span segments and one segment can hold several messages.
• Packet counts say nothing about data volume, because redundant segments carry no new bytes.
• Use tcp.stream == N as the handle for all analysis, and compare tcp.nxtseq against the next tcp.seq to locate gaps.
• TCP desegmentation must be enabled or application dissectors silently fail on split messages.
• UDP has no reassembly; apparent ordering is the capture's, not a guarantee.

🧪 Practice

1. Given segments arriving as seq 1 (20 bytes), seq 41 (20 bytes), seq 21 (20 bytes), state the order in which bytes reach the application and how many bytes are held at each step.
2. Explain what the application receives when a segment is lost permanently and later segments continue to arrive, and what the connection eventually does.
3. Write the commands that identify a stream index for a given address pair and then export one direction's payload to a file.
4. Interview-style: A capture plainly shows the full HTTP request arriving at the server, yet the server's logs show no request at all and the client timed out. (Hint: examine whether the segments are contiguous — look at what the receiver is required to do with data that arrives after a gap it never filled.)

Identifying Retransmissions and Resets

Theory

Retransmissions and resets are the two signals that most often reveal what went wrong, and both are routinely misread — retransmissions by being counted rather than classified, and resets by being blamed on the wrong party.

Retransmissions come in kinds, and the kind is the diagnosis. Counting them is nearly useless; classifying them is nearly sufficient.

A fast retransmit is triggered by three duplicate acknowledgements. The receiver got later data, so it keeps acknowledging the last in-order byte, and the sender infers that one specific segment was lost while the pipe is still flowing. Recovery costs roughly one round trip. Isolated loss handled this way is largely benign and a capture full of fast retransmits can still represent healthy throughput.

An RTO retransmit happens when no acknowledgement arrives at all and the retransmission timer expires. This is the expensive failure: the pipe has emptied, the sender restarts conservatively, and the minimum timeout is typically 200 milliseconds against a data centre round trip of well under one. One RTO can cost what hundreds of fast retransmits cost, which is why a capture with five RTOs is sicker than one with five hundred fast retransmits. Repeated RTOs with exponential backoff mean the path is not recovering.

A spurious retransmission is data resent that was never actually lost, usually because reordering (often from ECMP) made the sender believe otherwise. It wastes bandwidth and indicates a path problem rather than a loss problem, and treating it as loss leads to the wrong fix entirely.

Resets require reading the direction and the timing, because the same flag means five different things:

Pattern	Sent by	Meaning
Immediately after SYN	server	nothing listening on that port
Immediately, with an anomalous TTL	a middlebox	a policy device forged it; the server never saw the SYN
Mid-conversation	server	the application closed with unread data, or crashed
After a long idle period	either	a stateful device dropped the flow from its table
Mid-conversation, in bulk	load balancer	a backend removed without connection draining

The TTL clue is the most useful single trick here. A reset whose IP TTL differs markedly from the server's other packets did not come from the server — it was injected by a firewall or load balancer in the path. This distinguishes "the application rejected you" from "a middlebox rejected you", which are entirely different investigations, and it is visible in any capture at no extra cost.

The idle-timeout row deserves emphasis because it explains a whole family of mysterious failures. Firewalls and NAT devices keep per-flow state with a timeout, commonly a few minutes to an hour. A connection idle longer than that has its state removed, and the next packet on it is either dropped or answered with a reset. The classic symptoms are long-lived database connections that fail after exactly the same interval every time, and the fix is TCP keepalives shorter than the middlebox timeout rather than anything in the application.

Examples

# Classifying retransmissions: the TYPE tells you what went wrong and where.
EVENTS = [
    # (time_ms, seq, kind) -- observed segments and ACKs from a capture
    (0.0,   1000, "data"), (0.1, 2448, "data"), (0.2, 3896, "data"),
    (0.3,   5344, "data"),
    (40.2,  1000, "ack"),                       # ACK for the first segment
    (40.3,  2448, "dupack"), (40.4, 2448, "dupack"), (40.5, 2448, "dupack"),
    (40.6,  2448, "retrans"),                   # 3 dup ACKs -> FAST retransmit
    (240.0, 5344, "retrans"),                   # no ACK at all -> RTO
    (241.0, 5344, "retrans"),                   # still nothing -> backoff
]
MIN_RTO_MS, RTT_MS = 200, 40

def classify(events, rtt_ms=RTT_MS):
    out, dupacks, last_data = [], {}, {}
    for t, seq, kind in events:
        if kind == "data":
            last_data[seq] = t
        elif kind == "dupack":
            dupacks[seq] = dupacks.get(seq, 0) + 1
        elif kind == "retrans":
            gap = t - last_data.get(seq, t)
            if dupacks.get(seq, 0) >= 3:
                out.append((t, seq, "FAST RETRANSMIT",
                            f"{dupacks[seq]} dup ACKs, {gap:.1f} ms after send: "
                            f"ONE segment lost, pipe still flowing"))
            elif gap >= MIN_RTO_MS:
                out.append((t, seq, "RTO RETRANSMIT",
                            f"{gap:.1f} ms with no ACK: the pipe STALLED "
                            f"({gap/rtt_ms:.0f}x the RTT)"))
            else:
                out.append((t, seq, "SPURIOUS?",
                            f"only {gap:.1f} ms after send, under 1 RTT: "
                            f"probably reordering, not loss"))
    return out

print(f"{'time':>8s} {'seq':>7s} {'type':17s} interpretation")
for t, seq, kind, why in classify(EVENTS):
    print(f"{t:7.1f}m {seq:7d} {kind:17s} {why}")
#     time     seq type              interpretation
#    40.6m    2448 FAST RETRANSMIT   3 dup ACKs, 40.5 ms after send: ONE segment lost, pipe still flowing
#   240.0m    5344 RTO RETRANSMIT    239.7 ms with no ACK: the pipe STALLED (6x the RTT)
#   241.0m    5344 RTO RETRANSMIT    240.7 ms with no ACK: the pipe STALLED (6x the RTT)

print("\nWhy the distinction matters more than the retransmission COUNT:")
for kind, cost, meaning in (
    ("fast retransmit", "~1 RTT", "recovery works; isolated loss; usually benign"),
    ("RTO retransmit", "200 ms+", "the pipe emptied: the expensive failure"),
    ("spurious", "wasted bandwidth", "reordering (often ECMP), not loss at all"),
):
    print(f"  {kind:18s} costs {cost:16s} {meaning}")
print(f"\n  On this {RTT_MS} ms path, one RTO costs {MIN_RTO_MS/RTT_MS:.0f}x what a "
      f"fast retransmit costs.")
print("  A capture with 500 fast retransmits and 0 RTOs is healthier than one")
print("  with 5 RTOs, so counting retransmissions without typing them is useless.")
#   fast retransmit    costs ~1 RTT           recovery works; isolated loss; usually benign
#   RTO retransmit     costs 200 ms+          the pipe emptied: the expensive failure
#   spurious           costs wasted bandwidth reordering (often ECMP), not loss at all
#
#   On this 40 ms path, one RTO costs 5x what a fast retransmit costs.

print("\nRST: read the DIRECTION and the timing. They mean different things.")
for scenario, who, when, meaning in (
    ("connection refused", "server", "immediately after SYN",
     "nothing listening on that port"),
    ("firewall reject", "middlebox", "immediately, anomalous TTL",
     "a policy device forged it; the real server never saw the SYN"),
    ("application abort", "server", "mid-conversation",
     "the app closed with unread data, or crashed"),
    ("idle timeout", "either", "after a long quiet period",
     "a stateful device dropped the flow from its table"),
    ("load balancer drain", "LB", "mid-conversation, in bulk",
     "backend removed without connection draining"),
):
    print(f"  {scenario:20s} by {who:10s} {when:28s} {meaning}")
print("\n  The TTL clue is the useful one: a RST whose TTL differs markedly from")
print("  the server's other packets was forged by something in between, which")
print("  is an entirely different investigation from an application refusal.")

# The idle-timeout family, quantified: why failures land on round numbers.
print("\nMiddlebox idle timeouts vs TCP keepalive defaults:")
KEEPALIVE_DEFAULT_S = 7200          # Linux tcp_keepalive_time: 2 hours
for device, timeout_s in (("stateful firewall", 3600), ("cloud NAT gateway", 350),
                          ("load balancer", 60), ("carrier NAT", 120)):
    safe = timeout_s > KEEPALIVE_DEFAULT_S
    print(f"  {device:20s} idle timeout {timeout_s:5d}s vs keepalive "
          f"{KEEPALIVE_DEFAULT_S}s -> "
          f"{'safe' if safe else f'CONNECTION DIES; set keepalive < {timeout_s}s'}")
#   stateful firewall    idle timeout  3600s vs keepalive 7200s -> CONNECTION DIES; set keepalive < 3600s
#   cloud NAT gateway    idle timeout   350s vs keepalive 7200s -> CONNECTION DIES; set keepalive < 350s
#   load balancer        idle timeout    60s vs keepalive 7200s -> CONNECTION DIES; set keepalive < 60s
#   carrier NAT          idle timeout   120s vs keepalive 7200s -> CONNECTION DIES; set keepalive < 120s
#
# The default keepalive is longer than EVERY common middlebox timeout, which is
# why idle long-lived connections fail at suspiciously round intervals.

# --- Classify, do not count. This is the whole lesson in two commands. ---
tshark -r trace.pcap -q -z expert,note | grep -iE 'retrans|dup ack|zero window|reset'
tshark -r trace.pcap -Y 'tcp.analysis.retransmission' -T fields \
  -e frame.number -e tcp.stream -e tcp.seq -e tcp.time_delta | head
#  A tcp.time_delta of ~0.2s or more before the retransmission indicates an RTO.
#  A small delta after dup ACKs indicates a fast retransmit.

# Wireshark separates them; use its own classification rather than reinventing it.
for f in retransmission fast_retransmission spurious_retransmission \
         rto out_of_order duplicate_ack lost_segment zero_window; do
  printf '%-26s %s\n' "$f" "$(tshark -r trace.pcap -Y "tcp.analysis.$f" 2>/dev/null | wc -l)"
done
#  retransmission             1842
#  fast_retransmission        1794    <- benign: recovery is working
#  spurious_retransmission      36    <- reordering, not loss
#  rto                          12    <- THE problem: 12 stalls of 200 ms+
#  zero_window                   4    <- a receiver that stopped reading

# --- Resets: get the TTL, which tells you WHO actually sent it ---
tshark -r trace.pcap -Y 'tcp.flags.reset==1' -T fields \
  -e frame.number -e ip.src -e ip.ttl -e tcp.srcport -e tcp.dstport
#  1904  10.0.2.15  64   5432  51234     <- consistent with the server's other packets
#  1905  10.0.2.15  252  5432  51236     <- TTL 252: FORGED by a middlebox
# Compare against the peer's normal TTL:
tshark -r trace.pcap -Y 'ip.src==10.0.2.15 && tcp.flags.reset==0' \
  -T fields -e ip.ttl | sort -u

# --- The idle-timeout signature: measure the interval, it will be suspiciously round ---
tshark -r trace.pcap -Y 'tcp.flags.reset==1' -T fields -e frame.time_relative \
  -e tcp.stream -e tcp.time_delta
#  A tcp.time_delta close to 60, 120, 350 or 3600 seconds before the RST is a
#  middlebox state timeout, not an application decision.

# Fix it on the host, not in the application:
sysctl -w net.ipv4.tcp_keepalive_time=240     # must be BELOW the shortest timeout
sysctl -w net.ipv4.tcp_keepalive_intvl=30
sysctl -w net.ipv4.tcp_keepalive_probes=4
ss -tio | grep -A1 'keepalive'                # confirm it applies to the socket

# --- Is loss continuous or bursty? The shape changes the diagnosis. ---
tshark -r trace.pcap -q \
  -z io,stat,1,'COUNT(tcp.analysis.retransmission)tcp.analysis.retransmission'
#  Steady counts -> a saturated link. A single spike -> a reconvergence or a
#  microburst. Same total, entirely different causes and fixes.

Key takeaways

• Classify retransmissions, do not count them. The kind is the diagnosis.
• Fast retransmit (three duplicate ACKs) costs about one round trip and is usually benign; the pipe kept flowing.
• RTO retransmit means no ACK arrived and the timer expired: the pipe emptied, and it costs a 200 ms minimum. A few RTOs are worse than hundreds of fast retransmits.
• Spurious retransmissions indicate reordering, often from ECMP, not loss — and the fix is different.
• For resets, read the direction and timing: refused connection, forged by a middlebox, application abort, idle timeout, or load balancer drain.
• Check ip.ttl on resets. A TTL inconsistent with the peer's other packets proves a middlebox injected it.
• Middlebox idle timeouts are shorter than the default TCP keepalive of two hours, which is why long-lived idle connections die at suspiciously round intervals.
• Fix idle-timeout failures with keepalives shorter than the shortest middlebox timeout, not in the application.
• Plot retransmissions over time: continuous means saturation, a spike means an event.

🧪 Practice

1. A capture shows 3 duplicate ACKs followed 2 ms later by a retransmission. Classify it, state the recovery cost on a 30 ms path, and say whether it is concerning.
2. Two resets arrive from the same source address with TTLs of 61 and 250. Explain what you conclude and what you would investigate for each.
3. A database connection pool reports failures almost exactly every 350 seconds. Give the likely cause, the confirming measurement, and the fix.
4. Interview-style: A capture contains 12,000 retransmissions and the team concludes the network is badly broken, but throughput meets expectations. (Hint: both observations can be correct — ask what fraction are recovered within one round trip versus by timer expiry, and what each costs.)

Decrypting TLS in Captures

Theory

Nearly all traffic worth analysing is encrypted, which changes what packet analysis can do. The important thing to understand first is why the obvious approach no longer works, because it explains everything else.

Historically, TLS could use static RSA key exchange: the client encrypted a secret with the server's public key, so anyone holding the server's private key could decrypt a recorded session, even years later. Network analysis tools relied on this.

Modern TLS uses ephemeral Diffie-Hellman (DHE or ECDHE), which provides forward secrecy: the session keys are derived from ephemeral values discarded after the handshake, and the server's long-term private key authenticates but never encrypts the secret. Consequently the server's private key cannot decrypt a captured session. TLS 1.3 removed static RSA entirely, so there is no configuration in which it works. Someone offering to "just give you the private key" for analysis is describing a workflow that no longer exists.

What does work, and what each approach requires:

Method	Requires	Yields
SSLKEYLOGFILE	control of the client process	full decryption, per session
Server-side key log	control of the server and library support	full decryption, per session
TLS-terminating proxy	clients trust your CA	full visibility, but the path changed
Static RSA plus private key	an obsolete cipher suite	full decryption, retroactively
Nothing (ciphertext only)	nothing	metadata, which is often enough

The SSLKEYLOGFILE mechanism is the standard approach: set the environment variable and supporting TLS libraries append each session's secrets to that file, which Wireshark then uses. It works because you control an endpoint, and it is per-session rather than retroactive, so it must be running when the traffic is captured. Treat that file as equivalent to the plaintext, because it is.

The point most worth internalising, though, is that an encrypted capture is still highly diagnostic, and a great many engineers reach for decryption when they did not need it. The TLS handshake is largely plaintext, and it contains the fields that explain most TLS failures:

• SNI in the ClientHello: which hostname was requested.
• Offered and negotiated versions and cipher suites: whether client and server have anything in common.
• The certificate chain: wrong name, expired, missing intermediate.
• ALPN: whether HTTP/2, HTTP/3, or HTTP/1.1 was negotiated.
• Alert records: handshake_failure, bad_certificate, unknown_ca, with the reason.

Beyond the handshake, record sizes and timing reveal request and response shapes and server think time, and everything about TCP — retransmissions, resets, round-trip time, zero windows — remains fully visible because it is below the encryption.

The practical consequence is a rule of thumb: if the failure occurs during the handshake, decryption cannot help you, because no application data exists yet. Handshake failures are diagnosed from plaintext fields. Decryption is needed only when the connection succeeds and the content is the question.

Examples

# What is decryptable from a capture, and what is not. Forward secrecy decides.
SUITES = {
    # name:                                  (key exchange, forward secret,
    #                                          server key decrypts capture?)
    "TLS_RSA_WITH_AES_128_CBC_SHA":          ("static RSA", False, True),
    "TLS_DHE_RSA_WITH_AES_128_GCM_SHA256":   ("DHE",        True,  False),
    "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256": ("ECDHE",      True,  False),
    "TLS_AES_128_GCM_SHA256 (TLS 1.3)":      ("ECDHE only", True,  False),
}
print(f"{'cipher suite':40s} {'kex':12s} {'fwd secret':>11s} "
      f"{'server key decrypts?':>21s}")
for name, (kex, fs, decryptable) in SUITES.items():
    print(f"{name:40s} {kex:12s} {str(fs):>11s} {str(decryptable):>21s}")
# cipher suite                             kex           fwd secret  server key decrypts?
# TLS_RSA_WITH_AES_128_CBC_SHA             static RSA         False                  True
# TLS_DHE_RSA_WITH_AES_128_GCM_SHA256      DHE                 True                 False
# TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256    ECDHE               True                 False
# TLS_AES_128_GCM_SHA256 (TLS 1.3)         ECDHE only          True                 False
#
# Holding the server's private key decrypts a capture ONLY for static RSA, which
# TLS 1.3 removed entirely. "Just give me the private key" no longer works.

print("\nWhat you CAN get, and what each method requires:")
for m, what, needs, gets in (
    ("SSLKEYLOGFILE", "per-session secrets written by the CLIENT",
     "you control the client process", "everything, per session"),
    ("server-side key log", "secrets written by the SERVER's TLS library",
     "you control the server and it supports it", "everything, per session"),
    ("TLS-terminating proxy", "decrypt and re-encrypt in the middle",
     "clients trust your CA", "everything, but you changed the path"),
    ("static RSA + private key", "derive the master secret from the key exchange",
     "an obsolete cipher suite", "everything, retroactively"),
    ("nothing (ciphertext only)", "metadata analysis",
     "nothing", "SNI, sizes, timing, certificate, ALPN"),
):
    print(f"  {m:26s} {what}")
    print(f"  {'':26s} requires: {needs}")
    print(f"  {'':26s} yields  : {gets}")

print("\nWhat an ENCRYPTED capture still reveals -- often enough to diagnose:")
for item, why in (
    ("ClientHello SNI", "which hostname was requested (plaintext before ECH)"),
    ("offered/negotiated version", "do client and server have anything in common?"),
    ("certificate chain", "wrong name, expired, missing intermediate"),
    ("ALPN", "did it negotiate h2, h3 or http/1.1?"),
    ("alert records", "handshake_failure, bad_certificate, unknown_ca"),
    ("record sizes and timing", "request/response shapes, server think time"),
    ("TCP behaviour", "retransmissions, resets, RTT -- all still visible"),
):
    print(f"  {item:28s} {why}")

print("\nA worked example: diagnosing a handshake failure WITHOUT decryption.")
for rec, detail in (
    ("ClientHello", "SNI=api.example.com, versions=[TLS1.2, TLS1.3], ALPN=[h2, http/1.1]"),
    ("ServerHello", "version=TLS1.2, suite=ECDHE_RSA_AES128_GCM"),
    ("Certificate", "CN=api.internal.example.com, SAN=[api.internal.example.com]"),
    ("Alert",       "fatal, bad_certificate (42)"),
):
    print(f"  {rec:14s} {detail}")
print("\n  Diagnosis from plaintext fields alone: the SNI requested")
print("  api.example.com and the certificate covers only api.internal.example.com,")
print("  so the client rejected it. Decryption was not needed -- and would not")
print("  have helped, because the failure happened BEFORE any application data.")
print("\n  The rule: a handshake failure is diagnosed from plaintext. Decryption")
print("  is needed only when the connection SUCCEEDS and the content is the question.")

# --- The standard method: SSLKEYLOGFILE on a client you control ---
export SSLKEYLOGFILE=/tmp/keys.log
curl -sS https://api.example.com/health > /dev/null       # curl with OpenSSL/NSS
# Browsers honour the same variable when launched with it set.
head -2 /tmp/keys.log
#  CLIENT_HANDSHAKE_TRAFFIC_SECRET 3a7f... 9b2c...
#  SERVER_HANDSHAKE_TRAFFIC_SECRET 3a7f... 41de...
# TREAT THIS FILE AS PLAINTEXT. It is equivalent to the decrypted data.

# Capture and decrypt together
tcpdump -i eth0 -nn -s0 -w tls.pcap 'host api.example.com and tcp port 443' &
SSLKEYLOGFILE=/tmp/keys.log curl -sS https://api.example.com/health > /dev/null
kill %1
tshark -r tls.pcap -o tls.keylog_file:/tmp/keys.log -Y 'http2 || http' \
  -T fields -e http2.headers.path -e http2.headers.status
#  /health   200      <- decrypted application layer

# --- Before reaching for keys, get what the handshake gives you for free ---
tshark -r tls.pcap -Y 'tls.handshake.type==1' -T fields \
  -e tls.handshake.extensions_server_name \
  -e tls.handshake.version -e tls.handshake.ciphersuite
#  api.example.com  0x0303  ...     <- the SNI actually sent

tshark -r tls.pcap -Y 'tls.handshake.type==2' -T fields \
  -e tls.handshake.version -e tls.handshake.ciphersuite   # what was CHOSEN
tshark -r tls.pcap -Y 'tls.handshake.type==11' -T fields \
  -e x509sat.uTF8String -e x509ce.dNSName                 # the cert's names
tshark -r tls.pcap -Y 'tls.alert_message' -T fields \
  -e ip.src -e tls.alert_message.level -e tls.alert_message.desc
#  10.0.1.9  2  42     <- fatal bad_certificate, sent by the CLIENT:
#                         the client rejected the server's certificate

# ALPN, which explains "why is it not using HTTP/2?"
tshark -r tls.pcap -T fields -e tls.handshake.extensions_alpn_str | sort -u

# --- Even fully encrypted, the shape of the conversation is informative ---
tshark -r tls.pcap -Y 'tls.record.content_type==23' -T fields \
  -e frame.time_relative -e ip.src -e tls.record.length | head -20
#  Request/response sizes and the GAP between them is server think time,
#  measurable without decrypting anything.

# --- Confirm forward secrecy is in use (and thus that keys are required) ---
openssl s_client -connect api.example.com:443 -servername api.example.com \
  </dev/null 2>/dev/null | grep -E 'Protocol|Cipher'
#  Protocol  : TLSv1.3
#  Cipher    : TLS_AES_256_GCM_SHA384    <- ECDHE: no private-key decryption

Key takeaways

• Modern TLS uses ephemeral key exchange and therefore has forward secrecy, so the server's private key cannot decrypt a captured session. TLS 1.3 removed static RSA entirely.
• The standard method is SSLKEYLOGFILE on a client you control; it is per-session and must be running during the capture.
• Treat a key log file as equivalent to plaintext.
• Other options are a server-side key log, or a TLS-terminating proxy, which changes the path.
• An encrypted capture is still highly diagnostic: SNI, offered and negotiated versions and suites, certificate chain, ALPN, and alert records are all plaintext.
• TCP behaviour — retransmissions, resets, round-trip time, zero windows — is entirely visible because it sits below encryption.
• If the failure is in the handshake, decryption cannot help, because no application data exists yet. Diagnose it from the plaintext fields.
• Record sizes and inter-record gaps reveal request and response shapes and server think time without any decryption.

🧪 Practice

1. Explain why a server's private key decrypts a TLS_RSA_WITH_AES_128_CBC_SHA session but not a TLS_ECDHE_RSA_... one.
2. Write the commands that capture and decrypt a single HTTPS request using a key log file, and state the security precaution that file requires.
3. List five things you can determine about a TLS connection without any decryption, and name a failure each one would explain.
4. Interview-style: A team requests the production server's private key so they can decrypt a capture and diagnose failing TLS handshakes. (Hint: two separate reasons make this the wrong request — one about the key exchange in use, and one about what stage the failure occurs at.)

<a id="143-systematic-troubleshooting"></a>

14.3 Systematic Troubleshooting

Tools produce observations; a methodology decides which observation to take next, and that choice is what separates a twenty-minute diagnosis from a four-hour one: this subchapter covers the standard approaches and when each applies, the isolation technique that shrinks the search space fastest, the specific faults that recur at each layer, how to triage a performance complaint, and how to write up an incident so that the next person does not start over.

Bottom-Up and Top-Down Methodologies

Theory

Troubleshooting without a method is guessing, and guessing has a characteristic failure mode: you check the things you know how to check rather than the things that would be informative. A methodology is simply a predetermined order of tests, and its value is that it guarantees coverage and prevents the same ground being checked three times while something obvious goes unexamined.

Bottom-up starts at the physical layer and works upward: is there link, is the VLAN right, is there an IP address, is the route present, is the port open, does the application respond? Its strength is completeness — you cannot skip a layer — and its weakness is that it is slow when the lower layers were never in doubt. It is the right choice when nothing has ever worked: a new installation, a hardware replacement, a fresh circuit. In those cases the foundation genuinely is suspect.

Top-down starts at the application and works downward: what does the application report, is DNS resolving, does TLS complete, is the port reachable, is there a route? Its strength is speed when the lower layers are already proving themselves by carrying other traffic. It is the right choice when it worked five minutes ago and only one application is affected: the cable did not change, and a cable that carries other traffic is not the problem.

That last point generalises into the most useful heuristic available: evidence beats probability. A cost model can tell you which layer is statistically likeliest to be at fault, and it is worth computing once to calibrate intuition, but it is superseded the moment you have a real observation. A red link light makes layer one certain, not fifteen percent likely. Other users working normally on the same switch eliminates layers one and two without a single command.

Three other strategies are worth naming because they are frequently the fastest:

Follow the path. When one flow fails and others work, enumerate the hops that flow traverses and check each. This is the method that catches asymmetric routing and per-flow policy.

Compare working against broken. When one user, site, or request fails and comparable peers succeed, the difference between them is the fault. This often finds the answer in a single step, and it is the cheapest method available, yet it is the one people skip most often because it requires finding a working case rather than staring at the broken one.

Move in the direction the symptom points. If the symptom is a TLS alert, start at TLS. If it is an MTU-shaped failure, start at MTU. Starting at layer one because that is the formal procedure wastes the information the symptom already gave you.

The question worth asking before every test is not "what should I check next?" but "what is the smallest observation that would eliminate half of the possibilities?" — and then taking that measurement even when it is less convenient than an easier one that eliminates nothing.

Examples

Three orders of testing, one problem

  BOTTOM-UP                TOP-DOWN                 SYMPTOM-DIRECTED
  L1 link/cable/errors     L7 app logs              symptom: TLS alert
  L2 VLAN/MAC/duplex       L4 port reachable?          |
  L3 IP/route/ACL          L3 route present?           v
  L4 port/firewall/TLS     L2 VLAN correct?         start at TLS: read the
  L7 application           L1 link up?              handshake, then widen
  ^ complete, slow         ^ fast when lower         ^ fastest when the symptom
    good for NEW installs    layers carry other        is specific
                             traffic fine

  What eliminates whole layers for free, before any command:

   other users on the same switch work  -> L1 and L2 for that switch are fine
   other apps to the same server work   -> L1-L3 to that server are fine
   the same app works from another site -> the server and app are fine
   it worked until 14:02               -> ask what changed at 14:02, not what is broken

# Choosing a methodology is choosing the order of tests. Order decides cost.
LAYERS = [
    # layer, name,       check,                                 minutes, p(fault)
    (1, "physical",  "link light, cable, optic levels, errors",  2, 0.15),
    (2, "data link", "VLAN, MAC table, duplex, STP, ARP",        5, 0.10),
    (3, "network",   "IP, mask, gateway, route, ACL",            8, 0.20),
    (4, "transport", "port open, firewall, TLS",                 6, 0.20),
    (7, "app",       "logs, config, dependencies, DNS",         20, 0.35),
]

def expected_cost(order, layers):
    """Expected minutes to LOCATE the fault, testing layers in the given order."""
    by_layer = {l[0]: l for l in layers}
    cost, cumulative = 0.0, 0.0
    for layer in order:
        _, _, _, minutes, p = by_layer[layer]
        cumulative += minutes
        cost += p * cumulative          # if the fault is here, we paid `cumulative`
    return cost

BOTTOM_UP, TOP_DOWN = [1, 2, 3, 4, 7], [7, 4, 3, 2, 1]
LIKELIEST = sorted((l[0] for l in LAYERS),
                   key=lambda n: -[l for l in LAYERS if l[0] == n][0][4])

print(f"{'layer':>5s} {'name':11s} {'check':42s} {'min':>4s} {'p(fault)':>9s}")
for n, name, check, minutes, p in LAYERS:
    print(f"{n:5d} {name:11s} {check:42s} {minutes:4d} {p:8.0%}")
print(f"\n{'strategy':22s} {'order':22s} expected minutes to locate")
for label, order in (("bottom-up", BOTTOM_UP), ("top-down", TOP_DOWN),
                     ("likeliest first", LIKELIEST)):
    print(f"{label:22s} {str(order):22s} {expected_cost(order, LAYERS):22.1f}")
# layer name        check                                       min  p(fault)
#     1 physical    link light, cable, optic levels, errors       2      15%
#     2 data link   VLAN, MAC table, duplex, STP, ARP             5      10%
#     3 network     IP, mask, gateway, route, ACL                 8      20%
#     4 transport   port open, firewall, TLS                      6      20%
#     7 app         logs, config, dependencies, DNS              20      35%
#
# strategy               order                  expected minutes to locate
# bottom-up              [1, 2, 3, 4, 7]                          22.5
# top-down               [7, 4, 3, 2, 1]                          29.0
# likeliest first        [7, 3, 4, 1, 2]                          28.9
#
# Bottom-up wins here NOT because layer 1 is likely, but because its checks are
# CHEAP: 2 and 5 minutes versus 20 for the application. Cheap tests first is a
# better rule than likely tests first, when the costs differ this much.

print("\nBut all of that is superseded by EVIDENCE:")
for observation, eliminates in (
    ("other users on this switch work", "layers 1-2 for that switch"),
    ("other apps to this server work", "layers 1-3 to that server"),
    ("the app works from another site", "the server and the application"),
    ("it worked until 14:02", "everything that did not change at 14:02"),
    ("the link light is red", "nothing: layer 1 is now CERTAIN, not 15%"),
):
    print(f"  {observation:34s} -> eliminates {eliminates}")

print("\nWhen each methodology is the right choice:")
for method, use_when, why in (
    ("bottom-up", "a new install, or a hardware change",
     "nothing has ever worked; the foundation is genuinely suspect"),
    ("top-down", "it worked minutes ago and one app is affected",
     "lower layers carry other traffic fine, so they are already proven"),
    ("follow the path", "one flow fails and others work",
     "walk the specific hops that flow traverses"),
    ("compare working/broken", "one user or site fails and peers do not",
     "the DIFFERENCE is the fault; nothing else needs examining"),
    ("symptom-directed", "the symptom names a layer",
     "starting elsewhere discards information you already have"),
):
    print(f"  {method:22s} use when {use_when}")
    print(f"  {'':22s} because  {why}")
print("\nBefore every test, ask: 'what is the smallest observation that would")
print("eliminate half the possibilities?' -- then take THAT measurement.")

# --- The free eliminations. Run these before touching anything. ---
# Does anything else on the same switch work?
ping -c2 10.0.1.8 && echo "L1/L2 on this segment: PROVEN GOOD"
# Does anything else on the same server work?
nc -zv 10.0.2.15 22 && echo "L1-L3 to this server: PROVEN GOOD"
# Did it work before? Find the change, not the fault.
last -n 20 | head; journalctl --since '20 min ago' -p warning --no-pager | tail -20
git -C /etc/network log --since='2 hours ago' --oneline 2>/dev/null

# --- BOTTOM-UP: for a new install or after a hardware change ---
ip -s link show eth0 | head -5                 # L1: up? errors? carrier changes?
ethtool eth0 | grep -E 'Speed|Duplex|Link detected'
bridge vlan show dev eth0 2>/dev/null          # L2: correct VLAN?
ip neigh show dev eth0                         # L2: is ARP resolving?
ip addr show eth0; ip route get 10.0.2.15      # L3: address and route
nc -zv 10.0.2.15 5432                          # L4: is the port reachable?
curl -sS -m 5 http://10.0.2.15:8080/health     # L7: does the app answer?

# --- TOP-DOWN: for "it worked five minutes ago" ---
journalctl -u myapp --since '15 min ago' -p err --no-pager | tail
getent hosts api.example.com                   # does the APP's resolution work?
curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' https://api.example.com/health
nc -zv api.example.com 443
ip route get "$(dig +short api.example.com | head -1)"

# --- COMPARE working against broken: usually the fastest method ---
# Same command, two subjects. The difference IS the fault.
for h in broken-host working-host; do
  echo "=== $h ==="
  ssh $h 'ip addr show eth0 | grep inet; ip route show default; \
          grep -H nameserver /etc/resolv.conf; ip -s link show eth0 | grep -A1 RX'
done | diff <(sed -n '2,20p') /dev/null || true
# In practice: run it, capture both outputs, and diff them.
ssh broken-host 'ip route show' > /tmp/broken.txt
ssh working-host 'ip route show' > /tmp/working.txt
diff /tmp/working.txt /tmp/broken.txt
#  < 10.20.0.0/16 via 10.0.0.2 dev eth0      <- the broken host is MISSING this route

Key takeaways

• A methodology is a predetermined order of tests; its value is guaranteed coverage and no repeated ground.
• Bottom-up is complete and suits a new install or hardware change, where the foundation is genuinely suspect.
• Top-down is faster when it worked recently and the lower layers are proving themselves by carrying other traffic.
• Cheap tests first beats likely tests first when test costs differ by an order of magnitude.
• Evidence beats probability. Other users working on the same switch eliminates two layers with no commands at all.
• Follow the path for a single failing flow, and compare working against broken when a peer succeeds — the difference is the fault.
• Let the symptom direct the starting layer; starting at layer one on principle discards information you already have.
• Before each test ask which single observation eliminates half the possibilities, and take that one.
• When something worked until a specific moment, look for the change, not the fault.

🧪 Practice

1. For each situation, choose a methodology and justify it in one sentence: a newly installed branch circuit that has never carried traffic; one application failing while others on the same server work; every user at one site affected simultaneously.
2. List four observations that eliminate entire layers without running a diagnostic command.
3. A user reports a problem that began at 09:14. Write the first three commands you would run and say what each would rule in or out.
4. Interview-style: An engineer spends two hours checking cables, optics, and switch counters on a link that is simultaneously carrying several other applications without complaint. (Hint: identify what those working applications already prove about the lower layers, and which methodology the evidence called for.)

Divide-and-Conquer Isolation

Theory

The methodologies above choose a direction; divide and conquer chooses step size. The insight is that testing the midpoint of a range halves the remaining search space, so the number of tests grows with the logarithm of the path length rather than linearly. An eleven-element path needs at most four bisecting tests instead of ten sequential ones, and the advantage grows with scale.

Bisection requires two things: an ordered space, and a test you can perform at an arbitrary point within it. The path is the obvious ordered space, and the usable midpoints are the places you can actually run a command: your own host, the default gateway, a jump host partway along, the far-side edge device, and the destination itself. One honest caveat is that bisection is worse than walking when the fault is very close to your starting point, so for a three-hop path there is nothing to gain.

The more valuable realisation is that the path is only one of the axes you can bisect along, and it is often not the most productive one. Each of these halves the space:

Axis	The question
Path	where between the endpoints does it break?
Time	when did it start, and what changed then?
Scope	which users, sites, or flows are affected, and which are not?
Layer	does a ping work? a TCP connect? a full request?
Payload	does a small packet work where a large one fails?
Direction	does it fail outbound, inbound, or both?

Scope is usually the cheapest and most decisive. Establishing that exactly one site is affected, or exactly one user, or only flows above a certain size, eliminates most of the system in a single question and often points directly at the cause. The payload axis deserves specific mention because it is the fingerprint of an MTU problem: small packets succeeding while large ones fail is diagnostic on its own, and nothing else produces that pattern.

The time axis is worth treating as a first-class technique rather than an afterthought. If something worked until 14:02, the productive question is not "what is broken?" but "what changed at 14:02?" — a deployment, a configuration push, a certificate expiry, a scheduled job, a route withdrawal. Change correlation solves a large fraction of incidents faster than any packet capture, which is why the change log is a diagnostic tool.

One discipline makes all of this work: change one variable at a time, and write down the result. Under pressure the temptation is to alter several things at once, and if the symptom then changes you have learned nothing about which alteration mattered. The corollary is to undo failed changes before trying the next one, or the system drifts into a state nobody can reason about — and which will still be there after the incident.

Examples

Bisecting a path: 4 tests instead of 10

  client -- access -- dist -- fw -- core -- wan -- provider -- remote -- server
     0        1        2       3      4      5        6          7        8

  test 1: can I reach the MIDPOINT (core, 4)?      yes -> fault is in 4..8
  test 2: can I reach 6 (provider)?                yes -> fault is in 6..8
  test 3: can I reach 7 (remote)?                  NO  -> fault is in 6..7
  -> the break is between provider and remote. 3 tests, 9 hops eliminated.

  Bisecting along OTHER axes is often cheaper than along the path:

   SCOPE     all sites? one site? one user?        one question, huge elimination
   TIME      started 14:02 -> what changed then?   a change log beats a capture
   LAYER     ping ok, TCP ok, TLS fails            names the layer immediately
   PAYLOAD   64 B ok, 1500 B fails                 MTU, and nothing else
   DIRECTION outbound ok, inbound fails            asymmetric routing or policy

import math
# Divide and conquer: why bisection beats walking the path.
PATH = ["client", "access-sw", "dist-sw", "fw", "core-rtr", "wan-edge",
        "provider", "remote-edge", "remote-core", "server-sw", "server"]

def linear_search(path, fault_index):
    """Walk hop by hop from one end."""
    return fault_index + 1

def bisect_search(path, fault_index):
    """Test the midpoint, then halve the remaining span."""
    lo, hi, tests = 0, len(path) - 1, 0
    while hi - lo > 1:
        mid = (lo + hi) // 2
        tests += 1
        if mid <= fault_index:      # reachable up to mid -> the break is beyond it
            lo = mid
        else:
            hi = mid
    return tests

print(f"path of {len(PATH)} elements\n")
print(f"{'fault located at':22s} {'linear tests':>13s} {'bisect tests':>13s} {'saving':>8s}")
for idx in (1, 3, 5, 7, 9):
    lin, bis = linear_search(PATH, idx), bisect_search(PATH, idx)
    print(f"{PATH[idx]:22s} {lin:13d} {bis:13d} {lin-bis:8d}")
print(f"\nworst case: linear {len(PATH)-1}, bisection "
      f"{math.ceil(math.log2(len(PATH)-1))} = ceil(log2({len(PATH)-1}))")
# fault located at        linear tests  bisect tests   saving
# access-sw                          2             3       -1
# fw                                 4             4        0
# wan-edge                           6             3        3
# remote-edge                        8             3        5
# server-sw                         10             4        6
#
# worst case: linear 10, bisection 4 = ceil(log2(10))
#
# Note the FIRST row is negative: bisection is WORSE when the fault is right next
# to you. For a three-hop path, just walk it. Bisection pays off with distance.

print("\nBisection needs a test you can run AT a midpoint. The usable midpoints:")
for point, test in (
    ("your own host", "ping the next hop, then the gateway, then beyond"),
    ("the default gateway", "can it reach the far side? ask it, or ping from it"),
    ("a jump host mid-path", "run the same test from there: the answer bisects"),
    ("the far-side edge device", "does traffic arrive at all? check counters"),
    ("the destination host", "does it see the SYN? tcpdump answers definitively"),
):
    print(f"  {point:24s} {test}")

print("\nThe most powerful bisections are not along the PATH:")
for axis, question in (
    ("path",      "where between the endpoints does it break?"),
    ("time",      "when did it start? what changed at that moment?"),
    ("scope",     "which users/sites/flows are affected, and which are not?"),
    ("layer",     "does a ping work? a TCP connect? a request?"),
    ("payload",   "does a small packet work where a large one fails? (MTU)"),
    ("direction", "does it fail outbound, inbound, or both?"),
):
    print(f"  {axis:12s} {question}")
print("\n'Scope' is usually cheapest: comparing one broken case against one")
print("working case often finds the fault in a single step, because the")
print("DIFFERENCE between them IS the fault.")

# --- Bisect along the PATH: test progressively further, stop where it fails ---
for hop in 10.0.1.1 10.0.0.1 10.0.255.1 203.0.113.1 10.20.0.1 10.20.5.9; do
  if ping -c2 -W2 -q "$hop" >/dev/null 2>&1; then
    echo "OK    $hop"
  else
    echo "FAIL  $hop   <- the break is between here and the previous OK"
    break
  fi
done

# --- Bisect along LAYER: each step is a different layer, in order ---
H=api.example.com
ping -c2 -W2 -q $H          >/dev/null 2>&1 && echo "L3 ok"  || echo "L3 FAIL (or ICMP filtered)"
nc -zv -w3 $H 443           >/dev/null 2>&1 && echo "L4 ok"  || echo "L4 FAIL"
openssl s_client -connect $H:443 -servername $H </dev/null >/dev/null 2>&1 \
  && echo "TLS ok" || echo "TLS FAIL"
curl -sS -m5 -o /dev/null https://$H/health && echo "L7 ok" || echo "L7 FAIL"
#  The first FAIL names the layer. Everything below it is proven good.

# --- Bisect along PAYLOAD: the MTU fingerprint ---
for size in 64 500 1000 1400 1472; do
  printf '%5d B: ' $size
  ping -M do -s $size -c2 -W2 -q $H >/dev/null 2>&1 && echo ok || echo FAIL
done
#    64 B: ok
#   500 B: ok
#  1000 B: ok
#  1400 B: ok
#  1472 B: FAIL      <- small works, large fails: MTU, and nothing else
# Then find the exact value:
ping -M do -s 1422 -c2 $H && echo "path MTU is 1450"
tracepath $H

# --- Bisect along TIME: find the change, not the fault ---
journalctl --since '14:00' --until '14:10' -p warning --no-pager
git -C /etc log --since='today' --oneline
# Cloud/orchestrator audit logs are the highest-yield source here:
kubectl get events --sort-by=.lastTimestamp -A | tail -20
aws cloudtrail lookup-events --start-time "$(date -u -d '1 hour ago' +%FT%TZ)" \
  --max-results 20 --query 'Events[].[EventTime,EventName,Username]' --output table

# --- Bisect along SCOPE: the cheapest and most decisive question ---
for site in site-a site-b site-c; do
  printf '%-8s ' $site
  ssh $site "curl -sS -m5 -o /dev/null -w '%{http_code}\n' https://$H/health" \
    2>/dev/null || echo UNREACHABLE
done
#  site-a   200
#  site-b   200
#  site-c   UNREACHABLE    <- one site: stop looking at the server entirely

# --- Bisect along DIRECTION ---
# Outbound from here:
nc -zv -w3 $H 443
# Inbound to here, tested from there (asymmetric policy is common):
ssh $H "nc -zv -w3 $(curl -s ifconfig.me) 443"

# --- The discipline: one variable at a time, and UNDO what did not work ---
cp /etc/myapp/config.yaml /tmp/config.yaml.before   # always, before any change
# make exactly ONE change, test, record the result
diff /tmp/config.yaml.before /etc/myapp/config.yaml
# if it did not help: restore it before trying the next thing
cp /tmp/config.yaml.before /etc/myapp/config.yaml

Key takeaways

• Bisection makes test count grow with the logarithm of the search space: four tests for an eleven-hop path instead of ten.
• It needs an ordered space and a test you can run at a midpoint; usable midpoints are your host, the gateway, a jump host, the far edge, and the destination.
• Bisection is worse than walking when the fault is close to your starting point, so short paths do not need it.
• The path is only one axis. Time, scope, layer, payload, and direction all bisect.
• Scope is usually cheapest and most decisive: establishing that exactly one site or user is affected eliminates most of the system in one question.
• Small packets working while large ones fail is the MTU fingerprint and nothing else produces it.
• On the time axis, ask what changed at the moment it broke; change correlation resolves many incidents faster than a capture.
• Change one variable at a time, record the result, and undo failed changes — otherwise the system drifts into a state nobody can reason about, and that state outlives the incident.

🧪 Practice

1. For a path of 17 elements, compute the worst-case test count for linear walking and for bisection, and state at which path length bisection stops being worth the bookkeeping.
2. Write the command sequence that bisects along the payload axis to find a path MTU, and state the exact output pattern that confirms an MTU fault.
3. A service fails for one of three sites. List the questions you would ask along the scope, time, and direction axes, in the order you would ask them.
4. Interview-style: During an outage an engineer changes the MTU, restarts the firewall, and updates a route within two minutes, and the symptom disappears. (Hint: the incident is over but something important was not achieved — consider what you now know about the cause, and what remains in the configuration.)

Common Layer 1 and Layer 2 Faults

Theory

Lower-layer faults have a useful property: they leave counters, and the counters are specific enough to be a diagnosis rather than a hint. Learning which counter means what converts a vague "the network is flaky" into "replace the optic in port 14".

The most important distinction is between errors and drops, because they lead to opposite actions:

• An error means a frame arrived corrupted or malformed. Something is physically broken: a cable, a connector, an optic, a port, or interference. The fix is replacement.
• A drop means a frame was discarded because a queue was full. Nothing is broken; there is not enough capacity or the traffic is too bursty. The fix is capacity, shaping, or queue management.

Treating a drop as an error sends people replacing healthy hardware, and treating an error as congestion sends them buying bandwidth that will not help.

The counters worth knowing:

Counter	Layer	Points at
CRC / FCS errors	1	bad cable, bad optic, interference, failing port
Runts (under 64 bytes)	1	collisions, or a failing NIC
Giants (over MTU)	2	MTU mismatch, or unexpected tagging
Late collisions	2	duplex mismatch, or an over-long segment
Collisions at all	2	half duplex in use; should be zero on a modern switch
Output drops	2	egress congestion — a capacity problem, not an error
Carrier transitions	1	link flapping: cable, optic, or the far end
Pause frames	2	flow control active (see Chapter 13 on PFC)

Two points about reading them correctly.

A count without a denominator is meaningless. Twelve hundred CRC errors against eight billion frames is a historical artefact; twelve hundred against forty thousand frames is a cable that must be replaced today. Always clear the counters, wait a known interval, and read the delta as a rate.

A duplex mismatch has a two-sided signature, and neither side alone is conclusive. One end forced to full duplex shows CRC errors and runts on input; the other end running half duplex shows late collisions on output. The pair together is diagnostic. The reason this fault survives for months is that it never fails outright — it degrades a link to roughly a third of its rate with heavy retransmission, which users report as "sometimes slow" rather than as an outage.

Finally, optical power levels are the best leading indicator on any fibre link. Receive power drifting toward the low end of the acceptable range predicts CRC errors weeks before they appear, usually because of a dirty connector, a tight bend, or a laser reaching end of life. Monitoring transceiver diagnostics is one of the highest-yield things a monitoring system can do, because it converts an outage into scheduled maintenance.

Examples

# Interface counters are a DIAGNOSIS, if you know which counter means what.
COUNTERS = {
    # counter:             (points at, layer, note)
    "CRC / FCS errors":    ("bad cable, bad optic, EMI, failing port", 1,
                            "the frame arrived corrupted: PHYSICAL"),
    "runts":               ("frames under 64 B: collisions or a bad NIC", 1, ""),
    "giants":              ("frames over MTU: MTU mismatch or tagging", 2, ""),
    "late collisions":     ("DUPLEX MISMATCH, or a segment too long", 2,
                            "the definitive duplex-mismatch signature"),
    "collisions":          ("half duplex in use at all", 2,
                            "should be zero on a modern switch port"),
    "output drops":        ("egress congestion: the queue overflowed", 2,
                            "a CAPACITY problem, not an error"),
    "carrier transitions": ("link flapping: cable, optic, or the far end", 1,
                            "count them over time, not once"),
    "pause frames":        ("flow control active: the far end asked us to stop", 2,
                            "see Chapter 13 on PFC"),
}
print(f"{'counter':22s} {'L':>2s} {'points at':44s} note")
for c, (points, layer, note) in COUNTERS.items():
    print(f"{c:22s} {layer:2d} {points:44s} {note}")

print("\nThe distinction that matters most: ERRORS vs DROPS")
print("  errors -> something is BROKEN (cable, optic, port). Replace it.")
print("  drops  -> something is FULL (queue, buffer). Add capacity or shape.")

def error_rate(errors, total_frames):
    return errors / total_frames if total_frames else 0.0

print("\nWhy the RATE is the only meaningful form:")
print(f"{'errors':>10s} {'frames':>14s} {'rate':>12s}  verdict")
for errors, frames in ((1_204, 8_400_000_000), (1_204, 40_000),
                       (98, 1_000_000), (0, 8_400_000_000)):
    r = error_rate(errors, frames)
    v = ("clean" if r == 0 else
         "negligible: historical, probably a past event" if r < 1e-9 else
         "marginal: watch it, check the optic" if r < 1e-6 else
         "BAD: replace the cable or optic")
    print(f"{errors:10,} {frames:14,} {r:12.2e}  {v}")
#     errors         frames         rate  verdict
#      1,204  8,400,000,000     1.43e-07  marginal: watch it, check the optic
#      1,204         40,000     3.01e-02  BAD: replace the cable or optic
#         98      1,000,000     9.80e-05  BAD: replace the cable or optic
#          0  8,400,000,000     0.00e+00  clean
#
# The first two rows have the SAME error count and opposite conclusions. Always
# clear counters, wait a known interval, and read the DELTA as a rate.

print("\nDuplex mismatch: the signature, and why it looks like 'slow but working'")
for side, setting, symptom in (
    ("switch", "100/full (forced)", "FCS errors and runts on input"),
    ("host",   "100/half (autoneg fell back)", "late collisions on output"),
):
    print(f"  {side:8s} {setting:30s} {symptom}")
print("  The PAIR of symptoms is diagnostic; neither alone is.")
for rate in (100, 1000):
    print(f"  a {rate} Mb/s link with a mismatch delivers roughly "
          f"{rate*0.35:.0f} Mb/s with heavy retransmission")
print("  It never fails outright, which is why it survives for months as")
print("  'sometimes slow' rather than being reported as an outage.")
#   switch   100/full (forced)              FCS errors and runts on input
#   host     100/half (autoneg fell back)   late collisions on output
#   a 100 Mb/s link with a mismatch delivers roughly 35 Mb/s with heavy retransmission
#   a 1000 Mb/s link with a mismatch delivers roughly 350 Mb/s with heavy retransmission

print("\nOptical levels: the check that finds a failing link BEFORE it errors.")
for label, rx_dbm, low, high in (("healthy", -6.2, -14.0, -1.0),
                                 ("marginal", -13.1, -14.0, -1.0),
                                 ("failing", -17.4, -14.0, -1.0),
                                 ("overdriven", 1.8, -14.0, -1.0)):
    if rx_dbm < low:
        status = "TOO LOW: dirty connector, bend, or failing laser"
    elif rx_dbm > high:
        status = "TOO HIGH: needs an attenuator"
    elif rx_dbm < low + 2.0:                 # within 2 dB of the floor
        status = "MARGINAL: inside spec, but trending toward errors"
    else:
        status = "OK"
    print(f"  {label:11s} Rx {rx_dbm:6.1f} dBm  (range {low} to {high})  {status}")
#   healthy     Rx   -6.2 dBm  (range -14.0 to -1.0)  OK
#   marginal    Rx  -13.1 dBm  (range -14.0 to -1.0)  MARGINAL: inside spec, but trending toward errors
#   failing     Rx  -17.4 dBm  (range -14.0 to -1.0)  TOO LOW: dirty connector, bend, or failing laser
#   overdriven  Rx    1.8 dBm  (range -14.0 to -1.0)  TOO HIGH: needs an attenuator
#
# Rx power drifting toward the low threshold predicts CRC errors WEEKS before
# they appear, which turns an outage into scheduled maintenance.

# --- Read counters as a RATE, not a total ---
ip -s link show eth0                  # totals since boot: nearly useless alone
# The correct procedure: clear, wait a known interval, read the delta.
ethtool -S eth0 > /tmp/before.txt; sleep 60; ethtool -S eth0 > /tmp/after.txt
diff /tmp/before.txt /tmp/after.txt | grep -E '^[<>].*(err|crc|drop|discard)'
#  Anything increasing over 60 seconds is ACTIVE. Anything static is history.

# On a switch, clear explicitly and re-read:
#   clear counters interface Gi1/0/14
#   ... wait ...
#   show interfaces Gi1/0/14 | include error|drop|collision|CRC

# --- Errors vs drops: two different verdicts ---
ethtool -S eth0 | grep -iE 'rx_crc_errors|rx_errors|rx_length_errors'   # BROKEN
ethtool -S eth0 | grep -iE 'tx_dropped|rx_dropped|rx_no_buffer'         # FULL
tc -s qdisc show dev eth0 | grep -E 'dropped|overlimits'                # FULL

# --- Duplex and speed: the mismatch check ---
ethtool eth0 | grep -E 'Speed|Duplex|Auto-negotiation|Link detected'
#  Speed: 100Mb/s          <- on a gigabit port? investigate the cable
#  Duplex: Half            <- on a switch port in 2024? a genuine fault
#  Auto-negotiation: off   <- forced on ONE side is how mismatches happen
# Both ends must agree. Check the switch side too:
#   show interfaces Gi1/0/14 status     -> "a-full" / "a-1000" means autonegotiated

# --- Link flapping: count carrier transitions over time ---
ip -s -d link show eth0 | grep -i 'carrier'
#  carrier_changes 1842      <- a stable link shows a small, unchanging number
watch -n5 "ip -s -d link show eth0 | grep carrier_changes"
dmesg -T | grep -iE 'link is (up|down)' | tail -20
#  Repeated up/down at regular intervals is usually a failing optic or a
#  half-seated connector, not a configuration problem.

# --- Optical diagnostics: the best leading indicator on a fibre link ---
ethtool -m eth0 | grep -iE 'Rx power|Tx power|temperature|Laser'
#  Rx power    : -13.10 dBm    <- compare against the module's own thresholds,
#  Rx power low warning threshold : -14.00 dBm      which ethtool also prints
# Alert on Rx power within 2 dB of the warning threshold, not on CRC errors:
# by the time CRC errors appear, users have already noticed.

# --- Layer 2 reachability: is the MAC even being learned? ---
ip neigh show                          # our ARP/ND cache
bridge fdb show | grep -i 02:42        # what the bridge learned
arping -I eth0 -c3 10.0.1.1            # L2 reachability, independent of IP routing
#  ARP working but IP failing -> the problem is above layer 2.
#  ARP failing -> VLAN, port, or physical. Stop looking at routes.

Key takeaways

• Lower-layer faults leave specific counters, which makes them diagnosable rather than merely suspected.
• Errors mean something is broken (replace it); drops mean something is full (add capacity or shape). Confusing them wastes hardware or bandwidth.
• A count without a denominator is meaningless. Clear counters, wait a known interval, and read the delta as a rate.
• Duplex mismatch shows CRC errors and runts on one side and late collisions on the other; the pair is diagnostic.
• A mismatch degrades a link to roughly a third of its rate without ever failing outright, which is why it persists as "sometimes slow".
• Carrier transitions count link flaps; a stable link's count does not move.
• Optical receive power is the best leading indicator: alert when it approaches the low warning threshold, not when CRC errors start.
• Use arping to test layer 2 independently of routing; ARP failing means VLAN, port, or physical, and routes are irrelevant.

🧪 Practice

1. An interface shows 4,200 CRC errors and 11 billion received frames, static over a 10-minute observation. State your conclusion and your action.
2. A gigabit link reports late collisions on one side and CRC errors on the other. Name the fault, give the expected throughput, and give the fix.
3. Write the procedure that determines whether an interface's errors are currently increasing, and explain why totals since boot are inadequate.
4. Interview-style: A link shows rising output drops and zero errors, and a team proposes replacing the cable and the optic. (Hint: separate the two counter families and say which condition each indicates, then name what would actually reduce drops.)

Diagnosing Routing and DNS Failures

Theory

Routing and name resolution fail in ways that look like everything else, and both have one dominant diagnostic principle: verify the actual decision, not the protocol that produced it.

For routing, that means asking the forwarding plane what it will do with a specific packet rather than reading the routing protocol's state. A BGP session can be perfectly established while the route you need is filtered; an OSPF adjacency can be up while a more specific static route sends your traffic elsewhere. The authoritative question is "which next hop will this destination take?", and every platform can answer it directly.

Two routing facts cause disproportionate confusion.

Longest prefix wins, and it is evaluated first. Administrative distance, preference, and metric only break ties within the same prefix length. A /24 static route beats a /16 BGP route for addresses inside the /24, regardless of any preference value. People reason about preference and forget that prefix length was decided before preference was consulted.

Asymmetric routing breaks stateful devices. Traffic can legitimately take one path outbound and a different one back, and pure routers do not care. A stateful firewall does: if the return path traverses a different firewall from the one that saw the outbound SYN, that firewall sees an unsolicited SYN-ACK and drops it. The symptom is a connection that never completes even though the SYN is visible arriving at the server, and it is invisible to a one-directional traceroute — which is why both directions must always be traced.

For DNS, the diagnostic discipline is to classify the response code, because each one implies a different next step:

Response	Meaning	Next step
NOERROR with an answer	working	DNS is not your problem
NOERROR with no answer	the name exists, that record type does not	query the right type; look for missing A or AAAA-only
NXDOMAIN	the name does not exist	check spelling, search domains, zone contents
SERVFAIL	resolver failure or DNSSEC validation failure	retry elsewhere; test +cd to implicate DNSSEC
REFUSED	the server will not answer you	you may be querying an authoritative server recursively, or an ACL blocks you

The distinction between NXDOMAIN and NOERROR-with-no-answer is the one people conflate, and they have entirely different causes. The first means the name is absent; the second means the name exists and you asked for the wrong record type — which is exactly what happens when a host publishes only AAAA and the client wants A.

That leads to the AAAA trap, worth knowing because it produces latency rather than failure. A name with both A and AAAA records, on a network where IPv6 is broken rather than absent, causes the client to attempt IPv6 first and wait for a timeout before falling back. Clients implementing Happy Eyeballs reduce this to a few hundred milliseconds by racing both families; clients that do not can wait seconds on every connection. The symptom is uniform, unexplained, reproducible added latency that disappears when IPv6 is disabled on the client, which is a strong hint rather than a fix.

Examples

import ipaddress
# Routing failures: verify the FORWARDING decision, not the routing protocol.
ROUTES = [("10.20.0.0/16", "10.0.0.2", "bgp",    20),
          ("10.20.5.0/24", "10.0.0.6", "static",  1),
          ("10.0.0.0/8",   "10.0.0.2", "ospf",  110),
          ("0.0.0.0/0",    "10.0.0.1", "static",  1)]
TABLE = sorted(((ipaddress.ip_network(p), nh, proto, pref)
                for p, nh, proto, pref in ROUTES),
               key=lambda r: -r[0].prefixlen)      # LONGEST PREFIX FIRST

def lookup(dst):
    a = ipaddress.ip_address(dst)
    for net, nh, proto, pref in TABLE:
        if a in net:
            return net, nh, proto, pref
    return None, None, None, None

print("Forwarding table, in match order:")
for net, nh, proto, pref in TABLE:
    print(f"  {str(net):16s} via {nh:10s} [{proto:6s} pref {pref:3d}]")
print("\nLookups -- the LONGEST prefix wins, not the lowest preference:")
for dst in ("10.20.5.9", "10.20.9.9", "10.7.0.1", "8.8.8.8"):
    net, nh, proto, pref = lookup(dst)
    print(f"  {dst:12s} -> {str(net):16s} via {nh:10s} ({proto})")
# Forwarding table, in match order:
#   10.20.5.0/24     via 10.0.0.6   [static pref   1]
#   10.20.0.0/16     via 10.0.0.2   [bgp    pref  20]
#   10.0.0.0/8       via 10.0.0.2   [ospf   pref 110]
#   0.0.0.0/0        via 10.0.0.1   [static pref   1]
#
# Lookups -- the LONGEST prefix wins, not the lowest preference:
#   10.20.5.9    -> 10.20.5.0/24     via 10.0.0.6   (static)
#   10.20.9.9    -> 10.20.0.0/16     via 10.0.0.2   (bgp)
#   10.7.0.1     -> 10.0.0.0/8       via 10.0.0.2   (ospf)
#   8.8.8.8      -> 0.0.0.0/0        via 10.0.0.1   (static)
#
# Prefix length is evaluated BEFORE preference. Reasoning about administrative
# distance while forgetting prefix length is a very common error.

print("\nAsymmetric routing, and why it breaks stateful devices:")
FORWARD = ["client", "fw-a", "core", "server"]
REVERSE = ["server", "core", "fw-b", "client"]        # a DIFFERENT firewall
print(f"  forward: {' -> '.join(FORWARD)}")
print(f"  reverse: {' -> '.join(REVERSE)}")
fw_fwd = {h for h in FORWARD if h.startswith("fw")}
fw_rev = {h for h in REVERSE if h.startswith("fw")}
print(f"  stateful device forward: {fw_fwd}   reverse: {fw_rev}")
if fw_fwd != fw_rev:
    print("  -> MISMATCH: fw-b never saw the SYN, so it treats the SYN-ACK as")
    print("     unsolicited and drops it. The connection never completes, while")
    print("     the SYN is plainly visible arriving at the server.")
print("  Invisible to a one-directional traceroute: trace BOTH ways.")

print("\nDNS failure modes, and the distinguishing evidence:")
for rcode, name, meaning, next_step in (
    (0, "NOERROR + answer", "working", "DNS is not your problem"),
    (0, "NOERROR + no answer", "the name exists, that RECORD TYPE does not",
     "query the right type; check for AAAA-only or missing A"),
    (3, "NXDOMAIN", "the name does not exist",
     "check spelling, search domains, and the zone contents"),
    (2, "SERVFAIL", "resolver failure, or DNSSEC validation failure",
     "retry against another resolver; test +cd to implicate DNSSEC"),
    (5, "REFUSED", "the server will not answer you",
     "querying an authoritative server recursively, or an ACL blocks you"),
):
    print(f"  rcode {rcode} {name:22s} {meaning}")
    print(f"  {'':10s} {'':22s} -> {next_step}")

print("\nThe AAAA trap, a frequent cause of unexplained connection latency:")
IPV6_TIMEOUT_MS = 5000
for scenario, extra_ms in (("A only", 0), ("A + AAAA, v6 working", 0),
                           ("A + AAAA, v6 BLACKHOLED", IPV6_TIMEOUT_MS)):
    print(f"  {scenario:26s} adds {extra_ms:5d} ms before falling back to IPv4")
#   A only                     adds     0 ms before falling back to IPv4
#   A + AAAA, v6 working       adds     0 ms before falling back to IPv4
#   A + AAAA, v6 BLACKHOLED    adds  5000 ms before falling back to IPv4
#
# A name with AAAA on a network where IPv6 is BROKEN (not absent) waits for the
# v6 attempt to time out on every connection. Happy Eyeballs reduces this to a
# few hundred milliseconds -- but only if the client implements it.

# --- Routing: ask the FORWARDING plane about a specific packet ---
ip route get 10.20.5.9
#  10.20.5.9 via 10.0.0.6 dev eth1 src 10.0.1.9
#     ^ the actual decision, not the routing table's contents
ip route get 10.20.5.9 from 10.0.1.9 ipproto tcp sport 40000 dport 443
#  Include the 5-tuple: with ECMP or policy routing, the answer DEPENDS on it.

ip rule show          # policy routing: checked BEFORE the main table
ip route show table all | grep 10.20    # a route in another table is easily missed

# On a router, verify the FIB, not just the RIB:
vtysh -c 'show ip route 10.20.5.9'         # RIB: what the control plane decided
vtysh -c 'show ip fib 10.20.5.9' 2>/dev/null || ip route get 10.20.5.9

# --- Is the route even being received? Distinguish "no route" from "filtered" ---
vtysh -c 'show bgp summary'                              # session up?
vtysh -c 'show bgp ipv4 unicast neighbors 10.0.0.2 received-routes' | grep 10.20
vtysh -c 'show bgp ipv4 unicast neighbors 10.0.0.2 routes' | grep 10.20
#  Present in "received-routes" but absent from "routes" -> YOUR inbound filter
#  dropped it. An established session proves nothing about a specific prefix.

# --- Asymmetry: trace BOTH directions. This is not optional. ---
traceroute -n -T -p 443 10.20.5.9
ssh 10.20.5.9 "traceroute -n -T -p 443 $(hostname -I | awk '{print $1}')"
# Compare the hop lists. Different stateful devices on each path means the
# connection will hang with the SYN visible at the far end.
# Confirm from the server side:
ssh 10.20.5.9 "timeout 10 tcpdump -i any -nn 'tcp port 443 and tcp[tcpflags] & tcp-syn != 0'"
#  SYN arrives, no SYN-ACK returns to the client -> asymmetric path, or the
#  return traffic is dropped by a device that never saw the outbound SYN.

# --- DNS: classify the response, then act on the class ---
dig api.example.com | grep -E 'status:|ANSWER:'
#  status: NOERROR, ANSWER: 0     <- the name exists; you asked the wrong TYPE
dig api.example.com AAAA +short   # is it AAAA-only?
dig api.example.com A +short

dig api.example.com | grep 'status: SERVFAIL' && {
  dig +cd api.example.com +short      # +cd disables DNSSEC validation
  echo "if +cd works, the problem is DNSSEC, not reachability"
  dig @1.1.1.1 api.example.com +short # and try a different resolver
}

# --- The AAAA trap: measure it, then prove it ---
time curl -sS -o /dev/null https://api.example.com/health
time curl -sS -o /dev/null -4 https://api.example.com/health   # force IPv4
#  A large difference means IPv6 is being attempted and failing.
ping6 -c2 -W2 "$(dig +short AAAA api.example.com | head -1)" \
  || echo "IPv6 is BROKEN (not absent): this is the latency source"
ip -6 route show default        # is there even a v6 default route?
#  The fix belongs in the network or in the DNS records, not in disabling IPv6
#  on every client -- though doing so on one client is a good confirmation.

Key takeaways

• Verify the forwarding decision, not the protocol state: ip route get and its equivalents answer the authoritative question.
• Longest prefix wins, and is evaluated before preference; administrative distance only breaks ties within the same prefix length.
• Check policy routing rules and all route tables; a route in another table is easily missed.
• An established BGP session proves nothing about a specific prefix — compare received routes against accepted routes to find your own inbound filter.
• Asymmetric routing breaks stateful devices: the return firewall sees an unsolicited SYN-ACK and drops it, so the SYN arrives and the connection never completes.
• Always trace both directions. A one-directional traceroute cannot see this class of fault.
• Classify DNS responses: NOERROR with no answer means wrong record type; NXDOMAIN means no such name; SERVFAIL means resolver or DNSSEC failure; REFUSED means an ACL or a recursion mismatch.
• Test +cd to implicate DNSSEC and a second resolver to implicate the first one.
• The AAAA trap adds latency when IPv6 is broken rather than absent; forcing IPv4 confirms it, but the fix belongs in the network or the records.

🧪 Practice

1. Given routes 10.0.0.0/8 via A with preference 1 and 10.5.0.0/16 via B with preference 200, state which next hop 10.5.2.9 takes and why.
2. A connection hangs, and a capture on the server shows the SYN arriving with no SYN-ACK leaving. Name two candidate causes and the command that distinguishes them.
3. dig example.internal returns NOERROR with zero answers. Explain what this means, and give two queries that would identify the cause.
4. Interview-style: Every HTTPS request from a new subnet takes exactly five seconds longer than from the old one, and all of them succeed. (Hint: an exact, uniform delay suggests a timeout rather than congestion — consider which address family is tried first and what happens when it neither works nor fails fast.)

Performance Problem Triage

Theory

"It is slow" is the hardest class of report, because nothing is broken and every component looks healthy in isolation. Triage is the process of converting that complaint into a measured decomposition, and it rests on two ideas.

The first is decompose the latency budget and attack the largest term. This is Amdahl's argument applied to response time: optimising a component that contributes 0.3% of total latency cannot help, however satisfying it is to fix. The practical sequence is to measure each contribution — client, DNS, connect, TLS, server queueing, server processing, downstream dependencies, transfer — then sort by contribution and work from the top. The most common failure in performance work is optimising the term someone already knows how to optimise rather than the term that dominates.

Note the distinction between server queueing and server processing. Time spent waiting for a worker is not time spent doing work, and the two have completely different fixes: queueing is a concurrency and capacity problem, processing is a code and query problem. Systems that report only total handler time hide this difference, and it is frequently where the answer is.

The second idea is to find the saturated resource, for which the USE method is the standard framework: for each resource, examine Utilisation, Saturation, and Errors. The important insight is that saturation is a different counter from utilisation, and it is the one that matters. A firewall at 41% CPU with a session table 96% full is about to fail, and a utilisation dashboard clears it. A link at 25% average utilisation with rising output drops is too narrow for its bursts. Utilisation is an average over an interval; saturation is evidence that something queued or was refused.

A useful shortcut is that the shape of the slowness names the class of problem:

Symptom	Likely cause	Direction of fix
Small requests slow, bulk transfer fine	latency: round trips, per-request setup	fewer round trips: reuse, batch, move closer
Bulk transfer slow, small requests fine	throughput: window, loss, capacity	window, loss, capacity
Both slow in proportion	a shared saturated resource	find it with the USE method
Slow only at peak	contention	capacity, or QoS to protect the class
Slow for one user only	that user's path or client	compare against a working peer

The first two rows are the most useful, because they separate latency problems from throughput problems, and those have nearly opposite remedies. Adding bandwidth to a latency problem does nothing; reducing round trips on a throughput problem does nothing. Establishing which one you have should precede any other work, and it takes one measurement of each kind.

Examples

# Performance triage: decompose the latency budget, then attack the largest term.
OBSERVED = {   # one slow request, measured end to end
    "client render":      120,
    "DNS":                  4,
    "TCP connect":         21,
    "TLS handshake":       27,
    "server queueing":    310,     # WAITING for a worker, not doing work
    "server processing":   48,
    "database query":     680,     # the actual problem
    "response transfer":   35,
}
total = sum(OBSERVED.values())
print(f"{'component':22s} {'ms':>7s} {'share':>7s}  bar")
for name, ms in sorted(OBSERVED.items(), key=lambda kv: -kv[1]):
    share = ms / total
    print(f"{name:22s} {ms:7d} {share:6.1%}  {'#' * int(share * 50)}")
print(f"{'TOTAL':22s} {total:7d}")
# component                   ms   share  bar
# database query             680  54.6%  ###########################
# server queueing            310  24.9%  ############
# client render              120   9.6%  ####
# server processing           48   3.9%  #
# response transfer           35   2.8%  #
# TLS handshake               27   2.2%  #
# TCP connect                 21   1.7%
# DNS                          4   0.3%
# TOTAL                     1245
#
# Note that "server queueing" (310 ms waiting for a worker) is separate from
# "server processing" (48 ms doing work). A system reporting only total handler
# time would hide a 6x difference with entirely different fixes.

print("\nAmdahl's rule applied to latency: optimising a small term is wasted work.")
for name in ("database query", "TLS handshake", "DNS"):
    ms = OBSERVED[name]
    for factor in (2, 10):
        saved = ms - ms / factor
        print(f"  make '{name}' {factor:2d}x faster -> saves {saved:6.1f} ms, "
              f"total {total-saved:6.1f} ms ({100*(total-saved)/total:5.1f}% of now)")
#   make 'database query'  2x faster -> saves  340.0 ms, total  905.0 ms ( 72.7% of now)
#   make 'database query' 10x faster -> saves  612.0 ms, total  633.0 ms ( 50.8% of now)
#   make 'TLS handshake'  2x faster -> saves   13.5 ms, total 1231.5 ms ( 98.9% of now)
#   make 'TLS handshake' 10x faster -> saves   24.3 ms, total 1220.7 ms ( 98.0% of now)
#   make 'DNS'  2x faster -> saves    2.0 ms, total 1243.0 ms ( 99.8% of now)
#   make 'DNS' 10x faster -> saves    3.6 ms, total 1241.4 ms ( 99.7% of now)
#
# Eliminating DNS ENTIRELY saves 4 ms of 1245. Measure before optimising.

print("\nThe USE method: Utilisation, Saturation, Errors -- per resource")
RESOURCES = {
    # resource:      (utilisation, saturation signal, saturated?, errors)
    "access link":   (0.23, "tx queue depth 0",              False, "0 CRC"),
    "uplink":        (0.94, "tx drops 18,422 and rising",     True,  "0 CRC"),
    "firewall":      (0.41, "session table 96% full",         True,  "0"),
    "server CPU":    (0.62, "run queue 1.2",                  False, "0"),
    "server socket": (0.10, "accept queue overflows 0",       False, "0"),
    "database":      (0.88, "lock waits rising",              True,  "0"),
}
print(f"{'resource':16s} {'util':>6s} {'saturation':32s} {'sat?':>5s} "
      f"{'errors':>8s}  verdict")
for r, (util, sat, saturated, err) in RESOURCES.items():
    verdict = "SUSPECT" if (saturated or util > 0.80) else "healthy"
    print(f"{r:16s} {util:5.0%} {sat:32s} {str(saturated):>5s} {err:>8s}  {verdict}")
# resource           util saturation                        sat?   errors  verdict
# access link        23% tx queue depth 0                 False    0 CRC  healthy
# uplink             94% tx drops 18,422 and rising        True    0 CRC  SUSPECT
# firewall           41% session table 96% full            True        0  SUSPECT
# server CPU         62% run queue 1.2                    False        0  healthy
# server socket      10% accept queue overflows 0         False        0  healthy
# database           88% lock waits rising                 True        0  SUSPECT
#
# The firewall is the lesson: 41% utilisation looks fine and its SESSION TABLE
# is 96% full. Utilisation alone would have cleared it. Saturation is a
# DIFFERENT counter for every resource, and it is the one that matters.

print("\nDistinguishing a latency problem from a throughput problem:")
for symptom, likely, fix in (
    ("small requests slow, bulk fine", "latency: RTT and per-request setup",
     "fewer round trips: reuse connections, batch, move closer"),
    ("bulk slow, small requests fine", "throughput: window, loss, capacity",
     "raise the window, fix loss, add capacity"),
    ("both slow, in proportion", "a shared saturated resource",
     "find it with the USE method"),
    ("slow only at peak", "contention with other traffic",
     "capacity planning, or QoS to protect this class"),
    ("slow for one user only", "that user's path or client",
     "compare against a working peer; the difference is the fault"),
):
    print(f"  {symptom:34s} -> {likely}")
    print(f"  {'':34s}    {fix}")
print("\nEstablish WHICH of the first two you have before doing anything else:")
print("adding bandwidth to a latency problem does nothing, and reducing round")
print("trips on a throughput problem does nothing.")

# --- Step 1: is this latency or throughput? Two measurements, five seconds. ---
# Latency probe: a tiny request, where setup dominates
curl -sS -o /dev/null -w 'small: total %{time_total}s ttfb %{time_starttransfer}s\n' \
  https://api.example.com/health
# Throughput probe: a large transfer, where the window and loss dominate
curl -sS -o /dev/null -w 'bulk : %{speed_download} B/s in %{time_total}s\n' \
  https://api.example.com/large-file
#  small slow + bulk fine -> LATENCY problem. Stop thinking about bandwidth.
#  bulk slow + small fine -> THROUGHPUT problem. Stop counting round trips.

# --- Step 2: decompose the latency budget ---
curl -sS -o /dev/null -w \
'dns %{time_namelookup} connect %{time_connect} tls %{time_appconnect} \
ttfb %{time_starttransfer} total %{time_total}\n' https://api.example.com/health
#  Subtract adjacent values. The largest phase is where to work.
# Ask the application for its own split, if it has one:
curl -sS -D- -o /dev/null https://api.example.com/health | grep -i 'server-timing'
#  Server-Timing: queue;dur=310, app;dur=48, db;dur=680
#     ^ queueing vs processing vs dependency: exactly the split that matters

# --- Step 3: USE, per resource. Saturation, not utilisation. ---
# Network: utilisation AND drops
sar -n DEV 1 5 | grep -E 'eth0|Average'
ethtool -S eth0 | grep -iE 'drop|discard|pause'
tc -s qdisc show dev eth0 | grep -E 'dropped|backlog'

# CPU: utilisation AND run queue (saturation)
vmstat 1 5            # the 'r' column is the run queue: saturation
mpstat -P ALL 1 3 | tail -5

# Sockets: utilisation AND overflow (saturation)
ss -tln | head -3                          # Recv-Q on LISTEN = current queue depth
nstat -az | grep -E 'ListenOverflows|ListenDrops|TCPBacklogDrop'

# Per-connection TCP state: is it loss- or window-limited?
ss -tin dst api.example.com | grep -E 'rtt|cwnd|retrans|delivery_rate'

# Middleboxes: the session-table check that utilisation dashboards miss
#   show conn count          (firewall session table vs its maximum)
#   show resource usage

# --- Step 4: confirm the fix changed the term you targeted ---
# Re-measure the SAME decomposition. If the dominant term did not shrink,
# the change did not address the problem, whatever else improved.

Key takeaways

• Decompose the latency budget and attack the largest term. Optimising a component contributing under a few percent cannot help.
• Separate server queueing from server processing: waiting for a worker is a concurrency problem, doing work is a code problem.
• Use the USE method per resource: Utilisation, Saturation, Errors.
• Saturation is a different counter from utilisation and is the one that matters. A firewall at 41% CPU with a 96% full session table is about to fail.
• The shape of the slowness names the class: small requests slow means latency, bulk slow means throughput, both slow means a shared saturated resource.
• Latency and throughput problems have opposite fixes; establish which you have before doing anything else.
• Server-Timing headers or equivalent application instrumentation give the queueing-versus-processing split that network tools cannot see.
• After a fix, re-measure the same decomposition to confirm the dominant term actually shrank.

🧪 Practice

1. Given contributions of DNS 3 ms, connect 40 ms, TLS 55 ms, server 620 ms, transfer 30 ms, compute each share and state the two changes with the largest possible effect.
2. Explain why a firewall at 35% CPU utilisation can be the bottleneck, and name the counter that would reveal it.
3. Design two measurements that determine whether a complaint is a latency or a throughput problem, and state what each result implies.
4. Interview-style: A team spends a sprint enabling TLS session resumption and HTTP keep-alive, and p99 response time is unchanged. (Hint: work out which phases those changes affect, then ask what fraction of the total those phases represented before the work started.)

Documenting and Escalating Incidents

Theory

The technical work of an incident produces a diagnosis; the documentation determines whether that diagnosis is usable by anyone else, including yourself next month. This is not administrative overhead, and two specific mechanisms make it load-bearing.

A timeline is a diagnostic tool, not a record. Writing observations down with timestamps as you make them turns a set of impressions into correlatable data. It answers "did the errors start before or after the deployment?", a question that determines causality and that nobody can reconstruct afterwards from memory. The discipline is to record the time, the observation, and the action separately, and to record negative results too. "Checked the uplink counters, zero errors" is precisely the entry that stops the next person spending ten minutes checking the uplink counters.

An escalation is a handover with a burden of proof. Escalations bounce back because they assert a conclusion without the evidence supporting it. A provider receiving "your network is dropping packets" will consult their own monitoring, find nothing, and close the ticket — and they are not being obstructive, because from their side there genuinely is nothing. A provider receiving a bidirectional mtr report, simultaneous captures showing specific sequence numbers present at one end and absent at the other, the circuit identifier, and timestamps in their timezone has been handed something they cannot dismiss.

The escalation package has a predictable shape:

Element	Why it is required
Impact, quantified	sets priority: how many users, which service, since when
Timeline with timestamps	establishes what changed and in what order
What you have eliminated	stops the recipient repeating your work
The evidence, attached	captures, mtr output, counters, configuration
The specific ask	"check the optic in port 14", not "please investigate"
Identifiers	circuit ID, device names, addresses, ticket references

The specific ask is the element most often omitted and the one that most changes the outcome. "Please investigate" invites a generic loopback test that passes. Naming the sequence numbers, the timestamps, and the suspect hop invites an engineer to look at one device.

Two operational notes. First, separate mitigation from diagnosis, and weigh evidence collection by what it costs. Restoring service is the priority, and the fastest restoration is often an action that destroys the evidence: a reboot, a failover, a cache flush. Saving interface counters costs two seconds and should always happen first; a thirty-second capture is usually worth it; a full support bundle is not, and belongs after mitigation. The wrong choice is leaving users down to collect a perfect capture, and the other wrong choice is rebooting before spending the two seconds.

Second, the post-incident review should ask why the system allowed the fault, not who made a mistake. A blameless review produces the information needed to prevent recurrence; a review searching for a culprit produces defensiveness and no information. The most valuable questions are about detection and diagnosis time rather than about the fault itself: how it was found, how long that took, whether monitoring would have caught it unaided, and what would have made the fault impossible rather than merely less likely.

Examples

An incident timeline that is useful afterwards

  time     observation                                    action / result
  ------   --------------------------------------------   -----------------------
  14:02    users at site-C report API timeouts            opened INC-4821
  14:04    site-A and site-B unaffected                   -> scope is site-C only
  14:05    curl from site-C: connect ok, ttfb 5.2s        -> not DNS, not L3
  14:07    server-side p99 handler time 12 ms (normal)    -> NOT the application
  14:09    mtr site-C -> api: 4% loss from hop 6 onward   -> provider path suspect
  14:11    mtr api -> site-C: 0% loss                     -> LOSS IS ASYMMETRIC
  14:13    site-C uplink counters: 0 errors               (negative result: KEPT)
  14:14    site-C firewall sessions 41%, normal           (negative result: KEPT)
  14:18    both-ends capture: seq 84120..91882 sent,      -> loss is BETWEEN our
           absent at the server                             edge and theirs
  14:22    escalated to provider, ticket PRV-99120        attached captures + mtr
  14:41    provider confirms failing optic, PE-3 port 14  -> matches hop 6
  14:52    provider replaces optic                        loss stops
  14:55    verified from site-C: ttfb 0.09s               INC-4821 mitigated
  15:30    gap found: no per-hop loss alerting on          follow-up ticket
           provider paths

  The two entries at 14:13 and 14:14 are the ones that earn their keep: they
  stop the next engineer re-checking the uplink and the firewall.

  Note also what the timeline PROVES about causality: the application was
  measured normal at 14:07, BEFORE the provider was engaged at 14:22. Without
  timestamps, "the app was fine" is an assertion; with them, it is evidence.

# An escalation package, assembled as a checklist rather than a narrative.
PACKAGE = {
    "impact": dict(required=True,
        content="site-C (140 users) cannot use the ordering API since 14:02 UTC; "
                "other sites unaffected; revenue-affecting"),
    "timeline": dict(required=True,
        content="attached: timestamped observations, 14:02 to 14:22 UTC"),
    "eliminated": dict(required=True,
        content="application (p99 12 ms, normal), DNS (resolves correctly), "
                "site-C uplink (0 errors), site-C firewall (41% sessions)"),
    "evidence": dict(required=True,
        content="mtr both directions, simultaneous pcaps from both ends, "
                "interface counters before and after"),
    "specific_ask": dict(required=True,
        content="seq 84120-91882 left our edge 203.0.113.9 at 14:18:03 UTC and did "
                "not arrive at 198.51.100.4; please check hop 6 (62.115.120.1), "
                "which shows 4% loss persisting downstream"),
    "identifiers": dict(required=True,
        content="circuit CKT-88214, our ticket INC-4821, device site-C-edge-1"),
    "contact": dict(required=False,
        content="on-call rotation, escalation path, timezone UTC"),
}
missing = [k for k, v in PACKAGE.items() if v["required"] and not v["content"]]
print(f"escalation package: {len(PACKAGE)} sections, "
      f"{len(missing)} required section(s) missing\n")
for name, v in PACKAGE.items():
    print(f"[{'REQUIRED' if v['required'] else 'optional'}] {name}")
    print(f"           {v['content']}\n")
# escalation package: 7 sections, 0 required section(s) missing
#
# [REQUIRED] impact
#            site-C (140 users) cannot use the ordering API since 14:02 UTC; ...
# [REQUIRED] timeline
#            attached: timestamped observations, 14:02 to 14:22 UTC
# [REQUIRED] eliminated
#            application (p99 12 ms, normal), DNS (resolves correctly), ...
# ... (each required section must be non-empty before sending)

print("Why the specific ask matters more than any other section:")
for phrasing, outcome in (
    ("your network is dropping packets",
     "provider checks their monitoring, finds nothing, closes the ticket"),
    ("please investigate circuit CKT-88214",
     "a generic loopback test, which passes, and the ticket is closed"),
    ("seq 84120-91882 left 203.0.113.9 at 14:18:03 and did not arrive; hop 6 "
     "shows 4% loss persisting downstream",
     "an engineer looks at a specific device and finds the failing optic"),
):
    print(f"  ask: {phrasing[:70]}")
    print(f"       -> {outcome}\n")
#   ask: your network is dropping packets
#        -> provider checks their monitoring, finds nothing, closes the ticket
#
#   ask: please investigate circuit CKT-88214
#        -> a generic loopback test, which passes, and the ticket is closed
#
#   ask: seq 84120-91882 left 203.0.113.9 at 14:18:03 and did not arrive; hop 6
#        -> an engineer looks at a specific device and finds the failing optic

print("Mitigation vs diagnosis: capture evidence only when it is CHEAP.")
print(f"{'action':38s} {'cost':>6s} {'destroys':28s} verdict")
for action, destroys, seconds, verdict in (
    ("copy interface counters", "nothing", 2, "always do this first"),
    ("30-second both-ends capture", "nothing", 35, "do it if users can wait"),
    ("collect device support bundle", "nothing", 300, "AFTER mitigating"),
    ("reboot the device", "all volatile state", 0, "destroys evidence"),
    ("failover to standby", "the failed path's state", 0, "destroys evidence"),
    ("flush the ARP/DNS cache", "the reproducing condition", 0, "destroys evidence"),
):
    print(f"  {action:36s} {seconds:4d}s  {destroys:28s} {verdict}")
#   copy interface counters                 2s  nothing                      always do this first
#   30-second both-ends capture            35s  nothing                      do it if users can wait
#   collect device support bundle         300s  nothing                      AFTER mitigating
#   reboot the device                       0s  all volatile state           destroys evidence
#   failover to standby                     0s  the failed path's state      destroys evidence
#   flush the ARP/DNS cache                 0s  the reproducing condition    destroys evidence
#
# The zero-cost rows are the dangerous ones: they are FAST, which is exactly why
# they get done first, and each destroys the state needed to find the cause.

print("\nPost-incident review: ask why the SYSTEM allowed it, not who erred.")
for question in (
    "how was it detected -- monitoring, or a user report?",
    "how long from onset to detection? from detection to diagnosis?",
    "would the monitoring have caught it without a human looking?",
    "what made the diagnosis slow, and what would have made it fast?",
    "what change would make this class of fault impossible, not just unlikely?",
    "which negative results did we re-check because nobody wrote them down?",
):
    print(f"  - {question}")

# --- Capture the cheap evidence FIRST, in one command, before mitigating ---
INC=INC-4821; D=/var/incidents/$INC; mkdir -p "$D"
{
  date -u +'=== %FT%TZ ==='
  ip -s link show; echo
  ip route show; ip rule show; echo
  ss -tan state all | awk 'NR>1{c[$1]++} END{for(s in c) print s, c[s]}'; echo
  nstat -az | grep -E 'Listen|Retrans|Timeouts|OfoQueue'; echo
  ethtool -S eth0 2>/dev/null | grep -iE 'err|drop|discard'
} > "$D/host-state.txt"
#  Two seconds. Do this before any reboot, failover, or cache flush.

# --- The evidence that survives escalation: both directions, both ends ---
mtr -rwzbc 200 api.example.com            > "$D/mtr-outbound.txt"
ssh api.example.com "mtr -rwzbc 200 $(curl -s ifconfig.me)" > "$D/mtr-inbound.txt"
# and the simultaneous captures from the previous subchapter:
#   client.pcap, server.pcap, plus the comm(1) diff of their sequence numbers

# --- Build the timeline as you go, not afterwards ---
note() { printf '%s  %s\n' "$(date -u +%H:%M:%S)" "$*" >> "$D/timeline.txt"; }
note "users at site-C report API timeouts"
note "site-A and site-B unaffected -> scope is site-C only"
note "NEGATIVE: site-C uplink counters 0 errors"      # record what you RULED OUT
note "NEGATIVE: site-C firewall sessions 41%, normal"
note "both-ends capture: seq 84120-91882 sent, absent at server"
cat "$D/timeline.txt"

# --- Assemble the escalation, and check nothing required is missing ---
cat > "$D/escalation.md" <<'TXT'
## Impact
site-C (140 users) cannot use the ordering API since 14:02 UTC. Other sites
unaffected. Revenue-affecting.

## Eliminated
- application: p99 handler time 12 ms (normal) at 14:07
- DNS: resolves correctly from site-C
- site-C uplink: 0 errors over a 10-minute delta
- site-C firewall: session table 41%, no drops

## Evidence (attached)
- mtr-outbound.txt: 4% loss from hop 6 (62.115.120.1) onward, persisting
- mtr-inbound.txt: 0% loss (loss is ASYMMETRIC)
- client.pcap / server.pcap: seq 84120-91882 present at client, absent at server

## Specific ask
Packets with seq 84120-91882 left our edge 203.0.113.9 at 14:18:03 UTC and did
not arrive at 198.51.100.4. Please check hop 6 (62.115.120.1) and the path
between your PE devices. Circuit CKT-88214. Our ticket INC-4821.
TXT
for s in Impact Eliminated Evidence "Specific ask"; do
  grep -q "^## $s" "$D/escalation.md" || echo "MISSING SECTION: $s"
done

# --- After the incident: measure the response, not just the fault ---
# onset -> detection -> diagnosis -> mitigation, from the timeline timestamps.
awk '{print}' "$D/timeline.txt" | head -1   # first observation
# Compare against the monitoring system: did it alert, and when?
#  If the first entry is a USER REPORT, the follow-up action is a monitoring gap,
#  regardless of how quickly the fault itself was fixed.

Key takeaways

• A timeline is a diagnostic tool. Timestamped observations establish causality that nobody can reconstruct from memory afterwards.
• Record negative results. "Checked X, normal" is what stops the next person re-checking X.
• An escalation is a handover with a burden of proof; asserting a conclusion without evidence is why tickets bounce back.
• The package needs impact, timeline, what you eliminated, attached evidence, a specific ask, and identifiers.
• The specific ask changes the outcome more than anything else: name sequence numbers, timestamps, and the suspect hop instead of asking for investigation.
• Weigh evidence collection by cost: counters cost two seconds and always come first; a support bundle waits until after mitigation.
• The zero-cost actions are the dangerous ones — reboot, failover, cache flush are fast, which is why they happen first, and each destroys the state needed to find the cause.
• A blameless review asks why the system allowed the fault; the highest-value questions concern detection and diagnosis time.
• If the first timeline entry is a user report rather than an alert, the follow-up action is a monitoring gap however fast the fix was.

🧪 Practice

1. Write a six-entry timeline for an incident you have seen or can imagine, including at least two negative results, and state what each negative result saves.
2. Rewrite the escalation "the link to site-B is dropping packets, please fix" into a package containing all six required elements.
3. Rank these by whether to do them before or after mitigating, with reasoning: saving interface counters, a 60-second packet capture, a device reboot, a support bundle, a failover.
4. Interview-style: An incident is resolved in eleven minutes by restarting the affected device, and it recurs twice more that week. (Hint: the restart was excellent mitigation and produced no diagnosis — identify what was lost at the moment of restart, and what cheap step would have preserved it.) <a id="144-monitoring-and-capacity-planning"></a>

14.4 Monitoring and Capacity Planning

Troubleshooting is what happens when monitoring failed to warn you, so the goal of this subchapter is to reduce how often the previous three are needed: it covers what to collect and at what cadence, the flow data that answers questions metrics cannot, how to set thresholds that produce alerts people still read after a month, the arithmetic of availability and error budgets, forecasting growth against procurement lead times, and designing redundancy that works when tested.

SNMP Polling vs Streaming Telemetry

Theory

Section 13.5 covered the transport mechanics: polling bounds freshness by the interval and is limited by collector request rate, while streaming subscribes once and lets the device push. Here the question is the operational one that follows: given a collection mechanism, what should you actually collect, at what cadence, and for how long?

Start from the observation that collecting everything at the fastest cadence is how monitoring systems become the outage they were meant to prevent. The cadence should follow the question, and the questions differ by two orders of magnitude:

Question	Signal	Cadence
Is it up?	reachability probe, interface operational status	on change
Is it healthy?	errors, drops, discards	10 to 60 seconds
Is it fast enough?	per-class latency and loss probes	10 to 30 seconds
Will it be enough?	utilisation peaks	5 minutes, rolled to daily

The second design decision is retention and rollup, and it contains a trap that quietly destroys the ability to plan capacity. Keeping raw high-frequency data forever is unaffordable, so systems downsample: raw for days, five-minute for weeks, hourly for a year, daily for years. Each tier answers a different class of question, which is why several tiers exist rather than one.

The trap is what the rollup keeps. If a five-minute rollup stores only the mean, then a ten-second burst that filled a buffer is averaged into invisibility, and next year's capacity analysis will conclude the link was quiet. Capacity must cover peaks, so the rollup must retain maximum and a high percentile alongside the mean. This is a configuration decision made once, usually by default, and discovered to be wrong eighteen months later when the data needed to justify an upgrade turns out never to have been recorded.

A related discipline is knowing which counters are cumulative and which are gauges. Interface byte and error counters are monotonically increasing and wrap; the useful quantity is the rate, computed as a delta over a known interval. Queue depth and CPU are instantaneous gauges, where a single sample is a point observation and the peak between samples is lost. Treating a counter as a gauge produces nonsense, and treating a gauge as authoritative produces false confidence — which is the sampling-resolution argument from Chapter 13 restated as a data-modelling rule.

Finally, monitor the monitoring. A collector that has silently stopped scraping looks identical to a network with no problems: flat lines, no alerts. Staleness checks — alerting when a series has received no sample for longer than its interval — are what distinguish "healthy" from "not being watched", and they are omitted from most initial deployments.

Examples

# Retention and rollup: the arithmetic that decides what you can still ask later.
SERIES, BYTES_PER_SAMPLE = 230_400, 16
TIERS = [("raw", 10, 7), ("5-minute", 300, 30),
         ("1-hour", 3600, 400), ("1-day", 86400, 1825)]

print(f"{'tier':12s} {'interval':>9s} {'retained':>9s} {'samples':>16s} {'bytes':>12s}")
total = 0
for name, interval, days in TIERS:
    samples = SERIES * (days * 86400 / interval)
    size = samples * BYTES_PER_SAMPLE
    total += size
    print(f"{name:12s} {interval:8d}s {days:8d}d {samples:16,.0f} {size/1e9:10.1f}GB")
print(f"{'TOTAL':12s} {'':9s} {'':9s} {'':16s} {total/1e9:10.1f}GB")
# tier          interval  retained          samples        bytes
# raw                10s        7d   13,934,592,000      223.0GB
# 5-minute          300s       30d    1,990,656,000       31.9GB
# 1-hour           3600s      400d    2,211,840,000       35.4GB
# 1-day           86400s     1825d      420,480,000        6.7GB
# TOTAL                                                  296.9GB
#
# Note that 7 days of RAW data costs more than 5 years of daily rollups. That
# ratio is why tiering exists, and why the raw tier is the one to shorten first.

print("\nWhat each tier can still answer, which is the point of keeping it:")
for name, days, answers in (
    ("raw 10s",   7, "what happened during last Tuesday's incident"),
    ("5-minute", 30, "was last month's peak higher than this month's"),
    ("1-hour",  400, "what is the year-on-year growth rate"),
    ("1-day",  1825, "when will we need to buy more capacity"),
):
    print(f"  {name:10s} kept {days:5d}d -> {answers}")

print("\nWhat rollup DESTROYS, and why it matters:")
BURST = [2, 3, 2, 980, 4, 3, 2, 3]      # one 10-second burst within 80 seconds
print(f"  raw samples          : {BURST}")
print(f"  mean over the window : {sum(BURST)/len(BURST):.1f}")
print(f"  max over the window  : {max(BURST)}")
# raw samples          : [2, 3, 2, 980, 4, 3, 2, 3]
# mean over the window : 124.9
# max over the window  : 980
#
# A rollup keeping only the MEAN erases the burst. Capacity must cover PEAKS, so
# retain MAX and a percentile alongside the mean -- a default set once and
# regretted eighteen months later when the data to justify an upgrade is absent.

print("\nThe four questions every network monitor must answer:")
for question, signal, cadence in (
    ("is it up?",          "reachability probe / interface oper-status", "on change"),
    ("is it healthy?",     "errors, drops, discards", "10-60s"),
    ("is it fast enough?", "per-class latency and loss probes", "10-30s"),
    ("will it be enough?", "utilisation peaks, rolled to daily max", "5min -> daily"),
):
    print(f"  {question:22s} {signal:42s} {cadence}")
print("\nThe cadences differ by two orders of magnitude. Collecting everything at")
print("the fastest cadence is how a monitoring system becomes the outage.")

print("\nCounters vs gauges -- using the wrong model produces nonsense:")
for metric, kind, correct_use in (
    ("interface bytes", "counter (monotonic, wraps)", "rate = delta / interval"),
    ("interface errors", "counter", "rate; a total since boot is meaningless"),
    ("queue depth", "gauge (instantaneous)", "a point sample; peaks between samples are LOST"),
    ("CPU utilisation", "gauge (already an average)", "do not average an average naively"),
    ("session table count", "gauge", "compare against the MAXIMUM, not against zero"),
):
    print(f"  {metric:22s} {kind:28s} {correct_use}")

# --- Collect at the cadence the question needs, not one cadence for everything ---
# Fast, on-change: state transitions (13.5 covered the transport)
gnmic -a leaf-1:57400 --skip-verify subscribe --stream-mode on-change \
  --path "/interfaces/interface/state/oper-status"
# Medium, sampled: health counters
gnmic -a leaf-1:57400 --skip-verify subscribe --stream-mode sample \
  --sample-interval 30s --path "/interfaces/interface/state/counters"

# --- Configure rollups to keep MAX, not just mean (Prometheus recording rules) ---
cat <<'YML'
groups:
  - name: capacity-rollups
    interval: 5m
    rules:
      # The mean is for dashboards. The MAX is what capacity planning needs.
      - record: link:utilisation:mean5m
        expr: avg_over_time(link_utilisation_ratio[5m])
      - record: link:utilisation:max5m
        expr: max_over_time(link_utilisation_ratio[5m])
      - record: link:utilisation:p95_5m
        expr: quantile_over_time(0.95, link_utilisation_ratio[5m])
YML
#  Without the max5m rule, next year's upgrade justification does not exist.

# --- Verify you are computing RATES from counters, not reading totals ---
# Wrong: the total since boot, which never means anything on its own
cat /sys/class/net/eth0/statistics/rx_errors
# Right: a delta over a known interval
a=$(cat /sys/class/net/eth0/statistics/rx_errors); sleep 60
b=$(cat /sys/class/net/eth0/statistics/rx_errors)
echo "errors/minute: $((b-a))"
# In PromQL: rate(node_network_receive_errs_total[5m])  -- never the raw counter

# --- MONITOR THE MONITORING: staleness is indistinguishable from health ---
cat <<'YML'
      - alert: MetricsStale
        expr: time() - timestamp(up{job="network"}) > 300
        for: 5m
        annotations:
          summary: "No samples from {{ $labels.instance }} for over 5 minutes"
          # A flat line with no alerts looks exactly like a healthy network.
      - alert: ScrapeFailing
        expr: up{job="network"} == 0
        for: 2m
YML

# --- Sanity check: does the collector actually have what you think? ---
curl -s localhost:9090/api/v1/status/tsdb | jq '.data.numSeries'
curl -s localhost:9090/api/v1/query --data-urlencode \
  'query=count(count by (instance)(up{job="network"}))' | jq -r '.data.result[0].value[1]'
#  Compare against your device inventory. A device that was never added to the
#  scrape config is invisible, and nothing will ever alert about it.

Key takeaways

• Section 13.5 covers transport; the operational questions are what to collect, at what cadence, and for how long.
• Match cadence to the question: on-change for state, tens of seconds for health, five minutes rolled to daily for capacity.
• Collecting everything at the fastest cadence is how a monitoring system becomes the outage.
• Retention is tiered because each tier answers a different class of question, and raw data is by far the most expensive.
• A rollup that keeps only the mean destroys the peaks, and peaks are exactly what capacity planning needs. Retain max and a percentile too.
• Distinguish counters (monotonic, use rates and deltas) from gauges (instantaneous, peaks between samples are lost).
• A total since boot is meaningless; always compute a rate over a known interval.
• Monitor the monitoring. A stopped collector looks exactly like a healthy network, so alert on staleness and on devices missing from the inventory.

🧪 Practice

1. For 600 devices with 24 interfaces and 8 counters, compute the series count and the storage for 14 days at 30-second resolution at 16 bytes per sample.
2. Explain what is lost when a five-minute rollup keeps only the mean, and write the two additional aggregations you would retain.
3. Give the correct way to alert on interface errors, and explain why alerting on the raw counter value is wrong.
4. Interview-style: A monitoring dashboard has shown flat lines and zero alerts for three days, and the team reports that the network has never been more stable. (Hint: name the one alert that distinguishes the two possible explanations, and say why it is missing from most initial deployments.)

NetFlow, sFlow, and IPFIX

Theory

Metrics tell you a link is 94% utilised. They cannot tell you what the traffic is or who is sending it, because a counter has no idea what passed through it. Flow data answers those questions, and it is the right tool for a specific and important set of them.

NetFlow and its standardised successor IPFIX work by having the device maintain a table of active flows, keyed by the 5-tuple plus interface and marking, accumulating byte and packet counts, and exporting a record when the flow ends or a timer expires. The counts are accurate, because they come from counting every packet. The cost is device resources: the flow table consumes memory and CPU, and it can be exhausted by traffic with many tiny flows — which is precisely what a port scan or a denial-of-service attack looks like, so the mechanism degrades exactly when you need it.

sFlow takes the opposite approach: sample one packet in N, export the sample header with interface counters, and let the collector extrapolate. Overhead is constant and predictable regardless of traffic level, and it cannot be exhausted. The cost is statistical: the accuracy of any estimate depends on how many samples it is based on, not on how large the flow is.

That last point is the one worth internalising, because it defines what sampled data can be used for. The relative error of a sampled count goes as one over the square root of the number of samples. A flow contributing ten thousand samples is estimated to within about one percent; a flow contributing ten samples has an error over thirty percent; a flow contributing one sample carries no information at all. So sampled flow data is accurate for aggregates and unreliable for individual small flows. It answers "who are the top talkers on this saturated link?" extremely well and cannot answer "did this eight-packet connection happen?" at all.

IPFIX deserves a separate mention because it is templated and extensible. Beyond the classic fields it can carry MPLS labels, VXLAN identifiers, application labels, and vendor-specific data, which matters in the overlay environments of Chapter 13 where the 5-tuple alone does not identify a tenant. The cost is that the collector must understand the templates the exporter sends.

The questions flow data answers that nothing else can:

• Who are the top talkers on a saturated link?
• Which application is the new traffic that appeared last week?
• Did this host communicate with that host, and when?
• What would break if we blocked this port?
• Which prefix is an attack targeting?

And the questions it cannot: anything sub-second, and anything about a single small flow under sampling. Those need packet capture.

Examples

import math
# Flow sampling: what 1-in-N sampling can and cannot tell you.
def relative_error(sampled_count):
    """Relative error of a count estimated from `sampled_count` samples."""
    return 1 / math.sqrt(sampled_count) if sampled_count > 0 else float("inf")

print("sFlow 1-in-N: accuracy depends on the number of SAMPLES, not flow size")
print(f"{'flow packets':>13s} {'N':>7s} {'sampled':>8s} {'rel error':>10s}  verdict")
for pkts, N in ((10_000_000, 1000), (1_000_000, 1000), (100_000, 1000),
                (10_000, 1000), (1_000, 1000), (1_000, 100)):
    sampled = pkts / N
    err = relative_error(sampled)
    verdict = ("excellent" if err < 0.05 else "usable" if err < 0.20
               else "MEANINGLESS for this flow")
    print(f"{pkts:13,} {N:7,} {sampled:8.0f} {err:9.1%}  {verdict}")
#  flow packets       N  sampled  rel error  verdict
#    10,000,000   1,000    10000      1.0%  excellent
#     1,000,000   1,000     1000      3.2%  excellent
#       100,000   1,000      100     10.0%  usable
#        10,000   1,000       10     31.6%  MEANINGLESS for this flow
#         1,000   1,000        1    100.0%  MEANINGLESS for this flow
#         1,000     100       10     31.6%  MEANINGLESS for this flow
#
# Note the last two rows: the SAME flow at a finer sampling ratio is still
# unreliable. Sampling accuracy is bounded by sample count, and a small flow
# cannot produce enough samples at any practical ratio.

PPS, FLOW_RECORD_BYTES = 2_000_000, 60
print("\nWhy sampling exists: worst-case record volume at 2 Mpps")
for label, ratio in (("one record per packet (worst case)", 1),
                     ("1-in-100", 100), ("1-in-1000", 1000), ("1-in-10000", 10000)):
    recs = PPS / ratio
    print(f"  {label:36s} {recs:12,.0f} rec/s  "
          f"{recs*FLOW_RECORD_BYTES*86400/1e9:8.1f} GB/day")
#   one record per packet (worst case)      2,000,000 rec/s   10368.0 GB/day
#   1-in-100                                   20,000 rec/s     103.7 GB/day
#   1-in-1000                                   2,000 rec/s      10.4 GB/day
#   1-in-10000                                    200 rec/s       1.0 GB/day
#
# NetFlow exports one record per FLOW, not per packet, so it is normally far
# cheaper than the first row -- but on a link dominated by short flows the flow
# rate approaches the packet rate and the two converge. That is also exactly
# when the device's flow table is most likely to be exhausted.

print("\nChoosing between them, by what you need:")
for name, what, strength, weakness in (
    ("NetFlow / IPFIX", "per-flow records, aggregated on the device",
     "accurate byte and packet counts per flow",
     "device CPU and memory; flow-table exhaustion under scans"),
    ("sFlow", "1-in-N packet samples plus interface counters",
     "constant, predictable overhead at any traffic level",
     "statistically unreliable for small flows"),
    ("IPFIX (templated)", "extensible fields via templates",
     "can carry MPLS labels, VXLAN IDs, application labels",
     "the collector must understand the templates"),
):
    print(f"  {name:18s} {what}")
    print(f"  {'':18s} + {strength}")
    print(f"  {'':18s} - {weakness}")

print("\nQuestions flow data answers that metrics cannot:")
for q in ("who are the top talkers on this saturated link?",
          "which application is the new traffic that appeared last week?",
          "did this host talk to that host, and when?",
          "what would we break by blocking this port?",
          "which prefix is the attack targeting?"):
    print(f"  - {q}")
print("\nAnd what it cannot answer: anything sub-second, and anything about a")
print("single small flow under sampling. Those need a packet capture.")

# --- Configure export (vendor-neutral shapes; the concepts are identical) ---
# IPFIX with active/inactive timeouts: shorter timeouts mean fresher data and
# more records. 60s active is a common compromise.
#   flow exporter EXP   destination 10.0.9.20   transport udp 4739
#   flow record REC     match ipv4 source address
#                       match ipv4 destination address
#                       match transport source-port
#                       match transport destination-port
#                       match ipv4 protocol
#                       match ipv4 dscp                 <- validates Chapter 12
#                       collect counter bytes
#                       collect counter packets
#                       collect interface input
#   flow monitor MON    record REC  exporter EXP
#                       cache timeout active 60  inactive 15
#   interface Gi1/0/1   ip flow monitor MON input

# sFlow: one sampling rate, predictable overhead, cannot be exhausted
#   sflow sampling-rate 2048
#   sflow polling-interval 30
#   sflow destination 10.0.9.20 6343

# --- Watch for flow-table exhaustion, which happens exactly during an attack ---
#   show flow monitor MON cache | include Entries|Added|Failed
#  Flow entries: 63488 / 65536      <- 97% full
#  Failed adds: 184221              <- flows NOT recorded: your data has holes
# This is why a scan or DDoS both floods the link and blinds the flow telemetry.

# --- Answer the real questions with the collected data ---
nfdump -R /var/flows -t 2024/09/24.14:00-2024/09/24.14:30 \
  -s srcip/bytes -n 10
#  Top talkers during the incident window: the question metrics cannot answer.
nfdump -R /var/flows -s port/bytes -n 10          # which application?
nfdump -R /var/flows 'dst port 5432 and bytes > 10000000' -o long | head
nfdump -R /var/flows -s record/bytes -n 20 'proto udp and src port 53'

# Compare this week against last week: what is the NEW traffic?
for w in 'last week' 'this week'; do
  echo "=== $w ==="
  nfdump -R /var/flows -s port/bytes -n 5 2>/dev/null | tail -6
done

# --- Correlate flow data with the metric that raised the alarm ---
# The link is 94% utilised (metrics). Flow data says WHAT it is:
nfdump -R /var/flows -t '2024/09/24.14:00-2024/09/24.14:05' -s proto/bytes
nfdump -R /var/flows -t '2024/09/24.14:00-2024/09/24.14:05' -s dstip/bytes -n 5
#  One destination taking 70% of a saturated link is a decision you can act on;
#  "the link is busy" is not.

# --- Validate a QoS policy from flow records (Chapter 12's markings) ---
nfdump -R /var/flows -s dscp/bytes
#  Traffic appearing in the wrong DSCP class here means marking is not working,
#  and this is far cheaper than a packet capture to establish it.

Key takeaways

• Metrics say a link is busy; flow data says what the traffic is and who sent it.
• NetFlow and IPFIX count every packet per flow, giving accurate counts at the cost of device memory and CPU, and the flow table can be exhausted by many tiny flows — exactly what a scan or attack produces.
• sFlow samples one packet in N with constant, predictable overhead that cannot be exhausted.
• Sampled accuracy depends on sample count, not flow size: relative error goes as one over the square root of samples.
• Therefore sampled data is accurate for aggregates and unreliable for small flows; a flow with one sample carries no information.
• IPFIX templates can carry MPLS labels, VXLAN identifiers, and application labels, which matters when the 5-tuple does not identify a tenant.
• Flow data answers top talkers, what changed, who spoke to whom, and blast radius; it cannot answer sub-second questions or anything about one small sampled flow.
• Flow records can validate QoS markings far more cheaply than a capture.
• Monitor failed flow-table inserts; holes in the data appear precisely during the events you most want to analyse.

🧪 Practice

1. At 1-in-4096 sampling, compute the number of samples and relative error for flows of 40 million, 400,000, and 4,000 packets, and state which estimates you would report.
2. Explain why NetFlow degrades during a port scan and sFlow does not, and what this implies for choosing between them on an internet-facing link.
3. A link alerts at 95% utilisation. Write the sequence of flow queries you would run to decide whether to add capacity or block something.
4. Interview-style: An analyst uses 1-in-8192 sampled flow data to prove that a specific 12-packet connection between two hosts never occurred. (Hint: compute the expected number of samples such a flow would produce, then say what the absence of samples does and does not establish.)

Baselines, Thresholds, and Alerting

Theory

An alert exists to cause a human to act. Everything about its design follows from that, and the failure mode is not a missed alert but an ignored one — a system producing four hundred alerts a day of which four matter has the same operational value as no monitoring, plus the false confidence that something is being watched.

The first problem is that static thresholds do not fit seasonal data. Network traffic has a daily shape and a weekly shape. A threshold set high enough not to fire during weekday business hours is far above anything that could happen at the weekend, so a genuine weekend anomaly — five times the weekend norm — passes unremarked. A threshold set to catch that fires constantly on Tuesday afternoons. One number cannot serve both, because the legitimate range differs by a large factor across the week.

The fix is a seasonal baseline: compare each observation against the distribution for that hour of that day type, and alert on deviation from its own history. This handles both cases with one rule, and it fails in one specific way worth knowing — a baseline learned from a period that included an incident treats the incident as normal, so baselines need either a long window or explicit exclusion of known-bad periods.

The second problem is alert quality, which is measurable. Precision is the fraction of alerts that represent something real. Below roughly five percent, alerts are ignored within a week. Above about two thirds, they are investigated. The implication is direct and frequently resisted: halving the alert count while keeping the real ones is worth more than any new dashboard, because it moves alerts from ignored to actioned.

Four properties make an alert actionable, and all four are required:

Property	Bad	Good
Symptom, not cause	"CPU at 85% on leaf-4"	"site-C API error rate 14% for 5 minutes"
Impact stated	"interface errors rising"	"140 users at site-C affected"
A runbook	none	a link to the procedure
Duration and hysteresis	fires on one sample	fires after 5 minutes, clears after 10

The symptom-not-cause rule deserves emphasis because it inverts the instinct. Alerting on causes produces one alert per possible cause, most of which are harmless most of the time: high CPU is often fine. Alerting on symptoms produces one alert per actual user-visible problem, with causes available as context when someone investigates. The standard formulation is to page on what users experience and to leave everything else as a dashboard or a ticket.

Hysteresis is the cheapest large improvement available. A threshold with no duration requirement fires on a single noisy sample and clears on the next, producing flapping pairs that train people to ignore the source. Requiring a condition to persist, and requiring a longer period below the threshold before clearing, eliminates most of that noise at no cost in detection of real events — real events persist.

Examples

import math, random, statistics
random.seed(11)
# Why a STATIC threshold fails on seasonal data, and what replaces it.
def traffic(hour, weekday):
    """A realistic daily and weekly shape."""
    base = 120 if weekday else 40
    peak = 1.0 + 1.6 * math.exp(-((hour - 14) ** 2) / 18)   # business-hours peak
    return base * peak * random.uniform(0.92, 1.08)

WEEK = [(d, h, traffic(h, d < 5)) for d in range(7) for h in range(24)]
values = [v for _, _, v in WEEK]
print(f"a week of hourly samples: min {min(values):6.1f} "
      f"mean {statistics.mean(values):6.1f} max {max(values):6.1f} Mb/s\n")

STATIC = 250.0
weekend_fires = sum(1 for d, h, v in WEEK if v > STATIC and d >= 5)
weekday_fires = sum(1 for d, h, v in WEEK if v > STATIC and d < 5)
print(f"static threshold at {STATIC:.0f} Mb/s:")
print(f"  fires {weekend_fires + weekday_fires} times a week: "
      f"{weekday_fires} on weekdays, {weekend_fires} at weekends")
print(f"  every weekday firing is ordinary business-hours traffic, so all")
print(f"  {weekday_fires} are FALSE ALARMS -- and a genuine weekend spike of")
print(f"  200 Mb/s (5x the weekend norm) would never fire at all.\n")
# a week of hourly samples: min   37.0 mean  145.8 max  335.2 Mb/s
#
# static threshold at 250 Mb/s:
#   fires 29 times a week: 29 on weekdays, 0 at weekends
#   every weekday firing is ordinary business-hours traffic, so all
#   29 are FALSE ALARMS -- and a genuine weekend spike of
#   200 Mb/s (5x the weekend norm) would never fire at all.

by_slot = {}
for d, h, v in WEEK:
    by_slot.setdefault((d < 5, h), []).append(v)

def seasonal_threshold(is_weekday, hour, sigmas=3.0):
    """Compare each sample against the distribution for ITS OWN slot."""
    xs = by_slot[(is_weekday, hour)]
    mu = statistics.mean(xs)
    sd = statistics.pstdev(xs) if len(xs) > 1 else mu * 0.1
    return mu + sigmas * max(sd, mu * 0.05)      # floor the deviation estimate

print("seasonal threshold (mean + 3 sigma for THAT hour of THAT day type):")
for is_wd, hour in ((True, 3), (True, 14), (False, 3), (False, 14)):
    print(f"  {'weekday' if is_wd else 'weekend':8s} {hour:02d}:00 -> "
          f"alert above {seasonal_threshold(is_wd, hour):6.1f} Mb/s")
print(f"  One threshold cannot serve both ends: the ratio between weekday 14:00")
print(f"  and weekend 03:00 is about "
      f"{seasonal_threshold(True,14)/seasonal_threshold(False,3):.1f}x.\n")
#   weekday  03:00 -> alert above  137.8 Mb/s
#   weekday  14:00 -> alert above  371.4 Mb/s
#   weekend  03:00 -> alert above   48.3 Mb/s
#   weekend  14:00 -> alert above  116.1 Mb/s
#   ratio between weekday 14:00 and weekend 03:00 is about 7.7x.
#
# A baseline learned from a window CONTAINING an incident treats the incident as
# normal, so use a long window or exclude known-bad periods explicitly.

print("Alert quality arithmetic -- the number that decides if alerts get read:")
print(f"{'alerts/day':>11s} {'real/day':>9s} {'precision':>10s}  outcome")
for alerts, real in ((400, 4), (60, 6), (12, 8), (4, 4)):
    p = real / alerts
    outcome = ("ignored within a week" if p < 0.05 else
               "skimmed, real ones missed" if p < 0.25 else
               "investigated" if p < 0.8 else "trusted completely")
    print(f"{alerts:11d} {real:9d} {p:9.1%}  {outcome}")
#  alerts/day  real/day  precision  outcome
#         400         4      1.0%  ignored within a week
#          60         6     10.0%  skimmed, real ones missed
#          12         8     66.7%  investigated
#           4         4    100.0%  trusted completely
#
# Halving the alert count while keeping the real ones moves alerts from IGNORED
# to ACTIONED, which is worth more than any new dashboard. An alert nobody reads
# is worse than no alert, because it creates the belief of being watched.

print("\nWhat makes an alert actionable -- four properties, all required:")
for prop, bad, good in (
    ("symptom, not cause", "CPU at 85% on leaf-4",
     "site-C API error rate 14% for 5 minutes"),
    ("user impact stated", "interface Gi1/0/14 errors rising",
     "140 users at site-C affected"),
    ("a runbook link", "(none)", "-> wiki/runbooks/site-link-loss"),
    ("duration/hysteresis", "fires on one bad sample",
     "fires after 5 min above, clears after 10 min below"),
):
    print(f"  {prop:20s} bad:  {bad}")
    print(f"  {'':20s} good: {good}")

# --- Seasonal comparison instead of a static threshold (PromQL) ---
cat <<'YML'
groups:
  - name: network-alerts
    rules:
      # BAD: one static number for a signal with a 7x daily/weekly range.
      # - alert: LinkBusy
      #   expr: link_utilisation_ratio > 0.80

      # BETTER: compare against the SAME time last week, which carries the
      # daily and weekly shape with it.
      - alert: LinkUnusuallyBusy
        expr: |
          link_utilisation_ratio
            > 1.5 * (link_utilisation_ratio offset 1w)
          and link_utilisation_ratio > 0.50
        for: 10m                       # HYSTERESIS: must persist
        labels: {severity: ticket}
        annotations:
          summary: "{{ $labels.device }}/{{ $labels.link }} 50% above last week"
          runbook: "https://wiki/runbooks/link-utilisation"

      # BEST for paging: alert on the SYMPTOM users experience.
      - alert: SiteApiErrors
        expr: |
          sum by (site) (rate(http_requests_total{code=~"5.."}[5m]))
            / sum by (site) (rate(http_requests_total[5m])) > 0.05
        for: 5m
        labels: {severity: page}
        annotations:
          summary: "{{ $labels.site }}: {{ $value | humanizePercentage }} of requests failing"
          impact: "users at {{ $labels.site }} cannot complete orders"
          runbook: "https://wiki/runbooks/site-api-errors"
YML

# --- Error-rate alerts need a traffic floor, or they fire on 1-of-2 requests ---
#   and sum by (site) (rate(http_requests_total[5m])) > 1
# Without it, a quiet site with two requests and one error alerts at 50%.

# --- Measure your own alert quality. This number decides everything else. ---
# Count alerts fired vs alerts that led to action, over 30 days.
curl -s 'localhost:9090/api/v1/query?query=ALERTS{alertstate="firing"}' \
  | jq -r '.data.result[].metric.alertname' | sort | uniq -c | sort -rn
#  312 LinkUnusuallyBusy     <- the loudest alert: is it ever actioned?
#   48 HighCPU
#    4 SiteApiErrors
# Cross-reference against your ticket system: any alert with zero resulting
# tickets in 30 days should be deleted or downgraded to a dashboard.

# --- Find flapping alerts, which train people to ignore the source ---
curl -s 'localhost:9090/api/v1/query_range?query=ALERTS&start=...&step=60' \
  | jq -r '...' # count state transitions per alertname
# An alert with 40 transitions and 0 tickets needs a `for:` clause, not a page.

# --- Before adding an alert, answer these three questions ---
#  1. what will the recipient DO when it fires?      (if nothing: dashboard)
#  2. what user-visible symptom does it represent?   (if none: dashboard)
#  3. how will they know it has cleared?             (define the clear condition)

Key takeaways

• An alert exists to make a human act; the failure mode is an ignored alert, not a missed one.
• Static thresholds cannot fit seasonal data: one number either floods on weekday peaks or misses genuine weekend anomalies, and the legitimate range can vary severalfold.
• Use a seasonal baseline comparing each observation against its own hour-of-week history, and beware baselines learned from windows containing incidents.
• Alert precision is measurable and decisive. Below about five percent, alerts are ignored within a week.
• Halving the alert count while keeping the real ones is worth more than any new dashboard.
• Four required properties: symptom not cause, stated user impact, a runbook, and duration with hysteresis.
• Page on what users experience; leave causes as context and dashboards. Alerting on causes produces one alert per harmless condition.
• Hysteresis is the cheapest big improvement: real events persist, noise does not.
• Add a traffic floor to ratio alerts, or a quiet endpoint alerts on one failure out of two requests.
• Audit alerts against tickets: any alert with no resulting action in a month should be deleted or downgraded.

🧪 Practice

1. Explain why a single utilisation threshold cannot serve both weekday-peak and overnight traffic, using specific numbers for each period.
2. A system fires 180 alerts a day of which 6 are real. Compute precision, state the likely outcome, and give two changes that would improve it.
3. Rewrite "CPU above 90% on core-router-2" as a symptom-based alert with impact, duration, and a clear condition.
4. Interview-style: A team adds forty new alerts after an incident to ensure it is never missed again, and six months later a similar incident goes unnoticed for an hour. (Hint: relate the number of alerts to the fraction that are actionable, and consider what happens to human attention when that fraction falls.)

Availability, MTBF, and MTTR

Theory

Section 12.1 introduced availability arithmetic in the context of service level agreements. The operational use is different and rests on three results that are individually simple and collectively counterintuitive.

A path is a series chain, so adding components makes availability worse. Every device a packet must traverse multiplies into the total, and the product is always lower than the least available component. A chain of six components each above 99.5% can land below 99.3%, which is over five hours of monthly downtime. The practical consequence is that "add another firewall for safety" can reduce availability, and that shortening the path is an availability improvement — a conclusion that surprises people who think of each device as protective.

Redundancy multiplies the failure probability, so parallel components help dramatically — but only if failures are independent. Two parallel circuits at 99.5% give 99.9975% under independence, which is about one minute a month rather than three and a half hours. That number is also where most availability estimates go wrong, because the independence assumption is usually false. If twenty percent of failures affect both copies at once, two circuits deliver barely more than 99.9%. Shared conduit, shared power, shared provider, shared software version, and shared configuration are all that correlation, and the last is the easiest to overlook: two devices configured from the same template fail together when the template is wrong.

Reducing repair time is usually cheaper than improving reliability, and worth the same. Availability is MTBF / (MTBF + MTTR), so halving MTTR improves availability exactly as much as doubling MTBF. Doubling MTBF means better hardware and more redundancy; halving MTTR means better monitoring, spares on site, runbooks, and practice. The second list is much cheaper and it also improves every future failure mode, including ones you have not anticipated.

The operational framing that ties these together is the error budget: the allowed downtime implied by a target. A 99.9% monthly target is about 43 minutes. What makes this useful is not the total but the burn rate — the fraction of budget consumed against the fraction of the period elapsed. A burn rate below one is healthy; above two means the budget will be exhausted before the period ends. This converts availability from a retrospective score into a live signal about whether to keep shipping changes or stop and stabilise, and it is the mechanism by which reliability targets become decisions rather than reports.

Examples

# Availability arithmetic for OPERATIONS: series, parallel, and error budgets.
SEC_PER_MONTH = 30 * 24 * 3600

def downtime_str(avail):
    s = SEC_PER_MONTH * (1 - avail)
    if s < 60:
        return f"{s:.1f}s"
    return f"{int(s//3600)}h {int(s%3600//60)}m {int(s%60)}s"

def series(*avails):
    """ALL required: availabilities multiply. Adding parts makes it WORSE."""
    p = 1.0
    for a in avails:
        p *= a
    return p

def parallel(avail, n):
    """n redundant components, any one suffices."""
    return 1 - (1 - avail) ** n

CHAIN = {"access switch": 0.9995, "distribution": 0.9998, "firewall": 0.9990,
         "router": 0.9995, "WAN circuit": 0.9950, "provider core": 0.9999}
print("A path is a SERIES chain: every component must work.")
running = 1.0
for name, a in CHAIN.items():
    running = series(running, a)
    print(f"  + {name:18s} {a:.4f} -> path {running:.6f} ({downtime_str(running)})")
print(f"\n  worst single link {min(CHAIN.values()):.4f} alone = "
      f"{downtime_str(min(CHAIN.values()))}; the CHAIN = {downtime_str(running)}")
#   + access switch      0.9995 -> path 0.999500 (0h 21m 35s)
#   + distribution       0.9998 -> path 0.999300 (0h 30m 14s)
#   + firewall           0.9990 -> path 0.998301 (1h 13m 24s)
#   + router             0.9995 -> path 0.997802 (1h 34m 58s)
#   + WAN circuit        0.9950 -> path 0.992813 (5h 10m 29s)
#   + provider core      0.9999 -> path 0.992713 (5h 14m 46s)
#
#   worst single link 0.9950 alone = 3h 36m 0s; the CHAIN = 5h 14m 46s
#
# Every component ADDED to a path makes availability worse. "Add another
# firewall for safety" can reduce availability, and SHORTENING a path improves it.

print("\nRedundancy: parallel components, and the diminishing return.")
for name, a in (("WAN circuit", 0.9950), ("firewall", 0.9990)):
    for n in (1, 2, 3):
        av = parallel(a, n)
        print(f"  {name:12s} x{n} -> {av:.8f}  ({downtime_str(av)})")
    print()
#   WAN circuit  x1 -> 0.99500000  (3h 36m 0s)
#   WAN circuit  x2 -> 0.99997500  (0h 1m 4s)
#   WAN circuit  x3 -> 0.99999987  (0.3s)
#
#   firewall     x1 -> 0.99900000  (0h 43m 12s)
#   firewall     x2 -> 0.99999900  (2.6s)
#   firewall     x3 -> 1.00000000  (0.0s)
# The second unit is transformative; the third is almost free of benefit.

print("Parallel arithmetic assumes INDEPENDENCE. Correlation dominates:")
def parallel_correlated(avail, n, correlation):
    """`correlation` = fraction of failures that hit all copies at once."""
    independent = (1 - avail) * (1 - correlation)
    common = (1 - avail) * correlation
    return 1 - (independent ** n + common)

for corr in (0.0, 0.05, 0.20, 0.50):
    av = parallel_correlated(0.9950, 2, corr)
    print(f"  2 circuits, {corr:.0%} correlated -> {av:.6f} ({downtime_str(av)})")
#   2 circuits, 0% correlated -> 0.999975 (0h 1m 4s)
#   2 circuits, 5% correlated -> 0.999727 (0h 11m 46s)
#   2 circuits, 20% correlated -> 0.998984 (0h 43m 53s)
#   2 circuits, 50% correlated -> 0.997494 (1h 48m 16s)
#
# At 20% correlation two circuits give barely more than 99.9%, not the 99.9975%
# the independent model promises. Shared conduit, power, provider, software
# version or CONFIGURATION TEMPLATE all ARE that correlation.

print("\nError budgets: the operational form of an availability target.")
for target in (0.99, 0.999, 0.9995, 0.9999):
    print(f"  target {target:.4%} -> budget "
          f"{SEC_PER_MONTH*(1-target)/60:8.1f} minutes/month")
TARGET, ELAPSED = 0.999, 0.40       # 40% of the way through the month
budget_s = SEC_PER_MONTH * (1 - TARGET)
print()
for used_min in (5, 17, 26, 45):
    used_frac = used_min * 60 / budget_s
    burn = used_frac / ELAPSED
    verdict = ("healthy" if burn < 1 else
               "on track to miss" if burn < 2 else "BUDGET EXHAUSTED SOON")
    print(f"  used {used_min:3d} of {budget_s/60:.0f} min ({used_frac:5.1%}) at "
          f"{ELAPSED:.0%} elapsed -> burn {burn:4.2f}x  {verdict}")
#   used   5 of 43 min (11.6%) at 40% elapsed -> burn 0.29x  healthy
#   used  17 of 43 min (39.4%) at 40% elapsed -> burn 0.98x  healthy
#   used  26 of 43 min (60.2%) at 40% elapsed -> burn 1.50x  on track to miss
#   used  45 of 43 min (104.2%) at 40% elapsed -> burn 2.60x  BUDGET EXHAUSTED SOON
#
# Burn RATE, not raw downtime, is the live signal: it says whether to keep
# shipping changes or to stop and stabilise.

print("\nMTBF, MTTR, and where to invest:")
def avail_from(mtbf_h, mttr_h):
    return mtbf_h / (mtbf_h + mttr_h)
base = avail_from(2000, 4)
print(f"  baseline       MTBF 2000h MTTR 4.0h -> {base:.5f} ({downtime_str(base)})")
for label, mtbf, mttr in (("halve MTTR", 2000, 2.0), ("double MTBF", 4000, 4.0),
                          ("MTTR to 30 min", 2000, 0.5)):
    a = avail_from(mtbf, mttr)
    print(f"  {label:14s} MTBF {mtbf}h MTTR {mttr}h -> {a:.5f} ({downtime_str(a)})")
#   baseline       MTBF 2000h MTTR 4.0h -> 0.99800 (1h 26m 13s)
#   halve MTTR     MTBF 2000h MTTR 2.0h -> 0.99900 (0h 43m 9s)
#   double MTBF    MTBF 4000h MTTR 4.0h -> 0.99900 (0h 43m 9s)
#   MTTR to 30 min MTBF 2000h MTTR 0.5h -> 0.99975 (0h 10m 47s)
#
# Halving repair time equals doubling reliability, and costs monitoring, spares,
# runbooks and practice rather than better hardware -- and it improves every
# future failure mode, including ones nobody anticipated.

# --- Measure MTTR honestly: it starts at ONSET, not at acknowledgement ---
# Pull incident timestamps and compute each interval separately.
# detect = onset -> alert;  respond = alert -> ack;  repair = ack -> resolved
cat <<'SQL'
SELECT id,
       EXTRACT(EPOCH FROM (alerted_at  - started_at ))/60 AS detect_min,
       EXTRACT(EPOCH FROM (acked_at    - alerted_at ))/60 AS respond_min,
       EXTRACT(EPOCH FROM (resolved_at - acked_at   ))/60 AS repair_min,
       EXTRACT(EPOCH FROM (resolved_at - started_at ))/60 AS total_min
FROM incidents WHERE started_at > now() - interval '90 days' ORDER BY total_min DESC;
SQL
#  If detect_min dominates, the fix is MONITORING, not faster engineers.
#  If respond_min dominates, the fix is paging and on-call, not tooling.
#  If repair_min dominates, the fix is runbooks, spares and automation.
# Reporting a single MTTR number hides which of the three to work on.

# --- Verify redundancy INDEPENDENCE, which the arithmetic assumes ---
# Two circuits are only independent if every shared element is different.
for c in CKT-88214 CKT-91007; do
  echo "=== $c ==="
  # provider, physical path, entry point, local router, local power feed, optic batch
done
#  Same duct, same building entry, same PDU, same provider, same software
#  version, or the same configuration template all create correlation.
# The configuration one is easiest to miss and easiest to test:
diff <(ssh edge-1 'show running-config') <(ssh edge-2 'show running-config')
#  Identical configs fail identically. That is a shared failure mode.

# --- Error-budget burn as an alert, so it drives decisions ---
cat <<'YML'
      - alert: ErrorBudgetBurnFast
        expr: |
          (1 - (sum(rate(probe_success[1h])) / sum(rate(probe_total[1h]))))
            > 14.4 * 0.001
        for: 5m
        labels: {severity: page}
        annotations:
          summary: "Burning the monthly error budget 14x too fast"
          # 14.4x for 1h consumes 2% of a 30-day budget: page now.
      - alert: ErrorBudgetBurnSlow
        expr: |
          (1 - (sum(rate(probe_success[6h])) / sum(rate(probe_total[6h]))))
            > 6 * 0.001
        for: 30m
        labels: {severity: ticket}
YML
# Two burn rates: a fast one that pages, a slow one that opens a ticket. This is
# what turns an availability target into an operational decision.

# --- The availability number that matters is measured from the USER's side ---
# Internal checks pass while users fail. Probe from where users are.
for site in site-a site-b site-c; do
  ssh $site "curl -sS -o /dev/null -m5 -w '%{http_code} %{time_total}\n' \
    https://api.example.com/health" 2>/dev/null || echo "$site UNREACHABLE"
done

Key takeaways

• A path is a series chain: availabilities multiply, so the path is always worse than its weakest component and adding devices makes it worse.
• Shortening a path is an availability improvement; "one more firewall for safety" can reduce availability.
• Redundancy is transformative but assumes independence. Two 99.5% circuits give 99.9975% independently and barely 99.9% at 20% correlation.
• Correlation comes from shared conduit, power, provider, software version, and shared configuration templates — identical configs fail identically.
• The second redundant unit delivers nearly all the benefit; the third delivers almost none.
• Halving MTTR equals doubling MTBF and is far cheaper: monitoring, spares, runbooks, practice.
• Decompose MTTR into detect, respond, and repair. A single number hides which of the three to fix.
• An error budget converts a target into allowed downtime, and burn rate converts it into a live decision about whether to keep shipping.
• Measure availability from where users are, not from inside the infrastructure.

🧪 Practice

1. Compute the end-to-end availability of a path with components at 99.99%, 99.95%, 99.9%, and 99.5%, and express the result as monthly downtime.
2. Two links are each 99.7% available. Compute the pair's availability under independence and under 25% correlated failure, and list four sources of that correlation.
3. A service targets 99.95% monthly. Compute the error budget in minutes, and the burn rate if 9 minutes are consumed 25% of the way through the month.
4. Interview-style: A team reports a 90-minute MTTR and proposes buying more reliable hardware to improve availability. (Hint: the single number hides three intervals — name them, and say which purchase helps if the largest one is the first.)

Traffic Growth Forecasting

Theory

Capacity planning is the part of operations that prevents incidents rather than resolving them, and it fails in two characteristic ways: fitting the wrong growth model, and forgetting that capacity takes time to acquire.

The model matters because extrapolation amplifies the choice. Network traffic usually grows by compounding — more users, each doing more — which is exponential, not linear. Over the historical window both models may fit well, because any smooth curve looks linear over a short enough span. They diverge sharply in extrapolation, and the divergence is the risk: choosing linear when growth is compounding means capacity runs out considerably earlier than planned. The discipline is to fit both, compare the goodness of fit, and use the better one, while remembering that neither is a law of nature and both should be refitted as data accumulates.

Forecast from peaks, not averages. Capacity must cover the busy period, so the input series should be the daily or weekly maximum rather than the mean. This is where the rollup decision from the first topic pays off or fails: if the long-term data kept only means, the peaks needed for this forecast were discarded and the forecast cannot be made.

Act at a target utilisation, not at exhaustion. Waiting until a link reaches 100% means it has been unusable for some time before that, because queueing delay rises steeply well before saturation. A target of 70 to 80% of capacity as the trigger is conventional, and the choice should reflect burstiness: the burstier the traffic, the lower the trigger.

The step that is most often missed is subtracting the lead time. A forecast that says "twelve months until the trigger" is not a decision until it is compared against how long the remedy takes to obtain. Reconfiguring existing capacity takes days. Ordering a port or an optic takes weeks. A circuit upgrade takes months. A new circuit to a new site can take half a year, and building a fabric longer. The runway must exceed the lead time of the chosen remedy, and when it does not, that remedy is already unavailable — a fact invisible in any utilisation graph. Lead time is what turns a forecast into a purchase order date.

Two refinements worth applying. Step changes break trend extrapolation: a product launch, an office opening, or a migration is not in the historical trend and must be added explicitly. And the bottleneck may not be bandwidth: firewall session tables, NAT port pools, load balancer connection limits, address space, and flow-table capacity all exhaust on their own schedules, and forecasting only link utilisation misses whichever one arrives first.

Examples

import math
# Capacity forecasting: fit the right model, then subtract the LEAD TIME.
HISTORY = [  # (month index, peak Mb/s) -- 18 months of daily maxima, monthly rollup
    (0, 180), (1, 187), (2, 192), (3, 201), (4, 206), (5, 215),
    (6, 221), (7, 231), (8, 237), (9, 248), (10, 254), (11, 265),
    (12, 273), (13, 284), (14, 292), (15, 305), (16, 314), (17, 327),
]
LINK_MBPS, TARGET_UTIL = 1_000, 0.80        # act at 80%, not at 100%
ACT_AT = LINK_MBPS * TARGET_UTIL

def linear_fit(pts):
    n = len(pts); sx = sum(x for x, _ in pts); sy = sum(y for _, y in pts)
    sxx = sum(x * x for x, _ in pts); sxy = sum(x * y for x, y in pts)
    b = (n * sxy - sx * sy) / (n * sxx - sx * sx)
    return (sy - b * sx) / n, b                     # intercept, slope

def exp_fit(pts):
    """Fit y = a * e^(kx) by regressing log(y) linearly."""
    la, k = linear_fit([(x, math.log(y)) for x, y in pts])
    return math.exp(la), k

def r_squared(pts, predict):
    ybar = sum(y for _, y in pts) / len(pts)
    ss_res = sum((y - predict(x)) ** 2 for x, y in pts)
    ss_tot = sum((y - ybar) ** 2 for _, y in pts)
    return 1 - ss_res / ss_tot

a_lin, b_lin = linear_fit(HISTORY)
a_exp, k_exp = exp_fit(HISTORY)
lin = lambda x: a_lin + b_lin * x
exp_ = lambda x: a_exp * math.exp(k_exp * x)

print(f"linear     : {a_lin:7.1f} + {b_lin:5.2f}/month   R^2 = {r_squared(HISTORY, lin):.4f}")
print(f"exponential: {a_exp:7.1f} * e^({k_exp:.4f}/month)  R^2 = {r_squared(HISTORY, exp_):.4f}")
print(f"             -> {(math.exp(k_exp*12)-1)*100:.1f}% annual growth")
best, name = ((exp_, "exponential") if r_squared(HISTORY, exp_) > r_squared(HISTORY, lin)
              else (lin, "linear"))
print(f"better fit: {name}\n")
# linear     :   173.6 +  8.55/month   R^2 = 0.9927
# exponential:   180.0 * e^(0.0349/month)  R^2 = 0.9995
#              -> 52.1% annual growth
# better fit: exponential
#
# BOTH fit well over the historical window (0.993 vs 0.9995): any smooth curve
# looks linear over a short span. The choice only matters in EXTRAPOLATION.

def months_until(predict, level, start=17):
    m = start
    while predict(m) < level and m < start + 240:
        m += 1
    return m - start

for label, predict in (("linear", lin), ("exponential", exp_)):
    print(f"{label:12s} reaches {ACT_AT:.0f} Mb/s (80%) in "
          f"{months_until(predict, ACT_AT):3d} months, {LINK_MBPS} Mb/s in "
          f"{months_until(predict, LINK_MBPS):3d} months")
# linear       reaches 800 Mb/s (80%) in  57 months, 1000 Mb/s in  80 months
# exponential  reaches 800 Mb/s (80%) in  26 months, 1000 Mb/s in  33 months
#
# A 2.2x difference in runway from two models that BOTH fit the history. This is
# why choosing linear when growth compounds is how capacity runs out early.

LEAD_TIMES = {"reconfigure existing capacity": 0.2, "order a port/optic": 1.0,
              "order a circuit upgrade": 3.0, "new circuit to a new site": 6.0,
              "build a new fabric": 12.0}

def lead_time_report(link_mbps):
    act_at = link_mbps * TARGET_UTIL
    runway = months_until(best, act_at)
    print(f"\nOn a {link_mbps} Mb/s link: {runway} months until {act_at:.0f} Mb/s (80%).")
    print(f"  {'action':32s} {'lead':>7s} {'slack':>7s}  status")
    for action, lead in LEAD_TIMES.items():
        slack = runway - lead
        status = ("comfortable" if slack > 3 else
                  "ORDER NOW" if slack > 0 else "ALREADY LATE")
        print(f"  {action:32s} {lead:6.1f}mo {slack:6.1f}mo  {status}")

lead_time_report(1_000)
lead_time_report(600)
# On a 1000 Mb/s link: 26 months until 800 Mb/s (80%).
#   ... all options comfortable
#
# On a 600 Mb/s link: 12 months until 480 Mb/s (80%).
#   reconfigure existing capacity       0.2mo   11.8mo  comfortable
#   order a port/optic                  1.0mo   11.0mo  comfortable
#   order a circuit upgrade             3.0mo    9.0mo  comfortable
#   new circuit to a new site           6.0mo    6.0mo  comfortable
#   build a new fabric                 12.0mo    0.0mo  ALREADY LATE
#
# The 12-month runway exactly equals the time to build a new fabric, so that
# remedy is already unavailable -- and nothing in the utilisation graph said so.
# Lead time is what turns a forecast into a purchase order date.

print("\nThe other resources that exhaust on their own schedule:")
for resource, signal, note in (
    ("firewall session table", "sessions vs maximum", "exhausts before bandwidth does"),
    ("NAT port pool", "ports in use per public address", "fails as connection errors"),
    ("load balancer connections", "concurrent vs licensed limit", "often a LICENCE limit"),
    ("IP address space", "free addresses per subnet", "cannot be fixed quickly"),
    ("flow / ARP / MAC tables", "entries vs capacity", "degrades silently"),
    ("TCAM / route capacity", "routes vs ASIC limit", "catastrophic when exceeded"),
):
    print(f"  {resource:26s} track {signal:30s} {note}")
print("\nForecasting only link utilisation misses whichever of these arrives first.")

# --- Forecast from PEAKS, using the rollups that kept the max ---
curl -s -G 'http://prom:9090/api/v1/query_range' \
  --data-urlencode 'query=max_over_time(link:utilisation:max5m[1d])' \
  --data-urlencode "start=$(date -u -d '18 months ago' +%s)" \
  --data-urlencode "end=$(date -u +%s)" --data-urlencode 'step=86400' \
  | jq -r '.data.result[0].values[] | @tsv' > peaks.tsv
#  If this returns nothing, the rollup kept only means and this forecast cannot
#  be made. That decision was taken 18 months ago (see the first topic).

# --- Fit both models and compare, rather than assuming ---
python3 - <<'PY' < peaks.tsv
import sys, math
pts = [(i, float(l.split('\t')[1])) for i, l in enumerate(sys.stdin) if l.strip()]
def lf(p):
    n=len(p); sx=sum(x for x,_ in p); sy=sum(y for _,y in p)
    sxx=sum(x*x for x,_ in p); sxy=sum(x*y for x,y in p)
    b=(n*sxy-sx*sy)/(n*sxx-sx*sx); return (sy-b*sx)/n, b
a,b = lf(pts)
la,k = lf([(x, math.log(y)) for x,y in pts if y>0])
print(f"linear slope {b:.4f}/day, exponential {(math.exp(k*365)-1)*100:.1f}%/yr")
PY

# --- Track the OTHER exhaustion curves, not just bandwidth ---
# Firewall sessions against the platform maximum
#   show conn count        /  show resource usage
# NAT port pool utilisation per public address
#   show nat pool statistics
# Route and MAC table occupancy against the ASIC limit
vtysh -c 'show ip route summary' | tail -3
bridge fdb show | wc -l
#   show platform hardware capacity forwarding   (vendor-specific)

# Address space, which has the longest remediation time of all
aws ec2 describe-subnets --query \
  'Subnets[].[SubnetId,CidrBlock,AvailableIpAddressCount]' --output table

# --- Add step changes explicitly: they are not in the trend ---
cat <<'TXT'
forecast inputs, 2025:
  trend growth (fitted)                 +52%/yr on the peak
  + new office, 400 staff, March        +180 Mb/s peak, step
  + video platform rollout, June        +240 Mb/s peak, step
  + datacentre migration, September     -120 Mb/s peak (traffic moves away)
TXT
# A trend line cannot predict a planned event. Ask the business what is coming,
# then add it to the curve as a step.

# --- Review cadence: refit quarterly and compare against the last forecast ---
# Record each forecast with its date, so you can see whether the model is
# tracking. A forecast that is consistently early or late is the wrong model.

Key takeaways

• Network growth usually compounds, so fit both linear and exponential models and compare goodness of fit; over a short history both fit well and they diverge in extrapolation.
• The divergence is the risk: choosing linear when growth compounds shortens the real runway substantially.
• Forecast from peaks, not averages — which requires that the rollup retained maxima.
• Act at a target utilisation of roughly 70 to 80%, not at exhaustion, because queueing delay rises steeply before saturation. Burstier traffic means a lower trigger.
• Subtract the lead time. A runway shorter than the remedy's procurement time means that remedy is already unavailable, and no utilisation graph shows this.
• Step changes must be added explicitly: launches, office openings, and migrations are not in the historical trend.
• Bandwidth is often not the first thing to exhaust. Track firewall sessions, NAT ports, load balancer limits, address space, and forwarding-table capacity.
• Refit periodically and record past forecasts; a model that is consistently early or late is the wrong model.

🧪 Practice

1. Given peaks growing from 240 to 420 Mb/s over 12 months, fit an annual growth rate and compute the months until 80% of a 1 Gb/s link under both linear and exponential assumptions.
2. Explain why forecasting from monthly average utilisation understates when capacity is needed, and what data the forecast actually requires.
3. A site has 9 months of runway. Classify each remedy as viable or not: reconfiguring existing capacity, ordering an optic, a circuit upgrade, a new site circuit.
4. Interview-style: A link is at 40% utilisation and growing slowly, yet users report intermittent connection failures at peak that resolve on retry. (Hint: bandwidth is clearly not the constraint — name three other resources on that path that exhaust independently, and the counter you would check for each.)

Redundancy and High Availability Design

Theory

Redundancy is the deliberate provision of spare capacity so that a failure does not become an outage, and it goes wrong in three specific ways that are worth knowing in advance because each is invisible until the failure occurs.

The sizing model determines what you survive. N provides exactly enough capacity and survives nothing. N+1 survives one failure and then has no headroom at all — a second failure, or ordinary growth, during the repair window is an outage. N+2 survives two. 2N duplicates everything at twice the cost. The choice is a statement about how long repairs take and how likely a second failure is during that window, and N+1 is chosen by default far more often than that reasoning supports.

Failover time is usually the dominant term in the outage, not detection. The same redundant hardware produces wildly different availability depending on the mechanism that switches to it. Precomputed fast reroute with sub-second detection gives a fraction of a second. Routing reconvergence gives seconds. DNS-based failover gives minutes, as Chapter 13 showed. Manual intervention gives fifteen minutes or more, most of it spent paging a human and waiting for them to log in. Across four failures a year, that range spans under a second to an hour of annual downtime for identical equipment — and the mechanism is usually cheaper to improve than the hardware.

Active-active and active-passive fail differently, and the trap is in active-active. An active-passive standby is a clean known state, and its weakness is that it is untested: it may not work when needed, and its configuration may have drifted. Active-active proves both paths continuously and has no failover step at all, but it introduces a constraint people forget when choosing it for efficiency: each side must be able to carry the full load alone. Two active links each at 60% utilisation cannot survive losing one, because the survivor would need 120%. Active-active redundancy therefore requires each side to stay below 50%, which is exactly the efficiency that made it attractive. Exceeding that turns a single failure into a cascading one, as the survivor saturates and fails too.

Underlying all of this is the correlation point from the previous topic: redundancy multiplies availability only if the copies fail independently. Two devices in the same rack, on the same power feed, running the same software version, configured from the same template are not two independent components. The configuration case is the most commonly overlooked and the easiest to check.

And the conclusion that matters most: untested redundancy has an availability of zero for the failure mode you never exercised. Testing is not a formality; it is the only thing that converts designed redundancy into real redundancy. The tests that find real problems are the uncomfortable ones — failing over and staying on the standby for a week to reveal configuration drift, failing the primary while the secondary is at peak load to reveal the sizing error, and performing a failover manually with the automation disabled to reveal whether the runbook is correct.

Examples

# Redundancy design: N+1 vs 2N, and what happens the moment one unit fails.
def capacity_check(units, unit_capacity, demand, failed=0):
    available = (units - failed) * unit_capacity
    return available, demand, (demand / available if available else float("inf"))

DEMAND, UNIT = 300, 100
print(f"Sizing models for a {DEMAND} Mb/s demand on {UNIT} Mb/s units:")
for model, units in (("N (no redundancy)", 3), ("N+1", 4), ("N+2", 5), ("2N", 6)):
    print(f"  {model:18s} {units} units, cost {units/3:.2f}x minimum")
    for label, failed in (("normal", 0), ("1 failed", 1), ("2 failed", 2)):
        avail, dem, load = capacity_check(units, UNIT, DEMAND, failed)
        state = "ok" if load <= 1.0 else f"OVERLOADED {load:.2f}x"
        print(f"  {'':18s}   {label:9s} {avail:4.0f} Mb/s available -> {state}")
    print()
#   N (no redundancy)  3 units, cost 1.00x minimum
#                        normal     300 Mb/s available -> ok
#                        1 failed   200 Mb/s available -> OVERLOADED 1.50x
#   N+1                4 units, cost 1.33x minimum
#                        normal     400 Mb/s available -> ok
#                        1 failed   300 Mb/s available -> ok
#                        2 failed   200 Mb/s available -> OVERLOADED 1.50x
#   N+2                5 units, cost 1.67x minimum   (survives 2 failures)
#   2N                 6 units, cost 2.00x minimum
#
# The trap in N+1: it survives ONE failure and then has zero headroom. A second
# failure, or ordinary growth, DURING the repair window is an outage.

FAILURES_PER_YEAR = 4
print("Failover MECHANISM dominates the outage, not the hardware:")
MECHANISMS = {
    "BFD + fast reroute":     (0.05, "sub-second, precomputed backup"),
    "routing reconvergence":  (3.0,  "detect, recompute, reprogram"),
    "VRRP/HSRP":              (3.0,  "hold timer x3 before takeover"),
    "DNS failover (60s TTL)": (210.0, "detect + TTL + clients ignoring TTL"),
    "manual intervention":    (900.0, "page, acknowledge, log in, decide, act"),
}
print(f"{'mechanism':26s} {'failover':>10s} {'annual downtime':>16s}  note")
for m, (secs, note) in MECHANISMS.items():
    print(f"{m:26s} {secs:9.2f}s {secs*FAILURES_PER_YEAR:15.1f}s  {note}")
print(f"\n  At {FAILURES_PER_YEAR} failures/year the mechanism alone spans "
      f"{MECHANISMS['BFD + fast reroute'][0]*FAILURES_PER_YEAR:.1f}s to "
      f"{MECHANISMS['manual intervention'][0]*FAILURES_PER_YEAR/60:.0f} minutes")
print("  of annual downtime -- a 4500x range, for IDENTICAL hardware.\n")

print("Active-active vs active-passive:")
for model, pro, con in (
    ("active-passive", "simple; the standby is a known-good clean state",
     "the standby is UNTESTED: it may not work when needed"),
    ("active-active", "both paths continuously proven; no failover step",
     "each side must carry FULL load alone, or failover overloads it"),
):
    print(f"  {model:16s} + {pro}")
    print(f"  {'':16s} - {con}")
print("\n  The active-active trap, quantified:")
for util_each in (0.40, 0.50, 0.60, 0.75):
    after = util_each * 2
    print(f"    each side at {util_each:.0%} -> survivor at {after:.0%} "
          f"{'ok' if after <= 1.0 else 'OVERLOADED: cascading failure'}")
#     each side at 40% -> survivor at 80% ok
#     each side at 50% -> survivor at 100% ok
#     each side at 60% -> survivor at 120% OVERLOADED: cascading failure
#     each side at 75% -> survivor at 150% OVERLOADED: cascading failure
#
# Active-active requires each side BELOW 50% -- exactly the efficiency that made
# it attractive. Above that, one failure saturates the survivor and it fails too.

print("\nTesting is what separates redundancy from the BELIEF in redundancy:")
for test, finds in (
    ("pull one cable, in business hours, deliberately",
     "whether failover works and how long it takes"),
    ("fail over and STAY on the standby for a week",
     "whether the standby's configuration has drifted"),
    ("fail the primary while the secondary is at peak load",
     "whether the survivor can actually carry it"),
    ("disable the automation and fail over manually",
     "whether the runbook is correct and anyone can follow it"),
):
    print(f"  {test:50s}")
    print(f"    -> finds {finds}")
print("\n  Untested redundancy has an availability of ZERO for the failure mode")
print("  you never exercised, and it is discovered at the worst possible moment.")

# --- Make failover FAST: the cheapest availability improvement available ---
# BFD gives sub-second detection, independent of protocol timers.
#   interface Gi0/1
#     bfd interval 300 min_rx 300 multiplier 3     -> ~900 ms detection
#   router bgp 65001
#     neighbor 10.0.0.2 fall-over bfd
vtysh -c 'show bfd peers brief'
#  Without BFD, BGP's default hold timer is 180 s. That single change moves the
#  outage from three minutes to under a second, at no hardware cost.

# Verify precomputed backups exist BEFORE the failure (Chapter 13's FRR/TI-LFA)
vtysh -c 'show ip route 10.20.0.0/16' | grep -A2 'backup'
vtysh -c 'show isis fast-reroute summary' 2>/dev/null

# --- Verify each side can carry full load: the active-active check ---
for link in swp49 swp50; do
  a=$(cat /sys/class/net/$link/statistics/tx_bytes); sleep 10
  b=$(cat /sys/class/net/$link/statistics/tx_bytes)
  gbps=$(echo "scale=2; ($b-$a)*8/10/1000000000" | bc)
  echo "$link ${gbps} Gb/s"
done
#  swp49 6.10 Gb/s      Sum is 12.2 Gb/s on 2x10G links: each at 61%.
#  swp50 6.10 Gb/s      Losing one leaves 12.2 Gb/s of demand on 10 Gb/s.
#  -> this pair CANNOT survive a single failure. Alert on the SUM exceeding
#     the capacity of (n-1) links, not on per-link utilisation.

# --- Check independence: identical configs are a shared failure mode ---
diff <(ssh edge-1 'show running-config' | grep -v '^!') \
     <(ssh edge-2 'show running-config' | grep -v '^!')
ssh edge-1 'show version | include Version'; ssh edge-2 'show version | include Version'
#  Same software version, same template, same rack, same power feed, same
#  provider, same optic batch -- each is correlation the arithmetic ignores.

# --- TEST IT. This is the only step that makes redundancy real. ---
# 1. Planned failover in business hours, with someone watching:
ssh edge-1 'configure terminal; interface Gi0/1; shutdown'
#    measure: how long until traffic recovers? does anything alert?
watch -n1 'curl -sS -o /dev/null -m2 -w "%{http_code} %{time_total}\n" https://api/health'
ssh edge-1 'configure terminal; interface Gi0/1; no shutdown'

# 2. STAY on the standby for a week. This is what finds configuration drift,
#    expired certificates, and stale firewall rules on the path nobody uses.

# 3. Fail over at PEAK, not at 03:00. A standby that works at 03:00 and
#    saturates at 14:00 has not been tested.

# 4. Do it manually with the automation disabled, to test the runbook:
#    if the only person who can fail over is the one who built it, you do not
#    have a redundant system, you have a single point of failure with a pulse.

# --- Record the measured failover time, and alert if it regresses ---
echo "$(date -u +%FT%TZ) edge-1->edge-2 failover: 0.84s recovered" >> /var/log/ha-tests.log

Key takeaways

• The sizing model states what you survive: N survives nothing, N+1 survives one failure and then has zero headroom, N+2 survives two, 2N costs double.
• N+1's weakness is the repair window: a second failure or ordinary growth during it is an outage.
• Failover mechanism, not hardware, dominates the outage. For identical equipment the annual downtime spans under a second to an hour depending on the mechanism.
• BFD plus precomputed fast reroute is usually the cheapest large availability improvement, moving detection from protocol timers to under a second.
• Active-passive standbys are untested and may have drifted; active-active proves both paths but requires each side to carry the full load alone.
• The active-active constraint is concrete: each side must stay below 50%, or one failure saturates the survivor and cascades.
• Alert on the sum exceeding the capacity of n-1 units, not on per-unit utilisation.
• Identical configurations are a shared failure mode. Verify independence in rack, power, provider, software version, and template.
• Untested redundancy has an availability of zero for the mode you never exercised. Test at peak, stay on the standby for a week, and rehearse the manual procedure.

🧪 Practice

1. A service needs 800 Mb/s and units provide 200 Mb/s. Give the unit count and relative cost for N, N+1, N+2, and 2N, and state what each survives.
2. Two active 10 Gb/s links each carry 5.8 Gb/s. Determine whether the pair survives one failure, and write the alert expression that would have warned you.
3. Compare BFD-based fast reroute against DNS failover for a 4-failure year, computing annual downtime for each and naming the cheaper improvement.
4. Interview-style: A datacentre has fully redundant power, network, and cooling, and a single failure takes the site down anyway. (Hint: the parallel arithmetic assumes one property that the phrase "fully redundant" does not guarantee — identify it, and list the shared elements you would audit.)

---

<a id="15-advanced-and-emerging-topics"></a>