9. Reliability, Availability, and Fault Tolerance

Every component in a system eventually fails; reliability engineering is the discipline of making sure that a component failure is not a system failure. This chapter covers how availability is measured and budgeted, the patterns that stop one sick dependency from taking down everything that calls it, the failure modes that only appear at scale, and the operational practices — SLOs, error budgets, chaos experiments, postmortems — that keep reliability an explicit engineering target rather than a hope.

<a id="91-availability-engineering"></a>

9.1 Availability Engineering

Availability is the property that a system responds correctly when asked. This subchapter turns that vague goal into numbers you can budget against, then walks the structural choices — redundancy, failover topology, geographic spread — that actually move those numbers.

Availability Tiers and Nines

Availability is the fraction of time a system is usable, and it is conventionally quoted as a number of nines because the interesting differences live in the decimal places. The reason to speak in nines rather than percentages is that each additional nine is a tenfold reduction in downtime and, roughly, an order of magnitude more engineering effort and cost. Saying "we want high availability" is meaningless; saying "we want four nines" is a budget, a topology, and an on-call rotation.

The mental model that makes the numbers concrete is a downtime allowance. If you promise 99.9% availability, you are promising that the system is down for at most 43.2 minutes in a 30-day month. That is the entire budget for deploys gone wrong, dependency outages, certificate expiries, and bad database migrations combined. One botched release that takes an hour to roll back has already overspent it.

Availability	Downtime / year	Downtime / month	Downtime / week	Typical shape
90%	36.5 days	72 hours	16.8 hours	A hobby project
99%	3.65 days	7.2 hours	100.8 minutes	Single server, manual recovery
99.9%	8.76 hours	43.2 minutes	10.1 minutes	Redundant instances, one AZ, on-call
99.99%	52.6 minutes	4.32 minutes	60.5 seconds	Multi-AZ, automated failover
99.999%	5.26 minutes	25.9 seconds	6.05 seconds	Multi-region, no human in the loop

The step from 99.99% to 99.999% is the one that changes how you work. A four-minute monthly budget still permits a human to be paged, look at a dashboard, and act. A 26-second budget does not — anything that needs a person is already an SLO breach, so recovery must be fully automatic, which means the system must detect and route around its own failures without supervision.

# Availability from the two numbers that actually drive it.
#   MTBF = mean time between failures  (how often it breaks)
#   MTTR = mean time to recovery       (how long it stays broken)
def availability(mtbf_hours, mttr_hours):
    return mtbf_hours / (mtbf_hours + mttr_hours)

# Same failure rate -- one failure every 30 days -- two very different outcomes.
slow_recovery = availability(720, 4.0)        # 4 hours to recover
fast_recovery = availability(720, 1 / 3)      # 20 minutes to recover

print(f"{slow_recovery:.5f}")   # 0.99448 -> ~2.3 nines, ~48 hours down per year
print(f"{fast_recovery:.5f}")   # 0.99954 -> ~3.3 nines, ~4 hours down per year

# The lesson: recovery time is the cheaper lever. Making failures 12x rarer is
# a research project; making recovery 12x faster is usually automation you can
# build this quarter.

Two composition rules matter more than any single component's rating. Dependencies in series multiply, so a request path is always less available than its weakest link. Redundant components in parallel multiply their failure probabilities, so duplication is enormously effective — provided the failures are independent.

# Series: every hop must work for the request to work.
path = 0.999 * 0.999 * 0.999 * 0.999      # LB -> API -> cache -> DB
print(f"{path:.6f}")                      # 0.996006 -> ~35 hours of downtime/year
# Four "three nines" dependencies deliver well under three nines end to end.

# Parallel: the system is down only if EVERY replica is down.
replicas = 1 - (1 - 0.99) ** 3            # three independent 99% replicas
print(f"{replicas:.6f}")                  # 0.999999 -> six nines, from cheap parts

# The word doing all the work is "independent". Three replicas on the same
# host, the same rack, or the same bad deploy fail as one component, and the
# formula silently collapses back to 0.99.

A final caution: the number you promise must match the number you can measure. Availability measured as "the process was running" is nearly meaningless; the useful definition is success rate as seen by clients — the fraction of requests that returned a correct answer within the latency the user needed.

Key Takeaways

• Each additional nine cuts downtime tenfold and costs roughly an order of magnitude more effort; pick the tier the business actually needs.
• Availability = MTBF / (MTBF + MTTR), so shortening recovery is usually cheaper than making failures rarer.
• Dependencies in series multiply availabilities: end-to-end is always worse than any single hop.
• Redundancy multiplies failure probabilities and is extremely effective, but only to the degree failures are independent.
• Measure availability as client-observed success rate, not process uptime.

🧪 Practice

1. Compute the yearly downtime budget for 99.95% and decide whether a weekly 10-minute maintenance window fits inside it.
2. A request passes through a CDN (99.99%), an API gateway (99.95%), a service (99.9%), and a database (99.99%). Compute end-to-end availability and name the hop worth improving first.
3. Interview: Your service is rated 99.9% but a customer reports far more failures than that. What are the likely explanations? (Hint: think about what exactly is being measured, over what window, and whose requests are counted.)

Single Points of Failure

A single point of failure (SPOF) is any component whose loss takes down the whole system. Finding SPOFs is the highest-leverage reliability work available, because no amount of redundancy elsewhere compensates for one un-duplicated part on the critical path — the series rule guarantees the system inherits that component's availability exactly.

SPOFs are easy to name in a diagram and hard to find in a real system, because the dangerous ones are rarely the obvious boxes. Everyone remembers to duplicate the database. Fewer people notice that all three replicas authenticate against one identity server, that the deployment pipeline signs artifacts with one key in one region, or that the "redundant" load balancers share a single control plane that pushes configuration to both.

 WHERE SPOFs ACTUALLY HIDE

   [ clients ]
        |
   [ DNS ] <----------------- single zone, single provider, long TTL
        |
   [ LB-a ]   [ LB-b ]         redundant data plane...
        \       /
         \     /  <----------- ...but ONE control plane pushes their config
          \   /
      [ api x 20 ]  <--------- healthy redundancy
           |
      [ auth service ]  <----- every request needs it; one deployment
           |
   [ primary DB ] --> [ replica ]   failover exists, but is it tested?
           |
   [ shared config store ]  <--- read at startup by everything; if it is
                                 empty, a rolling restart takes out the fleet

The systematic way to find them is to walk the request path and ask, for each element, "if exactly this disappears right now, what still works?" — then repeat the walk for the control paths: deploys, configuration, secrets, DNS, service discovery, certificate renewal, and the monitoring that would tell you any of this had happened.

# A dependency audit expressed as data, so it can be asserted in CI.
COMPONENTS = {
    # name              (instances, failure_domains,            optional)
    "api":              (20,  ["az-a", "az-b", "az-c"], False),
    "postgres_primary": (1,   ["az-a"],                 False),
    "postgres_replica": (2,   ["az-b", "az-c"],         True),
    "auth":             (3,   ["az-a", "az-b", "az-c"], False),
    "config_store":     (1,   ["az-a"],                 False),
    "recommendations":  (4,   ["az-a", "az-b"],         True),   # degradable
}

def spofs(components):
    out = []
    for name, (instances, domains, optional) in components.items():
        if optional:                       # a failure here degrades, not breaks
            continue
        if instances == 1 or len(set(domains)) == 1:
            out.append(name)               # one instance OR one failure domain
    return out

print(spofs(COMPONENTS))
# ['postgres_primary', 'config_store']
#
# Note what the check counts: not just instance count but FAILURE DOMAIN count.
# Twenty replicas in a single availability zone are one SPOF in disguise.

Removing a SPOF has three possible answers, in descending order of preference: make the component redundant across independent failure domains; make the system able to run without it (degrade gracefully, cache its answers, fail open where safe); or accept it explicitly, with a documented recovery procedure and a recovery time everyone has agreed to. The third answer is legitimate — a single regional deployment may be a fine choice — but only when it is a decision rather than an oversight.

Key Takeaways

• A SPOF caps overall availability at its own, regardless of redundancy elsewhere on the path.
• The dangerous SPOFs usually sit in control planes — config, secrets, DNS, deploy pipelines — not in the boxes on the architecture diagram.
• Count independent failure domains, not instances: many replicas in one zone are still one point of failure.
• Every SPOF should be removed, degraded around, or explicitly accepted with a documented recovery plan.

🧪 Practice

1. Draw your current project's request path and mark every element whose loss would stop all traffic.
2. Extend the audit function so a component is also flagged when all its instances share one release version or one credential.
3. Interview: A team says "we have no SPOFs, everything runs three replicas." What questions would you ask? (Hint: ask what the three replicas share — a host, a zone, a config source, a release, a certificate.)

Redundancy and Failover Models

Redundancy is having more of something than you strictly need so that losing one copy is survivable. Failover is the mechanism that redirects work to the surviving copies. The two are separate concerns, and the common failure in practice is having the first without a tested version of the second: spare capacity nobody routes to is just an expensive idle machine.

Redundancy comes in levels, and the level determines which class of failure you survive. Duplicating a process survives a crash. Duplicating across hosts survives hardware failure. Duplicating across availability zones survives a power or network event in one datacenter. Duplicating across regions survives a regional outage or a bad regional configuration push. Each level costs more and covers a strictly larger set of failures.

Model	Spare capacity	Failover time	Cost	Typical use
Cold standby	Provisioned on demand	Minutes to hours	Lowest	Batch systems, DR sites
Warm standby	Running, not serving	Seconds to minutes	Medium	Databases, stateful services
Hot standby	Running, fully in sync	Sub-second	High	Payment ledgers, control planes
N+1	One spare unit	Immediate	1/N overhead	Stateless service fleets
N+M or 2N	M spares or full twin	Immediate	Up to 2x	Anything with strict SLOs

The capacity arithmetic is what people get wrong. "We run three availability zones, so we survive losing one" is only true if two zones can carry 100% of peak traffic — which means each zone must normally run at no more than about 67% utilization. Sizing a fleet to its average load and then claiming zone redundancy is a promise the system cannot keep.

import math

# Sizing a fleet that must survive the loss of one availability zone.
peak_rps     = 30_000
rps_per_node = 800
azs          = 3

nodes_at_peak = math.ceil(peak_rps / rps_per_node)      # 38 nodes carry peak
surviving_azs = azs - 1                                 # after one AZ is lost

# Each surviving AZ must carry its share of ALL the traffic, not its own share.
nodes_per_az = math.ceil(nodes_at_peak / surviving_azs) # 19
total_nodes  = nodes_per_az * azs                       # 57

print(nodes_at_peak, nodes_per_az, total_nodes)   # 38 19 57
print(f"{nodes_at_peak / total_nodes:.0%}")       # 67% -- the headroom IS the
                                                  # redundancy, not waste

Failover itself has two hard parts: deciding that a failure happened, and making sure only one component believes it is in charge. Detection trades speed against false positives — an aggressive health check flips over on a garbage-collection pause, a conservative one leaves users staring at errors. The second problem is split-brain: if the old primary is merely unreachable rather than dead, promoting a new one gives you two primaries accepting writes.

 SPLIT-BRAIN DURING FAILOVER

  Before:      [ primary A ] <== writes ==  clients
                    |
                [ replica B ]

  A partition isolates A from the failure detector, but NOT from clients:

   clients ==> [ A: "I am still primary" ]  X  [ B: promoted ] <== clients
                        \                              /
                         \------ both accept writes ---/
                            divergent history, silent data loss

  Defenses:
    - quorum: a node may act as primary only while a majority agrees it is
    - fencing token: a monotonically increasing epoch number; the storage layer
      rejects writes carrying a stale token, locking out the deposed primary
    - STONITH: forcibly power off or network-isolate the old primary first

The fencing-token idea is worth internalizing because it is the only defense that does not rely on timing assumptions: the shared resource itself refuses writes from an older epoch, so even a primary that was frozen for a minute and wakes up convinced it is in charge cannot corrupt anything.

Key Takeaways

• Redundancy without tested, automatic failover buys cost, not availability.
• Each redundancy level (process, host, zone, region) covers a strictly larger failure class at a strictly larger price.
• Surviving the loss of one of N failure domains requires running at (N-1)/N utilization or less; the unused headroom is the redundancy.
• Failover must solve detection (speed vs. false positives) and split-brain (quorum, fencing tokens, or forced isolation).

🧪 Practice

1. Recompute the sizing example for four AZs and explain why the per-AZ node count drops.
2. Write pseudocode for a storage layer that rejects any write carrying a fencing token lower than the highest it has seen.
3. Interview: Your database failover promoted a replica while the old primary was still taking writes. How would you prevent a repeat? (Hint: the fix cannot be "detect faster" — what must the storage layer itself enforce?)

Active-Active vs. Active-Passive

Once you have more than one copy of a system, you must decide whether the copies all serve traffic or whether some of them wait. Active-passive keeps one deployment live and one standing by; active-active runs both live and splits traffic between them. The choice looks like a capacity decision but is really a consistency and operations decision.

Active-passive is simpler because there is exactly one place where writes happen, so the hard questions about conflicting concurrent updates never arise. Its weaknesses are that the standby is idle capacity you pay for, and that a path which is never exercised is a path you cannot trust — untested failover is the most common cause of a failover that does not work.

Active-active removes both weaknesses and introduces one large one: two live copies accepting writes to the same logical data must reconcile. If both regions can update the same row, you need conflict resolution — last-writer-wins, CRDTs, or an application-level merge — and every one of those choices loses information in some scenario.

Dimension	Active-Passive	Active-Active
Capacity utilization	Half or less is idle	All copies serve traffic
Failover time	Detection + promotion: seconds to mins	Near zero; drain the failed copy
Write consistency	Simple: one writer	Hard: conflicts must be resolved
Failover confidence	Low unless drilled regularly	High: the path is used continuously
Cost	Paying for idle standby	Paying for capacity you actually use
Complexity	Low	High (routing, replication, conflicts)

 ACTIVE-PASSIVE                        ACTIVE-ACTIVE

    clients                                clients
       |                                  /       \
   [ router ]                       [ geo-router / anycast ]
       |                                /             \
  [ region A ] == replication =>  [ region A ] <= bi-directional => [ region B ]
   (serving)      [ region B ]     (serving)      replication        (serving)
                   (standby)
                                   Conflicts are possible whenever the same key
   One writer. Failover =          is written in both regions inside the
   promote B, repoint the          replication lag window.
   router. Nothing to merge.

A pattern that captures most of the benefit with much less of the risk is active-active with partitioned ownership: both regions serve traffic, but every piece of data has exactly one home region that owns its writes. Reads are served locally from replicas; writes are routed to the owner. There are no conflicts because there is still only one writer per key, yet both regions are live and the failover machinery runs every day.

# Sharded write ownership: both regions active, no write conflicts.
REGIONS = ["us-east", "eu-west"]

def owner_region(tenant_id: str) -> str:
    # Stable mapping from data to its single writer. In practice this lives in
    # a directory service so ownership can be MOVED during a failover.
    return REGIONS[stable_hash(tenant_id) % len(REGIONS)]

def handle_write(tenant_id, payload, local_region):
    owner = owner_region(tenant_id)
    if owner == local_region:
        return commit_locally(payload)      # fast path, no cross-region hop
    return forward_to(owner, payload)       # one extra ~80 ms round trip

def handle_read(tenant_id, local_region):
    # Reads are always local: fast, but behind by the replication lag.
    return read_local_replica(tenant_id)    # eventual consistency, accepted

The decision rule: if the workload is read-heavy and writes can be partitioned by tenant, user, or region, active-active with owned shards is usually right. If writes are global and must be strongly consistent — a single inventory count, a global sequence — active-passive with a fast, frequently drilled failover is the honest choice.

Key Takeaways

• Active-passive is simple because there is one writer, but its failover path is only as reliable as the last time it was tested.
• Active-active uses all capacity and exercises failover continuously, at the cost of conflict resolution.
• Partitioned write ownership gives active-active availability without write conflicts, and is the default for most multi-region designs.
• Truly global, strongly consistent writes push you back toward a single writer, whatever the topology is called.

🧪 Practice

1. List three data types in a typical SaaS product and decide, for each, whether its writes can be partitioned by tenant.
2. Modify the ownership example so a tenant can be transferred to the other region, and describe what must happen to in-flight writes during the move.
3. Interview: A stakeholder wants "active-active across two regions" for e-commerce checkout. What do you need to know before agreeing? (Hint: ask what happens when the same last unit of stock is sold in both regions.)

Multi-Region Architecture

Going multi-region is the point where physics enters the design. Within a region, round trips between services are under a millisecond and you can afford to be casual about how many of them a request makes. Between regions, a round trip costs tens to hundreds of milliseconds and no engineering effort reduces it, because it is bounded by the speed of light in fibre.

There are three legitimate reasons to go multi-region, and it is worth being explicit about which one applies, because they lead to different designs. Availability wants independent failure domains. Latency wants data close to users. Compliance wants data physically inside a jurisdiction. A design optimized for one is often wrong for the others.

 THE LATENCY BUDGET THAT DECIDES THE DESIGN

  intra-AZ round trip          ~0.5 ms     free; call as often as you like
  cross-AZ round trip          ~1-2 ms     cheap; synchronous replication is fine
  US-east to US-west            ~60 ms     one hop per request, at most
  US-east to Europe             ~80 ms     synchronous writes now hurt
  US-east to Asia-Pacific      ~150 ms     synchronous writes are not viable

  A checkout with a 300 ms budget can afford ONE cross-Atlantic round trip and
  nothing else. Two sequential ones and the budget is gone before rendering.

The core question in every multi-region design is what happens to writes. Synchronous cross-region replication gives zero data loss on a regional failure but pays the full round trip on every write and — worse — makes the write path depend on both regions being up, which lowers write availability rather than raising it. Asynchronous replication keeps writes fast and local but loses whatever had not replicated when the region died.

# What each replication choice actually costs you.
SCENARIOS = {
    # name                (write_latency_ms, data_loss_on_region_loss, write_availability)
    "single_region":      (   5, "total until restore",  "one region's"),
    "async_cross_region": (   5, "the replication lag",  "one region's"),
    "sync_cross_region":  (  85, "none",                 "BOTH regions'"),
    "quorum_3_regions":   (  85, "none",                 "any 2 of 3"),
}

for name, (lat, loss, avail) in SCENARIOS.items():
    print(f"{name:20} {lat:>3} ms  loss={loss:<22} avail={avail}")

# The quorum row is the interesting one: with three regions and a majority
# quorum, a write waits for the NEARER of the two remote regions, so it costs
# one cross-region round trip yet survives losing any single region. That is
# what Spanner-class systems sell, and why they are expensive.

Around that core, the standard multi-region toolkit is: route users to the nearest healthy region with latency-based DNS or anycast; keep session state out of region-local memory so any region can serve any user; replicate the data tier according to the decision above; and — the step most teams skip — make sure each region can be drained, meaning traffic can be shifted away from it deliberately, quickly, and without a deploy. A region you cannot drain on purpose is a region you cannot evacuate in an incident.

Two failure modes deserve advance thought. First, a partial region failure is harder than a total one: if a region is up but its database is slow, health checks may still pass while every request times out, so health signals must be end-to-end rather than per-process. Second, the failure that takes out two regions at once is almost never physical — it is a global configuration push, an expired certificate, or a bad deploy that reached everywhere. Regional independence must extend to the deployment pipeline, or the regions are not independent at all.

Key Takeaways

• Cross-region latency is a physical constant; count cross-region round trips as a hard budget item.
• Choose multi-region for availability, latency, or compliance — the three goals produce different designs.
• Synchronous cross-region writes eliminate data loss but make write availability depend on both regions; async keeps writes fast and loses the lag window.
• A three-region majority quorum costs one cross-region round trip and survives any single region loss.
• The realistic correlated multi-region failure is a global config or deploy, so stagger releases per region.

🧪 Practice

1. Given a 200 ms end-to-end budget and 80 ms cross-Atlantic round trips, decide whether an EU user can be served by a US-only write path.
2. Design the traffic-drain procedure for one region: what changes, who approves it, and how long until the region is empty.
3. Interview: Your service runs in three regions and still had a global outage. What are the likely causes? (Hint: look for what is identical in all three regions — code, config, certificates, and the pipeline that ships them.)

RTO and RPO

RTO and RPO are the two numbers that turn "we have backups" into a plan. Recovery Time Objective (RTO) is how long the business can tolerate being down. Recovery Point Objective (RPO) is how much recent data the business can tolerate losing. They are independent, they are business decisions rather than engineering ones, and most disaster-recovery arguments dissolve once someone states them.

They must be stated separately because they are bought with different mechanisms. RTO is bought with standby capacity and automation — how fast can you be serving again. RPO is bought with replication frequency — how recent is the newest copy you can restore. A nightly backup restored by hand might give RTO of six hours and RPO of 24 hours; continuous replication to a warm standby might give RTO of two minutes and RPO of five seconds.

 THE TWO OBJECTIVES ON ONE TIMELINE

   last durable copy                  outage                    service restored
          |                             |                              |
   -------+-----------------------------+------------------------------+------>
          |<-------- RPO window ------->|<--------- RTO window ------->|
          |                             |                              |
   everything written in here      the disaster            time spent detecting,
   is LOST if the primary is                               deciding, restoring,
   unrecoverable                                           and verifying

   RPO is set by replication frequency.  RTO is set by automation and practice.

Mechanism	Typical RPO	Typical RTO	Relative cost
Nightly backup to object store	Up to 24 hours	Hours	Very low
Hourly snapshots	Up to 1 hour	30-60 minutes	Low
Continuous log shipping	Seconds to minutes	10-30 minutes	Medium
Async replica, warm standby	Seconds	1-5 minutes	Medium-high
Sync replica or quorum writes	Zero	Seconds	High

from datetime import timedelta

def meets_objectives(snapshot_interval, restore_duration, detect_and_decide,
                     rpo_target, rto_target):
    # Worst case: the failure lands right before the next snapshot would run.
    worst_case_rpo = snapshot_interval
    # RTO is not just restore time -- detection and the decision to fail over
    # happen during the outage, and are usually the largest terms.
    worst_case_rto = detect_and_decide + restore_duration
    return {
        "rpo_ok": worst_case_rpo <= rpo_target,
        "rto_ok": worst_case_rto <= rto_target,
        "worst_case_rpo": worst_case_rpo,
        "worst_case_rto": worst_case_rto,
    }

print(meets_objectives(
    snapshot_interval=timedelta(hours=6),
    restore_duration=timedelta(minutes=40),    # measured in a drill, not assumed
    detect_and_decide=timedelta(minutes=25),   # page, triage, get approval
    rpo_target=timedelta(hours=1),
    rto_target=timedelta(hours=1),
))
# rpo_ok=False, rto_ok=True (65 minutes vs a 60-minute target is a near miss)
#
# Note WHY the RTO is tight: 25 of the 65 minutes are humans, not machines.
# Automating the failover decision buys more than a faster restore does.

Three practical rules. First, an untested restore is not a backup; the only evidence that RTO and RPO hold is a restore performed on a schedule, timed, from the real artifacts. Second, measure RTO from the moment the failure starts, not from the moment someone starts working — detection and escalation are usually the largest components. Third, set the objectives per data class: a shopping cart may tolerate an RPO of an hour while a payment ledger tolerates none, and forcing one number on both means overpaying for one and under-protecting the other.

Key Takeaways

• RTO is how long you can be down; RPO is how much data you can lose. They are bought with different mechanisms and must be stated separately.
• RPO is set by replication or snapshot frequency; RTO is set by automation, detection speed, and rehearsal.
• Worst-case RPO is the full interval between durable copies, not the average.
• Measure RTO from failure start, including detection and decision time.
• A backup that has never been restored under time pressure is an assumption, not a capability.

🧪 Practice

1. Assign RTO and RPO targets to three datasets in a system you know, and justify each from business impact rather than technical convenience.
2. Compute the worst-case RPO of a system that snapshots every 6 hours and ships write-ahead logs every 5 minutes, when the log shipper has been broken for a day.
3. Interview: The business asks for "zero downtime and zero data loss." How do you respond? (Hint: both are purchasable, but the price is a specific topology — name it and its cost, then ask which systems truly need it.)

<a id="92-resilience-patterns"></a>

9.2 Resilience Patterns

Redundancy handles components that stop. This subchapter handles the harder case: components that are still there but slow, flaky, or overloaded. Every pattern here exists to keep one degraded dependency from consuming the resources its callers need to serve everyone else.

Timeouts and Deadlines

A call without a timeout is a call that can hang forever, and a thread that hangs forever is a resource permanently removed from the pool. This is the mechanism behind most total outages: the failing dependency never returns errors, it simply stops answering, and the caller quietly runs out of threads, connections, or file descriptors while its own health checks still pass.

The naive fix is to put a timeout on every outbound call. That is necessary but not sufficient, because per-call timeouts do not compose. If service A has a 5 s timeout, calls B with a 5 s timeout, and B calls C with a 5 s timeout, then in the worst case the user has been waiting 5 s while A gave up long ago and every layer below is still doing work that nobody will read.

The composable version is a deadline: an absolute point in time, set at the edge, propagated with the request, and shortened at every hop. Each service computes its remaining budget rather than restarting a fixed timer, and any service that finds no budget left fails immediately instead of starting work that is guaranteed to be wasted.

 TIMEOUTS DO NOT COMPOSE; DEADLINES DO

 Per-call timeouts (each 5 s):
   t=0    gateway calls A (timeout 5 s)
   t=0    A calls B          (timeout 5 s)
   t=0    B calls C          (timeout 5 s)
   t=5    gateway gives up -> the user sees an error
   t=5..  A, B, and C are STILL WORKING on a dead request

 Deadline propagation (budget 1000 ms set once, at the edge):
   t=0    gateway: deadline = now + 1000 ms, sent in the request headers
   t=120  A receives it, 880 ms left, spends 200 ms, calls B with 680 ms
   t=340  B receives it, 660 ms left, calls C with 660 ms minus its own work
   t=900  C sees 100 ms left, knows its p99 is 300 ms -> FAILS FAST, frees the
          thread instead of burning it on work the caller will never read

import time

class Deadline:
    """An absolute time budget carried with the request across services."""

    def __init__(self, budget_ms: float):
        self.expires_at = time.monotonic() + budget_ms / 1000

    @classmethod
    def from_header(cls, header_value: str | None, default_ms: float):
        d = cls(default_ms)
        if header_value:                       # trust the caller's deadline, but
            incoming = float(header_value)     # never extend beyond our own max
            d.expires_at = min(d.expires_at, time.monotonic() + incoming / 1000)
        return d

    def remaining_ms(self) -> float:
        return max(0.0, (self.expires_at - time.monotonic()) * 1000)

def call_downstream(client, request, deadline: Deadline, expected_p99_ms: float):
    left = deadline.remaining_ms()
    # Reserve a slice for our own post-processing and the response trip home.
    usable = left - 30
    if usable <= expected_p99_ms * 0.5:
        # Not enough budget for this call to plausibly succeed: do not start it.
        raise DeadlineExceeded(f"{usable:.0f} ms left, need ~{expected_p99_ms} ms")
    return client.call(request, timeout_ms=usable,
                       headers={"x-deadline-ms": str(usable)})

Choosing the timeout value is its own decision. Set it from the dependency's measured latency distribution, not from a round number: a common starting point is p99 plus headroom, which fails the slowest 1% of calls rather than waiting for them. Too tight and you convert slow successes into errors and retries; too loose and you hold resources through the entire incident. Connect timeouts should be much shorter than read timeouts — a TCP connection either establishes quickly or the host is unreachable.

Key Takeaways

• Every outbound call needs a timeout; an unbounded call leaks the caller's thread and connection pool.
• Per-hop timeouts do not compose — total worst-case latency is their sum.
• Propagate an absolute deadline instead, and have each hop shrink it.
• A service with insufficient remaining budget should fail immediately rather than start doomed work.
• Derive timeout values from the measured latency distribution (p99 + headroom), and keep connect timeouts far shorter than read timeouts.

🧪 Practice

1. Given hops with p99s of 40 ms, 120 ms, and 250 ms in series, choose a sensible edge deadline and justify it.
2. Add deadline propagation to a two-service call chain and log how much budget each hop consumed.
3. Interview: A service has timeouts everywhere and still exhausted its thread pool during an incident. What could have happened? (Hint: consider what the timeout applies to, and what happens when retries sit inside it.)

Retries with Jitter

Retries exist because many failures are transient: a packet was dropped, an instance was restarting, a leader election was in progress. Retrying converts a brief blip into a slightly slower success, which is exactly what the user wants. Retries are also the single most common way a small incident becomes a large one, because a retrying client multiplies the load on a dependency at precisely the moment that dependency is struggling.

Three rules make retries safe. First, retry only what is safe to repeat — either the operation is idempotent, or the request carries an idempotency key so the server can deduplicate. Second, retry only what might succeed next time: a timeout or a 503 is worth retrying; a 400 or a 404 never is. Third, back off exponentially with jitter, so retries spread out in time instead of arriving as a synchronized wave.

The jitter part is not a detail. If a thousand clients all fail at the same instant and all retry after exactly 1 s, 2 s, 4 s, they arrive as three thundering herds, each landing on a dependency that has had no quiet moment in which to recover. Randomizing the delay converts a spike into a spread.

 WHY JITTER MATTERS

  No jitter (all clients failed at t=0):
     t=1s   ################################   1000 retries at once
     t=2s   ################################   1000 retries at once
     t=4s   ################################   1000 retries at once
            the dependency never gets a quiet interval to recover

  Full jitter (delay = random(0, min(cap, base * 2^n))):
     t=0-1s  ####  ###  ####  ###  ###  ####   spread across the window
     t=1-3s  ###  ####  ###  ####  ###   ##    load is smooth, recovery possible

import random

def backoff_delay(attempt, base_ms=100, cap_ms=10_000, strategy="full"):
    """Delay before retry number `attempt` (0-indexed)."""
    exponential = min(cap_ms, base_ms * (2 ** attempt))   # 100,200,400,...,10000
    if strategy == "none":
        return exponential                    # synchronized herds; avoid
    if strategy == "full":
        return random.uniform(0, exponential) # AWS "full jitter"; best spread
    if strategy == "equal":
        half = exponential / 2                # keeps a minimum wait, still spread
        return half + random.uniform(0, half)
    raise ValueError(strategy)

RETRYABLE = {408, 429, 500, 502, 503, 504}    # transient; a repeat may succeed

def call_with_retries(fn, deadline, max_attempts=3, budget=None):
    for attempt in range(max_attempts):
        if budget and not budget.allow():     # retry budget: see below
            raise RetriesExhausted("retry budget spent")
        try:
            return fn(timeout_ms=deadline.remaining_ms())
        except HttpError as e:
            if e.status not in RETRYABLE or attempt == max_attempts - 1:
                raise                          # non-retryable, or out of tries
        delay = backoff_delay(attempt)
        if delay > deadline.remaining_ms():    # never sleep past the deadline
            raise DeadlineExceeded()
        sleep_ms(delay)

The defense that matters most at scale is a retry budget: a cap on retries as a fraction of total requests, typically around 10%. Per-request retry limits do not bound systemic load, because "three attempts each" means the dependency sees 3x traffic when everything is failing. A budget makes the guarantee global — retries can rescue a small number of unlucky requests, but they can never triple the load.

class RetryBudget:
    """Allow retries only while they stay under `ratio` of normal traffic."""

    def __init__(self, ratio=0.1, window_requests=1000):
        self.ratio, self.window = ratio, window_requests
        self.requests = self.retries = 0

    def record_request(self):
        self.requests = min(self.window, self.requests + 1)

    def allow(self) -> bool:
        if self.retries >= self.requests * self.ratio:
            return False        # system-wide failure: stop amplifying it
        self.retries += 1
        return True

Finally, retries belong at one layer, not several. A client library that retries three times, calling a gateway that retries three times, calling a service that retries three times, produces 27 attempts for one user action. Pick the layer closest to the failure that can retry safely, and make every other layer pass failures through.

Key Takeaways

• Retry only idempotent (or idempotency-keyed) operations, and only on error classes that might succeed on a repeat.
• Exponential backoff with jitter turns synchronized herds into smooth load.
• A retry budget (roughly 10% of traffic) bounds systemic amplification in a way per-request limits cannot.
• Retries at multiple layers multiply: 3 x 3 x 3 = 27 attempts per user action.
• Never let a retry delay exceed the request's remaining deadline.

🧪 Practice

1. Implement full-jitter and equal-jitter backoff and plot the distribution of the first retry's delay for 1000 clients.
2. Add an idempotency key to a POST endpoint so retrying it cannot create two records.
3. Interview: A dependency has a 30-second brownout and your service stays down for 10 minutes afterwards. What is the likely mechanism? (Hint: what is the queue of retrying clients doing while the dependency tries to come back?)

Circuit Breaker

Retries assume the dependency will probably work next time. When a dependency is genuinely down, that assumption is false, and every retry costs a thread, a connection, and the full timeout duration for a result you can already predict. A circuit breaker is the component that notices this and stops calling.

The analogy is the electrical one it is named after. Rather than letting current keep flowing into a fault, the breaker trips and stays open until someone checks. In software the breaker sits in front of a dependency, watches the recent failure rate, and when that rate crosses a threshold it starts rejecting calls immediately, without attempting them. This converts slow failure into fast failure — which is what lets the caller stay healthy and shed load from the struggling dependency so it can recover.

 CIRCUIT BREAKER STATE MACHINE

           failure rate over threshold
                (within a window)
   [ CLOSED ] ---------------------------> [ OPEN ]
       ^                                       |
       |                                       | cool-down timer expires
       | probe succeeds                        v
       |                              [ HALF-OPEN ]
       +--------------------------------  |      |
                                          |      | probe fails
                                probe(s)  |      +---> back to OPEN
                                allowed   |            (timer restarts)
                                through   v

   CLOSED     calls pass through; failures are counted
   OPEN       calls fail instantly; the dependency gets zero load from us
   HALF-OPEN  a few trial calls decide whether it has recovered

import time

class CircuitBreaker:
    def __init__(self, failure_threshold=0.5, min_requests=20,
                 cool_down_s=30, half_open_probes=3):
        self.failure_threshold = failure_threshold   # trip above 50% failures
        self.min_requests = min_requests             # ignore tiny samples
        self.cool_down_s = cool_down_s
        self.half_open_probes = half_open_probes
        self.state = "CLOSED"
        self.failures = self.total = 0
        self.opened_at = 0.0
        self.probes_ok = 0

    def allow(self) -> bool:
        if self.state == "OPEN":
            if time.monotonic() - self.opened_at >= self.cool_down_s:
                self.state, self.probes_ok = "HALF_OPEN", 0
                return True                 # let one probe through
            return False                    # fail fast: no thread, no timeout
        return True

    def on_success(self):
        if self.state == "HALF_OPEN":
            self.probes_ok += 1
            if self.probes_ok >= self.half_open_probes:
                self._reset()               # dependency looks healthy again
        else:
            self.total += 1

    def on_failure(self):
        if self.state == "HALF_OPEN":
            self._trip()                    # still broken; restart the cool-down
            return
        self.total += 1
        self.failures += 1
        if (self.total >= self.min_requests
                and self.failures / self.total >= self.failure_threshold):
            self._trip()

    def _trip(self):
        self.state, self.opened_at = "OPEN", time.monotonic()

    def _reset(self):
        self.state, self.failures, self.total = "CLOSED", 0, 0

Three design points decide whether a breaker helps or hurts. The trip condition should be a failure rate over a rolling window with a minimum sample size, not a raw count — three failures out of three requests at 3 a.m. is noise, three hundred out of five hundred is a signal. The scope should be per dependency, and usually per endpoint or per instance, so a broken search endpoint does not stop calls to a healthy checkout endpoint on the same host. And an open breaker must have something to do instead of calling: pair it with a fallback (next topic), or it merely converts a slow error into a fast one.

Timeouts, retries, and breakers form a stack, and the order matters: a timeout bounds each attempt, retries handle isolated blips within the deadline, and the breaker watches the aggregate and cuts off retrying entirely once the failure is systemic.

Key Takeaways

• A breaker turns slow failure into fast failure, protecting the caller's resources and shedding load from the failing dependency.
• Trip on a failure rate over a rolling window with a minimum request count, never on a raw failure count.
• HALF-OPEN is what makes recovery automatic: a small number of probes decides whether to close.
• Scope breakers per dependency and endpoint, not per process.
• A breaker without a fallback only changes how fast the user sees an error.

🧪 Practice

1. Add a rolling time window to the breaker above so failures older than 60 seconds stop counting.
2. Explain what goes wrong if HALF-OPEN admits all traffic instead of a few probes.
3. Interview: How do you pick the failure-rate threshold for a breaker on a dependency that normally fails 2% of the time? (Hint: the threshold must sit well above the normal rate, and the minimum sample size decides how quickly noise can trip it.)

Bulkhead Isolation

A bulkhead is the wall that divides a ship's hull into watertight compartments; if one is breached, the others keep the ship afloat. The software pattern is the same idea applied to resources: partition your thread pools, connection pools, and queues so that one saturated dependency can only consume its own compartment.

The failure it prevents is specific and common. A service has one shared thread pool of 200 threads and calls five dependencies. One dependency slows from 20 ms to 5 s. Requests touching that dependency now occupy threads 250 times longer than before, and within seconds all 200 threads are waiting on it. The other four dependencies are perfectly healthy, and the service cannot call them, because there are no threads left. One slow dependency has taken down all functionality.

 SHARED POOL vs. BULKHEADS

  SHARED POOL (200 threads)
    [########################################]  100% occupied by calls to
                                                 the ONE slow dependency
    -> search, checkout, profile, and recommendations all fail together

  BULKHEADS
    payments        [####------]  40 threads, 12 busy   healthy
    search          [##########]  40 threads, 40 busy   saturated, contained
    profile         [###-------]  40 threads,  9 busy   healthy
    recommendations [##--------]  30 threads,  6 busy   healthy
    spare/critical  [#---------]  50 threads            reserved headroom

    -> only search degrades; everything else keeps serving

import threading
from contextlib import contextmanager

class Bulkhead:
    """A hard cap on concurrent calls to one dependency."""

    def __init__(self, name, max_concurrent, max_queued=0):
        self.name = name
        self.slots = threading.Semaphore(max_concurrent)
        self.queue_slots = threading.Semaphore(max_concurrent + max_queued)

    @contextmanager
    def acquire(self, wait_ms=0):
        # Reject immediately when even the queue is full: shedding load early is
        # cheaper than holding a caller's thread while it waits for a slot.
        if not self.queue_slots.acquire(blocking=False):
            raise BulkheadFull(self.name)
        try:
            if not self.slots.acquire(timeout=wait_ms / 1000):
                raise BulkheadFull(self.name)
            try:
                yield
            finally:
                self.slots.release()
        finally:
            self.queue_slots.release()

# Size each bulkhead with Little's Law: concurrency = throughput x latency.
#   payments: 200 rps x 0.15 s  = 30 concurrent  -> 40 slots with headroom
#   search:   800 rps x 0.04 s  = 32 concurrent  -> 40 slots with headroom
PAYMENTS = Bulkhead("payments", max_concurrent=40, max_queued=20)
SEARCH   = Bulkhead("search",   max_concurrent=40, max_queued=0)

def checkout(order):
    with PAYMENTS.acquire(wait_ms=50):     # never more than 40 threads in here
        return payments_client.charge(order)

Bulkheads exist at several granularities, and they compose. Within a process, separate pools per dependency. Within a fleet, separate instances per workload — a "critical" pool of servers that only handles checkout and a "bulk" pool that handles reports. Across tenants, per-tenant quotas so one customer's traffic spike cannot exhaust shared capacity — the "noisy neighbour" problem. At the extreme, cell-based architecture gives each group of customers a complete, independent copy of the stack, so the blast radius of nearly any failure is one cell.

The cost is utilization: partitioned resources cannot be borrowed, so some compartments sit idle while another is full. That is the trade being made deliberately — a few percent of efficiency in exchange for a bounded blast radius.

Key Takeaways

• Without isolation, one slow dependency consumes the shared pool and takes down unrelated functionality.
• Partition thread pools, connection pools, and queues per dependency; size each with Little's Law (concurrency = throughput x latency).
• Rejecting a call when a bulkhead is full is a feature: it sheds load instead of blocking a caller's thread.
• Isolation scales up through workload pools, per-tenant quotas, and cell-based architecture.
• The price is lower utilization; the purchase is a bounded blast radius.

🧪 Practice

1. Size bulkheads for a service handling 1200 rps where 30% of requests call a dependency with 250 ms p99 latency.
2. Extend the Bulkhead class to export the number of rejections so it can be alerted on.
3. Interview: A single tenant's batch job made your API unavailable for everyone. Which isolation boundaries would you add, and in what order? (Hint: start with the cheapest boundary that bounds one tenant's concurrency.)

Graceful Degradation

Availability is not binary. Between "fully working" and "returning 500s" there is a wide range of partially working, and the discipline of graceful degradation is deciding in advance which parts of the product to shed when resources are scarce so that the core function survives.

The reason this matters is that most systems have a small core and a large periphery. An e-commerce page needs the product, its price, and a working "add to cart" button. It also has recommendations, recently-viewed items, review summaries, inventory-at-nearby-stores, and a personalized banner. If any one of those secondary calls failing takes down the page, you have built a system whose availability is the product of a dozen availabilities — and you know from the series rule how bad that number is.

Degradation requires an explicit criticality classification, applied per feature, and enforced at the call site.

Tier	Meaning	Failure behaviour	Example
Critical	Product is useless without it	Fail the request	Price, add-to-cart, auth
Important	Noticeable but survivable loss	Serve stale, then omit	Inventory count, reviews
Optional	Enhancement only	Omit silently	Recommendations, banners
Deferrable	Must happen, not necessarily now	Enqueue for later	Analytics, emails, audit

CRITICAL, IMPORTANT, OPTIONAL = "critical", "important", "optional"

def render_product_page(product_id, user, load_shed_level=0):
    page = {}

    # Critical: no fallback exists. If this fails, the request fails.
    page["product"] = catalog.get(product_id, timeout_ms=200)
    page["price"]   = pricing.get(product_id, timeout_ms=150)

    # Important: try live, fall back to a cached value, then to omission.
    page["inventory"] = try_tiered(
        lambda: inventory.get(product_id, timeout_ms=100),
        lambda: cache.get(f"inv:{product_id}"),      # stale but useful
        default=None,
    )

    # Optional: skipped entirely under load. `load_shed_level` is raised by the
    # system itself when queue depth or CPU crosses a threshold -- degradation
    # is automatic, not something a human decides during an incident.
    if load_shed_level < 1:
        page["recommendations"] = try_tiered(
            lambda: recs.get(user.id, timeout_ms=80),
            lambda: recs.popular_in_category(page["product"].category),
            default=[],                               # empty list renders fine
        )
    return page

def try_tiered(*sources, default):
    for source in sources:                 # each tier is cheaper and staler
        try:
            value = source()
            if value is not None:
                return value
        except Exception:
            continue                                  # fall through to next tier
    return default

Two properties separate degradation that works from degradation that is only documented. First, it must be automatic and pre-decided: triggered by a measurable signal — queue depth, CPU, breaker state, error rate — not by a human choosing during an incident. Second, it must be visible: log and emit a metric whenever a feature is shed, or you will ship a release that silently disables recommendations for a month and nobody will notice until revenue moves.

It is also worth designing the degraded experience deliberately rather than letting it be an accident of which call failed. A read-only mode during a database failover, a "prices may be out of date" banner, a queued write acknowledged with "we'll process this shortly" — these are product decisions that need designers, and they convert a hard outage into a soft one.

Key Takeaways

• Classify every feature as critical, important, optional, or deferrable, and enforce the tier at the call site.
• A failing optional dependency must never fail the request; that is how a system inherits the product of a dozen availabilities.
• Degradation must trigger automatically from a measured signal, not from a human decision mid-incident.
• Emit metrics when features are shed, or silent degradation becomes permanent.
• Design the degraded experience (read-only mode, stale banners, queued writes) as a product feature.

🧪 Practice

1. Classify every backend call behind one page in a system you know, and justify each tier.
2. Add a load-shedding signal that raises load_shed_level when request queue depth exceeds a threshold, and drops it when the queue drains.
3. Interview: How would you make a checkout flow work while the reviews service is completely down? (Hint: begin by asking which of the data on that page the purchase actually depends on.)

Fallbacks and Default Responses

Degradation decides whether to serve something; a fallback decides what to serve instead. The two are separate skills, and fallbacks are where correctness risk concentrates, because a fallback is by definition an answer produced without the authoritative source.

Fallbacks are usefully ranked by how close they stay to the truth. Serving a slightly stale cached value is nearly always safe. Serving a computed approximation is often safe. Serving a static default is safe only when the default is conservative. And serving a wrong default can be far worse than serving an error — an authorization check that fails open, a fraud score that defaults to "clean," or a price that defaults to zero can each cost more than the outage would have.

Fallback	What it returns	Risk	Good for
Stale cache	The last known good value	Data is out of date	Profiles, catalogs, config
Approximation	A cheaper computed answer	Lower quality	Ranking, recommendations
Static default	A fixed conservative value	Wrong for some users	Feature flags, limits
Empty result	Nothing, rendered as absent	Missing functionality	Optional page sections
Queue for later	An acknowledgement, work deferred	Delayed effect, needs durability	Writes, notifications
Fail closed	An explicit error	Unavailable	Payments, authorization

The decision rule is to ask what a wrong answer costs compared to no answer. For a recommendation carousel, a stale or generic answer costs almost nothing and an error costs a broken page, so fall back aggressively. For an authorization decision, a wrong "allow" is a security incident, so fail closed — the correct fallback for a permission check is to deny, and the correct fallback for a rate limiter is usually to allow (protecting availability) unless the limiter guards something expensive.

def get_user_tier(user_id):
    """Tiered fallback chain, from authoritative to safe-but-generic."""
    try:
        return billing.get_tier(user_id, timeout_ms=100)      # authoritative
    except (Timeout, ServiceUnavailable):
        cached = cache.get(f"tier:{user_id}", allow_stale=True)
        if cached is not None:
            metrics.increment("fallback.tier.stale")          # ALWAYS measure
            return cached                                     # hours old, fine
        metrics.increment("fallback.tier.default")
        return "standard"      # conservative: never grants a tier they lack

def can_delete_account(user_id, target_id):
    try:
        return authz.check(user_id, "delete", target_id, timeout_ms=150)
    except (Timeout, ServiceUnavailable):
        metrics.increment("fallback.authz.denied")
        return False           # FAIL CLOSED: a wrong "allow" is unrecoverable

def allowed_by_rate_limiter(key):
    try:
        return limiter.check(key, timeout_ms=20)
    except (Timeout, ServiceUnavailable):
        metrics.increment("fallback.ratelimit.open")
        return True            # FAIL OPEN: the limiter protects capacity, and
                               # taking the site down to enforce it is worse

Two operational cautions. A fallback path that is only exercised during incidents is untested code running at the worst possible moment — exercise it deliberately (a small percentage of traffic, or a scheduled drill). And every fallback must be instrumented, because a system quietly serving stale data to 100% of users looks exactly like a healthy system on an availability dashboard. The metric that catches this is not "errors"; it is "fraction of responses served from a fallback."

Key Takeaways

• Choose the fallback by asking what a wrong answer costs versus no answer.
• Stale-but-correct beats fresh-but-absent for most read paths.
• Fail closed for security decisions; fail open for protective mechanisms whose absence is less harmful than an outage.
• Instrument every fallback: undetected fallback serving looks identical to health on an availability dashboard.
• Exercise fallback paths in normal operation, or they are untested code that runs only during incidents.

🧪 Practice

1. For five calls in a system you know, write down the fallback and the cost of it being wrong.
2. Add a metric and an alert that fires when more than 5% of responses are served from a fallback for over ten minutes.
3. Interview: Should a feature-flag service fail open or fail closed? (Hint: "it depends on the flag" is the start — what property of the flag decides it?)

Hedged Requests

Tail latency is a distinct problem from failure. A dependency can be up, healthy in aggregate, and still return the occasional very slow response — a garbage collection pause, a cold cache, a noisy neighbour, a queue behind one unlucky request. When a single user action fans out to many backend calls, those rare slow responses stop being rare: with 100 parallel calls each having a 1% chance of hitting the slow path, roughly 63% of user actions include at least one.

A hedged request attacks this directly. Send the request; if no answer has arrived by some threshold — typically the 95th percentile latency — send a second copy to a different replica and take whichever answer returns first. Because the threshold sits at p95, only about 5% of requests are ever duplicated, so the extra load is small, while the tail improves dramatically: the request now only appears slow if both attempts are slow, which for independent replicas is far less likely.

 HEDGING AT THE 95th PERCENTILE

  Without hedging:
    request ----------------------------------------------> 480 ms (p99 path)
                                                             user waits 480 ms

  With a hedge fired at p95 = 40 ms:
    attempt 1 ---------------------------------------------> 480 ms (ignored)
    attempt 2       |---------> 35 ms
                    ^
                    hedge sent at t=40 ms to a DIFFERENT replica
    response at t = 40 + 35 = 75 ms

  Cost: ~5% extra requests (only those slower than p95 are ever duplicated).
  Benefit: p99 collapses toward p95 + p50 rather than staying at the tail.

import asyncio

async def hedged_call(send, replicas, hedge_after_ms, deadline_ms):
    """Send to one replica; duplicate to another if it is slow. First wins."""
    tasks, delay = [], hedge_after_ms / 1000
    try:
        for replica in replicas:                     # usually 2 attempts, no more
            tasks.append(asyncio.create_task(send(replica)))
            done, pending = await asyncio.wait(
                tasks, timeout=delay, return_when=asyncio.FIRST_COMPLETED
            )
            if done:
                return done.pop().result()           # first answer wins
            delay = (deadline_ms - hedge_after_ms) / 1000   # wait out the rest
        done, _ = await asyncio.wait(tasks, timeout=delay,
                                     return_when=asyncio.FIRST_COMPLETED)
        if not done:
            raise DeadlineExceeded()
        return done.pop().result()
    finally:
        for t in tasks:
            t.cancel()          # ALWAYS cancel losers: an uncancelled hedge is
                                # duplicated work that the server still performs

Hedging comes with three hard preconditions. The operation must be idempotent or read-only, because two copies of it will sometimes both execute. The hedge must go to a different replica, or you are simply adding load to the machine that was already slow. And the system must bound the hedge rate — a global cap of a few percent — because during a broad slowdown every request exceeds the p95 threshold and naive hedging doubles the load on a system that is already struggling, which is a retry storm by another name.

A related and simpler variant is the tied request: send to two replicas immediately, and have each replica, on starting the work, tell the other to cancel. This trades a little coordination for a better tail than threshold-based hedging, and it is what some storage systems use for reads.

Key Takeaways

• Hedging targets tail latency, not failure: send a duplicate when the first attempt exceeds a threshold and take the first response.
• Setting the threshold at p95 costs about 5% extra requests and pulls p99 down toward p95 plus a typical response.
• Fan-out amplifies tails — with 100 parallel calls, a 1% slow path affects roughly 63% of user actions.
• Preconditions: idempotent work, a different replica, and cancellation of the loser.
• Cap the hedge rate globally, or a system-wide slowdown turns hedging into a load multiplier.

🧪 Practice

1. Compute the probability that at least one of 20 parallel calls hits a 1% slow path, and repeat for 50 and 200 calls.
2. Add a global rate cap to hedged_call so no more than 5% of requests are hedged in any minute.
3. Interview: Why is hedging dangerous for writes, and what would you need to change to make it safe? (Hint: think about what the server does when both copies of the request arrive and complete.)

<a id="93-failure-modes"></a>

9.3 Failure Modes

The patterns in the previous subchapter are answers. This subchapter is the set of questions — the specific ways distributed systems break at scale, most of which are emergent: no single component is defective, yet the system as a whole fails.

Cascading Failures

A cascading failure is one where the failure of a part shifts load onto the remaining parts, overloading them in turn, until the whole system is down. The defining property is that the system fails because it tried to compensate. This is why cascading failures are so damaging: the redundancy that was supposed to save you is the transmission mechanism.

The canonical form is capacity redistribution. Three nodes each handle 1000 rps at their limit, and the fleet serves 2100 rps — 700 per node, a comfortable 70%. One node dies. The remaining two now receive 1050 rps each, which is above their capacity. They slow down, their queues grow, health checks start failing, and one of them is marked unhealthy and removed. The last node now receives 2100 rps, and the system is gone. The trigger was one node; the cause was the absence of enough headroom to absorb it.

 CAPACITY REDISTRIBUTION CASCADE

  t=0   [A 700] [B 700] [C 700]     capacity 1000 each, 70% utilized
  t=1   [A  X ] [B1050] [C1050]     A dies; load redistributes above capacity
  t=2           [B slow] [C slow]   queues grow, latency climbs, timeouts begin
  t=3           [B  X  ] [C2100]    B fails a health check and is removed
  t=4                    [C  X  ]   total outage from ONE node failure

  The same fleet at 60% (1800 rps) survives: 900 rps each after A dies, under
  the 1000 limit. Headroom is not waste; it is the cascade brake.

Cascades have three other common shapes worth recognizing. Retry amplification, where clients respond to slowness by generating more load (next topic). Resource exhaustion, where one dependency's slowness consumes a shared pool (the bulkhead problem). And death spirals on recovery, where a restarted service is immediately hit by the full backlog of queued and retrying work, falls over again, and never gets a chance to warm caches or fill connection pools.

The mechanism underneath all of them is queueing. As utilization approaches 100%, queue length — and therefore latency — grows without bound. This is not a linear degradation; it is a cliff.

# Why load shedding beats queueing under overload.
arrival_rps, capacity_rps = 1200, 1000

# Option 1: queue the excess.
backlog = 0
for second in range(1, 61):
    backlog += arrival_rps - capacity_rps        # +200 requests every second
print(backlog)                                   # 12000 queued after one minute
print(f"{backlog / capacity_rps:.0f} s")         # 12 s of queue wait -- and by
# now every one of those queued requests has already timed out on the client,
# so the server spends 100% of its capacity producing answers nobody reads.

# Option 2: shed the excess immediately.
served, shed = capacity_rps, arrival_rps - capacity_rps
print(f"{served / arrival_rps:.0%} served fast, {shed / arrival_rps:.0%} rejected")
# 83% served fast, 17% rejected -- a far better outcome than 100% timing out.

# The queueing multiplier at various utilizations (M/M/1 intuition, 1/(1-u)):
for u in (0.5, 0.8, 0.9, 0.95, 0.99):
    print(f"utilization {u:.0%} -> latency multiplier {1 / (1 - u):>5.1f}x")
# 50% -> 2.0x   80% -> 5.0x   90% -> 10.0x   95% -> 20.0x   99% -> 100.0x

The defenses follow from the mechanism. Keep enough headroom that the loss of one failure domain does not push the rest above capacity. Shed load at the edge instead of queueing it, and shed by priority so the important work survives. Use bulkheads so exhaustion stays local. Make health checks shallow enough that an overloaded-but-working instance is not removed, worsening the very overload that caused it. And plan for recovery explicitly: bring capacity back gradually, drop the backlog rather than replaying it, and let caches warm before admitting full traffic.

Key Takeaways

• A cascading failure spreads through the compensation mechanism, which is why redundancy alone does not prevent it.
• The core arithmetic: after losing one of N domains, the rest must stay under capacity, so utilization must stay below (N-1)/N.
• Queueing under sustained overload is a cliff — latency grows without bound and the server does work no client is still waiting for.
• Shed load early and by priority; queueing excess work makes the outage worse.
• Design the recovery path: gradual traffic admission, backlog drops, cache warm-up.

🧪 Practice

1. For a 4-node fleet with 1000 rps capacity per node, find the maximum steady load that survives losing two nodes.
2. Add priority-aware shedding to a queue: drop background work before user requests when queue depth exceeds a threshold.
3. Interview: Your service crashed under load, you restarted it, and it crashed again immediately. What is happening and what do you do? (Hint: consider what is waiting for it at the moment it comes back, and what a cold cache means for its capacity.)

Retry Storms

A retry storm is the specific cascade in which clients, responding correctly at the individual level, collectively destroy the dependency they are trying to reach. Every client is behaving reasonably: the call failed, so it tries again. The aggregate behaviour is a load multiplier applied at exactly the moment the system has the least capacity to absorb it.

The arithmetic is brutal and it compounds across layers. Suppose a service normally receives 1000 rps and begins failing. Each client retries twice, so the service now sees 3000 rps. If the client is itself a service whose own callers retry twice, the chain multiplies again: three layers of "just two retries" is 27 attempts per original user request.

 RETRY AMPLIFICATION ACROSS LAYERS

   user request           1 attempt
       |
   mobile client   x3     3 attempts    (2 retries)
       |
   API gateway     x3     9 attempts    (2 retries)
       |
   service         x3    27 attempts    (2 retries)
       |
   database             27x its normal load, during an incident

  Each layer's policy is defensible in isolation. The product is not.

What makes storms self-sustaining is that they outlive their trigger. A dependency has a 30-second problem; by the time it recovers, a large backlog of retrying clients is waiting, and the instant it accepts connections it is hit with many times normal load and fails again. The system is now down not because of the original fault but because of the response to it — a metastable failure that persists until load is forcibly removed.

def storm_load(base_rps, retries_per_layer, layers, failure_rate):
    """Load reaching the bottom layer during a partial failure."""
    amplification = (1 + retries_per_layer) ** layers
    # Only failing requests are retried, so amplification scales with the
    # failure rate -- which is precisely why a partial failure escalates into
    # a total one as the failure rate climbs.
    effective = base_rps * (1 - failure_rate) + base_rps * failure_rate * amplification
    return effective

for rate in (0.0, 0.05, 0.25, 0.50, 1.0):
    print(f"failure rate {rate:>4.0%} -> {storm_load(1000, 2, 3, rate):>8.0f} rps")
# failure rate   0% ->     1000 rps
# failure rate   5% ->     2300 rps      <- already over capacity
# failure rate  25% ->     7500 rps
# failure rate  50% ->    14000 rps
# failure rate 100% ->    27000 rps      <- 27x, and it will not recover alone

The defenses are cumulative, and each one alone is insufficient:

Defense	What it bounds	Where it lives
Retry only one layer	The exponent in the amplification	Architecture rule
Exponential + jitter	The synchronization of retry waves	Client library
Retry budget (~10%)	Total systemic amplification	Client library
Circuit breaker	Retries against a dependency that is down	Caller
Load shedding / 429	What the server accepts at all	Server, edge
Retry-After header	When clients are allowed back	Server response
Randomized admission	The thundering herd on recovery	Server, edge

The server-side half is the part teams forget. A dependency that is failing should say so in a way clients must obey: return 429 or 503 with a Retry-After, and admit traffic gradually on recovery rather than opening the gates. You cannot rely on every client — especially third-party or mobile ones you cannot redeploy — to implement backoff correctly, so the server must be able to protect itself from badly behaved callers.

Key Takeaways

• Retry amplification is multiplicative across layers: 2 retries at 3 layers is 27x load.
• Storms are metastable — they persist after the original fault is fixed, because the recovering system is hit with the accumulated backlog.
• Retry at exactly one layer, with jitter and a global retry budget.
• Servers must defend themselves: 429/503 with Retry-After, load shedding, and gradual admission on recovery.
• You cannot assume clients behave; mobile and third-party callers cannot be redeployed mid-incident.

🧪 Practice

1. Compute the amplification for 1 retry across 4 layers, and compare it with 3 retries at 2 layers.
2. Implement Retry-After handling in a client so it waits the server-specified duration instead of its own backoff.
3. Interview: How would you bring a service back after a retry storm has taken it down? (Hint: it cannot simply start accepting traffic — what needs to happen to the waiting load first?)

Gray Failures and Slow Nodes

A crash is the easiest failure to handle: the process is gone, health checks fail, the load balancer removes it, and traffic moves. A gray failure is the difficult opposite — the component is up, responds to health checks, and is nevertheless failing to do its job for some or all of its users. It is the failure mode that produces the longest incidents, because monitoring says everything is fine.

Gray failures arise from a difference in perspective. The system's own view (a health-check endpoint returning 200) and the client's view (requests timing out) disagree, and automated recovery keys off the wrong one. The instance stays in rotation, keeps receiving its share of traffic, and keeps failing it.

 DIFFERENTIAL OBSERVABILITY: THE SIGNATURE OF A GRAY FAILURE

   health checker ---> GET /healthz ---> [ node C ] ---> 200 OK
                       (cheap, local, no dependencies)

   real client ------> POST /orders ---> [ node C ] ---> timeout
                       (needs the DB pool that node C has exhausted)

   Result: C stays in the load balancer pool, receives ~1/N of all traffic,
   and fails ~1/N of user requests indefinitely. Aggregate dashboards show
   an error rate of 1/N -- often small enough to sit under the alert threshold.

Common shapes: a node whose disk is failing and whose I/O now takes seconds; a process in a garbage-collection death spiral, alive but pausing constantly; a network interface with a high packet-drop rate; a node with a corrupted routing table that can reach some peers and not others; a replica that has fallen hours behind and is serving stale reads; a partially-deployed release where one instance runs incompatible code.

The counter-measures share one theme: judge health by what clients experience, per instance.

# Client-side outlier detection: peers vote on who is sick.
def detect_outliers(stats, min_requests=50, latency_factor=3.0, error_margin=0.10):
    """stats: {instance: (requests, errors, p99_latency_ms)}"""
    healthy = [s for s in stats.values() if s[0] >= min_requests]
    if len(healthy) < 3:
        return []                                  # too small a sample to judge
    median_p99 = sorted(s[2] for s in healthy)[len(healthy) // 2]
    median_err = sorted(s[1] / s[0] for s in healthy)[len(healthy) // 2]

    ejected = []
    for name, (reqs, errors, p99) in stats.items():
        if reqs < min_requests:
            continue
        # An instance is an outlier only relative to its PEERS -- this survives
        # a general slowdown, where every instance degrades together.
        if p99 > median_p99 * latency_factor or errors / reqs > median_err + error_margin:
            ejected.append(name)
    # Never eject so many that the survivors are overloaded: cap at ~10% of the
    # fleet, or outlier detection becomes the cause of a cascade.
    return ejected[: max(1, len(stats) // 10)]

print(detect_outliers({
    "node-a": (1000,  8,  45),
    "node-b": (1000, 11,  50),
    "node-c": (1000, 12, 900),     # 18x the median p99: a gray failure
    "node-d": (1000,  9,  47),
}))
# ['node-c']

Beyond outlier ejection, the practical toolkit is: deep health checks that actually touch critical dependencies (while keeping them cheap enough not to become a load source); per-instance metrics rather than fleet averages, since an average of 20 healthy nodes and one sick one looks healthy; latency percentiles per instance; replication-lag monitoring so a stale replica is removed from the read pool; and a low-friction "just restart or replace it" reflex — for a stateless node, replacement is almost always faster than diagnosis.

Key Takeaways

• Gray failures are partial: the component passes health checks while failing real work, so automation does not remove it.
• Their signature is differential observability — the system's view and the client's view disagree.
• Fleet averages hide them; measure error rate and latency per instance.
• Peer-relative outlier detection (with an ejection cap) removes sick nodes without over-reacting to a general slowdown.
• For stateless instances, replacement beats diagnosis.

🧪 Practice

1. Design a health check that would catch an exhausted database connection pool without itself becoming a significant load on the database.
2. Extend the outlier detector to require that an instance be an outlier in two consecutive windows before ejection, and explain why that helps.
3. Interview: One instance in a fleet of 30 is silently failing 100% of its requests. What does the aggregate error-rate dashboard show, and how would you have caught it? (Hint: compute the aggregate rate, then compare it with a typical alert threshold.)

Data Corruption and Bit Rot

Compute failures are loud and recoverable: restart the process and the bad state is gone. Data failures are quiet and permanent. A corrupted byte can sit undetected for months, be dutifully copied into every replica and every backup, and only surface when someone reads it — by which point every clean copy has aged out of retention.

Corruption is not rare at scale, because the underlying error rates are per-bit and the bit counts are enormous. Consumer drives are commonly specified around one unrecoverable read error per 10^14 bits, which is roughly one error per 12 TB read. Add cosmic-ray bit flips in non-ECC memory, firmware bugs, truncated writes during power loss, and — the most common source in practice — application bugs that write wrong data through a perfectly healthy storage stack.

# Why "rare" per-bit errors are not rare at fleet scale.
ure_rate_per_bit = 1e-14          # a common consumer-drive specification
bytes_read = 12e12                # 12 TB, one full drive scan

expected_errors = bytes_read * 8 * ure_rate_per_bit
print(f"{expected_errors:.2f}")   # 0.96 -- about one error per full-drive read

# A 1000-drive fleet scrubbed monthly reads 12 PB/month:
print(f"{1000 * expected_errors:.0f} errors/month expected")   # ~960
# Silent corruption is not an edge case; it is a monthly operational event.

The critical distinction is between corruption you detect and corruption you do not. Replication protects against loss but happily replicates corruption, so the defense must be verification, not duplication.

 DEFENSE IN DEPTH FOR DATA INTEGRITY

  write path
     application  --> checksum computed HERE, over the logical record
         |            (end-to-end: catches bugs in every layer below)
     serialization
         |
     filesystem   --> per-block checksums (ZFS, Btrfs)
         |
     device       --> ECC, T10-DIF
         v
     [ replica 1 ]  [ replica 2 ]  [ replica 3 ]
           \             |              /
            \            |             /
          background SCRUBBING: read every block, verify its checksum,
          repair from a good replica, and REPORT the rate

  read path: verify the end-to-end checksum before returning data to the user.

Four practices carry most of the weight. End-to-end checksums computed by the application and verified on read catch corruption introduced by any layer in between, including your own code. Background scrubbing finds errors while good copies still exist, rather than at read time when they may not. Immutable, versioned storage means a bad write creates a new version instead of destroying the old one. And backup verification — restoring and checking, on a schedule — is the only way to know a backup is not itself corrupt.

Two failure modes deserve specific mention. Backups that silently propagate corruption are defeated by keeping enough versions to reach back before the corruption began, which requires knowing when that was — hence the value of scrubbing metrics as a time series. And logical corruption caused by an application bug or a bad migration is invisible to every checksum in the stack, because the bytes were written correctly; catching it needs semantic validation: invariant checks, reconciliation jobs, and continuous comparison against a derived source of truth.

Key Takeaways

• Per-bit error rates make silent corruption a routine event at fleet scale, not an edge case.
• Replication copies corruption faithfully; only verification detects it.
• End-to-end application-level checksums catch errors that per-layer checksums cannot, including your own bugs.
• Scrub in the background so errors are found while a good copy still exists.
• Logical corruption passes every checksum; catching it requires semantic invariant checks and reconciliation.
• A backup is only verified once it has been restored and checked.

🧪 Practice

1. Compute the expected number of unrecoverable read errors when scrubbing 500 TB monthly at 10^-15 errors per bit.
2. Add an end-to-end checksum to a record write path and verify it on read, emitting a metric on mismatch.
3. Interview: A bad migration corrupted a column three weeks ago and all backups contain it. What now? (Hint: think about what other records of the same events exist — logs, event streams, downstream systems.)

Byzantine Failures

Most fault-tolerance work assumes fail-stop behaviour: components either work correctly or stop entirely. A Byzantine failure breaks that assumption — the component keeps running and produces wrong output, possibly different wrong output to different peers. The name comes from Lamport's Byzantine Generals problem, where generals must agree on a plan while some of them may be traitors sending contradictory messages.

This matters because ordinary redundancy is defenseless against it. Three replicas protect you against one crashing. They do not protect you against one returning plausible-but-wrong answers, because the failure detector cannot tell a wrong answer from a right one — nothing is timing out, nothing is erroring.

The classical result is the bound: tolerating f Byzantine faults requires 3f + 1 participants, versus 2f + 1 for crash faults. The extra replicas are what allow a correct majority to exist even when the faulty nodes actively lie.

Fault model	Nodes to tolerate f faults	Nodes for f=1	Typical protocol
Crash-stop	2f + 1	3	Raft, Paxos
Byzantine	3f + 1	4	PBFT, Tendermint

In practice, most engineers will never run a BFT consensus protocol; those live in blockchains, aerospace control systems, and a few financial settlement networks, because the cost — more replicas, more message rounds, cryptographic signatures on everything — is high. What is far more common is Byzantine-ish behaviour from ordinary bugs: a node running an old release computing prices with a stale tax table, a service returning HTTP 200 with a truncated body, a client sending malformed data that a lenient parser accepts, or a partially-failed disk returning data that is wrong but structurally valid.

# Practical Byzantine-resistant checks for ordinary systems.

def verified_response(responses):
    """Cross-check independent sources when a wrong answer is expensive."""
    # 1. Quorum agreement: require a majority to produce identical answers.
    from collections import Counter
    counts = Counter(canonical(r) for r in responses)
    answer, votes = counts.most_common(1)[0]
    if votes <= len(responses) // 2:
        raise NoQuorum(f"responses disagree: {dict(counts)}")
    return answer

def accept_write(record, signature, sender_key):
    # 2. Authentication: a wrong message from an untrusted source is rejected
    #    before it can influence anything.
    if not verify_signature(record, signature, sender_key):
        raise Untrusted(record.id)
    # 3. Validation: enforce invariants rather than trusting the sender.
    assert 0 <= record.amount <= MAX_TRANSFER, "amount out of range"
    assert record.currency in SUPPORTED, "unknown currency"
    assert record.version == EXPECTED_SCHEMA_VERSION, "schema mismatch"
    return commit(record)

def reconcile_daily(ledger_total, bank_statement_total):
    # 4. Reconciliation: an independently derived total is the cheapest possible
    #    Byzantine detector for a financial system.
    if abs(ledger_total - bank_statement_total) > TOLERANCE:
        raise ReconciliationBreak(ledger_total, bank_statement_total)

The practical takeaway is to spend the effort on the cheap defenses. Authenticate every internal call so a compromised or confused component cannot impersonate another. Validate inputs against invariants at every boundary, including from services you own. Use checksums and versions so a stale or truncated response is detectable. And reconcile against an independent source for anything financial. Full BFT consensus is reserved for the case where the participants are genuinely adversarial and mutually distrusting — which is exactly the case blockchains were built for.

Key Takeaways

• Byzantine failures produce wrong output rather than no output, so failure detectors and ordinary redundancy do not catch them.
• Tolerating f Byzantine faults needs 3f + 1 nodes, versus 2f + 1 for crashes.
• Full BFT protocols are rare outside adversarial settings because of their cost.
• The common real-world version is buggy or stale components, not malice.
• Cheap defenses — authentication, input validation, checksums, version checks, and reconciliation — cover most practical cases.

🧪 Practice

1. Explain why 3 nodes cannot tolerate 1 Byzantine fault, using a concrete scenario of contradictory messages.
2. Add schema-version validation to a service boundary so an old client's messages are rejected rather than misinterpreted.
3. Interview: A replica silently returns stale data after a failed upgrade. Is that a Byzantine failure, and how would you detect it? (Hint: ask whether the output is wrong versus absent, and what independent signal would reveal it.)

Correlated Failures

Every availability calculation in this chapter multiplies failure probabilities, and every one of those multiplications assumes independence. Correlated failures are what happens when that assumption is false — when one event takes out multiple "redundant" components at once — and they are the reason real systems fail far more often than their arithmetic predicts.

The arithmetic is unforgiving. Three replicas each with 1% failure probability give six nines if independent. If a single common cause has a 0.5% chance of taking out all three simultaneously, the system's failure probability is dominated entirely by that 0.5% — the redundancy contributes essentially nothing. Redundancy multiplies only the independent part of the risk; the correlated part it leaves untouched.

def system_failure_probability(p_individual, replicas, p_common_cause):
    independent = p_individual ** replicas       # all fail on their own
    return independent + p_common_cause          # plus the shared-fate event

print(f"{system_failure_probability(0.01, 3, 0.0):.8f}")     # 0.00000100
print(f"{system_failure_probability(0.01, 3, 0.005):.8f}")   # 0.00500100

# Adding a fourth and fifth replica moves the first term only:
print(f"{system_failure_probability(0.01, 5, 0.005):.8f}")   # 0.00500000
#
# The lesson: once a common cause dominates, MORE REPLICAS BUY NOTHING. The
# only useful work is removing the shared dependency.

Correlation hides in shared fate, and shared fate has more dimensions than people usually enumerate:

Dimension	Shared thing	Failure it causes
Physical	Rack, power feed, top-of-rack switch	All replicas in one rack disappear
Geographic	Datacenter, metro, power grid	A regional event takes the zone
Software	The same release, the same bug	A bad deploy fails everywhere at once
Configuration	One global config push	Instant, simultaneous, global misbehaviour
Time	Certificates, leap seconds, epochs	Everything expires at the same instant
Dependency	One DNS provider, one auth service	Every replica loses the same upstream
Behavioural	Synchronized retries, cron at :00	Self-inflicted load spikes
Capacity	A shared upstream at its limit	Failover exceeds the surviving capacity

The most common correlated failure in modern systems is not physical at all — it is a deploy or a configuration change, because those reach every replica by design. That is why the operational answer is to break the correlation in time: stagger rollouts across instances, zones, and regions, with soak intervals and automatic rollback, so that the blast radius of a bad change is one wave rather than the fleet.

# Turning a correlated risk into a sequential one.
ROLLOUT = [
    ("canary",  0.01, "30m"),   # 1% of one zone, watched closely
    ("zone-a",  0.33, "2h"),    # one failure domain at a time
    ("zone-b",  0.66, "2h"),    # a soak interval long enough for slow bugs
    ("zone-c",  1.00, None),    # (memory leaks, cron jobs, cache expiry)
]

def promote(stage, slo_burn_rate):
    # The gate is the SLO, not a human's impression of the dashboard.
    if slo_burn_rate > 2.0:
        return rollback(stage)
    return advance(stage)

# Config changes need the SAME treatment as code. A global config push is the
# fastest way ever invented to make an identical mistake in every region at
# the same instant.

Two further habits help. Make failure domains explicit in scheduling — anti-affinity rules so replicas of the same shard are never on one rack or one zone — and audit them, since a scheduler will happily co-locate everything if the constraint is not declared. And track the "expiry class" of correlated risks: certificates, tokens, license keys, and hardcoded dates fail simultaneously for everyone, and are best handled by automated rotation with alerting well before the deadline.

Key Takeaways

• Availability arithmetic assumes independence; correlated failures are where real systems deviate from the model.
• Once a common cause dominates, adding replicas buys nothing — remove the shared dependency instead.
• Shared fate spans physical, software, configuration, temporal, and dependency dimensions, not just hardware.
• The most common correlated failure today is a code or config rollout, so stagger both across failure domains with SLO-gated promotion.
• Declare anti-affinity explicitly and audit it; schedulers co-locate by default.

🧪 Practice

1. List every component your service depends on that has exactly one instance globally, including third parties.
2. Compute how much a fourth replica improves availability when a common cause contributes 0.2% failure probability.
3. Interview: Your three availability zones are independent, yet a single deploy took down all three. How do you prevent a recurrence without slowing releases to a crawl? (Hint: correlation in space was already broken — which dimension is left?)

<a id="94-reliability-practice"></a>

9.4 Reliability Practice

Architecture sets the ceiling on reliability; practice determines whether you reach it. This subchapter covers the operational machinery — how reliability is measured and negotiated, how failure is rehearsed deliberately, and how the organization learns from the failures it did not choose.

SLI, SLO, and SLA Definitions

Reliability discussions go in circles until three terms are separated. An SLI (Service Level Indicator) is a measurement: a number describing how well the service performed. An SLO (Service Level Objective) is an internal target for that number. An SLA (Service Level Agreement) is an external contract that attaches consequences — usually financial — to missing a target.

The order matters because each depends on the previous one. You cannot set a target for something you do not measure, and you must never sign a contract at the same threshold as your internal target: the SLO should be strictly stricter than the SLA, so that missing the SLO is an internal warning rather than a customer refund.

Term	Audience	Nature	Example
SLI	Engineers	Measurement	Fraction of requests under 300 ms, per 1 min
SLO	Internal	Target	99.9% of requests under 300 ms over 30 days
SLA	Customer	Contract	99.5% monthly, or a 10% service credit

A good SLI has a specific shape: a ratio of good events to valid events, measured where the user is. That definition rules out most of the metrics teams reach for first. CPU utilization is not an SLI — no user cares. Uptime of a process is not an SLI — the process can be up and useless. An average latency is a poor SLI, because an average hides exactly the tail the user notices.

# An SLI as a ratio of good events to valid events.
def availability_sli(requests):
    # "Valid" excludes what the service is not responsible for: client errors
    # (4xx, except 429), health checks, and traffic from load tests.
    valid = [r for r in requests
             if not r.is_synthetic and not (400 <= r.status < 500 and r.status != 429)]
    good = [r for r in valid if r.status < 500 and r.latency_ms < 300]
    return len(good) / len(valid) if valid else 1.0

# The threshold must come from user behaviour, not from what is convenient:
#   - 300 ms because measured abandonment climbs sharply past it
#   - measured at the load balancer, because that is closest to the user
#   - per endpoint class, because a search and a checkout are different products
SLO = {
    "checkout":  {"target": 0.999,  "latency_ms": 300,  "window_days": 30},
    "search":    {"target": 0.995,  "latency_ms": 500,  "window_days": 30},
    "reporting": {"target": 0.99,   "latency_ms": 5000, "window_days": 30},
}

Two choices inside an SLO definition do most of the work. The measurement point decides what you are actually promising: server-side metrics miss network and DNS failures that the client experiences, so client-side or load-balancer measurement is closer to the truth. The window decides how it behaves: a rolling 30-day window responds smoothly and is what engineering should watch, while calendar months are what contracts use because they are unambiguous for billing.

Finally, resist the temptation to set the target as high as possible. An SLO should sit just above the level at which users become unhappy, because every nine above that is spent effort that could have gone into features — and, more subtly, because users will come to depend on whatever reliability you actually deliver, regardless of what you promised.

Key Takeaways

• SLI = measurement, SLO = internal target, SLA = external contract with consequences; set the SLO strictly stricter than the SLA.
• A good SLI is a ratio of good events to valid events, measured close to the user, with an explicit definition of "valid".
• Percentile-based latency thresholds beat averages, which hide the tail.
• Set targets per endpoint class; a search and a checkout have different reliability requirements.
• Do not over-target: reliability above what users notice is spent budget.

🧪 Practice

1. Write an SLI definition for a file-upload endpoint, being explicit about which events count as valid.
2. Explain what changes if the same SLO is measured at the client rather than at the server, and which failures each misses.
3. Interview: A team proposes a 99.99% SLO for an internal batch reporting service. How do you respond? (Hint: ask what a user does differently when it is unavailable for ten minutes.)

Error Budgets

An error budget is the reframing that makes SLOs actionable. If the SLO is 99.9%, then 0.1% unreliability is not a failure — it is an allowance, deliberately granted. Over a 30-day window that allowance is 43.2 minutes of downtime, or 10,000 failed requests out of 10 million. Spending it is permitted. Running out has consequences.

This changes the conversation in a specific way. "Should we ship this risky feature?" is unanswerable in the abstract and usually resolved by whoever argues hardest. "Do we have budget remaining?" is a data question with an agreed-upon answer, which is why error budgets are the mechanism that stops the perpetual argument between teams that want to ship and teams that want stability. The policy, agreed in advance: budget remaining means ship; budget exhausted means the next work item is reliability, not features.

def error_budget_status(target, window_days, total_requests, failed_requests):
    allowed_failures = total_requests * (1 - target)
    consumed = failed_requests / allowed_failures if allowed_failures else 1.0
    return {
        "allowed":        round(allowed_failures),
        "used":           failed_requests,
        "remaining_pct":  round((1 - consumed) * 100, 1),
        "downtime_budget_min": round(window_days * 24 * 60 * (1 - target), 1),
    }

print(error_budget_status(0.999, 30, 10_000_000, 6_500))
# {'allowed': 10000, 'used': 6500, 'remaining_pct': 35.0,
#  'downtime_budget_min': 43.2}
#
# 35% of the month's budget left, with the month not yet over: this is the
# signal to slow risky changes down, before the budget is actually gone.

The operational half of error budgets is the burn rate: how fast the budget is being consumed relative to the rate that would exactly exhaust it over the window. A burn rate of 1 means you will finish the window exactly at the target. A burn rate of 14.4 means one hour of this consumes 2% of a 30-day budget. Alerting on burn rate rather than on a raw error threshold is what gives you alerts that are both fast for severe incidents and quiet for minor ones.

def burn_rate(observed_error_rate, target):
    return observed_error_rate / (1 - target)      # 1.0 = exactly on budget

# The multi-window, multi-burn-rate alerting policy (SRE standard practice):
ALERT_POLICY = [
    # burn  long window  short window  budget consumed  action
    (14.4,  "1h",        "5m",         "2%",            "page immediately"),
    (6.0,   "6h",        "30m",        "5%",            "page"),
    (3.0,   "24h",       "2h",         "10%",           "ticket"),
    (1.0,   "72h",       "6h",         "10%",           "ticket"),
]
# Two windows are required: the long one establishes that the burn is real,
# the short one confirms it is STILL happening, so the alert resolves once the
# incident ends instead of firing for hours afterwards.

print(f"{burn_rate(0.0144, 0.999):.1f}")   # 14.4 -> a 1.44% error rate pages
print(f"{burn_rate(0.0012, 0.999):.1f}")   #  1.2 -> slightly over budget, ticket

Three practices keep budgets honest. Agree the exhaustion policy in writing before it matters — a freeze on risky changes, or a fixed allocation of the next sprint to reliability work — since negotiating it during an incident guarantees it will not be honoured. Account for planned spend: deliberately taking a maintenance window is a legitimate budget expense, and one of the healthier uses of a surplus. And when a budget is consistently unspent, treat that as its own signal: either the target is too loose to be meaningful, or the team is being more conservative than the business needs.

Key Takeaways

• The error budget is the inverse of the SLO — an explicit, permitted allowance of unreliability.
• It converts "should we ship?" from an argument into a data question with a pre-agreed policy.
• Alert on burn rate, not raw error counts; multi-window policies page fast for severe burns and file tickets for slow ones.
• Write the exhaustion policy down before you need it.
• A budget that is never spent means the target is wrong or the team is too cautious.

🧪 Practice

1. Compute the request-count error budget for a 99.95% SLO over a month with 40 million requests.
2. Implement a burn-rate calculation over both a 1-hour and a 5-minute window and fire only when both exceed the threshold.
3. Interview: The error budget is exhausted on day 12 of a 30-day window and a major launch is scheduled for day 15. What do you do? (Hint: the policy should already exist — what should it say, and who agreed to it?)

Chaos Engineering

Chaos engineering is the practice of injecting failures into a system on purpose, in order to find out what actually happens rather than what the design document claims. Its premise is that every complex system contains failure modes nobody has thought of, and the choice is only between discovering them during a controlled experiment on a Tuesday afternoon or during a real incident at 3 a.m.

The name is unfortunate — the practice is not about randomness. A chaos experiment is a scientific experiment with a steady state, a hypothesis, a blast radius, and an abort condition. "Randomly break things in production" is not chaos engineering; it is an outage you caused yourself.

 THE CHAOS EXPERIMENT LOOP

  1. STEADY STATE     Define a measurable normal: "checkout success rate > 99.5%
                      and p99 < 400 ms", measured continuously.

  2. HYPOTHESIS       State the expected outcome BEFORE running: "killing one of
                      three payment-service instances will not change the steady
                      state, because the LB ejects it within 5 s."

  3. BLAST RADIUS     Choose the smallest injection that tests the hypothesis:
                      one instance, one AZ, 1% of traffic, one dependency.

  4. ABORT CONDITION  Define, in advance and automatically: "roll back if the
                      success rate drops below 99% for 30 s."

  5. INJECT           Run the fault. Observe.

  6. LEARN            Hypothesis held  -> confidence, widen the radius next time.
                      Hypothesis broke -> you found a real defect. Fix, re-run.

class ChaosExperiment:
    def __init__(self, name, hypothesis, steady_state, abort_if, blast_radius):
        self.name, self.hypothesis = name, hypothesis
        self.steady_state = steady_state      # callable -> bool
        self.abort_if = abort_if              # callable -> bool
        self.blast_radius = blast_radius

    def run(self, inject, revert, duration_s, sample_s=5):
        if not self.steady_state():
            return "SKIPPED: system was not healthy before the experiment"
        inject(self.blast_radius)
        try:
            for _ in range(duration_s // sample_s):
                if self.abort_if():
                    return f"ABORTED: {self.name} broke the abort condition"
                sleep(sample_s)
            return ("HELD" if self.steady_state() else "DISPROVED") + f": {self.hypothesis}"
        finally:
            revert()                          # ALWAYS revert, even on exception

experiment = ChaosExperiment(
    name="payment-instance-loss",
    hypothesis="Losing 1 of 3 payment instances does not affect checkout.",
    steady_state=lambda: checkout_success_rate(60) > 0.995,
    abort_if=lambda:     checkout_success_rate(30) < 0.99,
    blast_radius={"service": "payments", "instances": 1, "traffic_pct": 100},
)

The useful experiments follow a rough progression, and it is worth running them in this order because each level assumes the previous one works: kill a process; kill an instance; exhaust a resource (CPU, memory, disk, file descriptors); add latency to a dependency; make a dependency return errors; partition the network; lose an entire availability zone; expire a certificate or a credential; and finally, lose a region.

Two organizational points determine whether chaos engineering sticks. It must run in production eventually — staging environments differ in scale, traffic patterns, and data, and the failure modes that matter mostly live in those differences — but it should start in staging, and reach production only once the tooling, abort conditions, and observability are proven. And every experiment should be treated as a test that can regress: automate it, run it continuously, and alert when a hypothesis that used to hold stops holding.

Key Takeaways

• Chaos engineering is controlled experimentation, not randomness: steady state, hypothesis, blast radius, abort condition.
• State the hypothesis before injecting; a surprise is the finding.
• Start small and in staging; graduate to production, where the failure modes that matter actually live.
• Progress from process kills to resource exhaustion to dependency latency to zone and region loss.
• Automate experiments so a regression in resilience is caught like any other regression.

🧪 Practice

1. Write a full experiment definition — steady state, hypothesis, blast radius, abort condition — for adding 500 ms of latency to a cache.
2. Explain why an experiment must verify the steady state before injecting, and what a skipped run tells you.
3. Interview: Leadership refuses to allow chaos experiments in production. What do you propose? (Hint: what smaller-radius, lower-stakes version would build the evidence needed to change that decision?)

Disaster Recovery Planning

Disaster recovery is what happens when the resilience patterns are not enough: a region is gone, a database is unrecoverable, a bad migration destroyed a table, or a credential compromise requires rotating everything at once. DR planning is the work of deciding, in advance and calmly, what you will do — because the defining property of a disaster is that nobody will be thinking clearly when it starts.

A DR plan is built from the objectives set earlier: for each system, its RTO and RPO, its recovery mechanism, and the specific procedure that achieves them. What distinguishes a plan that works from a document that exists is whether it is specific enough to be executed by someone who did not write it, at 4 a.m., under pressure, possibly without access to the systems they normally use.

Disaster class	Typical trigger	Primary defense
Infrastructure loss	Zone or region outage	Multi-zone/region failover
Data loss	Storage failure, deletion	Backups, PITR, replication
Data corruption	Bad migration, application bug	Versioned backups, PITR
Dependency loss	Provider outage, expired contract	Fallbacks, second provider
Security incident	Compromised credentials or supply chain	Rotation, isolation, forensics
Human error	Accidental deletion or config push	Soft delete, staged rollout, undo

# A DR runbook expressed as executable structure, so it can be tested in CI
# and rehearsed as code rather than read as prose.
RUNBOOK = {
    "name": "primary-database-region-loss",
    "rto_target_min": 30,
    "rpo_target_min": 5,
    "preconditions": [
        "standby replica lag < 5 min",              # verified continuously
        "standby has current schema version",
        "DNS TTL for db endpoint <= 60 s",          # bounds client convergence
    ],
    "steps": [
        ("verify",   "confirm primary is unreachable from 2 independent probes"),
        ("fence",    "revoke primary's write credentials"),   # prevents split-brain
        ("promote",  "promote standby, record new epoch/fencing token"),
        ("repoint",  "update service discovery + DNS to the new primary"),
        ("verify",   "run smoke tests: read, write, and read-your-write"),
        ("scale",    "scale the standby region's app tier to full capacity"),
        ("announce", "update status page and notify support"),
    ],
    "rollback": "do NOT fail back automatically; the old primary may hold writes "
                "that were never replicated -- reconcile them manually first",
    "last_rehearsed": "2026-07-14",     # a plan not rehearsed this quarter is
                                        # an untested assumption
}

Three things are consistently missing from DR plans and consistently needed during disasters. Access: the credentials, VPN, and consoles required to execute the plan must not themselves depend on the system that is down — a plan that requires logging in through the failed SSO provider is not a plan. Communication: who declares the disaster, who has authority to trigger an irreversible failover, and where the team coordinates when the usual chat tool is part of the outage. Decision criteria: the specific, measurable condition under which you fail over rather than continue trying to repair, because the most expensive DR mistake is spending ninety minutes hoping the primary recovers.

Finally, treat fail-back as a separate plan needing its own care. Returning to the original region after a failover involves reconciling divergent data and is usually more dangerous than the failover itself; it should be scheduled deliberately, not performed in the middle of an incident.

Key Takeaways

• A DR plan turns RTO and RPO into a specific procedure, executable by someone who did not write it.
• Cover data corruption and human error, not only infrastructure loss — they need different mechanisms.
• Recovery access must not depend on the failed system, including SSO, VPN, and chat.
• Define in advance who declares a disaster and the measurable condition that triggers failover.
• Fail-back is a separate, more dangerous procedure; plan and schedule it apart.

🧪 Practice

1. Write the preconditions and steps for recovering from an accidentally dropped production table.
2. Identify every credential or tool your DR procedure requires and check whether each survives the failure it is meant to address.
3. Interview: Your primary region has been degraded for 20 minutes with no ETA. Do you fail over? (Hint: the answer should not depend on your mood — what should have been decided beforehand?)

Game Days and Failure Drills

A game day is a scheduled exercise in which a team responds to a simulated incident, end to end, using the real tools and the real runbooks. Chaos experiments test the system; game days test the sociotechnical system — the people, the documentation, the alerting, the escalation paths, and the handoffs. Most incident time is spent on detection, coordination, and decision-making, and those are exactly the parts that only a drill exercises.

The value comes from the gaps a drill exposes, which are reliably mundane and reliably serious: the runbook references a dashboard that was deleted; the on-call engineer lacks the permission the procedure requires; the alert fires to a channel nobody watches; the failover script has a syntax error introduced six months ago; nobody knows who is allowed to approve a customer-facing status update. None of these show up in design reviews, and all of them cost real minutes during a real outage.

 GAME DAY STRUCTURE (a half-day exercise)

  BEFORE
    - Choose a scenario tied to a real risk, not an exotic one
    - Decide the environment: production (highest fidelity, needs maturity)
      or a production-like staging environment
    - Assign roles: facilitator, incident commander, responders, scribe,
      and an observer whose only job is to note friction
    - Announce it. A surprise drill tests fear responses, not procedures.

  DURING
    - Facilitator injects the fault and answers questions in character
      ("the dashboard shows X") without giving away the cause
    - Responders follow the real runbooks, using the real tools
    - Scribe records a timeline with timestamps: every observation, decision,
      and action -- the timeline is the primary artifact
    - Facilitator can inject complications: "the on-call is unreachable",
      "the status page is also down"

  AFTER
    - Debrief within 24 hours, while memory is fresh
    - Produce action items with owners and dates, exactly as for a real
      incident -- an undone action item makes the next drill pointless
    - Measure: time to detect, time to diagnose, time to mitigate

# Tracking whether drills are actually improving response, not just occurring.
DRILLS = [
    # scenario                  date          detect_min  mitigate_min  gaps_found
    ("az-loss",                "2026-02-10",     8,          31,         6),
    ("db-failover",            "2026-03-24",    12,          44,         9),
    ("az-loss",                "2026-05-19",     3,          14,         2),   # repeat
    ("dependency-brownout",    "2026-07-07",    15,          52,        11),
]

for name, date, detect, mitigate, gaps in DRILLS:
    print(f"{date}  {name:24} detect {detect:>3}m  mitigate {mitigate:>3}m  gaps {gaps}")

# The repeated az-loss scenario is the useful signal: detection fell from 8 to
# 3 minutes and mitigation from 31 to 14, which means the action items from the
# first run were actually completed. Repeat scenarios deliberately -- a drill
# you never repeat cannot show improvement.

A few practical rules. Rotate the incident commander so expertise is not concentrated in one person who will inevitably be on holiday. Include the adjacent functions — support, communications, and whoever owns the status page — because their part of the response is usually the least rehearsed. Start with announced drills in staging and progress toward less-announced drills in production only as confidence grows. And treat the action items as real work with owners and deadlines; a drill whose findings are never fixed is theatre that consumes a day and improves nothing.

Key Takeaways

• Game days test people, runbooks, alerting, and escalation — the parts of incident response that chaos experiments do not touch.
• The gaps found are usually mundane and expensive: stale runbooks, missing permissions, unwatched alert channels.
• Assign explicit roles, and have a scribe produce a timestamped timeline.
• Repeat scenarios so improvement is measurable in detection and mitigation time.
• Action items must have owners and dates, or the exercise is theatre.

🧪 Practice

1. Design a two-hour game day for the loss of a critical third-party dependency, including roles and the complications the facilitator can inject.
2. Take an existing runbook and list every assumption it makes about access, tooling, and available people.
3. Interview: How would you run a game day for a team that has never done one, without breaking anything? (Hint: fidelity and risk are separate dials — which can you turn down first?)

Blameless Postmortems

A postmortem is the written analysis produced after an incident. "Blameless" means it is written on the premise that everyone involved acted reasonably given the information they had at the time, and that the useful question is therefore never "who did this" but "what made this the reasonable thing to do, and what would have made the harm impossible."

The justification is practical, not sentimental. Systems fail in ways that make the failing action look sensible from inside; hindsight makes it look obvious only because you now know the outcome. More importantly, blame changes behaviour in exactly the wrong direction: in a blaming culture people delay reporting, describe events vaguely, avoid touching fragile systems, and stop volunteering the details that would have prevented a recurrence. The information you need to fix the system is held by the people a blaming process punishes.

"Blameless" does not mean "consequence-free" or "no owners." Action items have owners and deadlines. What is removed is the search for a person to hold responsible for the outcome, and it is replaced with a search for the conditions that allowed the outcome.

 THE SAME EVENT, TWO FRAMINGS

 BLAMING
   "An engineer ran a migration without checking the row count, which locked
    the table for 40 minutes. Reminded the team to be careful with migrations."
   -> Learns nothing. Nothing changes. Next engineer makes the same mistake.

 BLAMELESS
   "A migration acquired an exclusive lock on a 400M-row table for 40 minutes.
    Contributing conditions:
      - The migration tool gives no warning about lock duration or table size.
      - Staging holds 10K rows, so the migration ran there in under a second.
      - There is no lock-timeout default, so the lock could be held unbounded.
      - No alert exists for lock waits, so detection came from user reports.
    Actions:
      - Set lock_timeout on all migrations (owner: A, due: 03-14)
      - Warn in CI when a migration targets a table over 1M rows (owner: B)
      - Seed staging with production-scale row counts (owner: C)
      - Alert on lock wait time over 30 s (owner: D)"
   -> Four defenses added. The same mistake now cannot cause the same outage.

A useful postmortem has a predictable structure: a short summary; user impact stated in concrete terms (how many users, which functionality, for how long, what it cost); a timestamped timeline from first anomaly to resolution; the analysis of contributing conditions; what went well, which matters because it identifies the defenses worth keeping; and action items with owners and dates. The timeline is the most valuable section, and the most commonly rushed — it is what exposes that detection took 22 of the incident's 45 minutes.

# Postmortem metrics turn individual incidents into a trend you can act on.
def incident_phases(timeline):
    """timeline: dict of phase -> minutes"""
    total = sum(timeline.values())
    for phase, minutes in timeline.items():
        print(f"{phase:12} {minutes:>4} min  ({minutes / total:>5.1%})")
    return total

incident_phases({
    "detect":   22,     # from first user impact to a human being aware
    "diagnose": 15,     # from awareness to knowing what to do
    "mitigate":  6,     # from decision to impact ending
    "resolve":  90,     # from mitigation to full repair (not user-visible)
})
# detect         22 min  (16.5%)
# diagnose       15 min  (11.3%)
# mitigate        6 min  ( 4.5%)
# resolve        90 min  (67.7%)
#
# Detection dominates the USER-VISIBLE portion (22 of 43 minutes). Investing in
# alerting would cut this incident's impact roughly in half; investing in a
# faster fix would not, because the fix was already quick once identified.

Two habits keep postmortems from becoming an archive nobody reads. Ask "why was this possible?" repeatedly rather than stopping at the first plausible cause — "the deploy was bad" is a stopping point, "a bad deploy could reach 100% of traffic in 90 seconds with no automatic rollback" is a finding. And track action items to completion with the same seriousness as feature work: the single strongest predictor of repeat incidents is a backlog of postmortem actions nobody closed.

Key Takeaways

• Blameless means assuming people acted reasonably given what they knew, and analysing the conditions instead of the person.
• Blame suppresses the information the process depends on; it makes future incidents more likely, not fewer.
• Blameless still means owners and deadlines for action items — it removes punishment, not accountability.
• A timestamped timeline is the highest-value section; it exposes where the time actually went, which is usually detection.
• Keep asking why the failure was possible until you reach a systemic condition, and close the action items.

🧪 Practice

1. Rewrite a blaming incident summary ("X pushed a bad config") into a blameless analysis with at least three contributing conditions.
2. Build a phase breakdown for a past incident you know and identify which phase would benefit most from investment.
3. Interview: An engineer's change caused a major outage. What does the postmortem say about them? (Hint: consider what the process should optimize for — the appearance of accountability, or the outage never being possible again.)

---

<a id="10-architectural-patterns-and-styles"></a>