13. Performance and Cost Engineering

Performance and cost are the same conversation held in different units: every millisecond you remove is bought with money, complexity, or reliability, and every dollar you save eventually shows up somewhere in the latency distribution. This chapter covers how to analyse where time actually goes, the optimisation techniques that buy the most latency per unit of complexity, the testing that proves a system holds up under load before your users discover otherwise, and the cost engineering that keeps a fast system affordable.

<a id="131-performance-analysis"></a>

13.1 Performance Analysis

Optimisation without measurement is superstition, so this subchapter starts with how to reason quantitatively about latency: budgeting it across a request path, understanding why the slow tail dominates user experience, the mathematical laws that cap what parallelism and scaling can buy, the queueing behaviour that explains why systems collapse at high utilisation, and the profiling tools that point at the code actually responsible.

Latency Budgets

Users do not experience your service; they experience a page, a tap, or an API call that fans out across many services. When a product decision says "search results in under 300 ms at the 95th percentile", that 300 ms has to be divided among everyone on the path — DNS, TLS, the gateway, the search service, the ranking call, the database, the render. A latency budget is that division written down, so each team knows how much time it owns and can tell when it has overspent.

The analogy is a household budget. Total income is fixed by the product requirement; each line item gets an allocation; if one line overspends, another must underspend or the total breaks. The value is not the arithmetic — it is that overspending becomes visible and attributable instead of being an argument about whose service is "slow".

Two rules govern how budgets compose:

• Sequential calls add. If A calls B and then C, A's latency is at least B + C plus A's own work.
• Parallel calls take the maximum, not the sum — but the maximum over several slow tails is worse than any individual tail (see the next topic).

 BUDGET FOR "SEARCH RESULTS" — 300 ms at p95, user-perceived

 client --> edge/CDN --> gateway --> search-svc --+--> index shard   (parallel)
                                                  +--> ranking-svc
                                                  +--> ads-svc

 +------------------------------+--------+-------------------------------+
 | Segment                      | Budget | Notes                         |
 +------------------------------+--------+-------------------------------+
 | Network client -> edge       |  60 ms | Geography, not code (5.4)     |
 | TLS + edge routing           |  15 ms | Session resumption helps      |
 | Gateway (authn, quota)       |  10 ms | Assumes token cache hit       |
 | search-svc own work          |  25 ms | Parse, plan, merge            |
 | MAX(index, ranking, ads)     | 120 ms | Parallel fan-out              |
 | Serialization + render start |  40 ms | Payload size matters          |
 | Reserve (headroom)           |  30 ms | ~10% unallocated on purpose   |
 +------------------------------+--------+-------------------------------+
 | TOTAL                        | 300 ms |                               |
 +------------------------------+--------+-------------------------------+

The reserve line matters. A budget allocated to 100% has no room for the day a dependency regresses by 20 ms, so every such day becomes an emergency. Ten to twenty percent of unallocated headroom converts emergencies into planned work.

from dataclasses import dataclass

@dataclass
class Segment:
    name: str
    budget_ms: float
    measured_p95_ms: float
    parallel_group: str | None = None    # segments sharing a group run concurrently

def evaluate(segments: list[Segment], target_ms: float) -> None:
    sequential = [s for s in segments if s.parallel_group is None]
    groups: dict[str, list[Segment]] = {}
    for s in segments:
        if s.parallel_group:
            groups.setdefault(s.parallel_group, []).append(s)

    # Sequential work adds; each parallel group contributes only its slowest leg.
    measured = sum(s.measured_p95_ms for s in sequential)
    measured += sum(max(g, key=lambda s: s.measured_p95_ms).measured_p95_ms
                    for g in groups.values())

    for s in segments:
        delta = s.measured_p95_ms - s.budget_ms
        flag = "OVER" if delta > 0 else "ok  "
        print(f"{flag} {s.name:<22} budget={s.budget_ms:6.1f} "
              f"actual={s.measured_p95_ms:6.1f} delta={delta:+6.1f}")

    headroom = target_ms - measured
    print(f"\ntotal p95 = {measured:.1f} ms / target {target_ms:.0f} ms "
          f"({'PASS' if headroom >= 0 else 'FAIL'}, headroom {headroom:+.1f} ms)")

evaluate([
    Segment("network_client_edge", 60, 58),
    Segment("tls_edge_routing",    15, 14),
    Segment("gateway_auth",        10, 22),             # over budget: cache misses
    Segment("search_own_work",     25, 21),
    Segment("index_shard",        120, 96,  "fanout"),
    Segment("ranking_svc",        120, 131, "fanout"),  # slowest leg sets the group
    Segment("ads_svc",            120, 44,  "fanout"),
    Segment("serialize_render",    40, 37),
], target_ms=300)

Budgets should be enforced, not merely documented: express each segment as an SLO (12.2), alert when a dependency exceeds its allocation, and derive client timeouts from the budget rather than from an arbitrary round number. A timeout larger than the budget guarantees that when a dependency is slow, the caller waits past the point where the answer is still useful to anyone.

Key Takeaways

• A latency budget divides an end-to-end target among every segment of the request path so overspending is visible and attributable.
• Sequential segments add; parallel fan-outs contribute only their slowest leg.
• Leave 10-20% unallocated as headroom, or every dependency regression becomes an incident.
• Budget at a percentile (p95/p99), never at the mean.
• Derive timeouts from the budget: waiting longer than the budget produces an answer nobody can still use.

🧪 Practice

1. Draw the request path for an endpoint you know and assign a budget to each segment totalling the product requirement, including headroom.
2. Measure the real p95 of two segments and compute how much budget the remaining segments still have.
3. Interview: A page must render in 200 ms but calls five services. How do you allocate the budget? (Hint: think about which calls can run concurrently, and what happens to the tail when you take a maximum over five distributions.)

Tail Latency and P99

Averages lie about user experience. If 99 requests take 10 ms and one takes 1 000 ms, the mean is about 20 ms — a number that describes no request that actually happened. Percentiles describe the distribution honestly: p50 is the median request, p99 is slower than 99% of the others, and everything above p95 is the tail.

Tails matter more than intuition suggests, for two reasons. First, tail events are not spread evenly across users — the same users hit them repeatedly, because slowness correlates with having more data, a worse connection, or a larger account. Second, tail latency amplifies under fan-out. If a request calls 100 independent services and each is slow 1% of the time, the chance that at least one is slow is 1 - 0.99^100 = 63%. A service's p99 becomes the user's median.

 LATENCY DISTRIBUTION

  count
    |############
    |############
    |############
    |############===
    |################=--      -    -         -
    +----+--------+-------+-------+---------+----------> latency
        p50      p90     p95     p99      p99.9
        10ms     40ms    75ms   380ms     2.1s
                                 |<------ the tail ------>|

 mean = 34 ms  <- describes nobody

 FAN-OUT AMPLIFICATION: P(at least one slow leg) = 1 - (1 - p)^N   [p = 1%]
   N=1 -> 1.0%     N=10 -> 9.6%     N=50 -> 39.5%     N=100 -> 63.4%

import asyncio

def tail_amplification(p_slow: float, fan_out: int) -> float:
    """Probability that at least one of `fan_out` independent legs is slow."""
    return 1 - (1 - p_slow) ** fan_out

for n in (1, 5, 10, 50, 100):
    print(f"fan-out {n:>3}: {tail_amplification(0.01, n):.1%} of requests hit a slow leg")

# ---- Hedged requests: send a second copy once the first exceeds p95 ---------
# Costs a few percent extra load and cuts the tail sharply, because the chance
# that BOTH copies are slow is far lower than the chance that one is.
async def hedged_call(send, hedge_after_ms: float, timeout_ms: float):
    first = asyncio.create_task(send())
    done, _ = await asyncio.wait({first}, timeout=hedge_after_ms / 1000)
    if done:
        return first.result()                       # fast path: no extra load
    second = asyncio.create_task(send())            # hedge only for the slow ~5%
    done, pending = await asyncio.wait(
        {first, second}, timeout=timeout_ms / 1000,
        return_when=asyncio.FIRST_COMPLETED)
    for task in pending:
        task.cancel()                               # always cancel the loser
    if not done:
        raise TimeoutError("both copies exceeded the budget")
    return done.pop().result()

Common causes of tail latency, roughly in order of how often they turn out to be the culprit: queueing behind other work, garbage collection or compaction pauses, cold caches, lock contention, retries after a timeout, noisy neighbours on shared hardware, and connection setup on a cold path. Note how many are shared-resource effects rather than slow code — which is why profiling one request in isolation often fails to explain a bad p99.

Technique	How it helps the tail	Cost
Hedged / tied requests	A second copy usually avoids the unlucky server	A few percent extra load
Narrower fan-out	Fewer chances to draw a slow leg	Less parallelism
Per-request timeout + retry	Caps the worst case	Retry storms if uncontrolled
Load-aware balancing (6.2)	Avoids servers already queueing	Balancer complexity
Prioritising interactive work	Batch jobs stop blocking user requests	Scheduling machinery
Cache warm-up on deploy	Removes cold-start outliers	Memory, slower rollouts

One aggregation rule deserves emphasis: percentiles do not average. The mean of ten servers' p99 values is not the fleet p99, and averaging p99 over 24 hourly windows is not the daily p99. Aggregate the underlying histograms and compute the percentile once, which is exactly why histogram metrics (12.1) exist.

Key Takeaways

• Report p50, p95, p99, and p99.9 — the mean describes no real request.
• Fan-out amplifies tails: with N parallel legs the chance of hitting a slow one is 1 - (1 - p)^N, so a service p99 becomes a user-visible median.
• Never average percentiles across servers or time windows; merge histograms instead.
• Most tail latency comes from shared-resource contention — queueing, GC, locks — not from slow application code.
• Hedged requests buy large tail reductions for a small, bounded load increase.

🧪 Practice

1. Take a service's latency histogram and compute p50/p95/p99, then describe which group of users each number represents.
2. Compute the probability of hitting a slow leg with a fan-out of 20 when each leg is slow 0.5% of the time, then propose two ways to reduce it.
3. Interview: Why is averaging the p99 values from ten servers wrong? (Hint: think about what a percentile is computed from, and whether that computation can be undone from the summary.)

Amdahl's and Universal Scalability Laws

Adding machines or threads feels like it should make things proportionally faster, and for a while it does. Two laws explain why it always stops, and why it sometimes goes backwards.

Amdahl's Law begins with the observation that every program has a part that cannot be parallelised — reading configuration, taking a lock, writing the final result. If a fraction s of the work is serial, then no matter how many workers N you add, speed-up is capped:

speedup(N) = 1 / (s + (1 - s) / N)      ->  1 / s   as N -> infinity

With 5% serial work the ceiling is 20x, however many cores you buy. The analogy is a kitchen: hiring more cooks parallelises the chopping, but there is one oven and one plating step, and eventually everyone is waiting on the oven.

The Universal Scalability Law (USL) adds the term Amdahl ignores: workers do not merely fail to help, they interfere with each other. Coordination — cache coherence, lock handoff, consensus rounds, cross-shard chatter — grows with the number of pairs of workers, so throughput eventually declines as capacity is added:

C(N) = N / (1 + a(N - 1) + b * N(N - 1))
       a = contention (serialisation)     b = coherency (crosstalk)

 THROUGHPUT vs WORKERS

  x |          ideal (linear)
    |        ,'
    |      ,'                       ___ Amdahl ceiling (1/s)
    |    ,'                 _.--''''
    |  ,'            _.--''
    |,'         _.-'         .------.   USL: peaks, then FALLS
    |        _.-'         .-'        `--.___
    |     .-'          .-'                   `-----____
    +---------------------------------------------------------> N workers
                          ^ optimal N when b > 0

def amdahl(n: int, serial_fraction: float) -> float:
    return 1 / (serial_fraction + (1 - serial_fraction) / n)

def usl(n: int, alpha: float, beta: float) -> float:
    """alpha = contention (queueing on shared resources),
       beta  = coherency  (cost of keeping workers consistent with each other)."""
    return n / (1 + alpha * (n - 1) + beta * n * (n - 1))

print(f"{'N':>5} {'Amdahl(s=5%)':>14} {'USL(a=.03,b=.0005)':>20}")
for n in (1, 2, 4, 8, 16, 32, 64, 128, 256):
    print(f"{n:>5} {amdahl(n, 0.05):>13.1f}x {usl(n, 0.03, 0.0005):>19.1f}x")

# Where does adding workers stop paying?
best = max(range(1, 513), key=lambda n: usl(n, 0.03, 0.0005))
print(f"\npeak throughput at N = {best}; past this point more workers do LESS work")

The practical readings:

Observation	Likely cause	What to do
Speed-up flattens at a fixed multiple	Serial fraction (Amdahl)	Shrink the serial section
Throughput peaks, then declines	Coherency term (USL b)	Remove cross-worker coordination
Latency rises while throughput is flat	Contention (a), queueing	Shard state, narrow lock scope

The lesson for system design is that scaling problems are usually coordination problems. Sharding by key so workers never touch the same state (4.5), eliminating shared counters, and accepting eventual consistency where the product allows (8.2) all reduce b — which raises the ceiling far more reliably than buying hardware does.

Key Takeaways

• Amdahl's Law caps speed-up at 1/s; a 5% serial fraction means a 20x ceiling regardless of machine count.
• USL adds a coherency term, so throughput can peak and then decline as workers are added.
• Fit both curves to real measurements — the parameters tell you whether to attack serialisation or coordination.
• The most durable scaling wins come from removing shared state, not from adding capacity.
• Always ask "what is the serial part?" before buying more machines.

🧪 Practice

1. Compute Amdahl speed-up at N = 8, 64, and 1024 for serial fractions of 1%, 5%, and 20%.
2. Measure throughput at several thread counts for a small program, fit USL parameters, and locate the peak.
3. Interview: Your service gets slower when you scale from 40 to 60 instances. What is happening? (Hint: think about what all those instances share — a database, a lock, a cache-invalidation broadcast — and how that cost grows with instance count.)

Queueing Theory Basics

Every component that can be busy is a queue: threads waiting for a CPU, requests waiting for a connection from the pool, jobs waiting for a worker. Queueing theory explains the most counter-intuitive fact in performance engineering — waiting time does not grow linearly with load, it grows hyperbolically, exploding as utilisation approaches 100%.

The intuition is a supermarket checkout. At 50% utilisation the cashier is often idle when you arrive, so you walk straight up. At 95% utilisation you almost always arrive mid-transaction with people ahead of you. The arrival rate merely doubled; your wait grew by an order of magnitude, because queues exist to absorb variability and there is no idle time left to absorb it with.

For the simplest model (M/M/1: random arrivals, random service times, one server) with utilisation rho = arrival rate / service rate:

average time in system   W = service_time / (1 - rho)

 RESPONSE TIME vs UTILISATION (service time = 10 ms)

  W |                                                       ,
 200|                                                     ,'
    |                                                   ,'
 150|                                                ,-'
    |                                            ,--'
 100|                                      _,---'
    |                              __,---''
  50|              ______,,,---''''
  10+----''''''
    +----+-----+-----+-----+-----+-----+-----+-----+-----+----> rho
        0.1   0.2   0.4   0.5   0.7   0.8   0.9  0.95  0.99

    rho=0.5 -> 20 ms     rho=0.9 -> 100 ms     rho=0.95 -> 200 ms
    The last 5% of capacity costs more latency than the first 90%.

import math

def mm1_response_time_ms(service_ms: float, utilisation: float) -> float:
    if utilisation >= 1:
        return float("inf")           # arrivals outpace service: the queue is unbounded
    return service_ms / (1 - utilisation)

for rho in (0.3, 0.5, 0.7, 0.8, 0.9, 0.95, 0.98, 0.99):
    w = mm1_response_time_ms(10, rho)
    print(f"rho={rho:>4}: response={w:8.1f} ms (of which queueing {w - 10:7.1f} ms)")

# Multiple servers behind ONE queue pool their variability, so the same total
# utilisation hurts less than N independent queues at that utilisation.
def servers_needed(arrival_rate: float, per_server_rate: float, target_rho: float) -> int:
    return math.ceil(arrival_rate / (per_server_rate * target_rho))

print(servers_needed(arrival_rate=900, per_server_rate=100, target_rho=0.7))  # 13

Three design consequences follow directly:

• Run below roughly 70% utilisation for latency-sensitive services. The remaining headroom is not waste; it is the buffer that keeps p99 bounded and absorbs spikes and instance failures. Batch systems, where only throughput matters, can safely run near 100%.
• Prefer one shared queue over per-server queues. A single queue feeding N workers pools variability, so no worker sits idle while another has a backlog. This is precisely why least-outstanding-requests balancing beats round-robin (6.2).
• Reduce variability, not just the mean. Queue wait depends on the variance of service times as much as on the average, so one pathological slow query degrades everyone behind it. Bounded queues with load shedding (6.3) let a saturated system degrade instead of collapsing.

Key Takeaways

• Wait time follows W = S / (1 - rho): it explodes as utilisation nears 1.
• Target roughly 70% utilisation for interactive services; headroom is latency insurance, not waste.
• Variance in service time inflates queueing as much as the mean does.
• One queue in front of N workers outperforms N separate queues.
• An unbounded queue turns overload into a total outage; bound queues and shed load instead.

🧪 Practice

1. Plot response time against utilisation from 0.1 to 0.99 for a 20 ms service time and mark where p99 stops being acceptable.
2. Compute how many workers a 900 rps workload needs at 70% utilisation if each worker sustains 100 rps.
3. Interview: CPU sits at 85% and latency is terrible, but adding two instances fixes it completely. Explain why. (Hint: split response time into service and waiting, and ask how each responds to lower utilisation.)

Little's Law

Little's Law is the closest thing performance engineering has to a free lunch: it holds for any stable system, regardless of arrival distribution, service distribution, or scheduling policy.

L = lambda * W          concurrency = throughput * latency

L is the average number of items in the system, lambda the average arrival rate, and W the average time each item spends inside. The power is that any two give you the third — which lets you size thread pools, predict queue depths, and sanity-check load-test output without simulating anything.

Think of a restaurant: if 30 diners arrive per hour and each stays 45 minutes, the average number of occupied tables is 30 * 0.75 = 22.5. With fewer tables than that, a queue forms at the door and never clears.

                lambda = 500 req/s               W = 0.2 s
    arrivals ----------------> [  SYSTEM  ] ----------------> departures

                        L = lambda * W = 500 * 0.2 = 100

    => 100 requests are in flight at steady state, so you need at least 100
       concurrent slots (threads, connections, goroutines) to sustain 500 rps
       at 200 ms. Fewer slots, and arrivals queue outside the system.

def concurrency(throughput_rps: float, latency_s: float) -> float:
    """L = lambda * W — how many requests are in flight at steady state."""
    return throughput_rps * latency_s

def max_throughput(pool_size: int, latency_s: float) -> float:
    """Invert the law: with a fixed pool, latency caps throughput."""
    return pool_size / latency_s

print(concurrency(500, 0.200))        # 100 in-flight requests
print(max_throughput(50, 0.200))      # 250 rps ceiling with a 50-slot pool

# A worked sizing problem ----------------------------------------------------
target_rps, p99_latency_s, safety = 1200, 0.150, 1.3
pool = concurrency(target_rps, p99_latency_s) * safety
print(f"size the pool for ~{pool:.0f} concurrent slots")     # ~234

# The trap: if downstream latency doubles, the SAME pool halves your throughput.
print(max_throughput(234, 0.300))     # 780 rps — a latency regression silently
                                      # became a capacity regression

That last line is the important one. Little's Law explains the most common cascading failure in distributed systems: a dependency slows down, W rises, in-flight requests L climb until the pool is exhausted, and the caller — whose own code did not change at all — starts rejecting traffic. This is why timeouts, bulkheads, and circuit breakers (9.2) are capacity controls as much as reliability controls: they cap W so L cannot grow without bound.

Question	Apply Little's Law as
How big should the thread pool be?	L = target_rps * p99_latency plus margin
What throughput can this pool sustain?	lambda = L / W
Why is my queue depth 4 000?	W = L / lambda — compute the implied wait
Are my load-test numbers self-consistent?	Check L = lambda * W across the metrics

Key Takeaways

• L = lambda * W holds for any stable system — no distributional assumptions required.
• Use it to size pools: required concurrency = target throughput * latency.
• Inverted, it shows a fixed pool turns any latency regression directly into a throughput regression.
• Queue depth plus arrival rate gives wait time without instrumenting it.
• It is the fastest sanity check on load-test output: if L != lambda * W, one of the reported numbers is wrong.

🧪 Practice

1. A service handles 2 000 rps at 50 ms average latency. How many requests are in flight, and what is the minimum worker count?
2. A queue holds 10 000 messages and consumers process 500/s. How long does a newly enqueued message wait?
3. Interview: A dependency's latency doubles while your traffic is unchanged, and your service starts shedding load. Explain the mechanism. (Hint: hold lambda fixed, double W, and ask what happens to L relative to your pool size.)

Profiling and Flame Graphs

Profiling answers the only question worth asking before optimising: where does the time actually go? Engineers are famously bad at guessing — the code that looks expensive is usually not the hot path, and the real cost often hides in serialisation, allocation, logging, or a lock nobody thought about.

There are two families, and confusing them wastes days:

Profiler type	How it works	Good for	Watch out for
Sampling	Interrupt periodically, record the stack	Low overhead, safe in production	Misses rare-but-slow events
Instrumenting	Record entry and exit of every function	Exact call counts	High overhead, distorts behaviour
On-CPU	Samples only running threads	Compute-bound hot spots	Blind to blocking
Off-CPU / wall	Samples blocked time as well	I/O waits, lock contention	Noisier, needs kernel support

The classic mistake is profiling on-CPU when the problem is off-CPU. If a service sits at 15% CPU with terrible latency, a CPU profile shows nothing useful, because the time is spent waiting — on a database, a lock, or a connection pool.

A flame graph turns a pile of stack samples into one picture. Each box is a function; a box sits on top of its caller; the width is the proportion of samples containing that frame. Width means cost. Order along the x-axis is alphabetical, not chronological — a common misreading.

 FLAME GRAPH — width = share of samples (cost), height = call depth

 |------------------------------------------------------------------| main
 |------------------------------------------------------------------| handle_request
 |--------------------|--------|----------|---------|---------------|
 | serialize_json     | authz  | db_query | render  | log_event     |
 |  42%               |  6%    |  18%     |  9%     |  25%          |
 |-------|            |        |          |         |----|          |
 | encode|            |        |          |         |fmt |          |
 |  38%  |            |        |          |         |22% |          |
 ^^^^^^^^                                            ^^^^
 The widest LEAF frames are where time is actually spent. Here JSON encoding
 and log formatting cost 60% between them — neither is "business logic".

# A reproducible micro-investigation: profile first, then act on the widest frame.
import cProfile, pstats, io, json, random

def build_payload(n=2000):
    return [{"id": i, "name": f"item-{i}", "tags": ["a", "b", "c"],
             "score": random.random()} for i in range(n)]

def handle_request(payload):
    return json.dumps(payload)            # the suspected hot spot

payload = build_payload()
profiler = cProfile.Profile()
profiler.enable()
for _ in range(200):
    handle_request(payload)
profiler.disable()

buf = io.StringIO()
pstats.Stats(profiler, stream=buf).sort_stats("cumulative").print_stats(10)
print(buf.getvalue())

# In production, use a CONTINUOUS sampling profiler instead (perf,
# async-profiler, py-spy, pprof) at around 99 Hz: cheap enough to leave on
# permanently, which is the only way to catch a regression you did not predict.
# 99 rather than 100 avoids sampling in lockstep with timers that fire on
# round intervals.

A disciplined optimisation loop looks like this: measure a baseline under realistic load, profile to find the widest frame, form a hypothesis about why it is wide, change one thing, re-measure, and keep the change only if the improvement exceeds measurement noise. Optimising anything other than the widest frame is bounded by Amdahl's Law — making 5% of the runtime twice as fast buys you 2.5%.

Key Takeaways

• Profile before optimising; intuition about hot paths is unreliable.
• Sampling profilers are cheap enough for continuous production use; instrumenting profilers distort what they measure.
• Low CPU with high latency means you need an off-CPU or wall-clock profile.
• In a flame graph, width is cost and x-axis order carries no meaning.
• Attack the widest frame first — Amdahl's Law caps what any narrow frame can return.

🧪 Practice

1. Profile a script that reads, transforms, and serialises data, and identify the top three frames by cumulative time.
2. Generate a flame graph for a running service and explain its widest leaf.
3. Interview: CPU utilisation is 20% but p99 latency is 3 seconds. How do you find the cause? (Hint: think about what a CPU profiler cannot see, and which signals show time spent blocked rather than running.)

<a id="132-optimization-techniques"></a>

13.2 Optimization Techniques

Once analysis has told you where the time goes, a small set of techniques accounts for most of the latency ever recovered from real systems. This subchapter covers reusing expensive connections, amortising per-operation overhead through batching, trading CPU for bandwidth with compression, moving work out of the request path by precomputing it, shaping the read and write paths independently, and tuning concurrency so the machine is busy without being congested.

Connection Reuse and Pooling

Opening a connection is disproportionately expensive compared to using one. A new TCP connection costs a round trip for the handshake; adding TLS costs one or two more; a database connection then costs authentication, session setup, and often server-side memory allocation. On a 40 ms round trip, that is 80-120 ms spent before a single byte of your query moves — for a query that takes 2 ms to run.

The fix is to treat connections as durable resources rather than per-request objects. Keep them open and hand them back and forth: HTTP keep-alive at the protocol level, connection pools at the client level.

 WITHOUT REUSE (per request)              WITH REUSE (per request)

 SYN -------->                            (connection already established)
      <-------- SYN/ACK                   query ------->
 ACK -------->                                 <------- result
 TLS ClientHello ->                       total: ~1 RTT + service time
      <-- ServerHello, cert
 TLS Finished ---->
      <-------- Finished
 auth handshake -->
      <-------- ok
 query ---------->
      <-------- result
 total: ~4-5 RTT + service time

Pool sizing is where most teams go wrong, in both directions. Too small and requests queue for a slot — Little's Law again, with the pool as L. Too large and you overwhelm the database: each connection consumes memory and a backend process or thread, and beyond the server's parallelism the extra connections only add contention. A database with 8 cores does not go faster with 500 connections than with 40; it goes slower.

from sqlalchemy import create_engine

engine = create_engine(
    "postgresql://app@db-primary/orders",
    pool_size=20,            # steady-state connections kept open per app instance
    max_overflow=10,         # temporary extras during bursts; closed after use
    pool_timeout=2,          # FAIL FAST when saturated instead of queueing forever
    pool_recycle=1800,       # reopen every 30 min: dodges silent NAT/firewall drops
    pool_pre_ping=True,      # cheap liveness check before handing out a connection
)

# Sizing sanity check across the fleet -------------------------------------
INSTANCES, DB_MAX_CONNECTIONS = 24, 500
per_instance = 20 + 10
print(f"worst case = {INSTANCES * per_instance} connections vs server limit "
      f"{DB_MAX_CONNECTIONS}")   # 720 > 500: the fleet can exhaust the database

# Two ways out:
#  1. shrink the pool so INSTANCES * (pool_size + overflow) < DB_MAX_CONNECTIONS
#  2. put a connection proxy (PgBouncer, ProxySQL, RDS Proxy) in front, so many
#     short app-side connections multiplex onto few real server connections

import httpx

# One long-lived client per process. Creating a client per request throws away
# the connection pool, the TLS session cache, and any HTTP/2 multiplexing.
client = httpx.Client(
    limits=httpx.Limits(max_connections=100, max_keepalive_connections=20),
    timeout=httpx.Timeout(connect=1.0, read=2.0, write=2.0, pool=0.5),
    http2=True,              # many requests share one connection; no head-of-line
)                            # blocking at the HTTP layer

def fetch_user(user_id: str) -> dict:
    return client.get(f"https://users.internal/v1/{user_id}").json()

Practical rules: one client or pool per process, created at start-up; pool timeouts short so saturation surfaces as a fast error rather than a hang; recycle connections periodically to survive network devices that drop idle flows; and separate pools for workloads with different criticality so a slow reporting query cannot starve checkout of connections (the bulkhead pattern, 9.2).

Key Takeaways

• Connection setup can cost several round trips; reuse turns that into zero.
• Create one pool or HTTP client per process and share it — never per request.
• Oversized pools hurt the server; size against the database's connection limit across the whole fleet, not per instance.
• Set a short pool checkout timeout so saturation fails fast instead of hanging.
• Use separate pools for workloads of different criticality, and a proxy when instance count makes direct pooling unworkable.

🧪 Practice

1. Measure the latency difference between 100 requests over a fresh connection each and 100 over one reused connection.
2. Compute the fleet-wide worst-case connection count for a service with 30 instances, pool size 15, overflow 10 — and decide whether it fits a 400 connection limit.
3. Interview: Your app has 50 instances each with a pool of 20 against one Postgres primary. What breaks and how do you fix it? (Hint: multiply, then think about what sits between the app and the database.)

Batching and Coalescing

Every operation carries fixed overhead: a network round trip, a syscall, a query parse, a lock acquisition, a durability flush. When that fixed cost dominates the marginal cost of the work itself, doing many small operations is enormously wasteful. Batching amortises the fixed cost over many items.

The analogy is grocery shopping: driving to the shop for one onion, then again for one carrot, is absurd — you make a list. The N+1 query problem is exactly that mistake expressed in code.

 N+1 QUERIES                            BATCHED

 SELECT * FROM orders LIMIT 100         SELECT * FROM orders LIMIT 100
 SELECT * FROM users WHERE id=1         SELECT * FROM users WHERE id IN (...)
 SELECT * FROM users WHERE id=2
 ... 100 more round trips               2 round trips total
 100 * 1.2 ms = 120 ms                  1.2 ms + 3 ms = 4.2 ms

Coalescing is batching's sibling for duplicate work: when many callers ask for the same thing at once, do the work once and share the answer. This is what prevents a cache expiry from becoming a stampede against the database (5.2).

import asyncio
from collections import defaultdict

class Batcher:
    """Collects keys for a few milliseconds, then issues one bulk load."""

    def __init__(self, bulk_load, max_batch=100, window_ms=5):
        self._bulk_load, self._max_batch, self._window = bulk_load, max_batch, window_ms / 1000
        self._pending: dict[str, asyncio.Future] = {}
        self._timer: asyncio.Task | None = None

    async def load(self, key: str):
        if key in self._pending:
            return await self._pending[key]        # COALESCING: share the in-flight call
        fut = asyncio.get_running_loop().create_future()
        self._pending[key] = fut
        if len(self._pending) >= self._max_batch:
            await self._flush()                    # size trigger: batch is full
        elif self._timer is None:
            self._timer = asyncio.create_task(self._after_window())  # time trigger
        return await fut

    async def _after_window(self):
        await asyncio.sleep(self._window)          # bounded added latency
        await self._flush()

    async def _flush(self):
        batch, self._pending = self._pending, {}
        if self._timer:
            self._timer.cancel()
            self._timer = None
        try:
            results = await self._bulk_load(list(batch))   # ONE round trip
            for key, fut in batch.items():
                fut.set_result(results.get(key))
        except Exception as exc:
            for fut in batch.values():             # never leave callers hanging
                fut.set_exception(exc)

async def bulk_load_users(ids: list[str]) -> dict:
    rows = await db.fetch("SELECT * FROM users WHERE id = ANY($1)", ids)
    return {r["id"]: r for r in rows}

users = Batcher(bulk_load_users)
# 100 concurrent calls to users.load(...) become one query with <= 5 ms added latency.

Every batch needs both a size trigger and a time trigger. Size alone means the last few items wait forever during a traffic lull; time alone means a burst builds an unbounded batch. The time window is a deliberate latency-for- throughput trade: 5 ms of added latency to remove 99 round trips is nearly always right, while 500 ms is not.

Layer	Batching mechanism	Typical win
Database reads	IN (...), bulk loaders	Removes N+1 round trips
Database writes	Multi-row INSERT, COPY	One flush amortised over rows
Message queues (7.1)	Producer batching, linger.ms	Fewer network and broker calls
RPC	Bulk endpoints, request pipelining	Fewer headers, handshakes
Logs and metrics	Buffered async export	Removes I/O from the hot path

Batching has real costs to weigh: added latency for the first item, larger blast radius if a batch fails (one poison item can fail 100), memory held while the batch fills, and harder retry semantics — a partial failure needs per-item results, not one exception.

Key Takeaways

• Batching amortises fixed per-operation cost; it wins whenever overhead dominates the work itself.
• Always combine a size trigger with a time trigger, and keep the window small.
• Coalescing deduplicates concurrent identical requests — the standard defence against cache stampedes.
• Batches enlarge the blast radius of a failure; return per-item results so retries stay precise.
• The N+1 query is the canonical batching bug; look for it in any ORM-backed list endpoint.

🧪 Practice

1. Find an N+1 query in a codebase and rewrite it as a single bulk query, measuring both.
2. Add a batcher with a 10 ms window in front of a hot key-value lookup and measure the change in round trips and p99.
3. Interview: What is the downside of increasing a batch window from 5 ms to 100 ms? (Hint: think about who pays for the wait, and what happens to memory and failure blast radius as the batch grows.)

Compression Trade-offs

Compression trades CPU time for bytes on the wire or on disk. Whether that trade pays depends entirely on the relative cost of the two, which varies by hop: over a mobile network, sending fewer bytes is almost always worth CPU; between two services on the same rack, spending 3 ms of CPU to save 1 ms of transfer is a loss.

The rule of thumb: compress when the link is slow, expensive, or metered (internet egress, mobile clients, cross-region traffic, archival storage) and think twice when it is fast and free (same-availability-zone RPC on small payloads).

Algorithm	Ratio	Compress speed	Decompress speed	Typical use
gzip	Good	Moderate	Fast	Universal HTTP default
brotli	Best text	Slow at high q	Fast	Static assets, precompressed
zstd	Good	Fast, tunable	Very fast	RPC, storage, logs — best default
lz4/snappy	Modest	Very fast	Very fast	Hot paths, in-memory, DB blocks

import gzip, zstandard, json, time

payload = json.dumps([{"id": i, "name": f"user-{i}", "email": f"u{i}@example.com"}
                      for i in range(5000)]).encode()

def bench(name, compress):
    t0 = time.perf_counter()
    out = compress(payload)
    ms = (time.perf_counter() - t0) * 1000
    ratio = len(payload) / len(out)
    # Break-even: does the time saved on the wire exceed the CPU time spent?
    for mbps in (10, 100, 1000):                 # link speed
        saved_ms = (len(payload) - len(out)) * 8 / (mbps * 1_000_000) * 1000
        verdict = "WORTH IT" if saved_ms > ms else "not worth it"
        print(f"{name:<12} ratio={ratio:4.1f}x cpu={ms:6.2f}ms "
              f"link={mbps:>5}Mbps saved={saved_ms:6.2f}ms  {verdict}")

bench("gzip-6", lambda b: gzip.compress(b, 6))
bench("zstd-3", lambda b: zstandard.ZstdCompressor(level=3).compress(b))
bench("zstd-19", lambda b: zstandard.ZstdCompressor(level=19).compress(b))

Several rules save real money and real incidents:

• Compress once for static content. Precompress assets at build time with a high, slow setting; you pay the CPU once and serve the small bytes forever. Use a fast level for dynamic responses compressed per request.
• Do not compress already-compressed data. JPEG, PNG, MP4, and ZIP payloads gain nothing and cost CPU.
• Skip tiny payloads. Below roughly 1 KB, header and framing overhead cancels the gain; most servers have a minimum-size threshold for exactly this reason.
• Choose a compact format before choosing a compressor. Protobuf or Avro (3.2) is often smaller than compressed JSON and cheaper to parse.
• Beware compression plus secrets on the same channel. CRIME/BREACH-style attacks infer secrets from compressed response sizes; do not compress responses that mix attacker-influenced input with secrets (11.3).

For storage, the calculus differs: compressed data means fewer bytes read from disk, so a compressed columnar table is frequently faster to scan than an uncompressed one, even after decompression — the disk, not the CPU, is the bottleneck.

Key Takeaways

• Compression trades CPU for bytes; whether it pays depends on link speed and cost, so decide per hop.
• zstd is the sensible modern default; lz4/snappy for hot paths, brotli for precompressed static assets.
• Precompress static content at a high level; use a fast level for dynamic responses.
• Never compress already-compressed or very small payloads.
• On storage, compression often improves read speed because I/O dominates CPU.

🧪 Practice

1. Benchmark gzip, zstd-3, and zstd-19 on a representative payload and compute the break-even link speed for each.
2. Configure a web server to compress responses above 1 KB, excluding images, and verify with response headers.
3. Interview: When would you deliberately not compress an API response? (Hint: consider payload size, payload type, where the two endpoints sit relative to each other, and one security case.)

Precomputation and Materialization

The cheapest work in a request is work that already happened. Precomputation moves expensive computation off the request path and onto a background path, trading storage and freshness for latency.

The analogy is meal prep. Cooking to order gives maximum freshness and maximum wait; preparing dishes in advance serves instantly, at the cost of storage and food that is a few hours old. Neither is universally right — the question is how stale an answer the product can tolerate.

The main forms, ordered by how far they push the work from the request:

Technique	What is stored	Freshness	Best when
Cached query result	The response	Seconds to minutes	Read-heavy, tolerant of staleness
Materialized view	A query's result table	Refresh interval	Expensive aggregations
Denormalized table	Pre-joined rows	Write-time	Joins dominate read cost
Precomputed feed/index	Per-user results	Write-time	Read amplification is extreme
Incremental aggregation	Running counters/rollups	Near real time	Metrics, counts, leaderboards

-- On-demand: correct, always fresh, and far too slow for a dashboard that
-- 10 000 people open every morning.
SELECT seller_id,
       date_trunc('day', created_at) AS day,
       count(*)                      AS orders,
       sum(total_cents)              AS revenue_cents
FROM orders
WHERE created_at >= now() - interval '90 days'
GROUP BY 1, 2;                        -- scans millions of rows per page view

-- Materialized: the same query, computed once per refresh and read as a table.
CREATE MATERIALIZED VIEW seller_daily_sales AS
SELECT seller_id,
       date_trunc('day', created_at) AS day,
       count(*)                      AS orders,
       sum(total_cents)              AS revenue_cents
FROM orders
GROUP BY 1, 2;

CREATE UNIQUE INDEX ON seller_daily_sales (seller_id, day);   -- required for
-- CONCURRENTLY, which refreshes without blocking readers:
REFRESH MATERIALIZED VIEW CONCURRENTLY seller_daily_sales;

# Incremental aggregation: update the rollup on write instead of rebuilding it.
# Cost moves from O(rows scanned per read) to O(1 per write).
async def record_order(order):
    async with db.transaction():                      # atomic with the write
        await db.execute("INSERT INTO orders (...) VALUES (...)", ...)
        await db.execute("""
            INSERT INTO seller_daily_sales (seller_id, day, orders, revenue_cents)
            VALUES ($1, date_trunc('day', $2), 1, $3)
            ON CONFLICT (seller_id, day) DO UPDATE SET
                orders        = seller_daily_sales.orders + 1,
                revenue_cents = seller_daily_sales.revenue_cents + EXCLUDED.revenue_cents
        """, order.seller_id, order.created_at, order.total_cents)

The hard part of precomputation is never the computing — it is invalidation and correctness. Every precomputed value can drift from its source: a refresh fails silently, an incremental update is applied twice, a backfill overlaps live writes. Three defences are worth building in from the start: make incremental updates idempotent or transactional with the source write, run a periodic reconciliation job that recomputes from truth and reports drift, and record the computation time alongside the value so consumers can see how stale it is.

The related decision is precompute for everyone or only for the hot set. A social feed precomputed for every user wastes enormous effort on dormant accounts; the standard answer is hybrid — precompute for active users, compute on demand for the rest, and switch strategy for outliers such as celebrity accounts whose fan-out is pathological (15.2).

Key Takeaways

• Precomputation trades storage and freshness for latency by moving work off the request path.
• Materialized views suit expensive aggregations; incremental rollups suit near-real-time counters.
• Correctness, not computation, is the hard part — build reconciliation from day one.
• Store a computed-at timestamp so consumers can reason about staleness.
• Precompute for the hot set and fall back to on-demand for the long tail.

🧪 Practice

1. Convert an expensive GROUP BY dashboard query into a materialized view and measure both latency and refresh cost.
2. Implement an incremental counter update that is safe against duplicate delivery.
3. Interview: How do you keep a materialized view from silently going stale? (Hint: think about what proves the refresh ran, and what proves the numbers still match the source of truth.)

Read and Write Path Optimization

Reads and writes want opposite things. Reads want data pre-joined, indexed, denormalized, and close by. Writes want a single place to write, few indexes to update, and no coordination. Every index that makes a read faster makes a write slower; every denormalized copy that removes a join adds a place to update. Optimising a system means deciding which path to favour — and often splitting them so each can be optimised independently.

Start by measuring the ratio. A system doing 100 reads per write should look very different from one doing 100 writes per read.

 READ-OPTIMISED                          WRITE-OPTIMISED

 many indexes                            few indexes
 denormalized / pre-joined               normalized, one place per fact
 materialized aggregates                 aggregate at read time
 read replicas + cache tiers             single writer, append-only
 B-tree (fast point lookups)             LSM-tree (fast sequential writes)
 cost: slower writes, storage,           cost: slower reads, compaction,
       consistency lag                         read amplification

The storage engine choice encodes this trade directly. A B-tree updates data in place — good for reads, and every write pays random I/O plus index maintenance. An LSM-tree appends to a memtable, flushes sorted files, and merges them later — excellent for writes, at the cost of read amplification and background compaction (4.1).

-- Write path: the index nobody audits.
-- Each additional index adds work to EVERY insert, update, and delete.
SELECT indexrelname,
       idx_scan,                          -- how often it was actually used
       pg_size_pretty(pg_relation_size(indexrelid)) AS size
FROM pg_stat_user_indexes
WHERE relname = 'orders'
ORDER BY idx_scan ASC;                    -- zero-scan indexes are pure write tax

-- Read path: a covering index answers the query from the index alone,
-- with no heap fetch at all.
CREATE INDEX idx_orders_seller_recent
    ON orders (seller_id, created_at DESC)
    INCLUDE (status, total_cents)         -- extra columns carried in the index
    WHERE status <> 'cancelled';          -- partial: smaller index, cheaper writes

# CQRS: split the paths outright. Writes go to a normalized store; a projection
# builds read models shaped exactly like the queries that serve them (10.3).
async def place_order(cmd):
    order = await write_store.insert_order(cmd)        # normalized, few indexes
    await events.publish("order.placed", order)        # async fan-out
    return order.id

async def project_order_placed(event):
    # One denormalized row per read pattern; each is cheap to query and
    # independently rebuildable by replaying the event log.
    await read_store.upsert_seller_dashboard_row(event)
    await read_store.upsert_buyer_history_row(event)

async def get_buyer_history(user_id):
    return await read_store.fetch_buyer_history(user_id)   # no joins, one index

Splitting the paths buys independent scaling and independent schemas, and it costs eventual consistency: a user may place an order and not see it in a list that is a few hundred milliseconds behind. That is often fine, and when it is not, the standard mitigations are read-your-own-writes routing (send a user's reads to the primary briefly after their write) or optimistic UI that shows the local result while the projection catches up (8.2).

Key Takeaways

• Reads and writes pull in opposite directions; measure the read/write ratio before optimising either.
• Every index accelerates some reads and taxes every write — audit unused ones.
• Covering and partial indexes give read speed for less write cost than a plain index.
• B-trees favour reads, LSM-trees favour writes; pick the engine to match the workload.
• CQRS separates the paths completely, buying independent optimisation at the price of eventual consistency.

🧪 Practice

1. List the indexes on a busy table with their scan counts and propose which to drop.
2. Rewrite a slow list endpoint's query to be served by a single covering index, and verify with EXPLAIN.
3. Interview: Your write throughput drops 40% after a feature launch that added no writes. What would you check? (Hint: think about what a new read feature usually adds to a table, and who pays for it on every insert.)

Concurrency and Parallelism Tuning

Concurrency is dealing with many things at once; parallelism is doing many things at once. A single-threaded event loop is concurrent but not parallel; a batch job across 16 cores is both. Getting this wrong produces two opposite failures: blocking work on an event loop starves everything else, and unbounded threads thrash the scheduler.

The first question is always what the work is waiting for:

Workload	Bottleneck	Right model	Sizing heuristic
CPU-bound	Cycles	Threads or processes = cores	N = cores (or cores - 1)
I/O-bound	Network/disk wait	Async, or many more threads than cores	N = target_rps * latency (Little)
Mixed	Both	Separate pools per class	Size each independently

For I/O-bound work the pool must be large enough to keep the required number of requests in flight — straight from Little's Law. For CPU-bound work, more threads than cores buys nothing and costs context switches and cache pollution.

import asyncio, os
from concurrent.futures import ProcessPoolExecutor

# Rule 1: never block the event loop. A CPU-heavy call inside an async handler
# stalls EVERY other in-flight request on that loop, not just this one.
cpu_pool = ProcessPoolExecutor(max_workers=os.cpu_count())

async def handle(request):
    rows = await db.fetch(...)                       # I/O: yields the loop
    loop = asyncio.get_running_loop()
    report = await loop.run_in_executor(             # CPU: pushed to processes
        cpu_pool, render_pdf, rows)                  # loop stays responsive
    return report

# Rule 2: bound concurrency toward every dependency, separately.
# One shared limit lets a slow dependency consume all the slots (9.2 bulkheads).
limits = {
    "search":   asyncio.Semaphore(50),
    "payments": asyncio.Semaphore(10),   # slow and critical: keep it isolated
    "images":   asyncio.Semaphore(200),  # fast and non-critical
}

async def call(dependency: str, coro):
    async with limits[dependency]:       # queues here rather than piling up
        return await coro                # downstream and hiding the backpressure

// Thread pools: the bounded queue and the rejection policy are the design.
// An unbounded queue turns overload into an OutOfMemoryError and a growing
// backlog of requests whose callers have already timed out.
ThreadPoolExecutor executor = new ThreadPoolExecutor(
    16,                                        // core threads (I/O-bound sizing)
    64,                                        // max threads under burst
    60L, TimeUnit.SECONDS,                     // idle threads above core exit
    new ArrayBlockingQueue<>(200),             // BOUNDED: backpressure, not memory
    new ThreadPoolExecutor.CallerRunsPolicy()  // on saturation, slow the producer
);                                             // down instead of dropping silently

Two failure modes are worth naming because they recur everywhere. False sharing happens when two threads write to different variables that live in the same cache line, so the hardware ships the line back and forth — the USL b term made concrete. Lock convoying happens when a lock is held across an I/O call, so every waiter inherits that latency; the fix is to shrink the critical section until it contains no I/O at all.

Tuning method: change one knob at a time under a realistic load test, watch throughput and latency together, and stop increasing concurrency at the point where throughput flattens but latency keeps rising. That knee is saturation; beyond it you are only lengthening queues.

Key Takeaways

• Size CPU-bound pools to core count and I/O-bound pools by Little's Law.
• Never run blocking or CPU-heavy work on an event loop; hand it to an executor.
• Bound concurrency per dependency so one slow service cannot consume every slot.
• Use bounded queues with an explicit rejection policy — unbounded queues turn overload into an outage.
• Stop increasing concurrency where throughput flattens and latency keeps climbing; that knee is saturation.

🧪 Practice

1. Measure throughput and p99 for a CPU-bound task at 1, N, and 4N threads on an N-core machine and explain the shape.
2. Add a per-dependency semaphore to a service that currently shares one global limit, and describe what changes during a downstream slowdown.
3. Interview: An async service has excellent throughput in tests but terrible p99 in production. What would you look for? (Hint: think about what one synchronous call inside a coroutine does to every other request sharing that loop.)

<a id="133-benchmarking-and-testing"></a>

13.3 Benchmarking and Testing

Analysis tells you where time goes and optimisation removes some of it, but only testing tells you what the system does under load it has not seen yet. This subchapter covers load testing to verify expected traffic, stress and soak testing to find breaking points and slow leaks, spike testing for sudden arrivals, shadow traffic for risk-free realism, the difference between synthetic probes and real user measurement, and how to turn all of it into a defensible capacity statement.

Load Testing

A load test asks a specific, falsifiable question: does the system meet its latency and error-rate targets at an expected level of traffic? It is not "see how fast it goes" — that is benchmarking, and it produces numbers nobody can act on.

The single most common way load tests mislead is unrealistic workload. Real traffic has a mix of endpoints in particular proportions, a key distribution with hot spots, a cache that is already warm, think time between user actions, and payload sizes drawn from real data. A test that hammers one endpoint with one key measures your cache, not your system.

 TEST SHAPE                         WHAT IT ANSWERS

 load     ____----''''''''''''      Does the system meet SLOs at expected peak?
          ramp    steady state      (ramp gently: instant load tests cold caches,
                                     not steady-state behaviour)

 stress   ____----''''----''''--/   Where does it break, and how?
          increase until failure

 soak     ------------------------  Does it degrade over hours? (leaks, fragmentation)
          steady, 8-24 hours

 spike    ___|''|______|''|_______  Can it absorb sudden arrivals and recover?
          instant 10x, then drop

// k6: a load test shaped like real traffic, with pass/fail thresholds.
import http from 'k6/http';
import { check, sleep, group } from 'k6';
import { Trend } from 'k6/metrics';

const checkoutLatency = new Trend('checkout_latency');

export const options = {
  stages: [
    { duration: '5m',  target: 500 },   // ramp: let caches and JIT warm up
    { duration: '20m', target: 500 },   // steady state: this is the measurement
    { duration: '3m',  target: 0 },     // ramp down
  ],
  thresholds: {
    // A test without thresholds is a demo. These make it a pass/fail gate.
    'http_req_duration{endpoint:browse}':   ['p(95)<200', 'p(99)<500'],
    'http_req_duration{endpoint:checkout}': ['p(95)<800'],
    'http_req_failed':                      ['rate<0.001'],   // < 0.1% errors
  },
};

export default function () {
  // Endpoint MIX mirrors production: 85% browse, 12% search, 3% checkout.
  const roll = Math.random();
  group('user_session', () => {
    if (roll < 0.85) {
      // Zipfian key choice: a few products are hot, most are cold — like reality.
      const id = zipfProductId();
      http.get(`https://staging.shop/api/products/${id}`, { tags: { endpoint: 'browse' } });
    } else if (roll < 0.97) {
      http.get(`https://staging.shop/api/search?q=${randomTerm()}`, { tags: { endpoint: 'search' } });
    } else {
      const res = http.post('https://staging.shop/api/checkout', payload(),
                            { tags: { endpoint: 'checkout' } });
      check(res, { 'checkout ok': (r) => r.status === 201 });
      checkoutLatency.add(res.timings.duration);
    }
  });
  sleep(Math.random() * 3 + 1);   // THINK TIME: users pause; load generators must too
}

Two methodological points separate credible tests from noise. First, open versus closed models: a closed model keeps a fixed number of virtual users who wait for a response before sending the next request, so when the system slows down, the offered load automatically drops — hiding overload. An open model sends at a fixed arrival rate regardless, which is how the internet actually behaves. Use open-model testing (k6's constant-arrival-rate executor, or equivalent) when you care about overload behaviour.

Second, coordinated omission: if your load generator waits for a slow response before scheduling the next request, the requests it never sent — the ones that would have queued during the stall — are missing from the results, and the measured p99 is far better than reality. Generators that record intended send time rather than actual send time correct for this.

A test is only meaningful against a comparable environment. Production-shaped data volumes matter more than production-shaped hardware: a query plan on 10 000 rows tells you nothing about the same query on 100 million.

Key Takeaways

• Load testing verifies SLOs at expected traffic; state thresholds up front so the test passes or fails rather than merely reporting.
• Model the real workload — endpoint mix, key skew, think time, payload sizes, warm caches.
• Prefer open-model (fixed arrival rate) generators; closed models mask overload by slowing themselves down.
• Watch for coordinated omission, which flatters tail latency.
• Data volume realism matters more than hardware parity.

🧪 Practice

1. Write a load test with an endpoint mix and think time, and add p95/p99 thresholds that fail the build.
2. Run the same test with a warm and a cold cache and explain the difference in p99.
3. Interview: Why can a closed-model load test report healthy latency for a system that is actually overloaded? (Hint: think about who decides when the next request is sent, and what happens to offered load when responses slow.)

Stress and Soak Testing

Load testing confirms the system works at expected traffic. Stress and soak testing answer the two questions load testing cannot: what happens beyond the expected, and what happens over time.

Stress testing increases load until something breaks. The valuable output is not the breaking number but the failure mode. A system that sheds excess load, keeps serving the requests it accepted, and returns to normal when load drops has a good failure mode. A system that accepts everything, queues without bound, times out on all requests including the ones it could have served, and needs a restart to recover has a bad one — and the difference is a design decision, not luck.

 GRACEFUL DEGRADATION                    CLIFF (BAD)

 good│----____                           good│--------
 put │        ''----____                 put │        |
     │                  ''-----              │        |
     │  sheds excess, serves the rest        │        +--------- 0 (collapse)
     └────────────────────────► load         └────────────────► load
        knee     overload                       knee  ^ everything times out,
                                                        recovery needs a restart

Soak testing runs a moderate, realistic load for hours or days to expose problems that need time to accumulate: memory leaks, unclosed file descriptors or connections, log volumes filling a disk, cache growth without eviction, thread leaks, database bloat from un-vacuumed dead tuples, and gradual index fragmentation. None of these show up in a 20-minute run.

// k6: stress to the breaking point using an OPEN model, so offered load keeps
// climbing even as the system slows down.
export const options = {
  scenarios: {
    stress: {
      executor: 'ramping-arrival-rate',   // arrival rate, not virtual users
      startRate: 100, timeUnit: '1s',
      preAllocatedVUs: 200, maxVUs: 5000, // enough VUs that the generator never
      stages: [                           // becomes the bottleneck itself
        { target: 500,  duration: '5m' },
        { target: 1000, duration: '5m' },
        { target: 2000, duration: '5m' },
        { target: 4000, duration: '5m' }, // find the knee, then push past it
      ],
    },
  },
};

# Soak test analysis: the trend matters, not the absolute value. Fit a line to
# memory over the run; a positive slope that does not flatten is a leak.
import statistics

def leak_slope(samples_mb: list[float], minutes_per_sample: float) -> float:
    n = len(samples_mb)
    xs = [i * minutes_per_sample for i in range(n)]
    mx, my = statistics.mean(xs), statistics.mean(samples_mb)
    num = sum((x - mx) * (y - my) for x, y in zip(xs, samples_mb))
    den = sum((x - mx) ** 2 for x in xs)
    return num / den                                    # MB per minute

slope = leak_slope([512, 528, 547, 561, 580, 599, 618, 640], minutes_per_sample=60)
print(f"{slope * 60:.1f} MB/hour")           # steady climb after warm-up = leak
print(f"OOM at 4 GB in ~{(4096 - 640) / (slope * 60):.0f} hours")

During a stress test, record what breaks first, because that component is your real capacity limit and everything else is theoretical. Common first failures: connection pool exhaustion, a downstream dependency without a circuit breaker, the log pipeline, disk I/O on the database, and — frequently — the load generator itself. Then verify recovery: drop the load and confirm the system returns to normal without human intervention. A system that needs a restart after every overload will need one during every incident.

Key Takeaways

• Stress testing finds the breaking point; the failure mode matters more than the number.
• Graceful degradation means shedding excess load while continuing to serve accepted requests, then recovering unaided.
• Soak testing exposes leaks, unbounded caches, descriptor exhaustion, and disk growth that short runs cannot.
• Record which component fails first — that is your actual capacity limit.
• Always test recovery, not just failure.

🧪 Practice

1. Stress a service until errors appear and document the first component to fail and how the system behaved past that point.
2. Run an 8-hour soak and plot memory, descriptors, and connection counts, fitting a trend line to each.
3. Interview: How would you tell a memory leak from normal cache growth in a soak test? (Hint: think about what each curve looks like over time and what a bounded cache is supposed to do at its limit.)

Spike Testing

Spike testing covers the case where load does not ramp at all: a marketing email goes out, a video goes viral, a competitor has an outage, or — the most common in practice — a dependency recovers and every client retries at once. Traffic goes from baseline to ten times baseline in seconds.

Sudden load is qualitatively harder than high load, because the mechanisms meant to help all have lag. Autoscaling needs to observe metrics, decide, provision, and boot — typically one to five minutes. Caches are cold for content nobody has requested recently. Connection pools must open new connections precisely when the database is busiest. JIT-compiled runtimes are slow until warm.

 SPIKE ARRIVES                     capacity vs demand over time

 load │      ,------------------------
      │      |                            <- demand: instant
      │      |
      │  ____|                            <- capacity: lags by the scaling delay
      │      :   ____----''''''''
      └──────:───────────────────────────► time
             t0    t0+90s   t0+3min
             |<-- THE GAP -->|
        Everything that protects you must work inside this window:
        queueing, load shedding, cached responses, pre-warmed headroom.

# The retry storm: the most common self-inflicted spike. When a dependency
# recovers, every client that was retrying hits it simultaneously unless the
# backoff is randomised.
import random

def backoff_delay(attempt: int, base: float = 0.1, cap: float = 30.0,
                  jitter: str = "full") -> float:
    exponential = min(cap, base * (2 ** attempt))
    if jitter == "none":
        return exponential          # every client retries at the SAME instant
    if jitter == "full":
        return random.uniform(0, exponential)   # spreads clients across the window
    return exponential / 2 + random.uniform(0, exponential / 2)   # equal jitter

# 10 000 clients, attempt 5: without jitter all 10 000 arrive at t = 3.2 s.
# With full jitter they arrive spread over 0-3.2 s: a ~3 000x lower peak rate.

# Kubernetes HPA tuned for spikes: scale out fast, scale in slowly.
# Symmetric behaviour is wrong — scaling in quickly after a spike guarantees
# you are undersized when the second wave arrives.
behavior:
  scaleUp:
    stabilizationWindowSeconds: 0        # react immediately
    policies:
      - type: Percent
        value: 100                       # allow doubling
        periodSeconds: 30
      - type: Pods
        value: 10                        # or +10 pods, whichever is larger
        periodSeconds: 30
    selectPolicy: Max
  scaleDown:
    stabilizationWindowSeconds: 600      # wait 10 minutes of calm before shrinking
    policies:
      - type: Percent
        value: 10                        # then shrink gently
        periodSeconds: 60

The defences that actually work in the scaling gap are the ones that need no scaling: a queue that absorbs the burst and lets workers drain it at their own rate (7.1), load shedding that rejects excess quickly rather than queueing it (6.3), cached or static responses served from the edge (5.4), pre-warmed headroom before predictable spikes such as a scheduled launch, and jittered client backoff so your own clients do not manufacture the spike.

Key Takeaways

• Spike testing checks behaviour when load arrives instantly, not gradually.
• Autoscaling lags by minutes; something else must cover the gap.
• Queues, load shedding, edge caching, and pre-warming are the in-gap defences.
• Retry storms are self-inflicted spikes; full jitter on exponential backoff is the fix.
• Scale out fast and scale in slowly — symmetric autoscaling policies leave you undersized for the second wave.

🧪 Practice

1. Run a test that jumps from 100 to 2 000 rps instantly and record the error rate and recovery time.
2. Simulate 5 000 clients retrying with and without full jitter, and plot the arrival rate at the dependency.
3. Interview: A dependency comes back after a 2-minute outage and immediately falls over again. Why? (Hint: think about what every client was doing during those two minutes and when their timers all expire.)

Shadow Traffic and Dark Launches

Synthetic load tests are approximations. Real traffic has a request mix, key skew, payload weirdness, and adversarial inputs that no generator reproduces faithfully. Shadow traffic gives you real traffic against a new system without risking a single user request.

Shadow traffic (mirroring): production requests are duplicated to a new service or version, whose responses are discarded. Users are served entirely by the existing system and never see the shadow's output — so the shadow can be wrong, slow, or crash without consequence.

Dark launch: new code is deployed and executed in production behind a flag, its results compared against the current implementation but not shown to users, until confidence is high enough to flip the flag (12.3).

                  +---------------------> [ PRIMARY ]-----> response to user
 request ----->  fork                          |
                  +- - - - - - - - - -> [ SHADOW ]          response DISCARDED
                     (async, best effort)      |
                                               v
                                        compare: latency, status, diff of results

import asyncio, hashlib, json, logging

async def handle_with_shadow(request):
    primary = asyncio.create_task(primary_service.handle(request))

    if should_shadow(request):
        # Fire and forget. The shadow must NEVER add latency to or fail the
        # user's request, so it is never awaited on the response path.
        asyncio.create_task(run_shadow(request, primary))

    return await primary

async def run_shadow(request, primary_task):
    try:
        shadow_req = scrub(request)      # strip or fake anything with side effects:
                                         # payment tokens, real email addresses
        shadow = await asyncio.wait_for(shadow_service.handle(shadow_req), timeout=5)
        primary = await primary_task
        if digest(primary) != digest(shadow):
            logging.warning("shadow_mismatch", extra={
                "path": request.path,
                "primary": digest(primary), "shadow": digest(shadow),
                "diff_sample": sample_diff(primary, shadow),   # bounded payload
            })
    except Exception:
        logging.warning("shadow_failed", exc_info=True)   # swallowed on purpose

def should_shadow(request) -> bool:
    # Deterministic sampling: the same user always shadows, so comparisons are
    # consistent across a session rather than a random half of each flow.
    if request.method != "GET":
        return False                     # start with reads: no side effects
    return int(hashlib.sha256(request.user_id.encode()).hexdigest(), 16) % 100 < 5

def digest(response) -> str:
    payload = normalise(response)        # drop timestamps, request ids, ordering
    return hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()

The hazard that defines shadow testing is side effects. A mirrored request that charges a card, sends an email, or writes to the production database has just done a real thing twice. Before mirroring anything, the shadow must run against a separate datastore, have external calls stubbed, and be scrubbed of anything that triggers a real-world action. The safe progression is: read-only traffic first, then writes against an isolated store, then a small percentage of real traffic behind a flag (canary), then full rollout.

Shadowing costs real money — you are running the workload twice — so sample rather than mirroring everything, and turn it off once the comparison is clean.

Key Takeaways

• Shadow traffic tests a new system with real production requests while users are served entirely by the old one.
• The shadow path must be asynchronous and failure-isolated; it can never add latency or errors to the user's request.
• Side effects are the danger: scrub, stub, and use a separate datastore before mirroring anything.
• Compare normalised digests, not raw responses, or every timestamp is a mismatch.
• Sample deterministically by user, and remember you are paying for the workload twice.

🧪 Practice

1. Add shadowing for 5% of GET traffic to a rewritten endpoint and log normalised response mismatches.
2. List every side effect in a write endpoint and describe how each would be neutralised in a shadow.
3. Interview: How do you shadow a payment service safely? (Hint: think about which calls leave your system, and what a shadow needs in place of each of them.)

Synthetic vs. Real User Monitoring

Both approaches measure performance in production, from opposite directions, and mature systems run both because each sees what the other cannot.

Synthetic monitoring runs scripted transactions from controlled locations on a schedule: hit the login page from four regions every minute, complete a checkout every five. It is a controlled experiment — same script, same place, same cadence — so any change in the number means a change in the system.

Real User Monitoring (RUM) instruments actual sessions and reports what real people experienced on their own devices and networks. It is ground truth about user experience, but it is uncontrolled: a bad number might mean your service regressed, or that a large cohort of users on slow mobile connections arrived.

Dimension	Synthetic	RUM
Traffic source	Scripted probes	Real users
Coverage	Only the flows you scripted	Everything users actually do
Baseline stability	High — controlled variables	Low — device, network, geography mix
Works with no users	Yes (nights, pre-launch, new region)	No
Catches	Availability, regressions, third parties	Real device and network reality
Misses	Unscripted flows, real conditions	Nothing users hit; everything they do not
Best used for	Alerting and uptime SLOs	SLOs about experience, prioritisation

// RUM: report real Core Web Vitals plus your own API timings from the browser.
import { onLCP, onINP, onCLS } from 'web-vitals';

function report(metric) {
  const body = JSON.stringify({
    name: metric.name,
    value: metric.value,
    rating: metric.rating,             // 'good' | 'needs-improvement' | 'poor'
    // Dimensions that let you SEGMENT — an aggregate p75 hides the cohort that
    // is actually suffering.
    connection: navigator.connection?.effectiveType,   // '4g', '3g', 'slow-2g'
    deviceMemory: navigator.deviceMemory,
    country: window.__geo,
    release: window.__release,
  });
  // sendBeacon survives page unload; fetch() during navigation is often dropped.
  navigator.sendBeacon('/rum', body);
}

onLCP(report);   // Largest Contentful Paint: when the main content appeared
onINP(report);   // Interaction to Next Paint: responsiveness to input
onCLS(report);   // Cumulative Layout Shift: visual stability

# Synthetic: a scheduled probe of a critical journey, run from several regions.
# Alert on the probe, investigate with RUM.
def synthetic_checkout_probe(region: str) -> dict:
    t0 = time.perf_counter()
    session = login(TEST_ACCOUNT)                    # a real, isolated test account
    add_to_cart(session, TEST_SKU)
    order = checkout(session, TEST_CARD_SANDBOX)     # sandbox payment credentials
    elapsed_ms = (time.perf_counter() - t0) * 1000
    emit_metric("synthetic.checkout.duration_ms", elapsed_ms, tags={"region": region})
    emit_metric("synthetic.checkout.success", 1 if order.ok else 0, tags={"region": region})
    return {"ok": order.ok, "ms": elapsed_ms}

Use them together in one workflow: alert on synthetic probes because their baseline is stable and a failure is unambiguous, then use RUM to judge impact and priority — a regression affecting 2% of sessions on one browser is a different matter from one affecting every mobile user. And always segment RUM data before concluding anything: a stable overall p75 can conceal a severe regression in one country, one device class, or one release.

Key Takeaways

• Synthetic monitoring is a controlled experiment: stable baseline, good for alerting, blind to unscripted reality.
• RUM is ground truth about experience but uncontrolled and unavailable without traffic.
• Run both: alert on synthetic, prioritise and quantify with RUM.
• Always segment RUM by device, network, geography, and release — aggregates hide cohorts.
• Synthetic probes need isolated test accounts and sandbox credentials so they never pollute business data.

🧪 Practice

1. Instrument a page with web-vitals and report LCP and INP with segmentation dimensions.
2. Write a synthetic probe for a critical journey and define the alert threshold for it.
3. Interview: Synthetic checks are green but users report slowness. What is going on? (Hint: think about where probes run, what devices they use, and which flows nobody scripted.)

Capacity Verification

Capacity planning produces a claim: "we can handle Black Friday". Capacity verification is the act of proving it, ideally before Black Friday. The difference matters because untested capacity plans are wrong in one direction almost every time — the bottleneck is never the resource the spreadsheet was about.

A defensible capacity statement has four parts: the load level, the workload shape, the SLOs that held, and the date it was verified. "We sustained 12 000 rps of production-shaped traffic for 30 minutes with p99 under 400 ms and errors under 0.05%, on 2026-08-14" is a capacity statement. "We can do about 12k" is a rumour.

 THE VERIFICATION LOOP

   forecast demand ---> compute required capacity ---> TEST IT
        ^                                                |
        |                                                v
   update model <--- compare predicted vs measured <--- record the
                                                        first bottleneck

from dataclasses import dataclass

@dataclass
class CapacityModel:
    peak_rps: float
    rps_per_instance: float          # from a measured single-instance test
    target_utilisation: float = 0.70 # queueing headroom (13.1)
    failure_domains: int = 3         # e.g. availability zones

    def instances_needed(self) -> int:
        import math
        base = self.peak_rps / (self.rps_per_instance * self.target_utilisation)
        # N+1 within the largest failure domain: survive losing one whole zone
        # while still meeting SLOs, not merely staying up.
        per_domain = math.ceil(base / (self.failure_domains - 1))
        return per_domain * self.failure_domains

model = CapacityModel(peak_rps=12_000, rps_per_instance=250)
print(model.instances_needed())      # 69 instances, zone-failure tolerant

# Verification: does reality match the model? Anything beyond ~15% error means
# the model is missing a bottleneck (a shared dependency, a lock, a quota).
def verify(predicted: int, measured_max_rps: float, model: CapacityModel) -> None:
    predicted_rps = predicted * model.rps_per_instance * model.target_utilisation
    error = (measured_max_rps - predicted_rps) / predicted_rps
    print(f"predicted {predicted_rps:,.0f} rps, measured {measured_max_rps:,.0f} rps "
          f"({error:+.1%}) -> {'model holds' if abs(error) < 0.15 else 'INVESTIGATE'}")

verify(69, measured_max_rps=9_800, model=model)   # -19%: something else binds first

Verification only counts if the whole path is exercised. Scaling the application tier to 69 instances proves nothing if the database connection limit, a third-party API quota, a NAT gateway's port range, or a load balancer's pre-warming limit binds first — and one of those usually does. Testing the full path at target load is what turns a plan into a fact.

Two production-grade practices extend this. A game day (9.4) runs the load test while deliberately failing a component — kill a zone at peak and confirm the remaining capacity holds. Load testing in production, done carefully with a small percentage of synthetic traffic tagged so it can be excluded from business metrics, is the only way to test the exact configuration users hit; it requires a kill switch and close monitoring, and it is how large systems verify capacity they cannot afford to replicate in staging.

Re-verify on a schedule and after any significant change. Capacity decays silently: a new feature adds a query per request, a dependency slows down, data volume grows past an index's efficiency. A capacity number older than a quarter is a historical fact, not a current one.

Key Takeaways

• A capacity statement needs a load level, a workload shape, the SLOs that held, and a verification date.
• Size for target utilisation around 70% and for surviving the loss of a whole failure domain.
• Compare predicted against measured capacity; a large gap means the model is missing a bottleneck.
• Test the entire path — quotas, connection limits, and gateways bind before the app tier does.
• Capacity decays; re-verify on a schedule and after major changes.

🧪 Practice

1. Build a capacity model for a service and compute the instance count needed to survive losing one of three zones at peak.
2. Run a test at the modelled peak and record which component saturates first.
3. Interview: How would you verify capacity for an event ten times your normal peak? (Hint: think about what you cannot replicate in staging, what quotas live outside your account, and how you would test the real path safely.)

<a id="134-cost-optimization"></a>

13.4 Cost Optimization

Cost is a non-functional requirement like latency or availability, and it is the one that most often goes unowned until a bill forces the conversation. This subchapter covers how to attribute cost to individual requests and features, the pricing models that make compute cheap or expensive, why storage bills are usually about access rather than capacity, how to size resources against real usage, how to reason about the three-way trade between cost, latency, and reliability, and the organisational practices that keep the whole thing under control.

Cost Modeling per Request

An infrastructure bill arrives as one large number, which is exactly the wrong unit for making decisions. The useful unit is cost per request, per user, or per business transaction, because that is what tells you whether a feature is worth running, which endpoint deserves optimisation, and whether growth improves or destroys your margins.

The mental model: divide total spend by the meaningful denominator, and then attribute it — cost per checkout matters more than cost per HTTP request, since one is tied to revenue and the other is not.

 COST OF ONE "SEARCH" REQUEST

 +-------------------------+-----------+--------------------------------------+
 | Component               | Cost (uc) | Basis                                |
 +-------------------------+-----------+--------------------------------------+
 | Compute (app tier)      |    12.0   | 40 ms CPU on a $0.04/vCPU-hr instance |
 | Search cluster          |    31.0   | Dominant: memory-resident index      |
 | Cache (Redis)           |     2.0   | Amortised cluster cost / requests    |
 | Database reads          |     4.0   | 3 reads at metered read units        |
 | Inter-AZ network        |     3.0   | 60 KB crossing a zone boundary       |
 | Egress to internet      |    18.0   | 200 KB at $0.09/GB                   |
 | Logging + tracing       |     7.0   | 4 KB of logs, 10% trace sampling     |
 +-------------------------+-----------+--------------------------------------+
 | TOTAL                   |    77.0   | = $0.00077 per search                |
 +-------------------------+-----------+--------------------------------------+
   uc = microcents. At 50M searches/day: $385/day, ~$11.5k/month.

   Note where the money is: the search cluster and EGRESS. Optimising the app
   tier's 12 uc caps out at 15% of the bill; halving payload size saves more.

from dataclasses import dataclass

@dataclass
class RequestCost:
    cpu_ms: float
    memory_gb_s: float
    db_reads: int
    db_writes: int
    egress_kb: float
    log_kb: float

    # Illustrative unit prices; substitute your provider's actual rates.
    CPU_PER_MS       = 0.04 / 3_600_000       # $/vCPU-hour -> $/vCPU-ms
    MEM_PER_GB_S     = 0.004 / 3600
    DB_READ          = 0.00000025             # $ per read unit
    DB_WRITE         = 0.00000125             # writes cost ~5x reads
    EGRESS_PER_KB    = 0.09 / 1_048_576       # $0.09/GB
    LOG_PER_KB       = 0.50 / 1_048_576       # ingest-priced logging is expensive

    def total(self) -> float:
        return (self.cpu_ms * self.CPU_PER_MS
                + self.memory_gb_s * self.MEM_PER_GB_S
                + self.db_reads * self.DB_READ
                + self.db_writes * self.DB_WRITE
                + self.egress_kb * self.EGRESS_PER_KB
                + self.log_kb * self.LOG_PER_KB)

    def breakdown(self) -> dict[str, float]:
        t = self.total()
        return {
            "compute": (self.cpu_ms * self.CPU_PER_MS) / t,
            "database": (self.db_reads * self.DB_READ + self.db_writes * self.DB_WRITE) / t,
            "egress":  (self.egress_kb * self.EGRESS_PER_KB) / t,
            "logging": (self.log_kb * self.LOG_PER_KB) / t,
        }

search = RequestCost(cpu_ms=40, memory_gb_s=0.02, db_reads=3, db_writes=0,
                     egress_kb=200, log_kb=4)
print(f"${search.total():.8f} per search")
for k, share in sorted(search.breakdown().items(), key=lambda kv: -kv[1]):
    print(f"  {k:<9} {share:5.1%}")          # optimise the top line, not the fun one

# Unit economics: the number that decides whether growth is good news.
REVENUE_PER_SEARCH = 0.0021                  # ad revenue per search
margin = (REVENUE_PER_SEARCH - search.total()) / REVENUE_PER_SEARCH
print(f"gross margin per search: {margin:.1%}")

Building the model requires attribution, which in practice means tagging every resource with a service, team, and environment, and enabling per-request telemetry for the metered components. Anything untagged lands in a "shared" bucket that grows until nobody can explain it; a good target is over 95% of spend attributable.

Two effects surprise people the first time they build this model. Observability often costs more than compute for chatty services — ingest-priced logs at scale are brutal, which is why sampling (12.1) is a cost decision as much as a signal-quality one. And egress frequently exceeds everything else for media-heavy or API-heavy products, which makes payload size and CDN offload the highest-leverage optimisations available.

Key Takeaways

• Model cost per request, per user, and per business transaction — the total bill is not an actionable unit.
• Include compute, memory, storage I/O, network egress, and observability; the last two are routinely the largest.
• Tag everything so spend is attributable; aim for 95%+ attribution.
• Compare unit cost against unit revenue to know whether growth helps margins.
• Optimise the largest line item, not the most interesting one.

🧪 Practice

1. Build a per-request cost model for one endpoint, including logging and egress, and rank the components.
2. Compute the monthly cost of a feature at current traffic and at 10x.
3. Interview: Your bill grew 40% while traffic grew 10%. Where do you look? (Hint: think about which costs scale with something other than request count — retained data, log volume, cross-zone chatter, idle reserved capacity.)

Compute Pricing Models

Cloud compute is sold several ways, and the same workload can differ in price by 5x depending on which model it uses. The choice is driven by one question: how predictable and how interruptible is this workload?

Model	Discount vs on-demand	Commitment	Interruptible	Fits
On-demand	Baseline	None	No	Spiky, unpredictable, short-lived
Reserved / savings plan	30-60%	1 or 3 years	No	Steady baseline capacity
Spot / preemptible	60-90%	None	Yes, minutes' notice	Batch, CI, stateless workers
Serverless (per-invocation)	Varies	None	No	Bursty, low-duty-cycle, event-driven

The standard architecture is layered: cover the steady baseline with commitments, handle predictable daily variation with on-demand autoscaling, and run everything interruption-tolerant on spot.

 CAPACITY LAYERS OVER A DAY

 instances
   120 |            .-''-.                 <- on-demand: the predictable daytime peak
       |         .-'      '-.
    80 |      .-'            '-.
       +---------------------------------- <- reserved / savings plan: 24x7 baseline
    40 |=================================
       |#########  spot: batch + async workers, shifted to off-peak  ########
     0 +---------------------------------------------------------------> time
       00:00        09:00        18:00        24:00

 Commit to the flat part only. Committing to peak means paying for peak at 03:00.

def annual_cost(hourly_on_demand: float, instances: int, hours: int = 8760) -> float:
    return hourly_on_demand * instances * hours

def layered_plan(baseline: int, peak: int, batch: int,
                 hourly: float, peak_hours_per_day: int = 9) -> dict:
    reserved   = annual_cost(hourly * 0.60, baseline)                     # ~40% off
    on_demand  = annual_cost(hourly, peak - baseline, peak_hours_per_day * 365)
    spot       = annual_cost(hourly * 0.25, batch)                        # ~75% off
    naive_peak = annual_cost(hourly, peak + batch)      # everything on-demand at peak
    total = reserved + on_demand + spot
    return {"reserved": reserved, "on_demand": on_demand, "spot": spot,
            "total": total, "naive": naive_peak, "saving": 1 - total / naive_peak}

plan = layered_plan(baseline=40, peak=120, batch=30, hourly=0.20)
for k, v in plan.items():
    print(f"{k:<10} {v:,.0f}" if k != "saving" else f"{k:<10} {v:.1%}")

# Commit only to what you are confident you will still be running. An unused
# 3-year reservation is worse than on-demand: you pay for capacity twice.
def safe_commitment(hourly_usage: list[int], confidence_percentile: int = 10) -> int:
    """Commit near the LOW percentile of observed usage, not the average."""
    s = sorted(hourly_usage)
    return s[int(len(s) * confidence_percentile / 100)]

Spot instances deserve a design note: they are reclaimed with roughly two minutes' warning, so using them safely means handling the interruption signal (drain, checkpoint, requeue), diversifying across instance types and zones so one capacity shortage does not take every worker, and keeping stateful or latency-critical components off them. With those in place, spot is the single largest cost lever in most architectures.

Serverless inverts the calculation: you pay per invocation and per GB-second with no idle cost, which is dramatically cheaper for spiky, low-duty-cycle workloads and dramatically more expensive for steady high-throughput ones. There is a crossover point — compute it rather than arguing about it (10.4).

Key Takeaways

• Layer pricing models: commitments for the 24x7 baseline, on-demand for the predictable peak, spot for interruptible work.
• Commit to a low percentile of observed usage, never to peak or even average.
• Spot saves 60-90% but requires interruption handling and instance-type diversification.
• Serverless wins on bursty low-duty-cycle workloads and loses on steady high-throughput ones; find the crossover.
• Graviton-class or newer instance families often beat the current generation on price-performance for the same money.

🧪 Practice

1. Take a service's hourly instance count for a month and compute the safe commitment level at the 10th percentile.
2. Estimate the annual saving from moving a batch pipeline to spot, including the cost of restarted work.
3. Interview: When is serverless more expensive than a reserved instance? (Hint: think about duty cycle — what fraction of the hour the code is actually executing — and where the crossover lies.)

Storage and Egress Costs

Storage bills confuse people because the headline price per gigabyte is almost never the whole cost. The real bill has four components: capacity, requests, retrieval, and transfer out. For a hot workload on a cheap archival tier, requests and retrieval can dwarf capacity by an order of magnitude.

Tier	Capacity $/GB-mo	Request cost	Retrieval	Min duration	Fits
Hot object store	~0.023	Low	None	None	Active data
Infrequent access	~0.0125	Higher	Per GB	30 days	Monthly-access data
Archive	~0.004	High	Per GB + hours to restore	90 days	Compliance retention
Block (SSD)	~0.08-0.125	IOPS-priced	None	None	Databases, low latency

The classic mistake is a lifecycle policy that moves objects to archive after 30 days when they are still read weekly. Capacity drops 80% and the bill goes up, because every read now pays retrieval plus per-request fees, and objects deleted before the minimum duration are billed for it anyway.

def monthly_storage_cost(gb: float, reads_per_month: int, tier: str) -> float:
    tiers = {
        # (capacity $/GB-mo, per-1000-GET, retrieval $/GB, min days)
        "standard": (0.023,  0.0004, 0.000, 0),
        "infrequent": (0.0125, 0.0010, 0.010, 30),
        "archive":  (0.004,  0.0025, 0.020, 90),
    }
    cap, per_1k_get, retrieval, _ = tiers[tier]
    avg_object_gb = 0.002                     # 2 MB objects
    return (gb * cap
            + reads_per_month / 1000 * per_1k_get
            + reads_per_month * avg_object_gb * retrieval)   # the line people forget

for tier in ("standard", "infrequent", "archive"):
    cold = monthly_storage_cost(10_000, reads_per_month=1_000,     tier=tier)
    warm = monthly_storage_cost(10_000, reads_per_month=5_000_000, tier=tier)
    print(f"{tier:<11} rarely-read=${cold:9,.2f}   frequently-read=${warm:12,.2f}")
# archive wins by ~5x when data is cold and loses by ~100x when it is not.

Egress — data leaving the cloud to the internet — is priced far above ingress, which is usually free, and it is the line item that most often surprises a growing product. Worse, internal transfer is metered too: cross-availability- zone traffic is billed in both directions, so a chatty microservice mesh spread across zones pays a permanent tax on every hop.

 NETWORK PRICING SHAPE (typical order of magnitude)

 ingress from internet ............ free
 same-AZ, private IPs ............. free
 cross-AZ (each direction) ........ ~$0.01/GB   <- the silent microservice tax
 cross-region .................... ~$0.02/GB
 internet egress ................. ~$0.09/GB   <- 9x cross-AZ, and it compounds
 via CDN egress .................. ~$0.02-0.085/GB, and CACHE HITS never touch
                                   your origin at all

The highest-leverage egress reductions, in order: serve static and cacheable content from a CDN so origin egress collapses to cache misses (5.4); shrink payloads (compression, a compact format, field filtering so clients fetch only what they render); keep chatty service-to-service traffic zone-local by preferring same-zone routing; and avoid cross-region replication of data nobody reads in the other region.

For storage capacity itself the levers are unglamorous and effective: lifecycle policies matched to actual access patterns rather than guesses, deduplication, compression at rest, deleting orphaned snapshots and unattached volumes (which quietly accumulate forever), and setting retention on logs and backups instead of keeping everything indefinitely (4.6).

Key Takeaways

• Storage cost is capacity plus requests plus retrieval plus transfer; capacity is often the smallest part.
• Match lifecycle tiers to real access patterns — archiving warm data increases the bill and adds minimum-duration charges.
• Internet egress is the expensive direction; ingress is typically free.
• Cross-AZ traffic is billed both ways and is a permanent tax on chatty cross-zone services.
• CDN offload plus smaller payloads is usually the biggest single egress win.

🧪 Practice

1. Compute the monthly cost of 5 TB read 100 000 times per month in each storage tier and identify the crossover point.
2. Estimate the cross-AZ transfer cost of a service that makes 10 downstream calls per request at 20 KB each, with a third of calls crossing a zone.
3. Interview: Your egress bill doubled but user count did not change. What would you investigate? (Hint: think about payload size, cache hit ratio, and whether something started bypassing the CDN.)

Right-Sizing Resources

Most cloud waste is not exotic; it is instances provisioned for a peak that never comes, sized by whoever created them from a number that felt safe. Right-sizing is the continuous practice of matching provisioned resources to observed usage plus a deliberate margin.

The reason it needs to be continuous is that the fit decays: traffic patterns change, code gets more efficient or less, and the instance families available get cheaper. A size that was correct a year ago is a guess today.

 THE WASTE PATTERN

 vCPU
   16 |=================================== provisioned (paid for, 24x7)
      |
      |                    ,--.
    4 |         ,-''-.  ,-'    '-.         actual peak usage
      |     _.-'      ''          '-._
    1 |__.-'                          '--.
      +----------------------------------------------> time
       waste = 12 vCPU * 24h * 365d = 75% of the bill for nothing

 Right-sized: 4 vCPU + autoscaling, or 6 vCPU for burst headroom.

def right_size(observed_p95: float, observed_max: float, provisioned: float,
               headroom: float = 1.30) -> dict:
    """Size against p95 with headroom, then sanity-check against the observed max.

    Sizing on the mean causes throttling; sizing on the max buys the worst
    minute of the year forever. p95 plus headroom is the usual compromise.
    """
    recommended = max(observed_p95 * headroom, observed_max * 1.05)
    return {
        "provisioned": provisioned,
        "recommended": round(recommended, 1),
        "waste_pct": round((1 - recommended / provisioned) * 100, 1),
    }

print(right_size(observed_p95=3.1, observed_max=4.4, provisioned=16))
# {'provisioned': 16, 'recommended': 4.6, 'waste_pct': 71.2}

# Memory needs a different rule: exceeding it kills the process, while exceeding
# CPU merely slows it. Use the observed maximum, never p95, and add more margin.
def right_size_memory(observed_max_gb: float, headroom: float = 1.5) -> float:
    return observed_max_gb * headroom

# Kubernetes: requests drive scheduling and cost; limits cap the blast radius.
resources:
  requests:
    cpu: "250m"            # what the scheduler reserves -> what you effectively pay
    memory: "512Mi"        # set from OBSERVED usage, not from a round number
  limits:
    cpu: "1000m"           # burst headroom; CPU over the limit is throttled
    memory: "768Mi"        # memory over the limit is an OOM kill, so keep this
                           # close to requests and generous versus real usage
# Requests far above real usage waste whole nodes: the scheduler reserves
# capacity nobody uses, and you pay for nodes running at 20% utilisation.

Beyond instance size, the recurring sources of waste are worth an explicit sweep: idle non-production environments running nights and weekends (schedule them off — an easy 60-70% saving on those accounts), unattached storage volumes and orphaned snapshots, load balancers and NAT gateways left behind by deleted stacks, over-provisioned database instances sized for a migration that finished, forgotten dev clusters, and log retention nobody ever shortened.

Right-sizing is safest when paired with autoscaling: the smaller instance is less frightening if the system adds instances under load. Sizing down without autoscaling converts a cost problem into an availability problem, which is a bad trade.

Key Takeaways

• Size CPU against p95 usage plus 20-30% headroom; size memory against the observed maximum, since exceeding it kills the process.
• Right-sizing decays — revisit it quarterly and after major changes.
• In Kubernetes, requests drive both scheduling and effective cost; inflated requests waste whole nodes.
• Sweep for idle non-production environments, orphaned volumes, snapshots, and forgotten load balancers.
• Pair downsizing with autoscaling so a cost win does not become an availability risk.

🧪 Practice

1. Pull two weeks of CPU and memory for a service and compute a right-sized recommendation for each.
2. Write a schedule that stops non-production environments outside working hours and estimate the annual saving.
3. Interview: Why is sizing memory with the same rule as CPU dangerous? (Hint: think about what happens when each resource is exceeded — one degrades, the other terminates.)

Cost vs. Latency vs. Reliability Trade-offs

Cost, latency, and reliability form a triangle in which improving one usually costs one or both of the others. Engineering maturity is not making the triangle disappear; it is making the trade explicit, quantified, and owned by someone with the authority to choose.

                     LATENCY
                       /\
                      /  \      caching, replicas, bigger instances,
                     /    \     edge presence  -> faster, MORE expensive
                    /      \
                   /        \
                  /__________\
              COST            RELIABILITY
                 multi-region, N+2 redundancy, backups, hot standby
                 -> more reliable, MORE expensive

    Pick a point deliberately. Every architecture is already a point on this
    triangle, chosen by accident if not on purpose.

Concrete examples of the same decision at different points:

Decision	Cheap option	Expensive option	What you buy
Availability target	99.9% (single region)	99.99% (multi-region active)	~43 min/month less downtime, ~2-3x cost
Read scaling	Cache with 5 min TTL	Read replicas, no staleness	Freshness, at replica cost
Backups	Nightly snapshot, RPO 24h	Continuous PITR, RPO 5 min	Data loss window
Deployment	Rolling, brief mixed state	Blue-green, instant rollback	Rollback speed, double capacity
Static content	Served from origin	Global CDN	Latency worldwide, offload egress

def availability_decision(annual_revenue: float, current_nines: float,
                          target_nines: float, extra_annual_cost: float) -> None:
    """Does buying more availability pay for itself in avoided lost revenue?"""
    hours = 8760
    downtime_now    = hours * (1 - current_nines)
    downtime_target = hours * (1 - target_nines)
    revenue_per_hour = annual_revenue / hours
    avoided = (downtime_now - downtime_target) * revenue_per_hour

    print(f"downtime {downtime_now:.1f}h -> {downtime_target:.1f}h per year")
    print(f"avoided loss ${avoided:,.0f} vs added cost ${extra_annual_cost:,.0f} "
          f"-> {'WORTH IT' if avoided > extra_annual_cost else 'NOT WORTH IT'}")
    # Caveats the arithmetic cannot capture: reputational damage, contractual
    # SLA penalties, and the fact that added complexity introduces its OWN
    # failure modes — a multi-region system can be less reliable in practice
    # than a well-run single-region one.

availability_decision(annual_revenue=40_000_000, current_nines=0.999,
                      target_nines=0.9999, extra_annual_cost=900_000)

Two heuristics make these decisions faster. First, not all traffic deserves the same treatment: the checkout path may justify multi-region redundancy while the recommendations service is allowed to be best-effort, and tiering by criticality avoids paying premium prices for everything. Second, degrade rather than fail: serving stale cached data, disabling personalisation, or falling back to a simpler ranking model keeps the product usable at a fraction of the cost of full redundancy — and it is often what users would choose if asked.

Finally, note the cases where the triangle collapses and one change helps everything at once. Caching reduces latency and cost and load on a fragile dependency. Smaller payloads help latency, egress cost, and mobile reliability together. Fixing an N+1 query improves all three. Look for those before trading anything away.

Key Takeaways

• Cost, latency, and reliability trade against each other; make the choice explicit rather than accidental.
• Quantify availability decisions against revenue at risk, while acknowledging what the arithmetic omits.
• Tier by criticality — premium reliability for revenue paths, best-effort for the rest.
• Graceful degradation buys most of the user-visible benefit of redundancy for a fraction of the price.
• Some changes — caching, smaller payloads, removing N+1 queries — improve all three at once; find those first.

🧪 Practice

1. Compute the annual revenue at risk for 99.9% versus 99.99% availability for a business you know, and decide whether the upgrade pays.
2. Classify a system's endpoints into criticality tiers and assign a different reliability target and budget to each.
3. Interview: How would you cut infrastructure cost by 30% without hurting user experience? (Hint: think about what is provisioned but idle, what is retained but unread, and which traffic never needed premium treatment.)

FinOps Practices

FinOps is the operational discipline that keeps cost decisions in the hands of the engineers who cause them. Its premise is simple: in the cloud, engineers make purchasing decisions every time they write a Terraform file, so cost has to be a visible engineering signal rather than a monthly surprise delivered to finance.

The practice runs as a loop, mirroring the observability loop of chapter 12.

 INFORM ---------------> OPTIMIZE ---------------> OPERATE ---------+
 visibility, tagging,    right-sizing, commitments,  budgets, alerts,|
 allocation, unit cost   architecture changes        policy, review  |
   ^                                                                 |
   +-----------------------------------------------------------------+
        each cycle updates the model and the forecast

Inform is the prerequisite for everything else: tag resources by service, team, and environment; allocate shared costs on a documented basis; publish per- team dashboards; and track unit cost (cost per request, per user, per order) rather than only total spend. Unit cost is the metric that distinguishes healthy growth from a leak — a total bill rising with traffic is fine, unit cost rising is not.

# Anomaly detection on daily spend: catch runaway cost within a day, not in the
# monthly invoice. The forgotten GPU cluster is a rite of passage.
import statistics

def cost_anomaly(daily: list[float], sigma: float = 3.0) -> None:
    baseline, today = daily[:-1], daily[-1]
    mean, sd = statistics.mean(baseline), statistics.pstdev(baseline)
    z = (today - mean) / sd if sd else 0.0
    if z > sigma:
        # Page a human only for genuine anomalies; monthly-close noise is not one.
        print(f"ALERT ${today:,.0f} vs baseline ${mean:,.0f} (z={z:.1f}), "
              f"annualised impact ${(today - mean) * 365:,.0f}")

cost_anomaly([4100, 4250, 4180, 4090, 4220, 4160, 9800])

# Policy as code: prevent waste at provisioning time instead of finding it later.
# Untagged resources cannot be attributed, and unattributable cost is
# unmanageable cost.
resource "aws_instance" "worker" {
  instance_type = var.instance_type

  tags = {
    Service     = var.service          # required by policy
    Team        = var.team             # required by policy
    Environment = var.environment      # required by policy
    CostCenter  = var.cost_center      # required by policy
    ExpiresAt   = var.expires_at       # non-prod: reaped automatically after this
  }

  lifecycle {
    precondition {
      condition     = var.environment == "prod" || var.expires_at != ""
      error_message = "Non-production resources must set an expiry date."
    }
  }
}

The practices that hold up over time: showback or chargeback so each team sees its own spend, budgets with alerts at 50/80/100% of forecast, cost as a line item in design reviews for anything material, an owner for the cloud bill who is an engineer rather than only an accountant, and automated reaping of untagged or expired non-production resources. Forecasting closes the loop, and it should be driven by unit cost and projected demand rather than by extrapolating last month's total.

Two anti-patterns are worth naming. Cost-cutting sprints that fix everything once and then let it decay are far less effective than continuous small adjustments. And optimising engineering time away to save infrastructure money is usually a loss: an engineer-week costs more than most of the savings people spend an engineer-week chasing — measure both sides before starting.

Key Takeaways

• FinOps runs a loop: inform (visibility), optimise (act), operate (govern).
• Tagging and allocation come first; unattributable cost cannot be managed.
• Track unit cost, not just total spend — rising unit cost is the real warning.
• Alert on daily anomalies rather than discovering them in the monthly invoice.
• Enforce tagging and expiry as policy at provisioning time, and prefer continuous small optimisation to periodic cost-cutting sprints.

🧪 Practice

1. Define a tagging standard for a system and identify which existing resources would fail it.
2. Build a daily cost anomaly check with a threshold and a defined owner for the alert.
3. Interview: How would you make engineers care about cloud cost without slowing them down? (Hint: think about which signals engineers already act on daily, and how cost could become one of them rather than a quarterly review.)

---

<a id="14-large-scale-data-and-intelligent-systems"></a>