12. Observability and Operations

A system you cannot see inside is a system you cannot operate, and every design decision in the previous chapters is a guess until production data confirms it. This chapter covers the telemetry that makes behaviour visible, the monitoring and alerting that turns telemetry into timely human action, the deployment techniques that let you change a running system without breaking it, and the infrastructure practices that keep environments reproducible rather than archaeological.

<a id="121-telemetry-pillars"></a>

12.1 Telemetry Pillars

Observability rests on three kinds of signal — logs, metrics, and traces — that answer different questions and cost very different amounts. This subchapter covers each, how context is carried across service boundaries to connect them, the standards that stop instrumentation from being vendor-specific, and the cardinality and sampling decisions that determine whether the bill is sustainable.

Structured Logging

A log is a record of a discrete event. The traditional form is a line of prose meant for a human to read, which works fine when one process runs on one machine and a person tails the file. It stops working the moment you have forty instances, because answering "how many times did this happen for enterprise customers in the last hour?" requires parsing prose with regular expressions — and those regexes break every time someone rewords a message.

Structured logging fixes this by emitting events as key-value data rather than sentences. The message becomes a stable event name and the variable parts become typed fields, so logs are queryable like a database rather than searchable like a book.

 UNSTRUCTURED                            STRUCTURED

 2026-08-26 10:15:04 ERROR Payment       {"ts":"2026-08-26T10:15:04Z",
 failed for user 8891 after 3            "level":"error","event":"payment_failed",
 retries (card declined)                 "user_id":"8891","retries":3,
                                          "reason":"card_declined",
 To count declines by reason:             "trace_id":"4bf92f...","amount":35.98}
   grep + regex + hope nobody
   reworded the message                  To count declines by reason:
                                           GROUP BY reason
 Fields are trapped in prose.            Fields are queryable, typed, stable.

import structlog, uuid

log = structlog.get_logger()

# ---- Bind context once; every subsequent line inherits it ------------------
def handle_request(request):
    request_log = log.bind(
        request_id=request.id,
        trace_id=request.trace_id,        # ties this log to a trace (see below)
        user_id=request.user.id,
        endpoint=request.path,
    )
    request_log.info("request_started", method=request.method)
    try:
        result = process(request, request_log)
        request_log.info("request_completed",
                         status=200,
                         duration_ms=result.elapsed_ms)   # a number, not text
        return result
    except PaymentDeclined as e:
        # The event name is STABLE; the details are fields. Renaming a field is
        # a breaking change you can search for; rewording prose is invisible.
        request_log.error("payment_failed",
                          reason=e.code, retries=e.attempts, amount=e.amount)
        raise

# ---- What not to do --------------------------------------------------------
# log.info(f"User {user.email} paid {amount}")
#   - interpolated: the fields cannot be aggregated
#   - contains PII: emails do not belong in logs at all (see 11.4)
#   - no event name: nothing stable to group by

Log levels exist to control volume and urgency, and using them consistently matters more than the exact taxonomy. A workable convention: ERROR means a request failed and someone should eventually look; WARN means something unexpected that the system handled; INFO records significant state changes; DEBUG is developer detail, off in production by default. The common failure is ERROR for conditions nobody acts on, which trains everyone to ignore errors.

Practice	Why it matters
Stable event names, variable fields	Aggregation survives rewording
Include trace_id and request_id	Connects logs to traces and to each other
Log durations as numbers	Enables percentiles without parsing
Never log PII, secrets, or tokens	Logs are widely readable and long-lived (11.2, 11.4)
Sample high-volume success logs	Cost scales with volume; failures matter more
One event per meaningful occurrence	Duplicate logging inflates cost and distorts counts

Logs are the most expensive pillar per unit of insight, because they are unaggregated: a million requests produce a million records. That is exactly what makes them irreplaceable for the specific case ("what happened to this request?") and a poor choice for the aggregate case ("what is the error rate?"), which metrics answer far more cheaply.

Key Takeaways

• Structured logs are queryable key-value events; prose logs trap their data in sentences that regexes must recover.
• Keep event names stable and put variability in typed fields.
• Bind request-scoped context once (trace_id, request_id, user) so every line can be correlated.
• Reserve ERROR for things someone will act on, or the level becomes noise.
• Logs answer "what happened to this one request?" — use metrics for aggregate questions, since logs cost per event.

🧪 Practice

1. Convert five interpolated log statements into structured events with stable names and typed fields.
2. Write a query over structured logs that counts payment failures by reason per hour, and explain why the same query is fragile against prose logs.
3. Interview: When would you choose a log over a metric? (Hint: think about whether the question is about one request or about all of them, and what the storage cost of each answer is.)

Metrics and Aggregation

A metric is a numeric measurement aggregated over time. Where a log records each event individually, a metric records a summary — a count, a sum, a distribution — so a million requests become a handful of numbers per interval. That compression is the entire point: metrics are cheap enough to keep for every service at all times, which makes them the right substrate for dashboards and alerts.

The trade is that aggregation discards detail. A metric can tell you the error rate tripled; it cannot tell you which customer was affected. The correct mental model is that metrics detect and quantify problems, while logs and traces diagnose them.

The four instrument types cover nearly everything:

Type	Semantics	Example	Query pattern
Counter	Monotonically increasing total	requests, errors, bytes sent	rate() over time
Gauge	A value that goes up and down	queue depth, memory, connections	current value, min/max
Histogram	Distribution bucketed at record time	request duration, payload size	percentiles, heatmaps
Summary	Client-side computed quantiles	latency (pre-aggregated)	read directly (rarely preferred)

from prometheus_client import Counter, Gauge, Histogram

# ---- Counter: only ever goes up; the DERIVATIVE is what you graph -----------
requests_total = Counter(
    "http_requests_total", "Total HTTP requests",
    ["method", "endpoint", "status"],      # labels: keep these LOW cardinality
)

# ---- Gauge: a point-in-time value that can decrease ------------------------
queue_depth = Gauge("job_queue_depth", "Jobs waiting", ["queue"])

# ---- Histogram: buckets chosen to bracket your SLO -------------------------
request_duration = Histogram(
    "http_request_duration_seconds", "Request duration",
    ["endpoint"],
    buckets=(.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10),
    # Buckets are fixed at definition time and cannot be changed retroactively.
    # Put a boundary exactly at your SLO threshold (e.g. 0.25) so the
    # "fraction of requests under SLO" query is exact rather than interpolated.
)

def handle(request):
    with request_duration.labels(endpoint=request.route).time():   # observes on exit
        response = process(request)
    requests_total.labels(method=request.method,
                          endpoint=request.route,       # the ROUTE PATTERN...
                          status=response.status).inc()
    # ...never request.path -- "/orders/8891" would create one time series per
    # order id. See "Cardinality and Sampling" below; this single mistake is
    # the most common cause of a monitoring system falling over.
    return response

# Rate of requests per second, averaged over 5 minutes.
# Always rate() a counter: the raw value is meaningless (it resets on restart,
# and rate() handles the reset correctly).
sum(rate(http_requests_total[5m])) by (endpoint)

# Error ratio -- the numerator and denominator must use the same window.
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))

# p99 latency from histogram buckets, aggregated across instances FIRST.
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, endpoint))
#   ^ summing buckets before computing the quantile is correct;
#     averaging per-instance p99s is not (see the next topic).

Two properties of metrics shape how they are used. They are pull or push based and time-aligned, so every series has a value per scrape interval — which makes them ideal for arithmetic across services. And they have fixed cost per series per interval, independent of traffic, which is why a metric can cover a million requests per second at the same cost as ten.

Key Takeaways

• Metrics aggregate at record time, giving fixed cost independent of traffic — the right substrate for dashboards and alerts.
• Counters go up and are always queried with rate(); gauges move both ways; histograms capture distributions.
• Choose histogram buckets deliberately and place one boundary at your SLO threshold — buckets cannot be changed retroactively.
• Label with route patterns and other low-cardinality values, never with IDs.
• Metrics detect and quantify; logs and traces diagnose.

🧪 Practice

1. Instrument an endpoint with a counter and a histogram, choosing buckets around a 250 ms SLO.
2. Write queries for request rate, error ratio, and p95 latency by endpoint, all over the same time window.
3. Interview: Why do you almost never graph a counter's raw value? (Hint: think about what happens to the number when a process restarts, and what question you were actually asking.)

Distributed Tracing

When a single user request touches eight services, neither logs nor metrics answer the question that matters: where did the time go, and which hop failed? Metrics show that the checkout endpoint is slow; logs show individual events in individual services with no inherent connection. Tracing supplies the missing structure by recording the causal tree of one request across every service it touched.

A trace is one request end to end. A span is one unit of work inside it — an HTTP handler, a database query, a call to another service — with a start time, a duration, attributes, and a parent. Spans form a tree, and rendering that tree against time produces the waterfall view that makes latency obvious.

 ONE TRACE, trace_id=4bf92f3577b34da6

  api-gateway  ├────────────────────────────────────────────────┤  310 ms
   orders      │  ├──────────────────────────────────────────┤     295 ms
    auth       │  │  ├──┤                                          12 ms
    pricing    │  │     ├────────┤                                 45 ms
    inventory  │  │              ├──────────────────────────┤     210 ms  <--
     db query  │  │              │  ├──┤                            8 ms
     vendor    │  │              │     ├───────────────────┤      195 ms  <--
    notify     │  │                                         ├─┤     5 ms
              0ms                                              310ms

  The waterfall answers instantly what metrics cannot: 195 of 310 ms is one
  third-party call inside inventory. Without the tree you would be reading
  five services' dashboards and guessing.

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def checkout(cart, user):
    # A span for the logical operation, not for every function call: spans have
    # cost, and a trace with 4,000 spans is unreadable as well as expensive.
    with tracer.start_as_current_span("checkout") as span:
        span.set_attribute("user.tier", user.tier)         # low cardinality: good
        span.set_attribute("cart.item_count", len(cart.items))
        span.set_attribute("user.id", user.id)  # high cardinality is FINE on a
        #                                          span (unlike a metric label):
        #                                          spans are individual records
        try:
            price = quote(cart)          # child spans are created automatically
            reserve(cart)                # by instrumented HTTP/DB clients
            order = place(cart, price)
            span.set_attribute("order.id", order.id)
            return order
        except Exception as e:
            span.record_exception(e)                 # attaches the stack trace
            span.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))
            raise                                    # never swallow to "fix" a span

def quote(cart):
    with tracer.start_as_current_span("pricing.quote") as span:
        span.set_attribute("pricing.rules_version", RULES_VERSION)
        ...

The single most valuable practice is to link the three pillars: put the trace_id in every log line (as in the logging topic) and attach exemplar trace IDs to metrics. That turns investigation into navigation — a latency spike on a dashboard leads to an exemplar trace, which leads to the slow span, which leads to that service's logs for that exact request.

Two cautions. Tracing has real overhead, in the instrumented process and in the backend, which is why sampling exists (below). And traces are only as complete as context propagation allows: one service that fails to forward the trace headers severs the tree, and everything beyond it becomes invisible — which is the subject of the next topic.

Key Takeaways

• A trace is one request's causal tree across services; a span is one unit of work with a parent, duration, and attributes.
• The waterfall view answers "where did the time go?" — a question metrics and logs cannot answer across service boundaries.
• Span attributes may be high cardinality (unlike metric labels) because spans are individual records.
• Record exceptions and set span status on failure so error traces are findable.
• Link pillars by putting trace_id in logs and exemplars on metrics; that is what makes investigation navigation rather than guesswork.

🧪 Practice

1. Instrument a two-service call path and view the resulting waterfall, identifying which span dominates.
2. Add span attributes that would let you distinguish slow requests by customer tier and by feature flag state.
3. Interview: A dashboard shows p99 latency doubled but every service's own latency looks normal. What is happening? (Hint: think about what a trace shows that per-service metrics cannot, including time spent between spans.)

Span Context Propagation

A trace only exists because every participant agrees on an identifier and passes it along. Span context propagation is the mechanism: a small piece of data — the trace ID, the current span ID, and sampling flags — travels with the request across every process boundary, so a receiving service knows which trace it is part of and which span it descends from.

Without it, each service starts its own unrelated trace and you get eight disconnected single-span traces instead of one tree. This is the most common reason tracing "does not work" in practice, and the break is usually one uninstrumented client, one message queue, or one thread pool.

 W3C TRACE CONTEXT -- the standard headers

  traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
               |  |                                |                |
               |  trace-id (16 bytes)              parent span-id   flags
               version                                              01 = sampled

  tracestate: vendor1=value,vendor2=value      (vendor-specific extras)

 PROPAGATION ACROSS A CALL

  [service A]  span 00f067aa (parent: none)
      |  HTTP request, injects headers
      |  traceparent: 00-4bf92f...-00f067aa-01
      v
  [service B]  extracts -> creates span a3ce929d with parent 00f067aa
      |  publishes a message, injects headers INTO THE MESSAGE
      v
  ( broker )   headers ride with the message, possibly for hours
      v
  [worker C]   extracts -> span continues the same trace_id
               (a "follows-from" link rather than a child, since the parent
                request has usually finished by the time the worker runs)

from opentelemetry import trace, context
from opentelemetry.propagate import inject, extract

# ---- Outbound: inject context into whatever carrier you are using ----------
def call_downstream(url, payload):
    headers = {}
    inject(headers)                        # writes `traceparent` / `tracestate`
    return requests.post(url, json=payload, headers=headers, timeout=2)

# ---- Inbound: extract before creating the server span ----------------------
def handle_http(request):
    ctx = extract(request.headers)         # None-safe: absent headers start a new trace
    with tracer.start_as_current_span("http.server", context=ctx):
        return route(request)

# ---- Asynchronous boundaries: the case people forget -----------------------
def publish(message, producer):
    carrier = {}
    inject(carrier)                        # context travels IN the message
    producer.send(topic="orders", value=message, headers=list(carrier.items()))

def consume(record):
    ctx = extract(dict(record.headers))
    # A queued job is not a child of a still-running request -- use a LINK so
    # the relationship is recorded without implying the parent is waiting.
    link = trace.Link(trace.get_current_span(ctx).get_span_context())
    with tracer.start_as_current_span("order.process", links=[link]):
        process(record.value)

# ---- Thread and task boundaries: context is thread-local -------------------
def offload(fn, executor):
    ctx = context.get_current()            # capture on the calling thread
    def wrapped():
        token = context.attach(ctx)        # restore inside the worker thread
        try:
            return fn()
        finally:
            context.detach(token)          # always detach, or context leaks
    return executor.submit(wrapped)

Two design consequences follow. Sampling decisions must propagate, because the flag in traceparent is what makes every service agree to record the same trace — if each service sampled independently at 10%, the probability of a complete eight-service trace would be one in ten million. And propagation is a cross-team contract: gateways, proxies, service meshes, and third-party SDKs must all forward the headers, so an allowlist-based proxy that strips unknown headers silently destroys tracing everywhere behind it.

Baggage is a related mechanism worth knowing: arbitrary key-value pairs that propagate alongside trace context, letting an upstream annotation (say, tenant.tier=enterprise) be readable by every downstream service. It is useful and easily abused — everything in baggage is copied on every hop, and it crosses trust boundaries, so it must never carry secrets or PII.

Key Takeaways

• Propagation carries trace ID, span ID, and sampling flags across every boundary; without it you get disconnected single-span traces.
• W3C traceparent/tracestate is the interoperable standard — prefer it over vendor-specific headers.
• Inject and extract at asynchronous boundaries too (queues, jobs), using links rather than parent-child when the parent has finished.
• Context is thread-local: capture and re-attach it explicitly when work moves to another thread or task.
• The sampling decision must propagate, or complete multi-service traces become statistically impossible; proxies that strip headers break tracing silently.

🧪 Practice

1. Trace a request through an HTTP call and a queued job, and verify all spans share one trace ID.
2. Break propagation deliberately in one service and observe exactly what the trace view shows afterwards.
3. Interview: Why must the sampling decision be made once and propagated? (Hint: compute the probability that eight services independently sampling at 10% all choose to record the same request.)

OpenTelemetry Standards

Before OpenTelemetry, instrumentation was vendor-specific: choosing a monitoring backend meant adopting its SDK throughout your code, and changing backends meant re-instrumenting everything. That coupling was expensive enough that teams tolerated bad tooling to avoid it, and it made polyglot systems especially painful — every language needed a different vendor library with different semantics.

OpenTelemetry (OTel) is a vendor-neutral standard covering all three signals. It defines an API you write against, an SDK that implements it, a wire protocol (OTLP), and — most valuable and least appreciated — semantic conventions: the agreed names for common attributes, so http.request.method means the same thing in every language and every backend.

 ARCHITECTURE

  your code -> OTel API  (stable; what you write against)
                  |
               OTel SDK  (sampling, batching, processors, resource detection)
                  |
                 OTLP    (the wire protocol)
                  |
          [ OTel Collector ]  <-- the seam that makes backends swappable
             /     |      \
        Prometheus Jaeger  vendor backend
        (metrics) (traces)  (anything)

  Change the backend: edit the Collector's config. Application code untouched.

from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import Resource
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Resource attributes identify the emitting entity and are attached to every
# signal, which is what lets you filter all telemetry by service or version.
resource = Resource.create({
    "service.name": "orders",              # required by convention
    "service.version": "1.14.2",
    "deployment.environment": "production",
})

provider = TracerProvider(resource=resource)
provider.add_span_processor(
    # Batching is not an optimization detail: exporting per span would add a
    # network round trip to every operation you are trying to measure.
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://collector:4317"),
                       max_queue_size=2048, schedule_delay_millis=5000)
)
trace.set_tracer_provider(provider)

# Semantic conventions: use the standard names so backends and dashboards
# understand your data without per-service configuration.
with trace.get_tracer(__name__).start_as_current_span("GET /orders/{id}") as s:
    s.set_attribute("http.request.method", "GET")        # NOT "method"
    s.set_attribute("url.path", "/orders/8891")          # NOT "path"
    s.set_attribute("http.response.status_code", 200)    # NOT "status"
    s.set_attribute("server.address", "orders.internal")

# The Collector: receive, process, export. Running one is what turns backend
# choice into a configuration change and gives you one place to enforce policy.
receivers:
  otlp:
    protocols: { grpc: { endpoint: 0.0.0.0:4317 }, http: {} }

processors:
  batch: { timeout: 10s, send_batch_size: 1024 }
  memory_limiter: { check_interval: 1s, limit_mib: 512 }   # protect the collector
  attributes:                       # central redaction: one place, every service
    actions:
      - key: user.email
        action: delete
      - key: http.request.header.authorization
        action: delete
  tail_sampling:                    # decide AFTER the full trace is assembled
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 1000 }
      - name: sample-the-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 1 }

exporters:
  prometheus: { endpoint: 0.0.0.0:8889 }
  otlp/traces: { endpoint: jaeger:4317 }

service:
  pipelines:
    traces:  { receivers: [otlp], processors: [memory_limiter, tail_sampling, batch], exporters: [otlp/traces] }
    metrics: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [prometheus] }

The architectural argument for running a Collector rather than exporting directly from each application: it decouples applications from backends, gives one place to apply redaction and sampling policy, buffers during backend outages so telemetry is not lost, and lets you fan out to several destinations during a migration. Applications then need to know one endpoint and nothing about where the data ultimately lands.

Maturity differs by signal — tracing and metrics are stable across major languages, while logging support arrived later and often still routes through existing logging libraries with an OTel bridge. Auto-instrumentation covers common frameworks and HTTP/DB clients with no code changes, which is the fastest way to get a useful trace tree; manual spans are then added for business-level operations that libraries cannot see.

Key Takeaways

• OpenTelemetry decouples instrumentation from backends: write once, export anywhere, change vendors by configuration.
• Semantic conventions are the underrated part — standard attribute names make telemetry portable and dashboards reusable.
• Set resource attributes (service.name, version, environment) so every signal can be filtered by its origin.
• Run a Collector as the seam: central redaction and sampling policy, buffering, and multi-backend fan-out.
• Start with auto-instrumentation for frameworks and clients, then add manual spans for business operations.

🧪 Practice

1. Instrument a service with auto-instrumentation plus one manual span, and export through a local Collector.
2. Configure the Collector to drop two sensitive attributes and verify they never reach the backend.
3. Interview: Why run a Collector instead of exporting straight to the backend? (Hint: think about what happens during a backend outage, and where you would otherwise have to change redaction policy.)

Cardinality and Sampling

Telemetry cost is not driven by traffic alone but by distinctness, and this is where monitoring bills and monitoring outages both originate. Cardinality is the number of unique time series a metric produces: the product of the number of distinct values across all its labels. Adding one unbounded label can multiply your series count by millions.

# Cardinality is multiplicative. Compute it before shipping the label.
def series_count(**label_value_counts):
    total = 1
    for _, n in label_value_counts.items():
        total *= n
    return total

# Reasonable
print(series_count(endpoint=40, method=5, status=6))            # 1,200 series

# One innocuous addition
print(series_count(endpoint=40, method=5, status=6, region=4))  # 4,800  fine

# The mistake that takes down a metrics backend
print(series_count(endpoint=40, method=5, status=6, user_id=500_000))
# 600,000,000 series -- each with its own memory, index entry, and retention.
#
# Labels that are ALWAYS wrong on a metric: user_id, order_id, session_id,
# request_id, email, full URL path, raw error message, timestamp.
# Those belong on SPANS and LOGS, which are individual records rather than
# time series -- high cardinality there is free by comparison.

The rule that follows: metrics need bounded labels; traces and logs carry the detail. If you want per-user investigation, find the user's trace, not their metric series. A useful discipline is to require that any new label has a known, enumerable set of values, and to normalize anything else — route patterns instead of paths, error classes instead of messages, bucketed sizes instead of exact byte counts.

Sampling is the corresponding lever for traces and logs, where volume rather than distinctness drives cost. The choice is when the decision is made.

Strategy	Decision point	Keeps errors?	Cost / complexity
Head-based, fixed	At trace start	Only by luck	Cheapest, simplest
Head-based, rate-limited	At trace start, per second budget	By luck	Cheap; protects the backend
Tail-based	After the trace completes	Yes, deterministically	Needs buffering in a collector
Adaptive	Dynamic rate per route	Depends	Complex; good at extreme scale

# Head-based sampling is a decision made before you know what happened,
# which is its whole weakness: errors are rare, and rare things get dropped.
import random

def head_sample(trace_id: str, rate: float) -> bool:
    # Deterministic on trace_id so every service makes the SAME decision from
    # the propagated ID -- never random per service (see propagation topic).
    return (int(trace_id[-8:], 16) / 0xFFFFFFFF) < rate

# Tail-based sampling, expressed as policy: keep everything interesting, and a
# small sample of the boring majority for baseline comparisons.
def tail_decision(trace) -> bool:
    if trace.has_error:                      return True     # 100% of failures
    if trace.duration_ms > 1000:             return True     # 100% of slow
    if trace.attributes.get("user.tier") == "enterprise":  return True
    return head_sample(trace.trace_id, 0.01)                 # 1% baseline
    # Result: near-complete coverage of what you investigate, at ~1-2% of the
    # storage of keeping everything. The cost is buffering every trace until
    # it completes, which is what the Collector does.

Three practical guidelines. Never sample away errors — they are the rare events you will need. Keep enough baseline that you can compare a failing trace against a healthy one, which is impossible at 0.01% sampling. And remember that metrics remain complete even when traces are sampled: rates and percentiles come from full aggregation, so heavy trace sampling does not distort your dashboards or alerts. That division of labour — complete metrics, sampled traces, targeted logs — is what makes observability affordable at scale.

Key Takeaways

• Metric cost is driven by cardinality, the multiplicative product of label value counts; one unbounded label can create millions of series.
• Never put IDs, emails, raw paths, or error messages in metric labels — put them on spans and logs instead.
• Sampling controls trace and log volume; head-based is cheap but drops errors by chance, tail-based keeps them deterministically.
• Sample errors and slow traces at 100% and keep a small baseline sample for comparison.
• Metrics stay complete under trace sampling, so alerting accuracy is unaffected — that split is what makes the total cost manageable.

🧪 Practice

1. Compute the series count for a metric with four labels and identify which one you would remove or normalize.
2. Configure tail-based sampling that keeps all errors, all traces over one second, and 1% of the rest, then measure the volume reduction.
3. Interview: Your metrics backend fell over after a deploy that added one label. Explain what happened and how you would prevent a recurrence. (Hint: think multiplicatively, and about what a code review would have to check.)

<a id="122-monitoring-and-alerting"></a>

12.2 Monitoring and Alerting

Telemetry is only useful if it reaches a human at the right moment with the right framing. This subchapter covers which signals to watch, how to aggregate latency without lying to yourself, how to build dashboards and alerts people trust, and what happens when the page fires.

Golden Signals

Given unlimited metrics, teams instrument everything and then cannot tell which numbers matter during an incident. The golden signals are the small set that detects nearly every user-visible problem, and their value is prioritization: if you can only build four graphs per service, build these four.

• Latency — how long requests take. Crucially, measure successful and failed requests separately, because a fast-failing service can look like a fast service.
• Traffic — demand on the system: requests per second, messages consumed, concurrent sessions.
• Errors — the rate of failed requests, including the ones that return HTTP 200 with a broken body.
• Saturation — how full the constrained resource is: CPU, memory, connection pool, queue depth, disk. This is the leading indicator; the other three tell you a problem has arrived, saturation tells you it is coming.

The related frameworks are worth knowing because they suit different targets. RED (Rate, Errors, Duration) is the request-service view — essentially the golden signals minus saturation. USE (Utilization, Saturation, Errors) is the resource view, applied per resource rather than per service.

Framework	Applies to	Signals	Best for
Golden	Any user-facing service	Latency, traffic, errors, saturation	General service health
RED	Request-driven services	Rate, errors, duration	Microservice dashboards
USE	Resources (CPU, disk, pool)	Utilization, saturation, errors	Capacity and infrastructure

# The four signals for one service, in queries you can paste into a dashboard.

# TRAFFIC
sum(rate(http_requests_total{service="orders"}[5m]))

# ERRORS -- as a RATIO, because the raw count rises with traffic and tells you
# nothing about whether the system is healthier or sicker.
sum(rate(http_requests_total{service="orders", status=~"5.."}[5m]))
  / sum(rate(http_requests_total{service="orders"}[5m]))

# LATENCY -- successful requests only, or fast failures flatter the graph.
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket{service="orders",
                                                status!~"5.."}[5m])) by (le))

# SATURATION -- the leading indicator. Whatever runs out FIRST is the one to
# graph; for most services that is a pool or a queue, not CPU.
max(db_connection_pool_in_use / db_connection_pool_size)
sum(job_queue_depth) by (queue)

# Saturation is only meaningful against a known limit, so record both.
# A gauge of "connections in use" with no denominator cannot be alerted on.
pool_in_use = Gauge("db_connection_pool_in_use", "Connections checked out", ["pool"])
pool_size   = Gauge("db_connection_pool_size",   "Configured pool size",    ["pool"])

def checkout_connection(pool):
    conn = pool.acquire()
    pool_in_use.labels(pool=pool.name).set(pool.in_use)
    pool_size.labels(pool=pool.name).set(pool.size)     # the denominator
    return conn

# Utilization vs saturation, which are not the same thing:
#   utilization = fraction of capacity in use          (80% of the pool busy)
#   saturation  = work that could NOT be served now    (requests WAITING for a
#                                                       connection)
# A pool at 100% utilization with no waiters is fine. A pool at 100% with a
# growing wait queue is an outage in progress. Instrument the wait.
pool_wait_seconds = Histogram("db_pool_wait_seconds", "Time waiting for a connection")

Two refinements make the signals genuinely useful. Measure from the user's side where you can — a server-side latency graph excludes DNS, TLS, network, and client rendering, so a service can look healthy while the product is unusable; synthetic probes and real-user monitoring close that gap. And define "error" by user impact rather than by status code: a 200 response containing an empty result set where data was expected is an error, and a 404 for a genuinely missing resource is not.

Key Takeaways

• Latency, traffic, errors, and saturation detect nearly every user-visible problem; build these before anything else.
• Measure latency for successful requests separately — fast failures otherwise improve the graph.
• Express errors as a ratio, not a count, so the signal is independent of traffic.
• Saturation is the leading indicator; instrument it against a known limit and measure waiting, not just utilization.
• Complement server-side signals with client-side or synthetic measurement, and define errors by user impact rather than status code.

🧪 Practice

1. Build the four golden-signal queries for a service you know and identify which resource is its true saturation point.
2. Add a wait-time histogram to a connection pool and show how it distinguishes healthy full utilization from saturation.
3. Interview: A service reports 99.99% success and users are complaining. What are the possible explanations? (Hint: think about where the measurement is taken and what counts as success.)

Percentile Latency vs. Averages

Averages hide the experiences that matter. If 99 requests take 50 ms and one takes 5 seconds, the mean is 99.5 ms — a number describing nobody's experience, comfortably under a 200 ms target, while one user in a hundred waited five seconds. Latency distributions are right-skewed and often multi-modal (cache hit versus miss, warm versus cold), and the mean is a poor summary of any of them.

Percentiles describe the distribution directly: p50 is the typical experience, p99 is the bad-but-not-freak experience, and p99.9 catches the tail that hits your highest-volume customers most often. Chapter 13.1 treats tail latency as a performance-engineering problem; here the concern is measuring and aggregating it correctly, which is where most teams go wrong.

 SAME MEAN, DIFFERENT SYSTEMS

  A: tight distribution            B: bimodal (cache hit / miss)
     requests                          requests
       |    ####                         |  ####            ##
       |   ######                        | ######          ####
       |  ########                       |########        ######
       +--------------- ms               +---------------------- ms
         40  50  60                        20  30      400  500

  mean(A) = 50 ms   p99(A) =  62 ms
  mean(B) = 50 ms   p99(B) = 480 ms      <- same average, different product

 The mean cannot distinguish these. The percentile can.

# THE AGGREGATION TRAP: percentiles do not average.
# Three instances, each reporting its own p99:
per_instance_p99 = [120, 130, 900]        # ms
print(sum(per_instance_p99) / 3)          # 383 ms -- a number with NO meaning

# The real p99 depends on how much traffic each instance served and on the
# full distributions. It could be higher or lower than 383; the average of
# percentiles is not an estimate of anything.
#
# The correct approach: aggregate the underlying HISTOGRAM BUCKETS across
# instances first, then compute the quantile once from the merged distribution.

# WRONG: computes a p99 per instance, then averages them.
avg(histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])))

# RIGHT: sums the bucket counts across instances, then takes one quantile.
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
#     ^ `le` must be preserved; every other label is summed away

Which percentile to target is a product decision with cost consequences, and the answer depends on how many requests a single user experience contains.

Percentile	Interpretation	Typical use
p50	The typical request	Capacity planning, general health
p90 / p95	Common bad case	Balanced SLO targets
p99	Rare bad case; still frequent at scale	Standard SLO for user-facing paths
p99.9	The extreme tail	High-volume APIs, paying enterprise
max	The single worst request	Debugging only — one outlier dominates

The reason p99 matters more than it sounds is fan-out amplification: if one page issues 20 backend calls, the chance that at least one lands on the p99 path is 1 - 0.99^20, about 18%. The p99 of a dependency becomes the near-typical experience of a page that depends on twenty of them — which is why chapter 9's hedged requests and this chapter's percentile discipline exist.

One further caution: percentiles over long windows hide short incidents. A p99 computed over 24 hours barely moves during a ten-minute outage. Alert on short windows (five minutes), and use long windows only for SLO reporting, where the burn-rate approach in the next topic handles the interaction properly.

Key Takeaways

• Averages hide skew and cannot distinguish a tight distribution from a bimodal one; percentiles describe the distribution.
• Percentiles cannot be averaged — aggregate histogram buckets across instances first, then compute the quantile once.
• Choose the target percentile by product impact; p99 is the usual default for user-facing paths.
• Fan-out amplifies the tail: 20 parallel calls make a p99 event roughly an 18% chance per page.
• Use short windows for alerting; long-window percentiles mask brief incidents.

🧪 Practice

1. Take two latency datasets with equal means and different shapes, and compute p50, p95, and p99 for each.
2. Find a dashboard query that averages per-instance percentiles and rewrite it to aggregate buckets correctly.
3. Interview: Your p50 is unchanged and p99 tripled. What kinds of causes fit that pattern? (Hint: think about what affects a minority of requests — a subset of instances, a cache, a lock, or one slow shard.)

Dashboard Design

A dashboard is a tool for answering questions under time pressure, and most are built as if they were art galleries — dozens of panels showing everything available, arranged by what was easy to query. During an incident that is worse than useless: the responder scrolls, hunts, and forms hypotheses from whatever panel happens to be red.

The design principle is to build dashboards around questions, in a hierarchy that supports the actual investigation path: is something wrong → what is wrong → where is it wrong → why.

 A THREE-LEVEL HIERARCHY

  L1  SERVICE HEALTH        one screen, no scrolling, answers "are we okay?"
      +---------------------------------------------------------------+
      | SLO burn: 1.2x  |  error rate: 0.4%  |  p99: 240ms  | RPS: 3.1k|
      +---------------------------------------------------------------+
      |  [ error ratio, 24h ]        |  [ p99 latency, 24h ]           |
      |  [ traffic, 24h ]            |  [ saturation, 24h ]            |
      +---------------------------------------------------------------+
                              |  drill down
  L2  SERVICE DETAIL          by endpoint, by dependency, by instance
      per-endpoint latency and errors | dependency latency | pool waits
                              |  drill down
  L3  DEBUG                   raw traces, logs filtered by trace_id,
                              per-instance resource graphs, recent deploys

  Each level answers ONE question and links to the next. The alert links to
  L1, L1 links to L2, L2 links to exemplar traces.

The rules that separate usable dashboards from decorative ones:

• One screen for the top level. If a responder must scroll, the layout has already failed; put the summary numbers top-left, where eyes land first.
• Every panel earns its place. If nobody can say what decision a panel informs, delete it. Unused panels dilute attention and slow queries.
• Annotate deploys and incidents. The single most common cause of a change in a graph is a change you made; overlaying deploy markers answers "what changed?" before anyone opens a ticket.
• Consistent layout across services. When every service dashboard has the same shape, responders read them without relearning, which matters at 3 a.m.
• Show the target on the graph. A latency panel with the SLO threshold drawn as a line converts "is 240 ms bad?" into a glance.
• Prefer ratios and rates over raw counts, so panels stay meaningful as traffic grows.

// A dashboard defined as code: versioned, reviewed, and reproducible across
// environments -- the same reason infrastructure is defined as code (12.4).
{
  "title": "Orders Service - Health",
  "templating": [
    { "name": "env",     "query": "label_values(deployment_environment)" },
    { "name": "service", "query": "label_values(service_name)" }
  ],
  "annotations": [
    { "name": "deploys",
      "datasource": "prometheus",
      "expr": "changes(build_info{service=\"$service\"}[5m]) > 0" }
  ],
  "panels": [
    { "title": "SLO burn rate (1h)", "type": "stat", "gridPos": {"x":0,"y":0,"w":6,"h":4},
      "targets": [{ "expr": "slo:error_budget_burn_rate_1h{service=\"$service\"}" }],
      "thresholds": [ {"value": 1, "color": "yellow"}, {"value": 6, "color": "red"} ] },

    { "title": "Error ratio", "type": "timeseries", "gridPos": {"x":6,"y":0,"w":9,"h":8},
      "targets": [{ "expr": "sum(rate(http_requests_total{service=\"$service\",status=~\"5..\"}[5m])) / sum(rate(http_requests_total{service=\"$service\"}[5m]))" }],
      "thresholds": [ {"value": 0.01, "color": "red"} ] },

    { "title": "Latency p50/p95/p99", "type": "timeseries", "gridPos": {"x":15,"y":0,"w":9,"h":8},
      "targets": [
        { "expr": "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket{service=\"$service\"}[5m])) by (le))", "legendFormat": "p50" },
        { "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{service=\"$service\"}[5m])) by (le))", "legendFormat": "p99" }
      ] }
  ]
}

A last distinction worth holding onto: dashboards are for known unknowns — questions you anticipated well enough to build a panel for. Genuinely novel failures require ad-hoc querying over high-cardinality data, which is what tracing and log exploration are for. A team that can only look at dashboards can only diagnose failures it has already seen.

Key Takeaways

• Build dashboards around questions in a drill-down hierarchy: are we okay, what is wrong, where, why.
• Keep the top level to one screen with summary numbers first; delete panels that inform no decision.
• Annotate deploys — the most common answer to "what changed?" is a deploy.
• Use a consistent layout across services and draw SLO thresholds on the graph.
• Dashboards answer anticipated questions; novel failures need ad-hoc queries over traces and logs.

🧪 Practice

1. Rebuild an existing dashboard as a three-level hierarchy and justify each surviving panel.
2. Add deploy annotations and find one past incident where the correlation is visible.
3. Interview: What makes a dashboard useful at 3 a.m. specifically? (Hint: think about cognitive load, consistency across services, and what a tired person can do without reading documentation.)

Alert Thresholds and Anomaly Detection

An alert should mean one thing: a human needs to do something now. Every alert that does not meet that bar erodes trust in all the others, so the design question is not "what can we detect?" but "what warrants waking someone?"

The traditional approach sets static thresholds on causes — CPU above 80%, disk above 90%, memory above some line. It fails in both directions. A service can run at 95% CPU perfectly happily, and it can be completely broken at 10% CPU. These are cause-based alerts, and they fire for conditions that may have no user impact while missing failures that do.

Symptom-based alerting inverts this: alert on what the user experiences — error rate, latency, unavailability — and use cause metrics for diagnosis rather than paging. The refinement is to tie the threshold to the SLO (chapter 9.4) through a burn rate: how fast the error budget is being consumed relative to the rate that would exhaust it exactly at period end.

# Burn rate: the rate of budget consumption relative to "on pace".
# burn_rate = observed_error_rate / error_budget_rate
#   1x  -> exactly on pace to exhaust the budget at the end of the window
#   14x -> the entire 30-day budget is gone in ~2 days

def burn_rate(observed_error_ratio, slo_target):
    budget = 1 - slo_target                   # 99.9% SLO -> 0.001 budget
    return observed_error_ratio / budget

print(burn_rate(0.014, 0.999))    # 14.0 -> page now
print(burn_rate(0.0012, 0.999))   # 1.2  -> ticket, not a page

# MULTI-WINDOW, MULTI-BURN-RATE: the standard pattern. Fast burns page quickly;
# slow burns open a ticket. Requiring BOTH a long and a short window to be
# breaching suppresses single-spike false positives.
ALERT_TIERS = [
    # (long window, short window, burn threshold, budget consumed, action)
    ("1h",  "5m",  14.4, "2%",  "page"),      # catastrophic: ~2 days to exhaust
    ("6h",  "30m",  6.0, "5%",  "page"),      # severe
    ("24h", "2h",   3.0, "10%", "ticket"),    # notable
    ("72h", "6h",   1.0, "10%", "ticket"),    # slow bleed
]

# Prometheus alerting rules implementing the top tier.
groups:
- name: slo
  rules:
  - alert: OrdersErrorBudgetFastBurn
    expr: |
      (
        sum(rate(http_requests_total{service="orders",status=~"5.."}[1h]))
          / sum(rate(http_requests_total{service="orders"}[1h])) > 14.4 * 0.001
      )
      and
      (
        sum(rate(http_requests_total{service="orders",status=~"5.."}[5m]))
          / sum(rate(http_requests_total{service="orders"}[5m])) > 14.4 * 0.001
      )
      # The `and` is the point: the long window confirms it is real, the short
      # window confirms it is still happening. Either alone is noisy.
    for: 2m
    labels: { severity: page }
    annotations:
      summary: "Orders burning error budget 14x -- ~2 days to exhaustion"
      runbook: "https://runbooks.example.com/orders/error-budget"
      dashboard: "https://grafana.example.com/d/orders-health"

Anomaly detection addresses what thresholds cannot: metrics with strong seasonal patterns, where "normal" at 3 a.m. Sunday differs from 2 p.m. Tuesday by an order of magnitude. A fixed threshold either misses a real drop at night or fires constantly during the day.

# Seasonal comparison: compare against the same time in previous weeks rather
# than against a constant. Simple, explainable, and usually sufficient.
import statistics

def seasonal_anomaly(current, same_hour_previous_weeks, sensitivity=3.0):
    baseline = statistics.median(same_hour_previous_weeks)   # median resists outliers
    spread = statistics.median(
        [abs(v - baseline) for v in same_hour_previous_weeks]) or 1e-9   # MAD
    z = abs(current - baseline) / (1.4826 * spread)          # robust z-score
    return z > sensitivity, baseline, z

# Detecting an ABSENCE is as important as detecting a spike: orders dropping to
# zero at noon is an outage that no error-rate alert will catch, because zero
# requests produce zero errors.
print(seasonal_anomaly(0, [1200, 1150, 1300, 1250]))    # (True, 1225, ~large)

Guidelines that keep an alerting system trustworthy: alert on symptoms, page only for user impact; every alert links to a runbook (12.4) and a dashboard; use for: durations so transient blips self-resolve; prefer ratios to absolutes so thresholds survive growth; and treat missing data explicitly — a metric that stops arriving usually means something worse than a metric that crossed a line.

Key Takeaways

• Alert on symptoms the user feels, not causes like CPU; use cause metrics for diagnosis.
• Burn-rate alerting ties thresholds to the SLO: fast burns page, slow burns open tickets.
• Multi-window conditions (long confirms real, short confirms ongoing) suppress single-spike false positives.
• Use seasonal baselines for metrics with daily and weekly cycles, and alert on absence as well as excess.
• Every alert needs a runbook link, a dashboard link, and an action a human can take; alerts without one of those should be deleted.

🧪 Practice

1. Compute burn rates for a 99.9% SLO at observed error ratios of 0.05%, 0.5%, and 2%, and assign each to page, ticket, or ignore.
2. Write a multi-window burn-rate alert rule and test it against a replayed traffic spike.
3. Interview: Why is a CPU alert usually a bad page? (Hint: think about whether a user can tell the difference, and what fraction of high-CPU moments require immediate human action.)

Alert Fatigue and Noise Reduction

Alert fatigue is the predictable outcome of alerting on too much: responders stop reading alerts carefully, start acknowledging without investigating, and eventually miss the one that mattered. It is the most common way a well-monitored system still has a long outage, and it is a systems problem rather than a discipline problem — no amount of individual attentiveness survives forty pages a night.

The measurable symptoms are useful because they turn a vague complaint into a tracked number.

Metric	Healthy target	What a bad value means
Pages per on-call shift	0–2	Above 5, sleep and judgment degrade
Actionable rate	> 90%	Low means alerts are informational
Auto-resolving pages	Near zero	Thresholds too tight or windows short
Alerts with a runbook	100%	Tribal knowledge is a single point of failure
Mean time to acknowledge	Stable	Rising means trust is falling

# Audit your alerts against evidence rather than opinion. Run this monthly.
from collections import Counter

def alert_audit(alert_history):
    """alert_history: [{name, fired_at, resolved_at, acknowledged, action_taken}]"""
    stats = {}
    for a in alert_history:
        s = stats.setdefault(a["name"], Counter())
        s["fired"] += 1
        if a["action_taken"]:                       s["actionable"] += 1
        if a["resolved_at"] - a["fired_at"] < 300:  s["self_resolved"] += 1
        if not a["acknowledged"]:                   s["ignored"] += 1

    for name, s in sorted(stats.items(), key=lambda kv: -kv[1]["fired"]):
        actionable = s["actionable"] / s["fired"]
        print(f"{name:35} fired={s['fired']:4} actionable={actionable:.0%} "
              f"self_resolved={s['self_resolved']:3} ignored={s['ignored']:3}")
        if actionable < 0.5:
            print("   -> DELETE or downgrade: half the pages needed no action")
        if s["self_resolved"] / s["fired"] > 0.3:
            print("   -> widen the `for:` duration or raise the threshold")

The techniques that actually reduce volume, roughly in order of impact:

Delete alerts nobody acts on. This is the largest and most-resisted win. An alert that has fired forty times and produced zero actions is not a safety net; it is noise that hides real signals.

Alert on the symptom once, not on every cause. A database outage that pages the database alert, the connection-pool alert, the latency alert, the error-rate alert, and eight dependent services' alerts is one incident and should be one page. Inhibition rules encode that dependency explicitly.

# Alertmanager: suppress downstream noise while the root cause is firing.
inhibit_rules:
  - source_matchers: [ 'alertname = DatabaseDown', 'cluster = prod' ]
    target_matchers: [ 'severity =~ "page|warning"', 'cluster = prod' ]
    equal: ['cluster']
    # While the database is down, every dependent alert is a consequence,
    # not new information. One page, not fourteen.

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s          # collect related alerts before the first notification
  group_interval: 5m       # batch updates for an ongoing group
  repeat_interval: 4h      # do not re-page every 5 minutes for the same thing
  routes:
    - matchers: [ 'severity = page' ]
      receiver: pagerduty
    - matchers: [ 'severity = ticket' ]
      receiver: jira                 # never wakes anyone
    - matchers: [ 'severity = info' ]
      receiver: slack_channel        # visible, silent

Tier by urgency and route accordingly. Only "a user is being harmed now and a human must act" earns a page. Everything else is a ticket or a dashboard, and routing enforces the distinction that good intentions do not.

Make alerts self-healing where possible. If the response to an alert is always "restart the process" or "scale up", automate that and alert only when the automation fails. An alert whose runbook has one deterministic step is a script waiting to be written.

Group and deduplicate. Twenty instances failing the same check is one alert with a count, not twenty pages.

Finally, treat alert quality as ongoing work with an owner. Review pages in every retrospective, add a tuning action item whenever a page was not actionable, and give any team consistently exceeding its page budget explicit time to fix the cause — the same error-budget logic from 9.4, applied to human attention.

Key Takeaways

• Alert fatigue is a systems problem: past a handful of pages per shift, responders stop reading them and real signals get missed.
• Track actionable rate, self-resolving pages, and pages per shift; audit alerts against that evidence monthly.
• Deleting non-actionable alerts is the highest-impact fix and the most resisted.
• Use inhibition and grouping so one incident produces one page rather than a cascade of consequences.
• Route by urgency — page only for user-affecting, human-required action — and automate any response that is always the same.

🧪 Practice

1. Audit a month of alert history, compute the actionable rate per alert, and propose three deletions.
2. Write inhibition rules so a dependency outage suppresses its dependents' pages.
3. Interview: How do you convince a team to delete alerts they feel safer having? (Hint: think about what the alert actually protects against, and what the measurable cost of keeping it has been.)

On-Call and Incident Response

On-call is the practice of having someone responsible for responding when production breaks. It exists because automated recovery cannot handle novel failures, and it is worth designing deliberately because a badly run rotation burns people out faster than almost anything else in engineering — and burned-out responders make outages longer.

A healthy rotation has recognizable properties: enough people (six to eight, so each person is on call roughly one week in seven), a sustainable page rate (the previous topic), explicit compensation or time off in lieu, follow-the-sun scheduling where the organization spans time zones so nobody is routinely woken, a secondary who is paged if the primary does not respond, and the authority to act — a responder who must wake a manager to approve a rollback is not really responding.

Incident response itself benefits from explicit roles, because the failure mode of ad-hoc response is five engineers investigating the same hypothesis while nobody talks to customers.

 INCIDENT ROLES (scale them to the incident; one person may hold several)

  INCIDENT COMMANDER   owns the incident, not the fix. Decides, delegates,
                       keeps the timeline. Does NOT debug -- the moment the IC
                       is head-down in a terminal, nobody is coordinating.

  OPERATIONS LEAD      makes the changes: rollbacks, scaling, failover.
                       Announces every action before taking it.

  COMMUNICATIONS       status page, customer support, internal stakeholders.
                       Frees the IC from answering "any update?" every 5 min.

  SCRIBE               timestamps every observation, action, and decision.
                       This becomes the postmortem timeline (see 9.4).

 THE FLOW
  detect -> acknowledge -> assess severity -> assemble -> MITIGATE -> resolve
                                                             ^
                        restore service FIRST; understand it afterwards.
                        Root-causing a live outage is the classic time sink.

Severity levels matter because they set the response, and they should be defined by user impact rather than by technical alarm.

Severity	Definition	Response
SEV1	Major functionality down for many users	Page immediately; all hands; exec comms
SEV2	Significant degradation or a subset down	Page; dedicated responders
SEV3	Minor impact, workaround exists	Business hours; ticket
SEV4	No user impact; risk or cleanup	Backlog

# Mitigation before diagnosis, encoded as an ordered checklist. Under stress,
# people follow lists better than they recall principles.
MITIGATION_LADDER = [
    ("Was there a recent deploy?",  "roll back first, investigate after"),
    ("Is one region/AZ affected?",  "fail over traffic away from it"),
    ("Is a dependency failing?",    "enable the fallback or degrade the feature"),
    ("Is it load-related?",         "shed load, scale out, tighten rate limits"),
    ("Is a feature flag involved?", "disable the flag"),
]
# Every rung restores users without knowing the root cause. The temptation is
# to find the bug first; the correct order is to stop the bleeding first.

def incident_record(severity, detected_at):
    return {
        "severity": severity,
        "detected_at": detected_at,        # feeds MTTD
        "acknowledged_at": None,
        "mitigated_at": None,              # user impact ENDS here -- the number
                                           # that matters most to users
        "resolved_at": None,               # full repair; often much later
        "timeline": [],                    # (ts, actor, observation_or_action)
        "customer_impact": {"users_affected": None, "requests_failed": None},
    }
    # Note the separation of mitigated_at from resolved_at: conflating them
    # makes incidents look longer and hides how effective mitigation was.

Practices that consistently shorten incidents: declare early — the cost of declaring an incident that turns out minor is far lower than the cost of a delayed response; one communication channel so context is not split across five threads; say what you are about to do before you do it, so two people do not restart the same service simultaneously; hand off explicitly with a written state summary when a shift ends; and stop when tired — extended incidents need rotation, because fatigued responders cause secondary incidents.

Everything after mitigation — the timeline, the contributing conditions, the action items — belongs to the blameless postmortem process covered in 9.4, and the scribe's notes are what make that process cheap rather than an archaeology project.

Key Takeaways

• A sustainable rotation needs enough people, a low page rate, compensation, a secondary, and responders empowered to act without approval.
• Separate roles: the incident commander coordinates and does not debug; operations changes things; comms handles stakeholders; a scribe keeps the timeline.
• Mitigate before diagnosing — roll back, fail over, degrade, shed load — and root-cause afterwards.
• Define severity by user impact and let it drive the response.
• Track mitigated_at separately from resolved_at; declare early, communicate in one channel, announce actions before taking them, and hand off in writing.

🧪 Practice

1. Write a severity matrix for a system you know with a concrete example incident at each level.
2. Build a mitigation ladder of five actions that restore service without knowing the cause.
3. Interview: Why should the incident commander not be the person fixing the problem? (Hint: think about what stops happening the moment that person opens a terminal and starts debugging.)

<a id="123-deployment-and-release"></a>

12.3 Deployment and Release

Every outage that is not caused by hardware or traffic is caused by a change, and most changes are deploys. This subchapter covers the pipeline that gets code to production, the strategies for swapping running versions without dropping requests, the separation of deployment from release via flags, and the two things that must always work: rolling back, and changing a schema underneath a live system.

CI/CD Pipeline Design

Continuous integration means every change is merged and verified against the main line frequently — many times a day — rather than accumulating on long-lived branches. Continuous delivery means every commit that passes verification is deployable; continuous deployment means it is actually deployed automatically. The distinction matters: delivery is a technical capability, deployment is a policy decision on top of it.

The motivation is integration risk. Two developers working for three weeks on separate branches produce a merge whose conflicts are semantic as well as textual, and the bugs surface all at once with no way to isolate which change caused what. Small, frequent, automatically verified changes invert every part of that: each deploy carries one change, so a failure points at a single suspect.

 PIPELINE STAGES -- ordered by cost, cheapest failures first

  commit
    |
  [ lint + typecheck ]     seconds      catches trivia before humans look
    |
  [ unit tests ]           1-2 min      most of your coverage lives here
    |
  [ build artifact ]       2-5 min      build ONCE, promote the same bytes
    |                                   through every environment
  [ integration tests ]    5-10 min     real DB, real broker, in containers
    |
  [ security scans ]       2-5 min      dependencies (SCA), static analysis,
    |                                   secret scanning, image CVEs
  [ deploy to staging ]
    |
  [ smoke / e2e tests ]    5-15 min     a handful of critical user journeys
    |
  [ deploy to prod ]       progressive (canary, then full -- see below)
    |
  [ verify in prod ]       automated checks against live signals

  Fast feedback is the design goal: a 45-minute pipeline gets bypassed, and a
  bypassed pipeline provides no safety at all.

# The build-once, promote-many principle expressed in a pipeline.
name: deploy
on: { push: { branches: [main] } }

jobs:
  build:
    runs-on: ubuntu-latest
    outputs:
      digest: ${{ steps.push.outputs.digest }}      # the immutable identity
    steps:
      - uses: actions/checkout@v4
      - run: make lint test                          # fail fast, fail cheap
      - id: push
        run: |
          IMAGE=registry.example.com/orders:${GITHUB_SHA}
          docker build -t "$IMAGE" .
          docker push "$IMAGE"
          # Record the DIGEST, not the tag: tags are mutable, digests are not.
          echo "digest=$(docker inspect --format='{{index .RepoDigests 0}}' "$IMAGE")" \
            >> "$GITHUB_OUTPUT"

  staging:
    needs: build
    steps:
      - run: ./deploy.sh staging "${{ needs.build.outputs.digest }}"
      - run: ./smoke-tests.sh https://staging.example.com

  production:
    needs: staging
    environment: production          # gate: approvals and secrets scoped here
    steps:
      # EXACTLY the same bytes that passed staging. Rebuilding for production
      # means production runs something no test ever ran.
      - run: ./deploy.sh production "${{ needs.build.outputs.digest }}"
      - run: ./verify-slo.sh orders --window 10m --max-error-rate 0.01

The design properties that make a pipeline trustworthy:

• Build once, promote the artifact. Rebuilding per environment reintroduces the "works on staging" class of bug; the digest is the identity that travels.
• Fast enough to keep. Target under ten minutes to a deployable artifact; parallelize, cache dependencies, and shard slow test suites.
• Deterministic and reproducible. Pin dependency versions and base images so the same commit produces the same artifact next month.
• No manual steps in the path to production. Every manual step is a place to be inconsistent under pressure.
• The pipeline is the only path. If someone can kubectl apply by hand, the gates are advisory.
• Secrets scoped by environment, injected at deploy time, never in the repo (11.2).
• Every artifact traceable to a commit, and every deploy recorded — this is the "what changed?" answer your dashboards annotate with.

Two organizational notes. Trunk-based development with short-lived branches is what makes CI meaningful; long-lived branches reintroduce exactly the integration risk CI removes. And a red main branch is an emergency — if the pipeline is broken, nobody can deploy anything, including the fix for the outage you are about to have.

Key Takeaways

• CI verifies every change against the main line frequently; CD makes every passing commit deployable, and deploying automatically is a separate policy.
• Build the artifact once and promote the same digest through environments.
• Order stages cheapest-failure-first and keep the whole pipeline fast enough that nobody wants to bypass it.
• Make the pipeline the only path to production, with no manual steps and environment-scoped secrets.
• Trunk-based development with short-lived branches is what makes CI work; a broken main branch blocks even emergency fixes.

🧪 Practice

1. Map an existing pipeline's stages and their durations, then reorder or parallelize to cut total time by a third.
2. Convert a pipeline that rebuilds per environment into build-once-promote and describe the class of bug this eliminates.
3. Interview: Why is it important that the same artifact goes to staging and production? (Hint: think about what staging is supposed to prove, and whether it proves it if the bytes differ.)

Blue-Green Deployment

Blue-green deployment runs two complete production environments. One (blue) serves all traffic; the other (green) is idle. You deploy the new version to green, verify it in a real production environment with real infrastructure, then switch all traffic at once. The previous version stays running and untouched, so rollback is switching back.

The property this buys is a fast, complete, and rehearsed rollback. Most deployment risk is not "will the new version start?" but "will it misbehave under real traffic?", and blue-green answers that with a switch you can reverse in seconds rather than a redeploy that takes minutes.

 BEFORE                     CUTOVER                   AFTER (v1 kept warm)

  [ router ]                 [ router ]                 [ router ]
      | 100%                     | 100%                     | 100%
      v                          v                          v
  [ BLUE  v1 ] <- live       [ GREEN v2 ]              [ GREEN v2 ] <- live
  [ GREEN --- ] idle         [ BLUE  v1 ] kept warm    [ BLUE  v1 ] next target

  Rollback = point the router back at BLUE. Seconds, not a deploy.
  Cost      = two full environments during the overlap.

#!/usr/bin/env bash
set -euo pipefail

LIVE=$(kubectl get service orders -o jsonpath='{.spec.selector.slot}')   # blue|green
IDLE=$([[ "$LIVE" == "blue" ]] && echo green || echo blue)

echo "live=$LIVE  deploying to idle=$IDLE"
kubectl set image "deployment/orders-$IDLE" app="$IMAGE_DIGEST"
kubectl rollout status "deployment/orders-$IDLE" --timeout=5m

# Verify the idle environment BEFORE it takes traffic: it is real production
# infrastructure with real dependencies, which staging never quite is.
./smoke-tests.sh "http://orders-$IDLE.internal"

# Atomic cutover: one selector change moves 100% of traffic.
kubectl patch service orders -p "{\"spec\":{\"selector\":{\"slot\":\"$IDLE\"}}}"
echo "traffic now on $IDLE"

# Watch real signals before declaring success. If these fail, switch back --
# the old environment is still running and still warm.
if ! ./verify-slo.sh orders --window 5m --max-error-rate 0.01; then
  kubectl patch service orders -p "{\"spec\":{\"selector\":{\"slot\":\"$LIVE\"}}}"
  echo "ROLLED BACK to $LIVE"; exit 1
fi
# Keep the old environment for at least one incident window before reusing it.

The costs and constraints are specific. Double the infrastructure during overlap, which for a large service is a real bill — mitigated by keeping the idle environment scaled down and scaling it up just before cutover. Shared state is the hard part: both environments talk to the same database, so schema changes must be compatible with both versions simultaneously (see zero-downtime migrations below), and any in-memory or sticky session state is lost at cutover unless it lives in a shared store. And the switch is all or nothing: a fault that only appears at full production load hits every user at once, which is precisely what canary releases address.

Aspect	Blue-green	Rolling update
Versions live at once	Two environments, one serving	Both serving during the roll
Rollback speed	Seconds (switch back)	Minutes (roll back through)
Infrastructure cost	2x during overlap	~1x
Blast radius at cutover	100% at once	Incremental
Pre-cutover testing	Full production environment	Not possible
DB compatibility	Must support both versions	Must support both versions

Blue-green fits well where rollback speed matters most and infrastructure cost is acceptable — payment systems, anything with a strict change window. It is commonly combined with a canary: shift a small percentage to green first, then complete the cutover.

Key Takeaways

• Two full environments; deploy to the idle one, verify it in real production infrastructure, then switch all traffic.
• The main benefit is rollback in seconds by switching back to a still-running previous version.
• Costs roughly double infrastructure during overlap; scale the idle environment down between deploys.
• Shared state is the constraint: the database must support both versions, and in-process state is lost at cutover.
• Cutover is all-or-nothing, so combine it with a canary phase when load-related faults are a concern.

🧪 Practice

1. Write a cutover script that verifies the idle environment, switches, checks live signals, and reverts automatically on failure.
2. List everything in a system you know that would break at cutover because it holds in-process state.
3. Interview: What does blue-green not protect you against? (Hint: think about what both environments share, and what a rollback cannot undo.)

Canary Releases

A canary release exposes a new version to a small fraction of traffic first, watches the signals, and expands only if they hold. The name comes from canaries in coal mines: a small, sensitive early warning that costs little when it goes wrong.

The reasoning is statistical. Testing environments cannot reproduce production's traffic mix, data shapes, load, and dependency behaviour, so some faults appear only in production. If a fault is going to appear, you want it to appear in front of 1% of users for two minutes rather than 100% for twenty.

 PROGRESSIVE ROLLOUT

  t=0    1%   ├─┤ analyze 10 min  ──> pass ──┐
  t=10   5%   ├───┤ analyze 10 min ──> pass ─┤
  t=20  25%   ├─────────┤ analyze ──> pass ──┤
  t=40 100%   ├────────────────────────────┤ done
                                            |
                        any stage FAILS ----+--> abort: route 0% to canary,
                                                 keep the stable version

  Exposure at fault detection:  1% x 10 min  instead of  100% x however long
  it takes a human to notice a dashboard.

# The analysis is the hard part: comparing the canary against the baseline
# rather than against a fixed threshold, because both move with traffic.
def analyze_canary(canary, baseline, min_requests=1000):
    """Return (verdict, reason). Compare like with like, over the same window."""
    if canary.request_count < min_requests:
        return "wait", "insufficient traffic for a decision"

    # 1. Error rate: compare RATIOS, and require a meaningful relative increase
    #    so ordinary noise at low volume does not abort the rollout.
    if canary.error_rate > max(baseline.error_rate * 1.5, 0.005):
        return "fail", f"error rate {canary.error_rate:.3%} vs {baseline.error_rate:.3%}"

    # 2. Latency: p99, not mean -- a regression usually shows in the tail first.
    if canary.p99_ms > baseline.p99_ms * 1.2:
        return "fail", f"p99 {canary.p99_ms}ms vs {baseline.p99_ms}ms"

    # 3. Resource regressions that would only bite at full scale.
    if canary.memory_rss_mb > baseline.memory_rss_mb * 1.3:
        return "fail", "memory regression"

    # 4. Business signals catch what technical ones cannot: a version that
    #    returns 200 with an empty cart is technically healthy and commercially
    #    broken.
    if canary.checkout_conversion < baseline.checkout_conversion * 0.9:
        return "fail", "conversion drop"

    return "pass", "all comparisons within tolerance"

# Progressive delivery as configuration (Argo Rollouts / Flagger style).
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: orders }
spec:
  strategy:
    canary:
      steps:
      - setWeight: 1
      - pause: { duration: 10m }        # long enough to accumulate signal
      - analysis:                        # automated gate, no human required
          templates: [{ templateName: success-rate-and-latency }]
      - setWeight: 5
      - pause: { duration: 10m }
      - setWeight: 25
      - pause: { duration: 20m }
      - setWeight: 100
      trafficRouting:
        istio: { virtualService: { name: orders } }
      analysis:
        templates: [{ templateName: success-rate-and-latency }]
        startingStep: 1                 # run continuously from step 1 onward

Three details determine whether a canary actually catches anything. Traffic must be representative — if routing sends the canary only internal or only cached requests, it proves nothing; hash-based routing on user ID gives a consistent, representative slice. The window must be long enough to accumulate statistically meaningful volume, which for a low-traffic service can mean hours, and for a very low-traffic service means canarying is not viable without synthetic load. And compare against a concurrent baseline, not against yesterday, so traffic shifts and dependency weather affect both sides equally.

Canaries have blind spots worth naming: slow-burn problems (a memory leak that takes six hours), issues affecting only a rare user segment that the sample missed, and anything involving data written by the new version that the old version cannot read — which no traffic-splitting strategy can protect you from.

Key Takeaways

• Expose a small traffic fraction first and expand only if signals hold; this bounds the blast radius of production-only faults.
• Compare the canary against a concurrent baseline, not against fixed thresholds or yesterday's numbers.
• Gate on error ratio, tail latency, resource usage, and at least one business metric.
• Route representatively with a stable hash, and hold each stage long enough for statistical significance.
• Canaries miss slow-burn faults, rare segments, and forward-incompatible data writes.

🧪 Practice

1. Define the analysis criteria for a canary of a checkout service, including one business metric.
2. Compute how long a 1% canary must run to see 1,000 requests at your traffic level, and decide whether the plan is viable.
3. Interview: Your canary passed and the full rollout caused an incident. What could explain that? (Hint: think about what changes between 1% and 100% — load, cache behaviour, connection counts, and time.)

Rolling Updates

A rolling update replaces instances of the old version with the new one incrementally: take a few out of service, upgrade them, put them back, repeat. It is the default strategy in most orchestrators because it needs no extra infrastructure and no traffic-splitting layer — capacity is maintained by never removing too many instances at once.

The mechanism depends entirely on health checks. The orchestrator adds a new instance to the load balancer only once it reports ready, and it removes an old instance only after connections drain. Both halves must work, or a rolling update becomes a rolling outage.

 ROLLING UPDATE, maxSurge=1  maxUnavailable=0   (4 replicas)

  step 0   [v1][v1][v1][v1]                     capacity 4/4
  step 1   [v1][v1][v1][v1][v2*]  surge one     capacity 4/4  (*not ready yet)
  step 2   [v1][v1][v1]     [v2]  drain one     capacity 4/4
  step 3   [v1][v1][v2*]    [v2]                capacity 4/4
  ...
  step n        [v2][v2][v2][v2]                capacity 4/4

  maxUnavailable=0 guarantees capacity throughout, at the cost of one extra
  instance's resources and a slower roll.

  DURING the roll, v1 and v2 both serve traffic -- so they must be mutually
  compatible: same API contract, same database schema expectations, same
  message formats. This is the constraint people forget.

apiVersion: apps/v1
kind: Deployment
metadata: { name: orders }
spec:
  replicas: 6
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1              # at most 1 extra pod above `replicas`
      maxUnavailable: 0        # never drop below `replicas` ready -- no capacity dip
  minReadySeconds: 30          # a pod must stay healthy 30s before counting as
                               # available; without this, a pod that crashes at
                               # t=35s is replaced only after the roll finished
  progressDeadlineSeconds: 600 # mark the rollout failed rather than hanging forever
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: app
        image: registry.example.com/orders@sha256:...
        readinessProbe:                 # gates traffic
          httpGet: { path: /readyz, port: 8080 }
          periodSeconds: 5
          failureThreshold: 2
        lifecycle:
          preStop:
            exec:
              # Fail readiness, then wait for the load balancer to notice BEFORE
              # the process starts shutting down. Skipping this is the single
              # most common cause of 502s during a deploy.
              command: ["/bin/sh", "-c", "touch /tmp/shutting-down && sleep 15"]

# The application half of the contract (see also twelve-factor, 10.4).
import os, signal, threading

_draining = threading.Event()

def readyz():
    # Readiness must go false the moment shutdown begins, and must not depend
    # on downstream health -- a dependency blip should not empty your fleet.
    if _draining.is_set() or os.path.exists("/tmp/shutting-down"):
        return 503, "draining"
    return 200, "ok"

def on_sigterm(signum, frame):
    _draining.set()          # 1. stop advertising readiness
    server.stop_accepting()  # 2. no new connections
    server.await_inflight(timeout=45)   # 3. finish what is in flight
    os._exit(0)

signal.signal(signal.SIGTERM, on_sigterm)

The parameters express a trade. maxUnavailable: 0 with maxSurge: 1 keeps full capacity but rolls slowly and needs headroom for the extra instance; maxUnavailable: 25% rolls faster and accepts a capacity dip, which is fine if you have headroom and dangerous during peak. For a stateful workload, ordered one-at-a-time replacement is usually mandatory.

The property to keep in mind is that a rolling update is not atomic and not instantly reversible. Both versions serve simultaneously, so every change must be backward compatible for the duration; and rolling back means rolling forward to the previous version, which takes as long as the deploy did. When rollback speed is critical, blue-green or a canary with instant traffic shift is the better tool.

Key Takeaways

• Rolling updates replace instances incrementally, maintaining capacity without extra environments or traffic splitting.
• Both versions serve traffic during the roll, so every change must be backward compatible with the version it replaces.
• Readiness probes gate traffic and liveness probes restart — conflating them causes mass restarts (see 10.4).
• Drain correctly: fail readiness, wait for the load balancer to notice, then finish in-flight requests; skipping the wait causes deploy-time 502s.
• Rollback means rolling forward to the old version and takes as long as the deploy; use blue-green or canary when seconds matter.

🧪 Practice

1. Configure a rolling update that never dips below full capacity and explain the resource cost of that guarantee.
2. Add a preStop hook and graceful shutdown, then verify no request fails during a rolling restart under load.
3. Interview: Why do requests fail during a deploy even with readiness probes configured? (Hint: think about the delay between a pod being told to stop and the load balancer learning about it.)

Feature Flags

A feature flag is a runtime condition that decides whether a code path is active. Its architectural significance is that it separates deployment from release: code ships to production disabled, and a separate, instant, reversible decision turns it on. Those two events no longer have to be the same event, which changes what deployment risk means.

That separation solves several problems at once. Long-lived feature branches disappear, because incomplete work can be merged behind a disabled flag. Rollback for a feature becomes a config change rather than a deploy. Gradual rollout by user segment becomes possible without traffic-level routing. And experimentation (A/B testing) uses the same mechanism.

# Flags with targeting: the interesting part is not the boolean but the rules.
import hashlib

class FeatureFlags:
    def __init__(self, config_source):
        self.source = config_source     # polled or streamed; NOT a deploy artifact

    def enabled(self, flag: str, ctx: dict) -> bool:
        rule = self.source.get(flag)
        if rule is None or not rule.get("active"):
            return False                # unknown flag -> off. Default safe.

        for override in rule.get("overrides", []):      # explicit allow/deny first
            if ctx.get(override["attribute"]) in override["values"]:
                return override["enabled"]

        pct = rule.get("percentage", 0)
        if pct <= 0:   return False
        if pct >= 100: return True

        # Stable bucketing: the SAME user always lands the same way, so nobody
        # sees the feature flicker between requests. Salting with the flag name
        # keeps different flags' cohorts independent rather than correlated.
        key = f"{flag}:{ctx.get('user_id', 'anonymous')}"
        bucket = int(hashlib.sha256(key.encode()).hexdigest()[:8], 16) % 100
        return bucket < pct

flags = FeatureFlags(config_source)

def checkout(cart, user):
    ctx = {"user_id": user.id, "tier": user.tier, "country": user.country}
    if flags.enabled("new_pricing_engine", ctx):
        price = new_pricing(cart)
    else:
        price = legacy_pricing(cart)     # the old path stays until the flag dies
    return place_order(cart, price)

// Flag configuration, changed without deploying code.
{
  "new_pricing_engine": {
    "active": true,
    "percentage": 10,
    "overrides": [
      { "attribute": "tier",    "values": ["internal"], "enabled": true  },
      { "attribute": "country", "values": ["DE"],       "enabled": false }
    ],
    "owner": "pricing-team",
    "created": "2026-08-01",
    "expires": "2026-10-01",
    "type": "release"
  }
}

Flags have distinct types with different lifetimes, and conflating them is how codebases end up unmaintainable.

Type	Purpose	Lifetime	Removal
Release	Ship dark, roll out gradually	Days to weeks	Mandatory once at 100%
Experiment	A/B test a hypothesis	Weeks	Remove when the test ends
Operational	Kill switch, load shedding	Permanent	Keep, and exercise it
Permission	Entitlement by plan or tenant	Permanent	Not a flag — that is authz

The dominant risk is flag debt. Every flag doubles the number of code paths, and ten independent flags describe 1,024 possible configurations, of which you test perhaps three. The discipline that prevents this: give every flag an owner and an expiry date at creation, alert on flags past expiry, remove release flags immediately after full rollout, and never nest flags inside flags.

Three operational rules round it out. Default to off on failure — if the flag service is unreachable, take the old path, never the new one. Cache flag evaluations locally with background refresh, because a network call per flag check puts your flag provider on the critical path of every request. And log flag state with requests, so an incident investigation can tell which cohort a failing request was in — otherwise "it works for me" becomes unresolvable.

Key Takeaways

• Feature flags separate deployment from release: ship disabled, enable instantly, disable instantly.
• Bucket users with a stable hash salted by flag name so experience is consistent and cohorts are independent.
• Distinguish release, experiment, and operational flags — the first two must be removed, the third is permanent.
• Flag debt is exponential: assign an owner and expiry at creation and enforce removal.
• Fail closed to the old path, evaluate from a local cache, and log flag state alongside requests.

🧪 Practice

1. Implement percentage rollout with stable per-user bucketing and prove the same user gets the same answer across restarts.
2. Audit a codebase for flags older than 90 days and write the removal plan for one of them.
3. Interview: What happens to your system if the feature flag service goes down? (Hint: think about defaults, caching, and whether "no flags" is a state your code handles.)

Rollback Strategies

Rollback is the mitigation that resolves most incidents, and the ability to perform it quickly and confidently is worth more than any amount of pre-deploy testing. The reason is arithmetic: testing reduces the probability of a bad deploy, while rollback reduces the duration of every bad deploy that still gets through — and duration is the term that dominates user impact.

The discipline is to roll back first and investigate afterwards. The instinct to understand the problem before reverting is natural and wrong: every minute spent diagnosing is a minute of user impact, and the diagnosis is easier once the system is stable and you can reproduce the fault off the critical path.

 WHAT CAN AND CANNOT BE ROLLED BACK

  EASY: stateless code                 revert the artifact
  EASY: configuration                  revert the config version
  EASY: feature flag                   flip it off (seconds, no deploy)

  HARD: additive schema change         old code ignores new columns -- usually fine
  HARD: data written in a new format   old code may not be able to read it
  IMPOSSIBLE: destructive migration    a dropped column has no undo
  IMPOSSIBLE: side effects already
              emitted                  sent emails, charged cards, published
                                       events that consumers already processed

  The design consequence: keep changes ROLLBACK-SAFE by construction --
  expand/contract migrations, backward-compatible formats, and idempotent
  side effects behind flags.

# Rollback decision as a checklist, because under stress people follow lists.
def rollback_plan(change):
    steps = []
    if change.feature_flagged:
        steps.append(("flip_flag_off", "seconds", "no deploy needed"))
        return steps                       # fastest possible path: stop here

    if change.config_only:
        steps.append(("revert_config", "under a minute", "restart may be needed"))
        return steps

    steps.append(("deploy_previous_digest", "2-10 min", "the standard path"))

    if change.schema_migration:
        # THE question, and it must be answerable before the deploy, not after.
        if change.migration_is_expand_only:
            steps.append(("leave_schema_alone", "-",
                          "old code tolerates the new column"))
        else:
            steps.append(("MANUAL", "unknown",
                          "destructive migration: restore or forward-fix"))
    if change.emits_new_event_format:
        steps.append(("check_consumers", "-",
                      "consumers may already have processed new-format events"))
    return steps

# Automated rollback: the fastest possible response is one that needs no human.
def watch_and_rollback(deploy, signals, window_s=600, threshold=1.5):
    baseline = signals.baseline_error_rate(deploy.service)
    for _ in range(window_s // 30):
        observed = signals.error_rate(deploy.service, window="2m")
        if observed > max(baseline * threshold, 0.01):
            deploy.rollback()                     # act first
            alert(f"auto-rolled back {deploy.version}: "
                  f"error rate {observed:.2%} vs baseline {baseline:.2%}")
            return "rolled_back"
        sleep(30)
    return "healthy"

What makes rollback reliable rather than theoretical:

• Keep the previous artifact deployable. Immutable, digest-addressed images and retained previous versions; if rollback requires rebuilding from an old commit, it is not a rollback.
• Version configuration too. A config change is a deploy and needs the same revert path.
• Rehearse it. A rollback path first exercised during an incident is a gamble; roll back deliberately in staging, and occasionally in production.
• Automate the trigger. Signal-based automatic rollback beats a human noticing by minutes, which is the whole game.
• Know the point of no return. Every change should be labelled before deploy with whether it is reversible, and irreversible ones get extra scrutiny, smaller batches, and a forward-fix plan.

Forward fix is the alternative when rollback is impossible — ship a corrective change instead. It is strictly worse when rollback is available, because it requires writing correct code under time pressure, but it is the only option once data has been transformed destructively or side effects have escaped. Planning which of the two applies is part of designing the change, not part of the incident.

Key Takeaways

• Rollback reduces the duration of every bad deploy, which matters more than reducing the probability of one.
• Roll back first, diagnose afterwards — diagnosis is easier and cheaper once the system is stable.
• Feature flags give the fastest rollback; artifact reverts are next; destructive migrations and emitted side effects may not be reversible at all.
• Keep previous artifacts deployable and versioned, and version configuration the same way.
• Rehearse rollbacks, automate the trigger on live signals, and label every change with whether it is reversible before it ships.

🧪 Practice

1. Classify the last ten changes in a repository as reversible, partially reversible, or irreversible, and say what would be needed for each.
2. Implement automatic rollback triggered by an error-rate comparison against a pre-deploy baseline.
3. Interview: When is forward-fixing the right choice over rolling back? (Hint: think about what state the old version can no longer read, and what has already left the system.)

Zero-Downtime Migrations

A schema change is the one deploy that cannot be atomic: the database is shared, and during any rolling update or blue-green overlap, two versions of the application run against one schema. Any migration that requires them to disagree about the schema causes errors, and any migration that locks a large table causes an outage regardless of application versions.

The technique that solves this is expand/contract (also called parallel-change): never change a thing in place; add the new thing, migrate to it, and only remove the old thing once nothing references it. Each step is individually backward compatible, so every intermediate state is deployable and rollback-safe.

 RENAMING A COLUMN, THE ONLY SAFE WAY  (username -> handle)

  step 1  EXPAND     add `handle` (nullable, no default backfill)
                     deploy N/A -- schema only, old code unaffected

  step 2  DUAL WRITE deploy code that writes BOTH columns, reads `username`
                     old code still running is fine: it ignores `handle`

  step 3  BACKFILL   copy username -> handle in batches, throttled
                     no lock held for long; safe to pause and resume

  step 4  SWITCH     deploy code that reads `handle`, still writes both
                     rollback is safe: `username` is still current

  step 5  STOP WRITE deploy code that only uses `handle`
                     `username` now stale but present -- still reversible-ish

  step 6  CONTRACT   drop `username` after a soak period
                     THIS is the irreversible step; do it last and separately

  Six deploys instead of one. That is the price of never having downtime and
  never having an unrecoverable moment.

# Step 2 and 4 in application code: the dual-write window.
def update_profile(user_id, new_handle, db, flags):
    # Write both columns while any reader of either might still be running.
    db.execute("UPDATE users SET username = %s, handle = %s WHERE id = %s",
               (new_handle, new_handle, user_id))

def read_profile(user_id, db, flags):
    row = db.query("SELECT username, handle FROM users WHERE id = %s", (user_id,))
    # The read switch is a FLAG, not a deploy: it flips in seconds and reverts
    # in seconds, which is exactly what you want for the riskiest step.
    if flags.enabled("read_handle_column", {"user_id": user_id}):
        return row["handle"] or row["username"]     # fall back while backfilling
    return row["username"]

-- Step 1: expand. Nullable, no default, no rewrite of existing rows.
ALTER TABLE users ADD COLUMN handle varchar(50);
-- A NOT NULL column WITH a default rewrites the whole table on older engines
-- and takes a long ACCESS EXCLUSIVE lock. Add nullable, backfill, then
-- constrain -- in that order.

-- Step 3: backfill in bounded batches so no single statement holds a long lock
-- and replication lag stays manageable.
UPDATE users SET handle = username
 WHERE handle IS NULL
   AND id IN (SELECT id FROM users WHERE handle IS NULL ORDER BY id LIMIT 5000);
-- Loop this with a pause between batches; monitor replica lag between runs.

-- Index creation must not block writes.
CREATE INDEX CONCURRENTLY idx_users_handle ON users (handle);
-- CONCURRENTLY is slower and can leave an INVALID index if it fails --
-- check for that and drop/retry rather than assuming success.

-- Adding a constraint without a long validation lock: two steps.
ALTER TABLE users ADD CONSTRAINT handle_not_null CHECK (handle IS NOT NULL) NOT VALID;
ALTER TABLE users VALIDATE CONSTRAINT handle_not_null;   -- takes a weaker lock

-- Step 6: contract, much later, as its own change.
ALTER TABLE users DROP COLUMN username;

Operation	Safe?	Note
Add nullable column	Yes	Metadata-only on modern engines
Add column with NOT NULL DEFAULT	Engine-dependent	May rewrite the table; check your version
Drop column	Irreversible	Contract step only, after a soak
Rename column or table	Never in place	Use expand/contract
Add index	Yes, concurrently	Non-concurrent blocks writes
Change column type	Usually not	New column + backfill + switch
Add foreign key	Careful	Add NOT VALID, then validate separately

Two further rules. Migrations run separately from application deploys, so a slow migration does not block a rollout and a rollback does not try to reverse a schema change. And every migration needs a tested rollback path or an explicit statement that it has none — for destructive steps, the honest answer is "restore from backup", which is only acceptable if you have rehearsed that restore (9.4).

Key Takeaways

• Two application versions always share one schema during a deploy, so every migration must be compatible with both.
• Use expand/contract: add, dual-write, backfill, switch reads, stop writing the old, then drop — six safe steps instead of one unsafe one.
• Never rename or retype in place; add a new column and migrate to it.
• Backfill in throttled batches, build indexes concurrently, and add constraints as NOT VALID then validate.
• Run migrations separately from deploys, and put the read-switch behind a feature flag so the riskiest step is instantly reversible.

🧪 Practice

1. Write the full six-step plan to rename a column on a 200-million-row table, including where each application deploy lands.
2. Implement a throttled backfill that monitors replica lag and pauses when it exceeds a threshold.
3. Interview: Why can you not just run ALTER TABLE ... RENAME COLUMN during a deploy? (Hint: think about which application versions are running at that exact moment and what each of them expects to find.)

<a id="124-infrastructure-management"></a>

12.4 Infrastructure Management

Infrastructure that exists only as the accumulated result of past manual changes cannot be reasoned about, reproduced, or safely modified. This subchapter covers defining infrastructure as code, replacing rather than patching servers, managing the configuration that varies between environments, keeping those environments close enough that testing means something, and writing down the operational knowledge that would otherwise live in one person's head.

Infrastructure as Code

Infrastructure as code (IaC) means defining servers, networks, databases, and permissions in version-controlled files, and having a tool make reality match those files. The alternative — clicking through a console or running ad-hoc commands — produces infrastructure nobody can describe: the configuration is whatever the last person did, there is no record of why, and reproducing the environment is archaeology.

The properties this buys are the same ones version control brought to application code, and they are worth naming individually because each solves a specific operational pain: reproducibility (build an identical environment on demand), reviewability (infrastructure changes go through pull requests), auditability (git history says who changed what and why), disaster recovery (rebuild from code rather than from memory), and testability (validate a change before applying it).

 IMPERATIVE                              DECLARATIVE

  "run these commands in order"           "this is the desired end state"

  create_vpc()                            resource "aws_vpc" "main" {
  create_subnet(vpc)                        cidr_block = "10.0.0.0/16"
  create_instance(subnet)                 }
  attach_security_group(instance)
                                          The tool computes the DIFF between
  Running it twice: errors, or            desired and actual, and applies only
  duplicate resources. You must           what is missing. Running it twice
  encode every current state.             changes nothing the second time.
                                          -> idempotent by construction

# Terraform: declarative, with the plan/apply cycle as the safety mechanism.
terraform {
  required_version = "~> 1.7"
  backend "s3" {
    bucket         = "example-tfstate"
    key            = "prod/orders/terraform.tfstate"
    dynamodb_table = "tfstate-locks"   # state locking: two applies cannot race
    encrypt        = true
  }
}

variable "environment" { type = string }
variable "instance_count" {
  type = number
  validation {                          # fail at plan time, not at 3 a.m.
    condition     = var.instance_count >= 2
    error_message = "At least two instances are required for availability."
  }
}

resource "aws_db_instance" "orders" {
  identifier     = "orders-${var.environment}"
  engine         = "postgres"
  engine_version = "16.3"                   # pin: 'latest' is not reproducible
  instance_class = var.environment == "production" ? "db.r6g.xlarge" : "db.t4g.small"

  backup_retention_period = 30
  deletion_protection     = var.environment == "production"
  storage_encrypted       = true            # see 11.2

  lifecycle {
    prevent_destroy = true                  # a typo must not delete the database
    ignore_changes  = [engine_version]      # let managed minor upgrades happen
  }

  tags = {
    Environment = var.environment
    Owner       = "orders-team"
    ManagedBy   = "terraform"               # marks what humans must not touch
  }
}

output "db_endpoint" { value = aws_db_instance.orders.endpoint }

The state file is the concept that surprises people. Terraform records what it believes exists so it can compute diffs; that state must be stored remotely, locked during operations, encrypted (it contains sensitive values), and backed up. State corruption or divergence — usually caused by someone making a manual change — is the most common operational problem with IaC.

Which brings up the discipline that determines whether IaC works at all: changes must go through code. A single manual console change creates drift, and the next apply will either revert it (surprising whoever made it) or fail. Detecting drift continuously and treating it as an incident is what keeps the files authoritative rather than aspirational.

# Drift detection and policy enforcement in CI.
name: infrastructure
on:
  pull_request:
  schedule: [{ cron: "0 */6 * * *" }]     # detect drift even without a PR

jobs:
  plan:
    steps:
      - run: terraform init
      - run: terraform validate
      - run: terraform plan -detailed-exitcode -out=tfplan
        # exit 0 = no changes, 2 = changes pending. On the schedule, exit 2
        # means someone changed something outside of code -> alert.
        continue-on-error: true
      - run: tfsec . && checkov -d .       # policy: public buckets, open SGs,
                                           # unencrypted volumes, wide IAM
      - run: infracost breakdown --path tfplan   # cost delta in the PR comment

Two design habits keep IaC maintainable at scale: modularize repeated patterns (a "service" module used by twenty services beats twenty copies), and separate state by blast radius — networking, data stores, and applications in different state files, so an application change cannot plan a VPC deletion.

Key Takeaways

• Define infrastructure in version-controlled files and let a tool reconcile reality to them; the console is for reading, not changing.
• Declarative definitions are idempotent by construction: the tool diffs desired against actual.
• Remote, locked, encrypted state is mandatory, and drift from manual changes is the main operational hazard.
• Review infrastructure changes as pull requests with automated policy, security, and cost checks on the plan.
• Modularize repeated patterns and split state by blast radius so a small change cannot plan a large deletion.

🧪 Practice

1. Define a database and its security group in Terraform with validation, deletion protection, and required tags.
2. Add a scheduled drift-detection job and deliberately create drift to confirm it is caught.
3. Interview: What goes wrong when someone changes a resource in the console? (Hint: think about what the tool believes exists and what the next apply will try to do about the difference.)

Immutable Infrastructure

Immutable infrastructure means servers are never modified after deployment. To change anything — a code version, a package, a kernel setting — you build a new image and replace the instance. The alternative, mutable infrastructure, patches running servers in place with configuration management or manual commands.

The problem immutability solves is configuration drift. Servers patched in place diverge: one missed a package update during a network blip, another has a manual fix from an incident two years ago, a third was built from an older base image. Eventually you have "snowflake" servers whose exact state nobody knows, which is why a bug reproduces on three hosts out of twenty and why rebuilding a failed host does not restore the same behaviour.

 MUTABLE                                  IMMUTABLE

  server-1  base + patch A + patch B      build IMAGE v42 (once, in CI)
  server-2  base + patch A                     |
  server-3  base + patch A + hotfix        +---+---+---+
            + manual change (2024)         v   v   v   v
                                        [v42][v42][v42][v42]   all identical
  divergent, unknowable state
  "works on server-2"                    change anything -> build v43,
  rebuild != same state                  replace instances, discard v42

                                         rollback = redeploy v42's image

# The image IS the unit of change. Everything variable comes in at run time.
FROM golang:1.22 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -ldflags="-X main.version=${VERSION}" -o /out/app ./cmd

FROM gcr.io/distroless/static:nonroot
COPY --from=build /out/app /app
USER nonroot:nonroot
ENTRYPOINT ["/app"]
# No SSH server, no package manager, no shell. You cannot log in and "just fix
# it", which is the point: the absence of a repair path forces replacement and
# keeps every instance identical.

# Immutability is a property you can assert, not just intend.
def verify_fleet_uniformity(instances):
    """Every instance in a group must run the exact same image digest."""
    digests = {i.id: i.image_digest for i in instances}
    unique = set(digests.values())
    if len(unique) > 1:
        # During a deploy this is expected and transient; outside a deploy it
        # means something was changed out of band, or a rollout stalled.
        return False, f"{len(unique)} distinct images: {unique}"
    return True, digests

# The operational rule that follows: no SSH into production. If you need to
# inspect a host, you need better telemetry (12.1) -- and if you need to fix
# a host, you need a new image. An instance you logged into is no longer
# identical to its siblings and should be replaced, not returned to service.

The consequences are worth being explicit about, because immutability changes how several ordinary tasks work:

• Patching becomes rebuilding the base image and rolling the fleet, so image builds must be fast and automated or security patching becomes slow.
• Debugging cannot rely on logging into a box; it relies on telemetry, and where deep inspection is needed, on removing an instance from service and quarantining it for analysis.
• State must live elsewhere — databases, object storage, caches — because an instance can be destroyed at any moment (the twelve-factor rule from 10.4).
• Boot time matters more, since scaling and replacement both wait on it; bake dependencies into the image rather than installing at startup.
• Image sprawl needs lifecycle management: retention policies, vulnerability rescanning of images still in use, and a promotion path from build to production-approved.

The payoff is that deployment, rollback, scaling, and disaster recovery all become the same operation — start instances from a known image — which is a substantial reduction in the number of distinct procedures a team must maintain and rehearse.

Key Takeaways

• Never modify running servers; build a new image and replace them.
• This eliminates configuration drift and snowflake servers, so any instance is interchangeable with any other.
• Deployment, rollback, scaling, and recovery collapse into one operation: launch from a known image.
• Removing shells and package managers from images enforces the discipline that convention alone will not.
• Requires state to live outside the instance, fast automated image builds for patching, and image lifecycle management.

🧪 Practice

1. Build a minimal image with no shell and describe how you would diagnose a memory issue in it.
2. Write a check that alerts when instances in one group run different image digests outside a deploy window.
3. Interview: How do you apply an urgent security patch under immutable infrastructure? (Hint: think about what must already be automated for this to take minutes rather than days.)

Configuration Management

Configuration is everything that varies between deployments of the same artifact: endpoints, credentials, timeouts, pool sizes, feature toggles, log levels. It needs managing because it changes independently of code, changes more often, and causes a large share of incidents — a wrong timeout or a bad connection string takes a system down as surely as a bad deploy, usually faster and with less review.

The foundational rule (10.4) is configuration lives in the environment, not in the artifact, so one image runs everywhere. Beyond that, the design questions are where configuration comes from, how it is validated, and whether it can change without a restart.

 PRECEDENCE, lowest to highest -- each layer overrides the one below

   defaults in code           safe, working values; the app runs with nothing else
        ^
   config file / ConfigMap    per-environment settings, in version control
        ^
   environment variables      per-deployment overrides
        ^
   secret store               credentials, injected separately (11.2)
        ^
   runtime overrides          feature flags, dynamic tuning -- no restart

 Later layers override earlier ones. Every value must be traceable to the
 layer it came from, or debugging "why is the timeout 30s here?" is guesswork.

from pydantic_settings import BaseSettings
from pydantic import Field, PostgresDsn, field_validator

class Settings(BaseSettings):
    """Validated at startup. An invalid config must fail immediately and
    loudly, not surface as a mysterious timeout under load three hours later."""

    database_url: PostgresDsn                              # required; no default
    db_pool_size: int = Field(default=10, ge=1, le=100)    # bounded
    http_timeout_s: float = Field(default=2.0, gt=0, le=30)
    log_level: str = Field(default="info", pattern="^(debug|info|warn|error)$")
    environment: str = Field(pattern="^(development|staging|production)$")
    cache_ttl_s: int = Field(default=300, ge=0)

    @field_validator("db_pool_size")
    @classmethod
    def pool_must_suit_environment(cls, v, info):
        # Cross-field validation catches the combinations that individually
        # look fine: 200 connections per instance x 40 instances exhausts the
        # database's connection limit long before anyone notices.
        if info.data.get("environment") == "production" and v < 5:
            raise ValueError("production requires a pool of at least 5")
        return v

    model_config = {"env_prefix": "APP_", "extra": "forbid"}
    # extra="forbid": a typo in APP_DB_POOL_SIZE_ fails at boot instead of
    # silently leaving the default in place.

settings = Settings()      # raises at import time if anything is wrong

# Configuration as versioned data, reviewed like code.
apiVersion: v1
kind: ConfigMap
metadata:
  name: orders-config
  annotations:
    # Change the annotation on every update so pods roll and actually pick it
    # up -- mounted ConfigMaps update lazily, and env-var ones never do.
    config-hash: "sha256:9f2b8c..."
data:
  APP_DB_POOL_SIZE: "20"
  APP_HTTP_TIMEOUT_S: "2.0"
  APP_LOG_LEVEL: "info"
  APP_CACHE_TTL_S: "300"
---

# Secrets come from a different object with different access control (11.2).
apiVersion: v1
kind: Secret
metadata: { name: orders-secrets }
stringData:
  APP_DATABASE_URL: "postgres://..."

The practices that prevent configuration incidents:

• Validate at startup, fail fast. A service that boots with an invalid configuration and fails later is much harder to diagnose than one that refuses to start.
• Version and review configuration changes through the same pipeline as code. Most organizations gate code review far more strictly than config review, and the incident data does not justify the asymmetry.
• Roll out config changes progressively. A bad value applied to the whole fleet simultaneously is a self-inflicted outage; treat it like a deploy.
• Never put secrets in configuration files — separate store, separate access control, separate audit.
• Make dynamic config explicit. Values that change without restart (flags, tuning knobs) need bounds, defaults on fetch failure, and change logging.
• Log effective configuration at startup, with secrets redacted, so an incident responder can see what the process actually believes.

Configuration type	Change frequency	Mechanism	Restart?
Build-time constants	Per release	Baked into the artifact	Yes
Environment settings	Rarely	ConfigMap / env vars	Usually
Secrets	On rotation	Secret store injection	Depends
Feature flags	Continuously	Flag service	No
Tuning parameters	Occasionally	Dynamic config service	No

Key Takeaways

• Configuration lives in the environment so one artifact runs everywhere; layer precedence must be explicit and traceable.
• Validate at startup with types, bounds, and cross-field rules, and refuse to start on invalid values.
• Review and version configuration changes as strictly as code, and roll them out progressively — they cause a large share of incidents.
• Keep secrets in a separate store with separate access control.
• Log effective configuration at boot with secrets redacted, and give dynamic values bounds plus a safe default on fetch failure.

🧪 Practice

1. Add startup validation to a service with bounds on at least four settings and one cross-field rule.
2. Design the precedence order for your configuration sources and write a function that reports which layer supplied each effective value.
3. Interview: Why do configuration changes cause so many incidents relative to the review they receive? (Hint: compare the review, testing, and rollout process a config change gets with what a code change gets.)

Environment Parity

Environment parity is the degree to which development, staging, and production resemble each other. It matters because the entire value of testing before production rests on the assumption that behaviour transfers — and every difference between environments is a place that assumption fails silently.

The classic divergences are familiar because they are all locally convenient: SQLite in development and Postgres in production, an in-memory queue instead of Kafka, one instance instead of forty, a 50-row dataset instead of 50 million, mocked third parties, and no TLS. Each saves setup time and each hides a class of bug — concurrency, connection limits, query plans on real data volumes, serialization formats, certificate handling.

 THE PARITY GAP AND WHAT HIDES IN IT

  dimension        dev            staging          production
  ---------------  -------------  ---------------  -----------------
  database         SQLite         Postgres 16      Postgres 16      <- SQL dialect,
  data volume      50 rows        10k rows         50M rows            query plans
  instances        1              2                40               <- concurrency,
  queue            in-memory      Kafka (1 broker) Kafka (3 brokers)    leader election
  third parties    mocked         sandbox          live             <- rate limits,
  TLS              off            on               on                  timeouts
  latency to DB    0.1 ms         1 ms             3 ms + failover

  Bugs that ONLY appear at the right-hand column: lock contention, index
  choice at scale, partial failure, clock skew, connection exhaustion.

# Docker Compose for local development: same engines, same versions, same
# protocols as production -- scaled down, not substituted.
services:
  postgres:
    image: postgres:16.3          # the exact production version, pinned
    environment: { POSTGRES_PASSWORD: local }
    ports: ["5432:5432"]
  redis:
    image: redis:7.2
  kafka:
    image: confluentinc/cp-kafka:7.6.0
  localstack:
    image: localstack/localstack:3   # S3/SQS with the real API surface
    environment: { SERVICES: s3,sqs }
  app:
    build: .
    environment:
      APP_DATABASE_URL: postgres://postgres:local@postgres:5432/app
      APP_KAFKA_BROKERS: kafka:9092
    depends_on: [postgres, redis, kafka, localstack]
# The differences that REMAIN are deliberate and known: single broker, tiny
# dataset, no TLS locally. Write them down rather than discovering them.

Complete parity is neither achievable nor desirable — nobody should run a 40-node production replica per developer. The workable strategy is to be exact about kinds and versions, approximate about scale, and explicit about what differs:

• Same backing service types and versions everywhere. This is the highest-value rule and the cheapest to follow with containers.
• Same deployment mechanism. If production uses Kubernetes and staging uses a docker-compose file, staging does not test your manifests, probes, or resource limits.
• Representative data volume in at least one pre-production environment, since query plans and index choices change with cardinality. Use anonymized or synthetic data (11.2), never a production copy.
• Real dependency behaviour where it matters — sandbox APIs over mocks for integration testing, because mocks encode your assumptions rather than the vendor's behaviour.
• Document the known gaps and treat each one as a risk with an owner.

Two techniques narrow the gap further. Ephemeral preview environments created per pull request from the same IaC as production give a real environment per change and are discarded afterwards. And testing in production — canaries, feature flags, shadow traffic — accepts that some behaviour is only observable with real traffic, and makes that observation safe rather than pretending staging covered it.

Key Takeaways

• Every environment difference is a place where pre-production testing silently stops predicting production behaviour.
• Match backing service types and versions exactly; approximate scale deliberately.
• Use the same deployment mechanism in staging as in production, or you are not testing your deployment configuration.
• Keep at least one environment with representative data volume, using anonymized or synthetic data.
• Full parity is impossible: write down the remaining gaps, and use preview environments plus canaries and flags to cover what staging cannot.

🧪 Practice

1. Build the parity table for a system you work on and mark each difference as acceptable or risky.
2. Create a local compose stack matching production service versions exactly, and list what still differs.
3. Interview: A bug appears only in production. What differences would you investigate first? (Hint: think about scale, concurrency, data shape, and which dependencies are real versus mocked.)

Runbooks and Operational Documentation

A runbook is a document telling a responder exactly how to handle a specific operational situation. It exists because incidents happen at 3 a.m. to whoever is on call, not to the person who built the system — and because the knowledge required is otherwise held in a few people's heads, which makes those people a single point of failure and guarantees slow responses whenever they are asleep, on holiday, or gone.

The distinction from general documentation is that a runbook is procedural and situational: it starts from a symptom or an alert, gives concrete commands, and leads to a decision. Architecture documents explain how the system works; runbooks explain what to do right now.

# Runbook: OrdersHighErrorRate

**Alert**: `OrdersErrorBudgetFastBurn`
**Severity**: SEV2 (SEV1 if checkout success < 90%)
**Owner**: orders-team   **Last verified**: 2026-08-12 (game day)

## Impact
Users see failed checkouts. Revenue impact is roughly $4k per minute at peak.

## Quick assessment (2 minutes)
1. Dashboard: https://grafana.example.com/d/orders-health
2. Recent deploys: `kubectl rollout history deployment/orders -n prod`
3. Dependency status: https://status.example.com/internal

## Decision tree
- Deploy in the last 30 minutes?
  -> **Roll back first**: `./scripts/rollback.sh orders`   (~2 min to effect)
- Errors concentrated in one AZ?
  -> Fail over: `./scripts/drain-az.sh <az>`
- `payments` dependency failing?
  -> Enable degraded mode: `./scripts/flag.sh set checkout_offline_queue true`
     Orders are queued and settled asynchronously. Expect support tickets.
- Database connection errors?
  -> Check pool saturation panel; if exhausted, scale down non-critical
     consumers first: `./scripts/scale.sh reporting 0`

## Escalation
- 15 min without mitigation -> page orders-team secondary
- 30 min or revenue impact > $100k -> page engineering director, open SEV1

## After mitigation
Open a postmortem (see the blameless process) and attach the incident timeline.

## Verification
This runbook's commands were last executed successfully during the 2026-08-12
game day. If any command fails, fix the runbook as part of the incident.

# The best runbook step is one that has become a script; the best script is one
# that has become automation. Track where each runbook sits on that ladder.
RUNBOOK_MATURITY = {
    "manual_prose":  0,   # "check the logs and see"
    "exact_commands": 1,  # copy-pasteable, tested
    "single_script": 2,   # one command does the whole procedure
    "automated":     3,   # runs without a human; alerts only on failure
}

def audit_runbooks(runbooks, alert_history):
    for rb in runbooks:
        fires = sum(1 for a in alert_history if a["name"] == rb["alert"])
        stale_days = (today() - rb["last_verified"]).days
        if fires > 10 and rb["maturity"] < 2:
            print(f"{rb['alert']}: fired {fires}x, still manual -> script it")
        if stale_days > 180:
            print(f"{rb['alert']}: unverified for {stale_days} days -> game day it")
        if rb["maturity"] == 3:
            print(f"{rb['alert']}: automated -> should this still page a human?")

What separates useful runbooks from documentation nobody reads:

• One runbook per alert, linked from the alert itself. A responder should never have to search.
• Start with impact and assessment, so the responder knows the stakes and can triage before acting.
• Concrete, copy-pasteable commands with expected output — not "restart the service" but the exact command and how to tell it worked.
• Decision trees over prose. Under stress people follow branches; they do not parse paragraphs.
• Include the escalation path and the criteria for using it, so escalation is a decision rather than an act of desperation.
• Verify by execution. A runbook is only trustworthy if someone has recently run it — game days (9.4) are the mechanism, and every incident that finds a wrong step should fix it before closing.

The broader documentation set that pays for itself: architecture decision records (chapter 1) for why things are the way they are, service catalogues mapping each service to its owner, dependencies, SLOs, and on-call rotation, and on-boarding documents that a new engineer follows to their first deploy. All of it decays, which is why the only reliable pattern is documentation kept next to the code, updated in the same pull request, and validated by someone actually using it.

Key Takeaways

• Runbooks turn one person's knowledge into a procedure anyone on call can follow; they are situational and procedural, not explanatory.
• Link a runbook from every alert, and lead with impact, assessment, and a decision tree rather than prose.
• Give exact commands with expected output, plus explicit escalation criteria.
• Track maturity: manual steps should become scripts, and frequently used scripts should become automation that pages only on failure.
• Runbooks decay — verify them in game days and fix them during the incident that exposed the error.

🧪 Practice

1. Write a runbook for an alert on a system you know, including a decision tree and escalation criteria.
2. Take a manual runbook step and convert it into a tested script invoked by one command.
3. Interview: How do you keep runbooks from going stale? (Hint: think about what forces someone to execute them on a schedule, and who fixes them when a step fails.)

---

<a id="13-performance-and-cost-engineering"></a>