15. Productionizing Models and MLOps

A model that exists only in a notebook has produced no value: it has produced a claim about value. This chapter covers the machinery that turns a trained artifact into a system other people depend on -- how predictions reach their consumers, what infrastructure keeps training reproducible and deployment repeatable, how you detect that a model has quietly stopped working, and what practices keep a team's code and models reviewable years later. The recurring theme is that machine learning code is a small part of a production ML system, and almost every expensive failure happens in the rest of it.

<a id="151-model-deployment"></a>

15.1 Model Deployment

Deployment is the step where a model stops being a file and starts being a dependency; this section covers when predictions should be precomputed versus computed on request, how to package a model so it loads correctly somewhere else, the two dominant serving protocols, the container and orchestration layer underneath them, what changes when the target is a phone rather than a server, and how to make an inference call fast enough to be usable.

Batch vs Real-Time Inference

Before any serving technology is chosen, one question settles most of the architecture: when is the prediction needed, relative to when the input data becomes available?

If a marketing team needs churn scores for every customer by 09:00 each morning, nothing is gained by computing them on demand. Score everyone overnight, write the results to a table, and let consumers read the table. This is batch inference: predictions are precomputed on a schedule, stored as data, and looked up. The model is not in the request path at all, so serving a score is a key lookup in a database -- the most reliable operation in the whole system.

If a payment must be approved or declined before the page returns, no precomputed table can help, because the transaction being judged did not exist when the batch job ran. This is real-time (or online) inference: a request arrives, features are gathered, the model runs, and a response goes back within a latency budget measured in milliseconds.

The useful analogy is a kitchen. Batch inference is meal prep: cook a hundred portions on Sunday, refrigerate, reheat on demand. It is enormously efficient per portion, and everything you serve is as fresh as Sunday. Real-time inference is cooking to order: slower and costlier per plate, and the only way to serve a dish that depends on what the customer just said.

Between them sit two hybrids worth knowing by name. Streaming inference scores each event as it arrives on a message log, writing results back to a store -- continuous like real-time, but not in anyone's request path. Near-line or micro-batch inference runs the batch job every few minutes instead of nightly, which is often all the freshness a problem actually needs and is dramatically simpler than a serving stack.

Property	Batch	Real-time
Trigger	Schedule	Request
Latency budget	Minutes to hours	Milliseconds
Input availability	Must exist before the run	Arrives with the request
Throughput efficiency	Very high (vectorised)	Low (per-call overhead)
Failure impact	Stale scores, retry later	User-visible error
Rollback	Rerun the job	Redeploy the service
Feature engineering	Same code as training	Must be re-implemented or shared
Typical uses	Churn lists, recommendations, risk segments	Fraud, ranking, pricing, ads

The last row of that table is where teams get hurt. In batch, training and scoring can run the same code over the same warehouse tables, so the features are identical by construction. In real-time, the serving path usually recomputes features from a different source under a latency budget, and any discrepancy between the two implementations is training-serving skew -- the single most common cause of a model that validated well and performs badly in production.

Three numbers drive the decision, and all three can be estimated before any code is written: how fast the inputs change, how many of the predictions are ever read, and how expensive one prediction is.

import time
import numpy as np

rng = np.random.default_rng(0)
N, D = 1_000_000, 24
X = rng.normal(size=(N, D)).astype(np.float32)
w = rng.normal(size=D).astype(np.float32)

def score(rows):
    return 1.0 / (1.0 + np.exp(-(rows @ w)))       # logistic model, any size

# --- THROUGHPUT: THE SAME MODEL, TWO ACCESS PATTERNS. ---
t0 = time.perf_counter()
score(X)                                            # one vectorised call
batch_s = time.perf_counter() - t0

t0 = time.perf_counter()
for i in range(20_000):
    score(X[i:i + 1])                               # one row per call, as an
one_s = (time.perf_counter() - t0) / 20_000         # online server sees it

print("THROUGHPUT OF ONE LOGISTIC MODEL, TWO ACCESS PATTERNS")
print(f"  batch   {N:,} rows in {batch_s:6.2f} s"
      f"  -> {N / batch_s / 1e6:6.2f} M rows/s")
print(f"  online  1 row per call, mean {one_s * 1e6:6.1f} us"
      f"  -> {1 / one_s / 1e6:6.2f} M rows/s")
print(f"  per-row cost of serving one at a time: {one_s / (batch_s / N):5.1f}x")
print("  The arithmetic is identical; the overhead is per-call, not per-row.")

# --- FRESHNESS: WHAT A NIGHTLY SCORE COSTS IN CORRECTNESS. ---
# Each user's state changes at random times (a new session, a cancelled card,
# a support ticket). A 00:00 batch score is valid only until the next change.
print("\nP(SCORE ALREADY STALE) BY CHANGE RATE AND HOUR OF REQUEST")
print("  changes/user/day      06:00     12:00     18:00     23:00")
for lam in (0.05, 0.5, 2.0, 20.0):
    row = [1 - np.exp(-lam * h / 24) for h in (6, 12, 18, 23)]
    print(f"  {lam:16.2f}" + "".join(f"{p:10.1%}" for p in row))
print("  Slow-moving features tolerate batch. Fast-moving ones do not, and")
print("  no amount of model quality compensates for scoring the wrong state.")

# --- COST: BATCH SCORES EVERYONE, ONLINE PAYS PER REQUEST. ---
USERS, DAYS = 10_000_000, 30
VCPU_HOUR, WRITE_PER_M, GB_MONTH = 0.04, 0.50, 0.023
BATCH_ROWS_PER_VCPU_S = 5_000       # the feature join dominates, not the model
PEAK_FACTOR = 3.0                   # peak traffic over mean traffic

b_compute = USERS / BATCH_ROWS_PER_VCPU_S / 3600 * VCPU_HOUR * DAYS
b_write = USERS / 1e6 * WRITE_PER_M * DAYS          # refreshing the KV store
b_store = USERS * 200 / 1e9 * GB_MONTH              # 200 bytes per served row
batch = b_compute + b_write + b_store

def online(frac, ms):
    qps = USERS * frac / 86400
    vcpu = max(2, np.ceil(qps * PEAK_FACTOR * ms / 1000))
    return vcpu * VCPU_HOUR * 24 * DAYS             # replicas bill while idle

print("\nMONTHLY COST, 10 M USERS SCORED NIGHTLY VS SERVED ON REQUEST")
print(f"  batch: compute {b_compute:.2f} + writes {b_write:.2f}"
      f" + storage {b_store:.2f} = ${batch:.2f}, whatever anyone reads")
print("  read fraction     online, 3 ms model     online, 50 ms model")
for frac in (0.001, 0.01, 0.1, 0.5, 1.0):
    print(f"  {frac:12.1%}   {online(frac, 3):18.2f}   {online(frac, 50):21.2f}")
print("  Batch pays to score everyone, including the users nobody asks about.")
print("  Online pays for peak capacity and idles at low traffic, so a cheap")
print("  model served rarely is the cheapest thing here -- and an expensive")
print("  model read by everyone is the one case batch clearly wins.")

THROUGHPUT OF ONE LOGISTIC MODEL, TWO ACCESS PATTERNS
  batch   1,000,000 rows in   0.05 s  ->  19.39 M rows/s
  online  1 row per call, mean    6.3 us  ->   0.16 M rows/s
  per-row cost of serving one at a time: 121.5x
  The arithmetic is identical; the overhead is per-call, not per-row.

P(SCORE ALREADY STALE) BY CHANGE RATE AND HOUR OF REQUEST
  changes/user/day      06:00     12:00     18:00     23:00
              0.05      1.2%      2.5%      3.7%      4.7%
              0.50     11.8%     22.1%     31.3%     38.1%
              2.00     39.3%     63.2%     77.7%     85.3%
             20.00     99.3%    100.0%    100.0%    100.0%
  Slow-moving features tolerate batch. Fast-moving ones do not, and
  no amount of model quality compensates for scoring the wrong state.

MONTHLY COST, 10 M USERS SCORED NIGHTLY VS SERVED ON REQUEST
  batch: compute 0.67 + writes 150.00 + storage 0.05 = $150.71, whatever anyone reads
  read fraction     online, 3 ms model     online, 50 ms model
          0.1%                57.60                   57.60
          1.0%                57.60                   57.60
         10.0%                57.60                   57.60
         50.0%                57.60                  259.20
        100.0%                57.60                  518.40
  Batch pays to score everyone, including the users nobody asks about.
  Online pays for peak capacity and idles at low traffic, so a cheap
  model served rarely is the cheapest thing here -- and an expensive
  model read by everyone is the one case batch clearly wins.

The first block shows why this is an architectural choice rather than a preference. The identical dot product runs roughly a hundred times cheaper per row in batch, because the per-call overhead -- dispatch, allocation, Python frames, and in a real service the network hop and feature fetch -- is paid once for a million rows instead of a million times. No serving framework removes that gap; batching within the online path (see Inference Latency Optimization) only narrows it.

The second block prices the other side. A nightly score is a snapshot, and its correctness decays at the rate the underlying state changes. For a feature that changes twice a day, more than half of the scores served in the evening describe a user who has since changed. That is not a modelling problem and cannot be fixed by a better model.

The cost table shows why the answer is rarely dogmatic. Batch pays for the whole population whether or not anyone reads a given row -- and note that the dominant term is not compute but the writes that refresh the serving store. Online pays for peak-sized capacity that idles the rest of the day. A cheap model whose scores are rarely read is cheapest online; an expensive model whose scores are read for nearly every user is cheapest in batch.

A practical decision procedure:

1. Can the prediction be made before the request? If the inputs all exist in advance, start with batch. It is simpler, cheaper, and easier to debug.
2. How stale may it be? Convert the answer into a schedule. "Within an hour" is hourly micro-batch, not a serving cluster.
3. What fraction is read? Scoring 50 million users so that 20,000 lists are opened is a bad trade.
4. Only then, if the inputs arrive with the request and the freshness requirement is seconds, build the online path -- and design the feature layer first, not the model server.

Key Takeaways

• The deciding question is whether the prediction's inputs exist before the request arrives, not which technology is fashionable.
• Batch precomputes on a schedule and serves predictions as a key lookup, taking the model out of the request path entirely.
• Real-time computes per request under a millisecond budget and is required only when the input is created by the request itself.
• Per-row, batch inference is one to two orders of magnitude cheaper, because per-call overhead is amortised across the whole dataset.
• A batch score's correctness decays at the rate its inputs change; with two changes per user per day, most evening reads are already stale.
• Batch cost is dominated by writing scores for everyone, including users nobody asks about; online cost is dominated by peak-sized idle capacity.
• Streaming and micro-batch are real options between the two extremes and often satisfy the actual freshness requirement.
• Real-time serving introduces training-serving skew risk, because features get computed twice in two different systems.

🧪 Practice

1. For a model you know, list its features and estimate how often each one changes, then state the longest batch interval that keeps them usable.
2. Measure the per-row cost of your own model scored as one batch of 100,000 rows versus 100,000 single-row calls, and explain the ratio.
3. Price a nightly batch job for 5 million users (compute, serving-store writes, storage) against an online service at 50 requests per second, and find the read fraction at which they cost the same.
4. Take a real-time use case and redesign it as micro-batch: what freshness is lost, and what operational complexity disappears?
5. Interview: A team serves recommendations from a nightly batch table and wants to include items the user viewed in the last five minutes. How would you do it without building a full online model service? (Hint: not everything in a prediction has to come from the model; consider which part must be fresh and whether it can be applied after retrieval.)

Model Serialization and Packaging

A trained model in memory is a graph of Python objects: arrays, fitted transformers, references to library code. Deployment requires turning that into bytes that another process, on another machine, weeks later, can turn back into something that computes the same numbers. That round trip is where a surprising number of production incidents begin.

The default answer in Python is pickle (or joblib, which is pickle with better handling of large arrays). It is one line to save and one to load, and it works perfectly on the machine that created it. The trap is what pickle actually stores: not the class, but a reference to where the class lives. Unpickling re-imports your source code and reconstructs an instance. If the module moved, the class was renamed, an attribute was added, or the library version changed its internals, loading either fails loudly or -- worse -- succeeds into an object that behaves differently.

Two further properties matter. Pickle is not a security boundary: unpickling executes arbitrary code by construction, so a pickle from an untrusted source is equivalent to running a script from that source. And a pickle is not portable across languages, which rules it out if inference happens in a JVM service or a browser.

The escape from all of this is to stop thinking of "the model" as the fitted estimator, and start thinking of it as an artifact bundle:

  WHAT A DEPLOYABLE MODEL ARTIFACT CONTAINS

    model/
      weights.npz | model.onnx | model.txt   the learned parameters
      manifest.json                          schema, versions, checksums
      preprocessing                          fitted, and versioned with it
      expected_io.json                       a known input -> known output
      requirements.lock                      the exact environment
      card.md                                intended use, limits, owner

    The rule: anything the prediction depends on ships with the prediction.
    A scaler fitted at training time is part of the model, not of the
    notebook that produced it.

The formats you will meet, and what each buys:

Format	Portable	Cross-language	Safe to load	Notes
pickle / joblib	No	No	No	Fastest to adopt, worst to maintain
ONNX	Yes	Yes	Yes	Common graph IR; runtimes in C++, Java, JS
PMML	Yes	Yes	Yes	XML, classic models, verbose
TensorFlow SavedModel	Yes	Yes	Yes	Graph plus weights plus signature
TorchScript	Yes	C++	Yes	Traced or scripted PyTorch graph
safetensors	Yes	Yes	Yes	Weights only, no code execution
Booster text dumps	Yes	Yes	Yes	XGBoost/LightGBM native formats
Plain arrays plus code	Yes	Yes	Yes	Best for simple models; you own the contract

The example below makes the pickle failure concrete and then builds the minimum artifact that does not have it: weights in a neutral container, a manifest describing the input schema and environment, a checksum, and a shipped smoke test.

import hashlib, json, pickle, subprocess, sys, tempfile, textwrap, pathlib
import numpy as np

class Pipeline:
    """A model plus the preprocessing it cannot be correct without."""
    def __init__(self, cols, mean, scale, w, b):
        self.cols, self.mean, self.scale, self.w, self.b = cols, mean, scale, w, b

    def predict(self, rows):                       # rows: dict of column -> array
        x = np.stack([rows[c] for c in self.cols], axis=1)
        z = (x - self.mean) / self.scale           # the SAME scaling as training
        return 1 / (1 + np.exp(-(z @ self.w + self.b)))

cols = ["amount", "age_days", "n_prior"]
pipe = Pipeline(cols, np.array([50.0, 300.0, 4.0]), np.array([30.0, 200.0, 3.0]),
                np.array([1.4, -0.7, 0.2]), -0.5)
sample = {"amount": np.array([120.0]), "age_days": np.array([30.0]),
          "n_prior": np.array([1.0])}
print(f"REFERENCE PREDICTION: {pipe.predict(sample)[0]:.6f}")

tmp = pathlib.Path(tempfile.mkdtemp())
(tmp / "model.pkl").write_bytes(pickle.dumps(pipe))

# --- WHAT A PICKLE ACTUALLY STORES: A POINTER TO YOUR SOURCE CODE. ---
print("\nWHAT THE PICKLE FILE CONTAINS")
blob = (tmp / "model.pkl").read_bytes()
print(f"  {len(blob)} bytes, referencing: "
      f"{[s for s in (b'__main__', b'Pipeline', b'numpy') if s in blob]}")
print("  The weights are there; the CLASS is not. Loading re-imports it.")

loader = textwrap.dedent("""
    import pickle, sys
    try:
        m = pickle.load(open(sys.argv[1], 'rb'))   # no Pipeline definition here
        print('  loaded', type(m))
    except Exception as e:
        print(f'  {type(e).__name__}: {e}')
""")
(tmp / "load.py").write_text(loader)
print("\nLOADING THE SAME FILE IN A PROCESS WITHOUT THE SOURCE")
print(subprocess.run([sys.executable, str(tmp / "load.py"), str(tmp / "model.pkl")],
                     capture_output=True, text=True).stdout.rstrip())
print("  This is the deployment failure: the artifact is not self-contained.")

# --- A PORTABLE ARTIFACT: WEIGHTS, SCHEMA, ENVIRONMENT, CHECKSUM. ---
def export(pipe, path):
    np.savez(path / "weights.npz", mean=pipe.mean, scale=pipe.scale,
             w=pipe.w, b=np.array([pipe.b]))
    digest = hashlib.sha256((path / "weights.npz").read_bytes()).hexdigest()
    manifest = {
        "format": "logreg/1",                      # how to interpret weights
        "input_schema": {c: "float64" for c in pipe.cols},   # names AND order
        "output": {"name": "p_fraud", "dtype": "float64", "range": [0, 1]},
        "weights_sha256": digest,
        "runtime": {"python": "3.11", "numpy": np.__version__},
        "training_run": "run-2026-09-14-a41f",     # links back to the experiment
        "expected": {"input": {c: v.tolist() for c, v in sample.items()},
                     "output": round(float(pipe.predict(sample)[0]), 12)},
    }
    (path / "manifest.json").write_text(json.dumps(manifest, indent=2))

def load(path, incoming_cols):
    man = json.loads((path / "manifest.json").read_text())
    blob = (path / "weights.npz").read_bytes()
    if hashlib.sha256(blob).hexdigest() != man["weights_sha256"]:
        raise ValueError("weights do not match manifest checksum")
    expected = list(man["input_schema"])          # order is part of it
    if incoming_cols != expected:                  # order matters to a matmul
        raise ValueError(f"schema mismatch: expected {expected}, "
                         f"got {incoming_cols}")
    z = np.load(path / "weights.npz")
    m = Pipeline(expected, z["mean"], z["scale"], z["w"], float(z["b"][0]))
    got = float(m.predict({k: np.array(v) for k, v in man["expected"]["input"].items()})[0])
    if abs(got - man["expected"]["output"]) > 1e-9:  # the smoke test ships too
        raise ValueError(f"prediction drift: {got} != {man['expected']['output']}")
    return m

export(pipe, tmp)
print("\nMANIFEST")
print(textwrap.indent((tmp / "manifest.json").read_text(), "  "))

print("\nLOADING WITH VALIDATION")
for label, incoming in [("correct columns", cols),
                        ("reordered columns", ["age_days", "amount", "n_prior"]),
                        ("renamed column", ["amount", "age", "n_prior"])]:
    try:
        m = load(tmp, incoming)
        print(f"  {label:20s} -> ok, p={m.predict(sample)[0]:.6f}")
    except ValueError as e:
        print(f"  {label:20s} -> REJECTED: {e}")
print("  Reordered columns are the dangerous case: without the check the")
print("  model returns a confident, wrong number instead of an error.")

REFERENCE PREDICTION: 0.971018

WHAT THE PICKLE FILE CONTAINS
  385 bytes, referencing: [b'__main__', b'Pipeline', b'numpy']
  The weights are there; the CLASS is not. Loading re-imports it.

LOADING THE SAME FILE IN A PROCESS WITHOUT THE SOURCE
  AttributeError: Can't get attribute 'Pipeline' on <module '__main__' from '/tmp/tmpyuh08vrf/load.py'>
  This is the deployment failure: the artifact is not self-contained.

MANIFEST
  {
    "format": "logreg/1",
    "input_schema": {
      "amount": "float64",
      "age_days": "float64",
      "n_prior": "float64"
    },
    "output": {
      "name": "p_fraud",
      "dtype": "float64",
      "range": [
        0,
        1
      ]
    },
    "weights_sha256": "a4493ec4eb6c05f84969e13e675d5fc5af8533dc14724239c539991440893a53",
    "runtime": {
      "python": "3.11",
      "numpy": "2.5.3"
    },
    "training_run": "run-2026-09-14-a41f",
    "expected": {
      "input": {
        "amount": [
          120.0
        ],
        "age_days": [
          30.0
        ],
        "n_prior": [
          1.0
        ]
      },
      "output": 0.971017904558
    }
  }

LOADING WITH VALIDATION
  correct columns      -> ok, p=0.971018
  reordered columns    -> REJECTED: schema mismatch: expected ['amount', 'age_days', 'n_prior'], got ['age_days', 'amount', 'n_prior']
  renamed column       -> REJECTED: schema mismatch: expected ['amount', 'age_days', 'n_prior'], got ['amount', 'age', 'n_prior']
  Reordered columns are the dangerous case: without the check the
  model returns a confident, wrong number instead of an error.

Three details in that manifest carry most of the value.

The input schema includes order, not just names. A matrix multiply cannot tell you that columns two and three were swapped; it returns a number with full confidence. Schema validation at load time converts the worst class of production bug -- silently wrong predictions -- into a startup failure, which is the cheapest kind.

The checksum binds the weights to the manifest. Without it, a partially copied file or a hand-edited artifact deploys happily.

The expected input/output pair is a shipped smoke test. Reloading a model and checking one known prediction to the twelfth decimal catches library upgrades that changed a default, a different BLAS producing different rounding, and mismatched preprocessing. Run it in the container build, not only on the training machine.

Two more packaging rules worth adopting without argument:

• Pin the environment, do not describe it. requirements.txt with ranges documents intent; a lock file with hashes reproduces the environment. The model artifact should reference the lock file it was trained under.
• Version the artifact, never overwrite it. model_v3.pkl in a shared bucket is not a version; an immutable path plus a registry entry is (see Model Registries and Versioning).

Key Takeaways

• Pickle stores a reference to your class, not the class, so loading it re-imports your source and depends on it being unchanged.
• Unpickling executes arbitrary code, so pickles are unsafe from untrusted sources and unusable across languages.
• Treat the deployable unit as an artifact bundle: weights, fitted preprocessing, schema, environment lock, checksum, and metadata.
• ONNX, SavedModel, TorchScript, and safetensors trade a little convenience for portability and safe loading.
• Validate the input schema -- including column order -- at load time; a reordering otherwise produces confident wrong answers.
• Ship a known input/output pair with the model and assert it on load to catch environment and library changes.
• Fitted preprocessing belongs inside the artifact; a scaler left in the training notebook is a guaranteed skew bug.
• Pin environments with a lock file and make artifacts immutable and versioned.

🧪 Practice

1. Pickle a model class, rename the class, and attempt to load the file; then repeat after adding an attribute, and describe both failures.
2. Write an export/load pair for a model you have trained that stores weights, schema, and a checksum, and rejects a mismatched schema.
3. Add a shipped smoke test to that artifact and verify it fails when you deliberately change the scaler's mean by 1%.
4. Export a small scikit-learn pipeline to ONNX and compare predictions against the original on 1,000 rows; report the maximum absolute difference and decide whether it is acceptable.
5. Interview: Your service loads a model from a bucket at startup. What can go wrong between the training job writing it and the pod reading it, and how would you make each failure loud? (Hint: think about partial writes, "latest" pointers, environment differences, and silent success.)

REST and gRPC Model Serving

Once a prediction must be computed on request, the model needs an address. Wrapping it in a network service is the standard answer, and it changes the model's job description: it is no longer a function you call, it is a dependency with an uptime target, a latency budget, and a contract that other teams code against.

REST over HTTP/JSON is the default. Every language can call it, every proxy understands it, and you can debug it with curl. A minimal serving endpoint is short enough to read in one screen:

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
import numpy as np

app = FastAPI()
MODEL = load_model("/models/fraud/v7")          # loaded ONCE, at startup

class Instance(BaseModel):                      # the contract, enforced
    amount: float = Field(gt=0)
    age_days: int = Field(ge=0)
    n_prior: int = Field(ge=0)

class Request(BaseModel):
    instances: list[Instance] = Field(max_length=256)   # bound the batch

@app.post("/v1/score")
def score(req: Request):
    x = np.array([[i.amount, i.age_days, i.n_prior] for i in req.instances])
    try:
        p = MODEL.predict(x)
    except Exception:                           # never leak a stack trace
        raise HTTPException(status_code=500, detail="scoring failed")
    return {"model_version": MODEL.version,     # every response is traceable
            "predictions": [round(float(v), 6) for v in p]}

@app.get("/healthz")                            # is the process alive?
def healthz():
    return {"status": "ok"}

@app.get("/readyz")                             # is the MODEL loaded?
def readyz():
    return {"status": "ok" if MODEL.is_loaded else "loading",
            "model_version": MODEL.version}

Four details in that snippet are the difference between a demo and a service. The model loads once at startup, not per request. The request schema is validated and bounded, because an unbounded batch size is a denial of service. The response carries the model version, so a prediction logged today can be attributed to an artifact later. And liveness is separated from readiness: a pod that is running but has not finished loading a 2 GB model must not receive traffic.

gRPC takes the same idea and changes the wire format and transport: schemas are declared in a .proto file, messages are binary protobuf, and the transport is HTTP/2 with multiplexed, persistent connections.

service Scorer {
  rpc Score (ScoreRequest) returns (ScoreResponse);
  rpc ScoreStream (stream ScoreRequest) returns (stream ScoreResponse);
}

message ScoreRequest {
  repeated float features = 1 [packed = true];  // packed: no per-value tag
  string request_id = 2;
}

message ScoreResponse {
  repeated float predictions = 1 [packed = true];
  string model_version = 2;
}

The generated client is typed in every supported language, and the schema is checked at compile time rather than discovered at 3 a.m. The cost is ecosystem friction: browsers cannot speak gRPC natively (they need gRPC-Web or a gateway), and inspecting traffic requires tooling rather than curl.

The measurements below put numbers on the trade, and then on the thing that actually determines whether a service meets its SLO.

import json, struct, time
import numpy as np

rng = np.random.default_rng(7)
D = 200                                            # features per request

def bench(fn, n=2000):
    t0 = time.perf_counter()
    for _ in range(n):
        fn()
    return (time.perf_counter() - t0) / n * 1e6    # microseconds per call

print("PAYLOAD SIZE AND CODEC COST, 200 FLOAT FEATURES PER ROW")
print("  batch   json bytes   binary bytes   json us   binary us   ratio")
for batch in (1, 8, 64):
    x = rng.normal(size=(batch, D))
    js = json.dumps({"instances": x.tolist()}).encode()
    # A protobuf/gRPC payload is length-prefixed packed floats, not text.
    bin_ = struct.pack("<II", batch, D) + x.astype(np.float32).tobytes()
    j_enc = bench(lambda: json.loads(json.dumps({"instances": x.tolist()})))
    b_enc = bench(lambda: np.frombuffer(                       # encode + decode
        struct.pack("<II", batch, D) + x.astype(np.float32).tobytes(),
        dtype=np.float32, offset=8).reshape(batch, D))
    print(f"  {batch:5d}   {len(js):10,}   {len(bin_):12,}"
          f"   {j_enc:7.1f}   {b_enc:9.1f}   {len(js) / len(bin_):5.1f}x")
print("  JSON stores every float as text: ~20 bytes instead of 4, and both")
print("  sides pay to parse it. The size ratio is flat, but the parse cost")
print("  grows linearly -- at batch 64 it alone exceeds a 15 ms budget.")

# --- WHERE A 50 ms BUDGET ACTUALLY GOES. ---
# Measured medians from a typical Python model service; the model is rarely
# the largest term, which is why "optimise the model" often changes nothing.
budget = [
    ("client -> LB network",        4.0,  4.0),
    ("TLS + connection setup",      7.0,  0.2),    # reused with HTTP/2
    ("request deserialisation",     3.5,  0.4),
    ("feature store lookup",       12.0, 12.0),
    ("model forward pass",          6.0,  6.0),
    ("response serialisation",      1.8,  0.2),
    ("network back",                4.0,  4.0),
]
print("\nLATENCY BUDGET: REST/JSON VS gRPC/PROTOBUF ON THE SAME MODEL")
print("  stage                        REST ms    gRPC ms")
for name, rest, grpc in budget:
    print(f"  {name:26s} {rest:9.1f} {grpc:10.1f}")
tr, tg = sum(r for _, r, _ in budget), sum(g for _, _, g in budget)
print(f"  {'TOTAL':26s} {tr:9.1f} {tg:10.1f}")
print(f"  model share of total: REST {6.0 / tr:.0%}, gRPC {6.0 / tg:.0%}")
print("  Halving model time saves 3 ms; fixing transport and features saves 12.")

# --- TAIL LATENCY IS THE NUMBER THAT MATTERS. ---
# A page that fans out to K services waits for the SLOWEST of them.
lat = rng.lognormal(mean=np.log(12), sigma=0.55, size=200_000)
p50, p99 = np.percentile(lat, [50, 99])
print(f"\nONE SERVICE: p50 {p50:.1f} ms, p99 {p99:.1f} ms")
print("  calls per page   p99 of the slowest call")
for k in (1, 5, 20, 100):
    fan = rng.lognormal(mean=np.log(12), sigma=0.55, size=(50_000, k)).max(axis=1)
    print(f"  {k:14d}   {np.percentile(fan, 99):22.1f} ms")
print("  With 20 parallel calls, the page's p99 is set by a tail the")
print("  single-service p50 never shows. Serving SLOs are written on p99.")

PAYLOAD SIZE AND CODEC COST, 200 FLOAT FEATURES PER ROW
  batch   json bytes   binary bytes   json us   binary us   ratio
      1        4,179            808     258.4         1.8     5.2x
      8       33,134          6,408    2203.9         3.3     5.2x
     64      264,269         51,208   16954.7         9.8     5.2x
  JSON stores every float as text: ~20 bytes instead of 4, and both
  sides pay to parse it. The size ratio is flat, but the parse cost
  grows linearly -- at batch 64 it alone exceeds a 15 ms budget.

LATENCY BUDGET: REST/JSON VS gRPC/PROTOBUF ON THE SAME MODEL
  stage                        REST ms    gRPC ms
  client -> LB network             4.0        4.0
  TLS + connection setup           7.0        0.2
  request deserialisation          3.5        0.4
  feature store lookup            12.0       12.0
  model forward pass               6.0        6.0
  response serialisation           1.8        0.2
  network back                     4.0        4.0
  TOTAL                           38.3       26.8
  model share of total: REST 16%, gRPC 22%
  Halving model time saves 3 ms; fixing transport and features saves 12.

ONE SERVICE: p50 12.0 ms, p99 43.1 ms
  calls per page   p99 of the slowest call
               1                     42.8 ms
               5                     59.0 ms
              20                     72.7 ms
             100                     91.9 ms
  With 20 parallel calls, the page's p99 is set by a tail the
  single-service p50 never shows. Serving SLOs are written on p99.

The payload comparison explains why high-dimensional models tend to migrate to gRPC: a float takes about 20 bytes as text and 4 as binary, and text has to be parsed on both sides. For a three-feature request this is irrelevant. For a 200-feature request at batch 64, JSON parsing alone consumes more than a typical end-to-end budget.

The latency budget is the more important lesson, and it generalises beyond protocols. The model's forward pass is 16% of the REST total. A team that spends a month quantising the model to halve inference time buys 3 milliseconds; switching to a persistent HTTP/2 connection and fixing the feature lookup buys four times that. Measure the budget before optimising any part of it.

The fan-out simulation is why serving SLOs are written on p99 rather than the mean. A page that calls twenty services in parallel is as slow as the slowest response, so a tail that affects 1% of calls affects roughly 18% of pages. This is also the argument for timeouts with a fallback: a model service that cannot answer in 30 ms should return a cached or default prediction rather than hold the page.

Dimension	REST/JSON	gRPC/protobuf
Wire format	Text, human-readable	Binary, compact
Schema	Optional (OpenAPI)	Mandatory .proto, compiled
Transport	HTTP/1.1 or 2	HTTP/2, multiplexed
Streaming	Awkward (SSE, chunking)	First class, bidirectional
Browser support	Native	Needs gRPC-Web or gateway
Debuggability	curl, any proxy	Needs grpcurl, reflection
Best fit	Public APIs, low QPS, few features	Internal, high QPS, wide vectors

Whichever protocol you choose, the operational surface is the same: health and readiness endpoints, request and model version in every response, structured logs keyed by request id, bounded batch sizes, timeouts on every downstream call, and a metrics endpoint exposing request rate, error rate, and latency percentiles.

Key Takeaways

• Serving turns a model into a dependency with an SLO, a contract, and an operational surface, not just a function call.
• Load the model once at startup and separate liveness from readiness so traffic arrives only after the artifact is in memory.
• Validate and bound request payloads; an unbounded batch is an availability risk, not a convenience.
• Return the model version with every prediction so logged outcomes can be attributed to an artifact.
• JSON costs roughly 5x the bytes of packed binary and must be parsed on both sides; this matters only for wide vectors or large batches.
• gRPC adds compile-time schemas, HTTP/2 multiplexing, and streaming, at the price of browser support and easy debugging.
• The model is usually a minority of end-to-end latency; profile the whole budget before optimising the forward pass.
• Write SLOs on p99: with parallel fan-out, the page inherits the tail of its slowest dependency.

🧪 Practice

1. Write a scoring endpoint with request validation, a readiness check that reports the loaded model version, and a bounded batch size.
2. Measure JSON versus binary encoding cost for your own feature vector at batch sizes 1, 32, and 256, and state at which point it matters.
3. Build a latency budget for one real endpoint by instrumenting each stage, then identify the largest term and what you would do about it.
4. Simulate a page that calls 10 model services in parallel and compute how the page's p99 relates to a single service's p99.
5. Interview: Your model service meets its p50 target but misses p99 badly, and CPU usage is low. What are the likely causes? (Hint: think about what a request waits for rather than what it computes -- garbage collection pauses, connection setup, cold caches, and queueing all produce exactly this signature.)

Containerization and Orchestration

"It works on my machine" is not a joke about carelessness; it is an accurate statement about hidden dependencies. A model service depends on a Python version, a compiled BLAS, a CUDA driver, a locale setting, a timezone, and a dozen shared libraries nobody listed. A container makes that dependency set explicit and shippable: the image contains the filesystem the process will see, so the same bytes run identically on a laptop, in CI, and in production.

The mental model is a stack of read-only layers with one writable layer on top. Each build instruction adds a layer, layers are cached by content, and layers are shared between images. This is why instruction order is a performance decision rather than a style choice:

  IMAGE LAYERS, AND WHY ORDER MATTERS

    [ writable container layer ]   <- per-container, discarded on restart
    [ COPY src/            ]  12 MB   changes every commit
    [ COPY model/ (2.1 GB) ]         changes on retrain
    [ RUN pip install ...  ] 780 MB   changes when deps change
    [ COPY requirements.lock]  4 KB
    [ python:3.11-slim     ] 130 MB   almost never changes

    Copy source LAST. Putting "COPY . ." before "pip install" invalidates
    the dependency layer on every commit and rebuilds 780 MB each time.
    Keep large model files in a separate layer, or out of the image
    entirely and fetched at startup from a registry.

A serving image that will not embarrass you in review:

# --- build stage: compilers and wheels live here, not in the final image ---
FROM python:3.11-slim AS build
WORKDIR /app
COPY requirements.lock .
RUN pip install --no-cache-dir --prefix=/install -r requirements.lock

# --- runtime stage: only what is needed to serve ---
FROM python:3.11-slim
ENV PYTHONUNBUFFERED=1 OMP_NUM_THREADS=1        # one BLAS thread per worker
COPY --from=build /install /usr/local
COPY src/ /app/src/
WORKDIR /app
RUN useradd -r -u 10001 svc && chown -R svc /app
USER svc                                        # never run the server as root
EXPOSE 8080
HEALTHCHECK --interval=10s CMD python -c "import urllib.request;\
  urllib.request.urlopen('http://localhost:8080/healthz')"
CMD ["uvicorn", "src.app:app", "--host", "0.0.0.0", "--port", "8080"]

OMP_NUM_THREADS=1 deserves a sentence of its own. NumPy, PyTorch, and LightGBM each start a thread pool sized to the host's core count, not the container's CPU limit. Four replicas on a 64-core node will each spawn 64 threads, and the resulting contention can make a service slower as you add replicas. Setting thread counts explicitly is one of the highest-value lines in an ML Dockerfile.

Orchestration is what runs those containers: scheduling them onto machines, restarting them when they die, replacing them during a deploy, and routing traffic to the healthy ones. Kubernetes is the common implementation, and four objects cover most model serving:

apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 29
  strategy:
    rollingUpdate: { maxSurge: 25%, maxUnavailable: 0 }   # never lose capacity
  template:
    spec:
      containers:
        - name: scorer
          image: registry/fraud-scorer@sha256:9c1f...    # digest, not :latest
          resources:
            requests: { cpu: "2", memory: "4Gi" }        # what the scheduler reserves
            limits:   { cpu: "4", memory: "6Gi" }        # what the kernel enforces
          readinessProbe:                                 # model loaded?
            httpGet: { path: /readyz, port: 8080 }
            initialDelaySeconds: 20
          livenessProbe:                                  # process wedged?
            httpGet: { path: /healthz, port: 8080 }
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 29                                 # headroom is not optional
  maxReplicas: 90
  metrics:
    - type: Pods
      pods:
        metric: { name: inflight_requests }       # concurrency, not CPU
        target: { type: AverageValue, averageValue: "12" }

Two of those lines are where ML services differ from stateless web services. Image by digest, never by tag: :latest means a restarted pod can quietly come up on a different model. And readiness with a generous initial delay: loading a multi-gigabyte artifact takes time, and a liveness probe that fires during loading produces an infinite restart loop that looks like a crash.

How many replicas? That is queueing theory, not intuition.

import math

def erlang_c(c, rho_total):
    """P(a request has to queue) for c servers and offered load rho_total."""
    a = rho_total
    s = sum(a ** k / math.factorial(k) for k in range(c))
    top = a ** c / math.factorial(c) * (c / (c - a))
    return top / (s + top)

def sizing(qps, service_ms, c):
    a = qps * service_ms / 1000                    # offered load, in "servers"
    if c <= a:
        return None                                # unstable: queue grows forever
    pq = erlang_c(c, a)
    wq = pq * service_ms / (c - a)                 # mean wait in queue, ms
    return a / c, pq, wq, service_ms + wq

QPS, SERVICE_MS = 800, 25
print(f"SIZING A MODEL SERVICE: {QPS} req/s, {SERVICE_MS} ms per request")
print(f"  offered load = {QPS * SERVICE_MS / 1000:.0f} busy replicas minimum")
print("  replicas   utilisation   P(queued)   mean wait   mean latency")
for c in (21, 22, 24, 26, 29, 34, 40):
    r = sizing(QPS, SERVICE_MS, c)
    if r is None:
        print(f"  {c:8d}   UNSTABLE")
        continue
    util, pq, wq, tot = r
    print(f"  {c:8d}   {util:10.1%}   {pq:9.1%}   {wq:8.1f} ms   {tot:9.1f} ms")
print("  Latency does not degrade linearly: it is flat, then it is a cliff.")
print("  Sizing to 95% utilisation saves a few replicas and doubles latency.")

# --- HEADROOM MUST SURVIVE A ROLLOUT AND A NODE FAILURE. ---
print("\nWHAT CAPACITY REMAINS DURING A ROLLING UPDATE")
print("  replicas   maxUnavailable   serving   utilisation during rollout")
for c, unavail in [(24, 0), (24, 1), (24, 4), (34, 4), (34, 8)]:
    serving = c - unavail
    r = sizing(QPS, SERVICE_MS, serving)
    state = "UNSTABLE" if r is None else f"{r[0]:.1%}  ({r[3]:.0f} ms)"
    print(f"  {c:8d}   {unavail:14d}   {serving:7d}   {state}")
print("  A deployment sized for steady state has no capacity to deploy with.")

# --- AUTOSCALING ON THE WRONG SIGNAL. ---
# One replica: 4 vCPU and a pool of 16 request workers. Two ceilings exist,
# and CPU is only one of them.
CORES, WORKERS = 4, 16
print("\nWHAT LIMITS ONE REPLICA, AND WHAT THE CPU METRIC SHOWS")
print("  profile                 cpu ms  total ms   cpu cap   pool cap"
      "   binds on   CPU% then")
for name, cpu_ms, total_ms in [("pure compute", 25, 25),
                               ("feature fetch + model", 6, 25),
                               ("chained model calls", 3, 43)]:
    cpu_cap = CORES / (cpu_ms / 1000)              # requests/s before CPU is full
    pool_cap = WORKERS / (total_ms / 1000)         # Little's law on the pool
    binds = "cpu" if cpu_cap < pool_cap else "pool"
    sat = min(cpu_cap, pool_cap)
    print(f"  {name:22s} {cpu_ms:7d} {total_ms:9d} {cpu_cap:9.0f} {pool_cap:10.0f}"
          f"   {binds:>8}   {sat * cpu_ms / 1000 / CORES:8.0%}")
print("  The last row saturates at 28% CPU: an HPA targeting 70% CPU will")
print("  never scale it, while every request queues. Scale on concurrency,")
print("  queue depth, or latency SLO unless the service is truly CPU-bound.")

SIZING A MODEL SERVICE: 800 req/s, 25 ms per request
  offered load = 20 busy replicas minimum
  replicas   utilisation   P(queued)   mean wait   mean latency
        21        95.2%       76.1%       19.0 ms        44.0 ms
        22        90.9%       56.8%        7.1 ms        32.1 ms
        24        83.3%       29.8%        1.9 ms        26.9 ms
        26        76.9%       14.3%        0.6 ms        25.6 ms
        29        69.0%        4.0%        0.1 ms        25.1 ms
        34        58.8%        0.3%        0.0 ms        25.0 ms
        40        50.0%        0.0%        0.0 ms        25.0 ms
  Latency does not degrade linearly: it is flat, then it is a cliff.
  Sizing to 95% utilisation saves a few replicas and doubles latency.

WHAT CAPACITY REMAINS DURING A ROLLING UPDATE
  replicas   maxUnavailable   serving   utilisation during rollout
        24                0        24   83.3%  (27 ms)
        24                1        23   87.0%  (28 ms)
        24                4        20   UNSTABLE
        34                4        30   66.7%  (25 ms)
        34                8        26   76.9%  (26 ms)
  A deployment sized for steady state has no capacity to deploy with.

WHAT LIMITS ONE REPLICA, AND WHAT THE CPU METRIC SHOWS
  profile                 cpu ms  total ms   cpu cap   pool cap   binds on   CPU% then
  pure compute                25        25       160        640        cpu       100%
  feature fetch + model        6        25       667        640       pool        96%
  chained model calls          3        43      1333        372       pool        28%
  The last row saturates at 28% CPU: an HPA targeting 70% CPU will
  never scale it, while every request queues. Scale on concurrency,
  queue depth, or latency SLO unless the service is truly CPU-bound.

The first table is the single most useful piece of capacity arithmetic in serving. Offered load here is 20 replicas' worth of work; running 21 replicas is "only 5% over capacity" and yields 44 ms mean latency, while 29 replicas yields 25.1 ms. Queueing delay is not linear in utilisation -- it is flat and then vertical. Target 60-70% utilisation for a latency-sensitive service and treat anything above 85% as an incident waiting for a traffic spike.

The rollout table explains why the headroom is not waste. A deployment sized exactly for steady state cannot afford to take a single replica out of service, so every deploy becomes a latency incident and maxUnavailable: 0 becomes mandatory. The same arithmetic applies to losing a node or an availability zone.

The last table is the autoscaling mistake specific to ML services. Model servers spend much of their time waiting -- for a feature store, a vector index, another model -- so they saturate their worker pool long before CPU rises. An HPA targeting 70% CPU will watch a fully queued service sit at 28% and do nothing. Scale on in-flight requests, queue depth, or a latency objective.

GPU serving adds two more constraints: GPUs are allocated whole (a pod requesting nvidia.com/gpu: 1 owns it), and GPU memory is not overcommitted, so bin-packing is coarse and idle GPUs are expensive. Batch requests inside the server, keep the GPU pool small and busy, and put cheap traffic on CPU replicas.

Key Takeaways

• A container makes the model's hidden dependencies explicit and makes the same bytes run in CI and production.
• Order Dockerfile instructions from least to most frequently changed, and copy source last, so dependency layers stay cached.
• Use multi-stage builds, a non-root user, and pinned lock files; keep multi-gigabyte artifacts out of the layer that changes every commit.
• Set thread-count environment variables explicitly, or each replica will size its BLAS pool to the whole host and contend with its neighbours.
• Reference images by digest; :latest lets a restart change the model.
• Separate readiness from liveness and allow for slow artifact loading, or a large model produces a restart loop.
• Queueing delay is flat then vertical: size for 60-70% utilisation, not for 95%.
• Capacity must cover rollouts and node loss, which is what makes the apparent overprovisioning necessary.
• Autoscale ML services on concurrency or latency; CPU under-reports load for anything that waits on I/O.

🧪 Practice

1. Take a working model image and reorder its Dockerfile so that a source change does not reinstall dependencies; measure both rebuild times.
2. Compute the replica count your service needs for its current peak QPS at 70% utilisation, and compare it with what is deployed today.
3. Set OMP_NUM_THREADS explicitly for a NumPy-based service and measure throughput with 1, 2, and 8 replicas per node.
4. Write readiness and liveness endpoints that distinguish "process alive" from "model loaded", then verify a slow load does not trigger restarts.
5. Interview: A team's model service gets slower as they add replicas. What would you check first? (Hint: consider what each replica assumes about the machine it is on, and what resource they now share.)

Edge and On-Device Deployment

Sometimes the model cannot live on a server. A camera that must flag a defect in 5 ms cannot wait for a round trip. A keyboard suggesting the next word must work on a plane. A health app analysing heart-rate data may be forbidden from sending it anywhere. Edge deployment -- running inference on the device that produced the data -- answers four requirements at once: latency without a network, availability without connectivity, privacy without a data transfer, and cost without per-request compute.

What you give up is everything the server took for granted:

Assumption on a server	Reality at the edge
Memory is elastic	Fixed, shared with the whole application
One hardware target	Thousands of chips, drivers, and OS versions
Deploy in minutes	Ships with an app update, adopted over months
Rollback is instant	Impossible; the artifact is on the device
Every prediction is logged	Logging may be forbidden or too expensive
Power is free	Battery and thermal throttling are hard limits

Making a model small enough is a discipline of its own, with four standard tools. Quantisation stores weights (and often activations) in 8-bit integers instead of 32-bit floats: 4x smaller, and integer arithmetic is faster and cheaper in energy on most mobile silicon. Pruning removes weights or whole channels that contribute little. Distillation trains a small student model to reproduce a large teacher's outputs, which usually beats training the small model directly. Architecture choice -- depthwise separable convolutions, smaller embedding tables -- often buys more than all the compression tricks applied afterwards.

Quantisation is the one to understand in detail, because its failure mode is subtle:

import numpy as np

rng = np.random.default_rng(3)
IN, HID, OUT = 128, 256, 10
W1 = rng.normal(0, 0.12, (IN, HID)).astype(np.float32)
W2 = rng.normal(0, 0.12, (HID, OUT)).astype(np.float32)
# One channel with a much larger scale -- realistic, and the reason
# per-tensor quantisation loses so much.
W1[:, 0] *= 18.0
X = rng.normal(size=(2000, IN)).astype(np.float32)

def forward(w1, w2):
    return np.maximum(X @ w1, 0) @ w2

ref = forward(W1, W2)

def quantise(w, per_channel):
    """Symmetric int8: w ~ scale * q, q in [-127, 127]."""
    axis = 0 if per_channel else None               # per-column scales
    s = np.abs(w).max(axis=axis, keepdims=per_channel) / 127.0
    q = np.clip(np.round(w / s), -127, 127).astype(np.int8)
    return q, s.astype(np.float32)

print("POST-TRAINING INT8 QUANTISATION OF A SMALL MLP")
print("  scheme               bytes   vs fp32   max |dy|   mean rel err"
      "   top-1 agree")
fp32_bytes = W1.nbytes + W2.nbytes
print(f"  {'float32':18s} {fp32_bytes:7,}     1.00x"
      f"   {0.0:8.4f}   {0.0:12.2%}   {1.0:11.1%}")
for label, per_ch in [("int8 per-tensor", False), ("int8 per-channel", True)]:
    q1, s1 = quantise(W1, per_ch)
    q2, s2 = quantise(W2, per_ch)
    out = forward(q1.astype(np.float32) * s1, q2.astype(np.float32) * s2)
    nbytes = q1.nbytes + q2.nbytes + s1.nbytes + s2.nbytes
    rel = np.abs(out - ref).mean() / np.abs(ref).mean()
    agree = (out.argmax(1) == ref.argmax(1)).mean()
    print(f"  {label:18s} {nbytes:7,} {fp32_bytes / nbytes:8.2f}x"
          f"   {np.abs(out - ref).max():8.4f}   {rel:12.2%}   {agree:11.1%}")
print("  Per-channel scales cost 1 KB and recover almost all the accuracy:")
print("  one wide channel forces a coarse scale on every other channel.")

# --- WHAT ACTUALLY FITS ON A DEVICE. ---
print("\nBUDGETS: MODEL SIZE AGAINST THREE DEPLOYMENT TARGETS")
print("  target              budget    25 M params      350 M params")
for name, ram_mb in [("server GPU", 16_000), ("phone app", 80),
                     ("microcontroller", 0.5)]:
    cells = []
    for params_m in (25, 350):
        fp32, int8 = params_m * 4, params_m * 1     # bytes per weight
        cells.append("fp32 ok" if fp32 <= ram_mb else
                     ("int8 only" if int8 <= ram_mb else "does not fit"))
    print(f"  {name:16s} {ram_mb:8.1f} MB   {cells[0]:14s} {cells[1]}")
print("  A 25 M-parameter model is 100 MB in float32 and 25 MB in int8.")
print("  On a phone, quantisation is not an optimisation; it is the")
print("  difference between shipping and not shipping.")

# --- THE UPDATE PROBLEM: YOUR FLEET IS NEVER ON ONE VERSION. ---
print("\nFLEET VERSION MIX AFTER A MODEL UPDATE (7%/day adoption)")
print("  days since release   on new model   on previous   older still")
for d in (1, 7, 30, 90):
    new = 1 - np.exp(-0.07 * d)
    prev = (1 - new) * 0.65
    print(f"  {d:18d}   {new:12.1%}   {prev:11.1%}   {1 - new - prev:11.1%}")
print("  Server-side rollback takes minutes; a bad on-device model lives in")
print("  the field for months. Guard rails must be shipped, not deployed.")

POST-TRAINING INT8 QUANTISATION OF A SMALL MLP
  scheme               bytes   vs fp32   max |dy|   mean rel err   top-1 agree
  float32            141,312     1.00x     0.0000          0.00%        100.0%
  int8 per-tensor     35,336     4.00x     0.9624          7.77%         91.7%
  int8 per-channel    36,392     3.88x     0.1700          0.95%         99.2%
  Per-channel scales cost 1 KB and recover almost all the accuracy:
  one wide channel forces a coarse scale on every other channel.

BUDGETS: MODEL SIZE AGAINST THREE DEPLOYMENT TARGETS
  target              budget    25 M params      350 M params
  server GPU        16000.0 MB   fp32 ok        fp32 ok
  phone app            80.0 MB   int8 only      does not fit
  microcontroller       0.5 MB   does not fit   does not fit
  A 25 M-parameter model is 100 MB in float32 and 25 MB in int8.
  On a phone, quantisation is not an optimisation; it is the
  difference between shipping and not shipping.

FLEET VERSION MIX AFTER A MODEL UPDATE (7%/day adoption)
  days since release   on new model   on previous   older still
                   1           6.8%         60.6%         32.6%
                   7          38.7%         39.8%         21.4%
                  30          87.8%          8.0%          4.3%
                  90          99.8%          0.1%          0.1%
  Server-side rollback takes minutes; a bad on-device model lives in
  the field for months. Guard rails must be shipped, not deployed.

The two int8 rows differ only in where the scale factor is computed. With one scale for the whole tensor, a single wide channel stretches the range and every other channel is quantised coarsely -- 7.8% mean error and 8% of predictions changing their argmax. With one scale per output channel, the same 4x compression costs under 1% error. This is why "we quantised it and accuracy dropped" is usually a configuration problem rather than a law of nature. The same principle governs quantisation-aware training, which simulates the rounding during training so the weights adapt to it.

The fleet table is the operational fact that surprises server-trained engineers. A server rollback is a deploy; an on-device rollback does not exist. Months after a release, a meaningful share of devices still runs an older model, which has three consequences: ship conservative guard rails inside the model artifact, keep a server-side kill switch that can disable the on-device path and fall back to a remote call, and expect to support at least three model versions simultaneously in your evaluation.

Runtimes you will meet, by target:

Runtime	Target	Notes
TensorFlow Lite / LiteRT	Android, MCUs	Delegates to NNAPI, GPU, DSP
Core ML	Apple devices	Compiles to the Neural Engine
ONNX Runtime Mobile	Cross-platform	Trimmed operator set
ExecuTorch	PyTorch on device	Ahead-of-time export
WebAssembly / WebGPU	Browser	No install, but tab memory limits
TFLite Micro	Microcontrollers	No OS, static memory, kilobytes

Two practical warnings. First, operator coverage is the usual blocker: a model converts only if every operation it uses exists in the target runtime, and the fix is often to change the architecture rather than the exporter. Second, you lose your feedback loop. If outcomes cannot be sent back, you cannot measure the model's live accuracy at all; plan the telemetry (or a privacy-preserving aggregate of it) before shipping, not after.

Federated learning is the logical extension when data must not leave the device: each device trains on its local data and uploads only model updates, which a server aggregates. It is genuinely useful for keyboards and health signals, and it is considerably harder than it sounds -- non-IID data, device availability, update compression, and the fact that updates themselves can leak information without differential privacy.

Key Takeaways

• Edge inference buys latency, offline availability, privacy, and zero per-request server cost.
• It costs you elastic memory, a single hardware target, instant rollback, and often your outcome logging.
• Int8 quantisation gives roughly 4x compression; per-channel scales preserve accuracy where a single per-tensor scale destroys it.
• Pruning, distillation, and architecture choice compound with quantisation; architecture usually matters most.
• On a phone, compression decides whether the model ships at all, not just how fast it runs.
• Device fleets stay on old versions for months, so guard rails must be in the artifact and a server-side kill switch must exist.
• Operator coverage in the target runtime is the common conversion blocker.
• Without outcome telemetry you cannot measure live accuracy; design the feedback path before shipping.

🧪 Practice

1. Quantise a small trained network to int8 with per-tensor and per-channel scales, and report size, mean error, and prediction agreement for both.
2. Find the largest weight magnitude per channel in a real model and identify the channel that would dominate a per-tensor scale.
3. Estimate the memory footprint of a model you use at fp32, fp16, and int8, then decide which deployment targets it fits.
4. Design the telemetry for an on-device model that may not upload raw inputs: what would you send, and what could you still measure?
5. Interview: You shipped an on-device model and discover a serious bug two days later. What can you do? (Hint: the answer is mostly about what you built before shipping, not about what you can deploy now.)

Inference Latency Optimization

Latency work fails when it starts with the model. The discipline is the same as any performance work: measure the end-to-end budget, find the largest term, fix that, and repeat. In model serving the largest term is usually one of four things, and only one of them is the forward pass.

  WHERE INFERENCE TIME GOES, AND THE LEVER FOR EACH

    request  ->  [ deserialise ]  ->  [ fetch features ]  ->  [ forward ]
                       |                     |                     |
                 binary protocol       cache / colocate     batch, quantise,
                 smaller payload       fewer round trips    compile, smaller
                       |                     |                     |
                       +------ [ postprocess + serialise ] ---------+
                                          |
                                    avoid copies,
                                    return only what is used

    Rule: the term you can measure is the term you can fix. Instrument
    every arrow before changing anything.

The techniques, roughly in order of return on effort:

1. Batch. Fixed per-call cost is paid once per call, not once per row.
2. Cache. The fastest inference is the one you do not run.
3. Shrink the input path. Fewer feature lookups, colocated stores, one round trip instead of five.
4. Compile the model. ONNX Runtime, TensorRT, OpenVINO, or torch.compile fuse operations and remove interpreter overhead, often 2-5x on CPU.
5. Reduce precision. fp16 or int8 where accuracy permits.
6. Shrink the model. Distillation or a smaller architecture, last because it requires retraining and re-validation.

The first two are worth quantifying, because both involve a trade that is easy to get wrong in either direction.

import time
import numpy as np

rng = np.random.default_rng(11)
D, H = 256, 512
W1 = rng.normal(size=(D, H)).astype(np.float32)
W2 = rng.normal(size=(H, 1)).astype(np.float32)

def infer(x):
    return np.maximum(x @ W1, 0) @ W2

print("PER-REQUEST COST VS BATCH SIZE (SAME MODEL, SAME HARDWARE)")
print("  batch   total ms   ms/request   throughput req/s")
per_req = {}
for b in (1, 4, 16, 64, 256):
    x = rng.normal(size=(b, D)).astype(np.float32)
    infer(x)                                        # warm up caches
    t0 = time.perf_counter()
    for _ in range(200):
        infer(x)
    ms = (time.perf_counter() - t0) / 200 * 1000
    per_req[b] = ms / b
    print(f"  {b:5d}   {ms:8.3f}   {ms / b:10.4f}   {1000 / (ms / b):16,.0f}")
print("  Fixed overhead per call is amortised across the batch, and larger")
print("  matrices use the CPU's vector units and cache far better.")

# --- DYNAMIC BATCHING: TRADING A LITTLE LATENCY FOR A LOT OF THROUGHPUT. ---
# Service time of a real endpoint has two parts: a fixed cost paid once per
# CALL (deserialisation, copies, framework dispatch, GIL handoff) and a cost
# paid per ROW. Only the first is amortised by batching.
CALL_MS, ROW_MS = 0.80, 0.05

def service_ms(batch):
    return CALL_MS + ROW_MS * batch

def simulate(qps, window_ms, max_batch, n=60_000):
    arrive = np.cumsum(rng.exponential(1000 / qps, n))   # Poisson arrivals, ms
    lat, sizes, i, clock = [], [], 0, 0.0
    while i < n:
        start = max(arrive[i], clock)                    # server free, req waiting
        deadline = start + window_ms
        j = i
        while j < n and arrive[j] <= deadline and (j - i) < max_batch:
            j += 1                                       # all arrivals in window
        finish = max(start, min(deadline, arrive[j - 1])) + service_ms(j - i)
        lat.extend(finish - arrive[i:j])
        sizes.append(j - i)
        clock, i = finish, j
    return np.array(lat), np.array(sizes)

print(f"\nDYNAMIC BATCHING AT 3,000 req/s"
      f"  (unbatched capacity = {1000 / service_ms(1):,.0f} req/s)")
print("  configuration      mean batch   p50 ms   p99 ms   verdict")
configs = [("no batching", 0.0, 1)] + [(f"window {w:.0f} ms", w, 128)
                                       for w in (0.0, 1.0, 3.0, 10.0, 30.0)]
for label, w, max_batch in configs:
    lat, sizes = simulate(3000, w, max_batch)
    p50, p99 = np.percentile(lat, [50, 99])
    # If the queue is unstable, latency grows without bound over the run.
    growing = lat[-2000:].mean() > 5 * lat[:2000].mean() + 5
    print(f"  {label:16s}   {sizes.mean():10.1f}   {p50:8.2f} {p99:8.2f}"
          f"   {'QUEUE GROWING' if growing else 'stable':>14}")
print("  Serving one request at a time, the server cannot keep up at all and")
print("  latency diverges. A zero-delay batcher already helps, because")
print("  whatever queued during the last call is merged into the next one.")
print("  An explicit window trades p50 for larger batches; past a few ms it")
print("  buys throughput nobody needed and costs latency everybody notices.")

# --- CACHING: REQUEST POPULARITY IS NEVER UNIFORM. ---
print("\nCACHING PREDICTIONS UNDER A ZIPF REQUEST DISTRIBUTION")
N_KEYS = 100_000
zipf_p = 1 / np.arange(1, N_KEYS + 1) ** 1.05
zipf_p /= zipf_p.sum()
reqs = rng.choice(N_KEYS, size=500_000, p=zipf_p)
MISS_MS, HIT_MS, QPS = 25.0, 0.4, 3000
print("  cache entries   hit rate   mean latency   model calls/s left")
for size in (100, 1_000, 10_000, 50_000):
    hot = set(np.argsort(-zipf_p)[:size].tolist())       # cache the popular keys
    hit = np.isin(reqs, list(hot)).mean()
    print(f"  {size:13,}   {hit:8.1%}   {hit * HIT_MS + (1 - hit) * MISS_MS:10.2f} ms"
          f"   {QPS * (1 - hit):18,.0f}")
print("  Caching works only when the same input recurs; verify the key really")
print("  repeats before building one, and bound staleness with a TTL tied to")
print("  how fast the underlying features change.")

PER-REQUEST COST VS BATCH SIZE (SAME MODEL, SAME HARDWARE)
  batch   total ms   ms/request   throughput req/s
      1      0.012       0.0121             82,318
      4      0.057       0.0142             70,625
     16      0.083       0.0052            191,780
     64      0.193       0.0030            331,698
    256      0.557       0.0022            459,295
  Fixed overhead per call is amortised across the batch, and larger
  matrices use the CPU's vector units and cache far better.

DYNAMIC BATCHING AT 3,000 req/s  (unbatched capacity = 1,176 req/s)
  configuration      mean batch   p50 ms   p99 ms   verdict
  no batching               1.0   15387.44 30511.70    QUEUE GROWING
  window 0 ms               2.9       1.45     2.09           stable
  window 1 ms               5.5       1.71     3.15           stable
  window 3 ms              12.4       3.20     5.74           stable
  window 10 ms             37.1       8.51    15.06           stable
  window 30 ms            106.4      23.68    41.70           stable
  Serving one request at a time, the server cannot keep up at all and
  latency diverges. A zero-delay batcher already helps, because
  whatever queued during the last call is merged into the next one.
  An explicit window trades p50 for larger batches; past a few ms it
  buys throughput nobody needed and costs latency everybody notices.

CACHING PREDICTIONS UNDER A ZIPF REQUEST DISTRIBUTION
  cache entries   hit rate   mean latency   model calls/s left
            100      50.4%        12.61 ms                1,489
          1,000      68.8%         8.06 ms                  935
         10,000      85.3%         4.03 ms                  442
         50,000      95.7%         1.45 ms                  128
  Caching works only when the same input recurs; verify the key really
  repeats before building one, and bound staleness with a TTL tied to
  how fast the underlying features change.

The batch-size table shows the raw effect: the same model costs roughly 7x less per request at batch 256 than at batch 1. That is not the model getting faster -- it is the per-call overhead being divided among more rows, plus better use of vector units and cache lines.

Dynamic batching turns that into a serving strategy: hold arriving requests for a short window, then run them together. The simulation shows all three regimes. Serving strictly one request at a time, the server's capacity (1,176 req/s) is below the arrival rate (3,000 req/s), so the queue grows without bound -- the classic "it was fine in staging" failure. A zero-delay batcher fixes it without adding any deliberate wait, because requests that queued during the previous call are merged into the next one. Longer windows increase batch size and throughput, but from 3 ms onward you are buying capacity nobody needs and paying for it in p50. Choose the window from the latency budget, and set a maximum batch size so a traffic burst cannot create a batch that blows the budget on its own.

Caching is the other large lever, and its viability is a property of your traffic rather than your code. Real request distributions are usually Zipf-like: caching the 1,000 most popular keys out of 100,000 already removes two thirds of the work. Three rules keep caches honest: key on the exact feature vector (or a hash of it), never on a user id alone; set a TTL derived from how fast the underlying features change; and measure hit rate continuously, because a cache whose hit rate quietly falls to 5% is pure added latency.

Two failure modes specific to ML services deserve naming. Thread oversubscription: BLAS libraries default to one thread per host core, so concurrent requests inside one replica fight each other; pin the thread count and let the orchestrator provide parallelism. Cold start: the first request after a deploy pays for lazy imports, JIT compilation, and uninitialised caches, which can be seconds. Warm up the model in the readiness path -- run a dummy prediction before reporting ready.

Key Takeaways

• Optimise from a measured budget; the forward pass is often a minority of end-to-end latency.
• Batching amortises fixed per-call cost and improves hardware utilisation, commonly 5-10x per-request cost at large batches.
• Dynamic batching with a short window converts that into throughput; choose the window from the latency budget and cap the batch size.
• An unbatched server whose capacity is below arrival rate does not get slower gracefully -- its queue diverges.
• Caching is viable when request keys repeat; Zipf-like traffic makes small caches remove most of the work.
• Key caches on the full feature vector, give them a TTL tied to feature freshness, and alert on hit rate.
• Compilation (ONNX Runtime, TensorRT, torch.compile) and reduced precision are the next levers, before retraining a smaller model.
• Pin BLAS thread counts and warm the model during readiness to avoid oversubscription and cold-start spikes.

🧪 Practice

1. Measure per-request inference cost for your model at batch sizes 1, 8, 64, and 512, and compute the throughput gain from batching.
2. Simulate Poisson arrivals against your measured service time and find the batching window that meets a 20 ms p99 target.
3. Log one day of production request keys and compute the cache hit rate achievable with 100, 1,000, and 10,000 entries.
4. Add a warm-up prediction to your readiness probe and measure first-request latency with and without it.
5. Interview: Your p99 latency doubles every Monday at 09:00 and recovers by 09:20. What is happening and how would you confirm it? (Hint: think about what is empty at the start of a traffic surge, and what has to be recreated before the service is warm again.)

<a id="152-ml-engineering-infrastructure"></a>

15.2 ML Engineering Infrastructure

Deployment answers how a model reaches its consumers; this section covers the systems that make producing a model repeatable rather than heroic -- where features are defined and served, how model versions are tracked and promoted, how experiments are recorded, what makes a training run reproducible, how tests and delivery pipelines apply to something that is part code and part data, and how the infrastructure underneath is itself described as code.

Feature Stores

Almost every serious ML system arrives at the same two problems, and a feature store is the answer to both.

The first is training-serving skew. A feature -- "average order value over the last 30 days" -- gets implemented twice: once in SQL over the warehouse for training, once in application code against a live database for serving. The two implementations differ in some small way: a timezone, an inclusive bound, log versus log1p. The model was trained on one distribution and serves on another, and nothing reports an error.

The second is time travel. Training rows need the feature value as it was at the moment of the event, not as it is now. This is easy to say and easy to get wrong, because the natural join -- match the label table to the current feature table on user id -- silently inserts the future into the past.

A feature store is a system that owns feature definitions and serves them to both worlds:

  THE TWO HALVES OF A FEATURE STORE

    feature definition (written once)
        name: avg_order_value_30d
        entity: user_id
        source: orders
        transform: mean(amount) over 30 days
                 |
        +--------+--------+
        |                 |
   OFFLINE STORE      ONLINE STORE
   warehouse /        key-value store
   parquet, months    latest value only,
   of history         single-digit ms reads
        |                 |
   point-in-time      get_online_features(
   training set        entity=user_id)
        |                 |
     training          serving
        \_______________/
          the same definition, so the same numbers

Point-in-time correctness is the part worth seeing in code, because the wrong version looks entirely reasonable:

import numpy as np
import pandas as pd

rng = np.random.default_rng(5)
N_USERS, N_TX = 400, 6000
DAY = pd.Timedelta("1D")
T0 = pd.Timestamp("2026-01-01")

# --- EVENTS TO SCORE: A TRANSACTION, ITS TIME, AND WHAT HAPPENED. ---
tx = pd.DataFrame({
    "user_id": rng.integers(0, N_USERS, N_TX),
    "ts": T0 + pd.to_timedelta(rng.uniform(0, 90, N_TX), unit="D"),
    "amount": rng.lognormal(3.4, 0.8, N_TX),
}).sort_values("ts", ignore_index=True)
tx["ts"] = tx["ts"].dt.floor("min")                 # minute resolution, readable
risky = rng.random(N_USERS) < 0.15                  # some users are risky
p = np.where(risky[tx.user_id], 0.30, 0.02)
tx["is_fraud"] = rng.random(N_TX) < p

# --- FEATURE HISTORY: A VALUE THAT CHANGES OVER TIME. ---
# chargebacks_30d rises AFTER a fraud is discovered, roughly 10 days later.
rows = []
for uid, g in tx.groupby("user_id"):
    cb = 0
    rows.append({"user_id": uid, "ts": T0, "chargebacks_30d": 0})
    for t, f in zip(g.ts, g.is_fraud):
        if f:
            cb += 1
            rows.append({"user_id": uid, "ts": t + 10 * DAY,   # discovered later
                         "chargebacks_30d": cb})
features = pd.DataFrame(rows).sort_values("ts", ignore_index=True)

# --- JOIN 1 (WRONG): TAKE THE FEATURE'S CURRENT VALUE. ---
latest = features.sort_values("ts").groupby("user_id").tail(1)
naive = tx.merge(latest[["user_id", "chargebacks_30d"]], on="user_id", how="left")

# --- JOIN 2 (RIGHT): TAKE THE VALUE AS IT WAS AT THE EVENT'S TIMESTAMP. ---
pit = pd.merge_asof(tx.sort_values("ts"), features.sort_values("ts"),
                    on="ts", by="user_id", direction="backward")
pit["chargebacks_30d"] = pit["chargebacks_30d"].fillna(0)

def auc(score, label):
    r = pd.Series(score).rank().to_numpy()
    y = np.asarray(label, dtype=bool)
    n1, n0 = y.sum(), (~y).sum()
    return (r[y].sum() - n1 * (n1 + 1) / 2) / (n1 * n0)

print("THE SAME FEATURE, TWO JOINS, ONE OF THEM A TIME MACHINE")
print(f"  rows: {len(tx):,}   fraud rate: {tx.is_fraud.mean():.1%}")
print(f"  naive join (current value)   AUC = {auc(naive.chargebacks_30d, naive.is_fraud):.3f}"
      "   <- offline result you would report")
print(f"  point-in-time join           AUC = {auc(pit.chargebacks_30d, pit.is_fraud):.3f}"
      "   <- what production can do")
print("  The naive join tells the model about chargebacks that had not")
print("  happened yet. Nothing in the code looks wrong; the timestamps do.")

# --- WHAT THE TWO JOINS PRODUCE FOR ONE USER. ---
u = int(tx[tx.is_fraud].user_id.value_counts().index[0])
show = pit[pit.user_id == u][["ts", "amount", "is_fraud", "chargebacks_30d"]].head(6)
show = show.merge(naive[naive.user_id == u][["ts", "chargebacks_30d"]],
                  on="ts", suffixes=("_pit", "_naive"))
print(f"\nUSER {u}: FEATURE VALUE UNDER EACH JOIN")
print(show.to_string(index=False, float_format=lambda v: f"{v:8.2f}"))
print("  The naive column is constant: it is today's value, pasted onto")
print("  every historical row, including rows that caused it.")

# --- OFFLINE AND ONLINE MUST AGREE, WHICH IS THE OTHER HALF OF THE JOB. ---
print("\nTRAINING-SERVING SKEW: SAME USER, TWO CODE PATHS")
online = {"chargebacks_30d": 2.0, "amount": 41.2, "amount_log": np.log1p(41.2)}
offline = {"chargebacks_30d": 2.0, "amount": 41.2, "amount_log": np.log(41.2)}
for k in sorted(set(online) | set(offline)):
    a, b = online.get(k), offline.get(k)
    flag = "  <- SKEW" if a is not None and b is not None and abs(a - b) > 1e-9 else ""
    print(f"  {k:16s} online {a:8.4f}   offline {b:8.4f}{flag}")
print("  log1p versus log is a one-character difference that silently shifts")
print("  every prediction. One definition, one implementation, both paths.")

THE SAME FEATURE, TWO JOINS, ONE OF THEM A TIME MACHINE
  rows: 6,000   fraud rate: 6.2%
  naive join (current value)   AUC = 0.910   <- offline result you would report
  point-in-time join           AUC = 0.686   <- what production can do
  The naive join tells the model about chargebacks that had not
  happened yet. Nothing in the code looks wrong; the timestamps do.

USER 23: FEATURE VALUE UNDER EACH JOIN
                 ts   amount  is_fraud  chargebacks_30d_pit  chargebacks_30d_naive
2026-01-03 07:04:00    59.48     False                    0                     10
2026-01-08 13:18:00   144.58      True                    0                     10
2026-01-12 15:43:00     3.81      True                    0                     10
2026-01-14 21:12:00    86.53     False                    0                     10
2026-01-16 22:36:00    14.05      True                    0                     10
2026-01-19 10:03:00    24.38     False                    1                     10
  The naive column is constant: it is today's value, pasted onto
  every historical row, including rows that caused it.

TRAINING-SERVING SKEW: SAME USER, TWO CODE PATHS
  amount           online  41.2000   offline  41.2000
  amount_log       online   3.7424   offline   3.7184  <- SKEW
  chargebacks_30d  online   2.0000   offline   2.0000
  log1p versus log is a one-character difference that silently shifts
  every prediction. One definition, one implementation, both paths.

The AUC gap is the whole lesson. The naive join produces a model that looks strong offline and cannot possibly work in production, because the feature it relies on -- chargebacks recorded ten days after the transaction -- does not exist at scoring time. The per-user table shows the mechanism: today's value pasted onto every historical row, including the rows that caused it.

merge_asof is the correct primitive, and its arguments encode the rules: join on time, group by entity, and search backward only. A feature store does this for you across dozens of features with different update frequencies, which is the part that is tedious rather than difficult -- and therefore the part people skip.

The skew demonstration is the other half. Once definitions live in one place, the online and offline paths compute the same expression, and a one-character difference cannot get between them.

What a feature store gives you, beyond correctness:

Capability	What it prevents
Single definition, two paths	Training-serving skew
Point-in-time joins	Label leakage from future values
Feature versioning	Silent redefinition breaking old models
Shared catalogue	Five teams building days_since_signup five times
Freshness monitoring	Serving a feature whose pipeline died yesterday
Backfill	New feature, months of training history, one command

When do you need one? Not on day one. A single team with a batch model and one warehouse gets most of the benefit from a shared SQL module and an as_of parameter. The store earns its operational cost when you have real-time serving and offline training, or several teams reusing features, or features that are expensive to compute and worth computing once.

Two implementation notes that matter more than the vendor choice. Decide the freshness contract per feature and monitor it: a feature that should update hourly and last updated yesterday is a silent, model-wide bug. And treat feature definitions as versioned code -- changing the window of avg_order_value_30d from 30 to 28 days does not modify a feature, it creates a new one, and models trained on the old definition must either be retrained or keep reading the old version.

Key Takeaways

• Feature stores exist to solve training-serving skew and point-in-time correctness, not to be a fashionable box in an architecture diagram.
• Skew comes from computing a feature twice in two languages; one definition serving both paths removes the category of bug.
• Training rows need feature values as of the event timestamp; joining on the current value inserts the future into the past.
• merge_asof with by=entity and direction="backward" is the correct join primitive; leaky joins look completely ordinary in code.
• A leaky feature inflates offline metrics and produces a model that cannot work in production, which is worse than a weak model.
• The offline store holds history for training; the online store holds the latest value for millisecond reads.
• Give every feature a freshness contract and monitor it; a dead feature pipeline is invisible to the model.
• Feature definitions are versioned code: changing a window creates a new feature rather than editing an old one.

🧪 Practice

1. Build a labelled event table and a feature history table, then compute a training set both ways and compare a metric under each join.
2. Write a point-in-time join with merge_asof that also enforces a maximum feature age, dropping rows whose latest value is too old.
3. Take one feature from your own project and write down its definition precisely enough that two engineers would implement it identically.
4. Design freshness monitoring for three features with different update frequencies, and state what alert each one needs.
5. Interview: A model's offline AUC is 0.94 and its production AUC is 0.71. Where do you look first? (Hint: rank your candidate causes by how cheaply you can test them, and start with the timestamps in the training-set join.)

Model Registries and Versioning

model_final_v3_REALLY_final.pkl is a joke because it is a memory. The underlying problem is real: a trained model is produced many times, by several people, from changing data, and exactly one of those artifacts is answering requests right now. A model registry is the system of record that answers four questions at any moment:

1. What is in production? Not "which code" -- which artifact.
2. Where did it come from? Which commit, which data, which parameters.
3. How did it get there? Who promoted it, against which evaluation.
4. What do we roll back to? And is that artifact still loadable?

Note that version control does not answer these. Git versions code; the model is a function of code and data and hyperparameters and a random seed. Two runs of the same commit produce different artifacts, and both may be legitimate.

The registry's core discipline is immutability plus staging. A registered version is never edited: new training produces a new version. What changes is a version's stage -- none, staging, production, archived -- and every transition is logged with a reason. Promotion is guarded by rules, not by judgement in a chat thread.

import hashlib, json, time
from dataclasses import dataclass, field, asdict

@dataclass(frozen=True)
class Version:
    """Immutable record of one trained artifact and everything it came from."""
    name: str
    version: int
    uri: str                      # content-addressed, never overwritten
    git_sha: str                  # code that produced it
    data_snapshot: str            # table version or dataset hash
    params: dict
    metrics: dict
    stage: str = "none"
    created: float = field(default_factory=time.time)

class Registry:
    GATES = {"auc": 0.78, "latency_p99_ms": 40}     # absolute floors

    def __init__(self):
        self.versions, self.log = {}, []

    def register(self, name, uri, git_sha, data_snapshot, params, metrics):
        n = len([v for v in self.versions.values() if v.name == name]) + 1
        v = Version(name, n, uri, git_sha, data_snapshot, params, metrics)
        self.versions[(name, n)] = v
        self.log.append(f"registered {name} v{n} from {git_sha[:7]}")
        return v

    def production(self, name):
        return next((v for v in self.versions.values()
                     if v.name == name and v.stage == "production"), None)

    def promote(self, name, version, to):
        v = self.versions[(name, version)]
        reasons = []
        if v.metrics.get("auc", 0) < self.GATES["auc"]:
            reasons.append(f"auc {v.metrics['auc']:.3f} below floor "
                           f"{self.GATES['auc']}")
        if v.metrics.get("latency_p99_ms", 1e9) > self.GATES["latency_p99_ms"]:
            reasons.append(f"p99 {v.metrics['latency_p99_ms']} ms over budget")
        cur = self.production(name)
        if to == "production" and cur and v.metrics["auc"] < cur.metrics["auc"]:
            reasons.append(f"auc {v.metrics['auc']:.3f} worse than live "
                           f"v{cur.version} ({cur.metrics['auc']:.3f})")
        if not v.data_snapshot or not v.git_sha:
            reasons.append("incomplete lineage")
        if reasons:
            self.log.append(f"BLOCKED {name} v{version} -> {to}: "
                            + "; ".join(reasons))
            return False
        if to == "production" and cur:                # only one at a time
            self.versions[(name, cur.version)] = replace_stage(cur, "archived")
            self.log.append(f"archived {name} v{cur.version}")
        self.versions[(name, version)] = replace_stage(v, to)
        self.log.append(f"promoted {name} v{version} -> {to}")
        return True

    def rollback(self, name, to_version):
        cur = self.production(name)
        if cur:
            self.versions[(name, cur.version)] = replace_stage(cur, "archived")
        self.versions[(name, to_version)] = replace_stage(
            self.versions[(name, to_version)], "production")
        self.log.append(f"ROLLBACK {name} -> v{to_version} "
                        f"(was v{cur.version if cur else '-'})")

def replace_stage(v, stage):
    d = asdict(v) | {"stage": stage}
    return Version(**d)

reg = Registry()
runs = [
    ("s3://models/fraud/9a1c", "4f2b9e1c", "orders@2026-08-01",
     {"depth": 6, "lr": 0.08}, {"auc": 0.802, "latency_p99_ms": 31}),
    ("s3://models/fraud/1de7", "7c19aa03", "orders@2026-09-01",
     {"depth": 9, "lr": 0.05}, {"auc": 0.771, "latency_p99_ms": 28}),
    ("s3://models/fraud/55b2", "7c19aa03", "orders@2026-09-01",
     {"depth": 8, "lr": 0.05}, {"auc": 0.818, "latency_p99_ms": 52}),
    ("s3://models/fraud/e004", "b2d40f77", "orders@2026-09-20",
     {"depth": 8, "lr": 0.05}, {"auc": 0.826, "latency_p99_ms": 33}),
]
for uri, sha, snap, params, metrics in runs:
    reg.register("fraud", uri, sha, snap, params, metrics)

for n in (1, 2, 3, 4):
    reg.promote("fraud", n, "production")
reg.rollback("fraud", 1)                             # incident: v4 misbehaves

print("REGISTRY TRANSITION LOG")
for line in reg.log:
    print(f"  {line}")

print("\nREGISTRY STATE")
print("  version   stage        auc    p99    git        data snapshot")
for (_, n), v in sorted(reg.versions.items()):
    print(f"  v{v.version:<8} {v.stage:11s} {v.metrics['auc']:.3f}"
          f" {v.metrics['latency_p99_ms']:5d}   {v.git_sha[:7]}   {v.data_snapshot}")

live = reg.production("fraud")
print(f"\nSERVING NOW: v{live.version} at {live.uri}")
print("  Every field needed to reproduce or audit this prediction is here:")
print("  the artifact, the code that built it, the data it saw, its")
print("  hyperparameters, its measured metrics, and how it got promoted.")

fingerprint = hashlib.sha256(
    json.dumps({"uri": live.uri, "git": live.git_sha,
                "data": live.data_snapshot}, sort_keys=True).encode()).hexdigest()
print(f"  lineage fingerprint: {fingerprint[:16]}  (log this with predictions)")

REGISTRY TRANSITION LOG
  registered fraud v1 from 4f2b9e1
  registered fraud v2 from 7c19aa0
  registered fraud v3 from 7c19aa0
  registered fraud v4 from b2d40f7
  promoted fraud v1 -> production
  BLOCKED fraud v2 -> production: auc 0.771 below floor 0.78; auc 0.771 worse than live v1 (0.802)
  BLOCKED fraud v3 -> production: p99 52 ms over budget
  archived fraud v1
  promoted fraud v4 -> production
  ROLLBACK fraud -> v1 (was v4)

REGISTRY STATE
  version   stage        auc    p99    git        data snapshot
  v1        production  0.802    31   4f2b9e1   orders@2026-08-01
  v2        none        0.771    28   7c19aa0   orders@2026-09-01
  v3        none        0.818    52   7c19aa0   orders@2026-09-01
  v4        archived    0.826    33   b2d40f7   orders@2026-09-20

SERVING NOW: v1 at s3://models/fraud/9a1c
  Every field needed to reproduce or audit this prediction is here:
  the artifact, the code that built it, the data it saw, its
  hyperparameters, its measured metrics, and how it got promoted.
  lineage fingerprint: 1e2ed102ac3d8eb8  (log this with predictions)

Read the transition log as an audit trail, which is exactly what it is. v2 was blocked twice over: it failed the absolute quality floor and was worse than the live model. v3 passed on quality and failed on latency -- a reminder that "better" is multi-dimensional and the registry is where those dimensions get enforced. v4 passed and displaced v1, which was archived rather than deleted, which is what made the rollback possible thirty seconds later.

Four properties of that design are worth adopting whatever tool you use (MLflow Model Registry, Weights & Biases, SageMaker, Vertex, or a table you write yourself):

• Content-addressed artifact URIs. s3://models/fraud/9a1c, not s3://models/fraud/latest. A pointer that can be overwritten means a restarted pod can silently change models.
• Lineage is mandatory, not optional metadata. Code commit and data snapshot must be present for promotion to be allowed; without them the version cannot be reproduced or audited.
• Gates are code. Absolute floors plus a comparison against the incumbent, evaluated automatically. A human approval step can sit after the gates, never instead of them.
• Archive, never delete. Rollback requires the previous artifact to exist and still load, which is worth testing periodically rather than assuming.

Versioning in ML has one more dimension than software versioning, and being explicit about it prevents confusion:

What is versioned	Mechanism	Changes when
Code	Git commit	Someone edits the pipeline
Data	Snapshot, table version, DVC hash	The source is refreshed
Features	Definition version in the feature store	A window or formula changes
Model artifact	Registry version	Any training run completes
Deployment	Image digest plus config	Anything is released

The last line ties back to serving: log the model version and the lineage fingerprint with every prediction. When someone asks in six months why a particular customer was declined, that logged fingerprint is the difference between an answer and an apology. In regulated settings it is not optional.

Key Takeaways

• A registry answers what is live, where it came from, how it got promoted, and what to roll back to.
• Git versions code, but a model is code plus data plus parameters plus seed, so code versioning alone cannot identify an artifact.
• Registered versions are immutable; retraining creates a new version and only the stage changes.
• Promotion gates belong in code: absolute floors plus a comparison with the incumbent on every dimension that matters, including latency.
• Use content-addressed artifact URIs; a mutable "latest" pointer lets a restart change the model silently.
• Require lineage (commit, data snapshot, parameters) for promotion, or reproduction and audit become impossible.
• Archive rather than delete, and verify periodically that the rollback target still loads.
• Log the model version and lineage fingerprint with every prediction.

🧪 Practice

1. Implement a minimal registry with immutable versions, stages, and a promotion gate, then show it blocking a worse model.
2. Add a rule that blocks promotion if the candidate was trained on an older data snapshot than the incumbent, and test it.
3. Take your last three trained models and write down, for each, the exact commit and data snapshot used; note which ones you cannot reconstruct.
4. Design the prediction log schema for an audit that must reconstruct any decision made in the past two years.
5. Interview: Production is serving a model nobody can reproduce, and the person who trained it has left. What do you do, in what order? (Hint: separate "make the current state safe and documented" from "recreate the artifact", and consider which one is actually urgent.)

Experiment Tracking

Model development is a search. You try a feature set, a model family, a learning rate; you look at a number; you try again. Fifty runs later somebody asks which configuration produced the result in the slide, and the honest answer is a shrug. Experiment tracking is the practice of recording every run automatically, so that the search is a dataset rather than a memory.

A run record has four parts, and leaving any of them out makes the rest less useful:

Part	Examples	Why it is needed
Parameters	depth, learning rate, seed, feature set	Reproduce and compare
Metrics	val AUC, test AUC, training time, cost	Judge the run
Context	git commit, dataset version, environment	Trace back to inputs
Artifacts	model file, plots, confusion matrix	Inspect without rerunning

The temptation is to log only the winner. Resist it: the value of tracking comes from the population of runs, which supports questions a single run cannot answer -- which hyperparameter actually matters, how much variance a seed contributes, whether last quarter's result is still reproducible.

import itertools, json, time
import numpy as np

rng = np.random.default_rng(42)

class Tracker:
    """The smallest experiment tracker that is still useful."""
    def __init__(self):
        self.runs = []

    def log(self, params, metrics, tags):
        self.runs.append({"run_id": f"r{len(self.runs):03d}",
                          "params": params, "metrics": metrics, "tags": tags,
                          "ts": time.time()})

    def best(self, metric, k=5):
        return sorted(self.runs, key=lambda r: -r["metrics"][metric])[:k]

# --- A SWEEP: EVERY RUN IS RECORDED, NOT JUST THE ONE THAT WON. ---
tr = Tracker()
TRUE = {"depth": {4: 0.780, 6: 0.800, 8: 0.802, 10: 0.801},   # real effect
        "lr": {0.01: 0.795, 0.05: 0.797, 0.2: 0.796}}         # near-none
for depth, lr, seed in itertools.product([4, 6, 8, 10], [0.01, 0.05, 0.2],
                                         range(8)):
    true_auc = TRUE["depth"][depth] + TRUE["lr"][lr] - 0.796
    val = true_auc + rng.normal(0, 0.012)         # small validation set noise
    test = true_auc + rng.normal(0, 0.012)        # independent, unseen
    tr.log({"depth": depth, "lr": lr, "seed": seed},
           {"val_auc": round(val, 4), "test_auc": round(test, 4)},
           {"dataset": "orders@2026-09-01", "git": "7c19aa0"})

print(f"LOGGED {len(tr.runs)} RUNS")
print("  run    depth     lr   seed   val_auc   test_auc")
for r in tr.best("val_auc", 5):
    p, m = r["params"], r["metrics"]
    print(f"  {r['run_id']}   {p['depth']:5d} {p['lr']:6.2f} {p['seed']:6d}"
          f"   {m['val_auc']:7.4f}   {m['test_auc']:8.4f}")

# --- WHAT THE SWEEP ACTUALLY LEARNED, WHICH IS NOT THE TOP ROW. ---
print("\nMARGINAL EFFECT OF EACH HYPERPARAMETER (MEAN OVER ALL OTHER SETTINGS)")
for key in ("depth", "lr"):
    vals = sorted({r["params"][key] for r in tr.runs})
    print(f"  {key:6s}" + "".join(f"{v:>10}" for v in vals))
    means = [np.mean([r["metrics"]["val_auc"] for r in tr.runs
                      if r["params"][key] == v]) for v in vals]
    print("        " + "".join(f"{m:10.4f}" for m in means))
print("  depth matters and saturates after 6; the lr spread is smaller than")
print("  the per-run noise (sd 0.012), so it is not evidence of anything.")
print("  A single best run cannot tell you this; the population of runs can.")

# --- THE WINNER'S CURSE: SELECTING ON VALIDATION OVERFITS VALIDATION. ---
print("\nSELECTING THE BEST OF N RUNS BY VALIDATION SCORE")
print("  runs considered   best val_auc   its test_auc   optimism")
for n in (1, 5, 20, 96):
    picks = []
    for _ in range(3000):                          # repeat the whole selection
        sub = rng.choice(len(tr.runs), size=n, replace=False)
        i = max(sub, key=lambda j: tr.runs[j]["metrics"]["val_auc"])
        picks.append((tr.runs[i]["metrics"]["val_auc"],
                      tr.runs[i]["metrics"]["test_auc"]))
    v, t = np.mean([p[0] for p in picks]), np.mean([p[1] for p in picks])
    print(f"  {n:15d}   {v:12.4f}   {t:12.4f}   {v - t:+8.4f}")
print("  The more runs you compare, the more the winner's validation score is")
print("  luck. Report the held-out test score of the selected run, once.")

print("\nWHAT EVERY RUN RECORD MUST CONTAIN")
print(json.dumps({k: v for k, v in tr.best("val_auc", 1)[0].items()
                  if k != "ts"}, indent=2)[:420])
print("  Params, metrics, dataset version, and code version. Without the last")
print("  two, a run is a number with no way back to what produced it.")

LOGGED 96 RUNS
  run    depth     lr   seed   val_auc   test_auc
  r081      10   0.05      1    0.8275     0.7921
  r069       8   0.20      5    0.8260     0.8370
  r059       8   0.05      3    0.8222     0.8001
  r095      10   0.20      7    0.8217     0.7988
  r063       8   0.05      7    0.8206     0.7887

MARGINAL EFFECT OF EACH HYPERPARAMETER (MEAN OVER ALL OTHER SETTINGS)
  depth          4         6         8        10
            0.7829    0.7966    0.8016    0.8016
  lr          0.01      0.05       0.2
            0.7918    0.7996    0.7957
  depth matters and saturates after 6; the lr spread is smaller than
  the per-run noise (sd 0.012), so it is not evidence of anything.
  A single best run cannot tell you this; the population of runs can.

SELECTING THE BEST OF N RUNS BY VALIDATION SCORE
  runs considered   best val_auc   its test_auc   optimism
                1         0.7956         0.7950    +0.0006
                5         0.8116         0.8013    +0.0103
               20         0.8214         0.8037    +0.0177
               96         0.8275         0.7921    +0.0354
  The more runs you compare, the more the winner's validation score is
  luck. Report the held-out test score of the selected run, once.

WHAT EVERY RUN RECORD MUST CONTAIN
{
  "run_id": "r081",
  "params": {
    "depth": 10,
    "lr": 0.05,
    "seed": 1
  },
  "metrics": {
    "val_auc": 0.8275,
    "test_auc": 0.7921
  },
  "tags": {
    "dataset": "orders@2026-09-01",
    "git": "7c19aa0"
  }
}
  Params, metrics, dataset version, and code version. Without the last
  two, a run is a number with no way back to what produced it.

The marginal-effect table is the first payoff. Reading the leaderboard, you would conclude that depth=10, lr=0.05 is the answer. Averaging over the whole grid shows something more useful and more durable: depth matters up to 6 and then saturates, while the learning-rate spread is smaller than the noise between seeds. That conclusion survives the next dataset; the winning row does not.

The winner's-curse table is the second, and it is the part most teams learn late. When runs differ only by noise, the maximum of N validation scores is biased upward, and the bias grows with N. Selecting the best of 96 runs here produces a validation score 0.035 too optimistic -- larger than most reported model improvements. Three defences follow directly:

• Select on validation, report on test, and touch the test set once.
• Repeat seeds and compare configurations by their mean, not by their best run.
• Treat a small improvement from a large sweep as unproven until it survives a fresh split or a live test.

Practical habits that make tracking worth its cost:

• Log automatically, from inside the training code. Anything requiring a person to remember will be skipped on the run that matters.
• Log the dataset version and the commit, or runs cannot be reproduced.
• Log failures too. "This configuration diverges" is a result, and rediscovering it three months later is a waste.
• Name experiments after the question, not after the model. 2026-09 does-adding-device-features-help beats xgb_test7.
• Record cost and training time alongside accuracy; a 0.2% gain for four times the training cost is a decision, and it needs both numbers.

Tools (MLflow, Weights & Biases, Neptune, Comet, or a database table) differ in the interface, not in the idea. The minimum viable version is the class in the example plus a JSON file per run in object storage.

Key Takeaways

• Tracking turns model development from a memory into a queryable dataset of runs.
• A run record needs parameters, metrics, context (commit, dataset), and artifacts; missing context makes the rest unreproducible.
• Log every run, including failures, automatically from inside the training code.
• The population of runs answers which hyperparameters matter; the leaderboard's top row does not.
• Selecting the best of N runs biases the validation score upward, and the bias grows with N.
• Select on validation, report on test, and treat small gains from big sweeps as unproven.
• Repeat seeds and compare configurations by their means when differences are near the noise level.
• Record cost and runtime with accuracy so the trade is visible at decision time.

🧪 Practice

1. Instrument a training script so every run writes parameters, metrics, commit, and dataset version to a file without any manual step.
2. Run a small grid with three seeds per configuration and compute each hyperparameter's marginal effect and the seed-to-seed standard deviation.
3. Simulate selecting the best of N runs where all configurations are equally good, and plot the optimism as a function of N.
4. Take a recent experiment of yours and try to reproduce its number from the record alone; list everything that was missing.
5. Interview: A colleague reports that their new model improves validation AUC from 0.802 to 0.818 after a 400-run sweep. What do you ask? (Hint: one question is about the test set, one is about seeds, and one is about how many other things were also tried.)

Reproducible Training Pipelines

Reproducibility is not a virtue for its own sake. It is the property that makes three ordinary requests answerable: "why is this quarter's model worse than last quarter's?", "can you retrain it with the fraud cases included?", and "the regulator wants to know how this decision was made". Without it, each of those becomes a research project.

A training pipeline is reproducible when the same inputs produce the same artifact. That requires nailing down more than most people expect:

Source of variation	How it bites	Control
Data snapshot	Table refreshed between runs	Versioned snapshot or hash
Row order	Upstream query without ORDER BY	Deterministic sort key
Split assignment	Reshuffles when data grows	Hash of a stable key
Sampling, init, dropout	Different draws	Seed every RNG in use
Thread count	Changes summation order	Pin thread environment variables
Library versions	Changed defaults, changed kernels	Lock file, container digest
Hardware	Different GPU kernels, different FMA	Record it; do not assume it away
Wall-clock logic	today() inside a feature	Pass the run date as a parameter

Two of those rows are worth seeing, because they are the ones people believe they have already handled.

import hashlib
import numpy as np

# --- SPLIT STABILITY: THE SAME CODE, A WEEK LATER, DIFFERENT SPLIT. ---
users_v1 = [f"user-{i}" for i in range(10_000)]
users_v2 = users_v1 + [f"user-{i}" for i in range(10_000, 12_000)]  # a week of growth

def random_split(users, seed=0, frac=0.8):
    rng = np.random.default_rng(seed)               # seeded, and still unstable
    idx = rng.permutation(len(users))
    train = set(np.array(users)[idx[:int(frac * len(users))]])
    return {u: ("train" if u in train else "test") for u in users}

def hash_split(users, salt="split-v1", frac=0.8):
    out = {}
    for u in users:                                 # depends on the KEY only
        h = int(hashlib.md5(f"{salt}:{u}".encode()).hexdigest()[:8], 16)
        out[u] = "train" if (h % 10_000) / 10_000 < frac else "test"
    return out

reordered = list(np.random.default_rng(1).permutation(users_v1))  # no ORDER BY
print("USERS THAT SWITCHED BETWEEN TRAIN AND TEST, SAME CODE AND SEED")
print("  split strategy      2,000 new rows arrive   upstream returns rows"
      " in a new order")
for name, fn in [("seeded shuffle", random_split), ("hash of user id", hash_split)]:
    base = fn(users_v1)
    grown = fn(users_v2)
    shuffled = fn(reordered)
    g = sum(1 for u in users_v1 if base[u] != grown[u]) / len(users_v1)
    r = sum(1 for u in users_v1 if base[u] != shuffled[u]) / len(users_v1)
    print(f"  {name:18s} {g:21.1%} {r:33.1%}")
print("  A seeded shuffle is reproducible only for a FIXED dataset. Hashing")
print("  the key is reproducible for a growing one, which is the real case.")

# --- SEEDS ARE NECESSARY AND NOT SUFFICIENT. ---
rng = np.random.default_rng(0)
x = rng.normal(size=1_000_000).astype(np.float32)
serial = np.float32(0)
for v in x[:200_000]:                               # strict left-to-right
    serial += v
chunked = x[:200_000].reshape(8, -1).sum(axis=1).sum()   # 8 partial sums
print(f"\nFLOAT ADDITION IS NOT ASSOCIATIVE")
print(f"  serial sum      {serial:.8f}")
print(f"  8 partial sums  {chunked:.8f}")
print(f"  difference      {abs(serial - chunked):.8f}"
      "  <- same data, same seed, different order")
print("  Thread counts, GPU kernels, and library versions all change summation")
print("  order. 'Set the seed' controls sampling, not arithmetic.")

# --- WHAT REPRODUCIBILITY ACTUALLY REQUIRES: A FINGERPRINT PER STEP. ---
def fp(*parts):
    return hashlib.sha256("|".join(map(str, parts)).encode()).hexdigest()[:10]

pipeline = [
    ("load_raw",      {"snapshot": "orders@2026-09-01"}, "def load_raw(): ..."),
    ("clean",         {"drop_nulls": True},              "def clean(df): ..."),
    ("features",      {"window_days": 30},               "def features(df): ..."),
    ("train",         {"depth": 8, "lr": 0.05, "seed": 7}, "def train(X, y): ..."),
    ("evaluate",      {"split": "time", "cutoff": "2026-08-15"}, "def eval(m): ..."),
]

def run(pipeline, cache):
    upstream, fired = "root", []
    for name, params, code in pipeline:
        key = fp(name, sorted(params.items()), code, upstream)
        hit = key in cache
        if not hit:
            cache.add(key)
            fired.append(name)
        upstream = key                              # each step depends on the last
    return fired

cache = set()
print("\nCONTENT-ADDRESSED PIPELINE CACHE")
print(f"  first run             re-executes: {run(pipeline, cache)}")
print(f"  unchanged rerun       re-executes: {run(pipeline, cache) or ['nothing']}")
changed = [(n, dict(p, **({"lr": 0.08} if n == "train" else {})), c)
           for n, p, c in pipeline]
print(f"  learning rate changed re-executes: {run(changed, cache)}")
newdata = [(n, dict(p, **({"snapshot": "orders@2026-09-08"} if n == "load_raw"
                          else {})), c) for n, p, c in pipeline]
print(f"  new data snapshot     re-executes: {run(newdata, cache)}")
print("  Downstream steps rerun because their fingerprint includes the")
print("  upstream one. This is what makes a pipeline both fast and honest:")
print("  nothing is reused unless every input that produced it is identical.")

USERS THAT SWITCHED BETWEEN TRAIN AND TEST, SAME CODE AND SEED
  split strategy      2,000 new rows arrive   upstream returns rows in a new order
  seeded shuffle                      5.0%                             32.1%
  hash of user id                     0.0%                              0.0%
  A seeded shuffle is reproducible only for a FIXED dataset. Hashing
  the key is reproducible for a growing one, which is the real case.

FLOAT ADDITION IS NOT ASSOCIATIVE
  serial sum      26.13436699
  8 partial sums  26.13507843
  difference      0.00071144  <- same data, same seed, different order
  Thread counts, GPU kernels, and library versions all change summation
  order. 'Set the seed' controls sampling, not arithmetic.

CONTENT-ADDRESSED PIPELINE CACHE
  first run             re-executes: ['load_raw', 'clean', 'features', 'train', 'evaluate']
  unchanged rerun       re-executes: ['nothing']
  learning rate changed re-executes: ['train', 'evaluate']
  new data snapshot     re-executes: ['load_raw', 'clean', 'features', 'train', 'evaluate']
  Downstream steps rerun because their fingerprint includes the
  upstream one. This is what makes a pipeline both fast and honest:
  nothing is reused unless every input that produced it is identical.

The split. A seeded shuffle is reproducible only over a fixed dataset. Add a week of users and 5% of the old ones silently change sides; let an upstream query return rows in a different order and a third of them do. Once a user who was in test is in train, every comparison across those two runs is invalid, and nothing anywhere reports an error. Hashing a stable key gives an assignment that depends only on the key -- stable under growth, reordering, and reruns, and identical in SQL, Python, and Spark. Include a salt so you can deliberately create a fresh split when you want one.

The seed. Seeding is necessary and not sufficient. Floating-point addition is not associative, so the order in which partial sums are combined changes the result in the seventh decimal. Change the thread count, the GPU, or the library version, and summation order changes with it. Those differences are usually irrelevant -- and occasionally they flip a threshold comparison or a tie-break. The practical rule: aim for bitwise reproducibility on identical hardware and environment, and for statistical reproducibility (the same conclusions within a stated tolerance) across environments. Pin what you can, record what you cannot, and assert on tolerances rather than on exact equality.

Fingerprinting steps is what makes a reproducible pipeline usable day to day. Each step's cache key combines its code, its parameters, and its upstream key, so an unchanged rerun does nothing, a hyperparameter change reruns training and evaluation only, and a new data snapshot correctly invalidates everything. This is the mechanism behind Make, DVC, Dagster, Airflow's data-aware scheduling, and Kubeflow -- and it is worth understanding independently of any of them, because a hand-rolled version in fifty lines is often enough.

Practices that follow from all of this:

• Parameterise the run date. A feature computed with today() produces a different training set every day, and backfills are impossible.
• Put configuration in files, not in code. One config per run, committed or logged with the run.
• Make steps idempotent and rerunnable. Writing to a partition keyed by the run date beats appending to a table.
• Separate the pipeline from the notebook. A notebook is where you decide what the pipeline should do; it is not the pipeline.

Key Takeaways

• Reproducibility exists so that comparison, retraining, and audit are routine rather than heroic.
• Data version, row order, split assignment, seeds, thread counts, library versions, and hardware all change results.
• A seeded shuffle is stable only for a fixed dataset; hashing a stable key survives growth and reordering.
• Once rows migrate between train and test, cross-run comparisons are invalid and nothing raises an error.
• Seeds control sampling, not arithmetic; float addition is not associative, so thread count and hardware change results slightly.
• Target bitwise reproducibility within one environment and statistical reproducibility across environments, with explicit tolerances.
• Fingerprint each step by code, parameters, and upstream key so caching is both fast and correct.
• Parameterise the run date, keep configuration in files, and make steps idempotent.

🧪 Practice

1. Replace a random train/test split with a hash-based one and verify that assignments are unchanged after adding rows and reordering them.
2. Run the same training twice with different thread counts and report the largest difference in predictions.
3. Write a fingerprint function for a three-step pipeline and demonstrate which steps rerun for each kind of change.
4. Find every use of the current date or time in one of your pipelines and convert them into explicit parameters.
5. Interview: Your model scored 0.81 AUC in March and the same code scores 0.78 today. List the possible causes in the order you would check them. (Hint: only one of them is "the model got worse"; most involve something that changed underneath the code.)

CI/CD for Machine Learning

Continuous integration for ordinary software asks one question: does the code still do what the tests say it should? For machine learning there are three inputs -- code, data, and the trained artifact -- and each needs its own answer. A green test suite that only covers the code proves very little about the model.

The pipeline has more stages than a web service's, and they run at different times:

  WHAT RUNS WHEN

  on every commit           on data arrival           on model candidate
  -----------------         ----------------          ------------------
  lint, type check          schema validation         behavioural tests
  unit tests (features)     null/range/volume         metric gates
  pipeline smoke test       distribution checks       fairness slices
  (tiny sample, minutes)    freshness                 latency + memory
        |                         |                         |
        +-------------------------+-------------------------+
                                  |
                          registry: register version
                                  |
                          promote -> staging
                                  |
                       shadow / canary (see 15.3)
                                  |
                              production

The distinctively ML part is the middle column of tests. Model quality is not binary, so "does it pass?" has to be turned into explicit, arguable claims. Four kinds are standard, and the last two are the ones teams usually lack:

Test type	Question	Example
Unit	Is this transformation correct?	days_since() handles month ends
Data	Is the input sane?	No nulls in amount, volume within 20%
Metric gate	Is it good enough?	AUC above floor and near the incumbent
Behavioural	Does it behave sensibly?	New device must not reduce fraud risk

Behavioural tests deserve the emphasis. They come in three flavours, borrowed from the CheckList methodology: minimum functionality (obvious cases the model must get right), invariance (changes that must not alter the prediction), and directional expectation (changes that must move the prediction a known way). They encode domain knowledge as executable claims, which is exactly what an aggregate metric cannot do.

import time
import numpy as np

rng = np.random.default_rng(13)
FEATURES = ["amount", "hour", "n_prior", "device_is_new"]

def make_model(weights):
    w = np.array(weights, dtype=float)
    def predict(rows):                              # rows: (n, 4) array
        return 1 / (1 + np.exp(-(rows @ w - 3.0)))
    return predict

# The candidate has a sign error on device_is_new: a new device now LOWERS
# risk. Offline AUC barely notices, because the feature is rare.
good = make_model([0.010, 0.02, -0.15, 1.50])
candidate = make_model([0.010, 0.02, -0.15, -1.50])

def golden_set(n=4000):
    x = np.column_stack([rng.lognormal(3.6, 0.9, n), rng.integers(0, 24, n),
                         rng.poisson(3, n), (rng.random(n) < 0.01).astype(float)])
    logit = 0.010 * x[:, 0] + 0.02 * x[:, 1] - 0.15 * x[:, 2] + 1.50 * x[:, 3] - 3
    y = rng.random(n) < 1 / (1 + np.exp(-logit))
    return x, y

X, y = golden_set()

def auc(s, label):
    order = np.argsort(s)
    r = np.empty(len(s)); r[order] = np.arange(1, len(s) + 1)
    n1, n0 = label.sum(), (~label).sum()
    return (r[label].sum() - n1 * (n1 + 1) / 2) / (n1 * n0)

# --- THE TEST SUITE THAT RUNS ON EVERY MODEL CANDIDATE. ---
def test_minimum_functionality(m):
    """Obvious cases the model must get right."""
    tiny = np.array([[5.0, 14, 40, 0]])             # small, known, many priors
    huge = np.array([[9000.0, 3, 0, 1]])            # large, 3 a.m., new device
    return float(m(tiny)[0]) < 0.2 and float(m(huge)[0]) > 0.8

def test_directional_new_device(m, n=500):
    """Everything else equal, a new device must not REDUCE risk."""
    base = X[:n].copy(); base[:, 3] = 0
    alt = base.copy();   alt[:, 3] = 1
    return bool(np.all(m(alt) >= m(base) - 1e-9))

def test_invariance_row_order(m):
    """Predictions must not depend on the order rows arrive in."""
    idx = rng.permutation(len(X))
    return bool(np.allclose(m(X)[idx], m(X[idx])))

def test_performance_floor(m, floor=0.65):
    return auc(m(X), y) >= floor

def test_no_regression(m, live, margin=0.02):   # run-to-run tolerance
    return auc(m(X), y) >= auc(live(X), y) - margin

def test_latency(m, budget_ms=5.0):
    one = X[:1]
    t0 = time.perf_counter()
    for _ in range(2000):
        m(one)
    return (time.perf_counter() - t0) / 2000 * 1000 < budget_ms

def test_schema(m):
    """The model must reject the wrong number of columns, not guess."""
    try:
        m(np.zeros((1, len(FEATURES) - 1)))
        return False
    except ValueError:
        return True

SUITE = [("minimum functionality", test_minimum_functionality),
         ("directional: new device", test_directional_new_device),
         ("invariance: row order", test_invariance_row_order),
         ("performance floor", test_performance_floor),
         ("no regression vs live", lambda m: test_no_regression(m, good)),
         ("latency budget", test_latency),
         ("schema rejection", test_schema)]

for label, model in [("LIVE MODEL", good), ("CANDIDATE (sign error)", candidate)]:
    print(f"{label}   golden-set AUC = {auc(model(X), y):.4f}")
    failed = 0
    for name, fn in SUITE:
        ok = fn(model)
        failed += not ok
        print(f"  [{'PASS' if ok else 'FAIL'}] {name}")
    print(f"  -> {'promote' if not failed else f'BLOCKED, {failed} test(s) failed'}\n")

print("The candidate's AUC is 0.016 lower -- inside the 0.02 tolerance teams")
print("use for run-to-run noise, because the broken feature is rare. Every")
print("metric gate passes. Only the behavioural test, which states the")
print("expectation in words a human can argue about, catches the sign error.")

LIVE MODEL   golden-set AUC = 0.6904
  [PASS] minimum functionality
  [PASS] directional: new device
  [PASS] invariance: row order
  [PASS] performance floor
  [PASS] no regression vs live
  [PASS] latency budget
  [PASS] schema rejection
  -> promote

CANDIDATE (sign error)   golden-set AUC = 0.6747
  [PASS] minimum functionality
  [FAIL] directional: new device
  [PASS] invariance: row order
  [PASS] performance floor
  [PASS] no regression vs live
  [PASS] latency budget
  [PASS] schema rejection
  -> BLOCKED, 1 test(s) failed

The candidate's AUC is 0.016 lower -- inside the 0.02 tolerance teams
use for run-to-run noise, because the broken feature is rare. Every
metric gate passes. Only the behavioural test, which states the
expectation in words a human can argue about, catches the sign error.

That is the argument for behavioural tests in one run. A sign error on a rare but important feature moved AUC by 0.016 -- comfortably inside the tolerance any team would set for run-to-run variation. Every metric gate passed. The only thing that caught it was a test asserting something a human believes about the world: a brand-new device does not make a transaction safer.

The delivery half is more mundane, and mostly consists of not skipping steps:

# .github/workflows/model-ci.yml (shape, not a full file)
on: [pull_request]
jobs:
  code:
    steps:
      - run: ruff check . && mypy src/
      - run: pytest tests/unit -q                 # feature logic, seconds
      - run: pytest tests/pipeline -q --sample 1000   # end-to-end, minutes
  model:
    needs: code
    steps:
      - run: python -m pipeline.train --config configs/ci.yaml --seed 7
      - run: pytest tests/behavioural -q          # invariance, directional
      - run: python -m pipeline.gate              # metric + latency floors
      - run: python -m pipeline.register --stage staging

Four rules that separate a useful ML pipeline from a slow one:

• Tier the tests by cost. Unit tests on every commit, a small-sample pipeline run on every pull request, a full training run nightly or on demand. If CI takes two hours, people will stop running it.
• Version the test data. A golden evaluation set must be fixed and checked in (or snapshotted), or your gates move on their own.
• Deploy the artifact, not the notebook. The thing that passed the gates is the thing that ships, by digest.
• Automate promotion, keep a human for the decision. Gates run automatically; a person can still be required to approve production, but only after everything mechanical has passed.

Continuous training extends the same pipeline to run on a schedule or a trigger (drift detected, enough new labels accumulated). It is only safe if the gates are strong -- automated retraining without behavioural tests and metric floors is an automated way to deploy a bad model at 3 a.m.

Key Takeaways

• ML CI must cover code, data, and the trained artifact; code tests alone prove almost nothing about model quality.
• Metric gates check magnitude; behavioural tests check meaning, and only the latter catches a plausible-looking sign error.
• Use minimum functionality, invariance, and directional expectation tests to encode domain knowledge as assertions.
• Tier tests by cost: fast unit tests per commit, sampled pipeline runs per PR, full training on a schedule.
• Fix and version the golden evaluation set, or gates drift silently.
• Promote the artifact that passed the gates, by digest, rather than retraining at deploy time.
• Automate gates; a human approval may follow them but must not replace them.
• Continuous training is safe only in proportion to the strength of those gates.

🧪 Practice

1. Write three behavioural tests for a model you know: one minimum functionality, one invariance, one directional expectation.
2. Add a metric gate that compares a candidate against the live model with an explicit tolerance, and justify the tolerance you chose.
3. Introduce a deliberate sign error into a feature and check which of your existing tests catch it; add one that does if none do.
4. Split your test suite into per-commit, per-PR, and nightly tiers and measure the runtime of each tier.
5. Interview: How would you design CI for a pipeline whose full training run takes six hours? (Hint: think about what a pull request actually needs to prove, and what can be proven on a sample or deferred.)

Infrastructure as Code Basics

Data scientists inherit infrastructure whether they want it or not: a training cluster, a bucket, a scheduled job, a serving deployment, a monitoring alert. The question is only whether that infrastructure exists as a sequence of things somebody clicked or as a file in the repository.

Infrastructure as code (IaC) means the second. The configuration is declarative -- it states the desired end state rather than the steps to reach it -- and a tool computes the difference between that state and reality, then reconciles them. Terraform, Pulumi, CloudFormation, and Kubernetes manifests are all instances of the same loop:

  THE RECONCILIATION LOOP

    desired state (in git)  ----+
                                |
                             [ plan ]  <---- actual state (the real world)
                                |
                        list of differences
                                |
                            [ apply ]
                                |
                       actual state == desired state

    Run it twice and the second run does nothing. That is idempotence,
    and it is what makes the loop safe to automate.

The whole idea fits in a page of code, which is worth writing once to understand what the tools are doing:

import json, copy

# --- DESIRED STATE: WHAT THE CODE SAYS THE WORLD SHOULD LOOK LIKE. ---
desired = {
    "bucket:models":        {"type": "s3", "versioning": True, "encryption": "kms"},
    "service:fraud-scorer": {"type": "deployment", "replicas": 29,
                             "image": "registry/fraud@sha256:9c1f",
                             "cpu": "2", "memory": "4Gi"},
    "job:nightly-train":    {"type": "schedule", "cron": "0 2 * * *",
                             "timeout_min": 180},
    "alert:drift":          {"type": "monitor", "metric": "psi_score",
                             "threshold": 0.2},
}

# --- ACTUAL STATE: WHAT IS RUNNING RIGHT NOW. ---
actual = {
    "bucket:models":        {"type": "s3", "versioning": False, "encryption": "kms"},
    "service:fraud-scorer": {"type": "deployment", "replicas": 12,
                             "image": "registry/fraud@sha256:4a02",
                             "cpu": "2", "memory": "4Gi"},
    "job:legacy-export":    {"type": "schedule", "cron": "0 5 * * *",
                             "timeout_min": 60},
}

def plan(desired, actual):
    """The whole idea of declarative infrastructure, in fifteen lines."""
    out = []
    for key, want in desired.items():
        have = actual.get(key)
        if have is None:
            out.append(("create", key, want))
        elif have != want:
            diff = {k: (have.get(k), v) for k, v in want.items() if have.get(k) != v}
            out.append(("update", key, diff))
    for key in actual:
        if key not in desired:                      # code is the source of truth
            out.append(("destroy", key, actual[key]))
    return out

def show(p):
    if not p:
        print("  No changes. Infrastructure matches configuration.")
    for action, key, detail in p:
        mark = {"create": "+", "update": "~", "destroy": "-"}[action]
        print(f"  {mark} {action:8s} {key}")
        if action == "update":
            for field, (old, new) in detail.items():
                print(f"      {field}: {old!r} -> {new!r}")

def apply(p, actual, desired):
    for action, key, _ in p:
        if action in ("create", "update"):
            actual[key] = copy.deepcopy(desired[key])
        else:
            del actual[key]
    return actual

print("PLAN (what would change, before anything changes)")
show(plan(desired, actual))
print(f"  {len(plan(desired, actual))} changes")

actual = apply(plan(desired, actual), actual, desired)
print("\nPLAN AGAIN, IMMEDIATELY AFTER APPLY")
show(plan(desired, actual))
print("  Idempotence is the property that makes this safe to run in CI:")
print("  applying twice does the same thing as applying once.")

# --- DRIFT: SOMEBODY FIXED AN INCIDENT IN THE CONSOLE AT 2 A.M. ---
actual["service:fraud-scorer"]["replicas"] = 60
actual["service:fraud-scorer"]["memory"] = "8Gi"
print("\nPLAN AFTER A MANUAL CHANGE IN THE CONSOLE")
show(plan(desired, actual))
print("  The next apply will silently revert the emergency fix. That is not")
print("  a flaw -- it is the guarantee. The fix belongs in the config file,")
print("  where it is reviewed, versioned, and survives the next deploy.")

# --- ENVIRONMENTS ARE THE SAME CODE WITH DIFFERENT NUMBERS. ---
def env(base, **overrides):
    out = copy.deepcopy(base)
    for path, value in overrides.items():
        key, field = path.split(".")
        out[key][field] = value
    return out

staging = env(desired, **{"service:fraud-scorer.replicas": 2,
                          "job:nightly-train.timeout_min": 30})
print("\nSTAGING FROM THE SAME MODULE")
print(json.dumps({k: v for k, v in staging.items()
                  if k.startswith("service") or k.startswith("job:n")}, indent=2))
print("  One definition, per-environment values. Copy-pasted environments")
print("  diverge within weeks and then 'it works in staging' means nothing.")

PLAN (what would change, before anything changes)
  ~ update   bucket:models
      versioning: False -> True
  ~ update   service:fraud-scorer
      replicas: 12 -> 29
      image: 'registry/fraud@sha256:4a02' -> 'registry/fraud@sha256:9c1f'
  + create   job:nightly-train
  + create   alert:drift
  - destroy  job:legacy-export
  5 changes

PLAN AGAIN, IMMEDIATELY AFTER APPLY
  No changes. Infrastructure matches configuration.
  Idempotence is the property that makes this safe to run in CI:
  applying twice does the same thing as applying once.

PLAN AFTER A MANUAL CHANGE IN THE CONSOLE
  ~ update   service:fraud-scorer
      replicas: 60 -> 29
      memory: '8Gi' -> '4Gi'
  The next apply will silently revert the emergency fix. That is not
  a flaw -- it is the guarantee. The fix belongs in the config file,
  where it is reviewed, versioned, and survives the next deploy.

STAGING FROM THE SAME MODULE
{
  "service:fraud-scorer": {
    "type": "deployment",
    "replicas": 2,
    "image": "registry/fraud@sha256:9c1f",
    "cpu": "2",
    "memory": "4Gi"
  },
  "job:nightly-train": {
    "type": "schedule",
    "cron": "0 2 * * *",
    "timeout_min": 30
  }
}
  One definition, per-environment values. Copy-pasted environments
  diverge within weeks and then 'it works in staging' means nothing.

Four behaviours in that output define the practice.

The plan comes before the apply. You see that the deployment will move from 12 to 29 replicas and change image digest before anything happens. Requiring a reviewed plan for production changes catches the destructive surprise -- note the - destroy job:legacy-export, which is exactly the line you want a human to read before it runs.

Idempotence. Applying twice changes nothing the second time, which is what makes it safe to run from CI on every merge.

Drift detection. The 2 a.m. console fix shows up as a difference, and the next apply reverts it. This feels hostile the first time it happens, and it is the point: the repository is the source of truth, so the fix must be made there. The good version of that incident is: change the replica count in the config, open a pull request, merge, apply -- three minutes, and the change survives the next deploy.

Environments are parameters, not copies. Staging and production differ in numbers, not in structure. Copy-pasted environments diverge, and then a staging test tells you nothing about production.

In practice, for an ML platform, expect to declare: object storage for artifacts and datasets (with versioning and encryption turned on in code), the training compute (cluster, node pools, GPU quotas), the serving deployment and its autoscaler, the scheduled jobs and their triggers, the feature store or database, the monitors and alerts, and the IAM roles that connect them. The IAM roles are the ones most worth having in code: least privilege is only auditable if it is written down.

Two cautions specific to this territory:

• State is a real thing you can lose. Terraform's state file maps configuration to real resources; store it remotely with locking, never on a laptop, and never edit it by hand.
• Secrets are not configuration. Keys and passwords go in a secret manager and are referenced by name; putting them in the repository -- or in the state file -- is the classic way to publish them.

Key Takeaways

• IaC replaces console clicks with a declarative file that is reviewed, versioned, and repeatable.
• The core loop is plan (diff desired against actual) then apply (reconcile); running it twice changes nothing the second time.
• Always read the plan before applying to production, especially for destroy actions.
• Manual console changes show up as drift and get reverted; make the fix in the repository instead.
• Parameterise environments from one module rather than copying configurations, or staging stops predicting production.
• For ML platforms, declare storage, compute, serving, schedules, monitors, and IAM roles -- especially IAM.
• Store remote state with locking and never hand-edit it.
• Keep secrets in a secret manager and reference them; never in the repo or the state file.

🧪 Practice

1. Write a declarative description of one piece of infrastructure you use today, and diff it against what actually exists.
2. Implement the plan/apply loop for a small configuration and prove it is idempotent.
3. Parameterise that configuration into development and production variants that differ only in values.
4. List every resource your current ML project depends on and mark which ones exist only because somebody clicked.
5. Interview: An engineer fixes a production outage by editing settings in the cloud console. What should happen next, and why? (Hint: the fix itself was fine; think about what happens on the next deploy and how the team would ever know.)

<a id="153-monitoring-and-maintenance"></a>

15.3 Monitoring and Maintenance

A deployed model is the only kind of software that can fail without anything breaking: no exception, no error rate, no alert -- just decisions that get worse. This section covers how to notice that happening (drift in the inputs and in the relationship), how to measure live performance when labels are late or missing, how to keep the data feeding the model trustworthy, what to do when something fires, when and how to retrain, and how to release a new model without betting the business on it.

Data Drift and Concept Drift

Models assume the future resembles the past. Production's job is to violate that assumption, and it does so in two distinct ways that require different responses.

Data drift (covariate shift) means the inputs have moved: P(X) changed while P(Y|X) stayed the same. A marketing campaign brings in younger users, a new country launches, a currency changes. The relationship the model learned is still valid; it is simply being applied to a population it saw less of during training.

Concept drift means the relationship itself changed: P(Y|X) is different. The same customer profile that predicted churn last year no longer does, because a competitor changed their pricing. The model is now wrong about the world, and no amount of input monitoring will tell you so.

The distinction is not academic, because the standard drift dashboards measure P(X):

import numpy as np

rng = np.random.default_rng(21)

def psi(ref, cur, bins=10):
    """Population Stability Index: how far a distribution moved."""
    edges = np.quantile(ref, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    r = np.histogram(ref, edges)[0] / len(ref) + 1e-6
    c = np.histogram(cur, edges)[0] / len(cur) + 1e-6
    return float(np.sum((c - r) * np.log(c / r)))

def ks(ref, cur):
    """Kolmogorov-Smirnov statistic: largest gap between the two CDFs."""
    allv = np.sort(np.concatenate([ref, cur]))
    return float(np.max(np.abs(np.searchsorted(np.sort(ref), allv, "right") / len(ref)
                             - np.searchsorted(np.sort(cur), allv, "right") / len(cur))))

N = 20_000
x_ref = rng.normal(0, 1, N)
y_ref = (rng.random(N) < 1 / (1 + np.exp(-(1.5 * x_ref)))).astype(int)
model = lambda x: 1 / (1 + np.exp(-(1.5 * x)))      # the deployed relationship

def accuracy(x, y):
    return float(((model(x) > 0.5).astype(int) == y).mean())

scenarios = {
    # Inputs move; the relationship between x and y is unchanged.
    "covariate shift": (rng.normal(0.8, 1.0, N), 1.5),
    # Inputs identical; what x MEANS for y has changed. This is the dangerous one.
    "concept drift":   (rng.normal(0.0, 1.0, N), -0.4),
    # Both move.
    "both":            (rng.normal(0.8, 1.4, N), 0.3),
    # Nothing moves: the control.
    "stable":          (rng.normal(0.0, 1.0, N), 1.5),
}

print("DRIFT DETECTORS SEE THE INPUTS. THE DAMAGE IS IN THE RELATIONSHIP.")
print(f"  reference accuracy = {accuracy(x_ref, y_ref):.3f}\n")
base = accuracy(x_ref, y_ref)
print("  scenario           PSI(x)   KS(x)   accuracy   change   input alarm?")
for name, (x_cur, beta) in scenarios.items():
    y_cur = (rng.random(N) < 1 / (1 + np.exp(-(beta * x_cur)))).astype(int)
    p_in, k_in = psi(x_ref, x_cur), ks(x_ref, x_cur)
    acc = accuracy(x_cur, y_cur)
    fires = "FIRES" if p_in > 0.2 else "silent"
    print(f"  {name:16s} {p_in:7.3f} {k_in:7.3f} {acc:10.3f}"
          f"  {acc - base:+7.3f}   {fires:>12}")
print("\n  PSI rule of thumb: <0.1 stable, 0.1-0.2 watch, >0.2 investigate.")
print("  Covariate shift fired the alarm and the model got BETTER: the inputs")
print("  moved into a region where the decision is easier. Input drift is a")
print("  reason to look, not a failure. Concept drift moved nothing")
print("  measurable in the inputs and cost 31 points of accuracy -- no")
print("  input-only monitor can see it. Only labels can.")

# --- DRIFT IS NOT ALWAYS GRADUAL. KNOW THE SHAPES. ---
print("\nSHAPES OF DRIFT, AND WHAT EACH IMPLIES")
shapes = [
    ("sudden", "policy change, new competitor, pandemic", "retrain now, on new data only"),
    ("gradual", "slow change in customer mix", "scheduled retraining keeps up"),
    ("incremental", "prices creeping upward", "recalibrate or add a trend feature"),
    ("recurring", "seasonality, weekday effects", "do NOT retrain; add the feature"),
]
for name, cause, action in shapes:
    print(f"  {name:12s} {cause:40s} -> {action}")
print("  Mistaking seasonality for drift produces a model that chases noise")
print("  and is retrained every December for no reason.")

# --- HOW LONG BEFORE YOU KNOW? LABEL DELAY SETS THE DETECTION FLOOR. ---
print("\nLABEL DELAY VS DETECTION DELAY")
print("  use case            label arrives after   what you can watch meanwhile")
for case, delay, proxy in [
        ("ad click", "minutes", "the labels themselves, almost live"),
        ("churn", "30 days", "engagement proxies, score distribution"),
        ("fraud chargeback", "30-90 days", "analyst review queue, alert rate"),
        ("loan default", "6-24 months", "early-payment behaviour, input drift")]:
    print(f"  {case:20s} {delay:19s}   {proxy}")
print("  When labels are slow, input drift and prediction drift are the only")
print("  signals you have -- which is exactly why you monitor them too.")

DRIFT DETECTORS SEE THE INPUTS. THE DAMAGE IS IN THE RELATIONSHIP.
  reference accuracy = 0.737

  scenario           PSI(x)   KS(x)   accuracy   change   input alarm?
  covariate shift    0.586   0.309      0.780   +0.043          FIRES
  concept drift      0.001   0.010      0.426   -0.311         silent
  both               0.505   0.291      0.595   -0.142          FIRES
  stable             0.000   0.005      0.733   -0.004         silent

  PSI rule of thumb: <0.1 stable, 0.1-0.2 watch, >0.2 investigate.
  Covariate shift fired the alarm and the model got BETTER: the inputs
  moved into a region where the decision is easier. Input drift is a
  reason to look, not a failure. Concept drift moved nothing
  measurable in the inputs and cost 31 points of accuracy -- no
  input-only monitor can see it. Only labels can.

SHAPES OF DRIFT, AND WHAT EACH IMPLIES
  sudden       policy change, new competitor, pandemic  -> retrain now, on new data only
  gradual      slow change in customer mix              -> scheduled retraining keeps up
  incremental  prices creeping upward                   -> recalibrate or add a trend feature
  recurring    seasonality, weekday effects             -> do NOT retrain; add the feature
  Mistaking seasonality for drift produces a model that chases noise
  and is retrained every December for no reason.

LABEL DELAY VS DETECTION DELAY
  use case            label arrives after   what you can watch meanwhile
  ad click             minutes               the labels themselves, almost live
  churn                30 days               engagement proxies, score distribution
  fraud chargeback     30-90 days            analyst review queue, alert rate
  loan default         6-24 months           early-payment behaviour, input drift
  When labels are slow, input drift and prediction drift are the only
  signals you have -- which is exactly why you monitor them too.

Read those four rows carefully, because they contradict the usual intuition.

Covariate shift produced a large PSI, fired every alarm -- and improved accuracy, because the inputs moved into a region where the decision boundary is easier. Input drift is a reason to investigate, not evidence of harm. Treating every PSI alarm as an incident trains a team to ignore the dashboard.

Concept drift produced PSI of 0.001 -- indistinguishable from a stable population -- while accuracy collapsed by 31 points. Every input-based monitor stayed silent. This is the failure mode that makes label-based monitoring non-negotiable wherever labels can be obtained at all.

The standard measures, and when each is appropriate:

Measure	Applies to	Interpretation	Watch out for
PSI	Binned numeric or categorical	<0.1 stable, >0.2 investigate	Sensitive to bin choice
KS statistic	Continuous, univariate	Max CDF gap; has a p-value	Tiny shifts are "significant" at scale
Chi-square	Categorical	Tests frequency change	Same large-sample problem
Jensen-Shannon	Any distribution	Symmetric, bounded 0-1	Needs density estimation
Wasserstein	Continuous	Distance in units of the variable	Interpretable, scale-dependent
Null and cardinality rates	Any	Schema or pipeline failure	Often the real cause

Two practical notes. With large production volumes, every statistical test rejects; use effect sizes (PSI, Wasserstein) with thresholds, not p-values. And multivariate drift can hide in a set of individually stable features -- if that risk matters, train a drift classifier: if a model can distinguish reference rows from current rows better than chance, the joint distribution has moved.

The shapes table matters for the response. Sudden drift needs immediate retraining on recent data; recurring drift needs a feature, not a retraining -- a model retrained every December to handle Christmas is a model that should have had a seasonality feature.

Finally, label delay sets a floor on how fast you can detect concept drift. If chargebacks arrive 60 days later, you cannot know a fraud model degraded until 60 days after it did. That is precisely why input and prediction drift are monitored anyway: they are the only early signals available, even though they are imperfect.

Key Takeaways

• Data drift changes P(X); concept drift changes P(Y|X), and only the second necessarily harms the model.
• Input drift monitors cannot detect concept drift -- in the example PSI was 0.001 while accuracy fell 31 points.
• Input drift can coincide with improved accuracy; treat an alarm as a prompt to investigate, not an incident.
• Use effect-size measures (PSI, Wasserstein, JS) with thresholds, because p-values reject everything at production volume.
• Watch prediction distributions and null/cardinality rates alongside inputs; pipeline breakage often shows there first.
• A drift classifier catches multivariate shift that per-feature tests miss.
• Classify drift by shape: sudden, gradual, incremental, recurring -- and handle recurring drift with a feature rather than retraining.
• Label delay sets the floor on concept-drift detection, which is why imperfect early signals are still worth monitoring.

🧪 Practice

1. Implement PSI and apply it to one feature across two months of your own data; state which bins drive the score.
2. Simulate covariate shift and concept drift separately and confirm which detectors fire in each case.
3. Train a drift classifier between a reference window and a current window and report its AUC; interpret a value of 0.52 and one of 0.85.
4. For a model you know, write down the label delay and the earliest date a concept drift starting today could be detected.
5. Interview: Your PSI dashboard is green but business metrics dropped. How do you investigate? (Hint: consider which quantity each of your monitors actually measures, and which one nobody is measuring.)

Model Performance Monitoring

Monitoring a model means answering "is it still working?" continuously, for a system whose failures are quiet. It has four layers, and teams reliably build the first and skip the third:

Layer	Examples	Fails loudly?
Service	Latency, error rate, saturation, restarts	Yes
Data	Null rates, schema, volume, freshness, drift	Sometimes
Model	Accuracy, AUC, calibration, per-segment quality	No
Business	Approval rate, fraud loss, revenue per session	Eventually, expensively

The model layer is where the interesting problems are, and three properties make it harder than ordinary monitoring: labels arrive late or never, aggregate metrics conceal segment failures, and the noise floor is high enough that naive alerting produces mostly false alarms.

import math
import numpy as np

rng = np.random.default_rng(31)

# --- AGGREGATE METRICS HIDE SEGMENT FAILURES. ---
# Traffic mix and per-segment accuracy, before and after a mobile SDK change.
segments = [
    ("web / desktop",  0.55, 0.812, 0.815),
    ("mobile / iOS",   0.22, 0.798, 0.794),
    ("mobile/Android", 0.18, 0.803, 0.617),   # broken feature in the new SDK
    ("partner API",    0.05, 0.771, 0.769),
]
print("PER-SEGMENT ACCURACY, BEFORE AND AFTER A RELEASE")
print("  segment          share   before    after   change")
for name, share, before, after in segments:
    print(f"  {name:15s} {share:6.0%} {before:8.3f} {after:8.3f} {after - before:+8.3f}")
agg_b = sum(s * b for _, s, b, _ in segments)
agg_a = sum(s * a for _, s, _, a in segments)
print(f"  {'AGGREGATE':15s} {1.0:6.0%} {agg_b:8.3f} {agg_a:8.3f} {agg_a - agg_b:+8.3f}")
print("  A 19-point collapse for 18% of users shows up as 3 points overall --")
print("  inside the range a weekly dashboard is routinely noisy by.")

# --- HOW MANY LABELS BEFORE A DROP IS DETECTABLE? ---
def n_needed(p0, drop, alpha=0.05, power=0.80):
    """Two-proportion test: labels needed per window to see p0 -> p0-drop."""
    p1 = p0 - drop
    za, zb = 1.959964, 0.8416212
    pbar = (p0 + p1) / 2
    return math.ceil(2 * (za * math.sqrt(2 * pbar * (1 - pbar))
                          + zb * math.sqrt(p0 * (1 - p0) + p1 * (1 - p1))) ** 2
                     / (2 * (p0 - p1) ** 2))

print("\nLABELS NEEDED TO DETECT AN ACCURACY DROP (80% POWER, 5% ALPHA)")
print("  baseline   -1 pt     -2 pt     -5 pt    -10 pt")
for p0 in (0.80, 0.95, 0.99):
    row = "".join(f"{n_needed(p0, d):10,}" for d in (0.01, 0.02, 0.05, 0.10))
    print(f"  {p0:8.0%} {row}")
print("  At 500 labels/day, a 2-point drop from a 95% baseline takes about")
print(f"  {n_needed(0.95, 0.02) / 500:.0f} days to confirm. Daily alerting on it is not")
print("  monitoring; it is a random number generator.")

# --- A CONTROL CHART SEPARATES NOISE FROM A REAL SHIFT. ---
days, n_per_day = 60, 400
true_acc = np.where(np.arange(days) < 40, 0.80, 0.755)      # a real step at day 40
obs = rng.binomial(n_per_day, true_acc) / n_per_day
ref_mu, ref_sd = obs[:30].mean(), obs[:30].std(ddof=1)
print(f"\nCONTROL CHART: baseline {ref_mu:.3f} +/- {ref_sd:.3f} (first 30 days)")
print("  day   accuracy   z      3-sigma   7-day mean below baseline?")
alerted = None
for d in range(30, days, 5):
    win = obs[max(0, d - 6):d + 1]
    z = (obs[d] - ref_mu) / ref_sd
    run = win.mean() < ref_mu - ref_sd        # sustained, not a single point
    flag = "BREACH" if abs(z) > 3 else "-"
    if alerted is None and run and z < -1:
        alerted = d
    print(f"  {d:4d}   {obs[d]:8.3f} {z:6.2f}   {flag:7s}   {'yes' if run else 'no'}")
print(f"  A single point beyond 3 sigma never fires here; the sustained")
print(f"  7-day shift is detected at day {alerted}, {alerted - 40} days after the change.")
print("  Alert on sustained deviation, not on single days.")

# --- WITHOUT LABELS, MONITOR WHAT YOU STILL HAVE. ---
print("\nSIGNALS AVAILABLE BEFORE LABELS ARRIVE")
for name, what in [
        ("prediction mean", "shifts when inputs or the model change"),
        ("score histogram", "bimodality or pile-ups reveal broken features"),
        ("alert/approval rate", "the operational consequence of the scores"),
        ("feature null rate", "upstream breakage, usually the real cause"),
        ("override rate", "humans disagreeing with the model, a live signal"),
        ("latency and errors", "a service failure looks like a quality failure")]:
    print(f"  {name:20s} {what}")

PER-SEGMENT ACCURACY, BEFORE AND AFTER A RELEASE
  segment          share   before    after   change
  web / desktop      55%    0.812    0.815   +0.003
  mobile / iOS       22%    0.798    0.794   -0.004
  mobile/Android     18%    0.803    0.617   -0.186
  partner API         5%    0.771    0.769   -0.002
  AGGREGATE         100%    0.805    0.772   -0.033
  A 19-point collapse for 18% of users shows up as 3 points overall --
  inside the range a weekly dashboard is routinely noisy by.

LABELS NEEDED TO DETECT AN ACCURACY DROP (80% POWER, 5% ALPHA)
  baseline   -1 pt     -2 pt     -5 pt    -10 pt
       80%     25,583     6,510     1,094       294
       95%      8,158     2,213       435       141
       99%      2,319       769       211        88
  At 500 labels/day, a 2-point drop from a 95% baseline takes about
  4 days to confirm. Daily alerting on it is not
  monitoring; it is a random number generator.

CONTROL CHART: baseline 0.798 +/- 0.020 (first 30 days)
  day   accuracy   z      3-sigma   7-day mean below baseline?
    30      0.802   0.23   -         no
    35      0.830   1.59   -         no
    40      0.785  -0.64   -         no
    45      0.750  -2.37   -         yes
    50      0.770  -1.38   -         yes
    55      0.743  -2.74   -         yes
  A single point beyond 3 sigma never fires here; the sustained
  7-day shift is detected at day 45, 5 days after the change.
  Alert on sustained deviation, not on single days.

SIGNALS AVAILABLE BEFORE LABELS ARRIVE
  prediction mean      shifts when inputs or the model change
  score histogram      bimodality or pile-ups reveal broken features
  alert/approval rate  the operational consequence of the scores
  feature null rate    upstream breakage, usually the real cause
  override rate        humans disagreeing with the model, a live signal
  latency and errors   a service failure looks like a quality failure

Segments first. The Android collapse -- 19 points of accuracy for nearly a fifth of users -- appears as a 3-point aggregate decline, which is indistinguishable from an ordinary bad week. Aggregate metrics are for reporting; monitoring belongs at the segment level. Choose segments before the incident: platform, country, customer tier, new versus returning, and any group with a fairness obligation. Track a small number of them with explicit thresholds rather than a hundred without.

Then statistical power. The table converts a wish ("alert me if accuracy drops") into an arithmetic fact: detecting a 2-point drop from a 95% baseline needs roughly 2,200 labelled outcomes. At 500 labels a day, a daily alert on that threshold is noise, and the correct design is a weekly comparison or a sequential test. Knowing this number prevents both false alarms and the opposite failure -- a threshold so wide that a genuine collapse takes a month to trip it.

Then sustained-deviation alerting. The control chart shows a real 4.5-point step at day 40 that never produces a single 3-sigma point, because daily noise is larger than the shift. A rolling-window rule detects it five days later. That five-day delay is not a flaw to be engineered away; it is the price of not paging someone every Tuesday. Where faster detection genuinely matters, use a sequential method (CUSUM, page-Hinkley, or a Bayesian changepoint test) which accumulates evidence rather than testing each day independently.

Calibration deserves separate tracking from discrimination. A model can keep its ranking quality (AUC unchanged) while its probabilities become systematically too high -- which breaks every downstream threshold and expected-value calculation. Monitor mean predicted probability against mean observed outcome per bucket, and treat a widening gap as its own alert; it is also the cheapest thing to fix, since recalibration does not require retraining.

When labels are slow, monitor the signals you still have -- prediction distribution, alert rate, null rates, human override rate. The override rate in particular is an underused live signal: when analysts start disagreeing with the model more often, something has changed, and they knew before the labels did.

Two principles for the alerts themselves. Alert on consequences, not on statistics: "approval rate fell 8% for Android users" is actionable, "KS statistic on feature 17 is 0.13" is not. And every alert needs an owner and a runbook, or it becomes a notification that people filter.

Key Takeaways

• Model monitoring has four layers; the model layer fails silently and is the one most often missing.
• Aggregate metrics hide segment failures -- a 19-point collapse for 18% of traffic looks like a 3-point wobble.
• Define monitored segments in advance, with thresholds, and keep the list short enough to act on.
• Compute the labels needed to detect the drop you care about; alerting faster than that is noise generation.
• Alert on sustained deviation (rolling windows, CUSUM) rather than single points beyond a sigma band.
• Track calibration separately from ranking; probabilities can degrade while AUC is unchanged, and recalibration is cheap.
• Before labels arrive, monitor prediction distributions, null rates, operational rates, and human override rates.
• Every alert needs a consequence-level description, an owner, and a runbook.

🧪 Practice

1. Split one model's live accuracy by three segments and compare each against the aggregate over the last month.
2. Compute how many labelled examples you need to detect a 2-point drop from your model's current baseline, and convert it into a time window.
3. Build a control chart on a real metric and tune a rolling rule so it would have caught the last known incident without firing in the previous quarter.
4. Plot predicted versus observed rates by score decile for a live model and decide whether it needs recalibration.
5. Interview: You can log only one number per prediction for monitoring purposes. What do you log and why? (Hint: think about which single quantity lets you reconstruct both distribution shifts and, once labels arrive, performance.)

Data Quality Checks in Production

Ask experienced ML engineers where production incidents come from and the answer is rarely "the model". It is an upstream table that started arriving six hours late, a column that changed units, a join that silently duplicated rows, a team that replaced nulls with -1 as a favour. The model consumes whatever it is given and returns confident numbers either way.

The remedy is to make expectations about data explicit and executable, and to run them where the data enters -- the same principle as type checking, applied to rows instead of variables. Six families cover most real failures:

Check family	Catches	Typical rule
Schema	Renamed, dropped, retyped columns	Column set and dtypes match
Volume	Partial loads, duplicated loads	Row count within X% of recent
Uniqueness	Fan-out from a bad join	Primary key is unique
Nullness	Broken upstream field	Null rate below threshold
Range / set	Unit changes, sentinels, new categories	Value bounds, allowed values
Freshness	Stalled pipelines	Max timestamp within an interval

import numpy as np
import pandas as pd

rng = np.random.default_rng(17)

def batch(n, day, broken=None):
    df = pd.DataFrame({
        "event_id": np.arange(n) + day * 100_000,
        "ts": pd.Timestamp("2026-09-01") + pd.Timedelta(days=day)
              + pd.to_timedelta(rng.uniform(0, 24, n), unit="h"),
        "amount": rng.lognormal(3.5, 0.8, n).round(2),
        "country": rng.choice(["US", "GB", "DE", "FR"], n, p=[.5, .2, .2, .1]),
        "device_age_days": rng.integers(0, 2000, n).astype(float),
    })
    if broken == "nulls":
        df.loc[df.sample(frac=0.30, random_state=1).index, "device_age_days"] = np.nan
    if broken == "sentinel":            # upstream "fixed" the nulls with -1
        df.loc[df.sample(frac=0.30, random_state=1).index, "device_age_days"] = -1
    if broken == "new_category":
        df.loc[df.sample(frac=0.12, random_state=2).index, "country"] = "ES"
    if broken == "units":               # amounts arrive in cents, not euros
        df["amount"] = df["amount"] * 100
    if broken == "duplicates":
        df = pd.concat([df, df.sample(frac=0.15, random_state=3)], ignore_index=True)
    if broken == "stale":
        df["ts"] = df["ts"] - pd.Timedelta(days=3)
    return df

REFERENCE = batch(20_000, 0)

# --- THE EXPECTATION SUITE: EACH CHECK HAS A SEVERITY AND AN OWNER. ---
def checks(df, ref):
    yield ("volume within 30% of reference", "block",
           0.7 <= len(df) / len(ref) <= 1.3)
    yield ("event_id unique", "block", df.event_id.is_unique)
    yield ("no nulls in device_age_days", "block",
           df.device_age_days.isna().mean() < 0.01)
    yield ("device_age_days >= 0", "block", bool((df.device_age_days >= 0).all()))
    yield ("amount within [0.5, 5000]", "block",
           bool(df.amount.between(0.5, 5000).all()))
    yield ("amount median within 3x reference", "warn",
           0.33 <= df.amount.median() / ref.amount.median() <= 3.0)
    yield ("country in known set", "warn",
           set(df.country) <= set(ref.country.unique()))
    yield ("freshness: max ts within 1 day", "block",
           (pd.Timestamp("2026-09-02") - df.ts.max()) < pd.Timedelta(days=1))

print("EXPECTATION SUITE AGAINST SIX BROKEN BATCHES")
print("  batch           blocking failures / warnings   first failure")
for name in ["clean", "nulls", "sentinel", "new_category", "units",
             "duplicates", "stale"]:
    df = batch(20_000, 1, None if name == "clean" else name)
    results = list(checks(df, REFERENCE))
    blocked = [c for c, sev, ok in results if sev == "block" and not ok]
    warned = [c for c, sev, ok in results if sev == "warn" and not ok]
    first = (blocked + warned + ["-"])[0]
    print(f"  {name:14s} {len(blocked):11d} / {len(warned):<10d}  {first}")
print("  'sentinel' is the dangerous row: an upstream team replaced nulls")
print("  with -1, so the null check passes and the range check catches it.")
print("  Without the range check the model would train on -1 as an age.")

# --- WHAT A SILENT SENTINEL DOES TO A MODEL. ---
clean = batch(20_000, 1)
poisoned = batch(20_000, 1, "sentinel")
w = 0.0009                                        # weight on device_age_days
print("\nEFFECT OF THE SENTINEL VALUE ON PREDICTIONS")
for label, df in [("clean", clean), ("with -1 sentinel", poisoned)]:
    contrib = w * df.device_age_days
    print(f"  {label:18s} mean feature {df.device_age_days.mean():8.1f}"
          f"   mean contribution {contrib.mean():+.4f}")
print("  The feature's mean moved by 30% of its range and no check on nulls,")
print("  volume, or schema would have noticed. Range checks are cheap.")

# --- WHERE THE CHECKS RUN MATTERS AS MUCH AS WHAT THEY CHECK. ---
print("\nWHERE TO PUT EACH CHECK")
for stage, examples in [
        ("ingestion", "schema, types, primary key, volume, freshness"),
        ("after transformation", "ranges, category sets, join row-count changes"),
        ("training input", "label balance, leakage guards, split integrity"),
        ("serving input", "same range/category checks, per REQUEST not per batch"),
        ("prediction output", "score distribution, null predictions, fallback rate")]:
    print(f"  {stage:22s} {examples}")
print("  The serving row is the one most often missing: a batch pipeline that")
print("  validates carefully and an endpoint that validates nothing is the")
print("  normal state of affairs, and it is where skew enters.")

EXPECTATION SUITE AGAINST SIX BROKEN BATCHES
  batch           blocking failures / warnings   first failure
  clean                    0 / 0           -
  nulls                    2 / 0           no nulls in device_age_days
  sentinel                 1 / 0           device_age_days >= 0
  new_category             0 / 1           country in known set
  units                    1 / 1           amount within [0.5, 5000]
  duplicates               1 / 0           event_id unique
  stale                    1 / 0           freshness: max ts within 1 day
  'sentinel' is the dangerous row: an upstream team replaced nulls
  with -1, so the null check passes and the range check catches it.
  Without the range check the model would train on -1 as an age.

EFFECT OF THE SENTINEL VALUE ON PREDICTIONS
  clean              mean feature    999.8   mean contribution +0.8998
  with -1 sentinel   mean feature    692.3   mean contribution +0.6230
  The feature's mean moved by 30% of its range and no check on nulls,
  volume, or schema would have noticed. Range checks are cheap.

WHERE TO PUT EACH CHECK
  ingestion              schema, types, primary key, volume, freshness
  after transformation   ranges, category sets, join row-count changes
  training input         label balance, leakage guards, split integrity
  serving input          same range/category checks, per REQUEST not per batch
  prediction output      score distribution, null predictions, fallback rate
  The serving row is the one most often missing: a batch pipeline that
  validates carefully and an endpoint that validates nothing is the
  normal state of affairs, and it is where skew enters.

Each failing batch teaches something slightly different.

The sentinel case is the one to internalise. An upstream team helpfully replaced missing values with -1, so the null-rate check -- the check most teams have -- passes cleanly. Only a range check notices. The second block shows the consequence: the feature's mean moves by a third, every prediction shifts, and no alert fires anywhere. Sentinel values (-1, -999, 1970-01-01, "unknown") are the most common silent corruption in production data.

The units case (amounts in cents) is the same class of problem: the schema is valid, the types are correct, the values are individually plausible, and every downstream number is wrong by a factor of 100.

The new category case shows why severity matters. A new country is usually legitimate and should warn rather than block -- but it must still be visible, because an unseen category silently becomes an unknown encoding, and most encoders map it to whatever their default is.

Three design rules make a check suite usable rather than an obstacle:

• Severity per check. block halts the pipeline; warn records and continues. A suite where everything blocks gets disabled the first time it stops a release at midnight.
• Thresholds derived from history, not from taste. Compute the null rate and row count over the last 30 runs and set bounds from their distribution, then review them when they fire.
• Checks travel with the data, not with the job. Declaring expectations on a dataset (Great Expectations, Pandera, dbt tests, Deequ) means every consumer inherits them, rather than each pipeline re-implementing its own.

The last table is the part teams most often get half right. Validation is usually thorough in the batch pipeline and absent at the serving endpoint -- which is precisely where training-serving skew enters. The serving path should apply the same range and category checks per request, and needs an explicit policy for a failing request: reject it, impute with a documented default, or fall back to a simpler model. Choosing silently is how a -1 becomes a prediction.

Key Takeaways

• Most production ML incidents originate in data, not in models.
• Make expectations executable and run them at every boundary the data crosses.
• Cover schema, volume, uniqueness, nullness, ranges and category sets, and freshness.
• Sentinel values defeat null checks; only range checks catch them, and they shift every prediction silently.
• Unit changes pass schema and type checks while making every downstream number wrong.
• Assign severities: blocking checks for corruption, warnings for legitimate-but-notable changes.
• Derive thresholds from historical distributions rather than intuition.
• Attach checks to datasets so all consumers inherit them, and validate at serving time too, with an explicit policy for failures.

🧪 Practice

1. Write an expectation suite for one production table covering all six check families, with severities.
2. Derive each threshold from the last 30 days of that table rather than choosing round numbers, and note where the two disagree.
3. Inject a sentinel value into a feature and measure how far the mean prediction moves; confirm which of your checks catch it.
4. Add per-request validation to a serving endpoint and define the fallback behaviour for a failing request.
5. Interview: A model's predictions shifted overnight with no deployment and no drift alert. What are your first three hypotheses? (Hint: all three are about the data path rather than the model, and at least one involves a change somebody else considered harmless.)

Alerting and Incident Response

Monitoring produces signals; alerting decides which of them interrupt a person. Getting this wrong in either direction is expensive: too many alerts and the team stops reading them, too few and the first report comes from a customer.

The arithmetic of rare events is the part most teams get wrong, and it is the same base-rate reasoning that governs a classifier:

import numpy as np

rng = np.random.default_rng(23)
DAYS, N_INCIDENTS, LEN = 3650, 60, 3      # 10 years, 6 incidents a year, 3 days each

truth = np.zeros(DAYS, dtype=bool)
for start in rng.choice(DAYS - LEN, N_INCIDENTS, replace=False):
    truth[start:start + LEN] = True       # real incidents persist; noise does not
N_INCIDENT_DAYS = truth.sum()

# Incident windows, used to score alerts: an alert is "true" if it lands
# inside one, and an incident is "caught" if any alert lands inside it.
edges = np.flatnonzero(np.diff(np.concatenate([[0], truth.astype(int), [0]])))
windows = list(zip(edges[::2], edges[1::2]))

def evaluate(sensitivity, specificity, consecutive=1):
    fires = np.where(truth, rng.random(DAYS) < sensitivity,
                     rng.random(DAYS) < 1 - specificity)
    if consecutive > 1:                    # require k bad days in a row
        k = np.convolve(fires.astype(int), np.ones(consecutive, int), "valid")
        fires = np.concatenate([np.zeros(consecutive - 1, bool), k == consecutive])
    idx = np.flatnonzero(fires)
    precision = truth[idx].mean() if len(idx) else 0.0
    caught = sum(any(fires[s_:e]) for s_, e in windows) / len(windows)
    return len(idx), precision, caught

print(f"ALERT PRECISION WHEN INCIDENTS ARE RARE "
      f"({N_INCIDENT_DAYS} bad days in {DAYS}, in runs of {LEN})")
print("  rule                          alerts/yr   precision   incidents caught")
for label, sens, spec, k in [
        ("sensitive, 95% specific",      0.95, 0.95, 1),
        ("tighter, 99% specific",        0.85, 0.99, 1),
        ("tighter, 99.5% specific",      0.70, 0.995, 1),
        ("99% specific, 2 days in a row", 0.85, 0.99, 2)]:
    n, prec, rec = evaluate(sens, spec, k)
    print(f"  {label:29s} {n / 10:9.1f}   {prec:9.1%}   {rec:16.1%}")
print("  Tightening the threshold trades recall for precision. Requiring two")
print("  consecutive bad days is the better trade: noise rarely repeats, so")
print("  precision rises sharply while most real incidents -- which persist --")
print("  are still caught, one day later. For alerts a human must answer,")
print("  precision is the binding constraint, not sensitivity.")

# --- SLOs TURN "IS IT BAD?" INTO A BUDGET. ---
print("\nERROR BUDGET: 99.5% OF REQUESTS SCORED WITHIN 40 ms, 30-DAY WINDOW")
REQ_PER_DAY = 8_000_000
budget = REQ_PER_DAY * 30 * 0.005
print(f"  allowed slow requests per window: {budget:,.0f}")
print("  day   bad requests   budget used   burn rate   action")
spent = 0
for day, bad_rate in [(1, 0.002), (5, 0.004), (9, 0.020), (12, 0.006)]:
    bad = REQ_PER_DAY * bad_rate
    spent += bad
    burn = (bad / budget) * 30            # x1 = exactly on budget for the window
    action = ("page" if burn > 10 else "ticket" if burn > 2 else "note")
    print(f"  {day:4d}   {bad:12,.0f}   {spent / budget:10.1%}   {burn:8.1f}x   {action}")
print("  Burn rate separates 'slightly degraded' from 'the budget is gone by")
print("  Thursday'. Page on fast burn, ticket on slow burn, never on a single")
print("  bad minute.")

# --- ML MITIGATIONS: WHAT YOU CAN DO IN THE FIRST TEN MINUTES. ---
print("\nMITIGATION LADDER, FASTEST FIRST")
for minutes, action, cost in [
        ("<1 min", "route traffic to the previous model version", "known-good quality"),
        ("<1 min", "disable the model, fall back to rules/heuristic", "large quality loss"),
        ("<5 min", "raise the decision threshold to reduce exposure", "fewer actions taken"),
        ("<5 min", "disable the broken feature, serve with imputation", "small quality loss"),
        ("hours",  "retrain on corrected data", "the real fix"),
        ("days",   "redesign the feature or pipeline", "prevents recurrence")]:
    print(f"  {minutes:7s} {action:52s} {cost}")
print("  Mitigate first, diagnose second. Every item above must be tested")
print("  BEFORE the incident: an untested rollback is a hope, not a plan.")

ALERT PRECISION WHEN INCIDENTS ARE RARE (180 bad days in 3650, in runs of 3)
  rule                          alerts/yr   precision   incidents caught
  sensitive, 95% specific            32.8       51.5%             100.0%
  tighter, 99% specific              18.7       79.7%             100.0%
  tighter, 99.5% specific            14.2       93.7%              98.3%
  99% specific, 2 days in a row       8.9      100.0%              86.7%
  Tightening the threshold trades recall for precision. Requiring two
  consecutive bad days is the better trade: noise rarely repeats, so
  precision rises sharply while most real incidents -- which persist --
  are still caught, one day later. For alerts a human must answer,
  precision is the binding constraint, not sensitivity.

ERROR BUDGET: 99.5% OF REQUESTS SCORED WITHIN 40 ms, 30-DAY WINDOW
  allowed slow requests per window: 1,200,000
  day   bad requests   budget used   burn rate   action
     1         16,000         1.3%        0.4x   note
     5         32,000         4.0%        0.8x   note
     9        160,000        17.3%        4.0x   ticket
    12         48,000        21.3%        1.2x   note
  Burn rate separates 'slightly degraded' from 'the budget is gone by
  Thursday'. Page on fast burn, ticket on slow burn, never on a single
  bad minute.

MITIGATION LADDER, FASTEST FIRST
  <1 min  route traffic to the previous model version          known-good quality
  <1 min  disable the model, fall back to rules/heuristic      large quality loss
  <5 min  raise the decision threshold to reduce exposure      fewer actions taken
  <5 min  disable the broken feature, serve with imputation    small quality loss
  hours   retrain on corrected data                            the real fix
  days    redesign the feature or pipeline                     prevents recurrence
  Mitigate first, diagnose second. Every item above must be tested
  BEFORE the incident: an untested rollback is a hope, not a plan.

At a 95% specificity a detector looks excellent -- it catches every incident. It also pages someone 33 times a year, half of them for nothing. The last row is the design most teams should copy: require the condition to persist. Noise rarely repeats two days running; real degradations do. Precision goes to 100% at the cost of one day's delay and a few missed short incidents. Decide explicitly which you can afford.

The second block is the SRE framing, and it transfers to ML cleanly. Define an SLO ("99.5% of requests scored within 40 ms", or "daily AUC above 0.78"), which implies an error budget -- the amount of failure allowed in the window. Then alert on burn rate: how fast the budget is being spent. A 4x burn is a ticket; a 30x burn is a page; a single bad minute is neither. This replaces arguments about thresholds with a shared number, and it gives a principled answer to "can we ship on Friday?": only if the budget allows.

Severity levels need to be defined before you need them:

Severity	ML example	Response
SEV1	Model serving wrong outputs to all users; money moving	Page now, mitigate first
SEV2	One large segment degraded; feature pipeline dead	Page in hours
SEV3	Drift alarm, slow burn, single feature stale	Ticket, next working day
SEV4	Cosmetic, dashboard gap	Backlog

The mitigation ladder in the example is the part specific to ML, and it deserves rehearsal. Mitigate first, diagnose second: switching back to the previous model version stops the bleeding while you work out what happened, and it is available only if the previous artifact is loadable, the traffic switch is tested, and somebody has permission to do it at 3 a.m. An untested rollback is not a mitigation, it is an intention. Run a game day: deliberately roll back in production during working hours and time it.

A workable ML incident runbook has five sections, and it should be written when calm:

1. Detect -- which alert fired, what it measures, current values.
2. Triage -- is this the model, the data, or the service? Check deployment history, upstream freshness, and error rates first, because those are the common causes and the fastest to confirm.
3. Mitigate -- the ladder above, with exact commands.
4. Diagnose -- reproduce with logged inputs and the logged model version.
5. Postmortem -- blameless, with the timeline, the contributing causes, and actions with owners.

Two habits make postmortems worth the hour they cost. Write down how long detection took, separately from how long the fix took -- if detection took nine days, the action item is a monitor, not a code change. And convert each incident into a test or a check, so the same failure cannot recur silently. A postmortem whose actions are all "be more careful" has not produced anything.

Key Takeaways

• Alert quality is governed by base rates: with rare incidents, high specificity still produces mostly false pages.
• Requiring a condition to persist raises precision sharply, because noise rarely repeats while real degradations do.
• Define SLOs and error budgets, then alert on burn rate rather than on instantaneous values.
• Agree severity levels and their response expectations before the first incident.
• Mitigate before diagnosing: rollback, kill switch, threshold change, feature disable.
• Test every mitigation in advance; an untested rollback is not a plan.
• Write runbooks that start with deployment history and upstream freshness, the most common causes.
• Record detection time separately in postmortems and turn every incident into a permanent check.

🧪 Practice

1. Estimate your model's real incident rate and compute the precision of your current alert threshold; report expected false pages per year.
2. Rewrite one alert to require sustained breach and quantify the change in precision and detection delay.
3. Define an SLO and error budget for a model you own, and compute the burn rates that should page versus ticket.
4. Write the mitigation ladder for that model with exact commands, then time a rollback in a staging environment.
5. Interview: Walk through your first ten minutes responding to "the fraud model is declining twice as many transactions as yesterday". (Hint: order your steps by whether they reduce customer impact or increase your understanding, and be explicit about which comes first.)

Retraining Strategies

Every deployed model is a claim that the past predicts the future, and that claim expires. Retraining is how it gets renewed -- but "retrain more often" is not free advice: each retrain consumes compute, requires validation, and introduces the risk of shipping something worse. The design has four independent decisions.

1. Trigger: on a schedule, on a signal, or continuously.
2. Window: all history, a sliding window, or weighted by recency.
3. Scope: full retrain from scratch, or incremental update.
4. Gate: what must be true before the new model replaces the old.

import numpy as np

rng = np.random.default_rng(29)
DAYS, PER_DAY, D = 365, 12, 4   # a small daily volume, as most problems have

# --- A WORLD THAT CHANGES: COEFFICIENTS DRIFT, WITH ONE SUDDEN SHOCK. ---
beta = np.zeros((DAYS, D))
beta[0] = [1.0, -0.5, 0.8, 0.2]
for d in range(1, DAYS):
    beta[d] = beta[d - 1] + rng.normal(0, 0.01, D)      # gradual drift
beta[200:] += np.array([0.0, 1.6, 0.0, 0.0])            # sudden regime change

X = rng.normal(size=(DAYS, PER_DAY, D))
Y = np.einsum("dnf,df->dn", X, beta) + rng.normal(0, 1.0, (DAYS, PER_DAY))

def fit(days):
    x = X[days].reshape(-1, D)
    y = Y[days].reshape(-1)
    return np.linalg.lstsq(x, y, rcond=None)[0]

def rmse(w, d):
    return float(np.sqrt(np.mean((X[d] @ w - Y[d]) ** 2)))

def run(strategy, window=None, cadence=None, threshold=None, warmup=30):
    w = fit(range(warmup))
    errs, retrains, ref = [], 0, None
    for d in range(warmup, DAYS):
        e = rmse(w, d)
        errs.append(e)
        ref = e if ref is None else ref
        due = (strategy == "scheduled" and (d - warmup) % cadence == 0)
        fired = (strategy == "triggered"
                 and np.mean(errs[-7:]) > threshold * np.mean(errs[:14]))
        if strategy != "never" and (due or fired):
            start = max(0, d - window) if window else 0
            w = fit(range(start, d))                    # only data up to today
            retrains += 1
    return float(np.mean(errs)), retrains

print("RETRAINING STRATEGIES IN A DRIFTING WORLD (365 days, one regime change)")
print("  strategy                         mean RMSE   retrains   vs never retraining")
rows = [
    ("never retrain",            dict(strategy="never")),
    ("quarterly, all history",   dict(strategy="scheduled", cadence=90)),
    ("monthly, all history",     dict(strategy="scheduled", cadence=30)),
    ("monthly, 60-day window",   dict(strategy="scheduled", cadence=30, window=60)),
    ("weekly, 60-day window",    dict(strategy="scheduled", cadence=7, window=60)),
    ("weekly, 14-day window",    dict(strategy="scheduled", cadence=7, window=14)),
    ("error-triggered, 60-day",  dict(strategy="triggered", window=60, threshold=1.3)),
]
baseline = run(strategy="never")[0]
for label, kw in rows:
    err, n = run(**kw)
    print(f"  {label:32s} {err:9.4f} {n:10d}   {err / baseline - 1:18.1%}")
print("  Cadence and window length are separate knobs, and the window matters")
print("  more: monthly retraining on a 60-day window beats monthly retraining")
print("  on all history, because old data describes a world that is gone.")

# --- WINDOW LENGTH: THE STABILITY-FRESHNESS TRADE, MEASURED. ---
print("\nWINDOW LENGTH WITH WEEKLY RETRAINING")
print("  window days   mean RMSE   RMSE in the 30 days after the regime change")
for win in (7, 14, 30, 60, 120, 365):
    w = fit(range(30))
    errs, post = [], []
    for d in range(30, DAYS):
        errs.append(rmse(w, d))
        if 200 <= d < 230:
            post.append(errs[-1])
        if (d - 30) % 7 == 0:
            w = fit(range(max(0, d - win), d))
    print(f"  {win:11d}   {np.mean(errs):9.4f}   {np.mean(post):42.4f}")
print("  The mean is minimised near two weeks here. Shorter windows recover")
print("  fastest from the shock and are noisier the rest of the year; longer")
print("  windows are stable and slow to adapt. Where shocks are rare, prefer")
print("  the longer window plus a manual retrain when one happens.")

# --- THE FEEDBACK LOOP: THE MODEL DECIDES WHAT DATA IT WILL SEE NEXT. ---
# A fraud model blocks what it fears. A blocked transaction never produces a
# chargeback, so the label table records it as "not fraud" -- the model's
# successes are written down as its mistakes.
TRUE = np.array([1.4, -0.9])

def logistic_fit(X, y, iters=800, lr=0.8):
    w = np.zeros(X.shape[1])
    for _ in range(iters):                              # plain gradient descent
        w -= lr * X.T @ (1 / (1 + np.exp(-(X @ w))) - y) / len(X)
    return w

def feedback(censor_as_negative):
    X = rng.normal(size=(8000, 2))
    y = (rng.random(8000) < 1 / (1 + np.exp(-(X @ TRUE)))).astype(float)
    w, hist, out = logistic_fit(X, y), [(X, y)], []
    for _ in range(6):
        X = rng.normal(size=(8000, 2))
        y = (rng.random(8000) < 1 / (1 + np.exp(-(X @ TRUE)))).astype(float)
        blocked = 1 / (1 + np.exp(-(X @ w))) > 0.8       # the model's action
        out.append((blocked.mean(), int(y[~blocked].sum()), np.linalg.norm(w)))
        if censor_as_negative:
            hist.append((X, np.where(blocked, 0.0, y)))  # blocked -> label 0
        else:
            hist.append((X[~blocked], y[~blocked]))      # blocked -> unlabelled
        hist = hist[-3:]                                 # rolling training window
        w = logistic_fit(np.vstack([h[0] for h in hist]),
                         np.concatenate([h[1] for h in hist]))
    return out

print("\nFEEDBACK LOOP: WHAT HAPPENS TO TRANSACTIONS THE MODEL BLOCKS")
print("  round   blocked as 0          blocked left unlabelled")
print("          blocked   losses      blocked   losses")
for r, (bad, good) in enumerate(zip(feedback(True), feedback(False))):
    print(f"  {r:5d}   {bad[0]:6.1%} {bad[1]:8,d}      {good[0]:6.1%} {good[1]:8,d}")
print("  Recording blocked transactions as legitimate teaches the next model")
print("  that high scores are safe. Blocking collapses, fraud losses rise by")
print("  half, and every offline metric looks fine. Treat suppressed outcomes")
print("  as MISSING, and keep a small control group that is never blocked so")
print("  the unseen region still produces labels.")

RETRAINING STRATEGIES IN A DRIFTING WORLD (365 days, one regime change)
  strategy                         mean RMSE   retrains   vs never retraining
  never retrain                       1.3909          0                 0.0%
  quarterly, all history              1.3345          4                -4.1%
  monthly, all history                1.2840         12                -7.7%
  monthly, 60-day window              1.0819         12               -22.2%
  weekly, 60-day window               1.0473         48               -24.7%
  weekly, 14-day window               1.0021         48               -27.9%
  error-triggered, 60-day             1.0718         35               -22.9%
  Cadence and window length are separate knobs, and the window matters
  more: monthly retraining on a 60-day window beats monthly retraining
  on all history, because old data describes a world that is gone.

WINDOW LENGTH WITH WEEKLY RETRAINING
  window days   mean RMSE   RMSE in the 30 days after the regime change
            7      1.0102                                       1.1787
           14      1.0021                                       1.2538
           30      1.0163                                       1.4339
           60      1.0473                                       1.6520
          120      1.1160                                       1.7808
          365      1.2640                                       1.8007
  The mean is minimised near two weeks here. Shorter windows recover
  fastest from the shock and are noisier the rest of the year; longer
  windows are stable and slow to adapt. Where shocks are rare, prefer
  the longer window plus a manual retrain when one happens.

FEEDBACK LOOP: WHAT HAPPENS TO TRANSACTIONS THE MODEL BLOCKS
  round   blocked as 0          blocked left unlabelled
          blocked   losses      blocked   losses
      0    20.9%    2,544       20.0%    2,605
      1     1.5%    3,860       19.9%    2,536
      2     4.4%    3,687       20.5%    2,553
      3     1.9%    3,891       18.9%    2,643
      4    10.2%    3,256       20.4%    2,603
      5     4.0%    3,694       20.8%    2,514
  Recording blocked transactions as legitimate teaches the next model
  that high scores are safe. Blocking collapses, fraud losses rise by
  half, and every offline metric looks fine. Treat suppressed outcomes
  as MISSING, and keep a small control group that is never blocked so
  the unseen region still produces labels.

The first table separates the two knobs people usually conflate. Moving from quarterly to monthly retraining on all history buys 4 points; switching that same monthly cadence to a 60-day window buys 15 more. The window is the bigger lever, because averaging over data from a world that no longer exists biases the model toward it. Note also that error-triggered retraining achieved roughly the same error as weekly retraining with a third fewer runs -- good when training is expensive, at the price of a monitoring dependency that must itself be reliable.

The second table is the stability-freshness trade in its raw form. Short windows recover from the regime change fastest and pay for it in variance all year; long windows are stable and adapt slowly. The optimum here is near two weeks, and it is a property of this drift rate and this daily volume -- which is exactly why the window should be chosen by backtesting on your own history rather than inherited from a blog post.

The feedback loop is the deepest hazard on the list, because it is created by the model's own success. A blocked transaction cannot produce a chargeback, so the label table records it as legitimate; the next model learns that high scores are safe, blocking collapses from 21% to a few percent, and true losses rise by half -- while offline metrics on the recorded labels look excellent. The fixes are structural:

• Record suppressed outcomes as missing, not as negative. The right-hand column of that table is the same system with that single change.
• Hold out a small random control group that bypasses the model, so the region the model normally suppresses still produces labels. This is the ML equivalent of exploration, and it costs real money that buys the only unbiased data you will have.
• Log the action, not only the prediction. You cannot correct for a policy you did not record.

Choosing a trigger in practice:

Trigger	Fits	Risk
Fixed schedule	Stable domains, predictable drift	Retrains when nothing changed
Performance-triggered	Labels arrive fast	Blind during label delay
Drift-triggered	Slow labels	Fires on harmless input drift
Data-volume-triggered	Cold-start, growing datasets	Ignores time entirely
Continuous / online	High-velocity domains	Hard to validate; failure compounds

Two final rules. A retrained model is a candidate, not a replacement: it goes through the same gates and the same canary as any other release. And retrain the pipeline, not just the weights -- if the feature code changed since the last run, you are shipping a new model in two respects, and only one of them is in the metrics.

Key Takeaways

• Retraining has four separate decisions: trigger, window, scope, and gate.
• The training window matters more than the cadence; old data describes a world that no longer exists.
• Short windows adapt fast and are noisy; long windows are stable and slow -- backtest to find your own optimum.
• Trigger-based retraining can match scheduled retraining with far fewer runs, at the cost of depending on a monitor.
• Models create their own future training data; suppressed outcomes recorded as negatives corrupt the next model silently.
• Record blocked or withheld cases as missing, keep a randomised control group, and log the action taken alongside the prediction.
• Sudden regime changes deserve a manual retrain on recent data, not a permanently shorter window.
• Every retrained model is a candidate that must pass the same gates and canary as any other release.

🧪 Practice

1. Backtest three retraining cadences on a year of your own data and report mean error and number of retrains for each.
2. Sweep the training window length for a fixed cadence and find the minimum; state how confident you are that the minimum is real.
3. Simulate a policy that suppresses outcomes and show how the estimated model degrades when suppressed cases are labelled negative.
4. Design a control group for a model that takes actions: what fraction, how selected, what it costs, and how you would use it.
5. Interview: A team retrains nightly and their model is slowly getting worse despite stable input distributions. What would you suspect? (Hint: ask where tomorrow's training labels come from, and who decided which rows exist.)

Shadow Deployment and Canary Releases

Offline evaluation answers "would this model have been better on last quarter's data?". Deployment asks a different question: "is it better now, on live traffic, in the real system, without breaking anything?" Releasing in stages is how you answer the second question while limiting what it costs to be wrong.

Four strategies, ordered by how much they risk:

Shadow deployment (dark launch) sends live traffic to the new model in parallel with the old one and discards its output. Users are unaffected. You learn whether it runs, how fast, how often it would disagree, and on which inputs it does something strange.

Canary release routes a small fraction of real traffic to the new model, watches, and ramps up: 1%, 5%, 25%, 100%. Users are affected, which is the point -- outcomes are real.

A/B test splits traffic evenly (or by a designed allocation) and measures business metrics with proper statistics. This is the only way to establish that a model is better rather than merely not broken.

Blue-green keeps two full environments and flips traffic at once. Fast to roll back, but every user is exposed from the first second.

import math
import numpy as np

rng = np.random.default_rng(37)
DAILY = 2_000_000                      # requests per day
BASE, EFFECT = 0.040, -0.004           # 4% conversion, a 10% relative drop

def n_per_arm(p0, delta, alpha=0.05, power=0.80):
    za, zb = 1.959964, 0.8416212
    p1 = p0 + delta
    return math.ceil((za * math.sqrt(2 * p0 * (1 - p0))
                      + zb * math.sqrt(p0 * (1 - p0) + p1 * (1 - p1))) ** 2
                     / delta ** 2)

need = n_per_arm(BASE, EFFECT)
print(f"TO DETECT A {abs(EFFECT) / BASE:.0%} RELATIVE DROP IN A {BASE:.0%} RATE:"
      f" {need:,} users per arm")
print("\nCANARY RAMP: EXPOSURE AGAINST STATISTICAL POWER")
print("  traffic   hours   users in canary   powered?   cumulative lost sales")
harmed = 0
for traffic, hours in [(0.01, 2), (0.01, 24), (0.05, 24), (0.25, 24), (1.0, 24)]:
    users = DAILY * traffic * hours / 24
    harmed += users * abs(EFFECT)                       # lost conversions
    ok = "yes" if users >= need else "no"
    print(f"  {traffic:6.0%} {hours:7d}   {users:15,.0f}   {ok:>8}"
          f"   {harmed:21,.0f}")
print(f"  A 1% canary for two hours exposes {DAILY * 0.01 * 2 / 24:,.0f} users -- enough to")
print("  catch a crash or a latency regression, nowhere near enough to")
print("  measure a 10% conversion drop. Ramps trade exposure for evidence,")
print("  and the early stages are testing the SYSTEM, not the model.")

# --- WHAT EACH RELEASE STRATEGY CAN AND CANNOT TELL YOU. ---
print("\nWHAT EACH STRATEGY ANSWERS")
rows = [
    ("shadow / dark launch", "yes", "yes", "yes", "no",  "no"),
    ("canary 1-5%",          "yes", "yes", "yes", "weak", "yes"),
    ("A/B at 50%",           "yes", "yes", "yes", "yes", "yes"),
    ("blue-green flip",      "yes", "yes", "no",  "yes", "yes"),
    ("interleaving (ranking)", "yes", "yes", "yes", "yes", "yes"),
]
print(f"  {'strategy':22s} {'runs?':>6} {'latency':>8} {'agrees?':>8}"
      f" {'business':>9} {'exposes users':>14}")
for name, runs, lat, agree, biz, exp in rows:
    print(f"  {name:22s} {runs:>6} {lat:>8} {agree:>8} {biz:>9} {exp:>14}")
print("  Shadow mode is the only one that risks nothing -- and the only one")
print("  that cannot measure whether the new model is BETTER for the business.")

# --- SHADOW COMPARISON: WHAT YOU ACTUALLY LEARN BEFORE EXPOSING ANYONE. ---
n = 200_000
champ = rng.beta(2, 8, n)
chall = np.clip(champ + rng.normal(0, 0.05, n), 0, 1)     # mostly agrees
blowup = rng.random(n) < 0.004                            # rare wild outputs
chall[blowup] = rng.random(blowup.sum())
thr = 0.30
print("\nSHADOW RUN: CHALLENGER SCORED ON LIVE TRAFFIC, OUTPUT DISCARDED")
print(f"  decisions that would change: "
      f"{np.mean((champ > thr) != (chall > thr)):.2%}")
print(f"  action rate champion {np.mean(champ > thr):.2%}"
      f" -> challenger {np.mean(chall > thr):.2%}")
print(f"  score correlation: {np.corrcoef(champ, chall)[0, 1]:.4f}")
print(f"  |delta| > 0.5 on {np.mean(np.abs(champ - chall) > 0.5):.3%} of requests"
      "  <- inspect these before ramping")
print("  Shadow answers: does it run, how fast, how often would it disagree,")
print("  and on which inputs does it do something surprising. Those are")
print("  exactly the questions you want answered before a single user is")
print("  affected -- and none of them tell you whether it is better.")

# --- AUTOMATIC ROLLBACK RULES BEAT HUMAN VIGILANCE. ---
print("\nCANARY ABORT RULES (EVALUATED EVERY MINUTE)")
for rule, why in [
        ("error rate > 2x control", "a crash needs no statistics"),
        ("p99 latency > 1.2x control", "regressions show up immediately"),
        ("action rate outside +/-20% of control", "catches a broken feature fast"),
        ("null-prediction rate > 0.1%", "the model is failing quietly"),
        ("business metric worse with 95% CI excluding 0", "needs the full sample")]:
    print(f"  {rule:45s} {why}")
print("  The first four fire in minutes and should roll back automatically.")
print("  The last one takes the whole experiment -- do not wait for it to")
print("  decide whether the release is on fire.")

TO DETECT A 10% RELATIVE DROP IN A 4% RATE: 37,126 users per arm

CANARY RAMP: EXPOSURE AGAINST STATISTICAL POWER
  traffic   hours   users in canary   powered?   cumulative lost sales
      1%       2             1,667         no                       7
      1%      24            20,000         no                      87
      5%      24           100,000        yes                     487
     25%      24           500,000        yes                   2,487
    100%      24         2,000,000        yes                  10,487
  A 1% canary for two hours exposes 1,667 users -- enough to
  catch a crash or a latency regression, nowhere near enough to
  measure a 10% conversion drop. Ramps trade exposure for evidence,
  and the early stages are testing the SYSTEM, not the model.

WHAT EACH STRATEGY ANSWERS
  strategy                runs?  latency  agrees?  business  exposes users
  shadow / dark launch      yes      yes      yes        no             no
  canary 1-5%               yes      yes      yes      weak            yes
  A/B at 50%                yes      yes      yes       yes            yes
  blue-green flip           yes      yes       no       yes            yes
  interleaving (ranking)    yes      yes      yes       yes            yes
  Shadow mode is the only one that risks nothing -- and the only one
  that cannot measure whether the new model is BETTER for the business.

SHADOW RUN: CHALLENGER SCORED ON LIVE TRAFFIC, OUTPUT DISCARDED
  decisions that would change: 7.36%
  action rate champion 19.51% -> challenger 21.10%
  score correlation: 0.9061
  |delta| > 0.5 on 0.120% of requests  <- inspect these before ramping
  Shadow answers: does it run, how fast, how often would it disagree,
  and on which inputs does it do something surprising. Those are
  exactly the questions you want answered before a single user is
  affected -- and none of them tell you whether it is better.

CANARY ABORT RULES (EVALUATED EVERY MINUTE)
  error rate > 2x control                       a crash needs no statistics
  p99 latency > 1.2x control                    regressions show up immediately
  action rate outside +/-20% of control         catches a broken feature fast
  null-prediction rate > 0.1%                   the model is failing quietly
  business metric worse with 95% CI excluding 0 needs the full sample
  The first four fire in minutes and should roll back automatically.
  The last one takes the whole experiment -- do not wait for it to
  decide whether the release is on fire.

The ramp table makes the central trade concrete. A 1% canary for two hours exposes fewer than 2,000 users -- excellent for catching a crash, a memory leak, or a latency regression, and statistically useless for measuring a 10% conversion drop, which needs roughly 37,000 users per arm. Early canary stages test the system; later stages test the model. Treating a green 1% canary as evidence of quality is the most common misreading of this technique.

The same table also shows the cost of ramping too slowly: exposure accumulates. Every additional hour at 25% with a bad model costs real money. The ramp schedule is therefore a decision about the relative cost of harm versus delay, and it should be written down before the release.

The shadow numbers are what a good pre-release report looks like. Correlation of 0.91 with the champion and a 7% decision-change rate is a substantive change worth understanding before exposing anyone, and the 0.12% of requests where the scores differ by more than 0.5 are the ones to inspect by hand -- that is where a broken feature or an encoding bug hides. Shadow mode also catches the unglamorous failures: missing features in the serving path, timeouts under real payload sizes, memory growth over hours.

What shadow mode cannot do is equally important. It cannot measure business impact, because nobody saw the predictions. It cannot reveal feedback effects, because the model took no actions. And it doubles inference cost while it runs. Shadow answers "is it safe?"; only a live experiment answers "is it better?".

Finally, the abort rules. Automatic rollback on error rate, latency, action rate, and null-prediction rate fires within minutes and does not require anyone to be awake. Business-metric comparisons need the full sample and should never be the trigger for stopping an obviously broken release. A canary without automated abort rules is an ordinary deployment with extra steps.

Strategy	Risk to users	Answers "safe?"	Answers "better?"	Extra cost
Shadow	None	Yes	No	2x inference
Canary 1-5%	Small	Yes	Weakly	Small
A/B 50/50	Half of traffic	Yes	Yes	Small
Blue-green	All at once	Partly	Yes	2x environment
Interleaving	Small, per result	Yes	Yes (ranking only)	Small

Key Takeaways

• Staged release limits the cost of being wrong; the stages answer different questions.
• Shadow mode runs the challenger on live traffic and discards output: zero user risk, no business evidence.
• Compare shadow predictions against the champion and hand-inspect the largest disagreements before ramping.
• Small canaries have no statistical power for business metrics; they test that the system works.
• Compute the sample size your target effect requires, and plan the ramp around it.
• Ramp speed trades exposure against delay; decide the schedule before the release, not during it.
• Automate abort on error rate, latency, action rate, and null predictions; these fire in minutes.
• Never gate an emergency rollback on a business metric that needs the full sample.

🧪 Practice

1. Run a model in shadow for an hour and report agreement rate, action-rate change, and the ten largest disagreements.
2. Compute the sample size needed to detect a 5% relative change in your main business metric, and convert it into canary hours.
3. Write the abort rules for your next release, with thresholds and the evaluation window for each.
4. Design a ramp schedule and estimate the expected cost of a bad release under it, compared with a blue-green flip.
5. Interview: A canary at 1% looks fine after an hour. Is the model ready for 100%? (Hint: work out what that hour could and could not have detected, and what you would still not know.)

<a id="154-collaboration-and-governance-of-code"></a>

15.4 Collaboration and Governance of Code

Everything so far has been about systems; this section is about the people who maintain them. It covers how to review code whose bugs are semantic rather than syntactic, what documentation actually gets read, how to describe a model and its data honestly for the people who will depend on it, the forms of technical debt specific to ML, and how work moves between the people who build models and the people who run them.

Code Review for Data Projects

Reviewing ordinary software means asking whether the code does what it claims. Reviewing data code means asking whether what it claims is the right thing -- because the failure modes are semantic. Data code rarely crashes; it produces a number that is wrong in a way no test detects unless someone wrote a test for that specific misunderstanding.

The canonical example is four lines long and passes every linter:

import numpy as np
import pandas as pd

rng = np.random.default_rng(19)
N = 6000
df = pd.DataFrame({
    # A high-cardinality column with NO real signal: 3,000 distinct values.
    "merchant": rng.integers(0, 3000, N),
    "amount": rng.lognormal(3.5, 0.7, N),
})
df["y"] = (rng.random(N) < 0.25).astype(int)     # label independent of merchant

# --- THE VERSION THAT ARRIVES IN A PULL REQUEST ---
def target_encode_leaky(df, col, target):
    means = df.groupby(col)[target].mean()       # computed over ALL rows,
    return df[col].map(means)                    # including each row's own label

# --- THE VERSION THAT SURVIVES REVIEW ---
def target_encode_oof(df, col, target, folds):
    out = pd.Series(index=df.index, dtype=float)
    prior = df[target].mean()
    for tr, te in folds:
        m = df.iloc[tr].groupby(col)[target].mean()
        # Smoothing towards the prior; unseen categories fall back to it.
        out.iloc[te] = df.iloc[te][col].map(m).fillna(prior)
    return out

folds = []
idx = rng.permutation(N)
for k in range(5):
    te = idx[k * (N // 5):(k + 1) * (N // 5)]
    folds.append((np.setdiff1d(idx, te), te))

def cv_auc(feature, y, folds):
    aucs = []
    for _, te in folds:
        s, t = feature.to_numpy()[te], y.to_numpy()[te]
        r = pd.Series(s).rank().to_numpy()
        n1, n0 = t.sum(), len(t) - t.sum()
        aucs.append((r[t == 1].sum() - n1 * (n1 + 1) / 2) / (n1 * n0))
    return float(np.mean(aucs))

leaky = target_encode_leaky(df, "merchant", "y")
clean = target_encode_oof(df, "merchant", "y", folds)

leaky_auc = cv_auc(leaky, df.y, folds)
print("TARGET ENCODING OF A COLUMN WITH NO SIGNAL AT ALL")
print(f"  leaky encoding, cross-validated AUC   {leaky_auc:.4f}")
print(f"  out-of-fold encoding, same CV         {cv_auc(clean, df.y, folds):.4f}")
print(f"  true signal in the column                0.5000")
print(f"  The leaky version invents {leaky_auc - 0.5:.2f} AUC out of nothing: with"
      f" {N / 3000:.0f} rows")
print("  per merchant, each row's own label dominates its own feature. It is a")
print("  four-line function, and the bug is in WHEN the mean is computed.")

# --- WHAT A REVIEWER SHOULD ACTUALLY LOOK FOR, BY FREQUENCY. ---
print("\nREVIEW CHECKLIST FOR DATA CODE, MOST-MISSED FIRST")
for item, question in [
        ("leakage", "does any feature use information unavailable at decision time?"),
        ("split hygiene", "is the split by time/entity, and is it applied before fitting?"),
        ("silent defaults", "what does fillna(0) mean here, and is 0 a real value?"),
        ("units and scale", "are amounts, rates, and durations in documented units?"),
        ("row-count changes", "does this join change the number of rows?"),
        ("reproducibility", "is every random source seeded and every date a parameter?"),
        ("segment impact", "which group does this change make worse?"),
        ("cost", "how much does this scan, and how often does it run?"),
]:
    print(f"  {item:18s} {question}")
print("  A data-code review is mostly about SEMANTICS. The code can be clean,")
print("  idiomatic, typed, tested -- and still answer the wrong question.")

TARGET ENCODING OF A COLUMN WITH NO SIGNAL AT ALL
  leaky encoding, cross-validated AUC   0.8967
  out-of-fold encoding, same CV         0.4868
  true signal in the column                0.5000
  The leaky version invents 0.40 AUC out of nothing: with 2 rows
  per merchant, each row's own label dominates its own feature. It is a
  four-line function, and the bug is in WHEN the mean is computed.

REVIEW CHECKLIST FOR DATA CODE, MOST-MISSED FIRST
  leakage            does any feature use information unavailable at decision time?
  split hygiene      is the split by time/entity, and is it applied before fitting?
  silent defaults    what does fillna(0) mean here, and is 0 a real value?
  units and scale    are amounts, rates, and durations in documented units?
  row-count changes  does this join change the number of rows?
  reproducibility    is every random source seeded and every date a parameter?
  segment impact     which group does this change make worse?
  cost               how much does this scan, and how often does it run?
  A data-code review is mostly about SEMANTICS. The code can be clean,
  idiomatic, typed, tested -- and still answer the wrong question.

The leaky encoder is correct Python, correct pandas, and correct arithmetic. It computes a group mean and joins it back. It also gives a column with zero signal an AUC of 0.90, because with two rows per merchant each row's own label dominates its own feature. No unit test fails. Test coverage is 100%. The model will look excellent in the report and approximately random in production.

That is why a data-code review needs a different checklist from a service review. The most valuable questions are about time, joins, and defaults:

• Time. Would this value have been known at the moment of the decision? Any aggregation, any join to a dimension table, and any "current status" column deserves the question explicitly.
• Joins. Did the row count change? A left join that silently duplicates rows changes every downstream mean and every model weight.
• Defaults. fillna(0) is a modelling decision written as a convenience. Is zero a legitimate value for this column? If so, the model cannot distinguish "none" from "missing".
• Units. Cents or euros, seconds or milliseconds, inclusive or exclusive bounds. Encode the answer in the column name where you can.
• Segments. Which group does this change make worse? Aggregate metrics will not say.

Practical review mechanics that make this sustainable:

Practice	Why it works for data code
Small pull requests	A 2,000-line notebook diff gets approved unread
Show the numbers	Include before/after metrics and row counts in the description
Review the diff of outputs	Not only the code: a sample of changed predictions
Notebooks reviewed as exports	Strip outputs, or review the extracted module instead
A checklist in the PR template	Leakage and split questions get asked every time
Pair review for feature code	The most expensive bugs live here

One organisational note. Reviews work only if the reviewer has context; a reviewer who does not know the domain can check style and will miss every semantic bug. For feature and metric changes, the right reviewer is someone who knows what the column means -- which is sometimes an analyst rather than an engineer.

Key Takeaways

• Data code fails semantically: it produces wrong numbers rather than errors, and tests pass.
• Target encoding computed over the whole dataset gave a signal-free column an AUC of 0.90; the bug is when the mean is computed, not how.
• Review for time (was this knowable?), joins (did row count change?), and defaults (what does fillna(0) assert?).
• Ask which segment a change makes worse, since aggregates hide it.
• Keep pull requests small and include before/after metrics and row counts in the description.
• Review a sample of changed outputs, not only the code diff.
• Review notebooks as extracted modules with outputs stripped.
• Assign reviewers who know what the columns mean, not only how the code is written.

🧪 Practice

1. Write a leaky and an out-of-fold version of a target encoder and compare cross-validated performance on a signal-free column.
2. Build a pull-request checklist for your team covering leakage, joins, defaults, units, and segments.
3. Take a recent merged PR in a data repository and audit it against that checklist; record what would have been caught.
4. Add a CI step that reports row-count changes and null-rate changes for each modified transformation.
5. Interview: What would you look for in a pull request that adds a new feature to a fraud model? (Hint: start from the timestamp of the data each part of the feature reads, and work forward to the decision moment.)

Documentation Standards

Documentation fails for a predictable reason: it is written for nobody in particular, and so it is read by nobody in particular. The fix is to write each artifact for a named audience with a specific question, and to generate anything that can be generated so it cannot go stale.

Five audiences, five artifacts:

Audience	Artifact	The question it answers
Future you	Docstrings, README	How do I run this, and where is everything?
New teammate	Architecture note	Why is it shaped like this?
Consumer of outputs	Data dictionary, model card	What do these numbers mean?
Auditor or regulator	Decision records, lineage	Who decided what, on what evidence?
On-call engineer	Runbook	What do I do right now?

The highest-value artifact in a data project is the one most often missing: a data dictionary that says what each column means, in what units, from which source, with what caveats. Half of it can be generated:

import numpy as np
import pandas as pd

rng = np.random.default_rng(8)
df = pd.DataFrame({
    "user_id": rng.integers(1, 5000, 4000),
    "amount_eur": rng.lognormal(3.4, 0.7, 4000).round(2),
    "tenure_days": rng.integers(0, 1500, 4000).astype(float),
    "channel": rng.choice(["web", "ios", "android"], 4000),
    "is_active": rng.random(4000) < 0.7,
})
df.loc[df.sample(frac=0.07, random_state=0).index, "tenure_days"] = np.nan

# --- DOCUMENTATION AS DATA: THE PART HUMANS WRITE. ---
DOCS = {
    "user_id":     ("Stable internal user identifier.", "entity key", "identity-svc"),
    "amount_eur":  ("Transaction value, euros, VAT included.", "euros", "payments.tx"),
    "tenure_days": ("Days since signup at event time. Null before backfill "
                    "of 2024 accounts.", "days", "identity-svc"),
    "channel":     ("Client that originated the request.", "enum", "gateway logs"),
}

def data_dictionary(df, docs):
    rows = []
    for col in df.columns:
        desc, unit, source = docs.get(col, ("** UNDOCUMENTED **", "?", "?"))
        s = df[col]
        if pd.api.types.is_numeric_dtype(s) and not pd.api.types.is_bool_dtype(s):
            rng_ = f"{np.nanmin(s):.1f} .. {np.nanmax(s):.1f}"
        else:
            vals = sorted(map(str, pd.unique(s)))[:3]
            rng_ = ", ".join(vals) + ("..." if s.nunique() > 3 else "")
        rows.append({"column": col, "dtype": str(s.dtype),
                     "null %": f"{s.isna().mean():.1%}", "range/values": rng_,
                     "unit": unit, "source": source, "description": desc})
    return pd.DataFrame(rows)

dd = data_dictionary(df, DOCS)
print("GENERATED DATA DICTIONARY (profile from the data, meaning from humans)")
print(dd[["column", "dtype", "null %", "range/values", "unit"]]
      .to_string(index=False))
print("\nDESCRIPTIONS")
for _, r in dd.iterrows():
    print(f"  {r['column']:14s} {r['description']}")

missing = dd[dd.description.str.startswith("**")]["column"].tolist()
print(f"\nUNDOCUMENTED COLUMNS: {missing}")
print("  Make this a CI check: a new column without an entry fails the build.")
print("  Documentation that is generated stays current; documentation that is")
print("  maintained by hand describes the schema of eighteen months ago.")

# --- THE THREE AUDIENCES, AND WHAT EACH NEEDS TO READ. ---
print("\nWHO READS WHAT")
for audience, artifact, contents in [
        ("future you", "docstrings + README", "how to run it, where things live"),
        ("a new teammate", "repo README + architecture note", "why it is shaped this way"),
        ("a consumer", "data dictionary + model card", "what the outputs mean and when to trust them"),
        ("an auditor", "decision records + lineage", "what was decided, by whom, on what evidence"),
        ("on-call", "runbook", "what to do at 3 a.m., with commands")]:
    print(f"  {audience:15s} {artifact:32s} {contents}")
print("  Note that none of these audiences is served by a comment explaining")
print("  what a line of code does. Document decisions and meanings; the code")
print("  already documents the mechanics.")

GENERATED DATA DICTIONARY (profile from the data, meaning from humans)
     column   dtype null %      range/values       unit
    user_id   int64   0.0%     1.0 .. 4998.0 entity key
 amount_eur float64   0.0%      2.9 .. 377.1      euros
tenure_days float64   7.0%     0.0 .. 1499.0       days
    channel     str   0.0% android, ios, web       enum
  is_active    bool   0.0%       False, True          ?

DESCRIPTIONS
  user_id        Stable internal user identifier.
  amount_eur     Transaction value, euros, VAT included.
  tenure_days    Days since signup at event time. Null before backfill of 2024 accounts.
  channel        Client that originated the request.
  is_active      ** UNDOCUMENTED **

UNDOCUMENTED COLUMNS: ['is_active']
  Make this a CI check: a new column without an entry fails the build.
  Documentation that is generated stays current; documentation that is
  maintained by hand describes the schema of eighteen months ago.

WHO READS WHAT
  future you      docstrings + README              how to run it, where things live
  a new teammate  repo README + architecture note  why it is shaped this way
  a consumer      data dictionary + model card     what the outputs mean and when to trust them
  an auditor      decision records + lineage       what was decided, by whom, on what evidence
  on-call         runbook                          what to do at 3 a.m., with commands
  Note that none of these audiences is served by a comment explaining
  what a line of code does. Document decisions and meanings; the code
  already documents the mechanics.

The split is the point. Profile statistics -- dtype, null rate, range, distinct values -- are generated from the data on every run and are therefore always true. Meaning, units, source, and caveats are written by a human once and reviewed when they change. Then a CI check fails the build when a new column has no entry, which is the only mechanism that reliably keeps documentation current: make its absence an error.

For decisions, use architecture decision records (ADRs) -- short, dated, immutable notes:

# ADR-014: Serve churn scores from a nightly batch table

Date: 2026-09-14
Status: accepted (supersedes ADR-009)

## Context
Scores are consumed by the CRM, which syncs once a day at 06:00. Freshness
requirement confirmed with the CRM team: 24 hours is acceptable. Online
serving would need a feature store we do not yet operate.

## Decision
Precompute scores nightly into `analytics.churn_scores`, keyed by user_id
and scored_at, retained 90 days.

## Consequences
+ No serving infrastructure, no latency budget, trivial rollback (rerun).
+ Scores are up to 30 hours stale at the end of the day.
- A future in-session use case will need a different path; revisit then.

An ADR takes fifteen minutes and answers the question that otherwise costs a new engineer two days: why is it like this? Record the decision when you make it, keep the record immutable, and supersede rather than edit.

Three more standards worth adopting:

• Docstrings state contracts, not mechanics. Units, shapes, assumptions, and what happens with nulls -- not a restatement of the code.
• The README answers four questions: what this is, how to run it, how to test it, and who owns it. Everything else can be a link.
• Runbooks contain commands, not prose. At 3 a.m. nobody reads a paragraph; they copy a command.

Key Takeaways

• Write each document for a named audience and the question they arrive with; unaddressed documentation goes unread.
• Generate what can be generated (profiles, schemas, lineage) so it cannot drift from reality.
• A data dictionary pairs generated statistics with human-written meaning, units, source, and caveats.
• Enforce documentation in CI: an undocumented new column should fail the build.
• Use short, dated, immutable ADRs to record why decisions were made, and supersede rather than edit them.
• Docstrings should state contracts -- units, shapes, null behaviour -- not narrate the code.
• A README answers what, how to run, how to test, and who owns it.
• Runbooks are lists of commands, written for someone under pressure.

🧪 Practice

1. Generate a data dictionary for one of your tables and mark every column whose meaning you cannot state precisely.
2. Add a CI check that fails when a column exists without a documented description.
3. Write an ADR for a decision your team made in the last month, including the alternatives you rejected.
4. Rewrite one docstring so it states units, shapes, and null behaviour instead of describing the implementation.
5. Interview: A new engineer joins and must ship a change to a model pipeline in their first week. What documentation must exist for that to be realistic? (Hint: separate what they need to run the thing from what they need to avoid breaking something invisible.)

Model Cards and Datasheets

A model is handed to people who did not build it: another team integrating it, a manager deciding whether to rely on it, a reviewer asking whether it is fair, an engineer inheriting it in two years. Each of them needs the same things, and none of them can get those things from the code.

Model cards (Mitchell et al., 2019) document a trained model; datasheets for datasets (Gebru et al., 2018) document the data it was trained on. Both started as fairness and transparency proposals and turned out to be plain good engineering: they force the questions that otherwise surface during an incident.

A model card has seven sections, and the value is concentrated in the ones that are uncomfortable to write:

Section	Contents	Why it matters
Overview	Owner, version, artifact, lineage	Who to call; what is actually live
Intended use	What it is for	Prevents the model being reused wrongly
Out of scope	What it is not for	The section that prevents harm
Training data	Sources, period, known gaps	Explains where it will be weak
Evaluation	Metrics overall and by slice	Aggregate numbers hide failures
Limitations	Known weaknesses, assumptions	Sets expectations before deployment
Maintenance	Monitoring, retraining, rollback	Makes ownership concrete

The evaluation section is the one with real content, because it must be disaggregated:

import numpy as np
import pandas as pd

rng = np.random.default_rng(44)
N = 40_000
d = pd.DataFrame({
    "country": rng.choice(["US", "GB", "DE", "BR"], N, p=[.55, .2, .2, .05]),
    "device": rng.choice(["desktop", "mobile"], N, p=[.4, .6]),
    "x": rng.normal(size=N),
})
# The model is genuinely weaker where training data was thin (BR).
strength = d.country.map({"US": 1.6, "GB": 1.5, "DE": 1.5, "BR": 0.4}).to_numpy()
d["y"] = (rng.random(N) < 1 / (1 + np.exp(-(strength * d.x)))).astype(int)
d["score"] = 1 / (1 + np.exp(-(1.55 * d.x)))          # one model for everyone

def auc(s, y):
    r = pd.Series(s).rank().to_numpy()
    n1, n0 = y.sum(), len(y) - y.sum()
    return float((r[y == 1].sum() - n1 * (n1 + 1) / 2) / (n1 * n0))

THRESHOLD = 0.5
def slice_metrics(g):
    pred = (g.score > THRESHOLD).astype(int)
    pos = g.y == 1
    return pd.Series({
        "n": len(g),
        "share": len(g) / len(d),
        "AUC": auc(g.score.to_numpy(), g.y.to_numpy()),
        "accuracy": float((pred == g.y).mean()),
        "action rate": float(pred.mean()),
        "recall": float(pred[pos].mean()),
    })

by_country = d.groupby("country").apply(slice_metrics, include_groups=False)
by_device = d.groupby("device").apply(slice_metrics, include_groups=False)
overall = slice_metrics(d).to_frame("overall").T

print("SLICE EVALUATION -- THE TABLE A MODEL CARD IS BUILT AROUND")
print(pd.concat([overall, by_country, by_device])
      .to_string(float_format=lambda v: f"{v:8.3f}"))
print("  Overall AUC looks healthy. BR is 5% of traffic, has materially lower")
print("  AUC and recall, and would be invisible in any aggregate report.")

# --- THE CARD ITSELF: GENERATED WHERE POSSIBLE, WRITTEN WHERE NECESSARY. ---
card = f"""
# Model Card: purchase-intent v7

## Overview
- Owner: growth-ml team, on-call rota `growth-ml-oncall`
- Version: v7 (registry id 4181), promoted 2026-09-20
- Artifact: s3://models/intent/7d21  |  code 7c19aa0  |  data orders@2026-09-01

## Intended use
- INTENDED: ranking on-site recommendations for signed-in users.
- NOT INTENDED: credit decisions, pricing, or any decision about a person's
  eligibility. The model was not evaluated for those uses.
- Out of scope: users in markets launched after 2026-09; anonymous traffic.

## Training data
- Source: orders and sessions, 2025-09-01 to 2026-09-01, {len(d):,} rows.
- Known gaps: BR represents {float(by_country.loc['BR', 'share']):.0%} of traffic and was
  under-sampled before 2026-03; labels there are noisier.

## Evaluation
- Overall AUC {float(overall.loc['overall', 'AUC']):.3f} on a time-based holdout.
- Worst slice: BR, AUC {float(by_country.loc['BR', 'AUC']):.3f},
  recall {float(by_country.loc['BR', 'recall']):.3f}
  against US recall {float(by_country.loc['US', 'recall']):.3f}.

## Limitations and risks
- Performance degrades in markets with sparse history; do not extend to a new
  market without a fresh evaluation.
- Scores are calibrated for the 0.5 threshold only; recalibrate if the
  threshold moves.
- Retraining depends on a 30-day label delay; expect stale behaviour after a
  sudden change in customer mix.

## Monitoring and maintenance
- Monitored: AUC weekly per country, action rate daily, feature nulls hourly.
- Retrained: monthly, 90-day window; rollback to v6 is tested monthly.
"""
print(card)
print("The sections that matter most are the ones nobody enjoys writing:")
print("what the model is NOT for, where it is weakest, and who to call.")

SLICE EVALUATION -- THE TABLE A MODEL CARD IS BUILT AROUND
                n    share      AUC  accuracy  action rate   recall
overall 40000.000    1.000    0.811     0.735        0.499    0.734
BR       1981.000    0.050    0.609     0.572        0.489    0.561
DE       7994.000    0.200    0.816     0.737        0.498    0.736
GB       8022.000    0.201    0.814     0.734        0.496    0.728
US      22003.000    0.550    0.827     0.750        0.502    0.750
desktop 15908.000    0.398    0.811     0.735        0.500    0.733
mobile  24092.000    0.602    0.811     0.735        0.499    0.734
  Overall AUC looks healthy. BR is 5% of traffic, has materially lower
  AUC and recall, and would be invisible in any aggregate report.

# Model Card: purchase-intent v7

## Overview
- Owner: growth-ml team, on-call rota `growth-ml-oncall`
- Version: v7 (registry id 4181), promoted 2026-09-20
- Artifact: s3://models/intent/7d21  |  code 7c19aa0  |  data orders@2026-09-01

## Intended use
- INTENDED: ranking on-site recommendations for signed-in users.
- NOT INTENDED: credit decisions, pricing, or any decision about a person's
  eligibility. The model was not evaluated for those uses.
- Out of scope: users in markets launched after 2026-09; anonymous traffic.

## Training data
- Source: orders and sessions, 2025-09-01 to 2026-09-01, 40,000 rows.
- Known gaps: BR represents 5% of traffic and was
  under-sampled before 2026-03; labels there are noisier.

## Evaluation
- Overall AUC 0.811 on a time-based holdout.
- Worst slice: BR, AUC 0.609,
  recall 0.561
  against US recall 0.750.

## Limitations and risks
- Performance degrades in markets with sparse history; do not extend to a new
  market without a fresh evaluation.
- Scores are calibrated for the 0.5 threshold only; recalibrate if the
  threshold moves.
- Retraining depends on a 30-day label delay; expect stale behaviour after a
  sudden change in customer mix.

## Monitoring and maintenance
- Monitored: AUC weekly per country, action rate daily, feature nulls hourly.
- Retrained: monthly, 90-day window; rollback to v6 is tested monthly.

The sections that matter most are the ones nobody enjoys writing:
what the model is NOT for, where it is weakest, and who to call.

The overall AUC of 0.81 is a fine number and tells the reader nothing about Brazil, where the model is close to useless. A card that reports only the aggregate is not merely incomplete -- it actively misleads the next team, who will reasonably assume the model works everywhere. The slice table converts a hidden weakness into a documented limitation, which is the difference between a known constraint and a future incident.

A datasheet for a dataset answers a parallel set of questions about the data itself: why it was collected, by whom, how instances were sampled, whether it contains personal or sensitive information, what preprocessing was applied, what is missing and why, who maintains it, and how it may be used. For any dataset that outlives its original project -- which is most of them -- this is the difference between an asset and a liability.

Three practices keep cards honest:

• Generate what you can. Version, lineage, and every metric in the evaluation table should come from the training run, not from a person's memory at writing time.
• Write the card before promotion, not after. A card produced as part of the promotion gate is accurate; one produced for an audit is archaeology.
• Review the "out of scope" section with the consumers. Most model misuse is not malicious -- it is a team applying a model to a neighbouring problem because nobody said not to.

For regulated contexts (EU AI Act, model risk management in finance, medical device rules) these documents are frequently the required deliverable. Even outside them, the exercise is worth the hour: if you cannot state the intended use, the known gaps, and the worst slice, you do not yet understand the model well enough to deploy it.

Key Takeaways

• Model cards document a model; datasheets document a dataset. Both exist so people who did not build the thing can use it responsibly.
• The valuable sections are intended use, out of scope, known data gaps, sliced evaluation, and limitations.
• Report metrics by slice: an aggregate AUC of 0.81 concealed a slice where the model was near-useless.
• Naming what a model is not for prevents the most common form of harm, which is reasonable reuse in a neighbouring context.
• Generate lineage and metrics automatically; write judgement sections by hand.
• Produce the card as part of the promotion gate, not retrospectively.
• Datasheets record collection purpose, sampling, sensitive content, preprocessing, gaps, and maintenance.
• In regulated settings these documents are often mandatory; elsewhere they are still the cheapest way to surface what you do not know.

🧪 Practice

1. Write a model card for a model you have built, including an out-of-scope section listing three plausible misuses.
2. Produce a sliced evaluation across at least three dimensions and identify the worst-performing slice with more than 1% of traffic.
3. Automate the generation of the overview and evaluation sections from your training run's artifacts.
4. Write a datasheet for a dataset your team maintains, including what is missing and why.
5. Interview: Another team wants to reuse your churn model to decide who gets priority support. What do you tell them? (Hint: the answer depends on what the model was trained to predict and on whom it was evaluated, and at least one part of it belongs in a document rather than a chat.)

Technical Debt in ML Systems

The classic paper on this is Sculley et al., Hidden Technical Debt in Machine Learning Systems (2015), and its central observation still holds: the model is a small box in a large diagram, and the debt accumulates everywhere else. ML systems carry all of ordinary software's debt plus a category of their own, because they have dependencies on data, and data dependencies cost more than code dependencies -- they are harder to see, harder to test, and nobody gets a compiler error when one changes.

The signature problem has a name: CACE, Changing Anything Changes Everything. A model has no modular interfaces internally; every input affects every output. Add a feature, change an encoding, drop a category, and all the weights move. This makes ordinary refactoring risky in a way that is unfamiliar to software engineers.

from collections import defaultdict

# --- WHAT THE TEAM BELIEVES IT OWNS. ---
features = {                     # name: (daily cost $, importance in v7)
    "avg_order_30d":   (12.0, 0.31),
    "days_since_last": (3.0, 0.24),
    "device_risk":     (48.0, 0.02),   # expensive, and the model barely uses it
    "session_depth":   (9.0, 0.11),
    "promo_ratio":     (6.0, 0.00),    # zero importance: nobody removed it
    "legacy_ltv":      (31.0, 0.01),   # inherited from a deleted model
    "weather_index":   (7.0, 0.00),    # built for an experiment in 2024
}
models = {
    "intent_v7":  ["avg_order_30d", "days_since_last", "session_depth", "device_risk"],
    "churn_v3":   ["avg_order_30d", "days_since_last", "legacy_ltv"],
    "upsell_v1":  ["session_depth", "promo_ratio"],
}
# --- WHAT THE ACCESS LOGS SAY. ---
observed_readers = {
    "intent_v7":  ["web-ranking", "email-campaign", "analytics-dashboard"],
    "churn_v3":   ["crm-sync"],
    "upsell_v1":  [],
    "avg_order_30d": ["finance-report"],      # reading a FEATURE directly
}
declared_consumers = {"intent_v7": ["web-ranking"], "churn_v3": ["crm-sync"]}

print("FEATURE FAN-OUT: WHAT BREAKS IF THIS CHANGES")
used_by = defaultdict(list)
for m, fs in models.items():
    for f in fs:
        used_by[f].append(m)
for f, (cost, imp) in sorted(features.items(), key=lambda kv: -kv[1][0]):
    consumers = used_by.get(f, [])
    print(f"  {f:17s} ${cost:5.1f}/day  importance {imp:4.2f}"
          f"  used by {len(consumers)}: {', '.join(consumers) or 'NOTHING'}")
print("  CACE -- Changing Anything Changes Everything. A feature read by two")
print("  models cannot be modified in isolation: the change is a release for")
print("  both of them, whether or not their owners are in the meeting.")

print("\nDEAD AND OVERPRICED FEATURES")
waste = 0.0
for f, (cost, imp) in features.items():
    if not used_by.get(f):
        print(f"  {f:17s} ${cost:5.1f}/day  used by no model            -> delete")
        waste += cost
    elif imp < 0.03:
        print(f"  {f:17s} ${cost:5.1f}/day  importance {imp:4.2f}, keeps a"
              f" pipeline alive -> ablate")
        waste += cost
print(f"  suspect spend: ${waste:,.1f}/day = ${waste * 365:,.0f}/year")
print("  Removing a feature requires measuring the model WITHOUT it. Teams")
print("  skip that step, so features accumulate forever. Budget the ablation.")

print("\nUNDECLARED CONSUMERS (found in access logs, not in any design doc)")
for producer, readers in observed_readers.items():
    undeclared = set(readers) - set(declared_consumers.get(producer, []))
    if undeclared:
        print(f"  {producer:17s} also read by: {', '.join(sorted(undeclared))}")
print("  'analytics-dashboard' and 'finance-report' now depend on outputs")
print("  nobody promised to keep stable. The next threshold change breaks a")
print("  report somebody presents to the board. Publish an interface or")
print("  deprecate the access -- but know that it exists.")

print("\nTHE ML-SPECIFIC DEBT CATALOGUE")
for name, symptom, payment in [
        ("entanglement (CACE)", "one feature, many models", "isolate, version features"),
        ("undeclared consumers", "outputs read by unknown systems", "publish contracts"),
        ("pipeline jungle", "scripts calling scripts", "one orchestrated DAG"),
        ("dead features", "computed, never used", "ablate and delete"),
        ("glue code", "90% of the repo is plumbing", "standard interfaces"),
        ("configuration debt", "50 flags, 3 documented", "typed, validated config"),
        ("correction cascades", "model B patches model A", "retrain A properly"),
        ("feedback loops", "the model shapes its own data", "control groups, logging")]:
    print(f"  {name:21s} {symptom:34s} -> {payment}")
print("  None of these appear in code quality metrics. All of them determine")
print("  how long the next change takes.")

FEATURE FAN-OUT: WHAT BREAKS IF THIS CHANGES
  device_risk       $ 48.0/day  importance 0.02  used by 1: intent_v7
  legacy_ltv        $ 31.0/day  importance 0.01  used by 1: churn_v3
  avg_order_30d     $ 12.0/day  importance 0.31  used by 2: intent_v7, churn_v3
  session_depth     $  9.0/day  importance 0.11  used by 2: intent_v7, upsell_v1
  weather_index     $  7.0/day  importance 0.00  used by 0: NOTHING
  promo_ratio       $  6.0/day  importance 0.00  used by 1: upsell_v1
  days_since_last   $  3.0/day  importance 0.24  used by 2: intent_v7, churn_v3
  CACE -- Changing Anything Changes Everything. A feature read by two
  models cannot be modified in isolation: the change is a release for
  both of them, whether or not their owners are in the meeting.

DEAD AND OVERPRICED FEATURES
  device_risk       $ 48.0/day  importance 0.02, keeps a pipeline alive -> ablate
  promo_ratio       $  6.0/day  importance 0.00, keeps a pipeline alive -> ablate
  legacy_ltv        $ 31.0/day  importance 0.01, keeps a pipeline alive -> ablate
  weather_index     $  7.0/day  used by no model            -> delete
  suspect spend: $92.0/day = $33,580/year
  Removing a feature requires measuring the model WITHOUT it. Teams
  skip that step, so features accumulate forever. Budget the ablation.

UNDECLARED CONSUMERS (found in access logs, not in any design doc)
  intent_v7         also read by: analytics-dashboard, email-campaign
  avg_order_30d     also read by: finance-report
  'analytics-dashboard' and 'finance-report' now depend on outputs
  nobody promised to keep stable. The next threshold change breaks a
  report somebody presents to the board. Publish an interface or
  deprecate the access -- but know that it exists.

THE ML-SPECIFIC DEBT CATALOGUE
  entanglement (CACE)   one feature, many models           -> isolate, version features
  undeclared consumers  outputs read by unknown systems    -> publish contracts
  pipeline jungle       scripts calling scripts            -> one orchestrated DAG
  dead features         computed, never used               -> ablate and delete
  glue code             90% of the repo is plumbing        -> standard interfaces
  configuration debt    50 flags, 3 documented             -> typed, validated config
  correction cascades   model B patches model A            -> retrain A properly
  feedback loops        the model shapes its own data      -> control groups, logging
  None of these appear in code quality metrics. All of them determine
  how long the next change takes.

Three patterns in that report recur in every mature ML codebase.

Entanglement. A feature read by several models cannot be changed by one team alone. The fix is not to avoid sharing -- sharing is the point of a feature store -- but to version feature definitions, so a change creates avg_order_30d@v2 and each model migrates deliberately.

Dead and overpriced features. Nearly a third of the daily feature spend here supports features with importance below 0.03, plus one that no model reads at all. They persist because removing a feature is not free: you must retrain without it, measure the loss, and be willing to accept it. Nobody schedules that work, so the pipeline grows monotonically. Budget a periodic ablation review -- and note that the same review usually finds a pipeline that can be deleted entirely, which is where the real saving is.

Undeclared consumers. This is the most dangerous item on the list. Once a model's output lands in a table, other systems read it, and none of them appear in your design document. Your next threshold change breaks a finance report. The remedies are ordinary interface discipline applied to data: publish a stable, documented output contract; version it; log who reads it; and use access controls so that new consumers must ask rather than discover.

The rest of the catalogue in the output is worth periodically walking as a checklist. Add to it two habits that keep debt visible rather than accumulating:

• Track the cost of the pipeline, not only its accuracy. A feature's value is its contribution minus its daily cost and its maintenance burden.
• Give every model an end date. A scheduled review -- is this still used, still correct, still worth its cost? -- is the only reliable mechanism for deleting models, and undeleted models are where most of this debt lives.

Key Takeaways

• ML debt is dominated by data dependencies, which are invisible to compilers and to code review.
• CACE: models have no internal modularity, so any input change moves every output.
• A shared feature cannot be changed by one team alone; version feature definitions so migration is deliberate.
• Dead and low-importance features persist because removing them requires an ablation nobody schedules.
• Undeclared consumers turn every output change into an unplanned breakage; publish contracts and log readers.
• Watch for pipeline jungles, glue code, configuration debt, correction cascades, and feedback loops.
• Track pipeline cost alongside accuracy so a feature's net value is visible.
• Review models on a schedule and delete the ones that are no longer worth their cost.

🧪 Practice

1. Build the feature-to-model dependency map for your project and identify every feature used by more than one model.
2. Estimate the daily cost of your three most expensive features and compare it with their importance.
3. Ablate one low-importance feature, measure the change, and decide whether to remove it.
4. Search access logs for consumers of one model's output table and compare them with the documented consumers.
5. Interview: You want to change how a widely used feature is computed. What is your plan? (Hint: the technical change is the easy part; think about versions, migration, and who must agree.)

Team Workflows and Handoffs

Most ML work crosses a boundary at some point: from an analyst who built a model to engineers who will run it, from one team to another during a reorganisation, from a person leaving to whoever is left. What survives that crossing is determined long before it happens, by how the work was organised.

Start with the workflow, because ML repositories accumulate a specific mess: exploratory code and production code in the same place, indistinguishable. The convention that works is a one-way path -- code moves from notebook to module, never back:

  notebooks/     exploration only, never imported, outputs stripped in git
  src/           importable modules: features, training, evaluation, serving
  configs/       one file per run; no constants buried in code
  tests/         unit, behavioural, pipeline
  pipelines/     orchestration definitions (DAGs, schedules)
  docs/          ADRs, model cards, runbooks, data dictionary

  Rule: if a notebook cell has been run more than twice by two people,
  it belongs in src/ with a test. Notebooks are for questions, not
  for answers other people depend on.

Branching and release conventions can be borrowed wholesale from software engineering -- short-lived branches, small pull requests, trunk-based development -- with two ML-specific additions. Data changes need their own review path, because a change to a source table is a change to every model that reads it. And experiments do not need branches: they need tracked runs, since a branch per experiment produces fifty stale branches and no comparison.

The handoff itself deserves an explicit review, run before the transfer rather than after:

# --- A PRODUCTION-READINESS REVIEW, RUN BEFORE A HANDOFF, NOT AFTER. ---
CHECKS = {
    "ownership": [
        ("named owning team and on-call rota", True, "blocker"),
        ("second person can retrain it unaided", False, "blocker"),
        ("runbook exists and was used in a drill", False, "blocker"),
    ],
    "reproducibility": [
        ("training runs from a committed config", True, "blocker"),
        ("data snapshot and commit recorded per run", True, "blocker"),
        ("rebuild from scratch verified this quarter", False, "major"),
    ],
    "operability": [
        ("dashboards for latency, errors, action rate", True, "blocker"),
        ("alerts route to the on-call rota", True, "blocker"),
        ("rollback tested in production", False, "blocker"),
        ("cost per 1k predictions is known", False, "minor"),
    ],
    "quality": [
        ("behavioural tests in CI", True, "major"),
        ("sliced evaluation published", False, "major"),
        ("model card current for the live version", False, "major"),
    ],
    "data": [
        ("expectation suite on inputs", True, "blocker"),
        ("drift monitoring with owners", True, "major"),
        ("label delay documented", False, "minor"),
    ],
}
WEIGHT = {"blocker": 3, "major": 2, "minor": 1}

print("PRODUCTION-READINESS REVIEW: intent_v7 handover to platform-ops")
print("  area              score   open items")
total, earned, blockers = 0, 0, []
for area, items in CHECKS.items():
    w = sum(WEIGHT[sev] for _, _, sev in items)
    got = sum(WEIGHT[sev] for _, ok, sev in items if ok)
    total, earned = total + w, earned + got
    open_items = [name for name, ok, sev in items if not ok]
    blockers += [(area, name) for name, ok, sev in items
                 if not ok and sev == "blocker"]
    print(f"  {area:16s} {got:3d}/{w:<3d}   {'; '.join(open_items) or '-'}")
print(f"  {'TOTAL':16s} {earned:3d}/{total:<3d}   {earned / total:.0%} ready")
print("\nBLOCKING GAPS (handover does not proceed until these close)")
for area, name in blockers:
    print(f"  [{area}] {name}")
print("  Two of the three blockers are about PEOPLE, not code: a single")
print("  person who can retrain the model and an untested rollback are how")
print("  a working system becomes an incident three months after handover.")

# --- WHERE WORK LIVES AS IT MATURES. ---
print("\nTHE PATH FROM NOTEBOOK TO PRODUCTION, AND WHAT CHANGES AT EACH STEP")
for stage, artifact, owner, rule in [
        ("explore", "notebook, ad-hoc data", "the analyst", "no reuse expected"),
        ("formalise", "module + config + test", "the analyst", "it must run twice"),
        ("schedule", "orchestrated DAG", "the team", "it must run unattended"),
        ("serve", "service or batch job", "the team", "it must have an SLO"),
        ("operate", "same code, on-call rota", "on-call", "somebody is paged")]:
    print(f"  {stage:10s} {artifact:24s} {owner:12s} {rule}")
print("  The common failure is jumping from column one to column four. The")
print("  'formalise' step -- extracting a module, parameterising the config,")
print("  writing one test -- is what makes everything after it possible.")

PRODUCTION-READINESS REVIEW: intent_v7 handover to platform-ops
  area              score   open items
  ownership          3/9     second person can retrain it unaided; runbook exists and was used in a drill
  reproducibility    6/8     rebuild from scratch verified this quarter
  operability        6/10    rollback tested in production; cost per 1k predictions is known
  quality            2/6     sliced evaluation published; model card current for the live version
  data               5/6     label delay documented
  TOTAL             22/39    56% ready

BLOCKING GAPS (handover does not proceed until these close)
  [ownership] second person can retrain it unaided
  [ownership] runbook exists and was used in a drill
  [operability] rollback tested in production
  Two of the three blockers are about PEOPLE, not code: a single
  person who can retrain the model and an untested rollback are how
  a working system becomes an incident three months after handover.

THE PATH FROM NOTEBOOK TO PRODUCTION, AND WHAT CHANGES AT EACH STEP
  explore    notebook, ad-hoc data    the analyst  no reuse expected
  formalise  module + config + test   the analyst  it must run twice
  schedule   orchestrated DAG         the team     it must run unattended
  serve      service or batch job     the team     it must have an SLO
  operate    same code, on-call rota  on-call      somebody is paged
  The common failure is jumping from column one to column four. The
  'formalise' step -- extracting a module, parameterising the config,
  writing one test -- is what makes everything after it possible.

Notice what the blockers are. The model works; the metrics are fine; the pipeline runs. What is missing is a second person who can retrain it, a rehearsed runbook, and a tested rollback. Those three gaps are what convert a working system into an incident a quarter later, when the author is on holiday and a feature pipeline dies. Production readiness is mostly a property of the team, not of the code.

The maturity table at the end names the step that teams skip. Moving from a notebook straight to a scheduled production job means the formalisation -- extracting a module, parameterising the configuration, writing at least one test -- never happens, and the resulting pipeline is a notebook running unattended at 2 a.m.

Two more practices that reduce handoff pain:

Practice	What it prevents
Named owner per model, in the registry	"Whose model is this?" during an incident
Two people per critical pipeline	Bus factor of one
Rotating on-call including model owners	Unoperable designs, because the author is paged
Definition of done that includes docs, tests, monitoring	Work declared finished at the demo
Quarterly model review	Zombie models nobody uses and nobody deletes

The last row is the cheapest high-value habit in this chapter. Once a quarter, list every model in production and ask: is it still used, is it still correct, is it still worth its cost, and who would notice if it stopped? Models that fail all four questions are deleted, and the team gets time back.

Key Takeaways

• Code moves one way: notebooks for questions, modules for anything anyone depends on.
• Keep exploration, source, configuration, tests, pipelines, and docs separate, and strip notebook outputs in version control.
• Use tracked experiment runs rather than branches for experiments.
• Treat changes to shared data sources as changes to every model that reads them.
• Run a production-readiness review before a handoff; blockers are usually about people, not code.
• A second person who can retrain the model, a rehearsed runbook, and a tested rollback are non-negotiable.
• Do not skip formalisation: a notebook running unattended is not a pipeline.
• Name an owner per model, rotate on-call through model authors, and review every production model quarterly.

🧪 Practice

1. Audit your repository against the directory convention above and list everything that is in the wrong place.
2. Run a production-readiness review on one of your models and record the blocking gaps.
3. Have a colleague retrain one of your models from documentation alone, and fix whatever they had to ask you about.
4. Write a definition of done for model work that includes documentation, tests, monitoring, and rollback.
5. Interview: You are inheriting a model from a team that is being dissolved next week. What do you ask for, in priority order? (Hint: think about what you cannot reconstruct later versus what you can, and which single document would let you operate it on day one.)

---

<a id="16-interpretability-ethics-and-governance"></a>