9. Machine Learning Fundamentals

Machine learning is the practice of fitting a function to data so that it works on data you have not seen, and almost everything difficult about it follows from that last clause. This chapter covers the vocabulary and problem types, the generalization theory that explains why models fail on new data, the metrics that reveal whether a model is actually any good, and the procedures for choosing among models without fooling yourself. Chapter 2 supplied the optimization machinery and chapter 6 the feature engineering; this chapter is about what you are optimizing and how you know it worked.

<a id="91-core-concepts"></a>

9.1 Core Concepts

The core vocabulary of machine learning encodes a set of design decisions, and knowing what each term commits you to is what makes the rest of the field navigable; this section covers the problem types, the objects they operate on, and the assumptions every model carries.

Supervised, Unsupervised, Reinforcement Learning

Machine learning divides into families by what kind of feedback the learner receives, and that single distinction determines what problems each family can solve, what data it needs, and how you evaluate it.

Supervised learning learns from labelled examples. You supply inputs paired with correct outputs, and the model learns a mapping from one to the other. The feedback is direct and complete: for every training example, you know the right answer. This is the best-understood and most widely deployed family, and if your problem can be framed as supervised learning with labels you can actually obtain, it usually should be.

It splits by the type of output. Classification predicts a category, such as spam or not spam, and regression predicts a continuous quantity, such as a price. The distinction matters because it changes the loss function, the metrics, and often the algorithm.

Unsupervised learning learns from inputs with no labels at all. There is no right answer to compare against, so the model finds structure instead: clusters of similar records, a lower-dimensional representation, association patterns, or anomalies. The absence of labels has a consequence people underestimate: evaluation is genuinely hard, because there is no ground truth to score against. A clustering can be measured for internal cohesion, but whether the clusters mean anything requires domain judgement.

Reinforcement learning learns from consequences. An agent takes actions in an environment and receives rewards, and must discover which actions lead to reward over time. The feedback is neither complete nor immediate: you learn that a sequence of moves led to a loss without being told which move was the mistake. This is the credit assignment problem, and it is why reinforcement learning is harder and more sample-hungry than supervised learning.

The analogy that holds up: supervised learning is studying with an answer key. Unsupervised learning is sorting a box of unlabelled photographs into piles that seem to belong together. Reinforcement learning is learning to cook by tasting the result, with no recipe and no indication of which step went wrong.

Several families sit between these and are worth knowing because they solve the labelling problem that dominates real projects.

Semi-supervised learning uses a small labelled set plus a large unlabelled one, which matches the common situation where data is abundant and labels are expensive. Self-supervised learning manufactures labels from the data's own structure, such as predicting a masked word from its context, and is the foundation of modern language and vision models. Active learning chooses which examples to label next, targeting the ones the model is most uncertain about, which can cut labelling cost dramatically.

import numpy as np
import pandas as pd

rng = np.random.default_rng(0)

# --- SUPERVISED: inputs paired with known outputs. ---
n = 600
X = pd.DataFrame({"sqft": rng.integers(600, 3200, n).astype(float),
                  "bedrooms": rng.integers(1, 6, n)})
price = 50_000 + 145 * X["sqft"] + 8_000 * X["bedrooms"] + rng.normal(0, 25_000, n)
is_luxury = (price > price.quantile(0.80)).astype(int)

from sklearn.linear_model import LinearRegression, LogisticRegression
# REGRESSION: the target is a continuous quantity.
reg = LinearRegression().fit(X, price)
print(f"regression   predicts a NUMBER: ${reg.predict(X.iloc[[0]])[0]:,.0f}")
# CLASSIFICATION: the target is a category.
clf = LogisticRegression(max_iter=1000).fit(X, is_luxury)
print(f"classification predicts a CLASS: {clf.predict(X.iloc[[0]])[0]}"
      f"  with P(luxury) = {clf.predict_proba(X.iloc[[0]])[0,1]:.3f}")
print("Same features, same rows. The TARGET TYPE changed everything downstream.")

# --- UNSUPERVISED: no target column exists at all. ---
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import silhouette_score, adjusted_rand_score

Xs = StandardScaler().fit_transform(X)
km = KMeans(n_clusters=3, n_init=10, random_state=0).fit(Xs)
print(f"\nclustering found {len(np.unique(km.labels_))} groups"
      f" of sizes {np.bincount(km.labels_)}")
# There is no accuracy to report. Only internal structure can be measured.
print(f"  silhouette (internal cohesion, -1 to 1): "
      f"{silhouette_score(Xs, km.labels_):.3f}")
print(f"  agreement with the luxury label: "
      f"{adjusted_rand_score(is_luxury, km.labels_):.3f}")
print("  The second number is only computable because we happen to HAVE a")
print("  label. In a real unsupervised problem you do not, which is why")
print("  evaluating clusters is a judgement call, not a metric.")

# --- REINFORCEMENT LEARNING: delayed, partial feedback. ---
# A two-state task: the reward arrives only at the END of a sequence, so the
# learner must work out which earlier action deserves the credit.
ACTIONS = ["left", "right"]
def episode_reward(seq):
    """Reward depends on the WHOLE sequence, not on any single action."""
    return 1.0 if seq == ["right", "right", "left"] else 0.0

# Monte Carlo credit assignment: distribute the final reward over the actions
# that produced it, and let repetition sort out which ones mattered.
value = {(t, a): 0.0 for t in range(3) for a in ACTIONS}
counts = {(t, a): 0 for t in range(3) for a in ACTIONS}
for ep in range(4000):
    eps = max(0.05, 1.0 - ep / 2000)              # explore, then exploit
    seq = []
    for t in range(3):
        if rng.random() < eps:
            seq.append(ACTIONS[rng.integers(2)])
        else:
            seq.append(max(ACTIONS, key=lambda a: value[(t, a)]))
    r = episode_reward(seq)
    for t, a in enumerate(seq):                   # the same r credited to all
        counts[(t, a)] += 1
        value[(t, a)] += (r - value[(t, a)]) / counts[(t, a)]

learned = [max(ACTIONS, key=lambda a: value[(t, a)]) for t in range(3)]
print(f"\nlearned policy: {learned}   (optimal: ['right', 'right', 'left'])")
print("Nothing told the agent which step was wrong. It inferred that from")
print("thousands of episodes, which is why RL needs so much more data.")

# --- THE LABELLING PROBLEM, and active learning as a response. ---
from sklearn.model_selection import train_test_split
big_X = pd.DataFrame(rng.normal(size=(4000, 8)))
big_y = (big_X[0] + big_X[1] * 0.8 + rng.normal(0, 0.5, 4000) > 0).astype(int)
pool_X, test_X, pool_y, test_y = train_test_split(big_X, big_y, test_size=0.3,
                                                  random_state=0)

def learn_with_budget(strategy: str, budget: int = 120, seed: int = 0) -> float:
    r = np.random.default_rng(seed)
    labelled = list(r.choice(len(pool_X), 20, replace=False))   # a seed set
    while len(labelled) < budget:
        m = LogisticRegression(max_iter=1000).fit(
            pool_X.iloc[labelled], pool_y.iloc[labelled])
        remaining = [i for i in range(len(pool_X)) if i not in set(labelled)]
        if strategy == "random":
            pick = list(r.choice(remaining, 10, replace=False))
        else:
            # ACTIVE: label the points the model is least certain about,
            # i.e. those whose predicted probability sits nearest 0.5.
            p = m.predict_proba(pool_X.iloc[remaining])[:, 1]
            pick = [remaining[j] for j in np.argsort(np.abs(p - 0.5))[:10]]
        labelled += pick
    final = LogisticRegression(max_iter=1000).fit(
        pool_X.iloc[labelled], pool_y.iloc[labelled])
    return final.score(test_X, test_y)

for strat in ("random", "active"):
    scores = [learn_with_budget(strat, seed=s) for s in range(5)]
    print(f"\n{strat:7s} sampling, 120 labels: accuracy {np.mean(scores):.4f}")
print("Active learning reaches higher accuracy from the same label budget by")
print("spending it near the decision boundary instead of uniformly.")

# --- FRAMING THE SAME BUSINESS PROBLEM THREE WAYS ---
framings = pd.DataFrame([
    ("Which customers will churn next month?", "supervised, classification",
     "historical churn labels", "AUC, precision at k"),
    ("What are our natural customer groups?", "unsupervised, clustering",
     "no labels needed", "silhouette + domain review"),
    ("What discount should we offer each user?", "reinforcement / bandit",
     "reward signal per action", "cumulative reward, regret"),
    ("Which customers should we call, given 50 calls?",
     "supervised + ranking + a budget", "churn labels + call outcomes",
     "precision at 50, uplift"),
], columns=["question", "framing", "data required", "how you evaluate"])
print("\n", framings.to_string(index=False))
print("\nThe framing is a choice, not a property of the data. It determines")
print("what you must collect and how you will know if it worked.")

  THE THREE FAMILIES, BY FEEDBACK

  family          feedback per example        knows the right answer?
  ---------------------------------------------------------------------------
  supervised      a label for every input     YES, completely
  unsupervised    none                        NO, there is no answer
  reinforcement   a delayed, aggregate        PARTIALLY: only the outcome
                  reward                      of a whole sequence

  SUPERVISED SPLITS BY OUTPUT TYPE

    classification  -> a category   spam / not spam, which of 10 digits
    regression      -> a quantity   price, duration, temperature

  THE CREDIT ASSIGNMENT PROBLEM

    supervised            reinforcement
    input -> label        action, action, action -> one reward
      |        |            ?       ?       ?        |
      +--------+            +-------+-------+--------+
      error known           WHICH action earned it?
      per example

    That gap is why RL needs orders of magnitude more interaction.

  THE LABEL-SCARCITY FAMILIES

  family             uses                        when to reach for it
  ---------------------------------------------------------------------------
  semi-supervised    few labels + many unlabelled labels are expensive
  self-supervised    labels derived from the      you have raw text/images
                     data's own structure         at scale
  active learning    the model picks what to      you control the labelling
                     label next                   queue
  transfer learning  a model trained elsewhere    your dataset is small

  CHOOSING A FRAMING (the same business problem)

    "who will churn?"          -> supervised classification, needs labels
    "what groups exist?"       -> unsupervised, needs domain review
    "what offer for whom?"     -> bandit / RL, needs a reward signal
    "who do we call, top 50?"  -> supervised + ranking under a budget

Key Takeaways

• The families differ by the feedback available: complete labels, no labels, or a delayed aggregate reward.
• Supervised learning splits into classification and regression, and the target type determines the loss, the metrics, and often the algorithm.
• Unsupervised learning has no ground truth, so evaluation requires domain judgement rather than a single metric.
• Reinforcement learning faces the credit assignment problem, which is why it needs far more interaction than supervised learning.
• The framing is a choice you make, and it determines what data you must collect and how you will know the model worked.

🧪 Practice

1. Take one dataset and fit both a regression and a classification model on it by transforming the target, then list what changed in the evaluation.
2. Cluster a dataset that happens to have labels, and compare an internal metric against agreement with the true labels.
3. Implement random and uncertainty-based sampling under a fixed label budget and compare final accuracy.
4. Interview: A stakeholder asks you to "use machine learning to understand our customers better". What do you ask before choosing an approach? (Hint: what determines whether this is supervised at all, and what would you evaluate against?)

Features, Labels, and Targets

The vocabulary here is simple and the decisions it encodes are not. A feature is an input variable, a label or target is what you are predicting, and an example or instance is one row pairing them. The standard notation is X for the feature matrix, one row per example and one column per feature, and y for the target vector.

Beyond the definitions, four decisions determine whether a supervised problem is well-posed at all.

What is one row? This is the grain, introduced in chapter 6, and it is the first thing to pin down. One row per customer, per customer-month, per transaction, or per customer-product pair are four different problems with different row counts, different independence properties, and different interpretations. A model whose grain is unstated cannot be evaluated correctly, because you do not know what the unit of prediction is.

What exactly is the label? "Churn" is not a label until you say that it means no purchase within 60 days of the last one, measured as of a specific date. Label definition is where most supervised projects are quietly decided, because a label that is easy to compute is often not the thing you care about. If you define churn as account deletion, you will build a model of a rare administrative action rather than of customers drifting away.

When is the label known relative to the prediction? Every feature must be available at prediction time, and the label must be observable afterwards. This is the as-of discipline from chapter 6, and violating it produces leakage, which chapter 6 covers in detail. The practical version is a table with one row per feature and one column for the timestamp at which its value becomes known.

How long must you wait to observe the label? This is the label latency or maturation period, and it sets an unavoidable floor on how fast you can iterate. If churn is defined over 90 days, your freshest usable training label is 90 days old, and a model trained today has never seen the last quarter.

Two further distinctions worth having.

Hard versus soft labels. A hard label is a single class; a soft label is a distribution, such as the fraction of annotators who chose each class. Soft labels carry information about ambiguity that hard labels discard, and training on them often improves calibration.

Label noise. Real labels are wrong sometimes, because annotators disagree, definitions drift, or the recorded outcome is a proxy. Label noise caps achievable accuracy at a level below 100%, and knowing roughly where that ceiling sits stops you chasing performance the data cannot support. Measuring inter-annotator agreement is how you estimate it.

import numpy as np
import pandas as pd

rng = np.random.default_rng(3)

# --- THE SAME RAW DATA, FOUR GRAINS, FOUR PROBLEMS ---
events = pd.DataFrame({
    "customer": rng.integers(1, 200, 5000),
    "product": rng.choice(list("ABCD"), 5000),
    "month": rng.choice(pd.period_range("2024-01", periods=8, freq="M"), 5000),
    "amount": rng.gamma(2, 40, 5000).round(2),
})
grains = {
    "one row per customer":
        events.groupby("customer").size(),
    "one row per customer-month":
        events.groupby(["customer", "month"]).size(),
    "one row per customer-product":
        events.groupby(["customer", "product"]).size(),
    "one row per transaction":
        events.groupby(events.index).size(),
}
print("THE GRAIN DECIDES THE PROBLEM")
for name, g in grains.items():
    print(f"  {name:32s} {len(g):>6,} rows")
print("\nThese are four different supervised problems with four different")
print("independence structures. Panel grains violate row independence, which")
print("matters for cross-validation (see 9.3) and for standard errors.")

# --- LABEL DEFINITION: one word, five labels, five different models. ---
AS_OF = pd.Timestamp("2024-06-01")
purchases = pd.DataFrame({
    "customer": rng.integers(1, 400, 3000),
    "ts": pd.Timestamp("2024-01-01") + pd.to_timedelta(
        rng.integers(0, 210, 3000), "D"),
})
last = purchases.groupby("customer")["ts"].max()
counts = purchases.groupby("customer").size()

definitions = {
    "no purchase in 30 days": (AS_OF - last).dt.days > 30,
    "no purchase in 60 days": (AS_OF - last).dt.days > 60,
    "no purchase in 90 days": (AS_OF - last).dt.days > 90,
    "only ever bought once": counts == 1,
    "never returned after first": (counts == 1) & ((AS_OF - last).dt.days > 60),
}
print("\n'CHURN' UNDER FIVE DEFINITIONS")
for name, mask in definitions.items():
    print(f"  {name:30s} positive rate {mask.mean():.3f}"
          f"  ({int(mask.sum()):>4} of {len(mask)})")
spread = max(m.mean() for m in definitions.values()) - \
         min(m.mean() for m in definitions.values())
print(f"\npositive rate spans {spread:.0%} across definitions. Each one trains")
print("a model of a different phenomenon. Write the definition down first.")

# --- LABEL LATENCY: how stale your freshest label must be. ---
print("\nLABEL LATENCY: the freshest usable training data")
today = pd.Timestamp("2024-09-01")
for horizon in (7, 30, 90, 365):
    newest_label = today - pd.Timedelta(days=horizon)
    print(f"  {horizon:>3}-day label horizon -> freshest complete label is"
          f" from {newest_label:%Y-%m-%d}"
          f" ({horizon} days of recent behaviour unusable)")
print("\nA 90-day label means a model trained today has never seen the last")
print("quarter. That is a hard constraint on how fast you can respond to")
print("a change in user behaviour, and no algorithm fixes it.")

# --- THE FEATURE AVAILABILITY TABLE (the leakage audit from chapter 6) ---
schema = pd.DataFrame([
    ("tenure_days", "feature", "signup date", "before prediction", "safe"),
    ("purchases_30d", "feature", "order log, trailing window",
     "before prediction", "safe"),
    ("support_tickets_7d", "feature", "ticket system", "before prediction", "safe"),
    ("avg_basket_value", "feature", "order log, trailing", "before prediction",
     "safe"),
    ("days_to_churn", "LABEL-DERIVED", "computed from the outcome",
     "AFTER the outcome", "LEAK"),
    ("cancellation_reason", "LABEL-DERIVED", "cancellation form",
     "AFTER the outcome", "LEAK"),
    ("churned_60d", "label", "order log, forward window",
     "60 days AFTER prediction", "the target"),
], columns=["column", "role", "source", "known when", "verdict"])
print("\nFEATURE AVAILABILITY AUDIT")
print(schema.to_string(index=False))

# --- HARD VS SOFT LABELS: annotator disagreement carries information. ---
n_items, n_annotators = 400, 5
true_difficulty = rng.beta(2, 2, n_items)     # how ambiguous each item is
votes = np.array([[rng.random() < (0.5 + 0.45 * (2*d - 1) * (1 if i % 2 else -1))
                   for i in range(n_annotators)] for d in true_difficulty])
soft = votes.mean(axis=1)
hard = (soft > 0.5).astype(int)
print(f"\nHARD VS SOFT LABELS ({n_annotators} annotators, {n_items} items)")
print(f"  unanimous items: {np.mean((soft == 0) | (soft == 1)):.1%}")
print(f"  split 3-2 or 2-3: {np.mean(np.isin(soft, [0.4, 0.6])):.1%}")
print("  soft label distribution:")
for v in sorted(np.unique(soft)):
    print(f"    {v:.1f} -> {np.mean(soft == v):.1%} of items")
print("\nHardening a 3-2 vote to a confident class discards the information")
print("that the item was genuinely ambiguous. Soft labels usually improve")
print("calibration (see 9.3).")

# --- LABEL NOISE SETS A CEILING ON ACHIEVABLE ACCURACY ---
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

m = 6000
Xc = pd.DataFrame(rng.normal(size=(m, 6)))
clean_y = (Xc[0] + 0.7 * Xc[1] - 0.4 * Xc[2] > 0).astype(int)
print("\nLABEL NOISE CEILING")
for noise in (0.00, 0.05, 0.15, 0.30):
    flip = rng.random(m) < noise
    noisy_y = np.where(flip, 1 - clean_y, clean_y)
    Xtr, Xte, ytr, yte = train_test_split(Xc, noisy_y, test_size=0.3,
                                          random_state=0)
    model = LogisticRegression(max_iter=1000).fit(Xtr, ytr)
    # Score against the NOISY labels (what you would observe) and the CLEAN
    # ones (the truth you cannot see).
    obs = model.score(Xte, yte)
    _, _, _, yte_clean = train_test_split(Xc, clean_y, test_size=0.3,
                                          random_state=0)
    true_acc = model.score(Xte, yte_clean)
    print(f"  {noise:.0%} label noise -> measured accuracy {obs:.3f},"
          f" true accuracy {true_acc:.3f}, ceiling ~{1-noise:.2f}")
print("\nAt 15% label noise, 0.85 measured accuracy is a PERFECT model.")
print("Estimate your ceiling from inter-annotator agreement before setting")
print("a performance target nobody can reach.")

# Cohen's kappa: agreement corrected for what chance alone would produce.
def cohens_kappa(a, b):
    observed = np.mean(a == b)
    pa, pb = np.mean(a), np.mean(b)
    expected = pa * pb + (1 - pa) * (1 - pb)
    return (observed - expected) / (1 - expected)

ann1, ann2 = votes[:, 0].astype(int), votes[:, 1].astype(int)
print(f"\ninter-annotator agreement: raw {np.mean(ann1 == ann2):.3f},"
      f" Cohen's kappa {cohens_kappa(ann1, ann2):.3f}")
print("Kappa corrects for chance agreement. Below ~0.6 the label definition")
print("itself needs work before any modelling is worthwhile.")

  THE OBJECTS

    X  feature matrix    n rows (examples) x p columns (features)
    y  target vector     n entries, one per row
    one row = one EXAMPLE = one unit of prediction

      X                                    y
    +----------+----------+---------+     +--------+
    | tenure   | spend_30 | tickets |     | churn  |
    +----------+----------+---------+     +--------+
    | 412      | 88.50    | 0       |     | 0      |   <- one example
    | 37       | 12.00    | 3       |     | 1      |
    +----------+----------+---------+     +--------+

  FOUR DECISIONS THAT MAKE A PROBLEM WELL-POSED

    1. GRAIN       one row per WHAT? customer, customer-month, transaction?
    2. DEFINITION  what exactly does the label mean, in executable terms?
    3. TIMING      is every feature known BEFORE the prediction moment?
    4. LATENCY     how long until the label is observable?

  THE SAME WORD, FIVE LABELS

    "churn" positive rate under each definition:
      no purchase in 30 days      0.52
      no purchase in 60 days      0.31
      no purchase in 90 days      0.14
      only ever bought once       0.22
      never returned after first  0.09

    A 43-point spread. Each trains a model of a different phenomenon.

  LABEL LATENCY IS A HARD CONSTRAINT

    today = 2024-09-01

    horizon    freshest complete label   recent data unusable
    7 days     2024-08-25                1 week
    30 days    2024-08-02                1 month
    90 days    2024-06-03                1 QUARTER
    365 days   2023-09-02                1 YEAR

    With a 90-day label, today's model has never seen the last quarter.

  LABEL NOISE CAPS ACCURACY

    noise   measured accuracy   true accuracy   ceiling
    0%      0.938               0.938           1.00
    5%      0.893               0.936           0.95
    15%     0.806               0.933           0.85
    30%     0.667               0.931           0.70

    The model barely degrades; the MEASUREMENT does. At 15% noise, 0.85 is
    a perfect score. Estimate the ceiling from annotator agreement first.

Key Takeaways

• Pin down the grain first, because one row per customer and one row per customer-month are different problems with different independence structures.
• A label is not defined until it is executable, and the positive rate can vary enormously across defensible definitions of the same word.
• Every feature must be known before the prediction moment, and the label observable after it, which is the leakage audit from chapter 6.
• Label latency sets a hard floor on data freshness that no algorithm can remove.
• Label noise caps achievable accuracy, so estimate it from inter-annotator agreement before setting a performance target.

🧪 Practice

1. Take an event log and compute the row count under four different grains, stating what one row means in each.
2. Define "churn" three ways on the same data and report the positive rate under each.
3. Build a feature availability table for a prediction problem, marking each column safe, leaking, or the target.
4. Inject 5%, 15%, and 30% label noise and show that measured accuracy falls while the model's true accuracy barely moves.
5. Interview: Your churn model reaches 0.82 accuracy and the team wants 0.95. What do you investigate before trying new algorithms? (Hint: what would two independent annotators agree on, and what does that imply about the maximum?)

Training, Validation, Test Splits

The entire point of a model is performance on data it has not seen, and the only way to estimate that is to hold some data back. That idea is simple; the discipline around it is where projects go wrong, and the failure is always the same shape: the held-out data influenced the model, so the estimate is optimistic.

The standard three-way split assigns each part a distinct job, and the jobs are what matter.

The training set fits the model parameters. This is the only data the fitting procedure sees directly.

The validation set chooses between models and tunes hyperparameters. Every time you look at validation performance and change something, you use a little of its independence. After a hundred such decisions, validation performance is optimistic about the same way a training score is, because you have effectively fitted your choices to it.

The test set estimates final performance. It must be used once, at the end, after everything is decided. A test set consulted repeatedly is a validation set, and you no longer have an unbiased estimate of anything.

That third rule sounds excessive until you see the mechanism. Each time you pick the better of two options based on test performance, you select for whichever one got luckier on that particular sample. With enough selections, you can produce impressive test performance on pure noise, exactly as the segment-fishing demonstration in chapter 8 does.

The split proportions depend on data volume rather than convention. With a few thousand rows, 60/20/20 is reasonable and the validation estimate is noisy. With ten million rows, 98/1/1 leaves a hundred thousand rows in each held-out set, which is plenty, and gives the model far more to learn from.

The harder question is how to split, and the right answer depends on the structure of the data. This connects to the temporal and panel material in chapter 1.3.

Random splitting is correct only when rows are independent and identically distributed and the deployment setting matches the sampled one.

Temporal splitting is required whenever you will predict the future from the past. A random split lets the model learn from data that postdates its test rows, which inflates performance and does not survive deployment.

Group splitting is required whenever rows share an entity. If one patient contributes several visits, a random split puts the same patient in train and test, and the model is scored on its ability to recognise a patient it has already seen.

Stratified splitting preserves class proportions, which matters for imbalanced targets and small samples.

Real problems often need several of these at once: split by time and by entity, for instance, when you must generalise to new users in future periods.

import numpy as np
import pandas as pd
from sklearn.model_selection import (train_test_split, StratifiedShuffleSplit,
                                     GroupShuffleSplit, TimeSeriesSplit)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(7)

# --- SPLIT SIZES SCALE WITH VOLUME, not with convention. ---
print("held-out rows under different split ratios:")
for n in (2_000, 50_000, 10_000_000):
    for ratio in [(0.60, 0.20, 0.20), (0.80, 0.10, 0.10), (0.98, 0.01, 0.01)]:
        tr, va, te = (int(n*r) for r in ratio)
        # Standard error of an accuracy estimate near 0.9 on the test set.
        se = np.sqrt(0.9 * 0.1 / max(te, 1))
        print(f"  n={n:>10,}  {ratio}  train {tr:>9,}  val {va:>7,}"
              f"  test {te:>7,}  accuracy se +/-{1.96*se:.4f}")
    print()
print("With 10M rows a 1% test set gives a +/-0.002 interval, so spending")
print("20% on holdout is wasted training data.")

# --- THE TEST SET DEGRADES WITH EVERY LOOK ---
# 200 useless model variants on data with NO signal at all.
n, p = 400, 30
X = pd.DataFrame(rng.normal(size=(n, p)))
y = rng.integers(0, 2, n)                      # pure noise target
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.4, random_state=0)

best_score, best_k, trace = 0.0, None, []
for k in range(1, 201):
    r = np.random.default_rng(k)
    cols = list(r.choice(p, 4, replace=False))
    m = LogisticRegression(max_iter=500).fit(Xtr[cols], ytr)
    s = m.score(Xte[cols], yte)                # LOOKING AT TEST each time
    if s > best_score:
        best_score, best_k = s, k
    trace.append(best_score)
print(f"\nTEST-SET EROSION on a target with NO signal (true accuracy 0.50):")
for looks in (1, 5, 20, 50, 200):
    print(f"  best of {looks:>3} variants: test accuracy {trace[looks-1]:.3f}")
print("\nSelecting on test performance manufactures skill from noise. That is")
print("why the test set is used ONCE, after all decisions are final.")

# --- TEMPORAL SPLIT: a random split lets the model see its own future. ---
m = 4_000
dates = pd.date_range("2023-01-01", periods=m, freq="3h")
drift = np.linspace(0, 2.5, m)                 # the relationship CHANGES
Xt = pd.DataFrame({"x1": rng.normal(size=m), "x2": rng.normal(size=m)})
logit = Xt["x1"] * (1 + drift) - Xt["x2"] * 0.5 - drift
yt = (rng.random(m) < 1/(1+np.exp(-logit))).astype(int)
panel = Xt.assign(date=dates, y=yt)

# WRONG: random split on time-ordered data with drift.
a, b, ya, yb = train_test_split(Xt, yt, test_size=0.3, random_state=0)
random_auc = roc_auc_score(yb, LogisticRegression(max_iter=1000)
                           .fit(a, ya).predict_proba(b)[:, 1])
# RIGHT: train on the past, test on the future.
cut = int(m * 0.7)
temporal_auc = roc_auc_score(
    yt[cut:], LogisticRegression(max_iter=1000)
    .fit(Xt[:cut], yt[:cut]).predict_proba(Xt[cut:])[:, 1])
print(f"\nrandom split AUC   {random_auc:.4f}   <- optimistic")
print(f"temporal split AUC {temporal_auc:.4f}   <- what production sees")
print(f"gap {random_auc - temporal_auc:+.4f}: every random training fold")
print("contains rows that postdate its own test rows.")

# --- GROUP SPLIT: the same entity must not appear on both sides. ---
k = 3_000
group = rng.integers(0, 150, k)                 # 150 users, ~20 rows each
user_effect = rng.normal(0, 2.2, 150)[group]    # a strong per-user signal
Xg = pd.DataFrame({"x": rng.normal(size=k) * 0.4 + user_effect * 0.1})
yg = (rng.random(k) < 1/(1+np.exp(-user_effect))).astype(int)

ga, gb, yga, ygb = train_test_split(Xg, yg, test_size=0.3, random_state=0)
naive_auc = roc_auc_score(ygb, LogisticRegression(max_iter=1000)
                          .fit(ga, yga).predict_proba(gb)[:, 1])
gss = GroupShuffleSplit(n_splits=1, test_size=0.3, random_state=0)
tr_i, te_i = next(gss.split(Xg, yg, groups=group))
grouped_auc = roc_auc_score(
    yg[te_i], LogisticRegression(max_iter=1000)
    .fit(Xg.iloc[tr_i], yg[tr_i]).predict_proba(Xg.iloc[te_i])[:, 1])
overlap = len(set(group[np.setdiff1d(np.arange(k), tr_i)]) & set(group[tr_i]))
print(f"\nrandom split AUC  {naive_auc:.4f}  <- users appear on BOTH sides")
print(f"group split AUC   {grouped_auc:.4f}  <- users appear on one side only")
print(f"users shared between the grouped train and test sets: {overlap}")
print("The gap is memorised users, not learned behaviour.")

# --- STRATIFIED: preserve class balance, which matters when positives are few. ---
imb_y = (rng.random(1500) < 0.03).astype(int)
imb_X = pd.DataFrame(rng.normal(size=(1500, 3)))
print(f"\noverall positive rate {imb_y.mean():.4f} ({imb_y.sum()} positives)")
for label, strat in [("unstratified", None), ("stratified", imb_y)]:
    rates = []
    for seed in range(20):
        _, _, _, yy = train_test_split(imb_X, imb_y, test_size=0.2,
                                       random_state=seed, stratify=strat)
        rates.append(yy.mean())
    print(f"  {label:12s} test positive rate: min {min(rates):.4f},"
          f" max {max(rates):.4f}, zero-positive splits:"
          f" {sum(1 for r in rates if r == 0)}")
print("An unstratified split can hand you a test set with no positives at all,")
print("which makes several metrics undefined.")

# --- A SPLITTER THAT RESPECTS BOTH TIME AND GROUPS ---
def time_group_split(df, date_col, group_col, test_frac=0.2, val_frac=0.2):
    """Chronological split with no entity crossing a boundary: needed when
    you must generalise to NEW entities in FUTURE periods."""
    d = df.sort_values(date_col).reset_index(drop=True)
    n = len(d)
    cut_val = int(n * (1 - test_frac - val_frac))
    cut_test = int(n * (1 - test_frac))
    train_groups = set(d.loc[:cut_val, group_col])
    # Drop later rows whose group already appeared in training.
    val = d.loc[cut_val:cut_test]
    test = d.loc[cut_test:]
    return (d.loc[:cut_val],
            val[~val[group_col].isin(train_groups)],
            test[~test[group_col].isin(train_groups)])

demo = pd.DataFrame({"date": dates[:2000],
                     "user": rng.integers(0, 400, 2000),
                     "y": rng.integers(0, 2, 2000)})
tr_d, va_d, te_d = time_group_split(demo, "date", "user")
print(f"\ntime+group split: train {len(tr_d):,}  val {len(va_d):,}"
      f"  test {len(te_d):,}")
print(f"  users in both train and test: "
      f"{len(set(tr_d['user']) & set(te_d['user']))}")
print(f"  train ends {tr_d['date'].max():%Y-%m-%d},"
      f" test starts {te_d['date'].min():%Y-%m-%d}")
print("Rows are dropped to enforce both constraints, which is the cost of")
print("measuring what you actually need to measure.")

  THREE SETS, THREE JOBS

    TRAIN        fits parameters                  seen constantly
    VALIDATION   selects models, tunes            seen many times
    TEST         estimates final performance      seen ONCE, at the end

    Each look at the test set spends a little of its independence.

  TEST-SET EROSION (target is pure noise, true accuracy 0.50)

    variants tried    best test accuracy
    1                 0.519
    5                 0.563
    20                0.600
    50                0.625
    200               0.663

    None of these models learned anything. Selection on test performance
    manufactured 0.66 out of noise.

  SPLIT SIZE SCALES WITH VOLUME

    n            ratio          test rows   accuracy interval
    2,000        60/20/20       400         +/-0.029
    50,000       80/10/10       5,000       +/-0.008
    10,000,000   98/1/1         100,000     +/-0.002

    Past a certain size, a bigger holdout buys precision you do not need
    and costs training data you do.

  HOW TO SPLIT, BY DATA STRUCTURE

  structure              splitter              what breaks otherwise
  ---------------------------------------------------------------------------
  independent rows       random                nothing
  time-ordered           chronological         model learns from its future
  repeated entities      by group              model memorises entities
  imbalanced target      stratified            a fold with zero positives
  new entities, future   time AND group        both of the above
  periods

  THE TWO GAPS, MEASURED

    random split on drifting time series:  AUC 0.842
    temporal split:                        AUC 0.761   gap -0.081

    random split on panel data:            AUC 0.889
    grouped split:                         AUC 0.648   gap -0.241

    The second gap is larger because entity memorisation is a stronger
    shortcut than temporal leakage.

Key Takeaways

• Training fits parameters, validation selects among options, and the test set is used once at the very end.
• Every decision made by looking at a held-out score spends its independence, and enough selections manufacture skill from pure noise.
• Split proportions should scale with data volume, since a 1% test set on ten million rows is already precise.
• Split by time when predicting the future, by group when entities repeat, and stratify when the target is imbalanced.
• Real problems often need several constraints at once, and enforcing them costs rows, which is the price of a usable estimate.

🧪 Practice

1. Generate a target with no signal, try 200 model variants selecting on test accuracy, and report how high the best test score climbs.
2. Build a drifting time series and compare the area under the curve from a random split against a chronological one.
3. Construct panel data with a strong per-entity effect and compare random against grouped splitting.
4. Write a splitter enforcing both a chronological boundary and no entity overlap, and report how many rows it discards.
5. Interview: A colleague reports 0.94 test accuracy after trying forty model configurations, all evaluated on the test set. What is your concern and what do you ask for? (Hint: how many times was the final estimate's independence spent, and what data remains untouched?)

Loss Functions and Objectives

A model learns by minimising something, and that something is the loss function. It is the most consequential choice in the entire modelling process, because it defines what "good" means. Every property you want the model to have must be expressed in the loss, or the optimiser will happily trade it away.

The gap worth naming immediately is between the loss you optimise and the metric you care about. Many metrics are unusable as losses because they are not differentiable or are piecewise constant: accuracy does not change at all for small parameter adjustments, so it provides no gradient to follow. So you optimise a differentiable surrogate, such as log loss, and evaluate on the metric you actually want. Keeping this distinction clear explains a lot of otherwise confusing behaviour, including why a model with better log loss can have worse accuracy.

For regression, the choice is mostly about how to treat large errors.

Squared error penalises the square of the residual, so a single large miss dominates. It is differentiable everywhere, it is the maximum likelihood estimator under Gaussian noise, and its minimiser is the conditional mean. Use it when large errors are genuinely disproportionately bad and the data has no severe outliers.

Absolute error penalises linearly, so outliers have bounded influence. Its minimiser is the conditional median, which is a substantively different prediction on skewed data. Use it when you want robustness.

Huber loss is quadratic near zero and linear in the tails, combining the gradient behaviour of squared error near the optimum with the robustness of absolute error far from it.

Quantile loss is the asymmetric generalisation: weight over- and under-predictions differently and the minimiser is the corresponding quantile. This is how you produce prediction intervals and how you handle genuinely asymmetric costs, such as inventory where a stockout costs more than overstock.

Log-transformed targets or a Poisson/Tweedie loss are the right answers when the target is a count or is multiplicative, because squared error on raw counts assumes constant variance that count data does not have.

For classification, the dominant choice is log loss, also called binary cross-entropy. It is the negative log likelihood under a Bernoulli model, and its defining property is that it is a proper scoring rule: it is minimised only by reporting the true probability, so it rewards calibration rather than just correct ranking. It punishes confident mistakes severely, which is usually what you want.

Hinge loss underlies support vector machines and cares only about the margin, producing no calibrated probability. Focal loss downweights easy examples to concentrate learning on hard ones, which helps under extreme imbalance.

Two additions apply across both.

Class or sample weights encode asymmetric costs directly in the objective, as covered in chapter 6.4. If a false negative costs fifty times a false positive, put that in the loss rather than correcting for it afterwards.

Regularisation terms added to the loss express a preference for simpler models, which the next section covers.

import numpy as np
import pandas as pd

rng = np.random.default_rng(11)

# --- REGRESSION LOSSES: what each one's minimiser actually is. ---
def squared(r): return r ** 2
def absolute(r): return np.abs(r)
def huber(r, delta=1.35):
    a = np.abs(r)
    # Quadratic inside delta, linear outside: gradients stay bounded.
    return np.where(a <= delta, 0.5 * r**2, delta * (a - 0.5 * delta))
def quantile(r, q=0.9):
    # Penalise under-prediction (r > 0) by q and over-prediction by (1-q).
    return np.maximum(q * r, (q - 1) * r)

skewed = np.concatenate([rng.normal(50, 8, 900), rng.normal(160, 25, 100)])
grid = np.linspace(skewed.min(), skewed.max(), 4000)
print("WHAT EACH LOSS PREDICTS on a right-skewed target")
print(f"  data: mean {skewed.mean():.2f}, median {np.median(skewed):.2f},"
      f" p90 {np.percentile(skewed, 90):.2f}")
for name, fn, expect in [("squared error", squared, "the MEAN"),
                         ("absolute error", absolute, "the MEDIAN"),
                         ("huber", huber, "between the two"),
                         ("quantile q=0.9", lambda r: quantile(r, 0.9),
                          "the 90th PERCENTILE")]:
    # Minimise total loss over a grid of candidate constant predictions.
    best = grid[np.argmin([fn(skewed - c).sum() for c in grid])]
    print(f"  {name:16s} minimiser {best:7.2f}   ({expect})")
print("\nThe loss is not a technical detail: it decides WHICH summary of the")
print("conditional distribution your model predicts.")

# --- OUTLIER SENSITIVITY, measured. ---
clean = rng.normal(100, 10, 500)
contaminated = np.append(clean, [900.0])           # one absurd value
print("\nEFFECT OF ONE OUTLIER on the fitted constant")
for name, fn in [("squared", squared), ("absolute", absolute),
                 ("huber", huber)]:
    g = np.linspace(80, 200, 3000)
    a = g[np.argmin([fn(clean - c).sum() for c in g])]
    b = g[np.argmin([fn(contaminated - c).sum() for c in g])]
    print(f"  {name:9s} clean {a:7.2f} -> contaminated {b:7.2f}"
          f"  (shift {b-a:+.2f})")
print("Squared error chases the outlier; absolute error barely notices it.")

# --- ASYMMETRIC COSTS: quantile loss for an inventory decision. ---
demand = rng.poisson(120, 20_000)
STOCKOUT_COST, HOLDING_COST = 18.0, 3.0
# The newsvendor solution: order the q-th quantile where q = cu / (cu + co).
q_star = STOCKOUT_COST / (STOCKOUT_COST + HOLDING_COST)
print(f"\nINVENTORY: stockout ${STOCKOUT_COST}, holding ${HOLDING_COST}")
print(f"  optimal service level q* = {q_star:.3f}"
      f"  -> order the {q_star:.1%} quantile")
for label, order in [("the mean", demand.mean()),
                     ("the median", np.median(demand)),
                     ("q* quantile", np.quantile(demand, q_star))]:
    short = np.maximum(demand - order, 0)
    excess = np.maximum(order - demand, 0)
    cost = (short * STOCKOUT_COST + excess * HOLDING_COST).mean()
    print(f"  order {label:12s} = {order:6.1f}  ->  expected cost ${cost:6.2f}")
print("Predicting the mean is the wrong target when the costs are asymmetric.")

# --- CLASSIFICATION: log loss is a PROPER scoring rule. ---
def log_loss_single(p, y, eps=1e-15):
    p = np.clip(p, eps, 1 - eps)
    return -(y * np.log(p) + (1 - y) * np.log(1 - p))

TRUE_P = 0.30
outcomes = (rng.random(400_000) < TRUE_P).astype(int)
print(f"\nLOG LOSS IS MINIMISED BY THE TRUTH (true p = {TRUE_P})")
for claim in (0.10, 0.20, 0.30, 0.40, 0.60):
    ll = log_loss_single(claim, outcomes).mean()
    brier = ((claim - outcomes) ** 2).mean()
    print(f"  reporting p={claim:.2f}: log loss {ll:.5f}  Brier {brier:.5f}")
print("Both are minimised at the true probability, so both reward HONEST")
print("probabilities rather than merely correct ranking. That is what")
print("'proper scoring rule' means.")

# Accuracy is NOT proper: it is indifferent to confidence.
print("\naccuracy cannot distinguish these two forecasters:")
for label, p in [("cautious (0.51 / 0.49)", 0.51), ("reckless (0.99 / 0.01)", 0.99)]:
    preds = np.where(outcomes == 1, p, 1 - p)
    acc = np.mean((preds > 0.5) == outcomes)
    print(f"  {label:24s} accuracy {acc:.4f}"
          f"  log loss {log_loss_single(preds, outcomes).mean():.4f}")
print("Identical accuracy, very different log loss. If you need probabilities,")
print("accuracy is the wrong objective AND the wrong metric.")

# --- WHY YOU OPTIMISE A SURROGATE: accuracy has no useful gradient. ---
print("\nACCURACY IS PIECEWISE CONSTANT, so there is nothing to descend:")
scores = rng.normal(0, 1, 2000)
labels = (rng.random(2000) < 1/(1+np.exp(-scores))).astype(int)
for shift in (0.0, 1e-6, 1e-4, 1e-2):
    acc = np.mean(((scores + shift) > 0) == labels)
    ll = log_loss_single(1/(1+np.exp(-(scores + shift))), labels).mean()
    print(f"  shift {shift:.0e}: accuracy {acc:.6f}   log loss {ll:.8f}")
print("Accuracy is identical for the first three shifts; log loss moves every")
print("time. Optimise the smooth surrogate, EVALUATE the metric you want.")

# --- THE LOSS IS WHERE COSTS BELONG ---
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix

Xc, yc = make_classification(n_samples=12_000, n_features=12, n_informative=5,
                             weights=[0.94, 0.06], random_state=0)
Xtr, Xte, ytr, yte = train_test_split(Xc, yc, test_size=0.3, random_state=0,
                                      stratify=yc)
COST_FN, COST_FP = 400.0, 8.0
print(f"\nPUTTING COSTS IN THE OBJECTIVE (FN ${COST_FN:.0f}, FP ${COST_FP:.0f})")
for label, weights in [("unweighted", None),
                       ("weighted by cost ratio", {0: 1.0, 1: COST_FN/COST_FP})]:
    m = LogisticRegression(max_iter=2000, class_weight=weights).fit(Xtr, ytr)
    tn, fp, fn, tp = confusion_matrix(yte, m.predict(Xte)).ravel()
    print(f"  {label:24s} FN {fn:>4} FP {fp:>4}"
          f"  total cost ${fn*COST_FN + fp*COST_FP:>9,.0f}")
print("Encoding the cost ratio as a class weight is the same idea as choosing")
print("a cost-optimal threshold (chapter 8.4), applied at training time.")

# --- A SELECTION TABLE ---
guide = pd.DataFrame([
    ("regression, clean data", "squared error", "the conditional mean"),
    ("regression, outliers", "absolute or Huber", "the conditional median"),
    ("regression, asymmetric cost", "quantile", "a chosen quantile"),
    ("regression, counts", "Poisson / Tweedie", "a rate, variance scales"),
    ("regression, multiplicative", "squared error on log(y)", "the geometric mean"),
    ("classification, need probabilities", "log loss", "calibrated P(y=1)"),
    ("classification, ranking only", "log loss or hinge", "a usable ordering"),
    ("classification, extreme imbalance", "focal or weighted log loss",
     "focus on hard cases"),
    ("classification, asymmetric cost", "weighted log loss",
     "cost-aware boundary"),
], columns=["situation", "loss", "what it predicts"])
print("\n", guide.to_string(index=False))

  THE LOSS DECIDES WHICH SUMMARY YOU PREDICT

  right-skewed target: mean 61.2, median 51.4, p90 92.7

    loss             minimiser   what you get
    squared error    61.2        the conditional MEAN
    absolute error   51.4        the conditional MEDIAN
    Huber            53.8        between the two
    quantile q=0.9   92.7        the 90th PERCENTILE

  LOSS SHAPES

  loss
    |  squared: r^2          |  absolute: |r|        |  Huber
    |     \    /             |    \    /             |    \__/
    |      \  /              |     \  /              |     \/
    |       \/               |      \/               |  quadratic inside,
    +--------------> r       +--------------> r      +  linear outside

    one outlier shifts the fitted constant by:
      squared  +1.57      absolute  +0.04      Huber  +0.11

  PROPER SCORING RULES REWARD HONESTY

    true p = 0.30, and you report:
      0.10  log loss 0.79924      <- punished for being wrong
      0.20  log loss 0.66774
      0.30  log loss 0.61086      <- MINIMUM, at the truth
      0.40  log loss 0.63189
      0.60  log loss 0.83178

    Accuracy is NOT proper:
      cautious (0.51)  accuracy 1.000   log loss 0.673
      reckless (0.99)  accuracy 1.000   log loss 0.010
    Same accuracy. Accuracy is blind to confidence.

  WHY A SURROGATE

    shift the scores by:      accuracy    log loss
    0                         0.746000    0.57142183
    1e-6                      0.746000    0.57142164
    1e-4                      0.746000    0.57140287
    1e-2                      0.746500    0.56957741

    Accuracy is flat, so gradient descent has nowhere to go. Optimise the
    smooth surrogate; evaluate the metric you actually care about.

Key Takeaways

• The loss defines what "good" means, and any property not expressed in it will be traded away by the optimiser.
• Squared error predicts the conditional mean, absolute error the median, and quantile loss a chosen quantile, so the loss selects which summary you get.
• Log loss is a proper scoring rule minimised only by the true probability, which is why it rewards calibration while accuracy does not.
• Accuracy is piecewise constant and unusable as a training objective, so you optimise a smooth surrogate and evaluate the metric separately.
• Asymmetric costs belong in the loss as class or sample weights, which is the training-time counterpart of a cost-optimal threshold.

🧪 Practice

1. Numerically minimise squared, absolute, and quantile loss over a skewed sample and confirm each minimiser is the mean, median, and quantile.
2. Add one extreme outlier and report how far the fitted constant moves under squared versus absolute loss.
3. Verify empirically that log loss is minimised at the true probability by scoring several fixed claims against simulated outcomes.
4. Show that accuracy is unchanged by a tiny shift in scores while log loss moves, and explain why that matters for training.
5. Solve an inventory problem where a stockout costs six times overstock, and show the quantile prediction beats the mean.
6. Interview: Your regression model's predictions look systematically low on high-value customers, and you are using squared error on a skewed target. What might be happening? (Hint: which summary of the conditional distribution is squared error actually fitting, and what does a long right tail do to it?)

Parametric vs Non-Parametric Models

Models divide by whether the number of things they learn is fixed in advance, and this distinction predicts more about a model's behaviour than the algorithm's name does.

A parametric model commits to a functional form with a fixed number of parameters. Linear regression with ten features learns eleven numbers, regardless of whether you train it on a thousand rows or a billion. Once fitted, the training data can be discarded: those eleven numbers are the entire model.

A non-parametric model does not fix its complexity in advance, and effectively lets the data determine it. K-nearest-neighbours stores every training point and has no parameters in the usual sense. A decision tree grows until a stopping criterion fires, so the number of splits depends on the data. Kernel density estimation and Gaussian processes scale their complexity with the sample.

The name is misleading, which is worth flagging: non-parametric does not mean "no parameters", it means the number is not fixed before seeing the data.

The consequences are systematic, and knowing them lets you predict how a model will behave before you fit it.

Data efficiency. Parametric models do well on small datasets, because the assumed form substitutes for data. Non-parametric models need more data and eventually surpass parametric ones, because they are not limited by an assumption that is only approximately true.

Bias and variance. A parametric model with the wrong form has irreducible bias no amount of data removes. A non-parametric model has low bias and high variance, which is the trade-off the next section covers in detail.

Prediction cost. A parametric model's prediction is a fixed computation. K-nearest-neighbours must search the training set at prediction time, so cost grows with training data, which is a deployment constraint people meet late and painfully.

Interpretability. Parametric coefficients answer "how much does this feature matter" directly. Non-parametric models generally do not, and need the interpretability tooling of chapter 16.

Extrapolation. This is the sharpest practical difference. A linear model extrapolates confidently beyond the training range, sometimes usefully and sometimes absurdly. A tree cannot extrapolate at all: outside the training range it returns the value of the nearest leaf, so its prediction is flat. Neither is right in general, and knowing which you have matters enormously when inputs drift.

Semi-parametric models combine both, such as a generalised additive model that fits a flexible smooth function per feature while keeping the additive structure.

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.neighbors import KNeighborsRegressor
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error
import time

rng = np.random.default_rng(13)

# --- PARAMETER COUNT: fixed, or growing with the data? ---
print("HOW MANY THINGS DOES THE MODEL STORE?")
for n in (200, 2_000, 20_000):
    X = rng.uniform(-3, 3, (n, 1))
    y = np.sin(X.ravel()) * 3 + rng.normal(0, 0.4, n)
    lin = LinearRegression().fit(X, y)
    tree = DecisionTreeRegressor(min_samples_leaf=5, random_state=0).fit(X, y)
    knn = KNeighborsRegressor(5).fit(X, y)
    print(f"  n={n:>6,}  linear: {lin.coef_.size + 1:>2} parameters"
          f"   tree: {tree.tree_.node_count:>5} nodes"
          f"   kNN: stores all {n:>6,} points")
print("\nThe linear model's size is independent of n. The others grow with it,")
print("which is what 'non-parametric' actually means.")

# --- DATA EFFICIENCY: who wins depends on how much data you have. ---
# The true function is genuinely non-linear, so the linear form is WRONG.
def truth(x):
    return np.sin(2 * x) * 2.5 + 0.3 * x ** 2

X_big = rng.uniform(-3, 3, (20_000, 1))
y_big = truth(X_big.ravel()) + rng.normal(0, 0.5, 20_000)
X_test = np.linspace(-3, 3, 2_000).reshape(-1, 1)
y_test = truth(X_test.ravel())

print("\nTEST ERROR BY TRAINING SIZE (true function is non-linear)")
print(f"  {'n':>7}  {'linear':>8}  {'kNN(5)':>8}  {'tree':>8}  {'forest':>8}")
for n in (20, 60, 200, 1_000, 8_000):
    Xs, ys = X_big[:n], y_big[:n]
    row = f"  {n:>7,}"
    for model in (LinearRegression(),
                  KNeighborsRegressor(min(5, max(n // 4, 1))),
                  DecisionTreeRegressor(min_samples_leaf=3, random_state=0),
                  RandomForestRegressor(n_estimators=60, random_state=0)):
        m = model.fit(Xs, ys)
        row += f"  {mean_absolute_error(y_test, m.predict(X_test)):8.3f}"
    print(row)
print("\nAt n=20 the linear model's wrong-but-simple form wins. By n=1,000 the")
print("flexible models have overtaken it, and the linear model has stopped")
print("improving: its error floor is BIAS, which data cannot remove.")

# --- THE BIAS FLOOR, made explicit. ---
lin_full = LinearRegression().fit(X_big, y_big)
print(f"\nlinear model trained on all 20,000 rows: MAE"
      f" {mean_absolute_error(y_test, lin_full.predict(X_test)):.3f}")
lin_small = LinearRegression().fit(X_big[:200], y_big[:200])
print(f"linear model trained on 200 rows:        MAE"
      f" {mean_absolute_error(y_test, lin_small.predict(X_test)):.3f}")
print("100x the data barely helps. The form is the limit, not the sample.")

# --- EXTRAPOLATION: the sharpest practical difference. ---
X_in = rng.uniform(0, 10, (1500, 1))
y_in = 3 + 2.0 * X_in.ravel() + rng.normal(0, 1.5, 1500)   # truly linear
X_out = np.array([[5.0], [10.0], [15.0], [25.0], [50.0]])
true_out = 3 + 2.0 * X_out.ravel()

models = {
    "linear (parametric)": LinearRegression(),
    "tree (non-parametric)": DecisionTreeRegressor(min_samples_leaf=10,
                                                   random_state=0),
    "forest (non-parametric)": RandomForestRegressor(n_estimators=80,
                                                      random_state=0),
    "kNN (non-parametric)": KNeighborsRegressor(10),
}
print("\nEXTRAPOLATION beyond the training range (trained on x in [0, 10])")
print(f"  {'x':>6}  {'truth':>8}" + "".join(f"  {k.split()[0]:>10}" for k in models))
fitted = {k: m.fit(X_in, y_in) for k, m in models.items()}
for i, x in enumerate(X_out.ravel()):
    row = f"  {x:>6.1f}  {true_out[i]:>8.1f}"
    for k, m in fitted.items():
        row += f"  {m.predict([[x]])[0]:>10.1f}"
    print(row)
print("\nThe linear model extrapolates correctly because its form happens to be")
print("right. The tree-based models are FLAT beyond the training range: they")
print("return the nearest leaf's value forever. Neither is safe by default,")
print("and you must know which behaviour you have when inputs drift.")

# --- PREDICTION COST: a deployment constraint, not a detail. ---
print("\nPREDICTION LATENCY for 5,000 rows")
for n_train in (1_000, 20_000, 200_000):
    Xt = rng.uniform(-3, 3, (n_train, 4))
    yt = truth(Xt[:, 0]) + rng.normal(0, .5, n_train)
    Xq = rng.uniform(-3, 3, (5_000, 4))
    row = f"  trained on {n_train:>7,}: "
    for label, model in [("linear", LinearRegression()),
                         ("kNN", KNeighborsRegressor(10)),
                         ("forest", RandomForestRegressor(n_estimators=50,
                                                          random_state=0))]:
        m = model.fit(Xt, yt)
        t0 = time.perf_counter()
        m.predict(Xq)
        row += f" {label} {1000*(time.perf_counter()-t0):7.1f}ms"
    print(row)
print("kNN's cost grows with the TRAINING set because it searches it at")
print("prediction time. This is the constraint that kills kNN in production.")

# --- A COMPARISON TABLE ---
table = pd.DataFrame([
    ("linear / logistic regression", "parametric", "p + 1", "excellent",
     "confident", "cheap"),
    ("polynomial regression", "parametric", "fixed by degree", "good", "wild",
     "cheap"),
    ("naive Bayes", "parametric", "fixed by classes x features", "excellent",
     "n/a", "cheap"),
    ("neural network", "parametric", "fixed by architecture", "poor-moderate",
     "unpredictable", "moderate"),
    ("decision tree", "non-parametric", "grows with data", "moderate", "FLAT",
     "cheap"),
    ("random forest / boosting", "non-parametric", "grows with data",
     "moderate", "FLAT", "moderate"),
    ("k-nearest neighbours", "non-parametric", "stores all data", "poor",
     "flat", "EXPENSIVE"),
    ("Gaussian process", "non-parametric", "O(n^2) kernel matrix", "good",
     "reverts to prior", "very expensive"),
    ("generalised additive model", "semi-parametric", "splines per feature",
     "good", "smooth but risky", "cheap"),
], columns=["model", "family", "what it stores", "small-data behaviour",
            "extrapolation", "prediction cost"])
print("\n", table.to_string(index=False))

  THE DEFINING QUESTION

    parametric      complexity FIXED before seeing data
    non-parametric  complexity DETERMINED BY the data

    linear regression, 10 features:  11 numbers, whether n=100 or n=10^9
    decision tree:                   nodes grow with n
    k-nearest neighbours:            stores every training point

    "Non-parametric" does NOT mean "no parameters". It means the count is
    not fixed in advance.

  DATA EFFICIENCY CROSSOVER (true function is non-linear)

  test MAE
    |  kNN / tree
    |   \
    |    \___
    |        \_____
    |  linear      \________  flexible models win here
    |  ____________________  linear model's BIAS FLOOR
    |
    +------------------------------> training size
       n=20      n=200      n=8000
       linear    roughly    flexible
       wins      equal      wins

    100x more data barely helps the linear model: its error is bias, and
    bias does not shrink with n.

  EXTRAPOLATION IS THE SHARPEST DIFFERENCE

    trained on x in [0, 10], truth is y = 3 + 2x

      x     truth    linear    tree    forest    kNN
      5.0    13.0     13.0     13.1    13.0      13.0
     10.0    23.0     23.0     22.4    22.3      22.2
     15.0    33.0     33.0     22.4    22.3      22.2   <- FLAT
     25.0    53.0     53.0     22.4    22.3      22.2   <- FLAT
     50.0   103.0    103.0     22.4    22.3      22.2   <- FLAT

    Trees return the nearest leaf forever. Linear models extrapolate the
    assumed form, which is right here and can be absurd elsewhere.

  PREDICTION COST SCALES DIFFERENTLY

    trained on    linear     kNN       forest
    1,000          0.3ms      8.1ms     11.4ms
    20,000         0.3ms     42.7ms     13.9ms
    200,000        0.3ms    418.0ms     15.2ms

    kNN searches the training set at prediction time. That is a deployment
    constraint, not an implementation detail.

Key Takeaways

• Parametric models fix their complexity in advance; non-parametric models let the data determine it, and the name means the count is not preset rather than absent.
• Parametric models win on small data because the assumed form substitutes for examples, and they hit a bias floor that more data cannot lower.
• Tree-based models cannot extrapolate and return a flat value outside the training range, while linear models extrapolate their assumed form confidently.
• Nearest-neighbour prediction cost grows with the training set, which is a deployment constraint rather than a detail.
• Semi-parametric models such as generalised additive models keep an interpretable structure while fitting flexible per-feature shapes.

🧪 Practice

1. Fit a linear model, a tree, and k-nearest-neighbours at five training sizes on a non-linear function, and identify the crossover point.
2. Show that a linear model on a non-linear target stops improving as data grows, and name what its remaining error is.
3. Train models on inputs restricted to one range and predict well outside it, tabulating which models go flat.
4. Measure prediction latency for nearest-neighbours at three training sizes and explain the scaling.
5. Interview: You must deploy a model that will see input values larger than anything in training, such as a sensor reading during an unusual event. Which model family concerns you and why? (Hint: what does a tree predict beyond its largest split, and what does a linear model predict?)

Inductive Bias

A learning algorithm sees finitely many examples and must choose a function defined everywhere. Infinitely many functions fit any finite dataset exactly, so something other than the data must decide which one to prefer. That something is the inductive bias: the set of assumptions a learner uses to generalise beyond what it has seen.

This is not a flaw to be engineered away. It is a logical necessity. A learner with no preference among functions consistent with the data has no basis to predict anything about a new point, because for every function that predicts one value, another equally consistent function predicts the opposite. No assumptions means no generalisation.

The formal statement is the no free lunch theorem: averaged over all possible problems, every algorithm performs identically, and none beats random guessing. The reason machine learning works anyway is that real problems are not drawn uniformly from all possible problems. They are smooth, structured, hierarchical, and sparse in ways that particular inductive biases match. Choosing a model is choosing which assumption about the world to import, and a model works when its bias matches the problem's structure.

Every familiar algorithm carries a recognisable bias:

Model	Inductive bias
Linear regression	the target is a weighted sum of features
Polynomial regression	smooth, of a specific degree
Decision tree	the target is piecewise constant on axis-aligned boxes
k-nearest neighbours	nearby points have similar targets
Naive Bayes	features are independent given the class
Convolutional network	local patterns matter, and are translation-invariant
Recurrent network / transformer	sequence order carries meaning
Ridge regression	coefficients should be small
Lasso	most coefficients should be exactly zero

Reading that table as a set of claims about the world is the useful move. A convolutional network works on images because a cat is a cat wherever it appears in the frame, and that translation invariance is a genuine property of images and not of, say, tabular data with named columns. Applying it where it does not hold gains nothing.

Two practical implications.

Match the bias to the structure, and check it. If a tree outperforms a linear model substantially, the relationship is probably not additive. If a linear model matches a boosted ensemble, the structure is probably close to additive and you should prefer the simpler model. The comparison is diagnostic, not just competitive.

Encode known structure rather than hoping it is learned. If you know an effect must be monotonic, use a model that enforces monotonicity. If you know a physical constraint, build it in. Every constraint you impose correctly is training data you did not need, which is why domain knowledge beats data volume so often on small problems.

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.tree import DecisionTreeRegressor
from sklearn.neighbors import KNeighborsRegressor
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_absolute_error

rng = np.random.default_rng(17)

# --- INFINITELY MANY FUNCTIONS FIT THE SAME POINTS ---
x_obs = np.array([0.0, 1.0, 2.0, 3.0, 4.0])
y_obs = np.array([0.0, 1.0, 4.0, 9.0, 16.0])          # looks like x^2
candidates = {
    "x^2": lambda x: x**2,
    "x^2 + sin(pi*x)": lambda x: x**2 + np.sin(np.pi * x),
    "degree-4 interpolant": np.poly1d(np.polyfit(x_obs, y_obs, 4)),
    "x^2 * cos(2*pi*x)... ": lambda x: x**2 * np.cos(2*np.pi*x)
                                        + x**2 * (1 - np.cos(2*np.pi*x)),
}
print("EVERY ONE OF THESE FITS ALL FIVE OBSERVED POINTS EXACTLY:")
for name, f in candidates.items():
    fit_err = np.abs(f(x_obs) - y_obs).max()
    print(f"  {name:24s} max error on observed points: {fit_err:.2e}"
          f"   prediction at x=2.5: {f(2.5):8.3f}")
print("\nThe data cannot choose between them. Only an ASSUMPTION can, and")
print("that assumption is the inductive bias.")

# --- THE SAME DATA, THREE STRUCTURES: which bias matches which? ---
n = 1200
X = rng.uniform(-3, 3, (n, 4))
targets = {
    # ADDITIVE and linear: matches a linear model's bias exactly.
    "additive linear": 2*X[:,0] - 1.5*X[:,1] + 0.8*X[:,2],
    # PIECEWISE CONSTANT on axis-aligned regions: a tree's native structure.
    "step function": (10 * (X[:,0] > 0) + 6 * (X[:,1] > 1)
                      - 4 * (X[:,2] < -1)),
    # SMOOTH and non-linear: neither linear nor axis-aligned.
    "smooth non-linear": np.sin(2*X[:,0]) * 3 + np.cos(X[:,1]) * 2,
    # MULTIPLICATIVE INTERACTION: additive models cannot express it.
    "pure interaction": X[:,0] * X[:,1] * 2,
}
models = {
    "linear": LinearRegression(),
    "poly-3": make_pipeline(PolynomialFeatures(3), Ridge(alpha=1.0)),
    "tree": DecisionTreeRegressor(min_samples_leaf=10, random_state=0),
    "kNN-10": make_pipeline(StandardScaler(), KNeighborsRegressor(10)),
    "boosting": GradientBoostingRegressor(random_state=0, n_estimators=150),
}
split = int(n * 0.7)
print("\nTEST MAE: each ROW is a structure, each COLUMN a model's bias")
header = f"  {'structure':20s}" + "".join(f"{k:>10}" for k in models)
print(header)
for tname, tvals in targets.items():
    y = tvals + rng.normal(0, 0.4, n)
    row = f"  {tname:20s}"
    scores = {}
    for mname, model in models.items():
        m = model.fit(X[:split], y[:split])
        e = mean_absolute_error(tvals[split:], m.predict(X[split:]))
        scores[mname] = e
        row += f"{e:10.3f}"
    best = min(scores, key=scores.get)
    print(row + f"   best: {best}")
print("\nNo model wins everywhere. The linear model is best on additive data")
print("and hopeless on interactions; the tree is best on steps and mediocre")
print("on smooth curves. That IS the no free lunch theorem, made concrete.")

# --- READ A MODEL'S BIAS FROM ITS PREDICTIONS ---
xg = np.linspace(-3, 3, 400).reshape(-1, 1)
y1 = np.where(xg.ravel() > 0, 5.0, 1.0) + rng.normal(0, .2, 400)  # a step
shapes = {
    "linear": LinearRegression().fit(xg, y1),
    "tree": DecisionTreeRegressor(min_samples_leaf=20, random_state=0).fit(xg, y1),
    "kNN-25": KNeighborsRegressor(25).fit(xg, y1),
}
print("\nfitting a STEP function, sampled at five x values:")
print(f"  {'x':>6}" + "".join(f"{k:>10}" for k in shapes))
for x in (-2.0, -0.3, 0.0, 0.3, 2.0):
    row = f"  {x:>6.1f}"
    for m in shapes.values():
        row += f"{m.predict([[x]])[0]:10.2f}"
    print(row)
print("The linear model must slope through the discontinuity; the tree")
print("reproduces it exactly; kNN smooths it over its neighbourhood width.")
print("Each is showing you its bias.")

# --- SPARSITY BIAS: L1 vs L2 make different claims about the world. ---
p = 60
Xs = rng.normal(size=(400, p))
true_coef = np.zeros(p)
true_coef[:4] = [3.0, -2.5, 2.0, 1.5]             # only 4 of 60 matter
ys = Xs @ true_coef + rng.normal(0, 1.0, 400)
print(f"\nTRUE MODEL: {int((true_coef != 0).sum())} of {p} coefficients are non-zero")
for label, model in [("ridge (L2): all small", Ridge(alpha=10.0)),
                     ("lasso (L1): most zero", Lasso(alpha=0.15, max_iter=5000))]:
    m = model.fit(Xs, ys)
    nz = int((np.abs(m.coef_) > 1e-6).sum())
    # Did it recover the right SUPPORT, not just a good fit?
    found = set(np.argsort(-np.abs(m.coef_))[:4])
    print(f"  {label:24s} non-zero {nz:>3}"
          f"  top-4 match truth: {found == {0,1,2,3}}")
print("Lasso's bias -- 'most features are irrelevant' -- is TRUE here, so it")
print("recovers the support. Ridge's bias -- 'all effects are small' -- is")
print("false here, and it spreads weight across all 60.")

# --- ENCODING KNOWN STRUCTURE beats hoping it is learned. ---
from sklearn.ensemble import HistGradientBoostingRegressor
k = 900
price = rng.uniform(5, 60, k)
# Domain fact: demand is MONOTONICALLY decreasing in price.
demand = 200 - 2.4 * price + rng.normal(0, 14, k)
Xp, yp = price.reshape(-1, 1), demand
tr = int(k * 0.7)
grid = np.linspace(5, 60, 60).reshape(-1, 1)

for label, model in [
        ("unconstrained", HistGradientBoostingRegressor(random_state=0,
                                                        max_iter=200)),
        ("monotonic constraint", HistGradientBoostingRegressor(
            random_state=0, max_iter=200, monotonic_cst=[-1]))]:
    m = model.fit(Xp[:tr], yp[:tr])
    pred = m.predict(grid)
    violations = int((np.diff(pred) > 0).sum())
    mae = mean_absolute_error(200 - 2.4*grid.ravel(), pred)
    print(f"\n{label:22s} monotonicity violations: {violations:>3}/{len(pred)-1}"
          f"   MAE vs truth {mae:.2f}")
print("\nThe constraint encodes a fact we KNOW. It removes nonsense predictions")
print("(demand rising with price) and improves accuracy, because the")
print("constraint is information the model did not have to learn from data.")

  WHY INDUCTIVE BIAS IS NECESSARY

  five observed points:  (0,0) (1,1) (2,4) (3,9) (4,16)

  all of these fit EXACTLY:
    x^2                      -> predicts 6.25 at x=2.5
    x^2 + sin(pi*x)          -> predicts 6.25 at x=2.5
    degree-4 interpolant     -> predicts 6.25 at x=2.5
    x^2 + 1000*sin(pi*x)     -> predicts 6.25 at x=2.5
                                 ...and wildly different values elsewhere

  The data is silent between the points. An assumption must speak.
  NO ASSUMPTIONS -> NO GENERALIZATION.

  NO FREE LUNCH, MADE CONCRETE (test MAE)

  structure             linear   poly-3    tree    kNN   boosting
  additive linear        0.32     0.34     1.42   0.95     0.47
  step function          3.18     2.41     0.29   0.88     0.31
  smooth non-linear      2.05     0.58     0.71   0.52     0.44
  pure interaction       1.63     0.21     0.66   0.74     0.28

  Every column wins somewhere and loses somewhere. A model is a claim
  about structure, and it works when the claim is true.

  EACH MODEL'S CLAIM ABOUT THE WORLD

    linear        "the target is a weighted SUM of features"
    polynomial    "the target is SMOOTH, of this degree"
    tree          "the target is CONSTANT on axis-aligned boxes"
    kNN           "NEARBY points have similar targets"
    naive Bayes   "features are INDEPENDENT given the class"
    CNN           "LOCAL patterns matter, wherever they appear"
    RNN/transformer "ORDER carries meaning"
    ridge         "all coefficients are SMALL"
    lasso         "most coefficients are exactly ZERO"

  BIAS IS VISIBLE IN THE PREDICTIONS (fitting a step at x=0)

       x      linear    tree    kNN-25
     -2.0       1.68     1.00     1.00
     -0.3       2.86     1.00     1.42
      0.0       3.01     1.00     3.00     <- kNN smooths the jump
      0.3       3.15     5.00     4.58
      2.0       4.33     5.00     5.00

    linear must slope through; the tree reproduces the step; kNN blurs it.

  THE DIAGNOSTIC USE

    tree >> linear         -> the relationship is not additive
    linear ~= boosting     -> structure is close to additive; ship the
                              simpler model
    lasso recovers support -> the problem really is sparse

Key Takeaways

• Infinitely many functions fit any finite dataset, so an assumption must choose among them, and that assumption is the inductive bias.
• The no free lunch theorem says no algorithm wins on average over all problems; models work because real problems have structure some biases match.
• Every algorithm's bias is a readable claim about the world, and choosing a model means choosing which claim to import.
• Comparing model families is diagnostic: a tree beating a linear model says the relationship is not additive.
• Encoding known structure such as monotonicity substitutes for training data and removes nonsensical predictions.

🧪 Practice

1. Construct three different functions that fit the same five points exactly and predict different values between them.
2. Build four datasets with different structures and fit five model families to each, identifying which bias wins where.
3. Fit a linear model, a tree, and nearest-neighbours to a step function and describe how each one's predictions reveal its bias.
4. Compare ridge and lasso on a genuinely sparse problem and check which recovers the correct set of non-zero coefficients.
5. Fit a gradient boosting model with and without a monotonic constraint on a relationship you know is monotonic, and compare both accuracy and constraint violations.
6. Interview: A colleague says they will avoid making assumptions by using a very flexible model. What is wrong with that reasoning? (Hint: what does a model with no preferences predict at a point it has never seen?)

<a id="92-generalization"></a>

9.2 Generalization

Generalization is the whole problem: a model that fits its training data perfectly and fails on new data has learned nothing useful; this section covers why that happens, how to diagnose which failure you have, and the techniques that trade a little fit for a lot of reliability.

Overfitting and Underfitting

A model can fail on new data in two opposite ways, and they demand opposite remedies, so distinguishing them is the first diagnostic step in any modelling problem.

Underfitting means the model is too simple to capture the real pattern. It performs badly on training data and on test data, and the two scores are close together. A straight line fitted to a curve underfits: it is wrong everywhere, and more data will not help because the limitation is the form, not the sample.

Overfitting means the model has captured patterns specific to the training sample that do not exist in the population. It performs excellently on training data and poorly on test data, and the gap between them is large. A decision tree grown until every leaf holds one example overfits: it has memorised the training set, including its noise.

The distinguishing signal is therefore not a single score but the relationship between two scores. Both bad and close together means underfitting. Training good and test much worse means overfitting. Reading only the test score tells you that something is wrong without telling you what.

The analogy that holds up: underfitting is a student who learned one rule and applies it everywhere. Overfitting is a student who memorised the answers to last year's exam paper, including the typos, and is helpless when the questions change.

Memorisation of noise is the precise mechanism of overfitting, and it is worth being concrete about. Real data contains signal, which repeats in new samples, and noise, which does not. A sufficiently flexible model cannot tell them apart and fits both. Every unit of noise it fits is a unit of error on new data, because that particular noise will not recur.

The controls that move a model along the simple-to-complex axis are collectively model capacity: polynomial degree, tree depth, number of neighbours, network size, regularisation strength, and training duration. Each one has a setting that underfits and a setting that overfits, and the job is to find the middle.

One important modern caveat. The classical picture says test error falls, reaches a minimum, then rises as capacity grows. For very over-parameterised models, particularly large neural networks, test error can fall again past the interpolation point, a phenomenon called double descent. The classical intuition remains the right default for tabular models and small data, but "more parameters always overfits" is not a law.

import numpy as np
import pandas as pd
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

rng = np.random.default_rng(0)

# --- ONE DATASET, CAPACITY SWEPT FROM TOO SIMPLE TO TOO COMPLEX ---
def truth(x):
    return np.sin(1.6 * x) * 2.0 + 0.25 * x

n = 90
x = rng.uniform(-4, 4, n)
y = truth(x) + rng.normal(0, 0.55, n)                # 0.55 is irreducible noise
xtr, xte, ytr, yte = train_test_split(x, y, test_size=0.4, random_state=0)
Xtr, Xte = xtr.reshape(-1, 1), xte.reshape(-1, 1)

print("POLYNOMIAL DEGREE: the classic capacity dial")
print(f"  {'degree':>7}  {'train MSE':>10}  {'test MSE':>10}  {'gap':>8}  verdict")
rows = []
for deg in (1, 2, 3, 5, 9, 15, 25):
    m = make_pipeline(PolynomialFeatures(deg), LinearRegression()).fit(Xtr, ytr)
    tr_mse = mean_squared_error(ytr, m.predict(Xtr))
    te_mse = mean_squared_error(yte, m.predict(Xte))
    gap = te_mse - tr_mse
    verdict = ("UNDERFIT: both bad" if tr_mse > 0.8
               else "OVERFIT: gap is large" if gap > 0.6
               else "reasonable")
    rows.append({"degree": deg, "train": tr_mse, "test": te_mse, "gap": gap})
    print(f"  {deg:>7}  {tr_mse:10.4f}  {te_mse:10.4f}  {gap:8.4f}  {verdict}")
best = min(rows, key=lambda r: r["test"])
print(f"\nbest test MSE at degree {best['degree']}"
      f" (irreducible noise variance is {0.55**2:.4f})")
print("Degree 1 is bad on BOTH sets: too simple. Degree 25 is near-perfect on")
print("train and terrible on test: it memorised the noise.")

# --- THE SIGNATURE: it is the GAP that identifies which failure you have. ---
print("\nTHE DIAGNOSTIC TABLE")
diag = pd.DataFrame([
    ("high", "high", "small", "UNDERFITTING", "more capacity, better features"),
    ("low", "high", "LARGE", "OVERFITTING", "less capacity, more data, regularise"),
    ("low", "low", "small", "good fit", "ship it"),
    ("high", "low", "negative", "a BUG or a leak", "check the split and the code"),
], columns=["train error", "test error", "gap", "diagnosis", "what to do"])
print(diag.to_string(index=False))
print("\nThe last row should be impossible. If test beats train substantially,")
print("suspect a split bug, a leak, or different preprocessing per split.")

# --- MEMORISING NOISE, demonstrated directly. ---
# Fit a high-capacity model to data with NO signal whatsoever.
pure_noise_x = rng.uniform(0, 1, (120, 1))
pure_noise_y = rng.normal(0, 1, 120)                 # unrelated to x
nx_tr, nx_te, ny_tr, ny_te = train_test_split(pure_noise_x, pure_noise_y,
                                              test_size=0.4, random_state=0)
print("\nFITTING DATA WITH NO SIGNAL AT ALL (true R-squared is 0)")
for leaf in (30, 10, 3, 1):
    t = DecisionTreeRegressor(min_samples_leaf=leaf, random_state=0).fit(nx_tr, ny_tr)
    print(f"  min_samples_leaf={leaf:>2}: train R2 {t.score(nx_tr, ny_tr):+.3f}"
          f"   test R2 {t.score(nx_te, ny_te):+.3f}"
          f"   leaves {t.get_n_leaves():>3}")
print("At leaf=1 the model explains 100% of the training variance and has")
print("NEGATIVE test R-squared, meaning it is worse than predicting the mean.")
print("Every bit of that training fit was noise.")

# --- AN ASCII VIEW of the three regimes ---
xg = np.linspace(-4, 4, 60)
fits = {}
for deg, label in [(1, "underfit (deg 1) "), (5, "good fit (deg 5) "),
                   (25, "overfit (deg 25) ")]:
    m = make_pipeline(PolynomialFeatures(deg), LinearRegression()).fit(Xtr, ytr)
    fits[label] = m.predict(xg.reshape(-1, 1))

print("\nfitted curves vs the truth (each column is an x value):")
lo, hi = -5.0, 5.0
def render(vals):
    out = ""
    for v in vals:
        level = int(np.clip((v - lo) / (hi - lo) * 8, 0, 8))
        out += "12345678#"[level]
    return out
print(f"  {'truth':18s}|{render(truth(xg))}|")
for label, vals in fits.items():
    print(f"  {label:18s}|{render(vals)}|")
print("  (digits encode height; compare each fitted row against 'truth')")
for label, vals in fits.items():
    print(f"  {label:18s} mean |error| vs truth: "
          f"{np.abs(vals - truth(xg)).mean():.3f}")

# --- MORE DATA FIXES OVERFITTING AND NOT UNDERFITTING ---
print("\nDOES MORE DATA HELP? (test MSE)")
print(f"  {'n':>6}  {'degree 1 (too simple)':>22}  {'degree 20 (too complex)':>24}")
for n_train in (30, 100, 400, 3000):
    xs = rng.uniform(-4, 4, n_train)
    ys = truth(xs) + rng.normal(0, 0.55, n_train)
    xv = rng.uniform(-4, 4, 3000)
    yv = truth(xv) + rng.normal(0, 0.55, 3000)
    simple = make_pipeline(PolynomialFeatures(1), LinearRegression()).fit(
        xs.reshape(-1, 1), ys)
    complex_ = make_pipeline(PolynomialFeatures(20), LinearRegression()).fit(
        xs.reshape(-1, 1), ys)
    xv2 = xv.reshape(-1, 1)
    print(f"  {n_train:>6,}  {mean_squared_error(yv, simple.predict(xv2)):>22.4f}"
          f"  {mean_squared_error(yv, complex_.predict(xv2)):>24.4f}")
print("The over-complex model improves steadily with data; the too-simple one")
print("plateaus immediately. This is the single most useful thing to know:")
print("MORE DATA CURES OVERFITTING AND DOES NOTHING FOR UNDERFITTING.")

# --- CAPACITY DIALS BY MODEL FAMILY ---
dials = pd.DataFrame([
    ("polynomial regression", "degree", "raise", "lower"),
    ("decision tree", "max_depth / min_samples_leaf", "deepen", "prune"),
    ("random forest", "max_features, min_samples_leaf", "raise features",
     "raise leaf size"),
    ("gradient boosting", "n_estimators, learning_rate, depth", "more trees",
     "fewer trees, lower rate"),
    ("k-nearest neighbours", "k", "lower k", "raise k"),
    ("linear / logistic", "regularisation strength", "weaken penalty",
     "strengthen penalty"),
    ("SVM (RBF)", "C and gamma", "raise both", "lower both"),
    ("neural network", "width, depth, epochs, dropout", "bigger, longer",
     "smaller, earlier stop, more dropout"),
], columns=["model", "capacity control", "to fix UNDERfitting",
            "to fix OVERfitting"])
print("\n", dials.to_string(index=False))

  THE TWO FAILURES ARE OPPOSITE

  UNDERFIT (degree 1)      GOOD (degree 5)        OVERFIT (degree 25)
    .    .                   .    .                 .    .
  ___________              _/‾\__/‾\_             _/\_/\/\_/\_
    .  .    .                .  .    .              .  .    .
  wrong everywhere         follows the signal     follows every point,
                                                  including the noise

  train MSE  1.24            0.28                  0.01
  test MSE   1.31            0.36                  2.87
  GAP        0.07            0.08                  2.86

  THE DIAGNOSTIC IS THE GAP, NOT THE SCORE

  train error   test error   gap        diagnosis        fix
  ---------------------------------------------------------------------------
  high          high         small      UNDERFITTING     more capacity
  low           high         LARGE      OVERFITTING      less capacity, data
  low           low          small      good             ship it
  high          low          negative   BUG or leak      check the split

  MEMORISING NOISE (data with ZERO real signal)

    min_samples_leaf    train R2    test R2   leaves
    30                    +0.08      -0.04      3
    10                    +0.41      -0.19     10
    3                     +0.83      -0.42     27
    1                     +1.00      -0.67     72

    At leaf=1 training R-squared is a PERFECT 1.00 and test R-squared is
    negative: worse than predicting the mean. All of that fit was noise.

  MORE DATA CURES ONE FAILURE, NOT THE OTHER

  test MSE
    |
    |  \                  degree 20 (too complex): improves with data
    |   \____
    |        \_____
    |  ____________________ degree 1 (too simple): FLAT, data cannot help
    |
    +-------------------------> training size
     30      100     400    3000

    Overfitting is a data problem. Underfitting is a model problem.

Key Takeaways

• Underfitting shows high error on both training and test data with a small gap; overfitting shows low training error and a large gap.
• The diagnosis comes from the relationship between two scores, so a test score alone tells you something is wrong without telling you what.
• Overfitting is literally the memorisation of noise, and a flexible enough model can reach perfect training fit on data with no signal.
• More data cures overfitting and does nothing for underfitting, which is the fastest way to decide whether to collect more.
• Every model family has explicit capacity dials, and each has a setting that underfits and one that overfits.

🧪 Practice

1. Sweep polynomial degree from 1 to 25 on a noisy non-linear target and tabulate training error, test error, and the gap.
2. Fit a decision tree with a minimum leaf size of one to data with no signal, and report both training and test R-squared.
3. Show that increasing training data improves an over-complex model's test error while leaving a too-simple model's unchanged.
4. For three model families, identify the capacity parameter and demonstrate one setting that underfits and one that overfits.
5. Interview: A model scores 0.99 on training data and 0.62 on test data. A colleague proposes trying a more powerful algorithm. What do you say? (Hint: which of the two failures does that gap indicate, and what does more capacity do to it?)

Bias-Variance Tradeoff

The bias-variance decomposition explains why the two failure modes exist, and it turns the vague advice "find the right complexity" into a statement about two quantities that move in opposite directions.

The setup is a thought experiment. Imagine drawing many different training sets from the same population, fitting your model on each, and predicting the same test point with all of them. You now have a distribution of predictions for that one point. Expected squared error decomposes exactly into three pieces:

Bias is how far the average prediction is from the truth. It measures systematic error from assumptions that are wrong. A linear model fitted to a curve has high bias: however much data you give it, its average prediction is off in the same direction.

Variance is how much predictions scatter across different training sets. It measures sensitivity to the particular sample. A deep tree has high variance: retrain it on a different draw and it produces a substantially different function.

Irreducible error is the noise in the target itself. No model removes it, and it is the floor on achievable error. Knowing roughly where that floor sits stops you chasing performance the data cannot support.

The identity is exact for squared loss: expected error equals bias squared plus variance plus noise. Its practical value is that capacity moves bias and variance in opposite directions. Adding complexity reduces bias and increases variance; removing it does the reverse. Total error is a U-shape, and the minimum is wherever the two curves cross in the right proportion.

This mapping is the payoff:

• Underfitting is high bias. The model cannot represent the pattern.
• Overfitting is high variance. The model is chasing the sample.

The connection to earlier material: a model's inductive bias determines where on this spectrum it starts, and regularisation is a deliberate increase in bias bought to reduce variance, which is worthwhile whenever the variance reduction is larger.

Two mechanisms for reducing variance without paying much bias are worth naming because they are why ensembles dominate tabular machine learning. Averaging many high-variance models reduces variance roughly in proportion to the number averaged, provided their errors are not perfectly correlated, which is exactly what bagging and random forests do. Boosting works the other way, reducing bias by adding many high-bias models in sequence, which is why its capacity control is about stopping in time.

Two caveats. The clean decomposition holds for squared loss and is messier for other losses. And the classical U-shaped curve is not the whole story for very over-parameterised models, as the double descent phenomenon shows.

import numpy as np
import pandas as pd
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor, BaggingRegressor

rng = np.random.default_rng(5)

# --- THE DECOMPOSITION, computed by simulation. ---
def truth(x):
    return np.sin(1.5 * x) * 2.0

NOISE_SD = 0.6
X_TEST = np.linspace(-3, 3, 120).reshape(-1, 1)
Y_TRUE = truth(X_TEST.ravel())

def decompose(make_model, n_train=60, n_datasets=300, seed=0):
    """Fit the same model on many independent training sets and split the
    expected squared error into bias^2 + variance + noise."""
    r = np.random.default_rng(seed)
    preds = np.empty((n_datasets, len(X_TEST)))
    for i in range(n_datasets):
        xs = r.uniform(-3, 3, n_train)
        ys = truth(xs) + r.normal(0, NOISE_SD, n_train)
        preds[i] = make_model().fit(xs.reshape(-1, 1), ys).predict(X_TEST)
    mean_pred = preds.mean(axis=0)
    bias_sq = np.mean((mean_pred - Y_TRUE) ** 2)      # systematic error
    variance = np.mean(preds.var(axis=0))             # scatter across datasets
    noise = NOISE_SD ** 2                             # irreducible
    return {"bias^2": bias_sq, "variance": variance, "noise": noise,
            "total": bias_sq + variance + noise}

print("BIAS-VARIANCE DECOMPOSITION BY POLYNOMIAL DEGREE")
print(f"  {'degree':>7} {'bias^2':>9} {'variance':>9} {'noise':>7} {'total':>9}")
rows = []
for deg in (1, 2, 3, 4, 6, 9, 13):
    d = decompose(lambda deg=deg: make_pipeline(PolynomialFeatures(deg),
                                                LinearRegression()), seed=deg)
    rows.append({"degree": deg, **d})
    print(f"  {deg:>7} {d['bias^2']:9.4f} {d['variance']:9.4f}"
          f" {d['noise']:7.4f} {d['total']:9.4f}")
df = pd.DataFrame(rows)
best = df.loc[df["total"].idxmin()]
print(f"\nminimum total error at degree {int(best['degree'])}")
print(f"bias falls monotonically: {df['bias^2'].round(3).tolist()}")
print(f"variance rises monotonically: {df['variance'].round(3).tolist()}")
print("They move in OPPOSITE directions, and the sum is a U-shape.")

# --- THE SAME TRADE-OFF ON A DIFFERENT CAPACITY DIAL ---
print("\nSAME DECOMPOSITION, TREE DEPTH AS THE DIAL")
print(f"  {'max_depth':>10} {'bias^2':>9} {'variance':>9} {'total':>9}")
for depth in (1, 2, 3, 5, 8, None):
    d = decompose(lambda dd=depth: DecisionTreeRegressor(max_depth=dd,
                                                         random_state=0), seed=7)
    label = "none" if depth is None else str(depth)
    print(f"  {label:>10} {d['bias^2']:9.4f} {d['variance']:9.4f} {d['total']:9.4f}")
print("An unrestricted tree has near-zero bias and the highest variance.")

# --- WHY AVERAGING WORKS: variance falls, bias does not. ---
print("\nAVERAGING HIGH-VARIANCE MODELS (unrestricted trees)")
print(f"  {'n_estimators':>13} {'bias^2':>9} {'variance':>9} {'total':>9}")
for n_est in (1, 3, 10, 50, 200):
    d = decompose(
        lambda k=n_est: BaggingRegressor(
            DecisionTreeRegressor(random_state=0), n_estimators=k,
            random_state=0),
        n_datasets=120, seed=11)
    print(f"  {n_est:>13} {d['bias^2']:9.4f} {d['variance']:9.4f} {d['total']:9.4f}")
print("\nBias is roughly flat and variance collapses. That is the entire")
print("argument for bagging and random forests: buy variance reduction")
print("without paying bias.")

# Variance reduction depends on the models being DECORRELATED.
print("\nwhy decorrelation matters: variance of an average of k models")
print("  = (rho + (1-rho)/k) * single-model variance")
single_var = 1.0
for rho in (0.0, 0.3, 0.7, 1.0):
    row = f"  correlation {rho:.1f}: "
    for k in (1, 10, 100):
        v = (rho + (1 - rho) / k) * single_var
        row += f" k={k:>3}: {v:.3f}"
    print(row)
print("At rho=1 averaging does nothing. Random forests inject randomness")
print("precisely to push rho down.")

# --- BOOSTING ATTACKS BIAS INSTEAD ---
from sklearn.ensemble import GradientBoostingRegressor
print("\nBOOSTING: many HIGH-BIAS models added in sequence (depth-2 stumps)")
print(f"  {'n_estimators':>13} {'bias^2':>9} {'variance':>9} {'total':>9}")
for n_est in (1, 10, 50, 200, 800):
    d = decompose(
        lambda k=n_est: GradientBoostingRegressor(
            n_estimators=k, max_depth=2, learning_rate=0.1, random_state=0),
        n_datasets=80, seed=13)
    print(f"  {n_est:>13} {d['bias^2']:9.4f} {d['variance']:9.4f} {d['total']:9.4f}")
print("\nBias falls steeply as trees are added, and variance creeps up. That")
print("asymmetry is why boosting's key control is WHEN TO STOP (see early")
print("stopping) while a forest's is how many trees you can afford.")

# --- MORE DATA REDUCES VARIANCE, NOT BIAS ---
print("\nEFFECT OF TRAINING SIZE (degree 12, a high-variance model)")
print(f"  {'n_train':>8} {'bias^2':>9} {'variance':>9} {'total':>9}")
for n in (30, 60, 200, 1000):
    d = decompose(lambda: make_pipeline(PolynomialFeatures(12),
                                        LinearRegression()),
                  n_train=n, n_datasets=150, seed=17)
    print(f"  {n:>8,} {d['bias^2']:9.4f} {d['variance']:9.4f} {d['total']:9.4f}")
print("\nVariance collapses with n while bias stays put. This is the formal")
print("version of 'more data cures overfitting, not underfitting'.")

# --- THE MAP FROM SYMPTOM TO CAUSE TO FIX ---
mapping = pd.DataFrame([
    ("high train AND test error, small gap", "HIGH BIAS",
     "more capacity, better features, less regularisation"),
    ("low train error, high test error, big gap", "HIGH VARIANCE",
     "more data, less capacity, regularise, ensemble/average"),
    ("test error plateaus well above zero with lots of data", "BIAS + NOISE",
     "change the model family or accept the floor"),
    ("scores swing a lot between random seeds", "HIGH VARIANCE",
     "average over seeds, or reduce capacity"),
    ("cross-validation folds disagree strongly", "HIGH VARIANCE or too little data",
     "more data, repeated CV, simpler model"),
], columns=["symptom", "cause", "fix"])
print("\n", mapping.to_string(index=False))

  THE EXACT DECOMPOSITION (squared loss)

    E[(y - f_hat(x))^2]  =  bias(x)^2  +  variance(x)  +  noise

      bias      how far the AVERAGE prediction is from the truth
      variance  how much predictions SCATTER across training sets
      noise     irreducible; the floor on achievable error

  THE THOUGHT EXPERIMENT

    draw 300 training sets -> fit 300 models -> predict ONE test point

    high bias, low variance      low bias, high variance
       predictions:                 predictions:
       ....|....                    .  . | .  .   .
           ^ truth                        ^ truth
       tightly clustered,            centred on the truth,
       in the WRONG place            but wildly scattered

  BY POLYNOMIAL DEGREE (noise variance = 0.36)

    degree   bias^2   variance   total
    1        1.7043    0.0290    2.0933
    2        1.6733    0.0455    2.0788
    3        0.1479    0.0621    0.5700
    4        0.1394    0.0801    0.5795
    6        0.0104    0.1373    0.5077   <- minimum
    9        0.0095    0.2638    0.6333
    13       0.0116    0.9002    1.2718

    bias falls monotonically | variance rises monotonically | sum is a U

  ERROR CURVES

  error
    |\                                    /
    | \                                  /  variance
    |  \                                /
    |   \___                      _____/
    |       \____            ____/
    |            \__________/               TOTAL (U-shaped)
    |  bias  ____/          \____
    |______/                     \______
    |- - - - - - - - - - - - - - - - - -   irreducible noise floor
    +------------------------------------> capacity
       underfitting      optimum      overfitting

  WHY AVERAGING WORKS

    variance of an average of k models = (rho + (1-rho)/k) * single variance

    rho     k=1     k=10    k=100
    0.0     1.000   0.100   0.010    <- ideal, fully decorrelated
    0.3     1.000   0.370   0.307
    0.7     1.000   0.730   0.703
    1.0     1.000   1.000   1.000    <- identical models: no benefit

    Random forests inject feature and row randomness to lower rho.

  BAGGING VS BOOSTING

    bagging   averages LOW-bias, HIGH-variance models  -> cuts VARIANCE
    boosting  sequences HIGH-bias, LOW-variance models -> cuts BIAS

    which is why a forest's main risk is cost and a boosted model's main
    risk is training too long.

Key Takeaways

• Expected squared error decomposes exactly into bias squared, variance, and irreducible noise.
• Capacity moves bias and variance in opposite directions, so total error is U-shaped in complexity.
• Underfitting is high bias and overfitting is high variance, which is what makes the decomposition practically useful.
• Averaging decorrelated models cuts variance without raising bias, which is the entire argument for bagging and random forests.
• More training data reduces variance and leaves bias untouched, which is the formal statement of when collecting data helps.

🧪 Practice

1. Fit the same model on 300 simulated training sets and compute bias squared, variance, and their sum at several polynomial degrees.
2. Repeat the decomposition using tree depth as the capacity dial and confirm the same opposite movement.
3. Show that bagging many unrestricted trees reduces variance while leaving bias roughly unchanged.
4. Compute the variance of an average of k models at correlations of 0, 0.3, and 0.7, and explain why forests randomise features.
5. Demonstrate that increasing training size reduces variance but not bias.
6. Interview: Your model's cross-validation scores range from 0.71 to 0.89 across five folds. What does that spread tell you, and what would you try? (Hint: which of the two components does fold-to-fold disagreement measure?)

Learning Curves

A learning curve plots model performance against the amount of training data, with separate curves for training and validation scores. It is the most informative single diagnostic in machine learning, and it answers a question that nothing else answers directly: would more data help?

That question has real money attached. Labelling is expensive, data collection takes months, and the alternative fixes, better features or a different model family, are also expensive. Guessing wrong wastes a quarter. The learning curve tells you which lever to pull, and it costs only the compute to refit your model at several training sizes.

Reading it comes down to three features.

Where the curves converge. If the training and validation curves have met and flattened, more data will not help. Whatever gap remains between that plateau and perfect performance is bias plus irreducible noise, and you need a different model or better features.

How large the gap is. A wide persistent gap between training and validation performance means variance, and variance is what more data reduces. If the validation curve is still rising when you run out of data, more data is the highest expected-value investment available.

Where the plateau sits. Both curves flat and close together at a poor score is the signature of underfitting, and it is the one case where more data is definitely not the answer.

A second kind of curve is worth distinguishing. A validation curve, sometimes called a complexity curve, plots performance against a hyperparameter rather than against data volume. It shows the U-shape from the previous topic directly and is how you pick a capacity setting. Learning curves answer "more data?", validation curves answer "more complexity?", and confusing them leads to the wrong fix.

Three practical notes.

Plot error bars. At small training sizes the estimate is noisy, so refit several times per size with different subsamples and show the spread. A single noisy curve invites over-reading.

Extrapolate cautiously. Learning curves often follow a power law, so fitting one can forecast the gain from doubling your data. That forecast is useful for a budget conversation and it is an extrapolation, so treat it as an estimate with a range.

Use the same splitting discipline. If your problem needs grouped or temporal splits, the learning curve must use them too, or it measures the wrong thing at every size.

import numpy as np
import pandas as pd
from sklearn.model_selection import learning_curve, validation_curve, KFold
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
from sklearn.ensemble import RandomForestRegressor
from sklearn.tree import DecisionTreeRegressor

rng = np.random.default_rng(23)

# --- THREE SCENARIOS THAT LOOK SIMILAR AND NEED OPPOSITE FIXES ---
def truth(x):
    return np.sin(1.4 * x[:, 0]) * 2.5 + 0.6 * x[:, 1] - 0.4 * x[:, 2] ** 2

N = 4_000
X = rng.uniform(-3, 3, (N, 3))
y = truth(X) + rng.normal(0, 0.5, N)

def curve(model, X, y, sizes=(0.02, 0.05, 0.1, 0.2, 0.4, 0.7, 1.0), reps=4):
    """Training and validation error at several training sizes, with spread."""
    out = []
    for frac in sizes:
        tr_scores, va_scores = [], []
        for rep in range(reps):
            r = np.random.default_rng(100 + rep)
            idx = r.permutation(len(X))
            n_tr = max(int(len(X) * 0.75 * frac), 8)
            tr, va = idx[:n_tr], idx[int(len(X) * 0.75):]
            m = model().fit(X[tr], y[tr])
            tr_scores.append(-m.score(X[tr], y[tr]))
            va_scores.append(-m.score(X[va], y[va]))
        out.append({"n_train": n_tr,
                    "train_err": np.mean(tr_scores),
                    "val_err": np.mean(va_scores),
                    "val_sd": np.std(va_scores),
                    "gap": np.mean(va_scores) - np.mean(tr_scores)})
    return pd.DataFrame(out)

scenarios = {
    "HIGH BIAS (linear on a curve)": lambda: LinearRegression(),
    "HIGH VARIANCE (deep tree)": lambda: DecisionTreeRegressor(
        min_samples_leaf=1, random_state=0),
    "WELL MATCHED (forest)": lambda: RandomForestRegressor(
        n_estimators=60, random_state=0),
}
for name, model in scenarios.items():
    c = curve(model, X, y)
    print(f"\n{name}")
    print("  (values are negative R-squared, so LOWER is better)")
    print(c.round(4).to_string(index=False))
    converged = abs(c["gap"].iloc[-1]) < 0.05
    still_improving = (c["val_err"].iloc[-2] - c["val_err"].iloc[-1]) > 0.005
    print(f"  curves converged: {converged}"
          f" | validation still improving: {still_improving}")
    if converged and not still_improving:
        print("  -> VERDICT: more data will NOT help. Change model/features.")
    elif not converged and still_improving:
        print("  -> VERDICT: MORE DATA is the highest-value lever.")
    elif not converged:
        print("  -> VERDICT: variance remains; regularise or add data.")
    else:
        print("  -> VERDICT: near the achievable floor.")

# --- AN ASCII LEARNING CURVE ---
c = curve(scenarios["HIGH VARIANCE (deep tree)"], X, y)
print("\nHIGH-VARIANCE learning curve (T = train error, V = validation error)")
lo = 0.0
hi = max(c["val_err"].max(), 0.6)
for _, row in c.iterrows():
    t_pos = int(np.clip(row["train_err"] / hi * 40, 0, 40))
    v_pos = int(np.clip(row["val_err"] / hi * 40, 0, 40))
    line = [" "] * 42
    line[t_pos] = "T"
    line[v_pos] = "V" if v_pos != t_pos else "*"
    print(f"  n={int(row['n_train']):>5,} |{''.join(line)}|"
          f" gap {row['gap']:.3f}")
print(f"  {'':>8} 0{'':>39}{hi:.1f}  (error ->)")
print("  The V marker walks LEFT as data grows while T stays near zero: the")
print("  gap is closing, so data is still buying accuracy.")

# --- VALIDATION CURVE: a different question entirely. ---
print("\nVALIDATION CURVE over polynomial degree (the COMPLEXITY question)")
degrees = [1, 2, 3, 4, 6, 8, 11]
train_sc, val_sc = validation_curve(
    make_pipeline(PolynomialFeatures(), Ridge(alpha=1e-6)),
    X[:800], y[:800], param_name="polynomialfeatures__degree",
    param_range=degrees, cv=5, scoring="neg_mean_squared_error")
vc = pd.DataFrame({"degree": degrees,
                   "train_mse": -train_sc.mean(axis=1),
                   "val_mse": -val_sc.mean(axis=1)})
vc["gap"] = vc["val_mse"] - vc["train_mse"]
print(vc.round(4).to_string(index=False))
print(f"best degree: {vc.loc[vc['val_mse'].idxmin(), 'degree']}")
print("\nLearning curve asks 'more DATA?'. Validation curve asks 'more")
print("COMPLEXITY?'. Answering the wrong one wastes a quarter.")

# --- FORECASTING THE GAIN FROM MORE DATA (a power law fit) ---
big = curve(scenarios["WELL MATCHED (forest)"], X, y,
            sizes=(0.05, 0.1, 0.2, 0.4, 0.7, 1.0), reps=3)
ns = big["n_train"].values.astype(float)
errs = big["val_err"].values
# Learning curves often follow err = a * n^(-b) + c. Fit log-log on the
# excess error above an assumed floor to estimate the exponent b.
floor = errs.min() * 0.75
excess = np.maximum(errs - floor, 1e-6)
b, log_a = np.polyfit(np.log(ns), np.log(excess), 1)
print(f"\npower-law fit: excess error ~ n^({b:.3f})")
for mult in (2, 5, 10):
    projected = floor + np.exp(log_a) * (ns[-1] * mult) ** b
    gain = errs[-1] - projected
    print(f"  {mult:>2}x the data -> projected error {projected:.4f}"
          f"  (improvement {gain:.4f}, {gain/errs[-1]:.1%})")
print("Useful for a budget conversation, and it IS an extrapolation. Report")
print("it as an estimate with a range, not a promise.")

# --- WHAT EACH SHAPE MEANS ---
shapes = pd.DataFrame([
    ("both flat and HIGH, tiny gap", "underfitting / high bias",
     "more capacity, better features", "NO"),
    ("train LOW, val HIGH, wide gap", "overfitting / high variance",
     "more data, regularise, simplify", "YES"),
    ("val still falling at max n", "variance not yet exhausted",
     "collect more data", "YES, clearly"),
    ("converged and flat at a good score", "near the achievable floor",
     "accept, or change the problem", "no"),
    ("val curve NON-monotonic / noisy", "too little data per point",
     "more repeats per size", "unclear yet"),
    ("val BELOW train", "leak, or a split bug",
     "audit the split and preprocessing", "fix the bug first"),
], columns=["curve shape", "diagnosis", "action", "would more data help?"])
print("\n", shapes.to_string(index=False))

  THE FOUR CANONICAL SHAPES

  HIGH BIAS (underfit)            HIGH VARIANCE (overfit)
  error                           error
    |                               |
    |V                              |V
    |T____________________          | V___
    |                               |     V____V____V
    |                               |T____________________
    +------------------> n          +------------------> n
    converged, but HIGH             wide GAP, V still falling
    -> more data will NOT help      -> MORE DATA WILL HELP

  GOOD FIT                        SOMETHING IS BROKEN
  error                           error
    |V                              |T
    | V__                           | T___
    |    V___T___T___T              |     T___
    |T___                           |V____V____V___
    +------------------> n          +------------------> n
    converged at a LOW error        validation BEATS training
    -> near the floor               -> leak or split bug

  READING A REAL CURVE (high-variance case, error axis 0 -> 0.6)

    n=   60 |T                     V           | gap 0.412
    n=  150 |T                 V               | gap 0.331
    n=  300 |T            V                    | gap 0.244
    n=  600 |T        V                        | gap 0.176
    n=1,200 |T     V                           | gap 0.121
    n=2,100 |T   V                             | gap 0.094
    n=3,000 |T  V                              | gap 0.081
             0                              0.6

    V marches left, T stays put: the gap is closing and data still pays.

  TWO DIFFERENT CURVES, TWO DIFFERENT QUESTIONS

    LEARNING curve     x-axis = training SIZE      "would more data help?"
    VALIDATION curve   x-axis = a HYPERPARAMETER   "more or less capacity?"

    Answering the wrong question is how teams spend a quarter collecting
    data for a model that was underfitting all along.

  FORECASTING THE GAIN (power law: excess error ~ n^-b)

    fitted exponent b = -0.38

    2x the data  -> error 0.1832 (improvement 6.1%)
    5x the data  -> error 0.1668 (improvement 14.5%)
    10x the data -> error 0.1571 (improvement 19.5%)

    Enough to decide whether a labelling budget is worth requesting.

Key Takeaways

• A learning curve plots training and validation performance against data volume and directly answers whether more data would help.
• Converged flat curves at a poor score mean high bias, so collecting data is the wrong investment.
• A wide gap with validation error still falling means variance remains and more data is the highest-value lever.
• Validation curves plot performance against a hyperparameter and answer the complexity question, which is a different question from the data question.
• Refit several times per training size to show spread, and treat any power- law extrapolation as an estimate rather than a promise.

🧪 Practice

1. Plot learning curves for a deliberately underfitting model and a deliberately overfitting model on the same data, and describe the two shapes.
2. Write a learning-curve function that repeats each training size several times and reports the standard deviation of the validation score.
3. Produce a validation curve over a capacity hyperparameter and identify the minimum.
4. Fit a power law to a learning curve and project the error at ten times the current data volume.
5. Interview: A team wants to spend three months labelling more data. Which single plot would you produce first, and what would convince you they are wrong? (Hint: what do the two curves look like when the limitation is the model rather than the sample?)

Regularization: L1, L2, Elastic Net

Regularisation adds a penalty on model complexity to the loss function, so the optimiser must balance fitting the data against keeping the model simple. In bias-variance terms it deliberately increases bias to reduce variance, and it pays whenever the variance reduction is larger, which on real data it very often is.

The mechanism for linear models is direct: add a term proportional to the size of the coefficients. Large coefficients mean a model that reacts sharply to small input changes, which is exactly the sensitivity that causes variance. Penalising their size buys stability.

The two standard penalties behave completely differently, and the difference is geometric rather than arbitrary.

L2, ridge, penalises the sum of squared coefficients. It shrinks all coefficients toward zero proportionally and sets none of them exactly to zero. Its particular strength is handling correlated features: where least squares would give two collinear features large opposite coefficients, ridge splits the weight between them, producing a stable and sensible solution. It also has a closed-form solution and is differentiable everywhere.

L1, lasso, penalises the sum of absolute coefficients. It drives coefficients to exactly zero, performing feature selection as part of fitting. The reason is the shape of the constraint region: the L1 ball has corners on the axes, and the optimum of a smooth loss over a region with corners tends to land on one, which means a coefficient of exactly zero. The L2 ball is round, has no corners, and so produces small-but-nonzero values instead. With correlated features lasso picks one somewhat arbitrarily, which is a real weakness when you want to interpret which feature mattered.

Elastic net combines both, keeping lasso's sparsity while retaining ridge's grouping behaviour on correlated features. It has two hyperparameters, the overall strength and the mix, and it is the sensible default when features are both numerous and correlated.

Three practical requirements.

Scale your features first. The penalty is on coefficient magnitude, and a coefficient's magnitude depends on its feature's units. Without standardisation the penalty falls unequally across features for no principled reason. This is covered in chapter 6.3 and it is the most common regularisation bug.

Do not penalise the intercept. It is a location parameter, not a measure of sensitivity, and shrinking it biases predictions toward zero.

Tune the strength by cross-validation. The right penalty depends on the signal-to-noise ratio and the sample size, which you cannot know in advance.

Regularisation is not only for linear models. Weight decay in neural networks is L2. Dropout randomly removes units during training, which acts like averaging an ensemble. Tree ensembles regularise through depth limits, minimum leaf sizes, subsampling, and shrinkage. The unifying idea is always the same: restrict what the model can express so it cannot express the noise.

import numpy as np
import pandas as pd
from sklearn.linear_model import (LinearRegression, Ridge, Lasso, ElasticNet,
                                  RidgeCV, LassoCV, ElasticNetCV)
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import mean_squared_error

rng = np.random.default_rng(29)

# --- A HARD PROBLEM: many features, few rows, mostly irrelevant, correlated. ---
n, p, k_true = 150, 80, 5
base = rng.normal(size=(n, k_true))
X = np.zeros((n, p))
X[:, :k_true] = base
# Features 5-9 are near-copies of the first five: collinearity on purpose.
X[:, k_true:2*k_true] = base + rng.normal(0, 0.15, (n, k_true))
X[:, 2*k_true:] = rng.normal(size=(n, p - 2*k_true))     # pure noise
true_coef = np.zeros(p)
true_coef[:k_true] = [3.0, -2.2, 1.8, -1.4, 1.0]
y = X @ true_coef + rng.normal(0, 1.5, n)

Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.35, random_state=0)
print(f"n={n}, p={p}, truly relevant: {k_true}, "
      f"and features 5-9 are near-copies of 0-4")

# --- UNREGULARISED LEAST SQUARES falls apart here. ---
ols = LinearRegression().fit(Xtr, ytr)
print(f"\nOLS: train MSE {mean_squared_error(ytr, ols.predict(Xtr)):8.3f}"
      f"  test MSE {mean_squared_error(yte, ols.predict(Xte)):8.3f}")
print(f"  largest |coefficient|: {np.abs(ols.coef_).max():.1f}"
      f"   (true maximum is {np.abs(true_coef).max():.1f})")
print("  Huge coefficients on noise features: the signature of variance.")

# --- THE THREE PENALTIES, compared on the same data. ---
print("\n{:<22} {:>9} {:>9} {:>8} {:>10} {:>12}".format(
    "model", "train MSE", "test MSE", "non-zero", "max |coef|", "found top-5"))
def report(label, model):
    pipe = make_pipeline(StandardScaler(), model).fit(Xtr, ytr)
    coef = pipe[-1].coef_
    nz = int((np.abs(coef) > 1e-6).sum())
    top5 = set(np.argsort(-np.abs(coef))[:5])
    print("{:<22} {:>9.3f} {:>9.3f} {:>8} {:>10.2f} {:>12}".format(
        label, mean_squared_error(ytr, pipe.predict(Xtr)),
        mean_squared_error(yte, pipe.predict(Xte)), nz, np.abs(coef).max(),
        str(len(top5 & set(range(2*k_true)))) + "/5 in 0-9"))
    return coef

c_ols = report("OLS (no penalty)", LinearRegression())
c_ridge = report("ridge (L2)", Ridge(alpha=10.0))
c_lasso = report("lasso (L1)", Lasso(alpha=0.35, max_iter=10_000))
c_enet = report("elastic net", ElasticNet(alpha=0.35, l1_ratio=0.5,
                                         max_iter=10_000))
print("\nRidge keeps all 80 features with small weights. Lasso zeroes most of")
print("them. Both beat OLS on test error by a wide margin.")

# --- SPARSITY: exactly how many coefficients hit zero? ---
print("\nNON-ZERO COEFFICIENTS AS THE PENALTY STRENGTHENS")
print(f"  {'alpha':>8} {'ridge nz':>10} {'lasso nz':>10} {'enet nz':>9}"
      f" {'ridge test':>11} {'lasso test':>11}")
for alpha in (0.001, 0.01, 0.1, 0.5, 1.0, 3.0):
    r = make_pipeline(StandardScaler(), Ridge(alpha=alpha)).fit(Xtr, ytr)
    l = make_pipeline(StandardScaler(),
                      Lasso(alpha=alpha, max_iter=20_000)).fit(Xtr, ytr)
    e = make_pipeline(StandardScaler(),
                      ElasticNet(alpha=alpha, l1_ratio=0.5,
                                 max_iter=20_000)).fit(Xtr, ytr)
    print(f"  {alpha:>8.3f} {int((np.abs(r[-1].coef_)>1e-6).sum()):>10}"
          f" {int((np.abs(l[-1].coef_)>1e-6).sum()):>10}"
          f" {int((np.abs(e[-1].coef_)>1e-6).sum()):>9}"
          f" {mean_squared_error(yte, r.predict(Xte)):>11.3f}"
          f" {mean_squared_error(yte, l.predict(Xte)):>11.3f}")
print("Ridge NEVER reaches zero; lasso does, and progressively more so.")

# --- CORRELATED FEATURES: where ridge and lasso genuinely differ. ---
print("\nCOEFFICIENTS ON FIVE NEAR-IDENTICAL FEATURES (0 and 5 are copies)")
pairs = [(0, 5), (1, 6), (2, 7)]
print(f"  {'pair':>8} {'true':>8} {'OLS':>16} {'ridge':>16} {'lasso':>16}")
scaler_sd = Xtr.std(axis=0)
for a, b in pairs:
    print(f"  ({a:>2},{b:>2}) {true_coef[a]:>8.2f}"
          f"  {c_ols[a]*scaler_sd[a]:>7.2f}/{c_ols[b]*scaler_sd[b]:>7.2f}"
          f"  {c_ridge[a]:>7.2f}/{c_ridge[b]:>7.2f}"
          f"  {c_lasso[a]:>7.2f}/{c_lasso[b]:>7.2f}")
print("\nRidge SPLITS the weight between each correlated pair. Lasso picks one")
print("and zeroes the other, which is fine for prediction and misleading if")
print("you report 'feature 0 matters and feature 5 does not'.")

# --- STABILITY UNDER RESAMPLING: the interpretability consequence. ---
print("\nHOW OFTEN IS EACH CORRELATED FEATURE SELECTED? (40 bootstrap fits)")
sel_lasso = np.zeros(p)
sel_enet = np.zeros(p)
for b_i in range(40):
    r = np.random.default_rng(b_i)
    idx = r.choice(len(Xtr), len(Xtr), replace=True)
    lm = make_pipeline(StandardScaler(),
                       Lasso(alpha=0.35, max_iter=20_000)).fit(Xtr[idx], ytr[idx])
    em = make_pipeline(StandardScaler(),
                       ElasticNet(alpha=0.35, l1_ratio=0.5,
                                  max_iter=20_000)).fit(Xtr[idx], ytr[idx])
    sel_lasso += np.abs(lm[-1].coef_) > 1e-6
    sel_enet += np.abs(em[-1].coef_) > 1e-6
print(f"  {'feature':>8} {'lasso':>8} {'elastic net':>13}")
for j in list(range(5)) + [5, 6, 7]:
    print(f"  {j:>8} {sel_lasso[j]/40:>8.0%} {sel_enet[j]/40:>13.0%}")
print("Lasso's choice between correlated twins flips between resamples.")
print("Elastic net keeps both more often, which is the 'grouping' property.")

# --- SCALING IS MANDATORY ---
X_unscaled = Xtr.copy()
X_unscaled[:, 0] *= 1000                # one feature in different units
print("\nWHY SCALING MATTERS (feature 0 multiplied by 1000)")
for label, pipe in [("ridge WITHOUT scaling", Ridge(alpha=10.0)),
                    ("ridge WITH scaling",
                     make_pipeline(StandardScaler(), Ridge(alpha=10.0)))]:
    m = pipe.fit(X_unscaled, ytr)
    coef = m.coef_ if hasattr(m, "coef_") else m[-1].coef_
    print(f"  {label:24s} coefficient on feature 0: {coef[0]:>10.5f}")
print("Unscaled, the penalty barely touches a large-unit feature, so the")
print("regularisation is applied unequally for no principled reason.")

# --- TUNE THE STRENGTH BY CROSS-VALIDATION ---
print("\nCROSS-VALIDATED PENALTY SELECTION")
for label, model in [
        ("RidgeCV", make_pipeline(StandardScaler(),
                                  RidgeCV(alphas=np.logspace(-3, 3, 40)))),
        ("LassoCV", make_pipeline(StandardScaler(),
                                  LassoCV(cv=5, random_state=0, max_iter=20_000))),
        ("ElasticNetCV", make_pipeline(
            StandardScaler(),
            ElasticNetCV(l1_ratio=[.1, .5, .7, .9, .99], cv=5,
                         random_state=0, max_iter=20_000)))]:
    m = model.fit(Xtr, ytr)
    est = m[-1]
    chosen = getattr(est, "alpha_", None)
    extra = f", l1_ratio={est.l1_ratio_:.2f}" if hasattr(est, "l1_ratio_") else ""
    print(f"  {label:14s} alpha {chosen:.4f}{extra}"
          f"  test MSE {mean_squared_error(yte, m.predict(Xte)):.3f}"
          f"  non-zero {int((np.abs(est.coef_) > 1e-6).sum())}")

  WHY THE PENALTIES DIFFER: GEOMETRY

  L2 (ridge): a round ball        L1 (lasso): a diamond with CORNERS
        beta2                            beta2
          |    ___                         |     /\
          |  /     \                       |   /    \
      ----+-(---o---)----  beta1      -----+-/---o----\----  beta1
          |  \     /                       |   \    /
          |    ---                         |     \/
      the loss contour touches         the loss contour touches a
      a SMOOTH boundary                CORNER, which sits ON an axis
      -> both coefficients small       -> one coefficient EXACTLY zero

    That is the whole reason lasso selects features and ridge does not.

  ON A HARD PROBLEM (n=150, p=80, 5 truly relevant, 5 near-duplicates)

    model              train MSE   test MSE   non-zero   max |coef|
    OLS (no penalty)       0.412     14.883        80        9.71
    ridge (L2)             1.106      3.291        80        1.54
    lasso (L1)             1.842      2.604        11        2.43
    elastic net            1.903      2.588        17        2.11

    OLS fits the training noise and is 5x worse on test data.

  SPARSITY AS THE PENALTY STRENGTHENS

    alpha    ridge nz   lasso nz   enet nz
    0.001          80         78        79
    0.01           80         62        71
    0.1            80         26        38
    0.5            80          8        13
    1.0            80          4         7
    3.0            80          0         2

    Ridge never reaches zero, by construction.

  CORRELATED PAIRS (features 0 and 5 are near-identical)

    pair     true      OLS              ridge           lasso
    ( 0, 5)  3.00      6.41 / -3.38     1.54 / 1.39     2.43 / 0.00
    ( 1, 6) -2.20     -4.77 /  2.62    -1.12 / -1.02   -1.88 / 0.00
    ( 2, 7)  1.80      3.92 / -2.10     0.91 / 0.84     0.00 / 1.51

    ridge SPLITS the weight; lasso picks ONE, and which one flips between
    resamples (selected 55% vs 45% of the time across bootstraps).

  CHOOSING

  situation                              use
  ---------------------------------------------------------------------------
  many correlated features               ridge, or elastic net
  want automatic feature selection       lasso
  many features, many correlated         ELASTIC NET (the usual default)
  p >> n                                 lasso or elastic net
  need stable coefficient interpretation ridge (lasso's choice is unstable)
  neural network                         weight decay (= L2) + dropout
  tree ensemble                          depth, min leaf, subsample, shrinkage

  NON-NEGOTIABLE: standardise features first, never penalise the intercept,
  and tune the strength by cross-validation.

Key Takeaways

• Regularisation adds a complexity penalty to the loss, trading a little bias for a larger reduction in variance.
• L2 shrinks all coefficients and never zeroes them, and it splits weight sensibly between correlated features.
• L1 drives coefficients to exactly zero because its constraint region has corners on the axes, giving feature selection during fitting.
• Lasso's choice among correlated features is unstable across resamples, so prefer ridge or elastic net when you will interpret coefficients.
• Standardise features before penalising, never penalise the intercept, and select the strength by cross-validation.

🧪 Practice

1. Build a problem with more features than rows and compare unregularised least squares against ridge and lasso on test error.
2. Sweep the penalty strength and tabulate how many coefficients each method leaves non-zero.
3. Create two near-identical features and compare how ridge and lasso divide the weight between them.
4. Bootstrap the fit forty times and report how often lasso selects each of a correlated pair.
5. Multiply one feature by 1000 and show that regularisation without standardisation treats it differently.
6. Interview: Your lasso model selects "days since last login" and drops "login frequency", and a stakeholder concludes frequency does not matter. What do you say? (Hint: what are those two features likely to be, and how does lasso behave in that situation?)

Early Stopping

For any model trained iteratively, capacity grows with training time. A neural network initialised near zero starts as a nearly linear function and becomes more expressive with each epoch. A boosted ensemble adds one tree per iteration, each correcting the last. In both cases the number of iterations is a capacity dial, and like every capacity dial it has a setting that underfits and a setting that overfits.

Early stopping means monitoring performance on held-out data during training and stopping when it stops improving. It is the cheapest and most widely used regularisation technique in practice, for two reasons: it requires no extra hyperparameter search over penalty strengths, and it saves compute rather than costing it.

The characteristic picture is two curves diverging. Training loss falls monotonically, because the optimiser is minimising it by construction. Validation loss falls, reaches a minimum, and then rises as the model begins fitting the training sample's noise. The minimum of the validation curve is the stopping point.

Making this work in practice requires four specifics.

Patience. Validation loss is noisy, so a single bad epoch is not evidence of overfitting. Patience is the number of epochs to wait without improvement before stopping. Too small and you stop on noise; too large and you waste compute. Ten to twenty epochs is typical for neural networks, and thirty to fifty rounds for boosting.

Restore the best weights. When you stop at epoch 60 because the best was epoch 45, the model currently in memory is the epoch-60 one, which is worse. Most frameworks have an option for this and it is easy to forget.

Use a separate split. The stopping decision uses the validation set, which means validation performance is now optimistically biased for the same reason as any other tuned quantity. If you need an unbiased estimate, keep a third split.

Monitor the right quantity. Stopping on the metric you care about is tempting, and metrics such as accuracy are noisier than the loss, which makes the stopping point less stable. Stopping on validation loss and reporting the metric is usually more reliable.

There is a deeper connection worth knowing: early stopping with gradient descent is approximately equivalent to L2 regularisation. Gradient descent from a small initialisation explores low-complexity solutions first, so stopping early is a way of restricting the effective model class. This is why the two techniques are partial substitutes and why stacking a lot of both can underfit.

import numpy as np
import pandas as pd
from sklearn.datasets import make_regression, make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.linear_model import SGDRegressor
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import mean_squared_error, log_loss, accuracy_score

rng = np.random.default_rng(31)

# --- BOOSTING: the number of trees IS the capacity dial. ---
n = 1_200
X = rng.uniform(-3, 3, (n, 5))
y = (np.sin(1.5 * X[:, 0]) * 3 + 0.8 * X[:, 1] - 0.5 * X[:, 2] ** 2
     + rng.normal(0, 1.0, n))
Xtr, Xtmp, ytr, ytmp = train_test_split(X, y, test_size=0.4, random_state=0)
Xva, Xte, yva, yte = train_test_split(Xtmp, ytmp, test_size=0.5, random_state=0)
print(f"train {len(Xtr)}, validation {len(Xva)}, test {len(Xte)}")

# staged_predict yields predictions after each successive tree, so one fit
# produces the whole curve.
model = GradientBoostingRegressor(n_estimators=1200, learning_rate=0.05,
                                  max_depth=3, random_state=0).fit(Xtr, ytr)
tr_curve = [mean_squared_error(ytr, p) for p in model.staged_predict(Xtr)]
va_curve = [mean_squared_error(yva, p) for p in model.staged_predict(Xva)]
best_iter = int(np.argmin(va_curve)) + 1

print("\nBOOSTING LOSS BY ITERATION")
print(f"  {'trees':>6} {'train MSE':>10} {'val MSE':>10}")
for i in (1, 10, 50, 100, best_iter, 400, 800, 1200):
    mark = "  <- validation minimum" if i == best_iter else ""
    print(f"  {i:>6} {tr_curve[i-1]:>10.4f} {va_curve[i-1]:>10.4f}{mark}")
print(f"\nbest iteration: {best_iter} of 1200")
print(f"  validation MSE at best: {va_curve[best_iter-1]:.4f}")
print(f"  validation MSE at 1200: {va_curve[-1]:.4f}"
      f"  ({va_curve[-1]/va_curve[best_iter-1]-1:+.1%} worse)")
print("Training MSE keeps falling forever; validation MSE turns around. The")
print("turning point is where the model starts fitting the sample's noise.")

# --- AN ASCII VIEW OF THE DIVERGENCE ---
print("\nloss curves (T = train, V = validation, error increases rightward)")
hi = max(va_curve[0], tr_curve[0])
for i in [1, 5, 20, 60, 150, best_iter, 500, 900, 1200]:
    t = int(np.clip(tr_curve[i-1] / hi * 40, 0, 40))
    v = int(np.clip(va_curve[i-1] / hi * 40, 0, 40))
    line = [" "] * 42
    line[t] = "T"
    line[v] = "V" if v != t else "*"
    star = " <- stop here" if i == best_iter else ""
    print(f"  {i:>5} |{''.join(line)}|{star}")
print(f"  {'':>5}  0{'':>38}{hi:.1f}")
print("  T marches left forever. V reverses direction and starts back right.")

# --- PATIENCE: how long to wait before believing the turn is real. ---
def early_stop(curve, patience: int):
    """Return the iteration chosen by a patience rule, plus how much compute
    it used. This is exactly what a framework's callback does."""
    best, best_i, waited = np.inf, 0, 0
    for i, v in enumerate(curve, start=1):
        if v < best - 1e-9:
            best, best_i, waited = v, i, 0
        else:
            waited += 1
            if waited >= patience:
                return best_i, i
    return best_i, len(curve)

print("\nPATIENCE CHOICE (true optimum is iteration "
      f"{best_iter})")
print(f"  {'patience':>9} {'chosen':>8} {'iterations run':>15} {'val MSE':>10}"
      f" {'test MSE':>10}")
for pat in (1, 5, 20, 50, 200):
    chosen, ran = early_stop(va_curve, pat)
    m = GradientBoostingRegressor(n_estimators=chosen, learning_rate=0.05,
                                  max_depth=3, random_state=0).fit(Xtr, ytr)
    print(f"  {pat:>9} {chosen:>8} {ran:>15} {va_curve[chosen-1]:>10.4f}"
          f" {mean_squared_error(yte, m.predict(Xte)):>10.4f}")
print("Patience 1 stops far too early on a noisy dip. Patience 50 finds the")
print("real minimum and costs only a little extra compute.")

# --- RESTORING THE BEST MODEL: easy to forget, and it matters. ---
final_model = GradientBoostingRegressor(n_estimators=1200, learning_rate=0.05,
                                        max_depth=3, random_state=0).fit(Xtr, ytr)
best_model = GradientBoostingRegressor(n_estimators=best_iter,
                                       learning_rate=0.05, max_depth=3,
                                       random_state=0).fit(Xtr, ytr)
print(f"\ntest MSE, model left at iteration 1200: "
      f"{mean_squared_error(yte, final_model.predict(Xte)):.4f}")
print(f"test MSE, model restored to iteration {best_iter}: "
      f"{mean_squared_error(yte, best_model.predict(Xte)):.4f}")
print("Stopping is not enough; you must ROLL BACK to the best checkpoint.")

# --- BUILT-IN EARLY STOPPING, which is what you should actually use. ---
from sklearn.ensemble import HistGradientBoostingRegressor
hgb = HistGradientBoostingRegressor(
    max_iter=2000, learning_rate=0.05,
    early_stopping=True,            # monitor an internal validation split
    validation_fraction=0.15,
    n_iter_no_change=40,            # patience
    random_state=0).fit(Xtr, ytr)
print(f"\nHistGradientBoosting stopped at {hgb.n_iter_} of 2000 iterations"
      f"  test MSE {mean_squared_error(yte, hgb.predict(Xte)):.4f}")

# --- NEURAL NETWORKS: epochs are the dial, and the loss is noisier. ---
Xc, yc = make_classification(n_samples=2_500, n_features=20, n_informative=8,
                             random_state=0)
cXtr, cXtmp, cytr, cytmp = train_test_split(Xc, yc, test_size=0.4, random_state=0)
cXva, cXte, cyva, cyte = train_test_split(cXtmp, cytmp, test_size=0.5,
                                          random_state=0)
net = MLPClassifier(hidden_layer_sizes=(128, 64), max_iter=1, warm_start=True,
                    learning_rate_init=0.01, random_state=0, alpha=0.0)
tr_ll, va_ll, va_acc = [], [], []
import warnings
warnings.filterwarnings("ignore")
for epoch in range(120):
    net.fit(cXtr, cytr)                          # warm_start: one epoch more
    tr_ll.append(log_loss(cytr, net.predict_proba(cXtr)))
    va_ll.append(log_loss(cyva, net.predict_proba(cXva)))
    va_acc.append(accuracy_score(cyva, net.predict(cXva)))
best_ll = int(np.argmin(va_ll)) + 1
best_acc = int(np.argmax(va_acc)) + 1
print(f"\nNEURAL NETWORK, 120 epochs")
print(f"  {'epoch':>6} {'train loss':>11} {'val loss':>10} {'val acc':>9}")
for e in (1, 5, 20, best_ll, 60, 120):
    print(f"  {e:>6} {tr_ll[e-1]:>11.4f} {va_ll[e-1]:>10.4f} {va_acc[e-1]:>9.4f}")
print(f"\nbest epoch by validation LOSS: {best_ll}")
print(f"best epoch by validation ACCURACY: {best_acc}")
print("The two disagree because accuracy is noisier and step-like. Stopping")
print("on the loss gives a more stable choice; report the metric separately.")

# --- EARLY STOPPING IS APPROXIMATELY L2 ---
print("\nEARLY STOPPING vs EXPLICIT L2 (linear model, gradient descent)")
Xr, yr = make_regression(n_samples=200, n_features=60, n_informative=8,
                         noise=12.0, random_state=0)
rXtr, rXte, rytr, ryte = train_test_split(Xr, yr, test_size=0.4, random_state=0)
print(f"  {'strategy':<34} {'test MSE':>10} {'||coef||':>10}")
for iters in (5, 25, 100, 2000):
    m = SGDRegressor(penalty=None, max_iter=iters, tol=None,
                     learning_rate="constant", eta0=0.002,
                     random_state=0).fit(rXtr, rytr)
    print(f"  {'no penalty, ' + str(iters) + ' iterations':<34}"
          f" {mean_squared_error(ryte, m.predict(rXte)):>10.1f}"
          f" {np.linalg.norm(m.coef_):>10.2f}")
from sklearn.linear_model import Ridge
for alpha in (0.1, 10.0, 200.0):
    m = Ridge(alpha=alpha).fit(rXtr, rytr)
    print(f"  {'ridge alpha=' + str(alpha) + ', converged':<34}"
          f" {mean_squared_error(ryte, m.predict(rXte)):>10.1f}"
          f" {np.linalg.norm(m.coef_):>10.2f}")
print("\nFewer gradient steps produce a smaller coefficient norm, exactly as a")
print("stronger L2 penalty does. Gradient descent from a small start explores")
print("simple solutions first, so stopping early restricts the model class.")

  THE CHARACTERISTIC DIVERGENCE

  loss
    |V
    | V
    |  V                                          V   <- validation turns up
    |   V                                    V
    |    V                              V
    |     V___                    __V
    |         V___V___V___V___V              <- the MINIMUM: stop here
    |T
    | T__
    |    T___
    |        T____
    |             T______T______T______T     <- training falls FOREVER
    +--------------------------------------> iterations

    Training loss is what the optimiser minimises, so it cannot rise.
    Validation loss is the only signal that capacity has gone too far.

  BOOSTING, MEASURED

    trees   train MSE   val MSE
    1          8.9124    9.0132
    10         4.2031    4.3910
    50         1.4820    1.6421
    100        1.0114    1.2887
    187        0.7203    1.1904   <- validation minimum
    400        0.3891    1.2455
    800        0.1702    1.3418
    1200       0.0894    1.4021   (+18% worse than the minimum)

  PATIENCE: HOW LONG TO WAIT BEFORE BELIEVING THE TURN

    patience   chosen iteration   iterations run   test MSE
    1          43                 44               1.3781
    5          122                127              1.2204
    20         171                191              1.1953
    50         187                237              1.1901   <- finds the true min
    200        187                387              1.1901   (wasted compute)

    Too little patience stops on a noisy dip. Too much wastes compute and
    changes nothing.

  THE FOUR THINGS TO GET RIGHT

    1. PATIENCE          10-20 epochs (nets), 30-50 rounds (boosting)
    2. RESTORE BEST      the model in memory at stop time is NOT the best
    3. SEPARATE SPLIT    the stopping set is now tuned on; keep a third set
    4. MONITOR THE LOSS  accuracy is noisier and gives an unstable stop point

  EARLY STOPPING IS APPROXIMATELY L2

    no penalty,    5 iterations   test MSE 3104.2   ||coef|| 12.4
    no penalty,   25 iterations   test MSE  481.7   ||coef|| 38.1
    no penalty, 2000 iterations   test MSE  392.6   ||coef|| 74.9

    ridge alpha=200, converged     test MSE  418.3   ||coef|| 21.7
    ridge alpha=10,  converged     test MSE  381.4   ||coef|| 52.6

    Fewer steps -> smaller norm, exactly like a stronger penalty. They are
    partial substitutes, so stacking heavy amounts of both underfits.

Key Takeaways

• For iteratively trained models the number of iterations is a capacity dial, and early stopping tunes it by watching held-out performance.
• Training loss falls monotonically by construction, so the validation curve's minimum is the only signal that capacity has gone too far.
• Set a patience window so a noisy dip does not stop training, and always restore the best checkpoint rather than the final one.
• The stopping set becomes a tuned quantity, so keep a third split for an unbiased estimate.
• Early stopping with gradient descent approximates L2 regularisation, which is why the two are partial substitutes.

🧪 Practice

1. Fit a boosted model with many iterations and use staged predictions to plot training and validation error, identifying the minimum.
2. Implement a patience rule and compare the chosen iteration at patience values of 1, 20, and 50.
3. Show that a model left at its final iteration performs worse on test data than one rolled back to the best checkpoint.
4. Train a neural network for many epochs and compare the best epoch chosen by validation loss against the best chosen by accuracy.
5. Demonstrate that stopping gradient descent early produces a smaller coefficient norm, like an L2 penalty.
6. Interview: A colleague uses early stopping on the validation set and reports that same validation score as the final result. What is wrong? (Hint: how many decisions did that split just make, and what does that do to its independence?)

Curse of Dimensionality

Adding features feels like adding information, and past a point it makes learning harder rather than easier. The collection of geometric effects responsible is called the curse of dimensionality, and it explains several otherwise surprising behaviours: why nearest-neighbour methods degrade, why every point looks equally far from every other, and why the sample size needed to cover a space grows explosively.

The root cause is that volume grows exponentially with dimension. To cover a one-dimensional interval at a resolution of 0.1 you need 10 cells. In two dimensions you need 100, in three 1,000, and in ten dimensions ten billion. Any fixed sample becomes vanishingly sparse as dimensions increase, so the data no longer covers the space it lives in. With 1,000 points in 10 dimensions, the average cell is empty, and a model asked to interpolate is really extrapolating.

Three consequences matter in practice.

Distances concentrate. In high dimensions the distance from a point to its nearest neighbour and to its farthest neighbour become almost the same. The ratio of the two approaches one. Since "nearest" is what nearest-neighbour methods, k-means, kernel methods, and radial basis functions all rely on, those methods lose their basis for discrimination. This is the sharpest and least intuitive effect.

Everything is in the shell. The volume of a high-dimensional ball is concentrated near its surface, so almost every point is near the boundary of the data's region rather than in the interior. Combined with the previous point, this means most predictions are made near the edge of the training distribution, where models are least reliable.

Everything becomes orthogonal. Random vectors in high dimensions are nearly perpendicular, so spurious correlations between unrelated features shrink toward zero while the maximum observed correlation across many pairs grows, which is the multiple comparisons problem from chapter 8 in geometric form.

The mitigations divide into three families.

Reduce the dimensions. Feature selection, covered in chapter 6.3, and dimensionality reduction such as principal component analysis, covered in chapter 11.2.

Use models with the right inductive bias. This is why the curse is less damaging than its name suggests. Real high-dimensional data usually lies on a much lower-dimensional manifold: images of faces occupy a tiny fraction of all possible pixel arrays. Models that exploit structure, such as tree ensembles doing axis-aligned splits or regularised linear models assuming sparsity, cope far better than distance-based methods. This is the manifold hypothesis, and it is why deep learning works on data with thousands of dimensions.

Add data, if you can. The required sample grows exponentially, so this is the weakest lever, but it is the only one that raises the ceiling rather than lowering the problem.

import numpy as np
import pandas as pd
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(37)

# --- VOLUME GROWS EXPONENTIALLY ---
print("SAMPLES NEEDED TO COVER THE UNIT CUBE AT RESOLUTION 0.1")
for d in (1, 2, 3, 5, 10, 20):
    cells = 10 ** d
    print(f"  {d:>2} dimensions: {cells:>25,} cells"
          f"   1,000 points fill {min(1000/cells, 1):.2e} of them")
print("\nA fixed sample becomes vanishingly sparse. A model asked to")
print("interpolate in 10 dimensions is usually extrapolating.")

# --- DISTANCE CONCENTRATION: the sharpest effect. ---
print("\nDISTANCE CONCENTRATION (1,000 uniform random points)")
print(f"  {'dim':>4} {'min dist':>10} {'max dist':>10} {'ratio max/min':>14}"
      f" {'rel. contrast':>14}")
for d in (1, 2, 5, 10, 50, 200, 1000):
    pts = rng.random((1000, d))
    q = rng.random((30, d))
    ratios, contrasts = [], []
    for point in q:
        dist = np.linalg.norm(pts - point, axis=1)
        dmin, dmax = dist.min(), dist.max()
        ratios.append(dmax / dmin)
        # Relative contrast: how much the nearest stands out from the mean.
        contrasts.append((dist.mean() - dmin) / dist.mean())
    dists = [np.linalg.norm(pts - point, axis=1) for point in q]
    print(f"  {d:>4} {np.mean([dd.min() for dd in dists]):>10.3f}"
          f" {np.mean([dd.max() for dd in dists]):>10.3f}"
          f" {np.mean(ratios):>14.3f} {np.mean(contrasts):>14.4f}")
print("\nIn 1 dimension the farthest point is ~100x the distance of the")
print("nearest. By 1,000 dimensions the ratio is ~1.1: 'nearest' has almost")
print("no meaning, which is exactly what kNN and k-means depend on.")

# --- VOLUME IS IN THE SHELL ---
print("\nFRACTION OF A BALL'S VOLUME IN THE OUTER 10% SHELL")
for d in (1, 2, 3, 10, 50, 200):
    # V(r) ~ r^d, so the inner 90% radius holds 0.9^d of the volume.
    shell = 1 - 0.9 ** d
    print(f"  {d:>3} dimensions: {shell:>7.2%} of the volume is in the shell")
print("\nIn 50 dimensions, 99.5% of points sit in the outer 10% of the radius.")
print("Almost every prediction is made near the EDGE of the training data.")

# --- RANDOM VECTORS BECOME ORTHOGONAL ---
print("\nANGLE BETWEEN RANDOM VECTORS")
for d in (2, 3, 10, 100, 1000):
    a, b = rng.normal(size=(500, d)), rng.normal(size=(500, d))
    cos = np.sum(a*b, axis=1) / (np.linalg.norm(a, axis=1)*np.linalg.norm(b, axis=1))
    print(f"  {d:>4} dimensions: mean |cos| {np.abs(cos).mean():.4f}"
          f"  -> mean angle {np.degrees(np.arccos(np.abs(cos).mean())):.1f} degrees")
print("Everything is perpendicular to everything else in high dimensions.")

# --- SPURIOUS CORRELATION: the multiple-comparison face of the curse. ---
print("\nLARGEST |CORRELATION| AMONG PURELY RANDOM FEATURES (n=200 rows)")
for p in (5, 20, 100, 500):
    M = rng.normal(size=(200, p))
    C = np.corrcoef(M, rowvar=False)
    iu = np.triu_indices(p, k=1)
    print(f"  {p:>4} features -> {len(iu[0]):>6,} pairs,"
          f" largest |r| = {np.abs(C[iu]).max():.3f}")
print("With 500 unrelated features you will always find a pair correlated at")
print("~0.35. That is geometry, not a finding.")

# --- WHICH MODELS SUFFER, AND WHICH DO NOT ---
print("\nACCURACY AS NOISE DIMENSIONS ARE ADDED (5 informative features, n=800)")
n = 800
informative = rng.normal(size=(n, 5))
y = (informative @ np.array([1.5, -1.2, 0.9, -0.7, 0.5])
     + rng.normal(0, 0.7, n) > 0).astype(int)
print(f"  {'total dims':>11} {'kNN-15':>9} {'logistic':>9} {'forest':>9}")
for extra in (0, 5, 20, 100, 500):
    X = np.hstack([informative, rng.normal(size=(n, extra))])
    row = f"  {5+extra:>11}"
    for model in (make_pipeline(StandardScaler(), KNeighborsClassifier(15)),
                  make_pipeline(StandardScaler(),
                                LogisticRegression(max_iter=2000, C=0.1)),
                  RandomForestClassifier(n_estimators=150, random_state=0)):
        row += f" {cross_val_score(model, X, y, cv=5).mean():>9.4f}"
    print(row)
print("\nkNN collapses toward chance: its distance metric is diluted by noise")
print("dimensions. The regularised linear model and the forest hold up,")
print("because their biases (sparsity, axis-aligned splits) ignore noise.")

# --- THE MANIFOLD HYPOTHESIS: why the curse is survivable. ---
print("\nDATA ON A LOW-DIMENSIONAL MANIFOLD INSIDE A HIGH-DIMENSIONAL SPACE")
t = rng.uniform(0, 4*np.pi, 900)               # ONE intrinsic dimension
# Embed a 1-D curve into 60 ambient dimensions via a random rotation.
curve = np.column_stack([t, np.sin(t)*3, np.cos(t)*3])
embed = curve @ rng.normal(size=(3, 60))
y_manifold = (np.sin(t) > 0).astype(int)

from sklearn.decomposition import PCA
pca = PCA().fit(embed)
cum = np.cumsum(pca.explained_variance_ratio_)
print(f"  ambient dimensions: {embed.shape[1]}")
print(f"  components for 99% of variance: {int(np.searchsorted(cum, 0.99)) + 1}")
print(f"  INTRINSIC dimensions: 3")
for model, label in [(make_pipeline(StandardScaler(), KNeighborsClassifier(10)),
                      "kNN on 60 raw dims"),
                     (make_pipeline(PCA(n_components=3), StandardScaler(),
                                    KNeighborsClassifier(10)),
                      "kNN on 3 PCA dims")]:
    print(f"  {label:22s} accuracy"
          f" {cross_val_score(model, embed, y_manifold, cv=5).mean():.4f}")
print("\nThe data only LOOKS 60-dimensional. Real high-dimensional data almost")
print("always lies on a lower-dimensional manifold, which is why deep")
print("learning works on images with a million pixels.")

  VOLUME GROWS EXPONENTIALLY

    resolution 0.1 per axis
     1 dim                    10 cells
     2 dim                   100 cells
     3 dim                 1,000 cells
     5 dim               100,000 cells
    10 dim        10,000,000,000 cells   <- 1,000 points fill 1e-7 of them
    20 dim   10^20            cells

  DISTANCE CONCENTRATION (1,000 uniform points)

    dim     min dist   max dist   max/min   relative contrast
    1          0.001      0.938   107.4     0.9957
    2          0.012      1.216    32.8     0.9243
    5          0.174      1.766     8.1     0.6017
    10         0.529      2.204     3.2     0.3510
    50         1.752      3.386     1.8     0.1157
    200        4.132      5.427     1.3     0.0533
    1000       9.673     11.084     1.1     0.0242

  the nearest and the farthest point become the SAME distance away

       low dimension              high dimension
         .  .                        . . . . . .
       .   Q   .                   . . . Q . . .
         .   .                       . . . . . .
      "nearest" is meaningful     everything is equidistant

  VOLUME IS IN THE SHELL

    dim     volume in the outer 10% of the radius
    1        10.0%
    2        19.0%
    3        27.1%
    10       65.1%
    50       99.5%    <- almost every point is near the boundary
    200     100.0%

  WHICH MODELS SURVIVE (5 informative features + noise dimensions)

    total dims    kNN-15   logistic   forest
    5             0.9013     0.8988   0.8925
    10            0.8700     0.8975   0.8888
    25            0.7975     0.8950   0.8850
    105           0.6488     0.8800   0.8675
    505           0.5563     0.8213   0.8388

    kNN collapses toward chance (0.50). The regularised linear model and
    the forest lose little, because their inductive biases ignore noise
    dimensions rather than averaging over them.

  THE MANIFOLD HYPOTHESIS: WHY ANY OF THIS WORKS

    ambient dimensions:                60
    components for 99% of variance:     3
    intrinsic dimensions:               3

    kNN on 60 raw dimensions   accuracy 0.7211
    kNN on 3 PCA dimensions    accuracy 0.9867

    Real data rarely fills its ambient space. Images of faces occupy a
    vanishing fraction of all possible pixel arrays.

Key Takeaways

• Volume grows exponentially with dimension, so any fixed sample becomes sparse and interpolation quietly becomes extrapolation.
• Distances concentrate, so the nearest and farthest neighbours become nearly equidistant, which undermines every distance-based method.
• Almost all volume sits near the boundary in high dimensions, so most predictions are made at the edge of the training distribution.
• Tree ensembles and regularised linear models cope far better than nearest- neighbour methods because their biases ignore noise dimensions.
• Real high-dimensional data usually lies on a much lower-dimensional manifold, which is why high-dimensional learning works at all.

🧪 Practice

1. Compute the ratio of farthest to nearest distance among random points at dimensions 2, 10, and 1,000, and describe the trend.
2. Calculate the fraction of a ball's volume inside the outer 10% of its radius at several dimensions.
3. Add increasing numbers of pure-noise features to a dataset and compare the accuracy decline of nearest-neighbours, a regularised linear model, and a forest.
4. Embed a one-dimensional curve in 50 dimensions and show that reducing to the intrinsic dimension restores nearest-neighbour accuracy.
5. Report the largest absolute correlation among 500 unrelated random features and explain why it is not a finding.
6. Interview: A team's nearest-neighbour recommender worked well on 20 features and degraded badly after they added 300 more. Explain what happened. (Hint: what happens to the distance between any two points as noise dimensions accumulate?)

<a id="93-model-evaluation"></a>

9.3 Model Evaluation

A number you cannot interpret is worse than no number, because it will be acted on; this section covers how to estimate performance reliably and which metric actually answers the question you have.

Cross-Validation Strategies

A single train-test split gives one estimate of performance, and that estimate is noisy. With a thousand test rows, an accuracy of 0.85 carries a confidence interval of roughly plus or minus 0.022, and the particular rows that landed in the test set matter. Cross-validation reduces that noise by splitting repeatedly, so every row serves as test data exactly once, and the estimate averages over the split.

The standard form is k-fold cross-validation: divide the data into k parts, train on k-1 and test on the remaining one, rotate, and average. It gives a lower variance estimate than a single split and it also gives you the spread across folds, which is itself informative because fold-to-fold disagreement measures the model's variance.

The choice of k is a bias-variance trade-off about the estimate rather than the model. A small k means each model trains on less data, so the estimate is pessimistic about the model you would actually deploy on all of it. A large k means each model trains on nearly everything, so bias is low, but the k models are highly correlated with each other and the estimate's variance does not fall as fast as you would hope. Five or ten is the usual compromise, and ten is the common default.

Leave-one-out is the extreme where k equals n. It is nearly unbiased, expensive, and has a well-known problem: the n models are so similar that the estimate has high variance, so it is often worse in practice than ten-fold despite being less biased.

The critical part is matching the splitter to the data's structure, which extends the split discipline from 9.1 and connects to the temporal and panel material in chapter 1.3:

Splitter	Use when	Prevents
KFold	rows are independent	nothing special
StratifiedKFold	classification, especially imbalanced	folds with no positives
GroupKFold	entities contribute several rows	entity memorisation
StratifiedGroupKFold	both of the above	both
TimeSeriesSplit	order matters	training on the future
Purged / embargoed CV	overlapping label windows	label leakage across the boundary
RepeatedKFold	small datasets	a noisy single-shuffle estimate

Three points that matter operationally.

Everything fitted goes inside the loop. Imputation, scaling, encoding, feature selection, and resampling are all fitted quantities, and fitting them before cross-validation leaks the held-out fold into training. Chapter 6.3 demonstrates this producing high accuracy on pure noise. Use a pipeline and the problem disappears mechanically.

For time series, the standard version has a subtlety. A forward-chaining split respects order, but if your label spans a window, such as "churn within 30 days", then a training row near the boundary has a label that depends on the test period. The fix is purging, removing rows whose label window overlaps the test set, plus an embargo gap.

Report the spread, not just the mean. A mean of 0.85 across folds ranging from 0.71 to 0.94 is a very different result from one ranging 0.84 to 0.86, and only the first tells you the model is unstable.

import numpy as np
import pandas as pd
from sklearn.model_selection import (KFold, StratifiedKFold, GroupKFold,
                                     TimeSeriesSplit, RepeatedStratifiedKFold,
                                     cross_val_score, train_test_split,
                                     LeaveOneOut)
from sklearn.linear_model import LogisticRegression, Ridge
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.datasets import make_classification
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(41)

# --- WHY ONE SPLIT IS NOT ENOUGH ---
X, y = make_classification(n_samples=1_500, n_features=15, n_informative=6,
                           random_state=0)
single = []
for seed in range(40):
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25,
                                          random_state=seed, stratify=y)
    single.append(LogisticRegression(max_iter=1000).fit(Xtr, ytr).score(Xte, yte))
print("SINGLE-SPLIT ESTIMATES ACROSS 40 RANDOM SEEDS")
print(f"  min {min(single):.4f}  max {max(single):.4f}"
      f"  spread {max(single)-min(single):.4f}  sd {np.std(single):.4f}")
cv10 = cross_val_score(LogisticRegression(max_iter=1000), X, y, cv=10)
print(f"  10-fold CV: mean {cv10.mean():.4f}  sd of the MEAN"
      f" {cv10.std()/np.sqrt(10):.4f}")
print("\nA single split's answer moves by 5 points depending on the seed. The")
print("cross-validated mean is far more stable, and you get the spread free.")

# --- CHOOSING k: a trade-off about the ESTIMATE, not the model. ---
print("\nEFFECT OF k (n=1,500)")
print(f"  {'k':>5} {'train rows/fold':>16} {'mean':>8} {'sd across folds':>17}"
      f" {'fits':>6}")
for k in (2, 3, 5, 10, 20):
    s = cross_val_score(LogisticRegression(max_iter=1000), X, y, cv=k)
    print(f"  {k:>5} {int(len(X)*(k-1)/k):>16,} {s.mean():>8.4f}"
          f" {s.std():>17.4f} {k:>6}")
loo_subset = cross_val_score(LogisticRegression(max_iter=1000), X[:300], y[:300],
                             cv=LeaveOneOut())
print(f"  LOO (on 300 rows only): mean {loo_subset.mean():.4f},"
      f" {len(loo_subset)} fits")
print("\nSmall k trains on less data, so the estimate is pessimistic about the")
print("model you will actually deploy. Large k costs k fits for a shrinking")
print("benefit, because the k models become nearly identical.")

# --- STRATIFICATION: mandatory when the target is imbalanced. ---
Xi, yi = make_classification(n_samples=1_000, weights=[0.97, 0.03],
                             n_features=10, n_informative=4, random_state=0)
print(f"\nIMBALANCED TARGET: {yi.sum()} positives in {len(yi)}")
for label, splitter in [("KFold", KFold(5, shuffle=True, random_state=3)),
                        ("StratifiedKFold",
                         StratifiedKFold(5, shuffle=True, random_state=3))]:
    counts = [int(yi[te].sum()) for _, te in splitter.split(Xi, yi)]
    print(f"  {label:18s} positives per test fold: {counts}"
          f"  {'<- a fold with 0 makes AUC undefined' if 0 in counts else ''}")

# --- GROUPED DATA: the same entity must not straddle a fold boundary. ---
n = 2_400
groups = rng.integers(0, 120, n)                  # 120 entities
entity_effect = rng.normal(0, 2.0, 120)[groups]
Xg = np.column_stack([rng.normal(size=n) * 0.5 + entity_effect * 0.15,
                      rng.normal(size=(n, 3))])
yg = (rng.random(n) < 1/(1+np.exp(-entity_effect))).astype(int)

print("\nPANEL DATA: 2,400 rows from 120 entities")
for label, cv, kw in [("KFold (WRONG)", KFold(5, shuffle=True, random_state=0), {}),
                      ("GroupKFold (right)", GroupKFold(5), {"groups": groups})]:
    s = cross_val_score(RandomForestClassifier(n_estimators=80, random_state=0),
                        Xg, yg, cv=cv, scoring="roc_auc", **kw)
    print(f"  {label:20s} AUC {s.mean():.4f} +/- {s.std():.4f}")
print("The gap is entity memorisation. The grouped number is the one that")
print("predicts performance on entities you have never seen.")

# --- TIME SERIES: forward chaining, and why folds grow. ---
m = 1_000
t = np.arange(m)
Xt = np.column_stack([rng.normal(size=m), np.sin(t/50)])
logit_t = Xt[:, 0] * (1 + t/m) + Xt[:, 1] - t/m*1.5
yt = (logit_t + rng.normal(0, .5, m) > 0).astype(int)
tscv = TimeSeriesSplit(n_splits=5)
print("\nTIME SERIES SPLIT (forward chaining)")
print(f"  {'fold':>5} {'train rows':>11} {'test rows':>10} {'train range':>16}"
      f" {'test range':>14}")
for i, (tr, te) in enumerate(tscv.split(Xt)):
    print(f"  {i:>5} {len(tr):>11} {len(te):>10}"
          f" {f'{tr[0]}-{tr[-1]}':>16} {f'{te[0]}-{te[-1]}':>14}")
print("Every test fold lies strictly AFTER its training data, and the training")
print("window grows. Compare against a shuffled split on the same series:")
lr = LogisticRegression(max_iter=1000)
shuffled = KFold(5, shuffle=True, random_state=0)
print(f"  shuffled KFold AUC   "
      f"{cross_val_score(lr, Xt, yt, cv=shuffled, scoring='roc_auc').mean():.4f}")
print(f"  TimeSeriesSplit AUC  "
      f"{cross_val_score(lr, Xt, yt, cv=tscv, scoring='roc_auc').mean():.4f}")

# --- PURGED CV: needed when the LABEL spans a window. ---
def purged_time_split(n, n_splits=5, label_window=40, embargo=10):
    """Forward chaining, but drop training rows whose LABEL WINDOW overlaps
    the test period, plus an embargo gap. Without this, a training row's
    label is partly determined by data inside the test set."""
    fold = n // (n_splits + 1)
    for i in range(n_splits):
        test_start = fold * (i + 1)
        test_end = test_start + fold
        # A training row at index j has a label observed up to j+label_window.
        train_end = max(test_start - label_window - embargo, 1)
        yield np.arange(0, train_end), np.arange(test_start, min(test_end, n))

print("\nPURGED SPLIT (label window 40 rows, embargo 10)")
print(f"  {'fold':>5} {'train':>14} {'GAP':>6} {'test':>14}")
for i, (tr, te) in enumerate(purged_time_split(1_000)):
    gap = te[0] - tr[-1] - 1
    print(f"  {i:>5} {f'0-{tr[-1]}':>14} {gap:>6}"
          f" {f'{te[0]}-{te[-1]}':>14}")
print("The gap removes rows whose outcome is still unfolding when the test")
print("period begins. Standard TimeSeriesSplit leaves that leak in place.")

# --- EVERYTHING FITTED MUST GO INSIDE THE LOOP ---
noise = rng.normal(size=(200, 3_000))              # NOTHING is predictive
rand_y = rng.integers(0, 2, 200)
sel = SelectKBest(f_classif, k=20).fit(noise, rand_y)     # sees ALL labels
leaky = cross_val_score(LogisticRegression(max_iter=1000),
                        noise[:, sel.get_support()], rand_y, cv=5).mean()
honest = cross_val_score(
    make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression(max_iter=1000)),
    noise, rand_y, cv=5).mean()
print(f"\nPURE NOISE, true accuracy 0.500")
print(f"  selection BEFORE cross-validation: {leaky:.4f}   <- leakage")
print(f"  selection INSIDE the pipeline:     {honest:.4f}   <- honest")
print("Any object with a .fit() belongs inside the pipeline. This is the same")
print("rule as chapter 6.3, and cross-validation is where it bites hardest.")

# --- REPEATED CV for small data, and the standard error of the estimate. ---
small_X, small_y = X[:300], y[:300]
print("\nSMALL DATA: a single 5-fold estimate depends on the shuffle")
means = []
for seed in range(8):
    s = cross_val_score(LogisticRegression(max_iter=1000), small_X, small_y,
                        cv=StratifiedKFold(5, shuffle=True, random_state=seed))
    means.append(s.mean())
print(f"  5-fold mean across 8 shuffles: min {min(means):.4f},"
      f" max {max(means):.4f}, spread {max(means)-min(means):.4f}")
rep = cross_val_score(LogisticRegression(max_iter=1000), small_X, small_y,
                      cv=RepeatedStratifiedKFold(n_splits=5, n_repeats=10,
                                                 random_state=0))
print(f"  repeated 5-fold x10: mean {rep.mean():.4f}"
      f"  sd {rep.std():.4f}  se of the mean {rep.std()/np.sqrt(len(rep)):.4f}")
print("Repeating averages over the shuffle, which is worth doing whenever the")
print("dataset is small enough for the shuffle to matter.")

# --- ALWAYS REPORT THE SPREAD ---
print("\nTWO MODELS WITH THE SAME MEAN, VERY DIFFERENT RELIABILITY")
for label, model in [("regularised linear",
                      make_pipeline(StandardScaler(),
                                    LogisticRegression(max_iter=1000, C=0.05))),
                     ("unrestricted tree",
                      RandomForestClassifier(n_estimators=3, max_depth=None,
                                             random_state=0))]:
    s = cross_val_score(model, X, y, cv=10)
    print(f"  {label:20s} mean {s.mean():.4f}  sd {s.std():.4f}"
          f"  range [{s.min():.4f}, {s.max():.4f}]")
print("Report the range. A model whose folds disagree is telling you it has")
print("high variance, which is information the mean alone hides.")

  WHY CROSS-VALIDATE

    single 25% split, 40 different seeds:
      min 0.8347   max 0.8853   spread 0.0506   sd 0.0131

    10-fold cross-validation:
      mean 0.8613   standard error of the mean 0.0041

    The single split's answer moves by 5 accuracy points on the seed alone.

  K-FOLD, DRAWN (k=5)

    fold 1  [TEST][----------- train -----------]
    fold 2  [train][TEST][------- train --------]
    fold 3  [----- train ----][TEST][-- train --]
    fold 4  [-------- train -------][TEST][train]
    fold 5  [----------- train ----------][TEST]

    every row is tested exactly once; average the five scores

  CHOOSING k

    k     train rows/fold   estimate bias   estimate variance   cost
    2     750               pessimistic     low                 2 fits
    5     1,200             slight          low                 5 fits
    10    1,350             small           low                 10 fits
    20    1,425             smallest        NOT lower           20 fits
    LOO   n-1               ~zero           HIGH                n fits

    5 or 10 is the compromise. LOO's models are so similar that averaging
    them barely reduces variance, despite the low bias.

  MATCH THE SPLITTER TO THE DATA

  structure                    splitter                 without it
  ---------------------------------------------------------------------------
  independent rows             KFold                    -
  imbalanced classification    StratifiedKFold          folds with 0 positives
  repeated entities            GroupKFold               entity memorisation
  both                         StratifiedGroupKFold     both
  time-ordered                 TimeSeriesSplit          training on the future
  labels span a window         PURGED + embargo         label leaks the test set
  small n                      RepeatedKFold            shuffle-dependent answer

  MEASURED GAPS

    panel data:    KFold AUC 0.921  vs  GroupKFold AUC 0.702
    time series:   shuffled 0.847   vs  TimeSeriesSplit 0.781

  PURGING, DRAWN (label window 40, embargo 10)

    [---------- train ----------]<-gap->[--- test ---]
                              ^ 50 rows dropped
    A training row at the boundary has a label that depends on data inside
    the test period. Standard forward chaining leaves that leak in place.

Key Takeaways

• A single split's estimate moves by several points on the random seed alone, so cross-validate and report the fold spread.
• Choosing k trades bias against variance in the estimate; five or ten is the compromise and leave-one-out is expensive with high-variance results.
• Stratify for imbalanced classification, group for repeated entities, and chain forward for time-ordered data.
• When labels span a forward window, purge and embargo the training rows near the test boundary.
• Every fitted transform belongs inside the pipeline, or cross-validation reports skill that does not exist.

🧪 Practice

1. Run a single train-test split across 40 seeds and compare the spread against a ten-fold cross-validated mean.
2. Cross-validate an imbalanced dataset with and without stratification and report the positive count in each test fold.
3. Build panel data with a strong per-entity effect and compare standard k-fold against grouped k-fold.
4. Implement a purged, embargoed time-series splitter and print the gap between each training and test window.
5. Reproduce the noise experiment where feature selection outside the loop reports accuracy well above chance.
6. Interview: A colleague reports 0.91 cross-validated accuracy but the model scores 0.73 in production on new customers. Name two splitting mistakes that would produce exactly that gap. (Hint: what do the production rows have in common that the validation rows did not?)

Regression Metrics

Regression metrics all summarise the residuals, the differences between predictions and actuals, and they differ in how they weight large errors and whether they are interpretable in the target's own units. Choosing among them is not cosmetic: a model tuned to minimise one can be visibly worse under another.

The core set:

Mean squared error averages squared residuals. It punishes large errors disproportionately and is in squared units, which makes it uninterpretable for stakeholders. Root mean squared error takes the square root, restoring the original units while keeping the disproportionate penalty. It is the default for most work and is minimised by predicting the conditional mean.

Mean absolute error averages absolute residuals. It treats all errors proportionally, is robust to outliers, is directly interpretable, and is minimised by the conditional median. The relationship between root mean squared error and mean absolute error is diagnostic: they are equal when all errors are the same size, and the ratio grows with the spread of errors. A ratio above about 1.5 means a few large errors dominate, which is worth investigating.

R-squared reports the fraction of variance explained relative to predicting the mean. Its appeal is being unitless and comparable, and it has two traps. It can be negative, which means the model is worse than the mean, and it depends on the variance of your particular test set, so it is not comparable across datasets. A model with an R-squared of 0.9 on a high-variance sample may have larger absolute errors than one at 0.5 on a narrow sample.

Mean absolute percentage error expresses error as a fraction of the actual value, which is attractive for business reporting and has serious defects. It is undefined when an actual is zero, explodes when actuals are near zero, and is asymmetric: it penalises over-prediction more heavily than under-prediction, which biases any model tuned on it toward under-forecasting. Symmetric variants partly address this and introduce their own oddities.

For counts and skewed targets, mean squared logarithmic error penalises relative rather than absolute error and is asymmetric in a way that under-predicts less harshly. For forecasting, mean absolute scaled error divides by the error of a naive baseline, which makes it comparable across series with different scales.

The reporting habit worth adopting: give at least one absolute metric in the target's units, one relative metric, and a residual diagnostic. A single number cannot tell you whether the errors are structured, and structure in residuals is the most actionable finding available, as chapter 7.3 covers.

import numpy as np
import pandas as pd
from sklearn.metrics import (mean_squared_error, mean_absolute_error, r2_score,
                             mean_absolute_percentage_error,
                             median_absolute_error, mean_squared_log_error)
from sklearn.linear_model import LinearRegression, QuantileRegressor
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(43)

# --- THE SAME MODEL, EVERY METRIC ---
n = 2_000
X = rng.uniform(1, 100, (n, 3))
y = 20 + 2.5*X[:, 0] + 1.2*X[:, 1] + rng.normal(0, 12, n)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
pred = LinearRegression().fit(Xtr, ytr).predict(Xte)

def regression_report(y_true, y_pred, label=""):
    resid = y_true - y_pred
    rmse = np.sqrt(mean_squared_error(y_true, y_pred))
    mae = mean_absolute_error(y_true, y_pred)
    return {
        "label": label,
        "RMSE": round(rmse, 3),
        "MAE": round(mae, 3),
        "median_AE": round(median_absolute_error(y_true, y_pred), 3),
        # A ratio near 1 means uniform errors; large means a few big misses.
        "RMSE/MAE": round(rmse / mae, 3),
        "R2": round(r2_score(y_true, y_pred), 4),
        "MAPE": round(mean_absolute_percentage_error(y_true, y_pred), 4),
        "bias": round(resid.mean(), 3),      # should be ~0 for a fitted model
    }

print("ONE MODEL, THE FULL REPORT")
for k, v in regression_report(yte, pred, "linear").items():
    print(f"  {k:12s} {v}")
print("\nRMSE and MAE are in the target's units. R-squared is unitless. MAPE is")
print("a percentage. Reporting only one hides a different failure mode.")

# --- RMSE VS MAE: the ratio is a diagnostic for error concentration. ---
print("\nRMSE/MAE RATIO ACROSS ERROR SHAPES (all with MAE ~ 4.0)")
shapes = {
    "all errors equal": np.full(1000, 4.0),
    "moderate spread": rng.normal(0, 5.0, 1000),
    "a few large misses": np.concatenate([rng.normal(0, 1.2, 970),
                                          rng.normal(0, 60, 30)]),
    "one catastrophic miss": np.append(rng.normal(0, 3.4, 999), [400.0]),
}
for name, errs in shapes.items():
    rmse = np.sqrt((errs ** 2).mean())
    mae = np.abs(errs).mean()
    print(f"  {name:24s} RMSE {rmse:8.2f}  MAE {mae:6.2f}"
          f"  ratio {rmse/mae:6.2f}")
print("\nA ratio above ~1.5 says a minority of rows drives the error. That is a")
print("prompt to look at those rows, not to pick a different metric.")

# --- SENSITIVITY TO ONE OUTLIER ---
y_out = yte.copy()
p_out = pred.copy()
y_out[0] += 800                                  # one absurd actual
print("\nEFFECT OF ONE BAD ROW IN 600")
before = regression_report(yte, pred)
after = regression_report(y_out, p_out)
for m in ("RMSE", "MAE", "median_AE", "R2"):
    print(f"  {m:12s} {before[m]:>10} -> {after[m]:>10}"
          f"   change {100*(after[m]-before[m])/abs(before[m]):+7.1f}%")
print("RMSE moves far more than MAE, and the median absolute error barely")
print("notices. Choose based on whether large errors are genuinely worse.")

# --- R-SQUARED IS NOT COMPARABLE ACROSS DATASETS ---
print("\nR-SQUARED DEPENDS ON THE TEST SET'S VARIANCE")
for label, spread in [("narrow range of y", 5.0), ("wide range of y", 60.0)]:
    Xs = rng.uniform(50 - spread, 50 + spread, (800, 1))
    ys = 10 + 2.0 * Xs.ravel() + rng.normal(0, 8, 800)
    a, b, ya, yb = train_test_split(Xs, ys, test_size=0.3, random_state=0)
    p = LinearRegression().fit(a, ya).predict(b)
    print(f"  {label:20s} R2 {r2_score(yb, p):6.3f}"
          f"   RMSE {np.sqrt(mean_squared_error(yb, p)):6.2f}"
          f"   sd(y) {yb.std():6.2f}")
print("\nIdentical noise and the SAME absolute error, and R-squared differs")
print("hugely. R-squared measures error relative to the spread of the sample,")
print("so it cannot compare models evaluated on different data.")

# R-squared can be negative: worse than predicting the mean.
bad_pred = np.full_like(yte, yte.mean() + 40)
print(f"\na constant prediction offset by 40: R2 {r2_score(yte, bad_pred):.3f}"
      "  <- negative means worse than the mean")

# --- MAPE IS ASYMMETRIC, and that asymmetry biases models. ---
print("\nMAPE PENALISES OVER- AND UNDER-PREDICTION UNEQUALLY")
actual = 100.0
print(f"  {'prediction':>11} {'error':>8} {'MAPE':>8}")
for p in (50, 80, 100, 120, 150, 200):
    print(f"  {p:>11} {p-actual:>8.0f} {abs(p-actual)/actual:>8.2%}")
print("  A prediction of 50 and one of 150 are both 50 off; MAPE agrees here.")
print("  But the WORST possible under-prediction is bounded at 100%, while")
print("  over-prediction is unbounded:")
for p in (0, 1, 300, 1000):
    print(f"  {p:>11} {p-actual:>8.0f} {abs(p-actual)/actual:>8.2%}")
print("  So a model tuned on MAPE learns that under-forecasting is SAFER.")

# The bias is measurable: fit a quantile model and compare MAPE.
Xm = rng.uniform(1, 50, (1500, 1))
ym = 5 + 2.0 * Xm.ravel() + rng.normal(0, 3, 1500)
ym = np.maximum(ym, 0.5)
mXtr, mXte, myTr, myTe = train_test_split(Xm, ym, test_size=0.3, random_state=0)
print("\n  which model wins under each metric?")
print(f"  {'model':<26} {'RMSE':>8} {'MAE':>8} {'MAPE':>8}")
for label, model in [("least squares (mean)", LinearRegression()),
                     ("quantile q=0.4 (low)",
                      QuantileRegressor(quantile=0.4, alpha=0.0,
                                        solver="highs")),
                     ("quantile q=0.5 (median)",
                      QuantileRegressor(quantile=0.5, alpha=0.0,
                                        solver="highs"))]:
    p = model.fit(mXtr, myTr).predict(mXte)
    print(f"  {label:<26} {np.sqrt(mean_squared_error(myTe, p)):>8.3f}"
          f" {mean_absolute_error(myTe, p):>8.3f}"
          f" {mean_absolute_percentage_error(myTe, p):>8.4f}")
print("  The deliberately LOW predictor wins on MAPE. If you optimise MAPE,")
print("  you have asked for an under-forecast.")

# --- SKEWED TARGETS: relative error metrics. ---
skew_y = rng.lognormal(3.0, 1.1, 1500)
skew_X = (np.log(skew_y) + rng.normal(0, 0.3, 1500)).reshape(-1, 1)
sXtr, sXte, syTr, syTe = train_test_split(skew_X, skew_y, test_size=0.3,
                                          random_state=0)
print("\nSKEWED TARGET (lognormal): raw vs log-space training")
for label, transform in [("fit on raw y", False), ("fit on log(y)", True)]:
    if transform:
        m = GradientBoostingRegressor(random_state=0).fit(sXtr, np.log1p(syTr))
        p = np.expm1(m.predict(sXte))
    else:
        m = GradientBoostingRegressor(random_state=0).fit(sXtr, syTr)
        p = m.predict(sXte)
    p = np.maximum(p, 1e-6)
    print(f"  {label:14s} RMSE {np.sqrt(mean_squared_error(syTe, p)):9.2f}"
          f"  MAE {mean_absolute_error(syTe, p):8.2f}"
          f"  MAPE {mean_absolute_percentage_error(syTe, p):7.3f}"
          f"  MSLE {mean_squared_log_error(syTe, p):7.4f}")
print("Fitting on the log scale wins on the relative metrics and loses on")
print("RMSE, because RMSE is dominated by the largest values. Both are right;")
print("they answer different questions.")

# --- SCALED ERROR: comparable across series. ---
def mase(y_true, y_pred, y_train, seasonality=1):
    """Mean absolute scaled error: the model's MAE divided by a naive
    forecast's MAE on the training data. Below 1 beats the naive baseline."""
    naive_mae = np.mean(np.abs(np.diff(y_train, n=seasonality)))
    return mean_absolute_error(y_true, y_pred) / naive_mae

series_a = np.cumsum(rng.normal(0, 1, 400)) + 100          # small scale
series_b = np.cumsum(rng.normal(0, 50, 400)) + 5_000       # large scale
print("\nMASE MAKES SERIES WITH DIFFERENT SCALES COMPARABLE")
for label, s in [("small-scale series", series_a), ("large-scale series", series_b)]:
    train, test = s[:350], s[350:]
    naive = np.full(len(test), train[-1])                  # last value carried
    print(f"  {label:20s} MAE {mean_absolute_error(test, naive):9.2f}"
          f"   MASE {mase(test, naive, train):6.3f}")
print("The MAEs differ by 50x and the MASEs are comparable, which is why")
print("forecasting competitions use scaled errors.")

  THE CORE METRICS

  metric   units       weights large errors   robust   minimised by
  ---------------------------------------------------------------------------
  MSE      squared     heavily (quadratic)    no       conditional MEAN
  RMSE     original    heavily                no       conditional MEAN
  MAE      original    proportionally         yes      conditional MEDIAN
  median AE original   barely at all          very     conditional median
  R^2      none        heavily                no       conditional mean
  MAPE     percent     proportionally, but    no       a LOW forecast
                       ASYMMETRICALLY
  MSLE     log units   relative error         partly   geometric-ish mean
  MASE     none        proportionally         yes      (comparable across
                                                        series)

  RMSE/MAE IS A DIAGNOSTIC

    error shape              RMSE     MAE    ratio
    all errors equal          4.00    4.00    1.00
    moderate spread           5.01    4.00    1.25
    a few large misses       10.68    3.98    2.68   <- investigate those rows
    one catastrophic miss    13.99    3.75    3.73   <- find that one row

  ONE BAD ROW IN 600

    metric        before      after    change
    RMSE          12.046     33.482   +177.9%
    MAE            9.611     10.941    +13.8%
    median AE      8.120      8.121     +0.0%
    R^2            0.9681     0.7542   -22.1%

  R-SQUARED IS NOT COMPARABLE ACROSS DATASETS

    narrow range of y   R2 0.372   RMSE 8.11   sd(y) 10.2
    wide range of y     R2 0.981   RMSE 8.06   sd(y) 69.4

    Identical absolute error. R-squared differs by 0.6 because it measures
    error RELATIVE TO the sample's spread.

  WHY MAPE BIASES TOWARD UNDER-FORECASTING

    actual = 100

    prediction     error    MAPE
    0              -100     100%     <- the WORST under-prediction is 100%
    50              -50      50%
    100               0       0%
    150             +50      50%
    300            +200     200%     <- over-prediction is UNBOUNDED
    1000           +900     900%

    A model tuned on MAPE learns that guessing low is safer. In the worked
    example the q=0.4 quantile model wins on MAPE and loses on RMSE.

  WHAT TO REPORT

    [ ] one ABSOLUTE metric in the target's units (RMSE or MAE)
    [ ] one RELATIVE metric (R^2, or MASE for forecasts)
    [ ] the RMSE/MAE ratio, to reveal error concentration
    [ ] mean residual, to reveal systematic bias
    [ ] a residual plot (chapter 7.3), to reveal structure

Key Takeaways

• Root mean squared error weights large errors heavily and targets the conditional mean; mean absolute error is robust and targets the median.
• The ratio of root mean squared error to mean absolute error diagnoses whether a few rows dominate the total error.
• R-squared depends on the test set's variance, so it cannot compare models evaluated on different data, and it can be negative.
• Mean absolute percentage error is asymmetric and unbounded above, so optimising it biases a model toward under-forecasting.
• Report an absolute metric, a relative metric, the mean residual, and a residual plot, because no single number reveals structure.

🧪 Practice

1. Compute the full metric report for one model and explain what each number adds that the others do not.
2. Construct four error distributions with the same mean absolute error and compare their root mean squared error ratios.
3. Add one extreme outlier and report the percentage change in root mean squared error, mean absolute error, and median absolute error.
4. Show that two models with identical absolute errors have very different R-squared values on test sets of different spread.
5. Fit a deliberately low-biased quantile model and show it wins on mean absolute percentage error.
6. Interview: Your demand forecast is evaluated on mean absolute percentage error and the business complains you consistently under-order. What is the likely cause? (Hint: what does the metric do to a prediction that is too high versus too low?)

Classification Metrics

Classification metrics are where most evaluation mistakes happen, because the default metric is accuracy and accuracy is misleading in exactly the situations that matter most. Getting this right requires understanding that a classifier produces a score, and that turning it into a decision requires a threshold, so most metrics are really properties of a score-plus-threshold pair.

Start with what goes wrong. Accuracy is the fraction of correct predictions, and under class imbalance it is dominated by the majority class. A model predicting "not fraud" for every transaction in a dataset with 0.5% fraud is 99.5% accurate and useless. This is covered in chapter 6.4 from the imbalance angle; here the point is that accuracy conflates two different error types that usually have very different costs.

The metrics that survive are all built from the four cells of the confusion matrix.

Precision is the fraction of positive predictions that were correct. It answers "when the model raises an alarm, how often is it right", which maps directly onto wasted reviewer effort.

Recall, also called sensitivity or the true positive rate, is the fraction of actual positives found. It answers "how much of the thing we care about did we catch", which maps onto missed cases.

These trade off. Lowering the threshold catches more positives and raises more false alarms. There is no setting that maximises both, and the choice between them is a business decision about relative costs, as chapter 8.4 covers.

F1 is the harmonic mean of precision and recall, useful as a single number and opaque about which side it came from. The harmonic mean is used rather than the arithmetic mean because it punishes imbalance: a model with precision 1.0 and recall 0.01 has an arithmetic mean of 0.505 and an F1 of 0.02, and the second number is the honest one. F-beta generalises it, weighting recall beta times as much as precision, which is how you encode an asymmetric preference without full cost numbers.

Specificity is the true negative rate, and balanced accuracy averages recall and specificity, which makes it robust to imbalance in a way plain accuracy is not.

Matthews correlation coefficient is the most underrated metric here. It uses all four confusion matrix cells, is symmetric in the classes, and is high only when the model does well on both. Under imbalance it is the most reliable single summary.

Cohen's kappa compares observed agreement against chance agreement, which is useful when the baseline is unclear.

Two structural points to keep straight.

Threshold-dependent versus threshold-free. Precision, recall, F1, and accuracy all depend on a threshold. Area under the receiver operating characteristic curve and average precision summarise performance across all thresholds, which makes them suitable for comparing models before an operating point is chosen. The next topic covers those curves.

Multi-class averaging. With more than two classes, precision and recall must be averaged, and the choice matters enormously. Macro averaging treats every class equally, so rare classes count as much as common ones. Micro averaging pools all predictions, so it is dominated by frequent classes and equals accuracy for single-label problems. Weighted averaging weights by class frequency. Reporting "F1 of 0.82" without saying which average is not a result.

import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.dummy import DummyClassifier
from sklearn.metrics import (accuracy_score, balanced_accuracy_score,
                             precision_score, recall_score, f1_score,
                             fbeta_score, matthews_corrcoef, cohen_kappa_score,
                             confusion_matrix, classification_report,
                             precision_recall_fscore_support)

rng = np.random.default_rng(47)

# --- WHY ACCURACY FAILS ---
X, y = make_classification(n_samples=20_000, n_features=18, n_informative=7,
                           weights=[0.99, 0.01], flip_y=0.01, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                      stratify=y)
print(f"positives: {yte.sum()} of {len(yte)} ({yte.mean():.3%})")

model = LogisticRegression(max_iter=2000).fit(Xtr, ytr)
proba = model.predict_proba(Xte)[:, 1]

def metric_row(y_true, y_pred, label):
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
    return {
        "model": label,
        "accuracy": round(accuracy_score(y_true, y_pred), 4),
        "balanced_acc": round(balanced_accuracy_score(y_true, y_pred), 4),
        "precision": round(precision_score(y_true, y_pred, zero_division=0), 4),
        "recall": round(recall_score(y_true, y_pred, zero_division=0), 4),
        "F1": round(f1_score(y_true, y_pred, zero_division=0), 4),
        "MCC": round(matthews_corrcoef(y_true, y_pred), 4),
        "kappa": round(cohen_kappa_score(y_true, y_pred), 4),
        "TP": tp, "FP": fp, "FN": fn,
    }

rows = [
    metric_row(yte, DummyClassifier(strategy="most_frequent").fit(Xtr, ytr)
               .predict(Xte), "always negative"),
    metric_row(yte, (proba >= 0.5).astype(int), "logistic @ 0.5"),
    metric_row(yte, (proba >= 0.05).astype(int), "logistic @ 0.05"),
    metric_row(yte, (proba >= yte.mean()).astype(int), "logistic @ base rate"),
]
print("\n", pd.DataFrame(rows).to_string(index=False))
print("\nThe useless 'always negative' model has the HIGHEST accuracy. Balanced")
print("accuracy, F1, and the Matthews coefficient all correctly give it zero.")

# --- WHY THE HARMONIC MEAN: F1 punishes imbalance between P and R. ---
print("\nARITHMETIC VS HARMONIC MEAN OF PRECISION AND RECALL")
print(f"  {'precision':>10} {'recall':>8} {'arithmetic':>11} {'F1 (harmonic)':>14}")
for p, r in [(0.9, 0.9), (0.95, 0.5), (1.0, 0.10), (1.0, 0.01), (0.5, 0.5)]:
    arith = (p + r) / 2
    harm = 2 * p * r / (p + r) if (p + r) else 0
    print(f"  {p:>10.2f} {r:>8.2f} {arith:>11.3f} {harm:>14.3f}")
print("\nA model with perfect precision and 1% recall gets 0.505 from the")
print("arithmetic mean and 0.02 from F1. The harmonic mean is the honest one,")
print("because it refuses to average away a near-zero component.")

# --- F-BETA: encoding an asymmetric preference. ---
print("\nF-BETA AT DIFFERENT THRESHOLDS (beta > 1 favours recall)")
print(f"  {'threshold':>10} {'precision':>10} {'recall':>8} {'F0.5':>7}"
      f" {'F1':>7} {'F2':>7}")
for t in (0.5, 0.2, 0.1, 0.05, 0.02):
    p = (proba >= t).astype(int)
    print(f"  {t:>10.2f} {precision_score(yte, p, zero_division=0):>10.4f}"
          f" {recall_score(yte, p, zero_division=0):>8.4f}"
          f" {fbeta_score(yte, p, beta=0.5, zero_division=0):>7.4f}"
          f" {f1_score(yte, p, zero_division=0):>7.4f}"
          f" {fbeta_score(yte, p, beta=2, zero_division=0):>7.4f}")
print("F0.5 peaks at a high threshold (precision matters), F2 at a low one")
print("(recall matters). The beta you choose IS a statement about costs.")

# --- THE PRECISION-RECALL TRADE-OFF IS UNAVOIDABLE ---
print("\nTHERE IS NO THRESHOLD THAT MAXIMISES BOTH")
grid = np.quantile(proba, np.linspace(0.5, 0.9995, 12))
for t in grid[::3]:
    p = (proba >= t).astype(int)
    pr = precision_score(yte, p, zero_division=0)
    rc = recall_score(yte, p, zero_division=0)
    bar_p = "#" * int(pr * 30)
    bar_r = "#" * int(rc * 30)
    print(f"  t={t:.4f}  P {pr:.3f} |{bar_p:<30}|")
    print(f"  {'':<12}  R {rc:.3f} |{bar_r:<30}|")

# --- MULTI-CLASS: the averaging choice changes the answer. ---
Xm, ym = make_classification(n_samples=6_000, n_features=15, n_informative=8,
                             n_classes=4, n_clusters_per_class=1,
                             weights=[0.70, 0.20, 0.07, 0.03], random_state=0)
mXtr, mXte, myTr, myTe = train_test_split(Xm, ym, test_size=0.3, random_state=0,
                                          stratify=ym)
mmodel = LogisticRegression(max_iter=3000).fit(mXtr, myTr)
mpred = mmodel.predict(mXte)

print("\nMULTI-CLASS: class distribution in the test set")
for c, n_c in zip(*np.unique(myTe, return_counts=True)):
    print(f"  class {c}: {n_c:>5} ({n_c/len(myTe):.1%})")

print("\nPER-CLASS PERFORMANCE")
p_c, r_c, f_c, s_c = precision_recall_fscore_support(myTe, mpred, zero_division=0)
per_class = pd.DataFrame({"class": range(len(p_c)), "precision": p_c.round(3),
                          "recall": r_c.round(3), "f1": f_c.round(3),
                          "support": s_c})
print(per_class.to_string(index=False))

print("\nTHE SAME MODEL UNDER THREE AVERAGES")
for avg in ("macro", "micro", "weighted"):
    print(f"  {avg:>9}: precision "
          f"{precision_score(myTe, mpred, average=avg, zero_division=0):.4f}"
          f"  recall {recall_score(myTe, mpred, average=avg, zero_division=0):.4f}"
          f"  F1 {f1_score(myTe, mpred, average=avg, zero_division=0):.4f}")
print(f"  accuracy: {accuracy_score(myTe, mpred):.4f}"
      "   <- equals MICRO for single-label problems")
print("\nMacro is much lower because the rare classes perform badly and macro")
print("weights them equally. 'F1 = 0.82' without naming the average is not a")
print("result. Report macro when rare classes matter.")

# --- WHICH SINGLE METRIC IS MOST RELIABLE UNDER IMBALANCE? ---
print("\nRANKING FOUR MODELS BY DIFFERENT METRICS (imbalanced binary)")
candidates = {
    "always negative": np.zeros(len(yte), int),
    "always positive": np.ones(len(yte), int),
    "threshold 0.5": (proba >= 0.5).astype(int),
    "threshold at base rate": (proba >= yte.mean()).astype(int),
}
comp = pd.DataFrame([metric_row(yte, p, k) for k, p in candidates.items()])
for metric in ("accuracy", "balanced_acc", "F1", "MCC"):
    order = comp.sort_values(metric, ascending=False)["model"].tolist()
    print(f"  by {metric:14s}: {' > '.join(order)}")
print("\nAccuracy ranks 'always negative' first. Balanced accuracy, F1, and MCC")
print("all rank it last. Under imbalance, prefer MCC as a single summary.")

  WHY ACCURACY FAILS (1% positive rate)

  model                   accuracy   balanced   precision   recall   MCC
  always negative           0.9900     0.5000      0.0000   0.0000   0.000
  logistic @ 0.5            0.9917     0.6529      0.7143   0.3061   0.464
  logistic @ 0.05           0.9832     0.8611      0.3200   0.7347   0.478
  logistic @ base rate      0.9651     0.9002      0.1985   0.8367   0.400

  The useless model has the second-highest accuracy. Accuracy is the wrong
  summary whenever the classes are imbalanced.

  THE FOUR CELLS EVERYTHING IS BUILT FROM

                    predicted neg    predicted pos
    actual neg           TN               FP        -> specificity = TN/(TN+FP)
    actual pos           FN               TP        -> recall      = TP/(TP+FN)
                          |                |
                          |          precision = TP/(TP+FP)

  WHY THE HARMONIC MEAN

    precision  recall   arithmetic   F1 (harmonic)
    0.90       0.90      0.900        0.900
    0.95       0.50      0.725        0.655
    1.00       0.10      0.550        0.182
    1.00       0.01      0.505        0.020   <- F1 refuses to hide this

    The harmonic mean is pulled toward the SMALLER value, so a near-zero
    component cannot be averaged away.

  THE TRADE-OFF IS STRUCTURAL

    threshold high    P |############################  |  R |###          |
    threshold mid     P |##############                |  R |###########  |
    threshold low     P |#####                         |  R |############ |

    No setting maximises both. Which one you want is a COST decision.

  MULTI-CLASS AVERAGING CHANGES THE ANSWER

    class   precision   recall    support
    0           0.891    0.951      1,260
    1           0.702    0.641        360
    2           0.481    0.317        126
    3           0.333    0.148         54

      macro     F1 0.547    <- every class counts equally
      micro     F1 0.823    <- pooled; equals accuracy
      weighted  F1 0.806    <- weighted by support

    A 0.28 spread on the same predictions. Always name the average.

  CHOOSING A SINGLE SUMMARY

  situation                         use
  ---------------------------------------------------------------------------
  balanced classes, equal costs     accuracy
  imbalanced, one summary needed    MATTHEWS CORRELATION COEFFICIENT
  imbalanced, care about positives  F1, or precision at a fixed recall
  recall matters more               F-beta with beta > 1
  precision matters more            F-beta with beta < 1
  comparing models pre-threshold    ROC AUC or average precision (next topic)
  rare classes matter in multiclass  MACRO averages

Key Takeaways

• Accuracy is dominated by the majority class, so under imbalance a useless model can score higher than a useful one.
• Precision measures wasted alarms and recall measures missed cases, and no threshold maximises both.
• F1 uses the harmonic mean specifically so that a near-zero precision or recall cannot be averaged away.
• Matthews correlation coefficient uses all four confusion cells and is the most reliable single summary under imbalance.
• In multi-class problems macro, micro, and weighted averaging give substantially different numbers, so always state which one you report.

🧪 Practice

1. Build a dataset with a 1% positive rate and show that a majority-class classifier beats a real model on accuracy while scoring zero on Matthews correlation.
2. Tabulate the arithmetic and harmonic means of several precision-recall pairs and explain why F1 uses the harmonic mean.
3. Sweep the threshold and report F0.5, F1, and F2, identifying where each peaks.
4. Train a four-class model with imbalanced classes and report macro, micro, and weighted F1, explaining the gap.
5. Rank four classifiers by accuracy and by Matthews correlation and find a case where the orderings disagree.
6. Interview: A model reports 0.94 F1 on a balanced validation set and performs poorly in production where positives are 2% of traffic. What happened? (Hint: which metric depends on the class balance of the evaluation set?)

Confusion Matrix Analysis

Every classification metric is a summary of the confusion matrix, which means the matrix contains strictly more information than any metric derived from it. Reading it directly is the most useful diagnostic habit in classification work, and it answers questions no scalar can: which errors is the model making, are they symmetric, and which specific confusions dominate.

For binary classification the four cells are true negatives, false positives, false negatives, and true positives. Every metric is a ratio of these. The cells themselves are what a decision maker needs, because they are counts of real consequences: false positives are reviewer hours, false negatives are missed fraud.

The habit worth building is to look at the matrix in three forms.

Raw counts show volume and are what cost calculations use.

Row-normalised, dividing by actual class totals, gives recall per class and answers "of the real positives, how many did we catch". This is the view for understanding coverage.

Column-normalised, dividing by predicted class totals, gives precision per class and answers "of the things we flagged, how many were right". This is the view for understanding alert quality.

Those last two are the same distinction as row and column percentages in chapter 7.1, and confusing them is just as common here.

For multi-class problems the matrix becomes genuinely irreplaceable. A ten-class problem has ninety possible confusion types, and they are never uniform. Almost always a small number of class pairs account for most errors, and that structure tells you what to do: merge classes that are not meaningfully distinct, collect more data for a specific confusion, or add a feature that separates one particular pair. No aggregate metric surfaces this.

Three analysis techniques extend the basic reading.

Find the dominant confusions. Sort the off-diagonal cells and look at the top few. If class 3 is mistaken for class 8 forty percent of the time and no other confusion exceeds five percent, you have one problem rather than a general accuracy problem.

Check symmetry. If class A is often called B but B is rarely called A, something asymmetric is happening, often a prior or threshold effect rather than genuine similarity.

Normalise for cost. Multiply the matrix elementwise by a cost matrix and sum. This converts the confusion matrix into the single number that actually matters for a decision, and it frequently ranks models differently from accuracy.

For imbalanced multi-class problems, plotting the matrix as a row-normalised heatmap is the standard presentation, and the diagonal should be the visibly dominant feature. Chapter 7.2 covers the encoding choices.

import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay

rng = np.random.default_rng(53)

# --- BINARY: the four cells, and the three normalisations. ---
X, y = make_classification(n_samples=12_000, n_features=14, n_informative=6,
                           weights=[0.92, 0.08], random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                      stratify=y)
proba = LogisticRegression(max_iter=2000).fit(Xtr, ytr).predict_proba(Xte)[:, 1]
pred = (proba >= 0.5).astype(int)

cm = confusion_matrix(yte, pred)
tn, fp, fn, tp = cm.ravel()
idx = ["actual neg", "actual pos"]
col = ["pred neg", "pred pos"]

print("RAW COUNTS (what cost calculations use)")
print(pd.DataFrame(cm, index=idx, columns=col).to_string())
print(f"\n  TN {tn:>5}  FP {fp:>5}   <- FP = wasted reviewer time")
print(f"  FN {fn:>5}  TP {tp:>5}   <- FN = missed cases")

print("\nROW-NORMALISED (divide by ACTUAL totals) -> recall per class")
print(pd.DataFrame(cm / cm.sum(axis=1, keepdims=True),
                   index=idx, columns=col).round(4).to_string())
print(f"  reads as: of real positives, we caught {tp/(tp+fn):.1%}")

print("\nCOLUMN-NORMALISED (divide by PREDICTED totals) -> precision per class")
print(pd.DataFrame(cm / cm.sum(axis=0, keepdims=True),
                   index=idx, columns=col).round(4).to_string())
print(f"  reads as: of our positive flags, {tp/(tp+fp):.1%} were right")
print("\nThese answer DIFFERENT questions. Stating one while the reader assumes")
print("the other is the most common confusion-matrix mistake.")

# --- COST-WEIGHTING: the number that actually decides. ---
COSTS = np.array([[0.0, 12.0],      # [actual neg]: FP costs 12
                  [450.0, 0.0]])    # [actual pos]: FN costs 450
print("\nCOST MATRIX APPLIED (FP $12, FN $450)")
print(f"  {'threshold':>10} {'TN':>6} {'FP':>6} {'FN':>5} {'TP':>5}"
      f" {'accuracy':>9} {'total cost':>12}")
best = None
for t in (0.5, 0.3, 0.15, 0.08, 0.04, 0.02):
    c = confusion_matrix(yte, (proba >= t).astype(int))
    cost = float((c * COSTS).sum())
    acc = np.trace(c) / c.sum()
    if best is None or cost < best[1]:
        best = (t, cost)
    print(f"  {t:>10.2f} {c[0,0]:>6} {c[0,1]:>6} {c[1,0]:>5} {c[1,1]:>5}"
          f" {acc:>9.4f} ${cost:>11,.0f}")
print(f"\nlowest cost at threshold {best[0]}, and accuracy is HIGHEST at 0.5.")
print("The cost-weighted matrix ranks thresholds differently from accuracy,")
print("which is the point of chapter 8.4's threshold formula.")

# --- MULTI-CLASS: where the matrix becomes irreplaceable. ---
Xm, ym = make_classification(n_samples=9_000, n_features=20, n_informative=12,
                             n_classes=6, n_clusters_per_class=1,
                             class_sep=0.9,
                             weights=[0.30, 0.25, 0.20, 0.12, 0.08, 0.05],
                             random_state=1)
mXtr, mXte, myTr, myTe = train_test_split(Xm, ym, test_size=0.3, random_state=0,
                                          stratify=ym)
mpred = RandomForestClassifier(n_estimators=200, random_state=0).fit(
    mXtr, myTr).predict(mXte)
mcm = confusion_matrix(myTe, mpred)
labels = [f"c{i}" for i in range(6)]

print("\nMULTI-CLASS CONFUSION MATRIX (raw counts)")
print(pd.DataFrame(mcm, index=[f"act_{l}" for l in labels],
                   columns=[f"prd_{l}" for l in labels]).to_string())
print(f"\noverall accuracy: {np.trace(mcm)/mcm.sum():.4f}")

# A row-normalised text heatmap: the standard presentation.
print("\nROW-NORMALISED, rendered as density (the diagonal should dominate)")
row_norm = mcm / mcm.sum(axis=1, keepdims=True)
ramp = " .:-=+*#%@"
print("            " + "".join(f"{l:>5}" for l in labels))
for i, l in enumerate(labels):
    cells = "".join(
        f"  {ramp[min(int(row_norm[i, j] * (len(ramp)-1)), len(ramp)-1)]}  "
        for j in range(6))
    print(f"  act_{l}   {cells}  recall {row_norm[i, i]:.3f}")

# --- FIND THE DOMINANT CONFUSIONS: the actionable output. ---
print("\nTOP OFF-DIAGONAL CONFUSIONS")
pairs = []
for i in range(6):
    for j in range(6):
        if i != j and mcm[i, j] > 0:
            pairs.append({"actual": labels[i], "predicted": labels[j],
                          "count": int(mcm[i, j]),
                          "share_of_class": round(row_norm[i, j], 4),
                          "share_of_all_errors": round(
                              mcm[i, j] / (mcm.sum() - np.trace(mcm)), 4)})
top = pd.DataFrame(pairs).sort_values("count", ascending=False)
print(top.head(6).to_string(index=False))
cum = top["share_of_all_errors"].head(3).sum()
print(f"\nthe top 3 confusions account for {cum:.1%} of ALL errors")
print("That is an actionable finding: fix one class pair, not 'accuracy'.")

# --- SYMMETRY CHECK: is the confusion mutual? ---
print("\nSYMMETRY OF THE TOP CONFUSIONS")
print(f"  {'pair':>10} {'i->j':>7} {'j->i':>7} {'ratio':>7}  interpretation")
seen = set()
for _, r in top.head(8).iterrows():
    i, j = labels.index(r["actual"]), labels.index(r["predicted"])
    if (min(i, j), max(i, j)) in seen:
        continue
    seen.add((min(i, j), max(i, j)))
    a, b = mcm[i, j], mcm[j, i]
    ratio = a / max(b, 1)
    note = ("mutual: classes genuinely overlap" if 0.5 <= ratio <= 2
            else "ASYMMETRIC: suspect a prior/threshold effect")
    print(f"  {labels[i]}<->{labels[j]} {a:>7} {b:>7} {ratio:>7.2f}  {note}")

# --- PER-CLASS SUMMARY derived entirely from the matrix. ---
print("\nPER-CLASS METRICS, computed from the matrix alone")
summary = []
for i, l in enumerate(labels):
    tp_i = mcm[i, i]
    fn_i = mcm[i].sum() - tp_i
    fp_i = mcm[:, i].sum() - tp_i
    tn_i = mcm.sum() - tp_i - fn_i - fp_i
    summary.append({
        "class": l, "support": int(mcm[i].sum()),
        "recall": round(tp_i / max(tp_i + fn_i, 1), 3),
        "precision": round(tp_i / max(tp_i + fp_i, 1), 3),
        "specificity": round(tn_i / max(tn_i + fp_i, 1), 3),
        "most_confused_with": labels[int(np.argmax(
            [mcm[i, j] if j != i else -1 for j in range(6)]))],
    })
print(pd.DataFrame(summary).to_string(index=False))
print("\nEvery classification metric is a ratio of these cells, which is why")
print("reading the matrix first tells you more than any metric can.")

  THE BINARY MATRIX AND ITS THREE READINGS

  RAW COUNTS                 what cost calculations use
                pred neg   pred pos
    actual neg     3,266        45
    actual pos        193        96

  ROW-NORMALISED (by ACTUAL)          -> RECALL per class
                pred neg   pred pos
    actual neg    0.9864     0.0136
    actual pos    0.6678     0.3322     "we caught 33% of real positives"

  COLUMN-NORMALISED (by PREDICTED)    -> PRECISION per class
                pred neg   pred pos
    actual neg    0.9441     0.3191
    actual pos    0.0559     0.6809     "68% of our flags were right"

    These answer different questions. Naming the wrong one is the most
    common confusion-matrix error.

  COST-WEIGHTING RANKS THRESHOLDS DIFFERENTLY FROM ACCURACY

    threshold    FP     FN   accuracy   total cost
    0.50         45    193   0.9339     $87,390
    0.30        104    154   0.9283     $70,548
    0.15        241    112   0.9019     $53,292
    0.08        512     71   0.8381     $38,094   <- lowest cost
    0.04      1,043     41   0.6989     $30,966
    0.02      1,764     24   0.5033     $31,968

    accuracy is highest at 0.50 and cost is lowest near 0.04-0.08

  MULTI-CLASS: THE STRUCTURE IS NEVER UNIFORM

              c0   c1   c2   c3   c4   c5
    act_c0    @    .    .         .        recall 0.915
    act_c1    .    @    :    .    .        recall 0.847
    act_c2    .    :    @    .    .    .   recall 0.781
    act_c3         .    .    %    =    .   recall 0.618
    act_c4    .    .    .    =    #    .   recall 0.542   <- c3<->c4 confusion
    act_c5    .    .    .    .    .    #   recall 0.489

    TOP CONFUSIONS
      actual  predicted  count  share of class  share of ALL errors
      c4      c3            61          0.2824              0.1877
      c3      c4            54          0.1667              0.1662
      c5      c2            29          0.2148              0.0892

    The top 3 confusions are 44% of all errors, and two of them are one
    mutual pair. That is a c3/c4 separability problem, not an "accuracy"
    problem, and the fix is specific: a feature that distinguishes them,
    more data for that pair, or merging them.

  SYMMETRY TELLS YOU WHICH KIND OF PROBLEM

    i->j and j->i similar    the classes genuinely overlap
    i->j >> j->i             a prior or threshold effect favours j

Key Takeaways

• The confusion matrix contains more information than any metric derived from it, so read it before reading scores.
• Row-normalised gives recall and column-normalised gives precision, and these answer different questions.
• Multiplying the matrix by a cost matrix produces the number that decides, and it frequently ranks thresholds differently from accuracy.
• In multi-class problems a few class pairs usually account for most errors, which turns a vague accuracy problem into a specific one.
• Checking whether a confusion is mutual distinguishes genuine class overlap from a prior or threshold effect.

🧪 Practice

1. Print a binary confusion matrix in raw, row-normalised, and column- normalised form, and write the sentence each version supports.
2. Apply a cost matrix across several thresholds and show that the lowest- cost threshold is not the most accurate one.
3. Train a six-class model and rank the off-diagonal cells, reporting what share of all errors the top three account for.
4. Check whether your largest confusions are symmetric and interpret the result.
5. Compute precision, recall, and specificity per class directly from a confusion matrix without calling a metric function.
6. Interview: A ten-class image classifier has 88% accuracy and the product team wants it improved. What is the first artefact you produce? (Hint: what would tell you whether this is one problem or ninety?)

ROC and Precision-Recall Curves

Every threshold-based metric describes a model at one operating point, which means comparing two models by their accuracy at 0.5 conflates two things: how well the model ranks cases, and where somebody happened to put the threshold. Curves separate those. They sweep the threshold across its whole range and plot the resulting trade-off, so a curve describes the model's ranking ability independently of any operating decision.

The receiver operating characteristic curve plots the true positive rate, which is recall, against the false positive rate, as the threshold moves. A perfect model reaches the top-left corner. The diagonal is random guessing. The area under the curve summarises it in one number with a genuinely useful interpretation: it is the probability that a randomly chosen positive receives a higher score than a randomly chosen negative. An area of 0.5 is random, 1.0 is perfect, and below 0.5 means the ranking is inverted, which usually indicates a sign error rather than a bad model.

Two properties make it popular. It is threshold-free, so it compares models before an operating point is chosen. And it is insensitive to class balance, because both axes are within-class rates. That second property is also its central weakness.

The precision-recall curve plots precision against recall. Under heavy imbalance it is the more informative view, and the reason is the denominator. The false positive rate divides by the number of negatives, which under imbalance is enormous, so thousands of false positives barely move it. Precision divides by the number of flagged cases, so those same false positives halve it. A fraud model can show 0.95 area under the receiver operating characteristic and produce an alert queue that is 95% noise, and only the precision-recall curve shows that.

The summary statistic is average precision, the precision-weighted mean across recall levels, and its baseline is the positive rate rather than 0.5. Reporting average precision without the positive rate is meaningless, because 0.3 is excellent at a 1% base rate and terrible at a 50% one.

Three further points.

Read the curve, not just the area. Two models with identical areas can have completely different shapes, and the shape is what matters operationally. If you will run at high precision, a model that is better in the low-recall region wins, regardless of total area.

Report the operating point you will actually use. "Precision at 90% recall" or "recall at 95% precision" or "precision in the top 500 alerts" are the numbers that map onto a deployment. Chapter 8.4 covers choosing among them by cost.

Put a confidence band on it. Area under the curve estimated on 200 positives is noisy, and bootstrapping the curve shows how noisy. Two models whose bands overlap heavily are not distinguishable.

import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (roc_curve, roc_auc_score, precision_recall_curve,
                             average_precision_score, precision_score,
                             recall_score)

rng = np.random.default_rng(59)

# --- THE PROBABILISTIC INTERPRETATION OF AUC ---
X, y = make_classification(n_samples=20_000, n_features=16, n_informative=7,
                           weights=[0.99, 0.01], flip_y=0.01, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                      stratify=y)
proba = LogisticRegression(max_iter=2000).fit(Xtr, ytr).predict_proba(Xte)[:, 1]
auc = roc_auc_score(yte, proba)

# Verify the interpretation directly: sample random pairs and count.
pos, neg = proba[yte == 1], proba[yte == 0]
draws = 200_000
wins = (rng.choice(pos, draws) > rng.choice(neg, draws)).mean()
print(f"ROC AUC from the formula:        {auc:.4f}")
print(f"P(random positive > random negative): {wins:.4f}")
print("They match, because that IS what the area under the ROC curve means.")

# --- WHY ROC FLATTERS IMBALANCED MODELS: the denominator. ---
print(f"\nclass balance: {yte.sum()} positives, {(yte==0).sum()} negatives")
fpr, tpr, thr = roc_curve(yte, proba)
prec, rec, pr_thr = precision_recall_curve(yte, proba)
ap = average_precision_score(yte, proba)
print(f"\nROC AUC {auc:.4f}   average precision {ap:.4f}"
      f"   (AP baseline = positive rate = {yte.mean():.4f})")

print("\nTHE SAME OPERATING POINTS, BOTH VIEWS")
print(f"  {'threshold':>10} {'flagged':>8} {'TP':>5} {'FP':>6} {'recall':>8}"
      f" {'FPR':>8} {'precision':>10}")
for t in (0.5, 0.2, 0.08, 0.03, 0.01):
    p = proba >= t
    tp = int((p & (yte == 1)).sum())
    fp = int((p & (yte == 0)).sum())
    print(f"  {t:>10.2f} {int(p.sum()):>8} {tp:>5} {fp:>6}"
          f" {tp/yte.sum():>8.3f} {fp/(yte==0).sum():>8.4f}"
          f" {tp/max(p.sum(),1):>10.3f}")
print("\nAt threshold 0.03 the false positive rate is a reassuring 0.026 while")
print("precision is 0.16: five in six alerts are wrong. ROC hides that because")
print("its denominator is 5,940 negatives; precision's denominator is the")
print("flagged set.")

# --- ASCII CURVES ---
def ascii_curve(xs, ys, xlabel, ylabel, width=44, height=11):
    rows = []
    for r in range(height, -1, -1):
        y_lo, y_hi = r / height, (r + 1) / height
        line = ""
        for c in range(width):
            x_lo, x_hi = c / width, (c + 1) / width
            hit = np.any((xs >= x_lo) & (xs < x_hi) & (ys >= y_lo) & (ys < y_hi))
            line += "#" if hit else " "
        rows.append(f"  {r/height:4.1f} |{line}|")
    rows.append(f"       +{'-'*width}+")
    rows.append(f"        0{' '*(width-4)}1.0  {xlabel}")
    return f"  {ylabel}\n" + "\n".join(rows)

print("\nROC CURVE (top-left is perfect; the diagonal is random)")
print(ascii_curve(fpr, tpr, "false positive rate", "true positive rate"))
print("\nPRECISION-RECALL CURVE (top-right is perfect)")
print(ascii_curve(rec, prec, "recall", "precision"))
print(f"  the dashed baseline for PR sits at the positive rate: {yte.mean():.4f}")

# --- IDENTICAL AUC, DIFFERENT SHAPES: why you read the curve. ---
print("\nTWO MODELS WITH NEARLY EQUAL AUC AND DIFFERENT OPERATING BEHAVIOUR")
rf = RandomForestClassifier(n_estimators=300, min_samples_leaf=4,
                            random_state=0).fit(Xtr, ytr)
p_rf = rf.predict_proba(Xte)[:, 1]
for label, p in [("logistic", proba), ("random forest", p_rf)]:
    a = roc_auc_score(yte, p)
    apr = average_precision_score(yte, p)
    # The operational numbers: what you get at a fixed recall, and vice versa.
    pr, rc, _ = precision_recall_curve(yte, p)
    at_r90 = pr[np.argmin(np.abs(rc - 0.90))]
    at_p50 = rc[np.argmin(np.abs(pr - 0.50))]
    top200 = yte[np.argsort(-p)[:200]].mean()
    print(f"  {label:15s} AUC {a:.4f}  AP {apr:.4f}"
          f"  precision@recall90 {at_r90:.3f}"
          f"  recall@precision50 {at_p50:.3f}"
          f"  precision in top 200 {top200:.3f}")
print("\nAUC can agree while the operational numbers differ. Report the point")
print("you will deploy at, not only the area.")

# --- AUC IS BALANCE-INVARIANT; AVERAGE PRECISION IS NOT. ---
print("\nSUBSAMPLING NEGATIVES CHANGES AP AND NOT AUC")
pos_idx = np.where(yte == 1)[0]
neg_idx = np.where(yte == 0)[0]
for keep in (1.0, 0.5, 0.2, 0.05):
    sel = np.concatenate([pos_idx,
                          rng.choice(neg_idx, int(len(neg_idx)*keep),
                                     replace=False)])
    yy, pp = yte[sel], proba[sel]
    print(f"  keep {keep:>5.0%} of negatives: positive rate {yy.mean():.4f}"
          f"   AUC {roc_auc_score(yy, pp):.4f}"
          f"   AP {average_precision_score(yy, pp):.4f}")
print("\nAUC is essentially unchanged; AP tracks the positive rate. That is why")
print("AP must always be reported WITH the base rate, and why AUC is the")
print("better choice for comparing across differently balanced datasets.")

# --- UNCERTAINTY: bootstrap the metric, especially with few positives. ---
def bootstrap_metric(y_true, y_score, fn, n_boot=600, seed=0):
    r = np.random.default_rng(seed)
    vals = []
    n = len(y_true)
    for _ in range(n_boot):
        idx = r.integers(0, n, n)
        if len(np.unique(y_true[idx])) < 2:
            continue
        vals.append(fn(y_true[idx], y_score[idx]))
    return np.mean(vals), np.percentile(vals, [2.5, 97.5])

print("\nBOOTSTRAP CONFIDENCE INTERVALS (positives drive the width)")
for label, size in [("full test set", len(yte)), ("small sample", 1500)]:
    sub = rng.choice(len(yte), size, replace=False)
    yy, pp = yte[sub], proba[sub]
    m_auc, ci_auc = bootstrap_metric(yy, pp, roc_auc_score)
    m_ap, ci_ap = bootstrap_metric(yy, pp, average_precision_score)
    print(f"  {label:14s} n={size:>5} positives={yy.sum():>4}")
    print(f"    AUC {m_auc:.4f}  95% CI [{ci_auc[0]:.4f}, {ci_auc[1]:.4f}]"
          f"  width {ci_auc[1]-ci_auc[0]:.4f}")
    print(f"    AP  {m_ap:.4f}  95% CI [{ci_ap[0]:.4f}, {ci_ap[1]:.4f}]"
          f"  width {ci_ap[1]-ci_ap[0]:.4f}")
print("With 15 positives the interval spans a third of the range. Two models")
print("whose bands overlap that much are not distinguishable.")

# --- CHOOSING THE OPERATING POINT FROM A CONSTRAINT ---
print("\nOPERATING POINTS UNDER REAL CONSTRAINTS")
capacity = 150                                    # reviews available per day
order = np.argsort(-proba)
print(f"  review capacity {capacity}: threshold {proba[order[capacity-1]]:.4f}"
      f"  precision {yte[order[:capacity]].mean():.3f}"
      f"  recall {yte[order[:capacity]].sum()/yte.sum():.3f}")
target_recall = 0.80
i = np.argmin(np.abs(rec - target_recall))
print(f"  required recall {target_recall:.0%}: precision {prec[i]:.3f}"
      f"  -> alerts per {len(yte):,} cases:"
      f" {int(yte.sum()*target_recall/max(prec[i],1e-9)):,}")
print("These are the numbers a stakeholder can act on. The area under a curve")
print("is not one of them.")

  THE TWO CURVES

  ROC                                PRECISION-RECALL
  true positive rate                 precision
   1.0|      ######                   1.0|##
      |    ##                            |  #
      |  ##                              |   ##
      | #                                |     ###
      |#         (diagonal = random)     |        #####
   0.0|________________                0.0|_____________####
      0   false positive rate  1.0        0     recall      1.0
                                          (baseline = positive rate)

  perfect = TOP-LEFT                  perfect = TOP-RIGHT

  AUC MEANS SOMETHING SPECIFIC

    AUC = P(score of a random positive > score of a random negative)

    formula:           0.9174
    simulated pairs:   0.9171      identical, because that is the definition

  WHY ROC FLATTERS AN IMBALANCED MODEL (59 positives, 5,941 negatives)

    threshold   flagged    TP     FP   recall     FPR   precision
    0.50             56    39     17    0.661  0.0029       0.696
    0.20            121    48     73    0.814  0.0123       0.397
    0.08            243    52    191    0.881  0.0322       0.214
    0.03            447    55    392    0.932  0.0660       0.123
    0.01            911    58    853    0.983  0.1436       0.064

    At 0.03 the FPR is a comfortable 0.066 and precision is 0.123: seven in
    eight alerts are wrong. FPR divides by 5,941; precision divides by 447.

  AUC vs AVERAGE PRECISION UNDER RESAMPLING

    keep 100% of negatives   positive rate 0.0098   AUC 0.9174   AP 0.3812
    keep  50%                positive rate 0.0195   AUC 0.9169   AP 0.4901
    keep  20%                positive rate 0.0474   AUC 0.9158   AP 0.6203
    keep   5%                positive rate 0.1653   AUC 0.9141   AP 0.7984

    AUC is balance-INVARIANT. AP tracks the base rate, so it must always be
    reported alongside it.

  WHAT TO REPORT

    AUC                       ranking quality, comparable across datasets
    average precision         + the positive rate, always
    precision @ target recall the operational number
    recall @ target precision the other operational number
    precision in the top k    when capacity is the binding constraint
    bootstrap CI on all of it especially when positives are few

Key Takeaways

• Curves describe ranking ability across all thresholds, separating it from where somebody set the operating point.
• Area under the receiver operating characteristic curve is the probability a random positive outranks a random negative, and it is invariant to class balance.
• Under heavy imbalance the false positive rate has an enormous denominator, so a strong area can coexist with an alert queue that is mostly noise.
• Average precision's baseline is the positive rate, so it is meaningless without it, and it changes when you resample negatives.
• Report the operating point you will deploy at with a bootstrap interval, since equal areas can hide very different operational behaviour.

🧪 Practice

1. Verify the probabilistic interpretation of the area under the curve by sampling random positive-negative pairs and counting wins.
2. Build an imbalanced problem and tabulate recall, false positive rate, and precision at five thresholds, explaining the divergence.
3. Subsample negatives at several rates and show that the area is stable while average precision moves.
4. Bootstrap confidence intervals for both summary metrics on a sample with few positives and report the interval widths.
5. Compute precision at 90% recall and recall at 50% precision for two models whose areas are nearly equal.
6. Interview: A vendor claims 0.93 area under the curve for their fraud model. What two questions do you ask? (Hint: what is the base rate, and what would the alert queue look like at the recall you need?)

Probability Calibration

A model that outputs 0.8 is making a claim: among all cases it scores 0.8, about 80% should be positive. Calibration is whether that claim is true. It is a distinct property from discrimination, which is whether the model ranks cases correctly, and the two can come apart completely. A model that outputs exactly half of every true probability ranks perfectly, achieving a full area under the curve of 1.0, and every one of its probabilities is wrong.

Whether you need calibration depends entirely on what consumes the output.

If a ranking is all you need, such as ordering leads for a sales team or choosing the top hundred cases to review, calibration is irrelevant. Only the order matters.

If the probability is multiplied by something, calibration is essential. Expected loss is probability times amount. Expected revenue is probability times value. Pricing, reserving, bidding, and any expected-value calculation from chapter 8.4 consume probabilities as numbers, and a miscalibrated input silently corrupts every downstream figure.

If the output is shown to a person who will act on it, calibration is essential for a subtler reason: a clinician told "20% risk" will interpret that literally, and systematic overstatement erodes trust in the model permanently.

Which models are miscalibrated, and how, is predictable:

Logistic regression is calibrated by construction on the training distribution, because its loss is a proper scoring rule fitted on exactly that parameterisation.

Naive Bayes is badly overconfident, pushing probabilities toward 0 and 1, because its independence assumption multiplies correlated evidence as though it were independent.

Support vector machines produce distances, not probabilities, and need conversion.

Boosted trees tend to be overconfident, since the loss rewards pushing probabilities outward.

Random forests are typically underconfident in the extremes, because averaging many trees pulls predictions toward the middle, so they rarely output values near 0 or 1.

Class weighting and resampling break calibration deliberately, because they change the effective base rate. Chapter 6.4 covers this; the consequence is that any imbalance treatment requires recalibration if the probabilities will be used as numbers.

Diagnosis uses a reliability diagram: bin predictions, plot mean predicted against observed frequency per bin, and compare against the diagonal. Summary statistics include the Brier score, a proper scoring rule which decomposes into calibration and refinement components, and expected calibration error, the weighted mean absolute gap across bins.

The fixes are post-hoc and cheap. Platt scaling fits a one-dimensional logistic regression mapping scores to probabilities, which is parametric, needs little data, and assumes a sigmoid distortion. Isotonic regression fits any monotonic mapping, which is more flexible, needs more data, and can overfit on small sets. Both must be fitted on data the model did not train on, or the calibration learns the training distortion rather than the real one.

import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.svm import SVC
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.metrics import brier_score_loss, roc_auc_score, log_loss

rng = np.random.default_rng(61)

# --- DISCRIMINATION AND CALIBRATION ARE DIFFERENT PROPERTIES ---
X, y = make_classification(n_samples=20_000, n_features=16, n_informative=8,
                           weights=[0.85, 0.15], random_state=0)
Xtr, Xcal, ytr, ycal = train_test_split(X, y, test_size=0.4, random_state=0,
                                        stratify=y)
Xcal, Xte, ycal, yte = train_test_split(Xcal, ycal, test_size=0.5, random_state=0,
                                        stratify=ycal)
true_p = LogisticRegression(max_iter=2000).fit(Xtr, ytr).predict_proba(Xte)[:, 1]

print("A PERFECT RANKER WITH WRONG PROBABILITIES")
halved = true_p / 2                            # same order, all values halved
print(f"  original: AUC {roc_auc_score(yte, true_p):.4f}"
      f"  Brier {brier_score_loss(yte, true_p):.5f}")
print(f"  halved:   AUC {roc_auc_score(yte, halved):.4f}"
      f"  Brier {brier_score_loss(yte, halved):.5f}")
print("  Identical AUC, because the ORDER is identical. Far worse Brier score,")
print("  because every probability is now wrong. AUC cannot see calibration.")

# --- MEASURING CALIBRATION ---
def calibration_report(y_true, p, n_bins=10):
    """Reliability table plus expected calibration error."""
    edges = np.quantile(p, np.linspace(0, 1, n_bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    bins = np.digitize(p, edges[1:-1])
    rows, ece = [], 0.0
    for b in range(n_bins):
        m = bins == b
        if m.sum() == 0:
            continue
        pred, obs = p[m].mean(), y_true[m].mean()
        rows.append({"bin": b, "n": int(m.sum()), "mean_pred": round(pred, 4),
                     "observed": round(obs, 4), "gap": round(pred - obs, 4)})
        ece += m.mean() * abs(pred - obs)
    return pd.DataFrame(rows), ece

tbl, ece = calibration_report(yte, true_p)
print(f"\nRELIABILITY TABLE, logistic regression (ECE {ece:.4f})")
print(tbl.to_string(index=False))
print("  gap near zero in every bin: this model is well calibrated.")

# --- WHICH MODELS ARE MISCALIBRATED, AND IN WHICH DIRECTION ---
models = {
    "logistic regression": LogisticRegression(max_iter=2000),
    "naive Bayes": GaussianNB(),
    "random forest": RandomForestClassifier(n_estimators=250, random_state=0),
    "gradient boosting": GradientBoostingClassifier(random_state=0),
    "SVM (rbf)": SVC(probability=True, random_state=0),
}
print("\nCALIBRATION BY MODEL FAMILY")
print(f"  {'model':<22} {'AUC':>7} {'Brier':>8} {'log loss':>9} {'ECE':>7}"
      f" {'min p':>7} {'max p':>7}  direction")
results = {}
for name, m in models.items():
    fitted = m.fit(Xtr, ytr)
    p = fitted.predict_proba(Xte)[:, 1]
    results[name] = p
    _, e = calibration_report(yte, p)
    t, _ = calibration_report(yte, p)
    # Positive mean gap = predictions above reality = overconfident.
    direction = ("overconfident" if t["gap"].mean() > 0.02
                 else "underconfident" if t["gap"].mean() < -0.02
                 else "well calibrated")
    print(f"  {name:<22} {roc_auc_score(yte, p):>7.4f}"
          f" {brier_score_loss(yte, p):>8.5f} {log_loss(yte, p):>9.4f}"
          f" {e:>7.4f} {p.min():>7.4f} {p.max():>7.4f}  {direction}")
print("\nNaive Bayes pushes to the extremes (its independence assumption")
print("multiplies correlated evidence). The forest compresses toward the")
print("middle (averaging many trees). Both have decent AUC.")

# --- AN ASCII RELIABILITY DIAGRAM ---
print("\nRELIABILITY DIAGRAM (o = observed, | = perfect calibration)")
for name in ("logistic regression", "naive Bayes", "random forest"):
    t, e = calibration_report(yte, results[name], n_bins=8)
    print(f"\n  {name} (ECE {e:.4f})")
    print(f"  {'predicted':>10} {'observed':>9}  0{'':>18}1")
    for _, r in t.iterrows():
        pos_pred = int(r["mean_pred"] * 20)
        pos_obs = int(r["observed"] * 20)
        line = [" "] * 21
        line[min(pos_pred, 20)] = "|"
        line[min(pos_obs, 20)] = "o" if pos_obs != pos_pred else "*"
        print(f"  {r['mean_pred']:>10.3f} {r['observed']:>9.3f}  {''.join(line)}")
    print("  (o left of | = overconfident; o right of | = underconfident)")

# --- FIXING IT: Platt scaling and isotonic regression. ---
print("\nPOST-HOC CALIBRATION (fitted on a held-out calibration split)")
print(f"  {'model':<20} {'method':<12} {'AUC':>7} {'Brier':>8} {'ECE':>7}")
for name, base in [("naive Bayes", GaussianNB()),
                   ("random forest",
                    RandomForestClassifier(n_estimators=250, random_state=0))]:
    raw = base.fit(Xtr, ytr)
    p_raw = raw.predict_proba(Xte)[:, 1]
    _, e_raw = calibration_report(yte, p_raw)
    print(f"  {name:<20} {'none':<12} {roc_auc_score(yte, p_raw):>7.4f}"
          f" {brier_score_loss(yte, p_raw):>8.5f} {e_raw:>7.4f}")
    for method in ("sigmoid", "isotonic"):
        # cv="prefit" calibrates an already-trained model on FRESH data.
        cal = CalibratedClassifierCV(raw, method=method, cv="prefit").fit(Xcal, ycal)
        p_cal = cal.predict_proba(Xte)[:, 1]
        _, e_cal = calibration_report(yte, p_cal)
        label = "Platt" if method == "sigmoid" else "isotonic"
        print(f"  {'':<20} {label:<12} {roc_auc_score(yte, p_cal):>7.4f}"
              f" {brier_score_loss(yte, p_cal):>8.5f} {e_cal:>7.4f}")
print("\nCalibration barely changes AUC, because it is a MONOTONIC transform")
print("and monotonic transforms preserve ranking. It substantially improves")
print("Brier score and calibration error.")

# --- CALIBRATION MUST BE FITTED ON UNSEEN DATA ---
print("\nCALIBRATING ON TRAINING DATA DOES NOT WORK")
nb = GaussianNB().fit(Xtr, ytr)
for label, (Xc, yc) in [("on TRAINING data (wrong)", (Xtr, ytr)),
                        ("on a held-out split (right)", (Xcal, ycal))]:
    cal = CalibratedClassifierCV(nb, method="isotonic", cv="prefit").fit(Xc, yc)
    p = cal.predict_proba(Xte)[:, 1]
    _, e = calibration_report(yte, p)
    print(f"  {label:28s} test ECE {e:.4f}"
          f"  Brier {brier_score_loss(yte, p):.5f}")
print("On training data the model's distortion is different (it has memorised")
print("those rows), so the learned correction is the wrong correction.")

# --- WHY IT MATTERS: expected value is probability TIMES amount. ---
amounts = rng.lognormal(4.0, 1.0, len(yte))
print("\nTHE COST OF MISCALIBRATION IN AN EXPECTED-VALUE DECISION")
print(f"  {'model':<24} {'predicted loss':>16} {'actual loss':>14} {'error':>10}")
actual_loss = (yte * amounts).sum()
for name in ("logistic regression", "naive Bayes", "random forest"):
    predicted = (results[name] * amounts).sum()
    print(f"  {name:<24} ${predicted:>15,.0f} ${actual_loss:>13,.0f}"
          f" {predicted/actual_loss-1:>+9.1%}")
cal_nb = CalibratedClassifierCV(GaussianNB().fit(Xtr, ytr), method="isotonic",
                               cv="prefit").fit(Xcal, ycal)
p_fixed = cal_nb.predict_proba(Xte)[:, 1]
print(f"  {'naive Bayes, calibrated':<24} ${(p_fixed*amounts).sum():>15,.0f}"
      f" ${actual_loss:>13,.0f} {(p_fixed*amounts).sum()/actual_loss-1:>+9.1%}")
print("\nThe uncalibrated reserve estimate is off by tens of percent. Every")
print("expected-value calculation inherits the calibration error directly.")

# --- WHEN YOU DO NOT NEED IT ---
print("\nWHEN CALIBRATION IS IRRELEVANT: a top-k ranking task")
k = 200
for name in ("logistic regression", "naive Bayes", "random forest"):
    top = np.argsort(-results[name])[:k]
    print(f"  {name:<24} positives in top {k}: {int(yte[top].sum()):>4}"
          f"  precision {yte[top].mean():.3f}")
print("The badly calibrated models do fine here, because only the ORDER is")
print("used. Calibrate when the number is consumed as a number.")

  TWO INDEPENDENT PROPERTIES

    DISCRIMINATION   does the model RANK correctly?      -> AUC
    CALIBRATION      are the probabilities TRUE?         -> Brier, ECE

    a model whose outputs are all HALVED:
      AUC    0.9432 -> 0.9432   unchanged (order preserved)
      Brier  0.0841 -> 0.1174   much worse (every value wrong)

    AUC is blind to calibration. That is not a flaw; it is the definition.

  RELIABILITY DIAGRAM

  observed
   1.0|                    /            perfectly calibrated sits ON
      |                  /  o           the diagonal
      |                /  o
      |              /  o               o BELOW the line = the model
      |            /  o                   predicts HIGHER than reality
      |          / o                      = OVERCONFIDENT
      |        /o
   0.0|______/______________
      0    predicted     1.0

  MISCALIBRATION IS PREDICTABLE BY FAMILY

  model                 AUC      ECE     min p    max p   direction
  logistic regression   0.9432   0.0071  0.0018   0.9886  well calibrated
  naive Bayes           0.9214   0.1783  0.0000   1.0000  OVERCONFIDENT
  random forest         0.9508   0.0402  0.0040   0.9880  underconfident
  gradient boosting     0.9531   0.0289  0.0009   0.9954  slightly over
  SVM (rbf)             0.9377   0.0521  0.0106   0.9799  over

  naive Bayes: independence assumption multiplies correlated evidence
  random forest: averaging trees pulls predictions toward the middle

  THE FIXES

  method              form                data needed   risk
  ---------------------------------------------------------------------------
  Platt (sigmoid)     1-D logistic        little        assumes a sigmoid
  isotonic            any monotonic step  more          overfits small sets

  both are MONOTONIC, so AUC is preserved and only calibration improves:

    naive Bayes, none        AUC 0.9214  Brier 0.1408  ECE 0.1783
    naive Bayes, Platt       AUC 0.9214  Brier 0.0952  ECE 0.0184
    naive Bayes, isotonic    AUC 0.9209  Brier 0.0938  ECE 0.0121

  MUST BE FITTED ON UNSEEN DATA

    calibrated on TRAINING data   test ECE 0.1521   (barely helps)
    calibrated on a held-out set  test ECE 0.0121   (works)

  DO YOU NEED IT?

  the output is...                     calibration
  a ranking (top-k, sort order)         NOT needed
  multiplied by a value (expected loss) ESSENTIAL
  compared to a cost threshold          ESSENTIAL
  shown to a human as a risk            ESSENTIAL
  fed to another model                  usually needed
  after class weighting or resampling    RE-CALIBRATE

Key Takeaways

• Calibration and discrimination are independent: halving every probability leaves the area under the curve unchanged and every probability wrong.
• Calibration is essential whenever the probability is multiplied by a value or shown to a person, and irrelevant for pure ranking.
• Naive Bayes is overconfident and random forests are underconfident at the extremes, for structural reasons rather than by accident.
• Platt scaling and isotonic regression are monotonic, so they fix calibration without changing ranking, and they must be fitted on held-out data.
• Class weighting and resampling break calibration deliberately, so any imbalance treatment requires recalibration if probabilities will be consumed as numbers.

🧪 Practice

1. Halve every predicted probability and confirm the area under the curve is unchanged while the Brier score worsens.
2. Build a reliability table with expected calibration error and run it on a logistic regression, a naive Bayes model, and a random forest.
3. Apply Platt scaling and isotonic regression to a miscalibrated model and report the change in both area and calibration error.
4. Show that calibrating on training data fails while calibrating on a held- out split works.
5. Compute an expected-loss figure using calibrated and uncalibrated probabilities and report the percentage error.
6. Interview: Your team applies class weighting to handle imbalance, and the finance team uses the model's probabilities to set reserves. What do you flag? (Hint: what does changing the effective base rate do to the meaning of the output?)

Ranking Metrics

Many problems are not really classification or regression at all. Search returns an ordered list, a recommender fills a limited number of slots, a fraud team reviews as many cases as it has capacity for, and a sales team calls the top of a lead list. In all of these, only the order matters and only the top of the order matters. Metrics that average over the whole dataset answer the wrong question.

Two properties define this class of metric.

Position matters. A relevant item at position 1 is worth far more than the same item at position 50, because users look at the top and rarely scroll. Metrics for ranking therefore apply a discount that decreases with position.

Only the top-k is evaluated. If your team reviews 200 cases a day, performance on case 5,000 is irrelevant. The cutoff is a real operational parameter, not a tuning choice.

The standard metrics, from simplest to most complete:

Precision at k is the fraction of the top k that are relevant. Simple, interpretable, and directly operational: "of the 200 alerts we review, how many are real". Its blind spot is that it ignores order within the top k and ignores how many relevant items existed.

Recall at k is the fraction of all relevant items appearing in the top k, which answers coverage.

Mean reciprocal rank is the average of one divided by the position of the first relevant item. This is the right metric when the user needs exactly one answer, such as a navigational search query.

Mean average precision averages precision computed at each position where a relevant item appears, then averages over queries. It rewards both getting relevant items and getting them early.

Normalised discounted cumulative gain is the most widely used metric for graded relevance. It sums each item's gain divided by a logarithmic position discount, then divides by the best possible arrangement so the score sits between 0 and 1. It handles multi-level relevance, such as "perfect, good, fair, irrelevant", which the binary metrics cannot.

Two practical points that matter more than the choice among these.

Position bias contaminates your labels. Click logs record what users clicked, and users click what was shown at the top. So an item at position 1 gets clicks partly because it was at position 1, which means training on clicks teaches the model to reproduce the existing ranking rather than to improve it. Correcting this needs either an explicit position-bias model with inverse propensity weighting, or randomised interleaving to collect unbiased data.

Offline ranking metrics correlate imperfectly with online outcomes. A model that improves normalised discounted cumulative gain does not always improve engagement, because the offline metric measures agreement with historical labels rather than user satisfaction. Online experiments from chapter 8 remain the arbiter, and offline metrics are for filtering candidates cheaply.

import numpy as np
import pandas as pd

rng = np.random.default_rng(67)

# --- THE METRICS, IMPLEMENTED ---
def precision_at_k(relevance, k):
    return float(np.mean(relevance[:k]))

def recall_at_k(relevance, k, total_relevant=None):
    total = total_relevant if total_relevant is not None else relevance.sum()
    return float(relevance[:k].sum() / max(total, 1))

def reciprocal_rank(relevance):
    hits = np.flatnonzero(relevance)
    return float(1.0 / (hits[0] + 1)) if len(hits) else 0.0

def average_precision(relevance):
    """Mean of precision@i at every position i where a relevant item sits."""
    hits = np.flatnonzero(relevance)
    if len(hits) == 0:
        return 0.0
    precisions = [(m + 1) / (i + 1) for m, i in enumerate(hits)]
    return float(np.mean(precisions))

def dcg(gains, k=None):
    """Discounted cumulative gain: each item's gain divided by log2(rank+1)."""
    g = gains[:k] if k else gains
    discounts = np.log2(np.arange(2, len(g) + 2))
    return float(np.sum(g / discounts))

def ndcg(gains, k=None):
    """DCG divided by the DCG of the IDEAL ordering, so it lands in [0, 1]."""
    ideal = dcg(np.sort(gains)[::-1], k)
    return float(dcg(gains, k) / ideal) if ideal > 0 else 0.0

# --- POSITION MATTERS: the same items, three orderings. ---
print("THE SAME FIVE RELEVANT ITEMS IN 20 SLOTS, THREE ORDERINGS")
orderings = {
    "relevant first":  np.array([1]*5 + [0]*15),
    "spread evenly":   np.array([1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0]),
    "relevant last":   np.array([0]*15 + [1]*5),
}
print(f"  {'ordering':<18} {'P@5':>6} {'P@10':>6} {'R@5':>6} {'MRR':>6}"
      f" {'AP':>6} {'nDCG@10':>8}")
for name, rel in orderings.items():
    print(f"  {name:<18} {precision_at_k(rel,5):>6.3f} {precision_at_k(rel,10):>6.3f}"
          f" {recall_at_k(rel,5):>6.3f} {reciprocal_rank(rel):>6.3f}"
          f" {average_precision(rel):>6.3f} {ndcg(rel.astype(float),10):>8.3f}")
print("\nAll three contain the SAME five relevant items. Accuracy and AUC-style")
print("aggregate metrics would rate them similarly; ranking metrics do not.")

# --- THE DISCOUNT: how fast position value decays. ---
print("\nPOSITION DISCOUNT (1 / log2(rank + 1))")
for pos in (1, 2, 3, 5, 10, 20, 50):
    d = 1 / np.log2(pos + 1)
    print(f"  position {pos:>3}: weight {d:.4f}"
          f"  ({d/1.0:.1%} of position 1)  {'#' * int(d * 40)}")
print("Position 10 is worth 30% of position 1, and position 50 about 18%.")
print("A log discount is gentler than real click-through decay, which is why")
print("some teams use a steeper custom discount.")

# --- GRADED RELEVANCE: what only nDCG can express. ---
print("\nGRADED RELEVANCE (0 = irrelevant ... 3 = perfect)")
graded = {
    "perfect order":      np.array([3., 3., 2., 2., 1., 1., 0., 0., 0., 0.]),
    "one perfect buried": np.array([2., 2., 1., 1., 0., 0., 0., 0., 0., 3.]),
    "good but not best":  np.array([2., 2., 2., 2., 1., 1., 1., 0., 0., 0.]),
}
print(f"  {'ordering':<22} {'DCG@10':>8} {'ideal':>8} {'nDCG@10':>9}"
      f" {'binary P@3':>11}")
for name, g in graded.items():
    ideal = dcg(np.sort(g)[::-1], 10)
    binary = precision_at_k((g > 0).astype(float), 3)
    print(f"  {name:<22} {dcg(g,10):>8.3f} {ideal:>8.3f} {ndcg(g,10):>9.3f}"
          f" {binary:>11.3f}")
print("\nThe binary metric cannot distinguish 'perfect' from 'merely relevant'.")
print("nDCG penalises burying the one perfect result at position 10.")

# --- A REALISTIC MULTI-QUERY EVALUATION ---
n_queries, n_docs = 400, 50
rows = []
for q in range(n_queries):
    # Graded relevance: mostly irrelevant with a few good documents.
    true_rel = rng.choice([0, 0, 0, 0, 1, 1, 2, 3], n_docs)
    # Three rankers of differing quality, all scoring the same documents.
    scores = {
        "perfect": true_rel + rng.normal(0, 0.01, n_docs),
        "good model": true_rel + rng.normal(0, 0.8, n_docs),
        "weak model": true_rel + rng.normal(0, 2.5, n_docs),
        "random": rng.normal(0, 1, n_docs),
    }
    for name, s in scores.items():
        order = np.argsort(-s)
        ranked_gains = true_rel[order].astype(float)
        ranked_binary = (ranked_gains > 0).astype(float)
        rows.append({
            "ranker": name,
            "P@5": precision_at_k(ranked_binary, 5),
            "P@10": precision_at_k(ranked_binary, 10),
            "R@10": recall_at_k(ranked_binary, 10, (true_rel > 0).sum()),
            "MRR": reciprocal_rank(ranked_binary),
            "MAP": average_precision(ranked_binary),
            "nDCG@10": ndcg(ranked_gains, 10),
        })
res = pd.DataFrame(rows).groupby("ranker").mean().round(4)
print(f"\nAVERAGED OVER {n_queries} QUERIES")
print(res.loc[["perfect", "good model", "weak model", "random"]].to_string())
print("\nEvery metric orders the rankers correctly here. They disagree when")
print("models differ in WHERE they place relevant items rather than whether.")

# --- CHOOSING k IS AN OPERATIONAL DECISION ---
print("\nPRECISION AND RECALL AT DIFFERENT CUTOFFS (the 'good model')")
good_rows = []
for q in range(200):
    true_rel = rng.choice([0, 0, 0, 0, 1, 1, 2, 3], n_docs)
    s = true_rel + rng.normal(0, 0.8, n_docs)
    ranked = (true_rel[np.argsort(-s)] > 0).astype(float)
    good_rows.append(ranked)
G = np.array(good_rows)
total_rel = G.sum(axis=1)
print(f"  {'k':>4} {'precision@k':>13} {'recall@k':>10} {'relevant found':>15}")
for k in (1, 3, 5, 10, 20, 50):
    p = G[:, :k].mean()
    r = (G[:, :k].sum(axis=1) / np.maximum(total_rel, 1)).mean()
    print(f"  {k:>4} {p:>13.4f} {r:>10.4f} {G[:, :k].sum(axis=1).mean():>15.2f}")
print("Precision falls and recall rises with k, exactly as in classification.")
print("Choose k from capacity or from how many slots the interface has.")

# --- POSITION BIAS: why click logs are contaminated labels. ---
print("\nPOSITION BIAS IN CLICK LOGS")
EXAMINE = np.array([1/np.log2(i + 2) for i in range(10)])   # P(user looks)
true_relevance = np.array([0.30]*10)                # ALL items equally relevant
observed_ctr = EXAMINE * true_relevance
print(f"  {'position':>9} {'P(examined)':>12} {'true relevance':>15}"
      f" {'observed CTR':>13}")
for i in range(6):
    print(f"  {i+1:>9} {EXAMINE[i]:>12.4f} {true_relevance[i]:>15.2f}"
          f" {observed_ctr[i]:>13.4f}")
print(f"\n  Every item is EQUALLY relevant, and observed click-through falls")
print(f"  from {observed_ctr[0]:.3f} to {observed_ctr[5]:.3f} purely from position.")
print("  Training on raw clicks teaches the model to reproduce the CURRENT")
print("  ranking, not to improve it.")

# Inverse propensity weighting: divide each click by P(examined).
ipw_estimate = observed_ctr / EXAMINE
print(f"\n  after inverse propensity weighting: "
      f"{np.round(ipw_estimate[:6], 3)}")
print("  Weighting each observation by 1/P(examined) recovers the truth.")
print("  The alternative is randomised interleaving to collect clean data.")

# --- OFFLINE METRICS ARE A FILTER, NOT A VERDICT ---
print("\nOFFLINE nDCG VS AN ONLINE OUTCOME (simulated correlation)")
offline = rng.uniform(0.60, 0.85, 30)
# Online lift correlates with offline gain but far from perfectly.
online = (offline - 0.7) * 0.4 + rng.normal(0, 0.025, 30)
corr = np.corrcoef(offline, online)[0, 1]
agree = np.mean(np.sign(offline - np.median(offline)) == np.sign(online))
print(f"  correlation between offline nDCG and online lift: {corr:.3f}")
print(f"  sign agreement: {agree:.0%}")
print("  Offline metrics cheaply eliminate bad candidates. They do not")
print("  substitute for the experiment (chapter 8).")

  THE SAME FIVE RELEVANT ITEMS, THREE ORDERINGS

  relevant first   #####...............   P@5 1.000  MRR 1.000  nDCG@10 1.000
  spread evenly    #...#...#...#...#...   P@5 0.400  MRR 1.000  nDCG@10 0.663
  relevant last    ...............#####   P@5 0.000  MRR 0.063  nDCG@10 0.000

  An aggregate metric sees the same five positives in all three. Only a
  ranking metric sees that the first is useful and the third is useless.

  THE POSITION DISCOUNT (1 / log2(rank + 1))

    position    weight    relative to position 1
    1           1.0000    100%   ########################################
    2           0.6309     63%   #########################
    3           0.5000     50%   ####################
    5           0.3869     39%   ###############
    10          0.3010     30%   ############
    20          0.2284     23%   #########
    50          0.1761     18%   #######

  WHICH METRIC FOR WHICH PROBLEM

  metric        answers                          use when
  ---------------------------------------------------------------------------
  precision@k   of the k we show, how many hit?   fixed slots, review capacity
  recall@k      of all good items, how many       coverage matters
                appear in the top k?
  MRR           how high is the FIRST hit?        the user needs ONE answer
  MAP           are hits both present AND early?  multiple relevant items
  nDCG@k        graded relevance + position       the general default

  ONLY nDCG HANDLES GRADED RELEVANCE

    ordering               DCG@10   nDCG@10   binary P@3
    perfect order           8.693    1.000      1.000
    one perfect buried      6.128    0.705      1.000   <- binary cannot tell
    good but not best       7.041    0.810      1.000      these apart

  POSITION BIAS CONTAMINATES CLICK LABELS

    every item equally relevant (0.30), yet:

    position   P(examined)   true relevance   observed CTR
    1            1.0000          0.30           0.3000
    2            0.6309          0.30           0.1893
    3            0.5000          0.30           0.1500
    4            0.4307          0.30           0.1292
    5            0.3869          0.30           0.1161
    6            0.3562          0.30           0.1069

    Observed click-through falls 2.8x from position alone. A model trained
    on raw clicks learns to REPRODUCE the current ranking.

    fix 1: inverse propensity weighting -- divide by P(examined)
    fix 2: randomised interleaving -- collect unbiased data directly

  OFFLINE METRICS ARE A FILTER

    correlation(offline nDCG, online lift) ~ 0.6, sign agreement ~ 75%

    Good enough to discard weak candidates cheaply. Not a substitute for
    the online experiment.

Key Takeaways

• Ranking problems care only about order and only about the top of it, so metrics averaged over the whole dataset answer the wrong question.
• Precision at k is the operational number when slots or review capacity are fixed, and k comes from the constraint rather than from tuning.
• Normalised discounted cumulative gain handles graded relevance and position discounting, which binary metrics cannot express.
• Click logs are contaminated by position bias, so training on raw clicks teaches the model to reproduce the existing ranking.
• Offline ranking metrics correlate imperfectly with online outcomes, so use them to filter candidates and an experiment to decide.

🧪 Practice

1. Implement precision at k, mean reciprocal rank, average precision, and normalised discounted cumulative gain, and score three orderings of the same relevant items.
2. Tabulate the logarithmic position discount and state what position 10 is worth relative to position 1.
3. Construct a graded-relevance example where a binary metric cannot distinguish two orderings that differ in normalised discounted cumulative gain.
4. Simulate position bias with equally relevant items and show that observed click-through still declines with position.
5. Apply inverse propensity weighting to recover the true relevance from biased click data.
6. Interview: Your recommender's offline normalised discounted cumulative gain improved by 8% and engagement did not move after launch. Name two explanations. (Hint: what were the offline labels collected under, and what does the metric actually measure agreement with?)

Baselines and Naive Benchmarks

A model's score means nothing on its own. An accuracy of 0.85 is excellent if the majority class is 50% of the data and embarrassing if it is 90%. A mean absolute error of 12 is good or bad entirely depending on what a trivial rule achieves. The only way a number becomes interpretable is by comparison, and the comparison must be established before you build anything, because afterwards you will be motivated to pick a flattering one.

Chapter 1.2 introduced baselines as part of the modelling stage. The point worth stressing here is that a baseline is not a formality. It regularly reveals that a complex model adds nothing, which is a valuable finding delivered cheaply, and it occasionally reveals a bug, because a model that cannot beat a constant is usually broken rather than merely weak.

The ladder of baselines, from trivial to demanding:

Constant predictions. For regression, predict the training mean, or the median if the target is skewed. For classification, always predict the majority class. These establish the absolute floor.

Random predictions matching the class distribution. Useful for confirming that your metrics behave as expected, which is worth checking.

Single-feature rules. A decision stump on the strongest feature, or a simple threshold a domain expert would suggest. This is frequently a strong baseline and it is often what the business is already doing.

Simple models. Logistic or linear regression on a handful of obvious features. This is the baseline that matters most, because a gradient boosted ensemble that barely beats it is not worth its operational cost.

The existing system. If a rules engine or a human process is in production, its performance is the real bar. Beating a constant is irrelevant when the alternative is a system that already works.

Domain-specific naive rules. For time series, predicting the last observed value is a genuinely hard baseline to beat, and a seasonal version, predicting the same period last year, is harder still. Many forecasting projects fail to beat it. For recommendation, "most popular item" is notoriously strong. For churn, "hasn't logged in for 30 days" is often close to a trained model's performance.

Three practices make baselines useful rather than ceremonial.

Evaluate baselines with the same pipeline and splits. A baseline scored on different data or without the same preprocessing is not a comparison.

Report the lift, not just the scores. "0.84 versus a 0.79 baseline" is more informative than either number, and converting the gap into business terms is what makes it actionable, as chapter 8.4 covers.

Ask whether the lift justifies the cost. A complex model carries training cost, inference cost, monitoring burden, drift risk, and interpretability loss. A one percent gain rarely pays for all of that, and saying so is part of the job.

import numpy as np
import pandas as pd
from sklearn.dummy import DummyClassifier, DummyRegressor
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression, LinearRegression
from sklearn.ensemble import (RandomForestClassifier, GradientBoostingClassifier,
                              GradientBoostingRegressor)
from sklearn.model_selection import cross_val_score, train_test_split, KFold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_classification
from sklearn.metrics import (accuracy_score, roc_auc_score, f1_score,
                             mean_absolute_error, matthews_corrcoef)

rng = np.random.default_rng(71)

# --- WHY A SCORE ALONE MEANS NOTHING ---
print("THE SAME ACCURACY, THREE DIFFERENT VERDICTS")
for majority in (0.50, 0.75, 0.90):
    verdict = ("excellent" if majority < 0.6
               else "decent" if majority < 0.8 else "WORSE THAN A CONSTANT")
    print(f"  majority class {majority:.0%}: a model at 0.85 accuracy is"
          f" {verdict}  (constant baseline = {majority:.2f})")

# --- THE BASELINE LADDER, on a classification problem. ---
X, y = make_classification(n_samples=8_000, n_features=14, n_informative=5,
                           weights=[0.78, 0.22], flip_y=0.03, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                      stratify=y)

def evaluate(name, model, Xtr=Xtr, ytr=ytr, Xte=Xte, yte=yte):
    m = model.fit(Xtr, ytr)
    p = m.predict(Xte)
    proba = (m.predict_proba(Xte)[:, 1] if hasattr(m, "predict_proba")
             else p.astype(float))
    return {"model": name,
            "accuracy": round(accuracy_score(yte, p), 4),
            "AUC": round(roc_auc_score(yte, proba), 4),
            "F1": round(f1_score(yte, p, zero_division=0), 4),
            "MCC": round(matthews_corrcoef(yte, p), 4)}

# A single-feature rule: a depth-1 tree on the best feature.
strongest = int(np.argmax(np.abs([np.corrcoef(Xtr[:, j], ytr)[0, 1]
                                 for j in range(Xtr.shape[1])])))
ladder = [
    evaluate("0. constant (majority)", DummyClassifier(strategy="most_frequent")),
    evaluate("1. random (stratified)", DummyClassifier(strategy="stratified",
                                                       random_state=0)),
    evaluate(f"2. one-feature stump (f{strongest})",
             DecisionTreeClassifier(max_depth=1, random_state=0)),
    evaluate("3. depth-3 tree", DecisionTreeClassifier(max_depth=3,
                                                       random_state=0)),
    evaluate("4. logistic regression",
             make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))),
    evaluate("5. random forest",
             RandomForestClassifier(n_estimators=300, random_state=0)),
    evaluate("6. gradient boosting",
             GradientBoostingClassifier(random_state=0)),
]
lad = pd.DataFrame(ladder)
print("\nTHE BASELINE LADDER")
print(lad.to_string(index=False))

# --- LIFT OVER THE RIGHT BASELINE, not over zero. ---
constant = lad.iloc[0]
simple = lad[lad["model"].str.startswith("4.")].iloc[0]
best = lad.iloc[-1]
print(f"\nLIFT ANALYSIS")
print(f"  gradient boosting vs CONSTANT:  accuracy"
      f" {best['accuracy']-constant['accuracy']:+.4f}")
print(f"  gradient boosting vs LOGISTIC:  accuracy"
      f" {best['accuracy']-simple['accuracy']:+.4f}"
      f"   AUC {best['AUC']-simple['AUC']:+.4f}")
print("\nThe first number looks impressive and is the wrong comparison. The")
print("second is the one that decides whether the complex model is worth it.")

# --- WHEN THE COMPLEX MODEL ADDS NOTHING: a genuinely linear problem. ---
print("\nA GENUINELY ADDITIVE PROBLEM: does complexity pay?")
n = 6_000
Xl = rng.normal(size=(n, 8))
yl = (Xl @ np.array([1.4, -1.1, 0.8, -0.6, 0.4, 0, 0, 0])
      + rng.normal(0, 1.0, n) > 0).astype(int)
cv = KFold(5, shuffle=True, random_state=0)
for label, model in [("logistic regression",
                      make_pipeline(StandardScaler(),
                                    LogisticRegression(max_iter=2000))),
                     ("random forest",
                      RandomForestClassifier(n_estimators=300, random_state=0)),
                     ("gradient boosting",
                      GradientBoostingClassifier(random_state=0))]:
    s = cross_val_score(model, Xl, yl, cv=cv, scoring="roc_auc")
    print(f"  {label:22s} AUC {s.mean():.4f} +/- {s.std():.4f}")
print("The ensembles do not beat logistic regression, because the structure IS")
print("additive. Chapter 9.1's point about inductive bias, seen as a baseline")
print("comparison: this is a diagnostic result, not a failure.")

# --- TIME SERIES: the naive baselines that are hard to beat. ---
T = 730
t = np.arange(T)
level = 100 + 0.05 * t
seasonal = 12 * np.sin(2 * np.pi * t / 7)                # weekly pattern
series = level + seasonal + rng.normal(0, 4, T)
split = 650
train, test = series[:split], series[split:]

def naive_last(train, h):
    return np.full(h, train[-1])

def naive_seasonal(train, h, period=7):
    reps = int(np.ceil(h / period))
    return np.tile(train[-period:], reps)[:h]

def naive_drift(train, h):
    slope = (train[-1] - train[0]) / (len(train) - 1)
    return train[-1] + slope * np.arange(1, h + 1)

h = len(test)
# A "real" model: gradient boosting on lag features.
lags = 14
rows = np.array([series[i-lags:i] for i in range(lags, split)])
targets = series[lags:split]
gb = GradientBoostingRegressor(random_state=0).fit(rows, targets)
preds, window = [], list(series[split-lags:split])
for _ in range(h):
    nxt = gb.predict(np.array(window[-lags:]).reshape(1, -1))[0]
    preds.append(nxt)
    window.append(nxt)

print("\nFORECASTING BASELINES (80-day horizon, weekly seasonality)")
print(f"  {'method':<28} {'MAE':>8} {'vs naive-last':>15}")
mae_last = mean_absolute_error(test, naive_last(train, h))
for label, p in [("naive: last value", naive_last(train, h)),
                 ("naive: seasonal (7-day)", naive_seasonal(train, h)),
                 ("naive: drift", naive_drift(train, h)),
                 ("gradient boosting on lags", np.array(preds))]:
    mae = mean_absolute_error(test, p)
    print(f"  {label:<28} {mae:>8.3f} {mae/mae_last:>14.2f}x")
print("\nThe seasonal naive forecast is genuinely hard to beat here. Many")
print("forecasting projects ship a model that loses to it and nobody checks.")

# --- POPULARITY: the recommender baseline that keeps winning. ---
n_users, n_items = 1_200, 400
popularity = rng.zipf(1.6, n_items).astype(float)
popularity /= popularity.sum()
true_pref = rng.random((n_users, n_items)) * 0.3 + popularity * 0.7
interactions = (rng.random((n_users, n_items)) < true_pref * 0.15).astype(int)
train_i = interactions.copy()
held = np.zeros_like(interactions)
for u in range(n_users):                          # hold out one item per user
    pos = np.flatnonzero(interactions[u])
    if len(pos):
        pick = rng.choice(pos)
        train_i[u, pick] = 0
        held[u, pick] = 1

pop_score = train_i.sum(axis=0)
K = 10
print("\nRECOMMENDER BASELINES (hit rate at 10)")
hits_pop, hits_rand = 0, 0
for u in range(n_users):
    if held[u].sum() == 0:
        continue
    seen = train_i[u] > 0
    cand = np.flatnonzero(~seen)
    top_pop = cand[np.argsort(-pop_score[cand])[:K]]
    top_rand = rng.choice(cand, K, replace=False)
    hits_pop += held[u, top_pop].sum()
    hits_rand += held[u, top_rand].sum()
n_eval = int((held.sum(axis=1) > 0).sum())
print(f"  random items:     hit rate {hits_rand/n_eval:.4f}")
print(f"  MOST POPULAR:     hit rate {hits_pop/n_eval:.4f}"
      f"  ({hits_pop/max(hits_rand,1):.1f}x random)")
print("'Most popular' is the baseline every recommender must beat, and it is")
print("stronger than people expect because popularity is genuinely predictive.")

# --- THE LIFT-VERSUS-COST TABLE, which is the actual deliverable. ---
decision = pd.DataFrame([
    ("constant", 0.780, 0, 0, "none"),
    ("logistic regression", 0.869, 2, 1, "coefficients, fully explainable"),
    ("random forest", 0.881, 8, 12, "needs SHAP or similar"),
    ("gradient boosting", 0.886, 14, 18, "needs SHAP or similar"),
    ("stacked ensemble", 0.889, 40, 60, "effectively opaque"),
], columns=["model", "AUC", "build_days", "monthly_ops_hours",
            "interpretability"])
decision["lift_vs_logistic"] = (decision["AUC"] - 0.869).round(4)
decision["auc_per_ops_hour"] = (decision["lift_vs_logistic"]
                                / decision["monthly_ops_hours"].replace(0, np.nan)
                                ).round(5)
print("\nLIFT VERSUS COST (the table to put in front of a decision maker)")
print(decision.to_string(index=False))
print("\nThe stacked ensemble buys 0.0200 AUC over logistic regression for 60")
print("ops hours a month and full loss of interpretability. Whether that is")
print("worth it is a business question, and presenting it as one is the job.")

  A SCORE IS MEANINGLESS WITHOUT ITS BASELINE

    accuracy 0.85, majority class 50%   -> excellent
    accuracy 0.85, majority class 75%   -> decent
    accuracy 0.85, majority class 90%   -> WORSE THAN A CONSTANT

  THE LADDER

  level  baseline                        typical AUC on the worked example
  0      constant / majority class       0.5000
  1      random, matching class rates    0.5021
  2      one-feature stump               0.7134
  3      depth-3 tree                    0.8402
  4      LOGISTIC REGRESSION             0.9231   <- the bar that matters
  5      random forest                   0.9388
  6      gradient boosting               0.9421

    lift vs CONSTANT:  +0.442   (impressive and irrelevant)
    lift vs LOGISTIC:  +0.019   (the number that decides)

  DOMAIN-SPECIFIC NAIVE RULES THAT ARE HARD TO BEAT

  domain           the baseline                     why it is strong
  ---------------------------------------------------------------------------
  time series      last observed value              series are persistent
  seasonal series  same period last cycle           seasonality dominates
  recommendation   most popular item                popularity is predictive
  churn            "inactive for 30 days"           recency is most of it
  credit           existing scorecard               already tuned for years
  any              the system in production         the real alternative

  FORECASTING, MEASURED (80-day horizon, weekly seasonality)

    method                        MAE    vs naive-last
    naive: last value           9.412         1.00x
    naive: seasonal (7-day)     3.688         0.39x   <- hard to beat
    naive: drift                9.221         0.98x
    gradient boosting on lags   4.107         0.44x   <- LOSES to seasonal

    A boosted model on lag features loses to a one-line seasonal rule. This
    happens constantly and goes unnoticed without the baseline.

  THE DELIVERABLE: LIFT VERSUS COST

  model                AUC     lift vs      build   ops hrs   interpretable
                               logistic     days    /month
  logistic regression  0.869   +0.0000        2        1       fully
  random forest        0.881   +0.0120        8       12       needs SHAP
  gradient boosting    0.886   +0.0170       14       18       needs SHAP
  stacked ensemble     0.889   +0.0200       40       60       opaque

    0.02 AUC for 60 ops hours a month and no explainability is a BUSINESS
    decision. Presenting it as one is part of the job.

Key Takeaways

• A score is uninterpretable without a baseline, and the baseline must be established before modelling so the comparison is not chosen to flatter.
• The baseline that matters is a simple model or the existing production system, not a constant prediction.
• Domain-specific naive rules such as the seasonal last value or most popular item are far stronger than people expect and are frequently not beaten.
• Evaluate baselines through the same pipeline and splits, or the comparison is invalid.
• Report lift against the right baseline alongside the operational cost, because whether a small gain justifies the complexity is a business decision.

🧪 Practice

1. Build a full baseline ladder from a constant prediction to a boosted ensemble, and report the lift of the best model over both the constant and the simple model.
2. Construct a genuinely additive problem and show that ensembles fail to beat logistic regression.
3. Implement last-value, seasonal, and drift forecasting baselines and compare a trained model against all three.
4. Build a most-popular-item recommender baseline and compute its hit rate at ten against random recommendation.
5. Produce a lift-versus-cost table for three models, including build effort and ongoing operational burden.
6. Interview: Your new model achieves 0.91 area under the curve and the team is ready to ship. What do you ask for first? (Hint: what does the thing it replaces score on the same split, and what would a single-feature rule get?)

<a id="94-model-selection-and-tuning"></a>

9.4 Model Selection and Tuning

Choosing among models and settings is itself a fitting procedure, and it overfits just as readily as the models it selects; this section covers the search methods and the validation discipline that keeps the final estimate honest.

Hyperparameters vs Parameters

Two kinds of number determine a model's behaviour, and conflating them produces both conceptual confusion and real leakage bugs.

Parameters are learned from the training data by the fitting procedure. The coefficients of a linear regression, the split thresholds in a decision tree, the weights of a neural network. You do not choose these; the optimiser finds them by minimising the loss. There are usually many of them, often millions.

Hyperparameters are set before fitting and control how the fitting happens. The regularisation strength, the tree depth, the learning rate, the number of neighbours, the network architecture. The fitting procedure cannot learn them, because for most of them the training loss moves monotonically in one direction: a deeper tree always fits training data better, so minimising training loss would always choose unlimited depth. There are usually few of them, from a handful to a few dozen.

The distinguishing test is clean: can the training loss select it? If minimising training error would pick the right value, it is a parameter. If minimising training error would pick an extreme value that overfits, it is a hyperparameter and needs held-out data to choose.

That last clause is the practical consequence. Because hyperparameters are chosen by looking at validation performance, the search is itself a fitting procedure, and validation performance after a search is optimistically biased for exactly the reason chapter 9.1 gives for test sets. The remedy is nested validation, covered later in this section.

A few categories are worth separating because they behave differently in search.

Continuous hyperparameters such as regularisation strength or learning rate usually matter on a logarithmic scale: the difference between 0.001 and 0.01 is far more consequential than between 0.1 and 0.11. Always sample them log-uniformly.

Integer hyperparameters such as tree depth or the number of neighbours are usually coarse, and adjacent values often perform indistinguishably.

Categorical hyperparameters such as the kernel type or the solver change the model's behaviour qualitatively, and they often make other hyperparameters meaningless: a linear kernel has no gamma.

Conditional hyperparameters only apply given another choice, which is why a search space is really a tree rather than a grid.

Finally, some hyperparameters barely matter and some dominate. Learning rate and regularisation strength typically dominate; the choice of solver or the random seed typically does not. Spending your budget on the ones that matter is most of what separates effective tuning from expensive tuning, and a quick sensitivity check tells you which is which.

import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score, KFold, train_test_split
from sklearn.linear_model import LogisticRegression, Ridge
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(73)

# --- PARAMETERS ARE LEARNED; HYPERPARAMETERS ARE CHOSEN. ---
X, y = make_classification(n_samples=4_000, n_features=20, n_informative=8,
                           random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                      stratify=y)

model = LogisticRegression(C=0.1, max_iter=2000).fit(Xtr, ytr)
print("LOGISTIC REGRESSION")
print(f"  hyperparameters SET by you: C={model.C}, penalty={model.penalty},"
      f" solver={model.solver}")
print(f"  parameters LEARNED from data: {model.coef_.size + 1} numbers")
print(f"    first three coefficients: {np.round(model.coef_[0][:3], 4)}")

tree = DecisionTreeClassifier(max_depth=4, min_samples_leaf=20,
                              random_state=0).fit(Xtr, ytr)
print("\nDECISION TREE")
print(f"  hyperparameters: max_depth={tree.max_depth},"
      f" min_samples_leaf={tree.min_samples_leaf}")
print(f"  parameters: {tree.tree_.node_count} nodes, each with a learned"
      f" feature index and threshold")

gb = GradientBoostingClassifier(n_estimators=50, learning_rate=0.1,
                                max_depth=3, random_state=0).fit(Xtr, ytr)
total_nodes = sum(est[0].tree_.node_count for est in gb.estimators_)
print("\nGRADIENT BOOSTING")
print(f"  hyperparameters: 3 that you chose")
print(f"  parameters: {total_nodes:,} learned split thresholds across"
      f" {len(gb.estimators_)} trees")

# --- THE TEST: CAN TRAINING LOSS CHOOSE IT? ---
print("\nWHY TRAINING LOSS CANNOT SELECT A HYPERPARAMETER")
print(f"  {'max_depth':>10} {'train accuracy':>16} {'CV accuracy':>13}")
for depth in (1, 3, 5, 10, 20, None):
    t = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(Xtr, ytr)
    cv = cross_val_score(DecisionTreeClassifier(max_depth=depth, random_state=0),
                         Xtr, ytr, cv=5).mean()
    label = "none" if depth is None else str(depth)
    print(f"  {label:>10} {t.score(Xtr, ytr):>16.4f} {cv:>13.4f}")
print("\nTraining accuracy rises MONOTONICALLY with depth, so minimising")
print("training loss would always choose unlimited depth. Cross-validated")
print("accuracy peaks and falls. That asymmetry IS the definition.")

# Contrast: a parameter, where training loss is the correct criterion.
print("\nBY CONTRAST, a PARAMETER: the loss has an interior minimum")
Xs = rng.normal(size=(500, 1))
ys = 3.0 * Xs.ravel() + rng.normal(0, 1, 500)
print(f"  {'candidate slope':>16} {'training MSE':>14}")
for slope in (0.0, 1.5, 2.5, 3.0, 3.5, 5.0):
    print(f"  {slope:>16.1f} {np.mean((ys - slope*Xs.ravel())**2):>14.4f}")
print(f"  fitted by least squares: "
      f"{np.linalg.lstsq(Xs, ys, rcond=None)[0][0]:.4f}")
print("The training loss has a genuine minimum, so the optimiser finds it.")

# --- SCALE MATTERS: sample continuous hyperparameters LOG-uniformly. ---
print("\nWHY LOG SCALE FOR REGULARISATION STRENGTH")
print(f"  {'C':>10} {'CV accuracy':>13}  {'linear-scale neighbours':>24}")
for C in (0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0):
    s = cross_val_score(make_pipeline(StandardScaler(),
                                      LogisticRegression(C=C, max_iter=3000)),
                        Xtr, ytr, cv=5).mean()
    print(f"  {C:>10} {s:>13.4f}")
print("\n  A LINEAR grid from 0.0001 to 100 in 10 steps gives:")
linear_grid = np.linspace(0.0001, 100, 10)
print(f"    {np.round(linear_grid, 2).tolist()}")
print("    -> 9 of 10 points are above C=11, and the interesting region")
print("       (0.0001 to 1) gets ONE point.")
log_grid = np.logspace(-4, 2, 10)
print(f"  A LOG grid gives: {np.round(log_grid, 5).tolist()}")
print("    -> even coverage of every order of magnitude.")

# --- WHICH HYPERPARAMETERS ACTUALLY MATTER? ---
print("\nSENSITIVITY: vary one hyperparameter, hold the rest at defaults")
BASE = dict(n_estimators=150, learning_rate=0.1, max_depth=3,
            subsample=1.0, min_samples_leaf=1)
base_score = cross_val_score(GradientBoostingClassifier(random_state=0, **BASE),
                             Xtr, ytr, cv=4, scoring="roc_auc").mean()
print(f"  baseline AUC {base_score:.4f}")
sweeps = {
    "learning_rate": [0.01, 0.05, 0.1, 0.3, 1.0],
    "max_depth": [1, 2, 3, 5, 8],
    "n_estimators": [25, 75, 150, 400],
    "subsample": [0.5, 0.7, 0.9, 1.0],
    "min_samples_leaf": [1, 5, 20, 50],
}
impact = []
for name, values in sweeps.items():
    scores = []
    for v in values:
        kw = dict(BASE)
        kw[name] = v
        scores.append(cross_val_score(
            GradientBoostingClassifier(random_state=0, **kw),
            Xtr, ytr, cv=4, scoring="roc_auc").mean())
    impact.append({"hyperparameter": name,
                   "min_AUC": round(min(scores), 4),
                   "max_AUC": round(max(scores), 4),
                   "range": round(max(scores) - min(scores), 4),
                   "best_value": values[int(np.argmax(scores))]})
imp = pd.DataFrame(impact).sort_values("range", ascending=False)
print(imp.to_string(index=False))
print("\nSpend the search budget on the top rows. The bottom rows barely move")
print("the score, and tuning them is how a search wastes a day.")

# --- CONDITIONAL HYPERPARAMETERS: the space is a TREE, not a grid. ---
print("\nCONDITIONAL STRUCTURE (an SVM's space)")
space = {
    "kernel=linear": ["C"],
    "kernel=rbf": ["C", "gamma"],
    "kernel=poly": ["C", "gamma", "degree", "coef0"],
}
for k, params in space.items():
    print(f"  {k:<16} relevant: {params}")
print("\n  gamma is MEANINGLESS for a linear kernel, so a naive full grid")
print("  wastes evaluations on duplicate configurations:")
n_C, n_gamma, n_degree = 5, 5, 3
naive = 3 * n_C * n_gamma * n_degree
smart = n_C + n_C*n_gamma + n_C*n_gamma*n_degree
print(f"    naive full grid:     {naive} configurations")
print(f"    respecting structure: {smart} configurations"
      f"  ({1 - smart/naive:.0%} fewer)")

# --- THE KEY CONSEQUENCE: a tuned validation score is biased. ---
print("\nTUNING BIASES THE VALIDATION SCORE IT SELECTED ON")
noise_X = rng.normal(size=(300, 40))
noise_y = rng.integers(0, 2, 300)               # NO signal at all
cv = KFold(5, shuffle=True, random_state=0)
best_score, tried = -np.inf, 0
for C in np.logspace(-4, 3, 12):
    for depth in (1, 2, 3, 5, 10):
        tried += 1
        s = cross_val_score(DecisionTreeClassifier(max_depth=depth,
                                                   random_state=0),
                            noise_X, noise_y, cv=cv).mean()
        best_score = max(best_score, s)
print(f"  {tried} configurations searched on data with NO signal")
print(f"  best cross-validated accuracy found: {best_score:.4f}"
      f"   (truth is 0.5000)")
print("\nThe search found 'skill' in noise, because selecting the maximum of")
print("many noisy estimates selects for luck. The score you tuned on is not")
print("an unbiased estimate of anything, which is why nested CV exists.")

  THE DISTINCTION

  PARAMETERS                        HYPERPARAMETERS
  learned from data                 set before fitting
  found by the optimiser            chosen by you or a search
  usually many (10s to millions)    usually few (1 to ~30)
  selected by TRAINING loss         selected by VALIDATION performance

    linear regression, 20 features:  21 parameters,  3 hyperparameters
    decision tree, depth 4:          31 nodes,       2 hyperparameters
    gradient boosting, 50 trees:     1,247 splits,   3 hyperparameters

  THE TEST: CAN TRAINING LOSS CHOOSE IT?

    max_depth   train accuracy   CV accuracy
    1              0.7219          0.7186
    3              0.8404          0.8290
    5              0.8981          0.8548
    10             0.9857          0.8341
    20             1.0000          0.8256
    none           1.0000          0.8241
                        ^               ^
              monotone: would    peaks at 5: needs
              always pick 'none' HELD-OUT data

    A parameter's training loss has an interior minimum. A hyperparameter's
    does not, which is exactly why it cannot be learned from training data.

  SAMPLE CONTINUOUS HYPERPARAMETERS ON A LOG SCALE

    linear grid, 0.0001 to 100 in 10 steps:
      0.0001, 11.11, 22.22, 33.33, 44.44, 55.56, 66.67, 77.78, 88.89, 100
      -> ONE point below 11, and the interesting region is below 1

    log grid, 1e-4 to 1e2 in 10 steps:
      0.0001, 0.0005, 0.0022, 0.01, 0.046, 0.215, 1.0, 4.6, 21.5, 100
      -> even coverage of every order of magnitude

  NOT ALL HYPERPARAMETERS MATTER EQUALLY

    hyperparameter      min AUC   max AUC   range    best
    learning_rate        0.8714    0.9402   0.0688   0.05
    max_depth            0.9021    0.9402   0.0381   3
    n_estimators         0.9188    0.9411   0.0223   400
    subsample            0.9352    0.9404   0.0052   0.9
    min_samples_leaf     0.9371    0.9402   0.0031   1

    The top two move the score 10x more than the bottom two. Budget
    accordingly.

  THE SEARCH SPACE IS A TREE

    kernel=linear  ->  C
    kernel=rbf     ->  C, gamma
    kernel=poly    ->  C, gamma, degree, coef0

    naive full grid:      225 configurations
    respecting structure:  105 configurations   (53% fewer)

  AND THE CRITICAL CONSEQUENCE

    60 configurations searched on data with NO signal
    best cross-validated accuracy found: 0.5933   (truth: 0.5000)

    Selecting the maximum of many noisy estimates selects for LUCK. The
    score you tuned on is optimistically biased -> use nested CV.

Key Takeaways

• Parameters are learned by minimising training loss; hyperparameters cannot be, because training loss moves monotonically in their extreme direction.
• The test is whether minimising training error would select the right value, and if not, the value needs held-out data.
• Sample continuous hyperparameters log-uniformly, since orders of magnitude matter more than linear increments.
• A few hyperparameters dominate performance and the rest barely matter, so run a sensitivity check before spending a search budget.
• Tuning is a fitting procedure, so the validation score you selected on is optimistically biased and can find skill in pure noise.

🧪 Practice

1. Print the count of learned parameters and chosen hyperparameters for a linear model, a tree, and a boosted ensemble.
2. Show that training accuracy rises monotonically with tree depth while cross-validated accuracy peaks, and explain what that proves.
3. Compare a linear and a logarithmic grid over a regularisation strength and count how many points each places in the useful region.
4. Run a one-at-a-time sensitivity sweep over five hyperparameters and rank them by the range of scores produced.
5. Search sixty configurations on a target with no signal and report the best cross-validated score found.
6. Interview: A colleague tunes a model's depth by picking the value with the lowest training error. What will they choose and why is it wrong? (Hint: what does training error do as depth increases without limit?)

Grid and Random Search

Once you know which hyperparameters matter, you need a procedure for searching them. The two classical approaches are exhaustive and random, and the comparison between them is one of the more counterintuitive results in applied machine learning.

Grid search evaluates every combination of a specified set of values. It is exhaustive over the grid, trivially parallel, reproducible, and easy to explain. Its fatal weakness is that cost multiplies: four hyperparameters with five values each is 625 configurations, and with five-fold cross-validation that is 3,125 model fits. Adding one more hyperparameter multiplies the cost again. This is the curse of dimensionality from 9.2 applied to a search space.

Random search samples configurations from distributions over the ranges. Bergstra and Bengio's 2012 result is that it is usually better than grid search for the same budget, and the reason is worth understanding properly because it is not obvious.

The argument has two parts. First, only a few hyperparameters matter, as the previous topic showed. A grid with five values per hyperparameter tries only five distinct values of the important one, no matter how many total configurations it evaluates, because the rest of the grid varies parameters that do not change the score. Random search with 100 samples tries 100 distinct values of the important one. It spends the same budget and explores the dimension that matters twenty times more finely.

Second, the probability argument: if the top 5% of the space is good enough, then a single random sample misses it with probability 0.95, and 60 samples miss it with probability 0.95 to the 60th power, which is about 4.6%. So 60 random samples find a top-5% configuration 95% of the time, regardless of how many dimensions the space has. That bound is dimension-free, which is exactly what grid search cannot claim.

Practical guidance for both:

Search log-uniformly over continuous scale parameters, as covered above.

Start coarse, then refine. A wide random search locates the good region cheaply, and a focused search inside it refines. Two rounds usually beat one exhaustive pass.

Use the one-standard-error rule. Among configurations within one standard error of the best, choose the simplest. The best score is partly luck, and the simpler model generalises more reliably.

Prune bad configurations early. Successive halving allocates a small budget to many configurations, discards the worst, and re-allocates to the survivors. This is often several times more efficient than either plain approach, and is available as halving grid and halving random search.

Parallelise, and mind the cost accounting. Both methods are embarrassingly parallel. The real cost is fits times folds, and it is worth computing before launching.

import numpy as np
import pandas as pd
import time
from sklearn.datasets import make_classification
from sklearn.model_selection import (GridSearchCV, RandomizedSearchCV,
                                     StratifiedKFold, train_test_split,
                                     cross_val_score)
from sklearn.experimental import enable_halving_search_cv  # noqa: F401
from sklearn.model_selection import HalvingRandomSearchCV
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.svm import SVC
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from scipy.stats import loguniform, randint, uniform

rng = np.random.default_rng(79)

X, y = make_classification(n_samples=3_000, n_features=20, n_informative=8,
                           flip_y=0.05, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25, random_state=0,
                                      stratify=y)
cv = StratifiedKFold(4, shuffle=True, random_state=0)

# --- THE COST OF A GRID MULTIPLIES ---
print("GRID SEARCH COST GROWS MULTIPLICATIVELY")
print(f"  {'hyperparameters':>16} {'values each':>12} {'configs':>10}"
      f" {'fits (4-fold)':>14}")
for n_hp in (1, 2, 3, 4, 5, 6):
    for vals in (5,):
        configs = vals ** n_hp
        print(f"  {n_hp:>16} {vals:>12} {configs:>10,} {configs*4:>14,}")
print("\nSix hyperparameters at five values each is 62,500 fits. This is the")
print("curse of dimensionality applied to a search space.")

# --- THE KEY INSIGHT: a grid explores the IMPORTANT dimension coarsely. ---
print("\nDISTINCT VALUES TRIED OF THE ONE HYPERPARAMETER THAT MATTERS")
print(f"  {'budget':>8} {'grid (4 hp x k values)':>24} {'random search':>15}")
for budget in (16, 81, 256, 625):
    k = round(budget ** 0.25)
    print(f"  {budget:>8} {f'{k} distinct values':>24} {budget:>15}")
print("\nA 625-point grid over 4 hyperparameters tries FIVE values of the")
print("important one. Random search with the same budget tries 625.")

# --- THE PROBABILITY ARGUMENT, and it is dimension-free. ---
print("\nP(FINDING A TOP-q CONFIGURATION) WITH n RANDOM SAMPLES")
print(f"  {'samples':>8}" + "".join(f"{f'top {q:.0%}':>10}"
                                    for q in (0.01, 0.05, 0.10)))
for n in (10, 30, 60, 100, 300):
    row = f"  {n:>8}"
    for q in (0.01, 0.05, 0.10):
        row += f"{1 - (1-q)**n:>10.3f}"
    print(row)
print("\n60 random samples find a top-5% configuration 95% of the time, and")
print("that bound does not depend on the number of dimensions at all.")

# --- HEAD TO HEAD ON THE SAME BUDGET ---
BUDGET = 48
grid_space = {
    "learning_rate": [0.01, 0.05, 0.1, 0.3],
    "max_depth": [2, 3, 4],
    "subsample": [0.7, 1.0],
    "min_samples_leaf": [1, 10],
}
n_grid = int(np.prod([len(v) for v in grid_space.values()]))
random_space = {
    "learning_rate": loguniform(0.005, 0.5),     # LOG scale: correct here
    "max_depth": randint(2, 7),
    "subsample": uniform(0.5, 0.5),              # 0.5 to 1.0
    "min_samples_leaf": randint(1, 40),
}
print(f"\nHEAD TO HEAD (grid has {n_grid} configs, random gets {BUDGET})")
results = []
for label, search in [
        ("grid search", GridSearchCV(
            GradientBoostingClassifier(n_estimators=120, random_state=0),
            grid_space, cv=cv, scoring="roc_auc", n_jobs=-1)),
        ("random search", RandomizedSearchCV(
            GradientBoostingClassifier(n_estimators=120, random_state=0),
            random_space, n_iter=BUDGET, cv=cv, scoring="roc_auc",
            random_state=0, n_jobs=-1)),
        ("halving random", HalvingRandomSearchCV(
            GradientBoostingClassifier(n_estimators=120, random_state=0),
            random_space, n_candidates=120, cv=cv, scoring="roc_auc",
            random_state=0, n_jobs=-1, factor=3))]:
    t0 = time.perf_counter()
    search.fit(Xtr, ytr)
    elapsed = time.perf_counter() - t0
    from sklearn.metrics import roc_auc_score
    test_auc = roc_auc_score(yte, search.best_estimator_.predict_proba(Xte)[:, 1])
    results.append({"method": label,
                    "configs_evaluated": len(search.cv_results_["params"]),
                    "best_cv_auc": round(search.best_score_, 4),
                    "test_auc": round(test_auc, 4),
                    "seconds": round(elapsed, 1)})
print(pd.DataFrame(results).to_string(index=False))
print("\nRandom search matches or beats the grid at a comparable budget, and")
print("halving does it faster by discarding weak candidates early.")

# --- WHY: random search explores the important dimension finely. ---
rs = RandomizedSearchCV(GradientBoostingClassifier(n_estimators=120,
                                                   random_state=0),
                        random_space, n_iter=BUDGET, cv=cv, scoring="roc_auc",
                        random_state=0, n_jobs=-1).fit(Xtr, ytr)
res = pd.DataFrame(rs.cv_results_["params"])
res["score"] = rs.cv_results_["mean_test_score"]
print(f"\ndistinct learning_rate values tried by random search:"
      f" {res['learning_rate'].nunique()}")
print(f"distinct learning_rate values in the grid:"
      f" {len(grid_space['learning_rate'])}")
print("\ncorrelation of each hyperparameter with the score:")
for c in ("learning_rate", "max_depth", "subsample", "min_samples_leaf"):
    print(f"  {c:<18} {res[c].astype(float).corr(res['score']):+.3f}")

# --- COARSE THEN FINE: two rounds beat one exhaustive pass. ---
print("\nTWO-STAGE SEARCH")
coarse = RandomizedSearchCV(
    GradientBoostingClassifier(n_estimators=120, random_state=0),
    {"learning_rate": loguniform(1e-3, 1.0), "max_depth": randint(1, 10),
     "min_samples_leaf": randint(1, 60)},
    n_iter=25, cv=cv, scoring="roc_auc", random_state=0, n_jobs=-1).fit(Xtr, ytr)
best = coarse.best_params_
print(f"  coarse ({25} samples, wide ranges): CV AUC {coarse.best_score_:.4f}")
print(f"    best region: {best}")
lr = best["learning_rate"]
fine = RandomizedSearchCV(
    GradientBoostingClassifier(n_estimators=120, random_state=0),
    {"learning_rate": loguniform(lr/3, lr*3),
     "max_depth": randint(max(best["max_depth"]-1, 1), best["max_depth"]+2),
     "min_samples_leaf": randint(max(best["min_samples_leaf"]-5, 1),
                                 best["min_samples_leaf"]+6)},
    n_iter=25, cv=cv, scoring="roc_auc", random_state=1, n_jobs=-1).fit(Xtr, ytr)
print(f"  fine ({25} samples, narrowed):      CV AUC {fine.best_score_:.4f}"
      f"   improvement {fine.best_score_-coarse.best_score_:+.4f}")
print("  50 fits total, split across two rounds, beats 50 in one pass.")

# --- THE ONE-STANDARD-ERROR RULE: prefer the simplest good model. ---
cvres = pd.DataFrame({
    "params": rs.cv_results_["params"],
    "mean": rs.cv_results_["mean_test_score"],
    "std": rs.cv_results_["std_test_score"],
})
cvres["depth"] = [p["max_depth"] for p in cvres["params"]]
best_row = cvres.loc[cvres["mean"].idxmax()]
se = best_row["std"] / np.sqrt(cv.get_n_splits())
within = cvres[cvres["mean"] >= best_row["mean"] - se]
simplest = within.loc[within["depth"].idxmin()]
print(f"\nONE-STANDARD-ERROR RULE")
print(f"  best:     AUC {best_row['mean']:.4f}  depth {best_row['depth']}")
print(f"  1-SE band: AUC >= {best_row['mean']-se:.4f}"
      f"  ({len(within)} of {len(cvres)} configs qualify)")
print(f"  simplest within the band: AUC {simplest['mean']:.4f}"
      f"  depth {simplest['depth']}")
print("  The peak score is partly luck. Among statistically indistinguishable")
print("  options, the simpler one generalises more reliably.")

# --- SUCCESSIVE HALVING, illustrated. ---
print("\nSUCCESSIVE HALVING ALLOCATION (factor 3)")
n_cand, resource = 81, 100
total = 0
while n_cand >= 1:
    total += n_cand * resource
    print(f"  {n_cand:>4} candidates x {resource:>5} samples each"
          f"  = {n_cand*resource:>7,} units")
    if n_cand == 1:
        break
    n_cand = max(n_cand // 3, 1)
    resource *= 3
print(f"  total: {total:,} units")
print(f"  a full search of 81 candidates at 8100 samples:"
      f" {81*8100:,} units ({81*8100/total:.1f}x more)")
print("Weak candidates are eliminated cheaply, and the survivors get the")
print("budget. This is usually the best default for expensive models.")

# --- CONDITIONAL SPACES: use a pipeline and a list of dicts. ---
print("\nSEARCHING ACROSS MODEL FAMILIES (a conditional space)")
pipe = Pipeline([("scale", StandardScaler()), ("clf", SVC())])
spaces = [
    {"clf": [SVC(kernel="linear")], "clf__C": loguniform(1e-2, 1e2)},
    {"clf": [SVC(kernel="rbf")], "clf__C": loguniform(1e-2, 1e2),
     "clf__gamma": loguniform(1e-4, 1e0)},
    {"clf": [GradientBoostingClassifier(random_state=0)],
     "clf__learning_rate": loguniform(1e-2, 5e-1),
     "clf__max_depth": randint(2, 6)},
]
search = RandomizedSearchCV(pipe, spaces, n_iter=30, cv=cv, scoring="roc_auc",
                            random_state=0, n_jobs=-1).fit(Xtr, ytr)
print(f"  best family: {type(search.best_estimator_['clf']).__name__}")
print(f"  best CV AUC: {search.best_score_:.4f}")
print("  A list of dicts encodes the conditional structure, so gamma is only")
print("  sampled where it means something.")

  GRID COST MULTIPLIES

    hyperparameters   values each   configs   fits (4-fold)
    1                 5                   5              20
    2                 5                  25             100
    3                 5                 125             500
    4                 5                 625           2,500
    5                 5               3,125          12,500
    6                 5              15,625          62,500

  THE COUNTERINTUITIVE RESULT: RANDOM BEATS GRID

  the reason, in one picture. only hyperparameter A matters:

  GRID (25 points)                 RANDOM (25 points)
  B  . . . . .                     B   .    .  .
     . . . . .                        .   .      .
     . . . . .                      .    .   .
     . . . . .                        .  .    .   .
     . . . . .                      .    .  .   .
     +---------- A                    +---------- A
     5 distinct values of A           25 distinct values of A

    Same budget. The grid explores the dimension that matters 5x more
    coarsely, because the other 4 axes vary things that do not help.

  THE PROBABILITY ARGUMENT IS DIMENSION-FREE

    P(at least one sample lands in the top q of the space)

    samples    top 1%    top 5%    top 10%
    10          0.096     0.401      0.651
    30          0.260     0.785      0.958
    60          0.453     0.954      0.998
    100         0.634     0.994      1.000
    300         0.951     1.000      1.000

    60 samples -> 95% chance of a top-5% configuration, in ANY number of
    dimensions. Grid search has no such guarantee.

  HEAD TO HEAD

    method           configs   best CV AUC   test AUC   seconds
    grid search           48        0.9361     0.9298      31.4
    random search         48        0.9374     0.9331      30.8
    halving random       120        0.9379     0.9340      14.2

  SUCCESSIVE HALVING (factor 3)

      81 candidates x   100 samples =   8,100 units
      27 candidates x   300 samples =   8,100 units
       9 candidates x   900 samples =   8,100 units
       3 candidates x 2,700 samples =   8,100 units
       1 candidate  x 8,100 samples =   8,100 units
      total                             40,500 units

      full search at max budget:       656,100 units  (16.2x more)

  THE ONE-STANDARD-ERROR RULE

    best config:               AUC 0.9374, depth 6
    within 1 SE (>= 0.9351):   14 of 48 configurations
    SIMPLEST within the band:  AUC 0.9358, depth 2

    The peak is partly luck. Among indistinguishable options, take the
    simplest.

  PRACTICAL DEFAULTS

    log-uniform for scale parameters (learning rate, C, alpha)
    coarse random search, then a focused second round
    halving when fits are expensive
    a list of dicts for conditional spaces
    the 1-SE rule to pick the final configuration

Key Takeaways

• Grid search cost multiplies with each hyperparameter, so it becomes infeasible past three or four.
• Random search usually beats grid search at the same budget because it explores the few important dimensions far more finely.
• Sixty random samples find a top-5% configuration about 95% of the time, and that guarantee does not depend on dimension.
• A coarse search followed by a focused one beats a single exhaustive pass for the same number of fits.
• Successive halving eliminates weak candidates cheaply, and the one- standard-error rule picks the simplest statistically indistinguishable configuration.

🧪 Practice

1. Tabulate grid search cost for one through six hyperparameters at five values each, including the fold multiplier.
2. Compute the probability that n random samples include a top-5% configuration for several values of n.
3. Run grid search and random search at the same budget on the same data and compare both the best cross-validated score and the test score.
4. Implement a two-stage coarse-then-fine random search and report the improvement over a single pass.
5. Apply the one-standard-error rule to a search result and report which configuration it selects instead of the peak.
6. Interview: A colleague plans a grid search over six hyperparameters with five values each and five-fold cross-validation. How many fits is that, and what do you suggest instead? (Hint: what fraction of those fits vary something that changes the score?)

Bayesian Optimization

Grid and random search share a weakness: they are memoryless. Every configuration is chosen without reference to what earlier evaluations revealed. If the first thirty samples all showed that a learning rate above 0.5 performs badly, random search will keep sampling there anyway.

Bayesian optimisation fixes this by building a model of the objective function and using it to choose where to evaluate next. The pieces are:

A surrogate model that predicts validation score as a function of hyperparameters, and crucially reports its own uncertainty. Gaussian processes are the classical choice because their uncertainty estimates are principled. Tree-structured Parzen estimators, used by Optuna and Hyperopt, scale better to larger and conditional spaces. Random forest surrogates are used by SMAC.

An acquisition function that converts the surrogate's prediction and uncertainty into a decision about where to sample. This is where the explore-exploit trade-off from chapter 8.4 reappears in a new setting. Expected improvement is the standard choice, integrating over the probability of beating the current best. Upper confidence bound adds a multiple of the standard deviation to the mean, which is optimism in the face of uncertainty. Probability of improvement is simpler and tends to under-explore.

The loop is: fit the surrogate to all evaluations so far, maximise the acquisition function to pick the next configuration, evaluate it for real, and repeat. Each real evaluation is expensive; maximising the acquisition function is cheap.

The payoff is sample efficiency. On expensive objectives, Bayesian optimisation typically reaches a given score in far fewer evaluations than random search. That matters when a single fit takes an hour.

Knowing when it does not help is equally important.

Cheap objectives. If a fit takes two seconds, the overhead of surrogate fitting and acquisition maximisation is a large fraction of the cost, and random search with more samples is simpler and competitive.

High dimensions. Gaussian process surrogates degrade past roughly twenty dimensions, for the reasons chapter 9.2 gives about distance in high dimensions.

Heavily conditional spaces. Tree-based surrogates handle these; Gaussian processes do not naturally.

Parallel budgets. Bayesian optimisation is inherently sequential, since each choice depends on previous results. With a hundred machines available, random search uses them fully and Bayesian optimisation needs special batch variants.

Very noisy objectives. If cross-validated scores vary by more than the differences between configurations, the surrogate is modelling noise.

In practice, Hyperband and its Bayesian hybrid BOHB are often the best default: they combine early stopping of bad configurations with model-guided selection, which gets both the sample efficiency and the cheap elimination.

import numpy as np
import pandas as pd
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import Matern, ConstantKernel, WhiteKernel
from scipy.stats import norm
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.ensemble import GradientBoostingClassifier

rng = np.random.default_rng(83)

# --- A CHEAP STAND-IN OBJECTIVE, so the mechanics are visible. ---
def objective(x):
    """Pretend this is 'cross-validate a model at hyperparameter x'.
    One global optimum near 0.62, a decoy near 0.15."""
    return (np.sin(3*x) * 0.3 + np.exp(-((x-0.62)**2)/0.02) * 0.6
            + np.exp(-((x-0.15)**2)/0.01) * 0.35)

grid_x = np.linspace(0, 1, 500)
true_best_x = grid_x[np.argmax(objective(grid_x))]
true_best = objective(grid_x).max()
print(f"true optimum: x={true_best_x:.3f}, value={true_best:.4f}")
print("(in reality this function is 'fit a model and cross-validate it')")

# --- THE ACQUISITION FUNCTION: expected improvement. ---
def expected_improvement(mu, sigma, best_so_far, xi=0.01):
    """How much better than the current best do we EXPECT this point to be,
    integrating over the surrogate's uncertainty? Zero-variance points get
    zero EI, so the search is pulled toward uncertain regions."""
    sigma = np.maximum(sigma, 1e-9)
    z = (mu - best_so_far - xi) / sigma
    return (mu - best_so_far - xi) * norm.cdf(z) + sigma * norm.pdf(z)

def bayes_opt(objective, n_init=4, n_iter=16, seed=0):
    r = np.random.default_rng(seed)
    X_obs = r.uniform(0, 1, n_init).reshape(-1, 1)
    y_obs = objective(X_obs.ravel())
    trace = [y_obs.max()]
    kernel = (ConstantKernel(1.0) * Matern(length_scale=0.1, nu=2.5)
              + WhiteKernel(noise_level=1e-4))
    for _ in range(n_iter):
        gp = GaussianProcessRegressor(kernel=kernel, normalize_y=True,
                                      n_restarts_optimizer=2,
                                      random_state=0).fit(X_obs, y_obs)
        mu, sigma = gp.predict(grid_x.reshape(-1, 1), return_std=True)
        ei = expected_improvement(mu, sigma, y_obs.max())
        nxt = grid_x[np.argmax(ei)]
        X_obs = np.vstack([X_obs, [[nxt]]])
        y_obs = np.append(y_obs, objective(nxt))
        trace.append(y_obs.max())
    return X_obs.ravel(), y_obs, np.array(trace)

xs, ys, bo_trace = bayes_opt(objective)
print(f"\nBAYESIAN OPTIMISATION, 20 evaluations")
print(f"  best found: x={xs[np.argmax(ys)]:.3f}, value={ys.max():.4f}"
      f"  ({ys.max()/true_best:.1%} of the optimum)")
print(f"  evaluations spent within 0.05 of the true optimum:"
      f" {int(np.sum(np.abs(xs - true_best_x) < 0.05))} of {len(xs)}")

# --- HEAD TO HEAD AGAINST RANDOM SEARCH ---
print("\nBEST-SO-FAR BY EVALUATION COUNT (mean of 30 runs)")
n_evals = 20
bo_curves, rand_curves = [], []
for s in range(30):
    _, _, tr = bayes_opt(objective, n_init=4, n_iter=n_evals-4, seed=s)
    bo_curves.append(tr)
    r = np.random.default_rng(1000 + s)
    samples = objective(r.uniform(0, 1, n_evals))
    rand_curves.append(np.maximum.accumulate(samples)[3:])
bo_mean = np.mean(bo_curves, axis=0)
rand_mean = np.mean(rand_curves, axis=0)
print(f"  {'evaluations':>12} {'random search':>14} {'Bayesian opt':>14}"
      f" {'advantage':>11}")
for i, n in enumerate([4, 6, 8, 12, 16, 20]):
    idx = min(n - 4, len(bo_mean) - 1)
    print(f"  {n:>12} {rand_mean[idx]:>14.4f} {bo_mean[idx]:>14.4f}"
          f" {bo_mean[idx]-rand_mean[idx]:>+11.4f}")
print(f"\n  evaluations for random search to reach what BO gets in 12: ", end="")
target = bo_mean[min(12-4, len(bo_mean)-1)]
reached = next((i+4 for i, v in enumerate(rand_mean) if v >= target), None)
print(f"{reached if reached else '>20'}")
print("BO concentrates its samples near the promising region instead of")
print("spreading them uniformly. That is what sample efficiency means.")

# --- WHERE DID EACH METHOD SPEND ITS EVALUATIONS? ---
print("\nSAMPLE LOCATIONS (the x axis, 0 to 1; optimum marked ^)")
def render_samples(locations, label):
    slots = [" "] * 50
    for loc in locations:
        idx = min(int(loc * 50), 49)
        slots[idx] = "#" if slots[idx] == " " else "@"
    return f"  {label:<16}|{''.join(slots)}|"
r = np.random.default_rng(1000)
print(render_samples(r.uniform(0, 1, 20), "random search"))
print(render_samples(xs, "Bayesian opt"))
marker = [" "] * 50
marker[min(int(true_best_x * 50), 49)] = "^"
print(f"  {'':<16}|{''.join(marker)}|  (^ = true optimum)")
print("  (# = one evaluation, @ = several)")

# --- THE EXPLORE-EXPLOIT KNOB ---
print("\nACQUISITION FUNCTIONS AND THEIR BEHAVIOUR")
acq = pd.DataFrame([
    ("expected improvement", "E[max(0, f(x) - best)]", "balanced",
     "the standard default"),
    ("upper confidence bound", "mu + kappa * sigma", "kappa controls it",
     "explicit, easy to tune"),
    ("probability of improvement", "P(f(x) > best)", "exploitative",
     "tends to get stuck"),
    ("entropy search", "information about the optimum", "exploratory",
     "expensive to compute"),
], columns=["acquisition", "formula", "explore/exploit", "note"])
print(acq.to_string(index=False))

# UCB with different kappa, to show the knob working.
kernel = ConstantKernel(1.0)*Matern(length_scale=0.1, nu=2.5)+WhiteKernel(1e-4)
gp = GaussianProcessRegressor(kernel=kernel, normalize_y=True,
                              random_state=0).fit(xs.reshape(-1, 1), ys)
mu, sigma = gp.predict(grid_x.reshape(-1, 1), return_std=True)
print("\n  next point chosen by UCB at different exploration weights:")
for kappa in (0.0, 1.0, 3.0, 10.0):
    nxt = grid_x[np.argmax(mu + kappa * sigma)]
    print(f"    kappa {kappa:>5.1f} -> x={nxt:.3f}"
          f"  (surrogate mean {mu[np.argmax(mu + kappa*sigma)]:.3f},"
          f" sd {sigma[np.argmax(mu + kappa*sigma)]:.3f})")
print("  kappa=0 is pure exploitation; large kappa chases uncertainty.")

# --- ON A REAL MODEL, with the cost accounting that decides. ---
X, y = make_classification(n_samples=2_500, n_features=18, n_informative=7,
                           flip_y=0.05, random_state=0)
cv = StratifiedKFold(3, shuffle=True, random_state=0)

def model_objective(lr_log, depth):
    """The real thing: cross-validate a boosted model. Expensive."""
    return cross_val_score(
        GradientBoostingClassifier(learning_rate=10**lr_log,
                                   max_depth=int(round(depth)),
                                   n_estimators=100, random_state=0),
        X, y, cv=cv, scoring="roc_auc").mean()

print("\nON A REAL OBJECTIVE (2 hyperparameters, 14 evaluations)")
bounds = np.array([[-2.5, -0.3], [1, 7]])            # log10(lr), depth
r = np.random.default_rng(5)
obs_X = np.column_stack([r.uniform(*bounds[0], 4), r.uniform(*bounds[1], 4)])
obs_y = np.array([model_objective(*row) for row in obs_X])
cand = np.column_stack([r.uniform(*bounds[0], 3000),
                        r.uniform(*bounds[1], 3000)])
for it in range(10):
    g = GaussianProcessRegressor(
        kernel=ConstantKernel(1.0)*Matern(length_scale=[0.5, 2.0], nu=2.5)
               + WhiteKernel(1e-4),
        normalize_y=True, random_state=0).fit(obs_X, obs_y)
    m, s = g.predict(cand, return_std=True)
    pick = cand[np.argmax(expected_improvement(m, s, obs_y.max()))]
    obs_X = np.vstack([obs_X, pick])
    obs_y = np.append(obs_y, model_objective(*pick))
best_i = int(np.argmax(obs_y))
print(f"  best AUC {obs_y.max():.4f} at learning_rate"
      f" {10**obs_X[best_i,0]:.4f}, depth {int(round(obs_X[best_i,1]))}")
rand_y = [model_objective(*row) for row in
          np.column_stack([r.uniform(*bounds[0], 14), r.uniform(*bounds[1], 14)])]
print(f"  random search, same 14 evaluations: best AUC {max(rand_y):.4f}")

# --- WHEN NOT TO USE IT ---
print("\nWHEN BAYESIAN OPTIMISATION IS THE WRONG TOOL")
guide = pd.DataFrame([
    ("a fit takes < 10 seconds", "random search", "surrogate overhead dominates"),
    ("more than ~20 hyperparameters", "random or halving", "GP surrogates degrade"),
    ("heavily conditional space", "TPE / tree surrogate", "GPs assume continuity"),
    ("100 machines available", "random or Hyperband", "BO is inherently sequential"),
    ("CV noise > config differences", "more folds first", "surrogate models noise"),
    ("a fit takes hours, few hyperparams", "BAYESIAN OPT", "exactly its niche"),
    ("expensive fits, can stop early", "BOHB / Hyperband", "best of both"),
], columns=["situation", "use", "why"])
print(guide.to_string(index=False))

  THE LOOP

    evaluate a few random points
         |
    +----v-----------------------------------+
    |  fit a SURROGATE to all results so far |
    |  (predicts score AND its uncertainty)  |
    |               |                        |
    |  maximise the ACQUISITION function     |  <- cheap
    |               |                        |
    |  evaluate that config for real         |  <- expensive
    +----|-----------------------------------+
         |  repeat
         v
      best found

  WHY IT BEATS RANDOM ON EXPENSIVE OBJECTIVES

    best-so-far by evaluation count (mean of 30 runs)

    evaluations   random search   Bayesian opt   advantage
    4                    0.6821         0.6834     +0.0013
    6                    0.7356         0.8102     +0.0746
    8                    0.7702         0.8914     +0.1212
    12                   0.8104         0.9331     +0.1227
    16                   0.8390         0.9402     +0.1012
    20                   0.8571         0.9418     +0.0847

    random search needs >20 evaluations to reach what BO gets in 12

  WHERE THE SAMPLES GO

    random search   | # #  ##   #  #   # #  #  ## #   #  #  # #  # |
    Bayesian opt    |#  #      #    #        @@@#@@       #     #  |
                    |                        ^                     |
                                        true optimum

    BO concentrates near the promising region; random spreads uniformly.

  ACQUISITION FUNCTIONS

  function                    formula                    behaviour
  ---------------------------------------------------------------------------
  expected improvement        E[max(0, f(x) - best)]     balanced (default)
  upper confidence bound      mu + kappa * sigma         kappa is the knob
  probability of improvement  P(f(x) > best)             exploitative
  entropy search              info about the optimum     exploratory, costly

    UCB's knob, made visible:
      kappa  0.0 -> x=0.622   pure exploitation (the current best region)
      kappa  1.0 -> x=0.618
      kappa  3.0 -> x=0.398   starts probing uncertainty
      kappa 10.0 -> x=0.914   chases the least-explored region

  WHEN NOT TO USE IT

  situation                         use instead        why
  ---------------------------------------------------------------------------
  fits take < 10 seconds            random search      overhead dominates
  > ~20 hyperparameters             random / halving   GP surrogates degrade
  heavily conditional space         TPE (Optuna)       GPs assume continuity
  100 machines available            random / Hyperband BO is SEQUENTIAL
  CV noise > config differences      more folds         it models the noise
  hours per fit, few hyperparams    BAYESIAN OPT       its actual niche
  expensive fits + early stopping   BOHB / Hyperband   best of both

Key Takeaways

• Bayesian optimisation replaces memoryless sampling with a surrogate model that learns from every evaluation.
• An acquisition function such as expected improvement converts predicted score and uncertainty into the next configuration, which is the explore- exploit trade-off again.
• The benefit is sample efficiency, so it pays when a single fit is expensive and the number of hyperparameters is modest.
• It is inherently sequential, degrades past roughly twenty dimensions, and is wasted on objectives cheap enough to sample densely.
• Hybrid methods combining early stopping with model-guided selection are often the best default for expensive tuning.

🧪 Practice

1. Implement expected improvement and run a Bayesian optimisation loop on a cheap one-dimensional objective with a decoy optimum.
2. Compare best-so-far curves for Bayesian optimisation and random search over twenty evaluations, averaged across runs.
3. Visualise or tabulate where each method placed its samples relative to the true optimum.
4. Vary the exploration weight in an upper confidence bound acquisition and show how the chosen next point moves.
5. Tune two hyperparameters of a real model with both methods at the same evaluation budget and compare.
6. Interview: Your team has 200 CPUs and each model fit takes 30 seconds. Would you use Bayesian optimisation? (Hint: what does it need that you cannot give it with 200 machines, and how many random configurations fit in the same wall clock time?)

Nested Cross-Validation

Hyperparameter tuning selects the configuration with the best validation score, and that score is therefore the maximum of many noisy estimates. The maximum of noisy estimates is biased upward, so the best cross-validated score from a search is not an unbiased estimate of the selected model's performance. Reporting it as one is among the most common methodological errors in applied machine learning, and it is invisible: nothing in the output looks wrong.

The mechanism is the same selection effect covered in chapter 8.2's peeking discussion and in 9.1's test-set erosion. Each configuration's cross-validated score has sampling noise. Taking the best of fifty such scores selects partly for configurations that got lucky on these particular folds. The bias grows with the number of configurations tried and shrinks with dataset size.

Nested cross-validation separates the two jobs into two loops.

The inner loop selects hyperparameters, using only the training portion of the current outer fold.

The outer loop estimates performance, evaluating the whole procedure including its tuning on data the inner loop never saw.

The result answers a subtly different and more useful question. It does not estimate "how good is this specific configuration", it estimates "how good is my model-building procedure, including its hyperparameter search". That is what you actually want, because when you retrain on new data you will run the procedure again, not reuse the configuration blindly.

Three points cause confusion and are worth stating plainly.

Nested cross-validation does not produce a final model. It produces an unbiased performance estimate. Once you accept the estimate, you refit the whole procedure on all the data to get the model you deploy, and the outer estimate is your honest claim about it.

The cost is the product of the loops. Five outer folds times five inner folds times fifty configurations is 1,250 fits. This is why people skip it, and why selecting a smaller search space or using halving inside the inner loop matters.

A single held-out test set is a cheaper approximation. Tune with cross-validation on the training portion and evaluate once on a test set never touched during tuning. This is unbiased too, and noisier than nested cross-validation because it rests on one split. For large datasets it is usually the right trade.

When the bias matters most is worth knowing: small datasets, many configurations, and flexible models. With 200 rows and 500 configurations the optimistic bias can exceed several points. With 500,000 rows and 20 configurations it is negligible.

import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import (GridSearchCV, RandomizedSearchCV,
                                     cross_val_score, StratifiedKFold,
                                     train_test_split)
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from scipy.stats import loguniform, randint

rng = np.random.default_rng(89)

# --- THE BIAS, DEMONSTRATED ON PURE NOISE ---
print("SELECTION BIAS ON DATA WITH NO SIGNAL (true accuracy = 0.500)")
print(f"  {'configs tried':>14} {'best inner CV':>15} {'NESTED estimate':>17}")
for n_configs in (1, 5, 20, 80):
    inner_best, nested = [], []
    for rep in range(12):
        r = np.random.default_rng(rep)
        Xn = r.normal(size=(150, 25))
        yn = r.integers(0, 2, 150)
        space = {"max_depth": list(range(1, 1 + min(n_configs, 12))),
                 "min_samples_leaf": [1, 3, 5, 10, 20][:max(1, n_configs // 12 + 1)]}
        gs = GridSearchCV(DecisionTreeClassifier(random_state=0), space,
                          cv=StratifiedKFold(4, shuffle=True, random_state=0))
        gs.fit(Xn, yn)
        inner_best.append(gs.best_score_)
        # The outer loop scores the WHOLE search procedure on unseen folds.
        nested.append(cross_val_score(
            gs, Xn, yn, cv=StratifiedKFold(4, shuffle=True, random_state=1)).mean())
    print(f"  {n_configs:>14} {np.mean(inner_best):>15.4f}"
          f" {np.mean(nested):>17.4f}")
print("\nThe inner score rises with the number of configurations tried, on data")
print("with NO signal. The nested estimate stays near 0.50, which is correct.")
print("The gap IS the selection bias, and it grows with search size.")

# --- THE SAME EFFECT ON REAL DATA ---
X, y = make_classification(n_samples=400, n_features=25, n_informative=6,
                           flip_y=0.10, random_state=0)
space = {"svc__C": loguniform(1e-2, 1e3), "svc__gamma": loguniform(1e-4, 1e0)}
pipe = make_pipeline(StandardScaler(), SVC())

inner_cv = StratifiedKFold(4, shuffle=True, random_state=0)
outer_cv = StratifiedKFold(5, shuffle=True, random_state=1)

print("\nREAL DATA (n=400): INNER SCORE VS NESTED ESTIMATE")
print(f"  {'n_iter':>8} {'best inner CV':>15} {'nested estimate':>17}"
      f" {'optimism':>10} {'fits':>8}")
for n_iter in (5, 20, 60):
    search = RandomizedSearchCV(pipe, space, n_iter=n_iter, cv=inner_cv,
                                random_state=0, n_jobs=-1)
    search.fit(X, y)
    nested = cross_val_score(
        RandomizedSearchCV(pipe, space, n_iter=n_iter, cv=inner_cv,
                           random_state=0, n_jobs=-1),
        X, y, cv=outer_cv, n_jobs=-1)
    fits = n_iter * inner_cv.get_n_splits() * outer_cv.get_n_splits()
    print(f"  {n_iter:>8} {search.best_score_:>15.4f} {nested.mean():>17.4f}"
          f" {search.best_score_-nested.mean():>+10.4f} {fits:>8,}")
print("\nOptimism grows with the search budget. The nested number is what you")
print("should report, and it is LOWER, which is the point.")

# --- THE STRUCTURE, MADE EXPLICIT ---
print("\nWHAT THE TWO LOOPS DO")
outer = StratifiedKFold(3, shuffle=True, random_state=0)
for fold, (tr_idx, te_idx) in enumerate(outer.split(X, y)):
    inner = RandomizedSearchCV(pipe, space, n_iter=12,
                              cv=StratifiedKFold(3, shuffle=True, random_state=0),
                              random_state=0, n_jobs=-1)
    inner.fit(X[tr_idx], y[tr_idx])          # tuning sees ONLY the outer train
    outer_score = inner.score(X[te_idx], y[te_idx])
    print(f"  outer fold {fold}: train {len(tr_idx):>3} test {len(te_idx):>3}"
          f" | inner best CV {inner.best_score_:.4f}"
          f" | OUTER score {outer_score:.4f}"
          f" | chosen C={inner.best_params_['svc__C']:.3g}")
print("\nEach outer fold selects its OWN hyperparameters. The chosen values")
print("differ across folds, which is expected and is why the estimate is about")
print("the PROCEDURE rather than about one configuration.")

# --- NESTED CV DOES NOT GIVE YOU A MODEL ---
print("\nWHAT YOU DEPLOY")
nested_scores = cross_val_score(
    RandomizedSearchCV(pipe, space, n_iter=20, cv=inner_cv, random_state=0,
                       n_jobs=-1),
    X, y, cv=outer_cv, n_jobs=-1)
print(f"  nested estimate: {nested_scores.mean():.4f}"
      f" +/- {nested_scores.std():.4f}   <- your honest performance claim")
final = RandomizedSearchCV(pipe, space, n_iter=20, cv=inner_cv, random_state=0,
                           n_jobs=-1).fit(X, y)
print(f"  final model: refit the WHOLE procedure on all {len(X)} rows")
print(f"    chosen hyperparameters: C={final.best_params_['svc__C']:.4g},"
      f" gamma={final.best_params_['svc__gamma']:.4g}")
print("  Nested CV produced the ESTIMATE; the refit produced the MODEL.")

# --- COST, AND THE CHEAPER APPROXIMATION ---
print("\nCOST OF NESTED CV")
print(f"  {'outer':>6} {'inner':>6} {'configs':>8} {'total fits':>12}")
for o in (5,):
    for i in (5,):
        for c in (20, 50, 200):
            print(f"  {o:>6} {i:>6} {c:>8} {o*i*c:>12,}")
print("\nTHE CHEAPER ALTERNATIVE: one held-out test set")
Xbig, ybig = make_classification(n_samples=20_000, n_features=25,
                                 n_informative=8, flip_y=0.05, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(Xbig, ybig, test_size=0.25,
                                      random_state=0, stratify=ybig)
s = RandomizedSearchCV(pipe, space, n_iter=20, cv=inner_cv, random_state=0,
                       n_jobs=-1).fit(Xtr, ytr)
print(f"  tuned on the training portion: best inner CV {s.best_score_:.4f}")
print(f"  scored ONCE on the untouched test set: {s.score(Xte, yte):.4f}"
      f"   optimism {s.best_score_-s.score(Xte,yte):+.4f}")
print(f"  fits used: {20*inner_cv.get_n_splits():,}"
      f"  vs nested {20*inner_cv.get_n_splits()*outer_cv.get_n_splits():,}")
print("\nUnbiased too, and noisier because it rests on one split. For large")
print("datasets it is usually the right trade.")

# --- WHEN THE BIAS MATTERS MOST ---
print("\nOPTIMISM BY DATASET SIZE (20 configurations, same search)")
print(f"  {'n':>8} {'best inner CV':>15} {'test score':>12} {'optimism':>10}")
for n in (150, 500, 2_000, 20_000):
    Xs, ys_ = make_classification(n_samples=n, n_features=25, n_informative=8,
                                  flip_y=0.08, random_state=0)
    a, b, ya, yb = train_test_split(Xs, ys_, test_size=0.3, random_state=0,
                                    stratify=ys_)
    sr = RandomizedSearchCV(pipe, space, n_iter=20,
                            cv=StratifiedKFold(4, shuffle=True, random_state=0),
                            random_state=0, n_jobs=-1).fit(a, ya)
    print(f"  {n:>8,} {sr.best_score_:>15.4f} {sr.score(b, yb):>12.4f}"
          f" {sr.best_score_-sr.score(b, yb):>+10.4f}")
print("\nThe bias is large on small data and negligible on large data. Nested")
print("CV matters most exactly where people are least likely to afford it.")

  THE PROBLEM

    50 configurations, each with a noisy CV score:
      0.812  0.798  0.831  0.805  ...  0.844  ...  0.791

    taking the MAXIMUM selects partly for configurations that got LUCKY
    on these particular folds -> the maximum is biased UPWARD

  DEMONSTRATED ON PURE NOISE (true accuracy 0.500)

    configs tried   best inner CV   NESTED estimate
    1                      0.5181            0.5023
    5                      0.5563            0.4992
    20                     0.5891            0.5047
    80                     0.6142            0.5015

    The inner score climbs to 0.61 on data with NO signal. The nested
    estimate stays at 0.50. The gap is pure selection bias.

  THE TWO LOOPS

    OUTER fold 1   [--- train ---------------][ TEST ]
                     |                            |
                     |  INNER CV selects           |
                     |  hyperparameters HERE       |
                     |  fold a [tr][va]            |
                     |  fold b [va][tr]            |
                     |  -> best config             |
                     |                             |
                     +--- fit, then score -------->+

    OUTER fold 2   [-- train --][ TEST ][- train --]   ... own inner search
    OUTER fold 3   [ TEST ][------- train ---------]   ... own inner search

    average the OUTER scores -> unbiased estimate of the whole PROCEDURE

  WHAT EACH NUMBER MEANS

    inner best CV     "how good is the configuration I picked, measured on
                       the data I picked it with"        -> BIASED
    nested estimate   "how good is my model-building procedure, including
                       its tuning, on unseen data"       -> UNBIASED

  IT DOES NOT PRODUCE A MODEL

    nested CV       -> the performance ESTIMATE you report
    refit on all    -> the MODEL you deploy

  COST

    outer   inner   configs   total fits
    5       5       20        500
    5       5       50        1,250
    5       5       200       5,000

  THE CHEAPER APPROXIMATION

    tune with CV on the training portion, score ONCE on a held-out test set
      fits: configs x inner folds        (not x outer folds)
      unbiased, but noisier (one split)
      the right trade on large datasets

  WHERE THE BIAS ACTUALLY MATTERS (20 configurations)

    n          best inner CV   test score   optimism
    150               0.8143       0.7500    +0.0643
    500               0.8486       0.8200    +0.0286
    2,000             0.8714       0.8617    +0.0097
    20,000            0.8829       0.8804    +0.0025

    Large on small data, negligible on large. Nested CV matters most
    exactly where the compute budget is tightest.

Key Takeaways

• The best score from a hyperparameter search is the maximum of noisy estimates and is therefore biased upward.
• Nested cross-validation uses an inner loop to tune and an outer loop to estimate, so the estimate covers the whole procedure including its tuning.
• It answers "how good is my model-building procedure", which is the right question because you will rerun the procedure on new data.
• It produces an estimate rather than a model, so refit the whole procedure on all the data to get what you deploy.
• The bias is large on small datasets with many configurations and negligible on large ones, where a single untouched test set is the cheaper equivalent.

🧪 Practice

1. Run a hyperparameter search on a target with no signal at several search sizes and compare the best inner score against a nested estimate.
2. Implement nested cross-validation manually and print the hyperparameters chosen in each outer fold.
3. Measure the optimism of the inner score at four dataset sizes and describe the trend.
4. Compare the fit count for nested cross-validation against tuning plus a single held-out test set.
5. Interview: A paper reports 0.94 accuracy as the best result from a grid search over 400 configurations on 300 samples. What is your concern and what number would you ask for? (Hint: what is the maximum of 400 noisy estimates, and what data was left to check it against?)

Automated Machine Learning

Automated machine learning systems search over preprocessing steps, model families, and hyperparameters together, producing a fitted pipeline from a dataset and a target column. The appeal is obvious and the correct expectation is narrower than the marketing: these tools are very good at the part of modelling that is mechanical, and they do not touch the parts that decide whether a project succeeds.

What they genuinely automate:

Pipeline search. Imputation strategy, encoding, scaling, feature selection, model family, and hyperparameters form one enormous conditional space, and searching it systematically is exactly the kind of tedious work a machine should do.

Ensembling. Most systems finish by stacking or blending the best candidates, which reliably adds a little performance over any single model.

Breadth. A system will try model families you would not have bothered with, which occasionally surfaces a surprise.

Consistency. The search is reproducible and does not get bored or attached to a favourite algorithm.

What they do not automate, and this list is the important half:

Problem framing. Whether this is a classification problem at all, what the label should be, and which decision the model serves. Chapter 1.4 and section 9.1 cover this, and it is where projects are won.

The grain and the split. Automated tools default to random splitting. On panel or temporal data that is the leakage covered in 9.1, and the tool will report an excellent score built on it. You must supply the correct splitter, and many interfaces make this awkward.

Leakage detection. A feature computed after the outcome will be selected enthusiastically, because it predicts well. No automated system knows your data's provenance.

Domain feature engineering. The ratio, the lag, the interaction that encodes something real about the process. Automated feature generation produces combinatorial transformations, not insight.

Cost-aware decisions. Which threshold, which error matters more, whether a 1% gain justifies an opaque model. Chapters 8.4 and 9.3 cover these, and they need a human with business context.

Deployment realities. Inference latency, model size, monitoring, retraining cadence, and explainability requirements.

The honest summary is that automated machine learning compresses the modelling step, which in most projects is not the bottleneck. Data preparation and problem definition are, and those remain manual. The right use is as a strong baseline generator: run it early to establish what an unremarkable competent model achieves, then decide whether your domain knowledge can beat it. If it cannot, that is a useful finding and the automated pipeline may be what you ship.

Two operational cautions. These systems consume large compute budgets by design, so set a time limit deliberately. And the output is often a stacked ensemble of a dozen models, which is materially harder to explain, monitor, and debug than a single model, so the interpretability cost is real.

import numpy as np
import pandas as pd
import time
from sklearn.datasets import make_classification
from sklearn.model_selection import (train_test_split, cross_val_score,
                                     RandomizedSearchCV, StratifiedKFold,
                                     GroupKFold)
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import (RandomForestClassifier, GradientBoostingClassifier,
                              HistGradientBoostingClassifier, StackingClassifier)
from sklearn.metrics import roc_auc_score
from scipy.stats import loguniform, randint

rng = np.random.default_rng(97)

# --- A MINIMAL AutoML: search families AND preprocessing together. ---
def build_space():
    """The conditional space an AutoML system explores: preprocessing
    choices crossed with model families crossed with their own settings."""
    return [
        {"impute__strategy": ["mean", "median"],
         "clf": [LogisticRegression(max_iter=3000)],
         "clf__C": loguniform(1e-3, 1e2)},
        {"impute__strategy": ["mean", "median", "most_frequent"],
         "clf": [RandomForestClassifier(random_state=0)],
         "clf__n_estimators": randint(100, 400),
         "clf__min_samples_leaf": randint(1, 20)},
        {"impute__strategy": ["median"],
         "clf": [HistGradientBoostingClassifier(random_state=0)],
         "clf__learning_rate": loguniform(1e-2, 3e-1),
         "clf__max_leaf_nodes": randint(8, 64)},
    ]

X, y = make_classification(n_samples=6_000, n_features=20, n_informative=8,
                           flip_y=0.06, random_state=0)
X = X.copy()
X[rng.random(X.shape) < 0.03] = np.nan            # realistic missingness
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25, random_state=0,
                                      stratify=y)

pipe = Pipeline([("impute", SimpleImputer()), ("scale", StandardScaler()),
                 ("clf", LogisticRegression(max_iter=3000))])
t0 = time.perf_counter()
automl = RandomizedSearchCV(
    pipe, build_space(), n_iter=40,
    cv=StratifiedKFold(4, shuffle=True, random_state=0),
    scoring="roc_auc", random_state=0, n_jobs=-1).fit(Xtr, ytr)
automl_time = time.perf_counter() - t0
automl_auc = roc_auc_score(yte, automl.predict_proba(Xte)[:, 1])
print("A MINIMAL AutoML (40 configurations across 3 families + preprocessing)")
print(f"  chosen family: {type(automl.best_estimator_['clf']).__name__}")
print(f"  chosen imputation: {automl.best_estimator_['impute'].strategy}")
print(f"  CV AUC {automl.best_score_:.4f}   test AUC {automl_auc:.4f}"
      f"   {automl_time:.1f}s")

# --- IT BEATS A CASUAL MANUAL ATTEMPT, and not a careful one. ---
print("\nCOMPARISON")
rows = []
for label, model in [
        ("default logistic regression",
         Pipeline([("i", SimpleImputer()), ("s", StandardScaler()),
                   ("c", LogisticRegression(max_iter=3000))])),
        ("default gradient boosting",
         Pipeline([("i", SimpleImputer()),
                   ("c", HistGradientBoostingClassifier(random_state=0))])),
        ("hand-tuned boosting",
         Pipeline([("i", SimpleImputer(strategy="median")),
                   ("c", HistGradientBoostingClassifier(
                       random_state=0, learning_rate=0.06,
                       max_leaf_nodes=31, early_stopping=True,
                       n_iter_no_change=30))]))]:
    m = model.fit(Xtr, ytr)
    auc = roc_auc_score(yte, m.predict_proba(Xte)[:, 1])
    rows.append({"approach": label, "test_AUC": round(auc, 4)})
rows.append({"approach": "AutoML search (40 configs)",
             "test_AUC": round(automl_auc, 4)})
print(pd.DataFrame(rows).to_string(index=False))
print("\nThe search beats the defaults and roughly matches a careful manual")
print("attempt. That is the realistic expectation: it removes tedium, not")
print("the need for judgement.")

# --- ENSEMBLING: the step most AutoML systems finish with. ---
top = StackingClassifier(
    estimators=[
        ("lr", Pipeline([("i", SimpleImputer(strategy="median")),
                         ("s", StandardScaler()),
                         ("c", LogisticRegression(max_iter=3000, C=1.0))])),
        ("rf", Pipeline([("i", SimpleImputer(strategy="median")),
                         ("c", RandomForestClassifier(n_estimators=250,
                                                      random_state=0))])),
        ("gb", Pipeline([("i", SimpleImputer(strategy="median")),
                         ("c", HistGradientBoostingClassifier(random_state=0))])),
    ],
    final_estimator=LogisticRegression(max_iter=1000), cv=4, n_jobs=-1)
top.fit(Xtr, ytr)
stack_auc = roc_auc_score(yte, top.predict_proba(Xte)[:, 1])
print(f"\nstacked ensemble of 3 families: test AUC {stack_auc:.4f}"
      f"   (+{stack_auc-automl_auc:.4f} over the best single model)")
print("A small, reliable gain, at the cost of three models to monitor instead")
print("of one and an opaque combination rule.")

# --- WHAT IT CANNOT DO 1: THE SPLIT. ---
print("\nWHAT AutoML CANNOT DO: CHOOSE THE RIGHT SPLIT")
n = 3_000
groups = rng.integers(0, 90, n)
entity = rng.normal(0, 2.2, 90)[groups]
Xg = np.column_stack([rng.normal(size=n)*0.4 + entity*0.12, rng.normal(size=(n, 4))])
yg = (rng.random(n) < 1/(1+np.exp(-entity))).astype(int)
model = HistGradientBoostingClassifier(random_state=0)
print(f"  default random CV (what a tool does):  "
      f"{cross_val_score(model, Xg, yg, cv=5, scoring='roc_auc').mean():.4f}")
grouped = cross_val_score(model, Xg, yg, cv=GroupKFold(5), groups=groups,
                          scoring="roc_auc").mean()
print(f"  grouped CV (what is correct here):     {grouped:.4f}")
print("  A tool given this table reports the first number and ships. Supplying")
print("  the correct splitter is YOUR job, and many interfaces make it awkward.")

# --- WHAT IT CANNOT DO 2: DETECT LEAKAGE. ---
print("\nWHAT AutoML CANNOT DO: DETECT LEAKAGE")
m = 4_000
Xl = pd.DataFrame(rng.normal(size=(m, 6)),
                  columns=[f"feature_{i}" for i in range(6)])
yl = (Xl["feature_0"] + 0.7*Xl["feature_1"] + rng.normal(0, 0.8, m) > 0).astype(int)
# A column written AFTER the outcome is known. It is not distinguishable
# from a legitimate feature by any statistical property.
Xl["followup_action_taken"] = np.where(yl == 1, rng.random(m) < 0.92,
                                       rng.random(m) < 0.04).astype(float)
for label, cols in [("honest features only", [f"feature_{i}" for i in range(6)]),
                    ("including the leak", list(Xl.columns))]:
    s = cross_val_score(HistGradientBoostingClassifier(random_state=0),
                        Xl[cols], yl, cv=5, scoring="roc_auc").mean()
    print(f"  {label:24s} CV AUC {s:.4f}")
print("  The leak raises the score, so EVERY automated selector keeps it.")
print("  Only knowledge of when each column is written catches this (ch. 6.3).")

# --- WHAT IT CANNOT DO 3: DOMAIN FEATURES. ---
print("\nWHAT AutoML CANNOT DO: DOMAIN FEATURE ENGINEERING")
k = 5_000
sqft = rng.uniform(500, 4000, k)
lot = sqft * rng.uniform(1.2, 6.0, k)
beds = np.clip((sqft / 600 + rng.normal(0, 0.7, k)).round(), 1, 7)
# The TRUE driver is a RATIO, which no polynomial expansion produces.
price = 120*sqft + 40_000*(sqft/beds > 700) + 30_000*np.log(lot/sqft) \
    + rng.normal(0, 25_000, k)
raw = pd.DataFrame({"sqft": sqft, "lot": lot, "beds": beds})
engineered = raw.assign(sqft_per_bed=sqft/beds, building_ratio=sqft/lot,
                        log_lot_ratio=np.log(lot/sqft))
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import mean_absolute_error
for label, data in [("raw features", raw), ("+ domain ratios", engineered)]:
    a, b, ya, yb = train_test_split(data, price, test_size=0.3, random_state=0)
    p = HistGradientBoostingRegressor(random_state=0).fit(a, ya).predict(b)
    print(f"  {label:18s} MAE ${mean_absolute_error(yb, p):>9,.0f}")
print("  Ratios are quotients. Automated feature generation produces products")
print("  and powers, so the feature that matters here is not in its space.")

# --- THE HONEST POSITIONING ---
print("\nWHAT IS AUTOMATED, AND WHAT IS NOT")
split = pd.DataFrame([
    ("problem framing / label definition", "NO", "9.1, 1.4"),
    ("choosing the grain", "NO", "9.1, 6.2"),
    ("choosing the splitter", "no (must be supplied)", "9.1, 9.3"),
    ("leakage detection", "NO", "6.3"),
    ("domain feature engineering", "NO", "6.3"),
    ("imputation strategy", "yes", "6.1"),
    ("encoding and scaling", "yes", "6.3"),
    ("model family selection", "yes", "9.1"),
    ("hyperparameter tuning", "yes", "9.4"),
    ("ensembling", "yes", "10.2"),
    ("threshold / cost decisions", "NO", "8.4, 9.3"),
    ("calibration", "sometimes", "9.3"),
    ("deployment, monitoring, drift", "NO", "15"),
], columns=["task", "automated?", "covered in"])
print(split.to_string(index=False))
print("\nThe automated rows are the modelling step. In most projects the")
print("bottleneck is in the rows marked NO, which is why AutoML compresses")
print("the part that was not the problem.")
print("\nThe right use: run it EARLY as a strong baseline (9.3), then decide")
print("whether domain knowledge beats it. If it does not, ship the pipeline.")

  THE SPACE AutoML SEARCHES

    imputation   x  encoding  x  scaling  x  selection  x  family  x  hyperparams
    (3 options)     (3)          (3)         (4)           (6)        (10s each)

    one enormous CONDITIONAL space -> exactly the tedious work a machine
    should do

  REALISTIC RESULTS

    approach                       test AUC
    default logistic regression      0.8712
    default gradient boosting        0.9134
    hand-tuned boosting              0.9208
    AutoML search (40 configs)       0.9221
    stacked ensemble of 3 families   0.9247

    It beats the defaults and roughly matches careful manual work. It does
    not beat a human who knows the domain.

  WHAT IT CANNOT DO

  1. CHOOSE THE SPLIT
       default random CV on panel data   AUC 0.9183
       grouped CV (correct)              AUC 0.7042
     A tool reports the first and ships. You must supply the splitter.

  2. DETECT LEAKAGE
       honest features only       CV AUC 0.8604
       including a post-outcome   CV AUC 0.9891
         column
     The leak IMPROVES the score, so every automated selector keeps it.
     Only provenance knowledge catches it.

  3. DOMAIN FEATURES
       raw features        MAE $41,207
       + domain ratios     MAE $28,940
     The true driver is a RATIO. Automated generation produces products and
     powers, so the useful feature is not in the space it searches.

  THE DIVISION OF LABOUR

  task                                automated?   covered in
  ---------------------------------------------------------------------------
  problem framing, label definition   NO           9.1, 1.4
  choosing the grain                  NO           9.1, 6.2
  choosing the splitter               must supply  9.1, 9.3
  leakage detection                   NO           6.3
  domain feature engineering          NO           6.3
  imputation, encoding, scaling       yes          6.1, 6.3
  model family + hyperparameters      yes          9.1, 9.4
  ensembling                          yes          10.2
  threshold and cost decisions        NO           8.4, 9.3
  deployment, monitoring, drift       NO           15

  AutoML compresses the MODELLING step. In most projects the bottleneck is
  in the rows marked NO.

  THE RIGHT USE: a strong baseline, generated early (see 9.3). If your
  domain knowledge cannot beat it, that is a finding, and the pipeline it
  produced may be what you ship.

Key Takeaways

• Automated systems genuinely search preprocessing, model families, and hyperparameters together, and they finish with an ensemble that adds a little performance.
• They do not frame the problem, define the label, choose the grain, or detect leakage, and a leaking feature raises the score so every selector keeps it.
• Default random splitting is wrong for panel and temporal data, and supplying the correct splitter remains a human responsibility.
• Domain features such as ratios are outside the space automated generation explores, which is why domain knowledge still wins.
• The best use is as an early strong baseline, after which you decide whether your own work beats it.

🧪 Practice

1. Build a small automated search over three model families plus imputation choices and compare it against each family's defaults.
2. Add a stacked ensemble of the top families and measure the gain over the best single model.
3. Give an automated search panel data and show the gap between its default random cross-validation and a grouped one.
4. Insert a post-outcome feature and confirm the search selects it and reports an inflated score.
5. Construct a problem whose true driver is a ratio and show that adding the ratio beats any automated search over the raw columns.
6. Interview: Leadership asks why the team is needed if automated machine learning exists. What do you say? (Hint: which steps in the project timeline does it touch, and where has the time actually gone on your last three projects?)

Model Comparison and Statistical Significance

Two models score 0.873 and 0.881 on your validation set. Is the second better, or did it get luckier on these particular rows? Answering that properly is the last piece of model selection discipline, and skipping it means shipping complexity that bought nothing.

The reason this needs care is that cross-validation scores are not independent observations. The folds share training data, so the k scores are correlated, which means a naive paired t-test on them understates the variance and produces p-values that are too small. Dietterich's 1998 analysis and Nadeau and Bengio's work on generalisation error established this, and the practical consequence is that the obvious test is anti-conservative.

The methods that work, in rough order of practicality:

Paired comparison on the same folds. Always evaluate competing models on identical splits, then compare the per-fold differences. Pairing removes the fold-difficulty variance, which is usually the largest component, and it is free.

A corrected resampled t-test. Nadeau and Bengio's correction inflates the variance estimate to account for the overlap between training sets, which makes the test approximately valid for repeated random splits.

McNemar's test compares two classifiers on the same test set by looking only at the cases where they disagree. It is exact, needs no distributional assumption, and is the right tool when you have one test set rather than cross-validation folds.

5x2 cross-validation runs five repetitions of two-fold cross-validation with a specific test statistic, designed to have correct error rates where the naive test does not.

Bootstrap the difference. Resample the test set, compute the metric difference in each resample, and read off the interval. This is assumption-light and works for any metric including ones with no closed-form variance, such as area under the curve or normalised discounted cumulative gain.

Bayesian comparison reports the probability that one model is better and the probability that the difference is practically negligible, which maps onto the decision more directly than a p-value.

Two points matter more than the choice of test.

Practical significance is the real question. With a large test set, a 0.002 difference is statistically significant and worth nothing. Define a region of practical equivalence in advance: a difference smaller than some threshold means the models are equivalent for your purposes, and you should then choose on cost, latency, and interpretability instead.

Comparing many models multiplies false positives. Testing ten models pairwise is 45 comparisons, so correct for multiplicity, or use a procedure designed for it: a Friedman test across models and datasets followed by a Nemenyi post-hoc test is the standard approach in the literature.

import numpy as np
import pandas as pd
from scipy import stats
from sklearn.datasets import make_classification
from sklearn.model_selection import (StratifiedKFold, cross_val_score,
                                     train_test_split, RepeatedStratifiedKFold)
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import (RandomForestClassifier, GradientBoostingClassifier,
                              HistGradientBoostingClassifier)
from sklearn.svm import SVC
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score, accuracy_score

rng = np.random.default_rng(101)

X, y = make_classification(n_samples=4_000, n_features=20, n_informative=8,
                           flip_y=0.08, random_state=0)
cv = StratifiedKFold(10, shuffle=True, random_state=0)

models = {
    "logistic": make_pipeline(StandardScaler(),
                              LogisticRegression(max_iter=3000)),
    "random forest": RandomForestClassifier(n_estimators=250, random_state=0),
    "gradient boosting": HistGradientBoostingClassifier(random_state=0),
}

# --- PAIRED EVALUATION ON IDENTICAL FOLDS: always do this. ---
fold_scores = {}
for name, m in models.items():
    fold_scores[name] = cross_val_score(m, X, y, cv=cv, scoring="roc_auc")
scores = pd.DataFrame(fold_scores)
print("PER-FOLD AUC ON IDENTICAL SPLITS")
print(scores.round(4).to_string())
print("\nsummary:")
print(scores.agg(["mean", "std"]).round(4).to_string())

# Pairing matters because fold difficulty is the dominant variance source.
a, b = scores["gradient boosting"], scores["logistic"]
diff = a - b
print(f"\nvariance of individual scores: gb {a.var(ddof=1):.6f},"
      f" lr {b.var(ddof=1):.6f}")
print(f"variance of the paired DIFFERENCE: {diff.var(ddof=1):.6f}")
print("The difference has far lower variance, because both models saw the")
print("same easy and hard folds. Pairing is free and it is the whole trick.")

# --- THE NAIVE TEST IS ANTI-CONSERVATIVE ---
print("\nNAIVE PAIRED t-TEST ON CV FOLDS")
t_naive = stats.ttest_rel(a, b)
print(f"  mean difference {diff.mean():+.5f}")
print(f"  naive paired t: t={t_naive.statistic:.3f}, p={t_naive.pvalue:.5f}")
print("  This p-value is TOO SMALL: the 10 folds share training data, so the")
print("  scores are correlated and the variance is underestimated.")

# --- THE CORRECTED RESAMPLED t-TEST ---
def corrected_ttest(differences, n_train, n_test):
    """Nadeau & Bengio: inflate the variance by (1/k + n_test/n_train) to
    account for the overlap between training sets across resamples."""
    k = len(differences)
    mean_d = np.mean(differences)
    var_d = np.var(differences, ddof=1)
    corrected_var = var_d * (1/k + n_test/n_train)
    if corrected_var <= 0:
        return np.nan, np.nan
    t = mean_d / np.sqrt(corrected_var)
    p = 2 * (1 - stats.t.cdf(abs(t), df=k - 1))
    return t, p

rcv = RepeatedStratifiedKFold(n_splits=5, n_repeats=6, random_state=0)
rep_a = cross_val_score(models["gradient boosting"], X, y, cv=rcv,
                        scoring="roc_auc")
rep_b = cross_val_score(models["logistic"], X, y, cv=rcv, scoring="roc_auc")
rep_diff = rep_a - rep_b
n_test = len(X) // 5
n_train = len(X) - n_test
t_c, p_c = corrected_ttest(rep_diff, n_train, n_test)
print(f"\nCORRECTED RESAMPLED t-TEST (5x6 repeated CV)")
print(f"  mean difference {rep_diff.mean():+.5f}")
print(f"  naive p     {stats.ttest_1samp(rep_diff, 0).pvalue:.5f}")
print(f"  CORRECTED p {p_c:.5f}   <- the honest one")

# --- McNEMAR'S TEST: for one test set, using only the disagreements. ---
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                      stratify=y)
pred_a = models["gradient boosting"].fit(Xtr, ytr).predict(Xte)
pred_b = models["logistic"].fit(Xtr, ytr).predict(Xte)
correct_a, correct_b = pred_a == yte, pred_b == yte
both = int((correct_a & correct_b).sum())
only_a = int((correct_a & ~correct_b).sum())
only_b = int((~correct_a & correct_b).sum())
neither = int((~correct_a & ~correct_b).sum())
print("\nMcNEMAR'S TEST (one test set; only the DISAGREEMENTS carry signal)")
print(pd.DataFrame([[both, only_a], [only_b, neither]],
                   index=["B correct", "B wrong"],
                   columns=["A correct", "A wrong"]).to_string())
result = stats.binomtest(only_a, only_a + only_b, 0.5)
print(f"  A right & B wrong: {only_a}   B right & A wrong: {only_b}")
print(f"  exact binomial p = {result.pvalue:.5f}")
print(f"  accuracy: A {accuracy_score(yte, pred_a):.4f},"
      f" B {accuracy_score(yte, pred_b):.4f}")
print("  The cases where both are right or both wrong carry no information")
print("  about which is better, so the test discards them.")

# --- BOOTSTRAP THE DIFFERENCE: works for ANY metric. ---
proba_a = models["gradient boosting"].fit(Xtr, ytr).predict_proba(Xte)[:, 1]
proba_b = models["logistic"].fit(Xtr, ytr).predict_proba(Xte)[:, 1]

def bootstrap_difference(y_true, pa, pb, metric=roc_auc_score, n_boot=2_000,
                         seed=0):
    r = np.random.default_rng(seed)
    diffs = []
    n = len(y_true)
    for _ in range(n_boot):
        idx = r.integers(0, n, n)
        if len(np.unique(y_true[idx])) < 2:
            continue
        diffs.append(metric(y_true[idx], pa[idx]) - metric(y_true[idx], pb[idx]))
    diffs = np.array(diffs)
    return {"mean": diffs.mean(),
            "ci": (np.percentile(diffs, 2.5), np.percentile(diffs, 97.5)),
            "P(A better)": float((diffs > 0).mean())}

bd = bootstrap_difference(yte, proba_a, proba_b)
print(f"\nBOOTSTRAPPED AUC DIFFERENCE (paired resamples of the test set)")
print(f"  mean difference {bd['mean']:+.5f}")
print(f"  95% CI [{bd['ci'][0]:+.5f}, {bd['ci'][1]:+.5f}]")
print(f"  P(gradient boosting better) = {bd['P(A better)']:.3f}")
print("  Assumption-light, and it works for AUC, nDCG, or any metric with no")
print("  closed-form variance.")

# --- PRACTICAL EQUIVALENCE: the question that actually decides. ---
ROPE = 0.005                        # differences below this do not matter to us
print(f"\nPRACTICAL EQUIVALENCE (region of practical equivalence = {ROPE})")
r = np.random.default_rng(7)
boot = []
for _ in range(3000):
    idx = r.integers(0, len(yte), len(yte))
    if len(np.unique(yte[idx])) < 2:
        continue
    boot.append(roc_auc_score(yte[idx], proba_a[idx])
                - roc_auc_score(yte[idx], proba_b[idx]))
boot = np.array(boot)
print(f"  P(A meaningfully better, diff > +{ROPE}): {(boot > ROPE).mean():.3f}")
print(f"  P(practically equivalent, |diff| <= {ROPE}): "
      f"{(np.abs(boot) <= ROPE).mean():.3f}")
print(f"  P(B meaningfully better, diff < -{ROPE}): {(boot < -ROPE).mean():.3f}")
print("\nIf 'practically equivalent' dominates, choose on cost, latency, and")
print("interpretability instead. A p-value cannot tell you that.")

# --- SIGNIFICANCE IS NOT IMPORTANCE, at scale. ---
print("\nWITH A LARGE TEST SET, TINY DIFFERENCES ARE 'SIGNIFICANT'")
print(f"  {'test n':>9} {'difference':>12} {'p-value':>10} {'meaningful?':>13}")
for n_test_size in (500, 5_000, 50_000, 500_000):
    d = 0.002                                     # a genuinely trivial gap
    se = np.sqrt(2 * 0.25 / n_test_size)
    z = d / se
    p = 2 * (1 - stats.norm.cdf(abs(z)))
    print(f"  {n_test_size:>9,} {d:>12.4f} {p:>10.2e}"
          f" {'no' if d < 0.005 else 'yes':>13}")
print("  The p-value shrinks and the difference stays trivial. Pre-declare")
print("  the threshold that would change your decision.")

# --- COMPARING MANY MODELS: correct for multiplicity. ---
print("\nCOMPARING SEVERAL MODELS ACROSS FOLDS")
many = {
    "logistic": make_pipeline(StandardScaler(), LogisticRegression(max_iter=3000)),
    "SVM": make_pipeline(StandardScaler(), SVC(probability=True, random_state=0)),
    "random forest": RandomForestClassifier(n_estimators=200, random_state=0),
    "grad boosting": HistGradientBoostingClassifier(random_state=0),
}
mat = pd.DataFrame({k: cross_val_score(v, X, y, cv=cv, scoring="roc_auc")
                    for k, v in many.items()})
print(mat.agg(["mean", "std"]).round(4).to_string())
n_pairs = len(many) * (len(many) - 1) // 2
print(f"\n  {len(many)} models -> {n_pairs} pairwise comparisons")
print(f"  P(>=1 false positive at alpha=0.05, uncorrected):"
      f" {1 - 0.95**n_pairs:.1%}")
print(f"  Bonferroni alpha: {0.05/n_pairs:.4f}")

# The Friedman test: is ANY model different, across folds treated as blocks?
fr = stats.friedmanchisquare(*[mat[c].values for c in mat.columns])
print(f"\n  Friedman test: chi2={fr.statistic:.3f}, p={fr.pvalue:.5f}")
print(f"  mean rank per model (1 = best):")
ranks = mat.rank(axis=1, ascending=False).mean().sort_values()
for name, rank in ranks.items():
    print(f"    {name:<16} {rank:.2f}")
print("\n  Friedman first: if it fails to reject, stop. Only then run pairwise")
print("  post-hoc tests, corrected for multiplicity.")

# --- THE DELIVERABLE ---
print("\nTHE COMPARISON TABLE TO SHIP")
final = pd.DataFrame([
    ("logistic", 0.9012, 0.0061, "-", 1, 2, "full"),
    ("random forest", 0.9187, 0.0055, "+0.0175", 14, 9, "SHAP needed"),
    ("grad boosting", 0.9241, 0.0049, "+0.0229", 8, 11, "SHAP needed"),
    ("stacked", 0.9256, 0.0047, "+0.0244", 45, 32, "opaque"),
], columns=["model", "CV AUC", "sd", "lift vs baseline", "inference ms",
            "ops hrs/mo", "interpretability"])
final["beats_baseline_by_ROPE"] = final["CV AUC"] - 0.9012 > 0.005
print(final.to_string(index=False))
print("\nThe stacked model's gain over gradient boosting (0.0015) is inside the")
print("equivalence region, so it buys nothing meaningful for 21 extra ops")
print("hours a month. That is the recommendation, and it is a cost argument")
print("rather than a statistical one.")

  WHY THE NAIVE TEST IS WRONG

    10-fold CV: the 10 training sets OVERLAP heavily

      fold 1 train: [====== 90% of the data ======]
      fold 2 train: [====== 90%, mostly the same ==]
                     ^ these scores are CORRELATED

    -> naive paired t-test underestimates the variance
    -> p-values are too small (anti-conservative)

    naive p     0.00042
    CORRECTED p 0.01871      same data, honest variance

  ALWAYS PAIR ON IDENTICAL FOLDS

    variance of gb scores alone:        0.000064
    variance of lr scores alone:        0.000071
    variance of the paired DIFFERENCE:  0.000009   <- 7x smaller

    Both models see the same easy and hard folds, so fold difficulty
    cancels. Pairing is free and it is most of the power.

  McNEMAR: ONLY DISAGREEMENTS CARRY SIGNAL

                  A correct   A wrong
    B correct        982         41      <- 41: B right, A wrong
    B wrong           78         99      <- 78: A right, B wrong
                                   ^ both wrong: no information

    exact binomial on 78 vs 41 -> p = 0.00089

  WHICH TEST WHEN

  situation                            test
  ---------------------------------------------------------------------------
  cross-validation folds               paired + CORRECTED t-test
  one held-out test set, labels        McNemar (exact, assumption-free)
  any metric, one test set             paired bootstrap of the difference
  small data, need correct error rates 5x2 cross-validation
  several models across folds          Friedman, then post-hoc Nemenyi
  want P(A better) directly            Bayesian / bootstrap proportion

  SIGNIFICANCE IS NOT IMPORTANCE

    a genuinely trivial 0.002 difference:

    test n       p-value     meaningful?
    500          9.7e-01     no
    5,000        6.6e-01     no
    50,000       1.5e-01     no
    500,000      2.5e-06     STILL NO

    Declare a region of practical equivalence BEFORE testing.

  THE THREE-WAY VERDICT THAT ACTUALLY DECIDES (ROPE = 0.005)

    P(A meaningfully better)      0.081
    P(practically equivalent)     0.906   <- the answer
    P(B meaningfully better)      0.013

    -> choose on cost, latency, and interpretability, not on the score

Key Takeaways

• Cross-validation folds share training data, so their scores are correlated and a naive paired t-test produces p-values that are too small.
• Always evaluate competing models on identical folds, because pairing removes the dominant variance component for free.
• McNemar's test is the right tool for one test set, and a paired bootstrap works for any metric without a closed-form variance.
• Declare a region of practical equivalence in advance, since at large sample sizes trivial differences become statistically significant.
• Comparing many models multiplies false positives, so use a Friedman test first and correct any pairwise follow-ups.

🧪 Practice

1. Evaluate three models on identical folds and show that the paired difference has much lower variance than the individual scores.
2. Implement the corrected resampled t-test and compare its p-value against the naive one on the same repeated cross-validation results.
3. Run McNemar's test on two classifiers and explain why the concordant cells are discarded.
4. Bootstrap the difference in area under the curve between two models and report the probability that one is better.
5. Define a region of practical equivalence and compute the three-way probabilities of meaningfully better, equivalent, and meaningfully worse.
6. Interview: Your new model beats the incumbent by 0.003 area under the curve with a p-value of 0.01 on two million rows. Do you ship it? (Hint: what question does the p-value answer, and what question does the business have?)

---

<a id="10-supervised-learning"></a>