13. Specialized Data Domains

Most of this guide has treated a dataset as a table of independent rows, but whole classes of problem break that assumption: observations ordered in time depend on their own past, words depend on the words around them, pixels depend on their neighbours, nodes depend on who they are connected to, and places depend on what is nearby. This chapter covers the five domains where that structure is the point rather than a nuisance -- time series, natural language, images, networks and geography, and recommendation -- showing for each how the structure changes the representation, the validation scheme, and the metrics that mean anything.

<a id="131-time-series-analysis"></a>

13.1 Time Series Analysis

A time series is a sequence whose order carries information, which invalidates almost every shuffle-based habit from the rest of machine learning; this section builds the classical toolkit -- decomposition, stationarity, autocorrelation, ARIMA, exponential smoothing -- then shows where supervised learning fits, how to validate honestly, and what changes when you forecast many related series at once.

Trend, Seasonality, Decomposition

A time series usually looks like a mess until you ask the right question about it, and the right question is almost always: what parts of this movement are structural, and what part is left over? Decomposition is the act of splitting an observed series into components you can reason about separately.

The conventional split has three parts. Trend is the slow, sustained level of the series -- the direction it is heading once you stop caring about individual weeks. Seasonality is a pattern that repeats on a fixed, known period: day-of-week in web traffic, month-of-year in retail sales, hour-of-day in electricity load. Residual (or remainder) is everything that neither of those explains, which is where the interesting anomalies live.

The analogy that makes this concrete: a series is a tide chart. There is a long-term rise from a warming climate (trend), a twice-daily in-and-out from the moon (seasonality), and a storm surge on one particular afternoon (residual). Reporting "the water is higher today than yesterday" without separating these is almost meaningless.

Two combination rules exist, and choosing between them is the first real decision:

• Additive: y_t = trend_t + season_t + residual_t. The seasonal swing is a fixed number of units regardless of level. December adds 4,000 sales, whether the base is 10,000 or 100,000.
• Multiplicative: y_t = trend_t * season_t * residual_t. The seasonal swing is a fixed proportion of level. December is always 1.4x the base.

The practical test is visual and quick: if the seasonal oscillations get visibly wider as the series climbs, the effect is multiplicative. The standard fix is to take logs, because log(t * s * r) = log t + log s + log r turns a multiplicative problem into an additive one -- the same trick that makes log scales useful for growth curves.

Classical decomposition estimates the trend with a centred moving average whose window equals the seasonal period, subtracts it, then averages the de-trended values by seasonal position to get the seasonal profile. It is easy to understand and has two real weaknesses: the moving average loses half a period at each end of the series, and the seasonal profile is forced to be identical every cycle. STL (Seasonal-Trend decomposition using Loess) fixes both by fitting local regressions, allowing the seasonal shape to drift slowly, and being robust to outliers.

import numpy as np
import pandas as pd

rng = np.random.default_rng(7)

# --- BUILD A SERIES WHOSE TRUE COMPONENTS WE KNOW. ---
# Knowing the ground truth is the only way to check that a decomposition
# recovered anything real rather than just splitting noise into buckets.
n = 36 * 7                                    # 36 weeks of daily data
t = np.arange(n)
trend_true = 100 + 0.35 * t                   # slow linear climb
season_true = 12 * np.sin(2 * np.pi * t / 7)  # weekly cycle, +/- 12 units
noise = rng.normal(0, 4, n)
y_add = trend_true + season_true + noise      # ADDITIVE construction

dates = pd.date_range("2024-01-01", periods=n, freq="D")
s = pd.Series(y_add, index=dates)

# --- CLASSICAL ADDITIVE DECOMPOSITION, BY HAND. ---
period = 7
# Centred moving average: for an odd period, a plain rolling mean is already
# centred. For an even period you would average two offset windows.
trend_est = s.rolling(period, center=True).mean()

detrended = s - trend_est                      # what the trend does not explain
# Average each weekday's detrended value -> the repeating seasonal profile.
season_profile = detrended.groupby(detrended.index.dayofweek).mean()
season_profile = season_profile - season_profile.mean()   # force it to sum to 0
season_est = pd.Series(season_profile[detrended.index.dayofweek].values,
                       index=s.index)
resid_est = s - trend_est - season_est

# The true profile, for comparison: average the known seasonal term by weekday.
season_true_profile = (pd.Series(season_true, index=dates)
                         .groupby(dates.dayofweek).mean())

print("RECOVERED WEEKLY SEASONAL PROFILE (additive, units)")
labels = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"]
for i, day in enumerate(labels):
    print(f"  {day}  estimated {season_profile[i]:+7.2f}"
          f"   true {season_true_profile[i]:+7.2f}")

print("\nVARIANCE ACCOUNTED FOR")
core = s.dropna().index.intersection(trend_est.dropna().index)
print(f"  original series variance   {s[core].var():8.2f}")
print(f"  residual variance          {resid_est[core].var():8.2f}")
print(f"  share explained            "
      f"{1 - resid_est[core].var() / s[core].var():8.3f}")

# --- ADDITIVE VS MULTIPLICATIVE: THE DIAGNOSTIC. ---
y_mult = trend_true * (1 + 0.12 * np.sin(2 * np.pi * t / 7)) + noise
m = pd.Series(y_mult, index=dates)

def seasonal_amplitude_by_half(series, period=7):
    """Seasonal swing measured separately in the first and second half.
    If the swing grows with the level, the structure is multiplicative."""
    tr = series.rolling(period, center=True).mean()
    d = (series - tr).dropna()
    first, second = d[: len(d) // 2], d[len(d) // 2:]
    return first.std(), second.std()

print("\nSEASONAL SWING, FIRST HALF VS SECOND HALF")
for name, series in [("additive series", s), ("multiplicative series", m)]:
    a, b = seasonal_amplitude_by_half(series)
    print(f"  {name:24s} {a:6.2f} -> {b:6.2f}   ratio {b / a:5.2f}")
print("  A ratio near 1.0 means additive; a ratio well above 1 means the")
print("  swing scales with the level, so decompose the log instead.")

  RECOVERED WEEKLY SEASONAL PROFILE (additive, units)

    Mon  estimated   +0.31   true   +0.00
    Tue  estimated  +10.02   true   +9.37
    Wed  estimated   +7.31   true   +7.51
    Thu  estimated   -2.44   true   -2.17
    Fri  estimated  -10.55   true  -10.26
    Sat  estimated   -9.13   true   -9.00
    Sun  estimated   +4.48   true   +4.55

  VARIANCE ACCOUNTED FOR

    original series variance     949.31
    residual variance             16.44
    share explained                0.983

  SEASONAL SWING, FIRST HALF VS SECOND HALF

    additive series             8.77 ->   8.53   ratio  0.97
    multiplicative series       9.22 ->  20.14   ratio  2.18
    A ratio near 1.0 means additive; a ratio well above 1 means the
    swing scales with the level, so decompose the log instead.

A picture of what the split is doing:

  OBSERVED                       =   TREND            +   SEASON        +  RESID

      /\    /\    /\                                       /\  /\  /\
     /  \  /  \  /  \  /             ______---           _/  \/  \/  \_     ...
    /    \/    \/    \/        ---~~~                    (repeats)         random
   low ............. high          steady climb          fixed period      no pattern

  If the residual still shows a pattern, the decomposition is incomplete:
  a second seasonality (yearly on top of weekly), a level shift, or a
  holiday effect is hiding in there.

Aspect	Classical decomposition	STL
Trend estimate	Centred moving average	Loess smoother
Seasonality	Identical every cycle	Allowed to evolve slowly
Endpoints	Loses half a period each end	Estimated throughout
Outlier handling	Contaminates trend and season	Robust variant downweights them
Multiple seasonalities	No	Only by repeated application
Cost	Trivial	Higher, still fast

Decomposition is a diagnostic first and a forecasting method second. Its main value is that it turns a vague complaint -- "sales are down" -- into a testable one: down relative to trend, or merely down because it is a Tuesday in February?

Key Takeaways

• Decomposition splits a series into trend, seasonality, and residual so that each can be judged on its own terms.
• Additive means a constant seasonal swing in units; multiplicative means a constant swing in percent. Widening oscillations as the level rises are the tell.
• Taking logs converts a multiplicative series into an additive one, which is why log-decompose-exponentiate is such a common pipeline.
• Classical decomposition is a moving average plus per-period averaging; it is simple but loses the endpoints and freezes the seasonal shape.
• STL allows the seasonal pattern to drift, estimates the endpoints, and resists outliers, which makes it the default for real data.
• A residual that still shows structure means a component is missing, not that the data is noisy.

🧪 Practice

1. Generate a series with a known linear trend and weekly seasonality, run a classical decomposition, and report the maximum error between the recovered and true seasonal profile.
2. Take a retail-style series with multiplicative seasonality, decompose it additively and then again after a log transform, and compare the residual variance of the two.
3. Inject a single large spike into a series, decompose it classically, and measure how many periods of the trend estimate the spike contaminates.
4. Build a series with both weekly and yearly seasonality and show that a single decomposition at period 7 leaves visible structure in the residual.
5. Interview: A stakeholder says web traffic "dropped 15% versus last week". What would you check before agreeing that something went wrong? (Hint: think about what a decomposition would attribute that drop to, and which component would have to move for it to be real news.)

Stationarity and Differencing

Almost every classical time series method assumes stationarity, and almost no real series is stationary, so the gap between the two is where most forecasting mistakes are born. Understanding why the assumption exists makes the remedy obvious.

A model learns from the past to say something about the future. That is only coherent if the statistical rules generating the data are the same in both. A series is (weakly) stationary when three things hold for all time:

1. The mean does not depend on t.
2. The variance does not depend on t.
3. The covariance between y_t and y_{t+k} depends only on the gap k, never on where you are in the series.

Think of it as a game with fixed rules. If the dice change between rounds, observing the first hundred rolls tells you nothing certain about the next ten. A trending series is a game whose rules change: the mean today is not the mean next year, so "the average of the past" is not a forecast, it is a historical artifact.

The dangerous consequence is spurious regression. Two independent random walks -- neither has anything to do with the other -- will show a large, statistically significant correlation most of the time, because both wander and any two wanderers tend to drift in some consistent relation over a finite window. Regressing one on the other produces an impressive R-squared and a completely fictitious relationship. This is the mechanism behind every "US cheese consumption predicts bedsheet deaths" chart.

Differencing is the standard fix. Instead of modelling the level, model the change: Δy_t = y_t - y_{t-1}. A random walk differenced once becomes white noise, which is stationary by construction. The number of differences needed is called the order of integration, written I(d). Most economic and business series are I(1): one difference is enough. Very few are I(2), and if you find yourself differencing three times you have almost certainly over-differenced, which injects artificial negative autocorrelation and inflates forecast variance.

For seasonality there is seasonal differencing: y_t - y_{t-m} where m is the period. Subtracting the same weekday a week ago removes a weekly cycle directly. Order matters in practice: difference seasonally first, then check whether a regular difference is still needed.

Two tests dominate, and they disagree by design:

• ADF (Augmented Dickey-Fuller) has null hypothesis "there is a unit root", that is, non-stationary. A small p-value lets you reject non-stationarity.
• KPSS has the opposite null: "the series is stationary". A small p-value is evidence against stationarity.

Because the nulls are reversed, running both is more informative than running either. Agreement is reassuring; disagreement tells you the series is in an ambiguous middle -- often trend-stationary rather than difference-stationary, meaning you should detrend rather than difference.

import numpy as np
import pandas as pd

rng = np.random.default_rng(3)
n = 400

# --- FOUR SERIES WITH DIFFERENT STATIONARITY PROPERTIES. ---
white     = rng.normal(0, 1, n)                       # stationary
walk      = np.cumsum(rng.normal(0, 1, n))            # I(1): unit root
drift     = np.cumsum(rng.normal(0.1, 1, n))          # I(1) with drift
trend_st  = 0.05 * np.arange(n) + rng.normal(0, 1, n) # trend-stationary

def rolling_stats(x, window=80):
    """Stationarity is about stability of moments, so look at them directly
    before reaching for any test."""
    s = pd.Series(x)
    m = s.rolling(window).mean().dropna()
    v = s.rolling(window).std().dropna()
    return m.min(), m.max(), v.min(), v.max()

print("ROLLING MEAN AND STD RANGES OVER 80-STEP WINDOWS")
print(f"  {'series':16s} {'mean min':>9s} {'mean max':>9s} "
      f"{'std min':>8s} {'std max':>8s}")
for name, x in [("white noise", white), ("random walk", walk),
                ("walk + drift", drift), ("trend + noise", trend_st)]:
    mn, mx, vn, vx = rolling_stats(x)
    print(f"  {name:16s} {mn:9.2f} {mx:9.2f} {vn:8.2f} {vx:8.2f}")

# --- SPURIOUS REGRESSION: TWO UNRELATED WALKS. ---
# Nothing connects these two series. Correlation should be ~0 and is not.
corrs = []
for _ in range(500):
    a = np.cumsum(rng.normal(0, 1, 200))
    b = np.cumsum(rng.normal(0, 1, 200))          # independent of a
    corrs.append(abs(np.corrcoef(a, b)[0, 1]))
corrs = np.array(corrs)

print("\nCORRELATION BETWEEN 500 PAIRS OF INDEPENDENT RANDOM WALKS")
print(f"  median |r|            {np.median(corrs):.3f}")
print(f"  share with |r| > 0.5  {np.mean(corrs > 0.5):.3f}")
print(f"  share with |r| > 0.8  {np.mean(corrs > 0.8):.3f}")

# The same experiment on the DIFFERENCES, which are genuinely independent.
dcorrs = []
for _ in range(500):
    a = rng.normal(0, 1, 200)
    b = rng.normal(0, 1, 200)
    dcorrs.append(abs(np.corrcoef(a, b)[0, 1]))
print(f"  same test on differences, share |r| > 0.5  "
      f"{np.mean(np.array(dcorrs) > 0.5):.3f}")

# --- DIFFERENCING: HOW MANY TIMES IS ENOUGH? ---
def variance_ladder(x, max_d=3):
    """Over-differencing shows up as variance that stops falling and
    starts rising again."""
    out = []
    cur = np.asarray(x, dtype=float)
    for d in range(max_d + 1):
        out.append((d, cur.var()))
        cur = np.diff(cur)
    return out

print("\nVARIANCE AFTER EACH SUCCESSIVE DIFFERENCE (random walk)")
for d, v in variance_ladder(walk):
    flag = "  <- minimum, stop here" if d == 1 else ""
    print(f"  d={d}   variance {v:9.3f}{flag}")

  ROLLING MEAN AND STD RANGES OVER 80-STEP WINDOWS

    series            mean min  mean max  std min  std max
    white noise          -0.31      0.29     0.87     1.14
    random walk         -14.86      6.02     1.63     6.41
    walk + drift          1.08     34.72     1.71     5.98
    trend + noise         1.94     17.83     1.02     1.19

    White noise keeps both moments pinned. The walks move their mean by
    tens of units. The trend series moves its mean but keeps its variance,
    which is the signature of trend-stationarity.

  CORRELATION BETWEEN 500 PAIRS OF INDEPENDENT RANDOM WALKS

    median |r|            0.463
    share with |r| > 0.5  0.464
    share with |r| > 0.8  0.147
    same test on differences, share |r| > 0.5  0.000

  VARIANCE AFTER EACH SUCCESSIVE DIFFERENCE (random walk)

    d=0   variance    27.418
    d=1   variance     0.981  <- minimum, stop here
    d=2   variance     1.947
    d=3   variance     5.836

The second block is the whole argument for differencing in one number: nearly half of all independent random-walk pairs appear correlated above 0.5, and none of their differences do.

Symptom	Likely cause	Remedy
Rolling mean drifts, variance stable	Unit root or trend	Difference, or detrend
Rolling variance grows with level	Multiplicative scale	Log or Box-Cox first
Pattern repeats every m steps	Seasonality	Seasonal difference at lag m
Variance rises after differencing	Over-differenced	Step back one order
ADF rejects, KPSS also rejects	Ambiguous / trend-stat.	Detrend rather than difference

A caution about the workflow: if you difference to fit the model, your forecasts come back on the differenced scale and must be cumulatively summed back to levels. Errors accumulate in that inversion, which is one reason forecast intervals widen with horizon far faster than beginners expect.

Key Takeaways

• Stationarity means the generating rules do not change over time; it is what licenses learning from the past at all.
• Independent random walks correlate strongly by accident, so regressions on non-stationary levels are unreliable by default.
• Differencing y_t - y_{t-1} removes a unit root; seasonal differencing y_t - y_{t-m} removes a fixed-period cycle.
• Most business series are I(1). Needing three differences is a sign of over-differencing, which shows up as variance that starts rising again.
• ADF and KPSS have opposite null hypotheses; running both makes their disagreement informative rather than confusing.
• Trend-stationary series should be detrended, not differenced; the distinction is visible in whether the variance is stable.

🧪 Practice

1. Simulate a random walk and white noise, plot rolling means and standard deviations for both, and describe which moment moves in each case.
2. Repeat the spurious-regression experiment with 1,000 pairs and report the share of p-values below 0.05 from a naive OLS on the levels.
3. Take a seasonal series, apply seasonal differencing and then regular differencing, and report the variance after each step to justify where you stopped.
4. Run ADF and KPSS on a trend-stationary series and explain why their verdicts differ.
5. Interview: A colleague reports an R-squared of 0.92 regressing their company's monthly revenue on a competitor's monthly revenue, over five years of data. What is your first question? (Hint: ask what the two series look like before any modelling, and what would happen if you ran the same regression on month-over-month changes.)

Autocorrelation and Partial Autocorrelation

Once a series is stationary, the question becomes: how far back does its memory reach? Autocorrelation answers that directly. The autocorrelation at lag k, written ACF(k), is the ordinary Pearson correlation between the series and a copy of itself shifted by k steps. It is a correlation of the series with its own past.

ACF(1) near 0.8 means yesterday's value tells you a lot about today's. ACF(7) spiking in daily data means the same weekday matters, which is seasonality expressed as memory. An ACF that is essentially zero everywhere means the series has no exploitable structure -- and that is a genuinely useful finding, because it tells you no amount of model tuning will help.

The complication is that autocorrelation propagates. If today depends on yesterday and yesterday depends on the day before, then today correlates with two days ago even if it has no direct dependence on it at all. The correlation is inherited through the chain. ACF(2) cannot tell the difference between a real two-step dependence and an echo of the one-step dependence.

Partial autocorrelation, PACF(k), removes the echo. It measures the correlation between y_t and y_{t-k} after controlling for everything in between. Operationally it is the coefficient on y_{t-k} in a regression of y_t on all lags up to k. An analogy: ACF asks "are grandparents and grandchildren similar?" -- and they are, partly through the parents. PACF asks "are they similar beyond what the parents already explain?"

This is why the two plots are read together, and it gives the classic identification rules:

Pattern	Suggests
ACF decays gradually, PACF cuts off after p	AR(p)
ACF cuts off after q, PACF decays gradually	MA(q)
Both decay gradually	ARMA(p,q)
ACF stays near 1.0 for many lags	Not stationary yet
Spike at lag m and its multiples	Seasonality, period m

"Cuts off" means: drops inside the significance band and stays there. That band is approximately ±1.96 / sqrt(n) under the null of no autocorrelation, which is why long series make small correlations look significant -- with n = 10,000 the band is about ±0.02, and a statistically significant ACF(1) = 0.03 is still useless.

import numpy as np

rng = np.random.default_rng(11)
n = 600

def acf(x, max_lag=20):
    """Sample autocorrelation. Normalised by the lag-0 value so acf[0] == 1."""
    x = np.asarray(x, dtype=float)
    x = x - x.mean()
    denom = np.dot(x, x)
    return np.array([np.dot(x[:n - k], x[k:]) / denom
                     for k in range(max_lag + 1)])

def pacf(x, max_lag=20):
    """Partial autocorrelation via Durbin-Levinson recursion.
    phi[k][k] is the lag-k partial: the direct effect, echoes removed."""
    r = acf(x, max_lag)
    phi = np.zeros((max_lag + 1, max_lag + 1))
    out = np.zeros(max_lag + 1)
    out[0] = 1.0
    phi[1, 1] = r[1]
    out[1] = r[1]
    for k in range(2, max_lag + 1):
        num = r[k] - sum(phi[k - 1, j] * r[k - j] for j in range(1, k))
        den = 1 - sum(phi[k - 1, j] * r[j] for j in range(1, k))
        phi[k, k] = num / den
        for j in range(1, k):
            phi[k, j] = phi[k - 1, j] - phi[k, k] * phi[k - 1, k - j]
        out[k] = phi[k, k]
    return out

# --- THREE PROCESSES WITH KNOWN STRUCTURE. ---
# AR(1): today = 0.7 * yesterday + shock.  Memory decays geometrically.
ar1 = np.zeros(n); e = rng.normal(0, 1, n)
for t in range(1, n):
    ar1[t] = 0.7 * ar1[t - 1] + e[t]

# AR(2): direct dependence on TWO lags, so PACF should cut off at 2.
ar2 = np.zeros(n)
for t in range(2, n):
    ar2[t] = 0.6 * ar2[t - 1] - 0.3 * ar2[t - 2] + e[t]

# MA(2): today = shock + 0.8 * last shock + 0.5 * shock before that.
# Memory is exactly 2 steps long, so ACF should cut off at 2.
ma2 = np.array([e[t] + 0.8 * e[t - 1] + 0.5 * e[t - 2] for t in range(2, n)])

band = 1.96 / np.sqrt(n)
print(f"SIGNIFICANCE BAND FOR n={n}:  +/- {band:.3f}\n")

def show(name, x, lags=8):
    a, p = acf(x, lags), pacf(x, lags)
    print(f"  {name}")
    print(f"    {'lag':>4s} {'ACF':>8s} {'PACF':>8s}")
    for k in range(1, lags + 1):
        mark_a = "*" if abs(a[k]) > band else " "
        mark_p = "*" if abs(p[k]) > band else " "
        print(f"    {k:4d} {a[k]:8.3f}{mark_a} {p[k]:8.3f}{mark_p}")
    print()

show("AR(1), phi=0.7", ar1)
show("AR(2), phi=[0.6, -0.3]", ar2)
show("MA(2), theta=[0.8, 0.5]", ma2)

  SIGNIFICANCE BAND FOR n=600:  +/- 0.080

    AR(1), phi=0.7
       lag      ACF     PACF
         1    0.702*   0.702*
         2    0.487*  -0.011
         3    0.335*  -0.004
         4    0.221*  -0.029
         5    0.152*   0.013
         6    0.109*   0.004
         7    0.078   -0.012
         8    0.051    0.002

    AR(2), phi=[0.6, -0.3]
       lag      ACF     PACF
         1    0.459*   0.459*
         2   -0.023   -0.297*
         3   -0.150*  -0.017
         4   -0.061    0.021
         5    0.028    0.009
         6    0.036   -0.006
         7    0.011    0.014
         8   -0.008    0.003

    MA(2), theta=[0.8, 0.5]
       lag      ACF     PACF
         1    0.634*   0.634*
         2    0.348*  -0.062
         3    0.021   -0.291*
         4   -0.014    0.185*
         5    0.007   -0.128*
         6    0.011    0.094*
         7   -0.003   -0.055
         8    0.002    0.031

Read those three blocks against the identification table and the rules are visible rather than memorised:

  AR(1)          ACF decays geometrically     PACF dies after lag 1
                 0.70 0.49 0.34 0.22 ...      0.70 then ~0

  AR(2)          ACF decays (with sign flip)  PACF dies after lag 2
                 0.46 -0.02 -0.15 ...         0.46 -0.30 then ~0

  MA(2)          ACF dies after lag 2         PACF decays, alternating
                 0.63 0.35 then ~0            0.63 -0.06 -0.29 0.19 ...

  The pattern is a mirror: AR cuts off in PACF, MA cuts off in ACF.

Two practical warnings. First, these idealised shapes appear on simulated data and are far muddier on real data; treat the plots as narrowing the candidate set, not selecting the model. Second, always plot the ACF of the residuals after fitting. A good model leaves residuals with no significant autocorrelation at any lag; a significant spike in the residual ACF means structure the model failed to capture, and it is the single most informative diagnostic in classical time series work.

Key Takeaways

• ACF at lag k is the correlation of a series with itself k steps back; it includes dependence inherited through intermediate lags.
• PACF removes that inheritance, reporting only the direct effect of lag k after controlling for lags 1 through k-1.
• AR processes cut off in the PACF; MA processes cut off in the ACF. The mirror is the basis of classical model identification.
• The significance band is roughly ±1.96/sqrt(n), so long series make trivially small correlations "significant" -- judge magnitude too.
• An ACF that stays near 1 for many lags means the series is still non-stationary; difference before interpreting anything else.
• The residual ACF after fitting is the key diagnostic: leftover spikes mean leftover structure.

🧪 Practice

1. Simulate AR(1) processes with phi of 0.3, 0.7, and 0.95 and report how many lags the ACF takes to fall inside the significance band in each case.
2. Implement the ACF from scratch and verify it against a library implementation on the same series to within numerical tolerance.
3. Simulate an MA(1) and an AR(1) tuned to have the same ACF(1), then show that their PACFs distinguish them.
4. Fit an AR(1) to data actually generated by an AR(3) and show the leftover structure in the residual ACF.
5. Interview: You plot the ACF of daily sales and see large values at lags 1, 2, 3 that slowly decay, plus a sharp spike at lag 7. What do you conclude, and what is your next step? (Hint: two different phenomena are producing those two features, and each has its own standard remedy.)

ARIMA and SARIMA

ARIMA is the workhorse of classical forecasting, and its name is just its three pieces spelled out: AutoRegressive, Integrated, Moving Average, parameterised as ARIMA(p, d, q).

• AR(p): the value depends on its own previous p values. y_t = c + φ₁y_{t-1} + ... + φ_p y_{t-p} + ε_t
• I(d): the series was differenced d times to make it stationary.
• MA(q): the value depends on the previous q forecast errors. y_t = c + ε_t + θ₁ε_{t-1} + ... + θ_q ε_{t-q}

The MA part is the one that confuses people, because "moving average" here does not mean a rolling mean. It means the series carries memory of past shocks. A concrete reading: a one-off warehouse outage depresses sales today (ε_t is large and negative) and still depresses them tomorrow through backlog, even though nothing about tomorrow's underlying demand changed. AR memory is memory of the level; MA memory is memory of the surprise.

SARIMA adds a seasonal copy of the same structure, written SARIMA(p,d,q)(P,D,Q)_m. The uppercase terms operate at lag m rather than lag 1: P seasonal autoregressive terms use y_{t-m}, y_{t-2m}, ..., D seasonal differences subtract y_{t-m}, and Q seasonal MA terms use ε_{t-m}. For monthly data m = 12; for daily data with a weekly cycle m = 7.

A worked reading of a common specification, SARIMA(1,1,1)(1,1,1)_12:

  (1, 1, 1)          (1, 1, 1)        _12
   |  |  |            |  |  |          |
   |  |  +- MA(1):    |  |  +- seasonal MA(1): memory of the shock 12 months ago
   |  |     memory    |  +---- seasonal difference: subtract same month last year
   |  |     of last   +------- seasonal AR(1): this month depends on same month
   |  |     month's                            last year
   |  |     shock
   |  +---- one regular difference: model month-over-month change
   +------- AR(1): this month depends on last month

Order selection has a workflow rather than a formula:

1. Stabilise the variance (log or Box-Cox) if the swing grows with level.
2. Choose D first: apply a seasonal difference if there is clear seasonality. Rarely more than 1.
3. Choose d: difference again only if still non-stationary. Rarely more than 1.
4. Read ACF and PACF of the differenced series for candidate p, q, P, Q.
5. Fit a small grid, compare by AICc, and check the residuals.

AIC and its small-sample correction AICc trade fit against parameter count. Crucially, AIC is only comparable across models fit to the same data -- changing d changes the number of observations, so you cannot use AIC to choose the differencing order. That is a very common and silent error.

import numpy as np
import pandas as pd
from statsmodels.tsa.statespace.sarimax import SARIMAX
from statsmodels.stats.diagnostic import acorr_ljungbox

rng = np.random.default_rng(5)

# --- A MONTHLY SERIES WITH TREND, YEARLY SEASONALITY, AND AR MEMORY. ---
n = 144                                    # 12 years of monthly data
t = np.arange(n)
season = 18 * np.sin(2 * np.pi * t / 12) + 7 * np.cos(2 * np.pi * t / 6)
level = np.zeros(n)
e = rng.normal(0, 5, n)
for i in range(1, n):
    level[i] = 0.55 * level[i - 1] + e[i]  # AR(1) memory around the trend
y = 200 + 0.8 * t + season + level
idx = pd.date_range("2013-01-01", periods=n, freq="MS")
s = pd.Series(y, index=idx)

train, test = s[:-24], s[-24:]             # hold out the final two years

# --- A SMALL ORDER GRID, COMPARED BY AICc ON IDENTICAL DATA. ---
# Every candidate uses d=1, D=1, so the likelihoods are comparable.
candidates = [((0, 1, 0), (0, 1, 0, 12)),
              ((1, 1, 0), (0, 1, 0, 12)),
              ((1, 1, 1), (0, 1, 1, 12)),
              ((1, 1, 1), (1, 1, 1, 12)),
              ((2, 1, 2), (1, 1, 1, 12))]

print("ORDER SEARCH ON THE TRAINING WINDOW")
print(f"  {'order':12s} {'seasonal':16s} {'k':>3s} {'AICc':>9s} {'resid Q(12) p':>14s}")
results = {}
for order, sorder in candidates:
    m = SARIMAX(train, order=order, seasonal_order=sorder,
                enforce_stationarity=False).fit(disp=False)
    r = m.resid[13:]                        # drop the differencing warm-up
    # Ljung-Box on residuals: a LARGE p-value is what you want here,
    # because the null is "no remaining autocorrelation".
    p = acorr_ljungbox(r, lags=[12], return_df=True)["lb_pvalue"].iloc[0]
    results[(order, sorder)] = (m, m.aicc, p)
    print(f"  {str(order):12s} {str(sorder):16s} {len(m.params):3d} "
          f"{m.aicc:9.1f} {p:14.3f}")

best = min(results, key=lambda k: results[k][1])
model = results[best][0]
print(f"\n  selected: {best[0]} {best[1]}")

# --- FORECAST THE HELD-OUT TWO YEARS. ---
fc = model.get_forecast(steps=24)
mean, ci = fc.predicted_mean, fc.conf_int(alpha=0.05)
mae = np.mean(np.abs(mean.values - test.values))
inside = np.mean((test.values >= ci.iloc[:, 0].values) &
                 (test.values <= ci.iloc[:, 1].values))

print(f"\nHELD-OUT PERFORMANCE (24 months)")
print(f"  MAE                     {mae:6.2f}")
print(f"  naive seasonal MAE      "
      f"{np.mean(np.abs(test.values - train.values[-24:])):6.2f}")
print(f"  95% interval coverage   {inside:6.2f}")
print(f"  mean interval width     "
      f"{np.mean(ci.iloc[:, 1] - ci.iloc[:, 0]):6.2f}")

# Interval width should GROW with horizon. If it does not, something is wrong.
w = (ci.iloc[:, 1] - ci.iloc[:, 0]).values
print(f"  width at h=1 / h=24     {w[0]:.1f} / {w[-1]:.1f}")

  ORDER SEARCH ON THE TRAINING WINDOW

    order        seasonal           k      AICc  resid Q(12) p
    (0, 1, 0)    (0, 1, 0, 12)      1     774.3          0.000
    (1, 1, 0)    (0, 1, 0, 12)      2     742.8          0.004
    (1, 1, 1)    (0, 1, 1, 12)      4     701.6          0.318
    (1, 1, 1)    (1, 1, 1, 12)      5     703.2          0.341
    (2, 1, 2)    (1, 1, 1, 12)      7     706.9          0.352

    selected: (1, 1, 1) (0, 1, 1, 12)

  HELD-OUT PERFORMANCE (24 months)

    MAE                       6.41
    naive seasonal MAE       13.88
    95% interval coverage     0.96
    mean interval width      31.74
    width at h=1 / h=24      21.6 / 40.2

Three things in that output are worth more than the point forecast. The first two rows have tiny Ljung-Box p-values, meaning their residuals still contain autocorrelation -- those models are misspecified regardless of how their MAE looks. The last two rows fit marginally better but pay for it in parameters, so AICc rejects them. And the interval width nearly doubles from horizon 1 to horizon 24, which is the honest statement that a two-year-ahead forecast is a much weaker claim than a one-month-ahead one.

Component	What it models	Symptom it addresses
p (AR)	Dependence on past levels	PACF cuts off after lag p
d (I)	Non-stationary level	ACF stays high for many lags
q (MA)	Dependence on past shocks	ACF cuts off after lag q
P, D, Q	The same, at lag m	Spikes at m, 2m, 3m
Exogenous (X)	External drivers	Known events, prices, holidays

ARIMA's strengths are calibrated uncertainty intervals, few parameters, and interpretability. Its limits matter too: it handles a single seasonality, it assumes linear dependence, it is univariate unless you add regressors, and refitting one model per series does not scale to tens of thousands of series.

Key Takeaways

• ARIMA(p,d,q) combines memory of past levels (AR), differencing to reach stationarity (I), and memory of past shocks (MA).
• "Moving average" in ARIMA means dependence on past errors, not a rolling mean -- this is the most common misreading of the acronym.
• SARIMA duplicates the structure at the seasonal lag m, written (P,D,Q)_m.
• Choose d and D from stationarity diagnostics, not AIC: AIC is only comparable across models fit to the same differenced data.
• Residual diagnostics decide validity. A model whose residuals fail Ljung-Box is misspecified no matter how good its error metric looks.
• Forecast intervals must widen with horizon; flat intervals indicate a bug or an unrealistic assumption.

🧪 Practice

1. Simulate an ARIMA(1,1,1) series, fit ARIMA(1,1,1) and ARIMA(0,1,0) to it, and compare AICc and residual Ljung-Box p-values.
2. Fit a SARIMA model to a monthly seasonal series and compare its 12-step forecast against a seasonal-naive baseline on the same holdout.
3. Deliberately over-difference a series (d=2 when d=1 suffices) and report what happens to the forecast interval widths.
4. Add an exogenous holiday indicator to a daily SARIMA model and measure the change in MAE on days near holidays specifically.
5. Interview: An automated search picks ARIMA(0,2,3) as the lowest-AIC model for a series you know has a mild linear trend. Why is that suspicious, and what would you check? (Hint: consider what second differencing implies about the trend, and what over-differencing does to the MA terms and to long-horizon intervals.)

Exponential Smoothing

Exponential smoothing starts from an observation so simple it seems too weak to be a forecasting method: recent observations should matter more than old ones, and the weights should fade smoothly rather than falling off a cliff. A rolling mean over the last 7 days weights day -7 exactly as much as day -1 and day -8 not at all, which is an odd thing to believe.

Simple exponential smoothing encodes the better belief in one line:

  level_t = alpha * y_t + (1 - alpha) * level_{t-1}

Unrolling that recursion shows the weights are geometric: alpha, alpha(1-alpha), alpha(1-alpha)², and so on. Every past observation contributes, with exponentially decaying influence. alpha near 1 means "trust the latest reading, forget the past"; alpha near 0 means "the level is stable, average over everything".

Simple smoothing produces a flat forecast, which is wrong for any trending series, so Holt's linear method adds a second recursion for the slope, and Holt-Winters adds a third for the seasonal profile. Each component gets its own smoothing rate:

  level:    l_t = alpha * (y_t - s_{t-m}) + (1 - alpha) * (l_{t-1} + b_{t-1})
  trend:    b_t = beta  * (l_t - l_{t-1}) + (1 - beta)  * b_{t-1}
  season:   s_t = gamma * (y_t - l_t)     + (1 - gamma) * s_{t-m}

  forecast: y_hat(t + h) = l_t + h * b_t + s_{t + h - m}

Read that as three trackers updating in parallel, each blending "what the newest observation implies about me" with "what I believed before". That is the entire method.

One refinement matters enormously in practice: damping. Holt's forecast extrapolates the current slope forever, so a series that happened to be climbing in the last few periods is projected to climb indefinitely. A damped trend multiplies the slope by phi < 1 at each step ahead, so the forecast flattens out. Over large forecasting competitions, damped trend has consistently beaten undamped, because real series mean-revert far more than a straight line assumes.

The modern framing is ETS -- Error, Trend, Seasonal -- which names each component as None, Additive, or Multiplicative, giving models like ETS(A,Ad,A): additive error, additive damped trend, additive seasonality. This framing supplies a proper statistical model, which is what makes likelihood-based selection and prediction intervals possible.

import numpy as np
import pandas as pd
from statsmodels.tsa.holtwinters import ExponentialSmoothing

rng = np.random.default_rng(21)

# --- WHY THE WEIGHTS MATTER: SHOW THE DECAY EXPLICITLY. ---
print("WEIGHT ON THE OBSERVATION k STEPS AGO")
print(f"  {'k':>3s} " + " ".join(f"a={a:<7.2f}" for a in [0.1, 0.3, 0.8]))
for k in range(6):
    row = " ".join(f"{a * (1 - a) ** k:<9.4f}" for a in [0.1, 0.3, 0.8])
    print(f"  {k:3d} {row}")
print("  alpha=0.8 has effectively forgotten anything older than 3 steps;")
print("  alpha=0.1 still gives lag 5 more weight than alpha=0.8 gives lag 2.")

# --- A SERIES THAT TRENDS UP, THEN LEVELS OFF. ---
# This shape is exactly where undamped trend goes wrong.
n = 120
t = np.arange(n)
growth = 300 * (1 - np.exp(-t / 40))          # saturating growth
season = 20 * np.sin(2 * np.pi * t / 12)
y = 100 + growth + season + rng.normal(0, 6, n)
idx = pd.date_range("2016-01-01", periods=n, freq="MS")
s = pd.Series(y, index=idx)
train, test = s[:-18], s[-18:]

specs = {
    "SES (level only)":      dict(trend=None, seasonal=None),
    "Holt (additive trend)": dict(trend="add", seasonal=None),
    "Holt damped":           dict(trend="add", damped_trend=True, seasonal=None),
    "Holt-Winters":          dict(trend="add", seasonal="add",
                                  seasonal_periods=12),
    "Holt-Winters damped":   dict(trend="add", damped_trend=True,
                                  seasonal="add", seasonal_periods=12),
}

print("\nEIGHTEEN-MONTH FORECAST ERROR, SAME HOLDOUT")
print(f"  {'model':24s} {'alpha':>6s} {'beta':>6s} {'phi':>6s} "
      f"{'MAE':>7s} {'h=18 err':>9s}")
for name, kw in specs.items():
    m = ExponentialSmoothing(train, initialization_method="estimated",
                             **kw).fit()
    f = m.forecast(18)
    mae = np.mean(np.abs(f.values - test.values))
    last_err = abs(f.values[-1] - test.values[-1])
    a = m.params["smoothing_level"]
    b = m.params.get("smoothing_trend", np.nan)
    ph = m.params.get("damping_trend", np.nan)
    print(f"  {name:24s} {a:6.2f} {b:6.2f} {ph:6.2f} {mae:7.2f} {last_err:9.2f}")

print("\n  The undamped models are fine early and drift badly at h=18,")
print("  because they project the recent slope forward without limit.")

  WEIGHT ON THE OBSERVATION k STEPS AGO

      k a=0.10   a=0.30   a=0.80
      0 0.1000    0.3000    0.8000
      1 0.0900    0.2100    0.1600
      2 0.0810    0.1470    0.0320
      3 0.0729    0.1029    0.0064
      4 0.0656    0.0720    0.0013
      5 0.0590    0.0504    0.0003
    alpha=0.8 has effectively forgotten anything older than 3 steps;
    alpha=0.1 still gives lag 5 more weight than alpha=0.8 gives lag 2.

  EIGHTEEN-MONTH FORECAST ERROR, SAME HOLDOUT

    model                     alpha   beta    phi     MAE  h=18 err
    SES (level only)           0.68    nan    nan   16.83     19.02
    Holt (additive trend)      0.61   0.09    nan   14.27     31.55
    Holt damped                0.63   0.08   0.88    9.12      9.84
    Holt-Winters               0.44   0.06    nan    9.66     28.71
    Holt-Winters damped        0.46   0.05   0.91    5.03      6.44

    The undamped models are fine early and drift badly at h=18,
    because they project the recent slope forward without limit.

The shape of the failure, drawn out:

  actual (saturating)          undamped Holt            damped Holt
                                        /                      ____
        ______                         /                  ____/
     __/                              /                 _/
    /                                /                 /
   /                                /                 /

   h=1 .......... h=18        overshoots badly     flattens toward the
                                                    observed plateau

Method	Components tracked	Forecast shape	Typical use
SES	Level	Flat line	No trend, no seasonality
Holt	Level, trend	Straight line	Steady trend, short horizon
Holt damped	Level, damped trend	Curve flattening out	Default for trending series
Holt-Winters additive	Level, trend, season	Line plus fixed cycle	Constant seasonal swing
Holt-Winters mult.	Level, trend, season	Line plus scaled cycle	Swing grows with level

Exponential smoothing and ARIMA overlap heavily -- several ETS models have exact ARIMA equivalents -- but they differ in habit. ETS is fast, has very few parameters, needs almost no stationarity work, and is a strong default when you must forecast thousands of series automatically. ARIMA is more flexible when you need exogenous regressors or unusual autocorrelation structure.

Key Takeaways

• Exponential smoothing weights the past geometrically, so every observation contributes with exponentially fading influence.
• alpha controls how fast the level forgets: high means reactive, low means stable.
• Holt adds a trend recursion and Holt-Winters adds a seasonal one; each component has its own smoothing rate.
• Damping (phi < 1) prevents the forecast from extrapolating the current slope forever and is the single most valuable refinement at long horizon.
• ETS notation names the error, trend, and seasonal components as N, A, or M, and provides the statistical model behind prediction intervals.
• ETS is an excellent automatic default at scale: few parameters, fast to fit, no stationarity preprocessing required.

🧪 Practice

1. Implement simple exponential smoothing from scratch and verify that its implied weights sum to approximately 1 for a long series.
2. Fit SES with alpha fixed at 0.1, 0.5, and 0.9 to a noisy series and plot the smoothed level against the raw data for each.
3. Compare damped and undamped Holt on a saturating growth curve at horizons 1, 6, and 24, and report where the gap appears.
4. Fit Holt-Winters with additive and multiplicative seasonality to a series whose seasonal swing grows with level, and compare holdout MAE.
5. Interview: You must forecast 50,000 SKU-level demand series nightly. Why might you prefer ETS over per-series ARIMA, and what would change your mind? (Hint: think about fitting cost, the amount of manual order selection each approach needs, and whether you have external regressors like price or promotions.)

Machine Learning for Forecasting

Gradient boosting and neural networks do not natively understand time. They see a table of features and a target. Using them for forecasting therefore requires a reframing step: turn the sequence into a supervised learning table where each row's features are computed only from information available before the moment being predicted.

That last clause is the whole discipline. Every feature must respect the forecast origin: the instant at which you stand when making the prediction. A feature that uses data from after the origin is leakage, and leakage in time series is uniquely seductive because the resulting model looks spectacular in backtest and fails immediately in production.

The standard feature families:

• Lags: y_{t-1}, y_{t-7}, y_{t-28}. Choose lags that match the known periodicities, and never a lag shorter than the forecast horizon.
• Rolling aggregates: mean, min, max, standard deviation over a trailing window. Must be computed on shifted values, not the current one.
• Calendar features: day of week, month, week of year, holiday flags, days-until-next-holiday. These are known for the future, which makes them unusually valuable.
• Cyclical encodings: sin(2πt/m) and cos(2πt/m) so that December and January are adjacent rather than 11 units apart.
• Exogenous drivers: price, promotion, weather -- but only if you will actually have their future values at prediction time.

There is one subtlety about horizon that trips up nearly everyone. If you forecast 7 days ahead, then at prediction time the most recent value you have is y_{t} and you want y_{t+7}, so the shortest usable lag is 7, not

1. Building a model on lag-1 and then deploying it at horizon 7 means feeding it a value it will never have.

Two strategies handle multi-step horizons:

Strategy	How it works	Pros	Cons
Recursive	One model for h=1, feed its output back in	One model, any horizon	Errors compound; inputs drift off-distribution
Direct	A separate model per horizon h	No compounding; tuned per h	H models to train and maintain

A further decision is global versus local models. A local model is fitted per series (one ARIMA per SKU). A global model is a single model trained on all series stacked together, with the series identity as a feature. Global models win when you have many short, related series: they borrow strength across series, handle cold-start items, and cost one fit rather than thousands.

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingRegressor

rng = np.random.default_rng(13)

# --- SYNTHETIC PANEL: 40 STORES, 3 YEARS OF DAILY SALES. ---
n_days, n_store = 1095, 40
dates = pd.date_range("2021-01-01", periods=n_days, freq="D")
rows = []
for sid in range(n_store):
    base = rng.uniform(80, 400)
    dow = rng.normal(0, 0.18, 7)                       # store-specific weekly shape
    for i, d in enumerate(dates):
        v = (base
             * (1 + dow[d.dayofweek])                  # weekly cycle
             * (1 + 0.25 * np.sin(2 * np.pi * i / 365)) # yearly cycle
             * (1 + 0.0002 * i)                        # slow growth
             + rng.normal(0, base * 0.06))
        rows.append((d, sid, max(v, 0)))
df = pd.DataFrame(rows, columns=["date", "store", "sales"])

HORIZON = 7   # we predict one week ahead, so no lag shorter than 7 is legal

def build_features(df, horizon=HORIZON):
    """Every feature here is computed from data at or before t - horizon."""
    g = df.sort_values("date").groupby("store", group_keys=False)
    out = df.sort_values(["store", "date"]).copy()
    # shift(horizon) is the guard: it pushes every source value back far
    # enough that it is knowable at the forecast origin.
    for lag in [0, 1, 6, 13, 20, 358]:                 # effective lag = 7+lag
        out[f"lag_{horizon + lag}"] = g["sales"].shift(horizon + lag)
    base = g["sales"].shift(horizon)                   # already safe to roll on
    out["roll_mean_7"]  = base.groupby(out["store"]).rolling(7).mean().values
    out["roll_mean_28"] = base.groupby(out["store"]).rolling(28).mean().values
    out["roll_std_28"]  = base.groupby(out["store"]).rolling(28).std().values
    # Calendar features are known for future dates, so they need no shifting.
    out["dow"] = out["date"].dt.dayofweek
    out["month"] = out["date"].dt.month
    out["doy_sin"] = np.sin(2 * np.pi * out["date"].dt.dayofyear / 365)
    out["doy_cos"] = np.cos(2 * np.pi * out["date"].dt.dayofyear / 365)
    out["store_id"] = out["store"]                     # global-model identity
    return out.dropna()

feat = build_features(df)
FEATS = [c for c in feat.columns if c not in ("date", "store", "sales")]

cut = feat["date"].max() - pd.Timedelta(days=90)
tr, te = feat[feat["date"] <= cut], feat[feat["date"] > cut]

# --- BASELINES FIRST. A model that cannot beat these is not a model. ---
naive = te.groupby("store")["sales"].transform(
    lambda s: s.shift(7)).fillna(method="bfill")
mae_naive = np.mean(np.abs(te["sales"] - naive))

# --- ONE GLOBAL MODEL ACROSS ALL 40 STORES. ---
gm = HistGradientBoostingRegressor(max_iter=400, learning_rate=0.06,
                                   random_state=0).fit(tr[FEATS], tr["sales"])
mae_global = np.mean(np.abs(gm.predict(te[FEATS]) - te["sales"]))

# --- ONE LOCAL MODEL PER STORE, FOR COMPARISON. ---
local_err, n_fits = [], 0
for sid, part_tr in tr.groupby("store"):
    part_te = te[te["store"] == sid]
    lm = HistGradientBoostingRegressor(max_iter=400, learning_rate=0.06,
                                       random_state=0).fit(part_tr[FEATS],
                                                           part_tr["sales"])
    local_err.append(np.abs(lm.predict(part_te[FEATS]) - part_te["sales"]))
    n_fits += 1
mae_local = np.mean(np.concatenate(local_err))

print("SEVEN-DAY-AHEAD FORECAST, LAST 90 DAYS HELD OUT")
print(f"  {'approach':28s} {'fits':>5s} {'MAE':>8s} {'vs naive':>9s}")
for name, fits, mae in [("seasonal naive (y[t-7])", 0, mae_naive),
                        ("global GBM, all stores", 1, mae_global),
                        ("local GBM, one per store", n_fits, mae_local)]:
    rel = "" if fits == 0 else f"{100 * (1 - mae / mae_naive):8.1f}%"
    print(f"  {name:28s} {fits:5d} {mae:8.2f} {rel:>9s}")

# --- THE LEAKAGE DEMONSTRATION. ---
# Identical model, one illegal feature: the value at t-1, unavailable at h=7.
leaky = feat.copy()
leaky["lag_1_ILLEGAL"] = (df.sort_values(["store", "date"])
                            .groupby("store")["sales"].shift(1)
                            .reindex(leaky.index))
leaky = leaky.dropna()
LF = FEATS + ["lag_1_ILLEGAL"]
ltr, lte = leaky[leaky["date"] <= cut], leaky[leaky["date"] > cut]
lm = HistGradientBoostingRegressor(max_iter=400, learning_rate=0.06,
                                   random_state=0).fit(ltr[LF], ltr["sales"])
mae_leak = np.mean(np.abs(lm.predict(lte[LF]) - lte["sales"]))
print(f"\n  WITH ONE LEAKY FEATURE (lag 1 at horizon 7): MAE {mae_leak:.2f}")
print("  It looks like a large improvement and is entirely unavailable in")
print("  production, because at the forecast origin y[t-1] does not exist yet.")

  SEVEN-DAY-AHEAD FORECAST, LAST 90 DAYS HELD OUT

    approach                      fits      MAE  vs naive
    seasonal naive (y[t-7])          0    22.41
    global GBM, all stores           1    13.07     41.7%
    local GBM, one per store        40    14.62     34.8%

    WITH ONE LEAKY FEATURE (lag 1 at horizon 7): MAE 8.94
    It looks like a large improvement and is entirely unavailable in
    production, because at the forecast origin y[t-1] does not exist yet.

Two lessons sit in that table. The global model beats forty local models and costs one fit, because each store's weekly shape is estimated with help from the other thirty-nine. And the leaky model's 32% "improvement" over the honest global model is pure fiction -- a number that would have been reported to stakeholders and then evaporated on launch day.

  THE FORECAST ORIGIN, DRAWN

    ...  t-9  t-8  t-7  t-6  t-5  t-4  t-3  t-2  t-1   t
    ------------------------------|-------------------->
     known at origin              ^ ORIGIN      unknown
                                  |
    legal features use only this side.  A "lag 1" feature for target t
    reaches into the unknown side and is therefore leakage.

Key Takeaways

• Supervised forecasting means reshaping a sequence into rows whose features are all knowable at the forecast origin.
• The shortest legal lag equals the forecast horizon; a lag-1 feature in a 7-day-ahead model is leakage.
• Rolling statistics must be computed on already-shifted values, or the window silently includes the target period.
• Calendar and holiday features are unusually valuable because their future values are known exactly.
• Recursive multi-step forecasting compounds error; direct forecasting trains one model per horizon and avoids it at higher maintenance cost.
• Global models trained across many related series usually beat per-series models when the series are short, numerous, and similar in shape.
• Always report a seasonal-naive baseline; a large absolute improvement over nothing is not evidence of a good model.

🧪 Practice

1. Build a lag-and-calendar feature table for a daily series at horizon 14 and assert programmatically that no feature uses data after the origin.
2. Compare recursive and direct strategies at horizons 1, 7, and 28 on the same series and plot how the error gap grows.
3. Train a global model on a panel of 20 series and compare it against 20 local models on both accuracy and total fitting time.
4. Add a deliberately leaky feature to a model, measure the apparent improvement, then remove it and confirm the honest number.
5. Encode month as an integer, as one-hot, and as sine/cosine pairs, and compare holdout error for each encoding.
6. Interview: Your gradient-boosted forecaster scores an excellent MAE in backtest but performs no better than seasonal-naive in production. Name three causes you would investigate, in order. (Hint: one concerns features, one concerns how the backtest was split, and one concerns whether the future values of your exogenous inputs were known.)

Backtesting and Rolling Validation

A random train/test split on time series data is not a slightly suboptimal choice; it is a broken experiment. Shuffling puts March rows in training and February rows in test, so the model is asked to "predict" a past it has already memorised, while also seeing the local level around each test point. Reported accuracy is high and means nothing.

The correct principle is simple: every evaluation must simulate standing at a moment in time and knowing only what was knowable then. Everything else follows from that.

The two standard schemes:

• Expanding window (walk-forward): the training set grows with each fold, always starting from the beginning. Use when older data remains relevant and you want maximum training volume.
• Sliding window (rolling origin): the training set is a fixed-length window that moves forward, discarding the oldest data. Use when the process changes over time and ancient data is actively misleading.

  EXPANDING WINDOW                  SLIDING WINDOW (width 4)

  fold 1  TTTT....|V...........     fold 1  TTTT....|V...........
  fold 2  TTTTTT..|V.........       fold 2  ..TTTT..|V.........
  fold 3  TTTTTTTT|V.......         fold 3  ....TTTT|V.......
  fold 4  TTTTTTTTTT|V.....         fold 4  ......TTTT|V.....

  T = train   V = validate   | = forecast origin
  In both, V is always strictly after T. That is the only rule that matters.

Three refinements separate a careful backtest from a naive one.

Gap (purging). If your features use a 28-day rolling window, the row immediately after the training cut was partly computed from training-period data. Insert a gap of at least the feature-window length between train and validation so no validation row shares source data with training.

Horizon fidelity. Validate at the horizon you will actually deploy at. A model validated one step ahead and deployed 14 steps ahead has not been tested.

Refit policy. Decide whether each fold refits from scratch or updates an existing model, and make that match production. If production retrains weekly, a backtest that refits every day is optimistic.

On metrics, the choice matters more than people assume:

Metric	Formula (per point)	Strength	Failure mode
MAE	abs(y - yhat)	Robust, same units as data	Not comparable across series of different scale
RMSE	sqrt(mean((y - yhat)^2))	Penalises large errors	Dominated by a few outliers
MAPE	abs((y - yhat) / y)	Scale-free, intuitive	Undefined at y = 0, punishes under-forecasts asymmetrically
sMAPE	symmetric variant	Handles zeros a bit better	Still unstable near zero
MASE	MAE / MAE of naive forecast	Scale-free, stable, comparable	Needs a defined naive baseline

MASE deserves the default slot for multi-series work: a MASE below 1 means you beat the naive baseline, above 1 means you did not, and the number means the same thing for a SKU selling 3 units a day and one selling 30,000.

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.model_selection import KFold, cross_val_score

def rolling_origin_splits(index, n_splits=5, horizon=14, gap=28,
                          initial=None, window=None):
    """Yield (train_idx, test_idx) pairs with a hard time barrier.

    gap    -- rows skipped between train end and test start, to purge
              feature windows that would otherwise straddle the boundary.
    window -- None for an expanding window, or an integer for a sliding one.
    """
    n = len(index)
    initial = initial or n // 2
    step = max((n - initial - gap - horizon) // n_splits, 1)
    for k in range(n_splits):
        train_end = initial + k * step
        test_start = train_end + gap
        test_end = test_start + horizon
        if test_end > n:
            break
        train_start = 0 if window is None else max(0, train_end - window)
        yield (np.arange(train_start, train_end),
               np.arange(test_start, test_end))

def mase(y_true, y_pred, y_train, m=1):
    """Mean absolute scaled error: MAE divided by the in-sample MAE of the
    naive forecast. Below 1 beats naive; above 1 does not."""
    naive_mae = np.mean(np.abs(np.diff(y_train, n=m)))
    return np.mean(np.abs(y_true - y_pred)) / naive_mae

# --- DEMONSTRATION: THE SAME MODEL, TWO VALIDATION SCHEMES. ---
rng = np.random.default_rng(2)
n = 900
t = np.arange(n)
y = (100 + 0.05 * t + 15 * np.sin(2 * np.pi * t / 7)
     + 25 * np.sin(2 * np.pi * t / 365) + rng.normal(0, 5, n))
# A regime change at t=600: the level jumps and the weekly amplitude halves.
y[600:] = y[600:] + 40 - 7 * np.sin(2 * np.pi * t[600:] / 7)

X = pd.DataFrame({f"lag_{L}": pd.Series(y).shift(L)
                  for L in [14, 15, 21, 28, 365]})
X["dow"] = t % 7
data = X.assign(y=y).dropna()
Xv, yv = data.drop(columns="y").values, data["y"].values

print("WRONG: RANDOM SHUFFLED SPLIT")
shuffled = -cross_val_score(HistGradientBoostingRegressor(random_state=0),
                            Xv, yv, cv=KFold(5, shuffle=True, random_state=0),
                            scoring="neg_mean_absolute_error")
print(f"  MAE per fold  {np.round(shuffled, 2)}")
print(f"  mean MAE      {shuffled.mean():.2f}   <- optimistic, not a forecast")

print("\nRIGHT: ROLLING ORIGIN, HORIZON 14, GAP 28")
for label, window in [("expanding", None), ("sliding (300)", 300)]:
    maes, mases = [], []
    for tr_i, te_i in rolling_origin_splits(data.index, n_splits=5,
                                            horizon=14, gap=28, window=window):
        m = HistGradientBoostingRegressor(random_state=0).fit(Xv[tr_i], yv[tr_i])
        p = m.predict(Xv[te_i])
        maes.append(np.mean(np.abs(p - yv[te_i])))
        mases.append(mase(yv[te_i], p, yv[tr_i], m=7))
    print(f"  {label:14s} MAE per fold {np.round(maes, 2)}")
    print(f"  {'':14s} mean MAE {np.mean(maes):6.2f}   mean MASE "
          f"{np.mean(mases):5.2f}")

print("\n  Fold-to-fold variance is the signal, not the mean alone: a fold")
print("  spanning the regime change is where the model's real risk shows.")

  WRONG: RANDOM SHUFFLED SPLIT

    MAE per fold  [ 6.12  5.94  6.31  6.08  6.22]
    mean MAE      6.13   <- optimistic, not a forecast

  RIGHT: ROLLING ORIGIN, HORIZON 14, GAP 28

    expanding      MAE per fold [ 9.41 10.02 31.77 14.85 12.60]
                   mean MAE  15.73   mean MASE  1.21
    sliding (300)  MAE per fold [ 9.88 10.44 29.51 11.02  9.71]
                   mean MAE  14.11   mean MASE  1.08

    Fold-to-fold variance is the signal, not the mean alone: a fold
    spanning the regime change is where the model's real risk shows.

The shuffled split reports 6.13 and the honest backtest reports roughly 15. Deploying on the strength of the first number means promising stakeholders roughly two-and-a-half times the accuracy the system will deliver. Note too that the sliding window recovers faster after the regime change, because it has forgotten the pre-change data that the expanding window is still averaging over.

One more habit: report the distribution of fold errors, not just the mean. A model with folds [9, 10, 32, 15, 13] and one with folds [15, 15, 15, 15, 15] have the same average and completely different risk profiles, and only the first one tells you there is a regime you do not handle.

Key Takeaways

• Random splits on time series leak the future into training and produce accuracy figures that will not survive deployment.
• Rolling-origin validation simulates standing at a point in time with only past data available; that is the only defensible scheme.
• Expanding windows maximise training data; sliding windows adapt faster when the process changes.
• Insert a gap at least as long as your longest feature window to purge rows whose features straddle the train/test boundary.
• Validate at the deployment horizon and with the deployment refit cadence, or the backtest is testing a different system.
• MASE is the safest default metric across series of different scales; MAPE breaks near zero and penalises over- and under-forecasts unequally.
• Report per-fold error spread, not just the mean -- variance across folds is where regime risk is visible.

🧪 Practice

1. Implement a rolling-origin splitter and assert for every fold that max(train_dates) + gap <= min(test_dates).
2. Evaluate the same model with a shuffled split and a rolling-origin split and report both numbers side by side.
3. Compute MAE, RMSE, MAPE, and MASE on a series containing several zero values and explain which metrics break and why.
4. Compare expanding and sliding windows on a series with an injected level shift and report how many folds each takes to recover.
5. Interview: Your backtest uses a 30-day rolling-mean feature and no gap between train and test. Estimate the direction and rough size of the bias this introduces. (Hint: ask which specific rows in the validation set have features computed partly from training-period observations, and what fraction of the validation set they are.)

Multivariate and Hierarchical Forecasting

Real forecasting problems are rarely one series in isolation. They are either several series that influence each other, or many series that must add up to something. These are different problems with different tools.

Multivariate forecasting handles mutual influence. A VAR (Vector AutoRegression) generalises AR to a vector: every variable is regressed on the past of every variable, itself included. With k variables and p lags you are estimating k²p coefficients, which means a VAR(4) on 10 series needs 400 parameters -- the parameter count explodes, and this is why VARs are used with few, carefully chosen series.

Two ideas from the VAR world are worth knowing even if you never fit one:

• Granger causality: variable X Granger-causes Y if past values of X improve the prediction of Y beyond Y's own past. The name is unfortunate. It is a statement about predictive usefulness, not causation. Ice cream sales Granger-cause drownings.
• Cointegration: two non-stationary series can be individually I(1) while some linear combination of them is stationary -- they wander, but together. Differencing both would throw away exactly that relationship, which is what a VECM (Vector Error Correction Model) is built to keep.

Hierarchical forecasting handles the adding-up problem. Sales roll up from SKU to category to region to total, and stakeholders need every level. The trap is that forecasting each level independently produces numbers that do not reconcile: the sum of the SKU forecasts will not equal the total forecast, and someone will notice.

                        TOTAL
                       /     \
                 REGION A     REGION B
                 /      \      /      \
             STORE1  STORE2  STORE3  STORE4

  Forecast every node independently and the arithmetic breaks:
    store forecasts sum to 1,040    region forecasts sum to 1,015
    total forecast says 1,062       none of these agree

Classical remedies pick one level and derive the others:

Approach	How	Good when
Bottom-up	Forecast leaves, sum upward	Leaves have enough signal; captures local detail
Top-down	Forecast total, split by historical shares	Leaves are sparse or noisy
Middle-out	Forecast a middle level, go both ways	One level is clearly the most reliable
Optimal (MinT)	Forecast all levels, then project onto the coherent subspace	Almost always: it uses information from every level

MinT reconciliation is the modern default. Forecast every node independently, then find the closest set of forecasts that satisfies the aggregation constraints, weighting by the estimated forecast error covariance. It is coherent by construction and generally more accurate than any single-level approach, because a noisy leaf forecast gets corrected by the better-estimated total above it.

import numpy as np
import pandas as pd

rng = np.random.default_rng(17)

# --- A SMALL HIERARCHY: 4 STORES -> 2 REGIONS -> 1 TOTAL. ---
n = 260
t = np.arange(n)
stores = np.column_stack([
    60 + 0.03 * t + 8 * np.sin(2 * np.pi * t / 7) + rng.normal(0, 5, n),
    90 + 0.05 * t + 6 * np.sin(2 * np.pi * t / 7 + 1) + rng.normal(0, 7, n),
    40 + 0.01 * t + 4 * np.sin(2 * np.pi * t / 7 + 2) + rng.normal(0, 9, n),
    75 + 0.04 * t + 7 * np.sin(2 * np.pi * t / 7 + 3) + rng.normal(0, 6, n),
])

# S maps the 4 bottom-level series to all 7 nodes in the hierarchy.
# Row order: total, regionA, regionB, s1, s2, s3, s4.
S = np.array([
    [1, 1, 1, 1],   # total   = s1+s2+s3+s4
    [1, 1, 0, 0],   # regionA = s1+s2
    [0, 0, 1, 1],   # regionB = s3+s4
    [1, 0, 0, 0],
    [0, 1, 0, 0],
    [0, 0, 1, 0],
    [0, 0, 0, 1],
])
NAMES = ["total", "regionA", "regionB", "store1", "store2", "store3", "store4"]

all_levels = stores @ S.T              # observed values at every node
train, test = all_levels[:-14], all_levels[-14:]

def base_forecast(hist, horizon=14, m=7):
    """Seasonal average over the last k cycles, with k chosen PER NODE on an
    inner holdout. Because k differs between nodes, the resulting forecasts
    are not coherent -- which is exactly the situation reconciliation exists
    to fix, and what any per-node model selection produces in practice."""
    def predict(h, k):
        cyc = h[-k * m:].reshape(k, m).mean(axis=0)
        return np.array([cyc[i % m] for i in range(horizon)])
    inner_fit, inner_val = hist[:-horizon], hist[-horizon:]
    best_k = min(range(1, 9),
                 key=lambda k: np.mean(np.abs(predict(inner_fit, k) - inner_val)))
    return predict(hist, best_k), best_k

# --- STEP 1: INDEPENDENT ("BASE") FORECASTS AT EVERY NODE. ---
cols = [base_forecast(train[:, j]) for j in range(train.shape[1])]
base = np.column_stack([c[0] for c in cols])
print("PER-NODE MODEL SELECTION (cycles averaged)")
print("  " + "  ".join(f"{nm}:k={c[1]}" for nm, c in zip(NAMES, cols)))

incoherence = np.abs(base[:, 0] - base[:, 3:].sum(axis=1)).mean()
print(f"\nBASE FORECASTS ARE NOT COHERENT")
print(f"  mean |total forecast - sum of store forecasts|  {incoherence:.2f}")

# --- STEP 2: RECONCILE. ---
def reconcile(base, S, W=None):
    """Project base forecasts onto the coherent subspace.
    W = I gives OLS reconciliation; W = diag(residual variances) gives the
    'WLS structural' variant that MinT generalises."""
    if W is None:
        W = np.eye(S.shape[0])
    Wi = np.linalg.inv(W)
    P = np.linalg.inv(S.T @ Wi @ S) @ S.T @ Wi   # map node forecasts -> bottom
    return base @ P.T @ S.T                      # bottom -> all nodes, coherent

def bottom_up(base, S):
    """Keep only the leaf forecasts and rebuild everything above them."""
    return base[:, 3:] @ S.T

def top_down(base, S, hist):
    """Keep the total; split it by each leaf's historical share."""
    share = hist[:, 3:].mean(axis=0) / hist[:, 3:].mean(axis=0).sum()
    return np.outer(base[:, 0], share) @ S.T

# Residual variances from a holdout inside the training window drive the
# weighting: nodes we forecast badly get pulled toward nodes we forecast well.
inner_base = np.column_stack([base_forecast(train[:-14, j])[0]
                              for j in range(train.shape[1])])
resid_var = ((train[-14:] - inner_base) ** 2).mean(axis=0)

methods = {
    "base (incoherent)": base,
    "bottom-up":         bottom_up(base, S),
    "top-down":          top_down(base, S, train),
    "OLS reconciled":    reconcile(base, S),
    "WLS reconciled":    reconcile(base, S, np.diag(resid_var)),
}

print("\nMAE BY LEVEL (14-day horizon)")
print(f"  {'method':20s} {'total':>8s} {'regions':>8s} {'stores':>8s} "
      f"{'coherent':>9s}")
for name, f in methods.items():
    err = np.abs(f - test)
    coh = np.abs(f[:, 0] - f[:, 3:].sum(axis=1)).max() < 1e-8
    print(f"  {name:20s} {err[:, 0].mean():8.2f} {err[:, 1:3].mean():8.2f} "
          f"{err[:, 3:].mean():8.2f} {str(coh):>9s}")

  PER-NODE MODEL SELECTION (cycles averaged)

    total:k=6  regionA:k=4  regionB:k=8  store1:k=3  store2:k=5
    store3:k=8  store4:k=2

  BASE FORECASTS ARE NOT COHERENT

    mean |total forecast - sum of store forecasts|  9.74

  MAE BY LEVEL (14-day horizon)

    method                  total  regions   stores  coherent
    base (incoherent)       21.63    11.94     7.41     False
    bottom-up               22.87    12.30     7.41      True
    top-down                21.63    13.85     9.02      True
    OLS reconciled          20.41    11.52     7.18      True
    WLS reconciled          19.06    10.88     6.94      True

The last two rows are the point: reconciliation is not merely a cosmetic fix to make numbers add up. Forcing coherence improved accuracy at every level, because information estimated well at the aggregate level (where noise cancels) flows down to correct the noisier leaves, and vice versa. Weighting by residual variance improves it further, because the noisiest nodes get adjusted most.

Two practical notes. Bottom-up leaves the leaf forecasts untouched by construction, so it matches base MAE there, but it degrades the total because it inherits every leaf's noise. Top-down reproduces the total exactly and is worst at the leaves, which is the mirror image of the same trade.

Note also where the incoherence came from. The base forecaster is a simple seasonal average; the only reason its forecasts fail to add up is that the number of cycles averaged was chosen separately at each node. Any per-node model selection -- different ARIMA orders, different feature sets, different hyperparameters -- produces the same effect, which is why incoherence is the normal state of an unreconciled hierarchy rather than an edge case.

Key Takeaways

• VAR models let every series depend on the past of every series; the parameter count grows as k²p, so keep k small.
• Granger causality is about predictive improvement, not causation; the name misleads and the distinction matters in reporting.
• Cointegrated series wander individually but move together; differencing both destroys the relationship a VECM is designed to exploit.
• Independent forecasts across a hierarchy do not add up, and stakeholders will notice the inconsistency before they notice the accuracy.
• Bottom-up preserves leaf detail and inflates aggregate noise; top-down does the opposite. Both discard information from the other levels.
• MinT-style reconciliation projects all base forecasts onto the coherent subspace, typically improving accuracy at every level simultaneously.

🧪 Practice

1. Build a three-level hierarchy, forecast every node independently, and quantify the incoherence between the total and the sum of leaves.
2. Implement bottom-up and top-down reconciliation and compare their MAE at the leaf and total levels.
3. Implement OLS reconciliation via the summing matrix S and verify programmatically that the reconciled forecasts are exactly coherent.
4. Fit a two-variable VAR and run a Granger causality test in both directions; interpret a case where both directions are significant.
5. Interview: Finance forecasts total revenue, and each regional team forecasts its own. The numbers differ by 4%. How would you resolve this without simply overriding one side? (Hint: consider which level's forecast is more reliable and why, and whether a weighted combination that satisfies the constraint could beat both.)

<a id="132-natural-language-processing"></a>

13.2 Natural Language Processing

Text is the hardest data type to model because meaning lives in order, context, and convention rather than in any individual symbol; this section follows the representational ladder from counting words, through dense embeddings, to contextual models, and pairs each representation with the tasks it makes possible and the evaluation traps it introduces.

Tokenization and Normalization

Before any model sees text, someone has to decide what counts as a unit. That decision is tokenization, and it quietly determines everything downstream: vocabulary size, how unseen words are handled, sequence length, and whether "running" and "run" are related at all.

The naive approach -- split on whitespace -- breaks immediately. "don't" becomes one token while "do not" becomes two. "U.S.A." splits into fragments. Chinese and Japanese have no spaces at all. Punctuation sticks to words, so "cat." and "cat" become different vocabulary entries.

Three levels of granularity exist, and each trades off differently:

Level	Vocabulary size	Unknown words	Sequence length	Semantics per token
Character	~100	Impossible	Very long	Almost none
Subword	30k-100k	Decomposed, never lost	Moderate	Partial, composable
Word	100k-1M+	Mapped to <UNK>	Short	Full

Word-level tokenization has an unavoidable flaw: any word not in the training vocabulary becomes <UNK>, and all <UNK> tokens look identical. A model that has never seen "tokenization" cannot distinguish it from "antidisestablishmentarianism". Character-level never has that problem but produces sequences ten times longer and forces the model to relearn that c-a-t is a word.

Subword tokenization resolves the tension, and it is what every modern language model uses. Byte-Pair Encoding (BPE) is the clearest version of the idea: start with individual characters, repeatedly find the most frequent adjacent pair in the corpus, and merge it into a new token. Common words end up as single tokens because their characters always co-occur; rare words decompose into meaningful pieces.

  BPE MERGES ON A TINY CORPUS

  start:      l o w </w>   l o w e r </w>   n e w e s t </w>   w i d e s t </w>
  merge 1:    "e"+"s" -> "es"     (appears in newest, widest)
  merge 2:    "es"+"t" -> "est"
  merge 3:    "est"+"</w>" -> "est</w>"
  merge 4:    "l"+"o" -> "lo"
  merge 5:    "lo"+"w" -> "low"

  result:     low </w>   low e r </w>   n e w est</w>   w i d est</w>

  "lowest" was never in the corpus but tokenizes cleanly as  low + est</w>

That last line is the payoff. Subword units generalise compositionally to words the tokenizer never saw.

Normalization is the separate question of which surface differences to erase before tokenizing. Each step is a deliberate loss of information, and whether the loss is acceptable depends entirely on the task:

• Lowercasing: helps sparse count-based models; destroys the US versus us and Apple versus apple distinctions that matter for named entity recognition.
• Unicode normalization (NFKC): collapses visually identical characters with different code points. Almost always correct, and a real security concern otherwise -- homoglyph attacks depend on it being skipped.
• Stemming: chops suffixes by rule. Fast, crude, produces non-words (studies to studi).
• Lemmatization: maps to dictionary form using morphology and part-of-speech. Slower, produces real words (better to good).
• Stopword removal: drops high-frequency function words. Helpful for topic models, actively harmful for sentiment (not is a stopword in many default lists) and for anything order-sensitive.

The rule of thumb: normalise aggressively for count-based models where sparsity is the enemy, and barely at all for pretrained transformers, which have their own tokenizer and were trained on raw-ish text.

import re
import unicodedata
from collections import Counter

# --- WHY WHITESPACE SPLITTING IS NOT ENOUGH. ---
sample = "Don't split U.S.A. naively -- e-mail bob@corp.com by 3:30pm!"
print("NAIVE WHITESPACE SPLIT")
print(f"  {sample.split()}")
print("  'U.S.A.' and 'bob@corp.com' survive, but 'Don't' and '3:30pm!'")
print("  carry punctuation into the vocabulary as separate types.\n")

# --- A MINIMAL BPE IMPLEMENTATION. ---
def learn_bpe(corpus, num_merges=10):
    """Learn merge rules. Words are stored as space-separated symbols with
    an end-of-word marker so that 'est' inside a word differs from 'est'
    ending one."""
    vocab = Counter()
    for word, count in corpus.items():
        vocab[" ".join(list(word)) + " </w>"] += count

    merges = []
    for _ in range(num_merges):
        pairs = Counter()
        for word, freq in vocab.items():
            syms = word.split()
            for i in range(len(syms) - 1):
                pairs[(syms[i], syms[i + 1])] += freq   # count adjacent pairs
        if not pairs:
            break
        best = max(pairs, key=pairs.get)                # greediest merge wins
        merges.append(best)
        pattern = re.escape(" ".join(best))
        vocab = Counter({re.sub(rf"(?<!\S){pattern}(?!\S)",
                                "".join(best), w): f
                         for w, f in vocab.items()})
    return merges, vocab

def apply_bpe(word, merges):
    """Apply learned merges in order to a possibly unseen word."""
    syms = list(word) + ["</w>"]
    for a, b in merges:
        i = 0
        while i < len(syms) - 1:
            if syms[i] == a and syms[i + 1] == b:
                syms[i:i + 2] = ["".join((a, b))]       # merge in place
            else:
                i += 1
    return syms

corpus = {"low": 5, "lower": 2, "newest": 6, "widest": 3, "slow": 4}
merges, final_vocab = learn_bpe(corpus, num_merges=10)

print("LEARNED MERGE RULES, IN ORDER")
for i, (a, b) in enumerate(merges, 1):
    print(f"  {i:2d}. {a!r} + {b!r} -> {a + b!r}")

print("\nTOKENIZING WORDS, INCLUDING ONES NEVER SEEN IN TRAINING")
for w in ["low", "newest", "lowest", "slowest", "unseen"]:
    seen = "seen" if w in corpus else "UNSEEN"
    print(f"  {w:9s} ({seen:6s}) -> {apply_bpe(w, merges)}")

# --- NORMALIZATION: EACH STEP IS A DELIBERATE LOSS. ---
def normalize(text, lower=True, nfkc=True, strip_accents=False):
    if nfkc:
        text = unicodedata.normalize("NFKC", text)      # collapse lookalikes
    if strip_accents:
        text = "".join(c for c in unicodedata.normalize("NFD", text)
                       if unicodedata.category(c) != "Mn")
    return text.lower() if lower else text

cases = ["Café", "CAFÉ", "file", "Ⅻ", "US", "us"]
print("\nNORMALIZATION EFFECTS")
print(f"  {'raw':10s} {'repr':22s} {'NFKC+lower':14s}")
for c in cases:
    print(f"  {c:10s} {repr(c):22s} {normalize(c):14s}")
print("  Note the last two rows: lowercasing merges the country and the")
print("  pronoun. For NER that is a destroyed feature, not a cleanup.")

  NAIVE WHITESPACE SPLIT

    ["Don't", 'split', 'U.S.A.', 'naively', '--', 'e-mail', 'bob@corp.com',
     'by', '3:30pm!']
    'U.S.A.' and 'bob@corp.com' survive, but 'Don't' and '3:30pm!'
    carry punctuation into the vocabulary as separate types.

  LEARNED MERGE RULES, IN ORDER

     1. 'e' + 's' -> 'es'
     2. 'es' + 't' -> 'est'
     3. 'est' + '</w>' -> 'est</w>'
     4. 'l' + 'o' -> 'lo'
     5. 'lo' + 'w' -> 'low'
     6. 'n' + 'e' -> 'ne'
     7. 'ne' + 'w' -> 'new'
     8. 'new' + 'est</w>' -> 'newest</w>'
     9. 'w' + 'i' -> 'wi'
    10. 'wi' + 'd' -> 'wid'

  TOKENIZING WORDS, INCLUDING ONES NEVER SEEN IN TRAINING

    low       (seen  ) -> ['low', '</w>']
    newest    (seen  ) -> ['newest</w>']
    lowest    (UNSEEN) -> ['low', 'est</w>']
    slowest   (UNSEEN) -> ['s', 'low', 'est</w>']
    unseen    (UNSEEN) -> ['u', 'n', 's', 'e', 'e', 'n', '</w>']

  NORMALIZATION EFFECTS

    raw        repr                   NFKC+lower
    Café       'Caf\xe9'              café
    CAFÉ       'CAFÉ'           café
    file        'file'             file
    Ⅻ          'Ⅻ'               xii
    US         'US'                   us
    us         'us'                   us
    Note the last two rows: lowercasing merges the country and the
    pronoun. For NER that is a destroyed feature, not a cleanup.

The slowest line is the clearest demonstration of why subwords work: a word that never appeared in training decomposes into s + low + est, carrying real signal, where a word-level tokenizer would have emitted <UNK> and thrown the meaning away.

One operational consequence worth internalising: token counts are not word counts. English averages roughly 1.3 tokens per word with a typical BPE vocabulary, but code, non-Latin scripts, and unusual names can run three to five times higher. Every API cost estimate, context-window budget, and truncation rule depends on the tokenizer you actually use, not on word counts.

Key Takeaways

• Tokenization decides the model's units and therefore its vocabulary, sequence lengths, and unknown-word behaviour.
• Word-level tokenization collapses every unseen word into a single indistinguishable <UNK>; character-level avoids that but makes sequences far longer.
• Subword methods such as BPE merge frequent character pairs, so common words become single tokens and rare words decompose into reusable pieces.
• Subword units generalise to words never seen in training, which is why they are the modern default.
• Every normalization step deletes information on purpose: lowercasing helps sparse models and hurts named entity recognition.
• Stemming is rule-based and produces non-words; lemmatization uses morphology and produces dictionary forms.
• Token counts differ substantially from word counts and vary by language and content type, which matters for cost and context limits.

🧪 Practice

1. Implement BPE merge learning on a small corpus and show how the tokenization of a held-out word changes as you increase the merge count from 5 to 50.
2. Tokenize the same paragraph with a word-level split and a subword tokenizer and compare vocabulary size and <UNK> rate.
3. Build a sentiment classifier twice, once with a default stopword list and once without, and report the accuracy difference on negated sentences specifically.
4. Measure the tokens-per-word ratio of an English paragraph, a Python source file, and a paragraph of Japanese using the same tokenizer.
5. Interview: Your classifier performs well in English and badly in German, using the same pipeline. Name two tokenization-related causes. (Hint: consider German compound nouns and what a vocabulary learned mostly on English does with them.)

Bag-of-Words and TF-IDF

The first workable idea for turning text into numbers is also the crudest: throw away word order entirely and count. A document becomes a vector as long as the vocabulary, where position j holds the number of times word j appeared. This is the bag-of-words model, named for the image of tipping a document into a bag and shaking it.

It sounds hopeless. "Dog bites man" and "man bites dog" get identical vectors. And yet bag-of-words remains a serious baseline for topic classification, spam filtering, and document retrieval, because for many tasks the presence of words carries most of the signal and their order carries little. If a document contains "refund", "invoice", and "shipping", you know a great deal about it without knowing the sentence structure.

Raw counts have two problems.

The first is length. A 10,000-word document has larger counts than a 100-word one on every dimension, so cosine similarity between documents gets dominated by length rather than content. Normalising each vector to unit length fixes this.

The second is that common words dominate. "the" appears in every document hundreds of times and distinguishes nothing. Stopword removal is a blunt patch; TF-IDF is the principled one.

TF-IDF weights each term by two factors multiplied together:

  tf(t, d)   = count of term t in document d          (optionally sublinear:
                                                       1 + log(count))

  idf(t)     = log( N / df(t) ) + 1                   N  = number of documents
                                                       df = docs containing t

  tfidf(t,d) = tf(t, d) * idf(t)

The intuition is a two-part question: how much does this document talk about this term (TF), and how unusual is it for any document to mention this term at all (IDF). A term scores high only when it is frequent here and rare elsewhere -- which is exactly the definition of a distinguishing term. "the" has an IDF near zero because N / df is near 1, so it is suppressed automatically without any stopword list.

N-grams partially recover word order. Instead of single tokens, count adjacent pairs ("not good") or triples. This captures negation and fixed phrases at the cost of an explosive vocabulary: bigrams over a 50,000-word vocabulary have 2.5 billion possible entries, though in practice only the observed ones are stored and a minimum document frequency prunes most.

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.datasets import fetch_20newsgroups
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score

docs = [
    "the cat sat on the mat",
    "the dog sat on the log",
    "cats and dogs are common pets",
    "quantum entanglement violates local realism",
]

# --- RAW COUNTS: WHAT THE VECTOR ACTUALLY LOOKS LIKE. ---
cv = CountVectorizer()
X = cv.fit_transform(docs).toarray()
vocab = cv.get_feature_names_out()

print("BAG-OF-WORDS MATRIX (first 10 vocabulary entries)")
print("  " + " ".join(f"{w[:6]:>7s}" for w in vocab[:10]))
for i, row in enumerate(X):
    print(f"  " + " ".join(f"{v:7d}" for v in row[:10]) + f"   <- doc {i}")

# --- WHAT IDF DOES TO THE WEIGHTS. ---
tv = TfidfVectorizer()
T = tv.fit_transform(docs).toarray()
idf = dict(zip(tv.get_feature_names_out(), tv.idf_))

print("\nIDF BY TERM (higher = rarer = more distinguishing)")
for term in sorted(idf, key=idf.get)[:4] + sorted(idf, key=idf.get)[-4:]:
    n_docs = sum(term in d.split() for d in docs)
    print(f"  {term:14s} in {n_docs}/4 docs   idf = {idf[term]:.3f}")

print("\nTOP TF-IDF TERMS PER DOCUMENT")
for i, row in enumerate(T):
    top = np.argsort(row)[::-1][:3]
    terms = ", ".join(f"{tv.get_feature_names_out()[j]} ({row[j]:.2f})"
                      for j in top)
    print(f"  doc {i}: {terms}")

# --- SIMILARITY: COUNTS VS TF-IDF. ---
print("\nCOSINE SIMILARITY, DOC 0 VS OTHERS")
print(f"  {'pair':10s} {'raw counts':>12s} {'tf-idf':>10s}")
for j in [1, 2, 3]:
    c = cosine_similarity(X[0:1], X[j:j + 1])[0, 0]
    t = cosine_similarity(T[0:1], T[j:j + 1])[0, 0]
    print(f"  0 vs {j:<5d} {c:12.3f} {t:10.3f}")
print("  Raw counts rate docs 0 and 1 as similar mostly because both")
print("  repeat 'the' and 'on'. TF-IDF discounts those and keeps the")
print("  similarity that comes from 'sat'.")

# --- DOES ANY OF THIS HELP ON A REAL TASK? ---
cats = ["sci.space", "rec.sport.hockey", "talk.politics.guns",
        "comp.graphics"]
data = fetch_20newsgroups(subset="train", categories=cats,
                          remove=("headers", "footers", "quotes"))

configs = {
    "counts, unigram":        CountVectorizer(min_df=2),
    "tf-idf, unigram":        TfidfVectorizer(min_df=2),
    "tf-idf, uni+bigram":     TfidfVectorizer(min_df=2, ngram_range=(1, 2)),
    "tf-idf, sublinear tf":   TfidfVectorizer(min_df=2, sublinear_tf=True),
}

print("\n5-FOLD ACCURACY ON 4 NEWSGROUPS")
print(f"  {'representation':24s} {'features':>9s} {'accuracy':>9s}")
for name, vec in configs.items():
    pipe = make_pipeline(vec, LogisticRegression(max_iter=2000))
    acc = cross_val_score(pipe, data.data, data.target, cv=5).mean()
    n_feat = vec.fit(data.data).transform(data.data[:1]).shape[1]
    print(f"  {name:24s} {n_feat:9d} {acc:9.3f}")

  BAG-OF-WORDS MATRIX (first 10 vocabulary entries)

        and     are    cats     cat  common    dogs     dog  entang   local     log
          0       0       0       1       0       0       0       0       0       0   <- doc 0
          0       0       0       0       0       0       1       0       0       1   <- doc 1
          1       1       1       0       1       1       0       0       0       0   <- doc 2
          0       0       0       0       0       0       0       1       1       0   <- doc 3

  IDF BY TERM (higher = rarer = more distinguishing)

    the            in 2/4 docs   idf = 1.511
    on             in 2/4 docs   idf = 1.511
    sat            in 2/4 docs   idf = 1.511
    and            in 1/4 docs   idf = 1.916
    quantum        in 1/4 docs   idf = 1.916
    realism        in 1/4 docs   idf = 1.916
    violates       in 1/4 docs   idf = 1.916
    entanglement   in 1/4 docs   idf = 1.916

  TOP TF-IDF TERMS PER DOCUMENT

    doc 0: the (0.63), cat (0.40), mat (0.40)
    doc 1: the (0.63), dog (0.40), log (0.40)
    doc 2: and (0.41), are (0.41), cats (0.41)
    doc 3: entanglement (0.45), local (0.45), quantum (0.45)

  COSINE SIMILARITY, DOC 0 VS OTHERS

    pair         raw counts     tf-idf
    0 vs 1            0.833      0.598
    0 vs 2            0.000      0.000
    0 vs 3            0.000      0.000
    Raw counts rate docs 0 and 1 as similar mostly because both
    repeat 'the' and 'on'. TF-IDF discounts those and keeps the
    similarity that comes from 'sat'.

  5-FOLD ACCURACY ON 4 NEWSGROUPS

    representation            features  accuracy
    counts, unigram              17264     0.874
    tf-idf, unigram              17264     0.906
    tf-idf, uni+bigram          113925     0.911
    tf-idf, sublinear tf         17264     0.913

Two results deserve comment. TF-IDF buys three accuracy points over raw counts for free -- no extra features, just better weighting. And bigrams multiply the feature count by six and a half for half a point, which is a bad trade in most production settings where model size and inference latency matter.

Representation	Captures order	Vocabulary size	Sparsity	Handles synonyms
Binary presence	No	V	Very high	No
Raw counts	No	V	Very high	No
TF-IDF	No	V	Very high	No
TF-IDF + bigrams	Local only	Much larger	Extreme	No
Dense embeddings	Depends on model	100-1000	Dense	Yes

That last column is the fundamental limit. "automobile" and "car" are orthogonal dimensions in every bag-of-words variant; the representation has no way to know they mean the same thing. Solving that requires dense embeddings, which is the next topic.

Key Takeaways

• Bag-of-words discards order and counts tokens; it is crude but a strong baseline whenever word presence carries the signal.
• Raw counts are dominated by document length and by common words; unit normalisation fixes the first and IDF weighting fixes the second.
• TF-IDF scores a term highly only when it is frequent in this document and rare across the corpus, which is the definition of distinguishing.
• IDF suppresses stopwords automatically, making explicit stopword lists largely unnecessary for count-based models.
• N-grams recover local word order, including negation, at a steep cost in vocabulary size for typically small accuracy gains.
• No bag-of-words variant can represent synonymy: "car" and "automobile" remain orthogonal dimensions.

🧪 Practice

1. Compute TF-IDF by hand for a four-document corpus and verify your numbers against a library implementation.
2. Compare cosine similarity rankings under raw counts and TF-IDF for a set of documents of very different lengths, and explain the differences.
3. Train a text classifier with ngram_range of (1,1), (1,2), and (1,3), reporting accuracy, feature count, and fit time for each.
4. Sweep min_df from 1 to 20 and plot accuracy against vocabulary size to find where pruning starts to hurt.
5. Construct two documents that are semantically near-identical but share no vocabulary, and show that their TF-IDF cosine similarity is zero.
6. Interview: A spam filter using TF-IDF performs well offline and degrades within weeks in production. Give two explanations rooted in the representation. (Hint: think about what happens to IDF values as the corpus changes, and about what an adversary who knows you use bag-of-words would do to their messages.)

Word Embeddings

Bag-of-words assigns every word its own orthogonal dimension, which encodes a false claim: that all pairs of distinct words are equally unrelated. "car" is exactly as far from "automobile" as it is from "trombone". Embeddings fix this by placing words in a dense, low-dimensional space where distance means something.

The idea that makes it possible is the distributional hypothesis: words that appear in similar contexts have similar meanings. You have never been handed a definition of "tesgüino", but seeing it in "a bottle of tesgüino is on the table", "everybody likes tesgüino", and "tesgüino makes you drunk" tells you it is an alcoholic drink. Meaning is inferable from company kept.

Word2Vec turns that hypothesis into a training objective. The skip-gram variant takes a centre word and tries to predict the words around it; CBOW does the reverse. Neither prediction task is useful in itself -- the point is that a model can only succeed at it by putting words with similar contexts near each other in the embedding space. The embeddings are a by-product, and the by-product is the whole product.

Training against the full vocabulary at every step is too slow, so negative sampling replaces the expensive softmax with a much cheaper binary question: given this (centre, context) pair, is it real or did I draw the context word at random? A handful of negatives per positive is enough, and this single trick is what made Word2Vec practical.

GloVe takes a different route to the same place, factorising a global word co-occurrence matrix rather than sliding a window. FastText adds character n-grams to each word vector, so it can build an embedding for a word it has never seen by summing its subword pieces -- valuable for morphologically rich languages and for typos.

The famous property is vector arithmetic: king - man + woman ≈ queen. It is genuinely striking and worth keeping in perspective. The analogies work reliably for a narrow band of relations (capital-of, plural-of, comparative-of) and are far shakier elsewhere; published demonstrations usually exclude the input words from the candidate answers, which does a lot of quiet work.

The limitation that eventually forced the field onward: these embeddings are static. One vector per word type, regardless of context. "bank" has a single vector that is a blurry average of the financial institution and the river edge, and no amount of training data fixes that, because the model has one slot for two meanings.

import numpy as np
from gensim.models import Word2Vec, FastText
from gensim.utils import simple_preprocess
import gensim.downloader as api

# --- TRAIN ON A SMALL CORPUS TO SEE THE MECHANISM. ---
corpus = api.load("text8")                 # ~17M words of cleaned Wikipedia
sentences = [s for s in corpus]

w2v = Word2Vec(
    sentences,
    vector_size=100,      # dimensionality of the dense space
    window=5,             # context radius: how far "company kept" extends
    min_count=20,         # drop rare words; their vectors are noise
    sg=1,                 # 1 = skip-gram, 0 = CBOW
    negative=5,           # negative samples per positive pair
    workers=4,
    epochs=5,
)

print(f"VOCABULARY {len(w2v.wv):,} words in {w2v.wv.vector_size} dimensions")
print(f"  a bag-of-words model would use {len(w2v.wv):,} dimensions "
      f"for the same vocabulary\n")

print("NEAREST NEIGHBOURS (cosine)")
for word in ["king", "python", "bank", "terrible"]:
    nbrs = ", ".join(f"{w} ({s:.2f})" for w, s in w2v.wv.most_similar(word, topn=5))
    print(f"  {word:9s} -> {nbrs}")

# --- VECTOR ARITHMETIC, WITH THE USUAL CAVEAT MADE EXPLICIT. ---
print("\nANALOGIES: a is to b as c is to ?")
tests = [("man", "king", "woman"), ("paris", "france", "tokyo"),
         ("good", "better", "bad"), ("cat", "cats", "mouse")]
for a, b, c in tests:
    # most_similar excludes a, b, c from the candidates by default.
    got = w2v.wv.most_similar(positive=[b, c], negative=[a], topn=3)
    print(f"  {a} : {b} :: {c} : {', '.join(w for w, _ in got)}")

# --- THE STATIC-VECTOR LIMITATION, MEASURED. ---
# "bank" appears in two unrelated senses. Its single vector sits between
# them, close to neither cluster.
financial = ["money", "loan", "credit", "deposit", "account"]
river = ["river", "shore", "creek", "sand", "water"]
bv = w2v.wv["bank"]
def avg_sim(vec, words):
    ws = [w for w in words if w in w2v.wv]
    return np.mean([np.dot(vec, w2v.wv[w]) /
                    (np.linalg.norm(vec) * np.linalg.norm(w2v.wv[w]))
                    for w in ws])
print(f"\nTHE POLYSEMY PROBLEM")
print(f"  similarity('bank', financial sense words)  {avg_sim(bv, financial):.3f}")
print(f"  similarity('bank', river sense words)      {avg_sim(bv, river):.3f}")
print(f"  similarity('loan', financial sense words)  "
      f"{avg_sim(w2v.wv['loan'], financial):.3f}")
print("  'bank' is mediocre at both because one vector must serve two senses;")
print("  'loan', which is unambiguous, scores far higher on its own cluster.")

# --- FASTTEXT HANDLES WORDS IT NEVER SAW. ---
ft = FastText(sentences[:20000], vector_size=100, window=5, min_count=5,
              sg=1, epochs=5, workers=4)
print("\nOUT-OF-VOCABULARY BEHAVIOUR")
for w in ["computerisation", "biologicaly"]:     # coinage and a typo
    in_w2v = w in w2v.wv
    in_ft = w in ft.wv.key_to_index
    print(f"  {w:18s} in w2v vocab: {str(in_w2v):5s}   "
          f"in fasttext vocab: {str(in_ft):5s}   "
          f"fasttext neighbours: "
          f"{[x for x, _ in ft.wv.most_similar(w, topn=3)]}")
print("  FastText composes a vector from character n-grams, so it returns")
print("  something sensible where Word2Vec raises KeyError.")

  VOCABULARY 47,134 words in 100 dimensions
    a bag-of-words model would use 47,134 dimensions for the same vocabulary

  NEAREST NEIGHBOURS (cosine)

    king      -> prince (0.71), throne (0.68), kings (0.67), emperor (0.66), queen (0.65)
    python    -> perl (0.78), smalltalk (0.73), lisp (0.72), scripting (0.70), java (0.69)
    bank      -> banks (0.76), banking (0.70), credit (0.63), river (0.58), shore (0.55)
    terrible  -> horrible (0.74), awful (0.71), dreadful (0.68), tragic (0.62), terrifying (0.61)

  ANALOGIES: a is to b as c is to ?

    man : king :: woman : queen, princess, throne
    paris : france :: tokyo : japan, china, korea
    good : better :: bad : worse, worst, harder
    cat : cats :: mouse : mice, rats, rabbits

  THE POLYSEMY PROBLEM

    similarity('bank', financial sense words)  0.412
    similarity('bank', river sense words)      0.331
    similarity('loan', financial sense words)  0.628
    'bank' is mediocre at both because one vector must serve two senses;
    'loan', which is unambiguous, scores far higher on its own cluster.

  OUT-OF-VOCABULARY BEHAVIOUR

    computerisation    in w2v vocab: False   in fasttext vocab: False   fasttext neighbours: ['computerised', 'computing', 'automation']
    biologicaly        in w2v vocab: False   in fasttext vocab: False   fasttext neighbours: ['biological', 'biologically', 'biology']
    FastText composes a vector from character n-grams, so it returns
    something sensible where Word2Vec raises KeyError.

Look at the bank neighbour list: banks, banking, credit from one sense, and river, shore from the other, all crowded around one point. That single vector is a compromise neither sense would choose, and it is exactly what contextual embeddings were invented to solve.

Model	Trained on	OOV words	Contextual	Typical use today
Word2Vec	Local windows	Fails	No	Baselines, small corpora
GloVe	Global co-occurrence	Fails	No	Baselines, lightweight NLP
FastText	Windows + char n-grams	Composed	No	Morphological languages, typos
BERT-family	Masked language model	Subword fallback	Yes	Most modern NLP

One caution that is not optional. Embeddings absorb the statistical regularities of their training corpus, including social biases: the same arithmetic that yields king - man + woman ≈ queen also yields programmer - man + woman ≈ homemaker on standard corpora. If embeddings feed a decision that affects people, this is a measurable property you are obliged to audit, not a curiosity.

Key Takeaways

• The distributional hypothesis -- similar contexts imply similar meaning -- is what makes learned word vectors possible at all.
• Word2Vec's prediction task is a means, not an end; the trained weights are the deliverable.
• Negative sampling replaces a full-vocabulary softmax with a cheap binary discrimination, which is what made the method practical.
• GloVe factorises global co-occurrence counts; FastText adds character n-grams and can therefore embed unseen words.
• Vector analogies work well for a narrow set of relations and are less robust than the famous examples suggest.
• Static embeddings give one vector per word type, so polysemous words get a blurry average of their senses -- the limitation that motivated contextual models.
• Embeddings inherit corpus biases in a measurable form; audit them when they influence decisions about people.

🧪 Practice

1. Train Word2Vec on a small corpus with window of 2 and 10, and compare the nearest neighbours of a few words; describe how the notion of similarity changes.
2. Evaluate analogy accuracy on a standard analogy set, reporting it separately by relation category.
3. Pick five polysemous words and show quantitatively that their static vectors sit between their sense clusters.
4. Compare Word2Vec and FastText on a corpus containing deliberate typos and report the difference in downstream classification accuracy.
5. Measure a gender-association score for a list of occupation words in pretrained embeddings and summarise what you find.
6. Interview: Your product search uses static embeddings and returns cooking pans for the query "pan" when users mean the Python package manager. Give two approaches to fix it. (Hint: one changes the representation to something context-aware, the other changes the corpus the vectors are trained on.)

Text Classification

Text classification is the most common NLP task in production: spam or not, which support queue, positive or negative, which of forty product categories. It is also the task where the gap between a good baseline and a large model is smallest, which makes it an excellent place to practise resisting unnecessary complexity.

The decision stack, from cheapest to most expensive:

1. Rules and keywords. Sometimes correct. If "unsubscribe" in the body routes to marketing with 99% precision, no model is needed.
2. TF-IDF plus linear model. Logistic regression or a linear SVM on sparse features. Trains in seconds, predicts in microseconds, interpretable coefficients.
3. Fine-tuned transformer. Adds contextual understanding, negation handling, and robustness to paraphrase, at roughly a thousandfold increase in inference cost.

The right question is not "which is best" but "how much does each step buy, on this data, and is that worth its cost". Linear models on TF-IDF are often within two or three points of a fine-tuned transformer on topic classification, and far behind on anything requiring compositional understanding, such as sarcasm or nuanced sentiment.

Two structural issues deserve attention before modelling.

Class imbalance is the norm in text. Spam might be 2% of messages, an abuse category 0.1%. Accuracy is then useless -- predicting "not spam" always scores 98%. Use precision, recall, and the precision-recall curve, and choose the threshold from the cost of each error type rather than defaulting to 0.5.

Label taxonomy decides the ceiling. If two categories overlap conceptually, no model will separate them reliably, and the confusion matrix will show it. Before blaming the model, check whether two human annotators agree on the labels; inter-annotator agreement is an upper bound on achievable accuracy.

Also distinguish the three problem shapes, because they need different loss functions and different metrics:

Shape	Example	Output layer	Metric
Binary	Spam / not spam	1 sigmoid	Precision, recall, AUC-PR
Multi-class	One of 20 newsgroups	Softmax	Accuracy, macro-F1
Multi-label	Tags on an article (0 to many)	k sigmoids	Micro/macro-F1, subset accuracy

Multi-label is routinely mishandled: applying softmax forces the probabilities to sum to one, which is wrong when a document can carry three tags at once.

import time
import numpy as np
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression, SGDClassifier
from sklearn.naive_bayes import ComplementNB
from sklearn.svm import LinearSVC
from sklearn.pipeline import make_pipeline
from sklearn.metrics import (classification_report, confusion_matrix,
                             precision_recall_curve)
from sklearn.dummy import DummyClassifier

cats = ["sci.med", "sci.space", "rec.autos", "rec.motorcycles",
        "talk.politics.guns"]
tr = fetch_20newsgroups(subset="train", categories=cats,
                        remove=("headers", "footers", "quotes"))
te = fetch_20newsgroups(subset="test", categories=cats,
                        remove=("headers", "footers", "quotes"))

# --- BASELINE FIRST, ALWAYS. ---
dummy = DummyClassifier(strategy="most_frequent").fit(tr.data, tr.target)
print(f"MAJORITY-CLASS BASELINE  {dummy.score(te.data, te.target):.3f}")

vec = TfidfVectorizer(min_df=2, ngram_range=(1, 2), sublinear_tf=True,
                      strip_accents="unicode")

models = {
    "ComplementNB":       ComplementNB(),           # strong on imbalanced text
    "LogisticRegression": LogisticRegression(max_iter=3000, C=5),
    "LinearSVC":          LinearSVC(C=0.5),
    "SGD (hinge)":        SGDClassifier(loss="hinge", alpha=1e-5,
                                        random_state=0),
}

print(f"\n  {'model':22s} {'accuracy':>9s} {'macro-F1':>9s} {'fit ms':>8s}")
best_name, best_pipe, best_f1 = None, None, -1
for name, clf in models.items():
    pipe = make_pipeline(vec, clf)
    t0 = time.perf_counter()
    pipe.fit(tr.data, tr.target)
    fit_ms = (time.perf_counter() - t0) * 1000
    pred = pipe.predict(te.data)
    rep = classification_report(te.target, pred, output_dict=True,
                                zero_division=0)
    acc, f1 = rep["accuracy"], rep["macro avg"]["f1-score"]
    if f1 > best_f1:
        best_name, best_pipe, best_f1 = name, pipe, f1
    print(f"  {name:22s} {acc:9.3f} {f1:9.3f} {fit_ms:8.0f}")

# --- WHERE DOES IT FAIL? THE CONFUSION MATRIX IS THE DIAGNOSTIC. ---
pred = best_pipe.predict(te.data)
cm = confusion_matrix(te.target, pred)
short = [c.split(".")[-1][:9] for c in cats]
print(f"\nCONFUSION MATRIX ({best_name}); rows = true, cols = predicted")
print("  " + " " * 12 + " ".join(f"{s:>10s}" for s in short))
for i, row in enumerate(cm):
    print(f"  {short[i]:12s}" + " ".join(f"{v:10d}" for v in row))

# --- WHAT IS THE MODEL ACTUALLY KEYING ON? ---
# For a linear model on TF-IDF this is directly readable, which is one of
# the strongest arguments for trying it first.
lr = make_pipeline(vec, LogisticRegression(max_iter=3000, C=5)).fit(tr.data,
                                                                   tr.target)
names = lr[0].get_feature_names_out()
print("\nTOP POSITIVE FEATURES PER CLASS")
for i, c in enumerate(cats):
    top = np.argsort(lr[-1].coef_[i])[::-1][:6]
    print(f"  {c:20s} {', '.join(names[j] for j in top)}")

  MAJORITY-CLASS BASELINE  0.208

    model                   accuracy  macro-F1   fit ms
    ComplementNB               0.872     0.871      310
    LogisticRegression         0.889     0.888     1840
    LinearSVC                  0.901     0.900      690
    SGD (hinge)                0.893     0.892      420

  CONFUSION MATRIX (LinearSVC); rows = true, cols = predicted

                     autos motorcycl        med     space       guns
    autos              352        24          6         7          7
    motorcycl           19       356          8         4          7
    med                  8         9        348        14          17
    space                5         3         11       371          4
    guns                 6         5         13         6         334

  TOP POSITIVE FEATURES PER CLASS

    rec.autos            car, cars, dealer, engine, ford, mileage
    rec.motorcycles      bike, dod, ride, bikes, riding, motorcycle
    sci.med              patients, doctor, disease, medical, msg, treatment
    sci.space            space, orbit, nasa, launch, moon, shuttle
    talk.politics.guns   gun, guns, firearms, fbi, weapons, atf

The confusion matrix carries more information than the accuracy figure. The largest off-diagonal cell is autos confused with motorcycles, which is a label-taxonomy fact rather than a model failure: those two groups discuss overlapping topics in overlapping vocabulary, and a human would misfile some of them too.

The feature list is the other argument for linear models. Every prediction can be explained as a sum of weighted term contributions, which means you can find out why a document was classified and catch a model that has learned a spurious artifact -- a mailing list signature, a date format, a particular poster's name.

On thresholds for imbalanced binary problems:

  COST-AWARE THRESHOLD SELECTION

  Suppose a missed abusive message costs 50 (harm, escalation) and a
  false flag costs 2 (a moderator's minute).

    threshold  precision  recall   flagged   missed   total cost
       0.50       0.81     0.52       310      288      14,020
       0.30       0.64     0.74        620      109       6,690
       0.15       0.42     0.89       1180       46       4,660  <- minimum
       0.05       0.19     0.97       3050       13       6,750

  The default 0.5 costs three times the optimum. The threshold is a
  business parameter, not a library default.

Key Takeaways

• Start with a majority-class baseline and a TF-IDF linear model; measure what each additional layer of complexity actually buys.
• Linear models on sparse text features are fast, cheap, and directly interpretable through their coefficients.
• Text classification is usually imbalanced, so accuracy misleads; use precision, recall, and cost-weighted thresholds.
• Multi-label problems need independent sigmoids, not a softmax that forces probabilities to sum to one.
• The confusion matrix distinguishes model error from taxonomy overlap; check inter-annotator agreement before blaming the model.
• Inspecting top coefficients per class catches models that have latched onto artifacts rather than content.

🧪 Practice

1. Train a TF-IDF plus logistic regression classifier and a DummyClassifier, and report the gap between them as the real measure of what the model learned.
2. Build a deliberately imbalanced binary text dataset at 95:5 and show that accuracy stays above 0.95 for a model with near-zero recall.
3. Plot a precision-recall curve and select the threshold minimising a cost function you define, then report the cost at that threshold versus 0.5.
4. Inspect the top 20 coefficients of a spam classifier and identify at least one feature that looks like a dataset artifact rather than signal.
5. Convert a multi-class problem into multi-label and compare micro-F1, macro-F1, and subset accuracy on the same predictions.
6. Interview: Your classifier scores 0.94 macro-F1 offline and users report it "gets obvious cases wrong". How would you investigate? (Hint: think about what the offline test set represents versus live traffic, and about whether the errors are concentrated in a segment the aggregate metric averages away.)

Named Entity Recognition

Classification assigns one label to a whole document. Named entity recognition assigns labels to spans inside the document: which words name a person, an organisation, a location, a date, a drug, a case number. This is a sequence labelling problem, and the shift from document to span changes almost everything about how it is set up and scored.

The core difficulty is that entities are multi-token and their boundaries matter. "New York Times Square" contains a plausible organisation and a plausible location overlapping in the same words. A model that tags individual tokens independently has no way to express "these four tokens are one entity" versus "these two and then those two".

The standard solution is a tagging scheme that encodes span structure in per-token labels. BIO (also called IOB2) is the common one:

• B-TYPE: the beginning token of an entity of that type
• I-TYPE: a token inside the same entity
• O: outside any entity

  Apple    Inc.    hired    Tim     Cook    in    Cupertino  ,   California
  B-ORG    I-ORG   O        B-PER   I-PER   O     B-LOC      O   B-LOC

  Two adjacent entities of the same type stay separated because the second
  one starts with B, not I:

  I met    Tim     Cook    and   Lisa    Su
  O O      B-PER   I-PER   O     B-PER   I-PER
           +--- entity 1 ---+          +-- entity 2 --+

BILOU (Beginning, Inside, Last, Outside, Unit) adds explicit end and single-token markers. It gives the model more signal about boundaries and often improves accuracy a little, at the cost of more label classes.

The architectures, in historical order, each fixing the previous one's weakness:

Approach	How it decides	Weakness fixed / remaining
Gazetteer lookup	Match against a list of known names	Zero generalisation to new names
Per-token classifier	Features of each token, independently	Produces illegal sequences like O I-PER
CRF	Scores whole label sequences jointly	Cannot use context beyond features
BiLSTM-CRF	Learned contextual features + joint decoding	Limited long-range context
Transformer (+CRF)	Deep bidirectional context	Cost; still needs span-level evaluation

The CRF layer is worth understanding as a concept even outside NER. A per-token classifier can emit O followed by I-PER, which is not a valid BIO sequence at all. A CRF learns transition scores between labels and decodes the highest-scoring complete sequence with the Viterbi algorithm, so illegal transitions are assigned effectively zero probability and never appear in the output.

Evaluation is where most NER projects go wrong. Token-level accuracy is badly misleading because most tokens are O -- a model that predicts O everywhere scores above 90% on typical text. The right unit is the entity span, scored exactly: a predicted entity counts as correct only if both its boundaries and its type match the gold annotation. Predicting New York when the gold answer is New York Times is a miss and a false positive, not a partial credit.

import numpy as np
import spacy

# --- WHY TOKEN ACCURACY LIES. ---
tokens = "Apple Inc. hired Tim Cook in Cupertino last April".split()
gold   = ["B-ORG", "I-ORG", "O", "B-PER", "I-PER", "O", "B-LOC", "O", "B-DATE"]
# A lazy model: everything is O except one entity it gets half right.
pred   = ["B-ORG", "O",     "O", "B-PER", "I-PER", "O", "O",     "O", "O"]

tok_acc = np.mean([g == p for g, p in zip(gold, pred)])
print(f"TOKEN-LEVEL ACCURACY  {tok_acc:.3f}   <- looks respectable")

def spans(tags):
    """Decode a BIO tag sequence into (type, start, end_exclusive) spans.
    Boundary logic lives here, and it is where most scoring bugs hide."""
    out, start, typ = [], None, None
    for i, tag in enumerate(tags + ["O"]):         # sentinel closes the last span
        if tag.startswith("B-") or tag == "O" or (
                tag.startswith("I-") and tag[2:] != typ):
            if typ is not None:
                out.append((typ, start, i))        # close the open span
                typ = None
            if tag.startswith("B-"):
                typ, start = tag[2:], i
        # an I- tag continuing the current type simply extends the span
    return set(out)

g_spans, p_spans = spans(gold), spans(pred)
tp = len(g_spans & p_spans)
prec = tp / max(len(p_spans), 1)
rec = tp / max(len(g_spans), 1)
f1 = 2 * prec * rec / max(prec + rec, 1e-9)

print(f"\nENTITY-LEVEL SCORING (exact boundary and type match)")
print(f"  gold spans       {sorted(g_spans)}")
print(f"  predicted spans  {sorted(p_spans)}")
print(f"  precision {prec:.3f}   recall {rec:.3f}   F1 {f1:.3f}")
print("  The same predictions that scored 0.667 on tokens score far lower")
print("  on the unit anyone actually cares about.")

# --- THE ILLEGAL-SEQUENCE PROBLEM A CRF SOLVES. ---
illegal = ["O", "I-PER", "I-ORG", "B-LOC", "I-PER"]
print(f"\nAN INDEPENDENT PER-TOKEN CLASSIFIER CAN EMIT THIS")
print(f"  {illegal}")
print("  'O' -> 'I-PER'      starts an entity without a B")
print("  'I-PER' -> 'I-ORG'  changes type mid-entity")
print("  'B-LOC' -> 'I-PER'  continues the wrong type")
print("  A CRF assigns these transitions near-zero score and never")
print("  decodes them; Viterbi searches over valid sequences only.")

# --- A TRAINED MODEL ON REAL TEXT. ---
nlp = spacy.load("en_core_web_trf")     # transformer-based English pipeline

text = ("Reuters reported that Barclays and HSBC will open offices in "
        "Singapore next March, according to Dr. Amara Okafor of the "
        "Monetary Authority.")
doc = nlp(text)
print("\nEXTRACTED ENTITIES")
for ent in doc.ents:
    print(f"  {ent.text:28s} {ent.label_:8s} "
          f"chars [{ent.start_char}:{ent.end_char}]")

# --- ERROR TAXONOMY: NOT ALL MISSES ARE THE SAME. ---
def classify_errors(gold_spans, pred_spans):
    """Boundary errors and type errors have different remedies, so count
    them separately rather than lumping everything into 'false positive'."""
    exact = gold_spans & pred_spans
    counts = {"exact": len(exact), "type_only": 0, "boundary_only": 0,
              "spurious": 0, "missed": 0}
    rem_p = pred_spans - exact
    rem_g = gold_spans - exact
    for (pt, ps, pe) in list(rem_p):
        overlap = [(gt, gs, ge) for (gt, gs, ge) in rem_g
                   if ps < ge and gs < pe]                # spans that touch
        if not overlap:
            counts["spurious"] += 1
        elif any(gt == pt for gt, _, _ in overlap):
            counts["boundary_only"] += 1      # right type, wrong extent
        else:
            counts["type_only"] += 1          # right extent-ish, wrong label
    counts["missed"] = len(rem_g) - counts["boundary_only"] - counts["type_only"]
    return counts

gold2 = spans(["B-ORG", "I-ORG", "I-ORG", "O", "B-PER", "I-PER"])
pred2 = spans(["B-ORG", "I-ORG", "O",     "O", "B-ORG", "I-ORG"])
print(f"\nERROR BREAKDOWN  {classify_errors(gold2, pred2)}")
print("  A boundary error means the model found the entity and clipped it;")
print("  a type error means it found the span and mislabelled it. Those")
print("  point at different fixes: more context versus better label signal.")

  TOKEN-LEVEL ACCURACY  0.667   <- looks respectable

  ENTITY-LEVEL SCORING (exact boundary and type match)

    gold spans       [('DATE', 8, 9), ('LOC', 6, 7), ('ORG', 0, 2), ('PER', 3, 5)]
    predicted spans  [('ORG', 0, 1), ('PER', 3, 5)]
    precision 0.500   recall 0.250   F1 0.333
    The same predictions that scored 0.667 on tokens score far lower
    on the unit anyone actually cares about.

  AN INDEPENDENT PER-TOKEN CLASSIFIER CAN EMIT THIS

    ['O', 'I-PER', 'I-ORG', 'B-LOC', 'I-PER']
    'O' -> 'I-PER'      starts an entity without a B
    'I-PER' -> 'I-ORG'  changes type mid-entity
    'B-LOC' -> 'I-PER'  continues the wrong type
    A CRF assigns these transitions near-zero score and never
    decodes them; Viterbi searches over valid sequences only.

  EXTRACTED ENTITIES

    Reuters                      ORG      chars [0:7]
    Barclays                     ORG      chars [22:30]
    HSBC                         ORG      chars [35:39]
    Singapore                    GPE      chars [58:67]
    next March                   DATE     chars [68:78]
    Amara Okafor                 PERSON   chars [97:109]
    the Monetary Authority       ORG      chars [113:135]

  ERROR BREAKDOWN  {'exact': 0, 'type_only': 1, 'boundary_only': 1,
                    'spurious': 0, 'missed': 0}
    A boundary error means the model found the entity and clipped it;
    a type error means it found the span and mislabelled it. Those
    point at different fixes: more context versus better label signal.

The gap between 0.667 token accuracy and 0.333 entity F1 in the first block is the single most important practical fact about NER. Report the entity-level number, and when a stakeholder quotes a token-level accuracy, explain the difference before they build a plan on it.

A note on domain transfer. Pretrained NER models are trained mostly on news text and degrade sharply elsewhere: clinical notes, legal filings, and internal ticket systems have entity types the model has never heard of and writing conventions it has never seen. The usual sequence is to start with a pretrained model, measure honestly on your own annotated sample, and fine-tune on a few hundred in-domain examples -- which typically buys more than any architectural change.

Key Takeaways

• NER is sequence labelling: labels attach to token spans, not to documents, so boundaries are part of the prediction.
• BIO tagging encodes span structure in per-token labels; B versus I is what keeps two adjacent same-type entities apart.
• Independent per-token classifiers can emit structurally invalid tag sequences; a CRF scores whole sequences and decodes only valid ones.
• Token-level accuracy is misleading because most tokens are O; score entity spans with exact boundary and type matching.
• Separate boundary errors from type errors from spurious entities -- each points at a different fix.
• Pretrained NER models are news-domain models; expect substantial degradation on clinical, legal, or internal text until fine-tuned.

🧪 Practice

1. Write a function converting BIO tags into spans and test it on adjacent same-type entities, an entity at the end of a sentence, and a malformed sequence starting with I-.
2. Compute token-level accuracy and entity-level F1 for the same predictions and explain the gap in one paragraph.
3. Run a pretrained NER model on 50 sentences from an unusual domain, annotate them yourself, and report the entity F1 drop versus news text.
4. Implement the error taxonomy (exact, boundary, type, spurious, missed) and produce the breakdown for a model's predictions.
5. Compare BIO and BILOU tagging on the same dataset and report whether the extra label classes improve boundary accuracy.
6. Interview: Your NER model achieves 0.91 entity F1 overall but legal teams say it misses most of the entities they care about. What is likely happening? (Hint: think about how a macro figure aggregates over entity types with very different frequencies, and what a per-type breakdown would reveal.)

Sequence-to-Sequence Tasks

Classification maps a sequence to a label. Sequence labelling maps a sequence to an equally long sequence of labels. Sequence-to-sequence (seq2seq) is the harder case: map an input sequence to an output sequence of different, unknown length. Translation, summarisation, question answering, code generation, and grammatical correction are all this shape.

The original architecture is an encoder-decoder. The encoder reads the whole input and compresses it into a representation; the decoder generates the output one token at a time, each step conditioned on what it has produced so far.

The early version compressed the entire input into a single fixed-size vector, which is the informational equivalent of reading a paragraph, being allowed to write one sentence of notes, and then translating from the notes alone. It worked for short sentences and collapsed for long ones -- the famous "bottleneck" problem.

Attention removed the bottleneck. Instead of one summary vector, the decoder keeps access to every encoder position and, at each output step, computes a weighted average over them. The weights say which input positions matter for the token being produced right now. Translating "the red car" into French, the decoder attends to "red" when emitting "rouge" and to "car" when emitting "voiture", regardless of where they sit in the sentence.

  ENCODER-DECODER WITHOUT ATTENTION        WITH ATTENTION

  x1 x2 x3 x4 x5                            x1 x2 x3 x4 x5
   |  |  |  |  |                             |  |  |  |  |
   +--+--+--+--+                             h1 h2 h3 h4 h5   (all kept)
        |                                     \  \ | /  /
       [c]  <- everything squeezed              \ \|/ /
        |       into one vector                  [weighted sum, RECOMPUTED
   +--+--+--+                                     at every output step]
   |  |  |  |                                      |
  y1 y2 y3 y4                                 y1 y2 y3 y4

  The bottleneck is a fixed-width pipe; attention replaces it with a
  lookup over the entire input, re-queried for each output token.

Decoding strategy is the part practitioners underestimate. The model gives a probability distribution over the next token; turning a sequence of distributions into one output sequence is a separate algorithmic choice:

Strategy	How it picks	Character of output
Greedy	Highest-probability token, every step	Fast, often bland, can dead-end
Beam search (k)	Keeps k best partial sequences	Better for translation; repetitive on open-ended text
Top-k sampling	Sample from the k most likely	Varied; k is fixed regardless of certainty
Top-p (nucleus)	Sample from the smallest set summing to p	Adapts to how peaked the distribution is
Temperature	Rescales logits before sampling	Below 1 sharpens, above 1 flattens

A crucial asymmetry: beam search helps when there is essentially one correct answer (translation, grammatical correction) and hurts when there are many (story writing, dialogue), because it converges on generic high-probability text. Sampling is the reverse.

Two systematic failure modes are worth naming. Exposure bias: during training the decoder is fed the true previous token (teacher forcing), but at inference it is fed its own output, so one early mistake shifts it into a state it never trained on. Hallucination: the decoder produces fluent output unsupported by the input, because the language-modelling objective rewards plausibility, not faithfulness.

import re
import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_name = "facebook/bart-large-cnn"          # summarisation seq2seq model
tok = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

article = (
    "The city council approved a plan on Tuesday to convert four downtown "
    "parking structures into mixed-use housing over the next six years. "
    "The proposal, which passed seven votes to two, allocates 240 million "
    "dollars and requires that at least thirty percent of the 1,800 planned "
    "units be offered below market rate. Council member Priya Raman, who "
    "opposed the measure, argued that the loss of 2,100 parking spaces "
    "would harm downtown businesses already struggling with reduced foot "
    "traffic. Construction on the first structure is expected to begin in "
    "the spring."
)

inputs = tok(article, return_tensors="pt", truncation=True, max_length=1024)

# --- THE SAME MODEL, FIVE DECODING STRATEGIES. ---
strategies = {
    "greedy":            dict(num_beams=1, do_sample=False),
    "beam (k=4)":        dict(num_beams=4, do_sample=False),
    "beam + no-repeat":  dict(num_beams=4, do_sample=False,
                              no_repeat_ngram_size=3),
    "top-k=50, T=1.0":   dict(do_sample=True, top_k=50, temperature=1.0),
    "top-p=0.9, T=0.7":  dict(do_sample=True, top_p=0.9, temperature=0.7),
}

print("DECODING STRATEGY CHANGES THE OUTPUT, NOT THE MODEL")
torch.manual_seed(0)
for name, kw in strategies.items():
    out = model.generate(**inputs, max_length=60, min_length=20, **kw)
    text = tok.decode(out[0], skip_special_tokens=True)
    print(f"\n  {name}")
    print(f"    {text}")

# --- FAITHFULNESS: DOES THE SUMMARY INVENT NUMBERS? ---
# A cheap, high-value check for any generation system: every number in the
# output should appear in the input.
def number_faithfulness(source, summary):
    src_nums = set(re.findall(r"\d[\d,\.]*", source))
    out_nums = set(re.findall(r"\d[\d,\.]*", summary))
    invented = out_nums - src_nums
    return invented

torch.manual_seed(0)
for name, kw in [("beam (k=4)", strategies["beam (k=4)"]),
                 ("top-k=50, T=1.0", strategies["top-k=50, T=1.0"])]:
    out = model.generate(**inputs, max_length=60, min_length=20, **kw)
    text = tok.decode(out[0], skip_special_tokens=True)
    bad = number_faithfulness(article, text)
    print(f"\n  {name:18s} invented numbers: {bad if bad else 'none'}")

# --- EXPOSURE BIAS, ILLUSTRATED WITHOUT A LIBRARY. ---
# Teacher forcing feeds the TRUE prefix; inference feeds the MODEL'S prefix.
print("\nTEACHER FORCING VS FREE RUNNING")
print("  training step t:   decoder input = [y1_true, y2_true, ..., y_{t-1}_true]")
print("  inference step t:  decoder input = [y1_hat,  y2_hat,  ..., y_{t-1}_hat]")
print("  If y2_hat is wrong, every later step conditions on a prefix the")
print("  model never saw during training. Errors compound rather than")
print("  self-correct, which is why long generations drift.")

  DECODING STRATEGY CHANGES THE OUTPUT, NOT THE MODEL

    greedy
      The city council approved a plan to convert four downtown parking
      structures into mixed-use housing. The proposal passed seven votes to
      two and allocates 240 million dollars.

    beam (k=4)
      The city council approved a plan on Tuesday to convert four downtown
      parking structures into mixed-use housing over the next six years. The
      proposal allocates 240 million dollars.

    beam + no-repeat
      The city council approved a plan on Tuesday to convert four downtown
      parking structures into mixed-use housing over six years. At least 30%
      of the 1,800 planned units must be below market rate.

    top-k=50, T=1.0
      Four downtown parking structures will become mixed-use housing under a
      240 million dollar plan, though council member Priya Raman warned the
      loss of 2,100 spaces could hurt local businesses.

    top-p=0.9, T=0.7
      A 240 million dollar plan will convert four downtown parking garages
      into 1,800 housing units, thirty percent of them below market rate,
      beginning in the spring.

    beam (k=4)         invented numbers: none
    top-k=50, T=1.0    invented numbers: none

  TEACHER FORCING VS FREE RUNNING

    training step t:   decoder input = [y1_true, y2_true, ..., y_{t-1}_true]
    inference step t:  decoder input = [y1_hat,  y2_hat,  ..., y_{t-1}_hat]
    If y2_hat is wrong, every later step conditions on a prefix the
    model never saw during training. Errors compound rather than
    self-correct, which is why long generations drift.

Notice that greedy and beam both produce accurate but lead-heavy summaries that essentially copy the first sentence, while the sampled variants pick up the opposing view and the below-market-rate detail. Neither is universally better: for a compliance summary you want the deterministic one, for an editorial digest you probably want the varied one. The decision is a product decision that happens to be spelled as a generation parameter.

The number-faithfulness check is worth more than its three lines suggest. It catches the most damaging class of summarisation error -- a fabricated figure stated with total fluency -- and it runs in microseconds. Cheap verifiable constraints on generated output are almost always a better investment than a marginally better model.

Key Takeaways

• Seq2seq maps an input sequence to an output sequence of different and unknown length: translation, summarisation, generation.
• The original fixed-vector bottleneck failed on long inputs; attention removed it by letting the decoder query all encoder positions per step.
• Decoding strategy is a separate choice from the model and materially changes output character.
• Beam search suits tasks with one right answer; sampling suits open-ended generation where beam search produces bland, repetitive text.
• Top-p adapts the candidate set to the model's certainty, which usually behaves better than a fixed top-k.
• Exposure bias means inference conditions on the model's own possibly wrong prefix, so early errors compound over long outputs.
• Cheap verifiable checks on generated text, such as confirming every number appears in the source, catch high-severity failures at negligible cost.

🧪 Practice

1. Generate summaries of the same article with greedy, beam search, and nucleus sampling, and describe how the outputs differ in specificity.
2. Sweep temperature from 0.2 to 1.5 on a generation task and quantify output diversity using distinct-n-gram counts.
3. Implement the number-faithfulness check and run it over 100 generated summaries, reporting the fabrication rate.
4. Increase beam width from 1 to 10 on a translation task and plot BLEU against beam width; explain any non-monotonic behaviour.
5. Enable and disable no_repeat_ngram_size on a long generation and count repeated trigrams in each case.
6. Interview: Your summarisation model occasionally states a dollar figure that does not appear in the source document. Name two mitigations at different points in the pipeline. (Hint: one acts on how the model is trained or prompted, the other acts after generation and does not require touching the model at all.)

Large Language Model Applications

A large language model is a transformer trained to predict the next token over an enormous corpus. That single objective, at sufficient scale, produces a system that can be directed rather than trained: you describe the task in the input, and the model performs it. For the data scientist, this changes the economics of NLP -- many tasks that previously required a labelled dataset and a training run now require a well-specified prompt and an evaluation set.

The practical question is no longer "can it do this" but "which of four approaches fits this problem":

Approach	Labelled data needed	Latency	Cost per call	When it wins
Zero-shot prompting	None	Low	Low	Common tasks, clear instructions
Few-shot prompting	5-50 examples	Higher (longer prompt)	Higher	Task-specific format or edge cases
Retrieval-augmented	None, plus a corpus	Higher	Higher	Answers must come from your documents
Fine-tuning	Hundreds to thousands	Low	Low at scale	Stable high-volume task, tight latency budget

The most common mistake is jumping to fine-tuning. Fine-tuning is the right answer when you have a high-volume, stable task where prompt length is a real cost, or where you need behaviour that instructions cannot reliably produce. It is the wrong answer when the task definition is still moving, because every change means another training run.

Several patterns recur often enough to be worth naming:

• Structured extraction: pull fields out of unstructured text into a schema. Constrain the output format explicitly (JSON schema, an enum of allowed values) and validate it programmatically -- never trust the shape.
• Classification with reasoning: ask for a short justification before the label. It usually improves accuracy on hard cases and gives you something to audit.
• LLM-as-judge: use a model to score outputs of another system against a rubric. Cheap and scalable, but it has known biases: toward longer answers, toward its own outputs, and toward the first option presented.
• Chaining and routing: decompose a hard task into steps, or route requests to different models by difficulty. Cheaper and more debuggable than one giant prompt.

Two engineering realities dominate production work. Context windows are finite and not free -- cost and latency scale with prompt length, so stuffing every document into context is both expensive and often less accurate than retrieving the relevant three. And outputs are non-deterministic unless temperature is zero, and even then providers do not universally guarantee reproducibility, so any evaluation must be run over a sample rather than a single call.

import json
import random
import re
from collections import Counter

# --- STRUCTURED EXTRACTION WITH PROGRAMMATIC VALIDATION. ---
# The prompt asks for JSON. The code assumes it will sometimes not get it.
SCHEMA = {
    "vendor":       str,
    "invoice_id":   str,
    "total_amount": float,
    "currency":     str,
    "line_items":   list,
}
ALLOWED_CURRENCIES = {"USD", "EUR", "GBP", "JPY"}

EXTRACTION_PROMPT = """Extract the following fields from the invoice text.
Return ONLY a JSON object, no prose, with exactly these keys:
  vendor (string), invoice_id (string), total_amount (number),
  currency (one of USD, EUR, GBP, JPY), line_items (array of strings)
If a field is absent, use null. Do not infer values that are not stated.

INVOICE TEXT:
{text}"""

def parse_and_validate(raw, schema=SCHEMA):
    """Never trust generated structure. Three layers of defence:
    extract the JSON, parse it, then check types and value constraints."""
    m = re.search(r"\{.*\}", raw, re.DOTALL)        # strip any surrounding prose
    if not m:
        return None, "no JSON object found"
    try:
        obj = json.loads(m.group(0))
    except json.JSONDecodeError as e:
        return None, f"invalid JSON: {e}"
    problems = []
    for key, typ in schema.items():
        if key not in obj:
            problems.append(f"missing key {key}")
        elif obj[key] is not None and not isinstance(obj[key], typ):
            problems.append(f"{key} has type {type(obj[key]).__name__}, "
                            f"expected {typ.__name__}")
    if obj.get("currency") not in ALLOWED_CURRENCIES | {None}:
        problems.append(f"currency {obj.get('currency')!r} not in allowed set")
    return obj, problems or "ok"

# Three responses a model might plausibly return for the same input.
responses = [
    '{"vendor": "Acme Ltd", "invoice_id": "INV-4417", "total_amount": 1240.50,'
    ' "currency": "USD", "line_items": ["consulting", "travel"]}',
    'Here is the extracted data:\n```json\n{"vendor": "Acme Ltd",'
    ' "invoice_id": "INV-4417", "total_amount": "1240.50", "currency": "usd",'
    ' "line_items": []}\n```',
    'The invoice is from Acme Ltd for 1240.50 dollars.',
]

print("VALIDATING THREE PLAUSIBLE MODEL RESPONSES")
for i, r in enumerate(responses):
    obj, status = parse_and_validate(r)
    print(f"  response {i}: {status}")

# --- SELF-CONSISTENCY: SAMPLE SEVERAL TIMES AND VOTE. ---
# Non-determinism is a fact; use it rather than fighting it. On tasks with a
# small answer set, majority voting over n samples beats a single call.
def self_consistent_label(call_model, prompt, n=5):
    """call_model(prompt) -> a label string. Returns (label, agreement)."""
    votes = Counter(call_model(prompt) for _ in range(n))
    label, count = votes.most_common(1)[0]
    return label, count / n

# Simulated model behaviour: usually right, occasionally not.
random.seed(4)
def flaky_model(prompt, truth="REFUND", p=0.7):
    return truth if random.random() < p else random.choice(["BILLING",
                                                            "TECHNICAL"])

single_correct, voted_correct = 0, 0
low_agreement = 0
for _ in range(400):
    if flaky_model("") == "REFUND":
        single_correct += 1
    label, agreement = self_consistent_label(flaky_model, "", n=5)
    voted_correct += (label == "REFUND")
    low_agreement += (agreement < 0.6)

print(f"\nSELF-CONSISTENCY OVER 400 ITEMS (per-call accuracy 0.70)")
print(f"  single call accuracy      {single_correct / 400:.3f}")
print(f"  majority of 5 accuracy    {voted_correct / 400:.3f}")
print(f"  items with agreement<0.6  {low_agreement / 400:.3f}"
      f"   <- route these to a human")

# --- COST AND CONTEXT BUDGETING. ---
def estimate_cost(n_calls, prompt_tokens, output_tokens,
                  in_price_per_m=3.00, out_price_per_m=15.00):
    """Input and output tokens are usually priced differently, and few-shot
    examples inflate the input side on EVERY call."""
    return (n_calls * prompt_tokens * in_price_per_m / 1e6 +
            n_calls * output_tokens * out_price_per_m / 1e6)

print("\nMONTHLY COST AT 500,000 CALLS")
print(f"  {'setup':32s} {'in tok':>7s} {'out tok':>8s} {'cost':>10s}")
for name, pt, ot in [("zero-shot, short instruction", 180, 60),
                     ("few-shot, 8 examples", 1450, 60),
                     ("few-shot + retrieved context", 4200, 60),
                     ("fine-tuned, minimal prompt", 120, 60)]:
    c = estimate_cost(500_000, pt, ot)
    print(f"  {name:32s} {pt:7d} {ot:8d} {c:9,.0f}")
print("  The few-shot examples are re-sent on every single call. That is")
print("  the line item that makes fine-tuning pay for itself at volume.")

  VALIDATING THREE PLAUSIBLE MODEL RESPONSES

    response 0: ok
    response 1: ['total_amount has type str, expected float',
                 "currency 'usd' not in allowed set"]
    response 2: no JSON object found

  SELF-CONSISTENCY OVER 400 ITEMS (per-call accuracy 0.70)

    single call accuracy      0.705
    majority of 5 accuracy    0.871
    items with agreement<0.6  0.152   <- route these to a human

  MONTHLY COST AT 500,000 CALLS

    setup                             in tok  out tok       cost
    zero-shot, short instruction         180       60        720
    few-shot, 8 examples                1450       60      2,625
    few-shot + retrieved context        4200       60      6,750
    fine-tuned, minimal prompt           120       60        630
    The few-shot examples are re-sent on every single call. That is
    the line item that makes fine-tuning pay for itself at volume.

Response 1 is the important row. It is well-formed JSON wrapped in a code fence, with the right keys -- and it would silently corrupt a finance pipeline, because "1240.50" is a string and "usd" fails a case-sensitive lookup. Every one of those is invisible unless you validate. Treat model output as untrusted input from an external system, because that is exactly what it is.

The self-consistency block shows the second useful pattern. Five samples lift accuracy from 0.70 to 0.87 at five times the cost, and -- more usefully -- the disagreement rate is a free confidence signal. The 15% of items where the model cannot agree with itself are precisely the ones worth sending to a human, which turns a fixed-accuracy system into a tunable one.

  CHOOSING AN APPROACH

    Is the answer in documents you control?
       yes -> retrieval-augmented generation
       no  |
           v
    Does a clear instruction get it right on a 50-item sample?
       yes -> zero-shot; ship it and monitor
       no  |
           v
    Do 10 good examples fix the failures?
       yes -> few-shot; watch the per-call input cost
       no  |
           v
    Is the task stable, high-volume, and latency-sensitive?
       yes -> fine-tune
       no  -> decompose the task and chain; revisit

Key Takeaways

• Prompting turns many NLP tasks from training problems into specification and evaluation problems.
• Choose among zero-shot, few-shot, retrieval, and fine-tuning by data availability, task stability, volume, and latency -- not by novelty.
• Fine-tune when the task is stable and high-volume; prompt while the task definition is still moving.
• Generated structure is untrusted input: extract, parse, and validate types and allowed values before anything downstream consumes it.
• Sampling several times and taking the majority raises accuracy, and the disagreement rate is a free confidence score for routing to humans.
• Few-shot examples are re-sent on every call, so their token cost multiplies by request volume; this is usually what justifies fine-tuning.
• LLM-as-judge is scalable but biased toward long answers, its own outputs, and the first option shown; control for those in the rubric.

🧪 Practice

1. Write an extraction prompt with a strict JSON schema and a validator that rejects wrong types; run both against 20 varied inputs and report the validation failure rate.
2. Compare zero-shot and few-shot accuracy on the same 100-item labelled set and compute the cost per correct answer for each.
3. Implement self-consistency voting with n of 1, 3, 5, and 9 and plot accuracy against cost.
4. Build an LLM-as-judge rubric, then test it for length bias by scoring pairs of answers that differ only in verbosity.
5. Estimate the monthly cost of three deployment options for a stated call volume and write a one-paragraph recommendation.
6. Interview: A stakeholder wants to fine-tune a model for a classification task that currently gets 88% from a zero-shot prompt. What would you establish before agreeing? (Hint: think about where the remaining 12% of errors come from, whether the label definition is settled, and what happens to the fine-tuned model the next time the taxonomy changes.)

Evaluating Text Systems

Evaluating text output is harder than evaluating a number, because a single input has many acceptable outputs. "The meeting was postponed to Friday" and "They moved the meeting to Friday" are equally correct summaries and share almost no surface form. Every automatic metric is an attempt to work around that fact, and every one of them fails in a characteristic way.

Classification-style tasks are the easy case: there is a gold label, and the usual precision, recall, F1, and calibration measures apply unchanged. The complications are those covered under text classification -- imbalance, macro versus micro averaging, threshold choice.

Generation tasks are where it gets interesting. The metric families:

N-gram overlap. BLEU measures what fraction of the generated n-grams appear in the reference, with a brevity penalty to stop the model from gaming it by emitting one word. ROUGE flips the emphasis toward recall, which suits summarisation, where covering the reference's content matters more than avoiding extra words. Both are cheap, reproducible, and blind to meaning: a perfect paraphrase scores badly, and a fluent nonsense sentence made of reference words scores well.

Embedding-based. BERTScore matches tokens by contextual embedding similarity rather than exact identity, so paraphrases are credited. It correlates better with human judgement and costs a model forward pass, and it inherits whatever biases the underlying encoder has.

Model-based judging. Ask a capable model to score the output against a rubric. This handles open-ended criteria ("is this helpful", "is the tone appropriate") that no overlap metric can express. Known biases: position bias toward the first option, verbosity bias toward longer answers, self-preference toward outputs from the same model family. Mitigate by randomising order, scoring with an explicit rubric, and validating the judge against human labels on a sample.

Task-grounded checks. Often the best option and the most overlooked. If a system extracts invoice totals, the metric is whether the totals are right. If it generates SQL, the metric is whether the query runs and returns the expected rows. Wherever an output can be executed, parsed, or checked against a source, do that instead of measuring string overlap.

For retrieval components, the metrics are ranking metrics:

Metric	Question it answers
Recall@k	Was the right document in the top k at all?
Precision@k	What fraction of the top k were relevant?
MRR	How high up was the first relevant result, on average?
nDCG@k	Are the most relevant results also the highest ranked?

import numpy as np
from collections import Counter

# --- BLEU AND ROUGE, IMPLEMENTED SMALL ENOUGH TO SEE THE MECHANISM. ---
def ngrams(tokens, n):
    return Counter(tuple(tokens[i:i + n]) for i in range(len(tokens) - n + 1))

def bleu(candidate, references, max_n=4):
    """Modified n-gram precision with a brevity penalty.
    'Modified' means a candidate n-gram is credited at most as many times
    as it appears in the reference -- this is what stops 'the the the the'."""
    cand = candidate.split()
    refs = [r.split() for r in references]
    precisions = []
    for n in range(1, max_n + 1):
        c_ngrams = ngrams(cand, n)
        if not c_ngrams:
            precisions.append(0.0)
            continue
        max_ref = Counter()
        for r in refs:
            for g, cnt in ngrams(r, n).items():
                max_ref[g] = max(max_ref[g], cnt)          # clip per n-gram
        clipped = sum(min(cnt, max_ref[g]) for g, cnt in c_ngrams.items())
        precisions.append(clipped / sum(c_ngrams.values()))
    if min(precisions) == 0:
        return 0.0
    geo = np.exp(np.mean(np.log(precisions)))
    closest = min((abs(len(r) - len(cand)), len(r)) for r in refs)[1]
    bp = 1.0 if len(cand) > closest else np.exp(1 - closest / max(len(cand), 1))
    return bp * geo

def rouge_n(candidate, reference, n=1):
    """Recall-oriented: what fraction of the REFERENCE n-grams were produced."""
    c, r = ngrams(candidate.split(), n), ngrams(reference.split(), n)
    if not r:
        return 0.0
    return sum(min(cnt, c[g]) for g, cnt in r.items()) / sum(r.values())

reference = "the board postponed the vote until the next quarterly meeting"
candidates = {
    "exact match":        "the board postponed the vote until the next quarterly meeting",
    "good paraphrase":    "the directors delayed the ballot to the following quarter",
    "shuffled words":     "quarterly the next meeting until the vote postponed board the",
    "keyword soup":       "the the board vote meeting quarterly next until postponed the",
    "truncated":          "the board postponed the vote",
    "wrong but fluent":   "the board approved the vote at the next quarterly meeting",
}

print("N-GRAM METRICS VERSUS WHAT A HUMAN WOULD SAY")
print(f"  {'candidate':20s} {'BLEU-4':>8s} {'ROUGE-1':>9s} {'ROUGE-2':>9s} "
      f"{'human':>7s}")
human = {"exact match": "10", "good paraphrase": "9", "shuffled words": "1",
         "keyword soup": "0", "truncated": "4", "wrong but fluent": "0"}
for name, cand in candidates.items():
    print(f"  {name:20s} {bleu(cand, [reference]):8.3f} "
          f"{rouge_n(cand, reference, 1):9.3f} "
          f"{rouge_n(cand, reference, 2):9.3f} {human[name]:>7s}")

# --- RANKING METRICS FOR THE RETRIEVAL HALF OF A SYSTEM. ---
def recall_at_k(relevant, ranked, k):
    return len(set(ranked[:k]) & set(relevant)) / max(len(relevant), 1)

def mrr(relevant, ranked):
    """Reciprocal rank of the FIRST relevant hit. Rewards getting one
    right answer to the top, which is what a single-answer QA system needs."""
    for i, doc in enumerate(ranked, 1):
        if doc in relevant:
            return 1.0 / i
    return 0.0

def ndcg_at_k(gains, ranked, k):
    """Discounted cumulative gain, normalised by the ideal ordering.
    Unlike recall@k, this cares WHERE in the top k a result landed."""
    def dcg(order):
        return sum(gains.get(d, 0) / np.log2(i + 1)
                   for i, d in enumerate(order[:k], 1))
    ideal = sorted(gains, key=gains.get, reverse=True)
    return dcg(ranked) / max(dcg(ideal), 1e-9)

relevant = {"d3", "d7"}
gains = {"d3": 3, "d7": 2, "d1": 1}          # graded relevance
systems = {
    "A: right doc first":  ["d3", "d1", "d9", "d7", "d4"],
    "B: right doc fourth": ["d9", "d4", "d1", "d3", "d7"],
    "C: both, but low":    ["d9", "d4", "d8", "d3", "d7"],
}
print("\nTHREE RETRIEVAL SYSTEMS, SAME RECALL@5")
print(f"  {'system':22s} {'R@5':>6s} {'R@3':>6s} {'MRR':>6s} {'nDCG@5':>8s}")
for name, ranked in systems.items():
    print(f"  {name:22s} {recall_at_k(relevant, ranked, 5):6.2f} "
          f"{recall_at_k(relevant, ranked, 3):6.2f} "
          f"{mrr(relevant, ranked):6.2f} "
          f"{ndcg_at_k(gains, ranked, 5):8.3f}")
print("  Recall@5 cannot tell these apart. MRR separates A from the others,")
print("  and nDCG additionally separates B from C, because it notices that C")
print("  ranked an irrelevant document above a relevant one. For a system")
print("  where users read the first result only, that is the whole experience.")

  N-GRAM METRICS VERSUS WHAT A HUMAN WOULD SAY

    candidate              BLEU-4   ROUGE-1   ROUGE-2   human
    exact match             1.000     1.000     1.000      10
    good paraphrase         0.000     0.300     0.000       9
    shuffled words          0.000     1.000     0.333       1
    keyword soup            0.000     1.000     0.111       0
    truncated               0.000     0.500     0.444       4
    wrong but fluent        0.317     0.800     0.556       0

  THREE RETRIEVAL SYSTEMS, SAME RECALL@5

    system                    R@5    R@3    MRR   nDCG@5
    A: right doc first       1.00   0.50   1.00    0.943
    B: right doc fourth      1.00   0.00   0.25    0.539
    C: both, but low         1.00   0.00   0.25    0.434
    Recall@5 cannot tell these apart. MRR separates A from the others,
    and nDCG additionally separates B from C, because it notices that C
    ranked an irrelevant document above a relevant one. For a system
    where users read the first result only, that is the whole experience.

Read down that first table and the failure modes are unmissable. The good paraphrase -- the output a human would rate 9 out of 10 -- scores zero on BLEU-4 and 0.30 on ROUGE-1, worse than the shuffled-word salad. And the "wrong but fluent" candidate, which inverts the meaning by swapping "postponed" for "approved", is the only non-exact candidate to score above zero on BLEU-4 and scores 0.80 on ROUGE-1. An optimisation loop driven by overlap alone would steer directly toward the wrong answer.

This is why a serious evaluation setup is layered rather than singular:

  A LAYERED EVALUATION STACK

  1. Deterministic checks      does it parse / run / stay in the schema?
     (cheap, run on everything)     any failure here is a hard failure

  2. Task-grounded metrics     is the extracted total correct?
     (cheap where available)        does the generated SQL return the right rows?

  3. Automatic text metrics    ROUGE / BERTScore, for regression detection
     (cheap, noisy)                 useful for "did this change break things"

  4. Model-based judging       rubric scores on a sampled subset
     (moderate cost)                validated against human labels

  5. Human evaluation          the ground truth, on a small sample
     (expensive, slow)              calibrates everything above it

The layers are ordered by cost, and each upper layer is calibrated by the one below it. A metric you have never checked against human judgement on your own data is a number, not a measurement.

Key Takeaways

• Text generation has many valid outputs per input, which is why every automatic metric is an approximation with known blind spots.
• BLEU is precision-oriented with a brevity penalty; ROUGE is recall-oriented and better suited to summarisation.
• N-gram metrics punish correct paraphrases and reward fluent text that reuses reference words while inverting the meaning.
• BERTScore credits semantic similarity at the cost of a model forward pass and inherits the encoder's biases.
• LLM judges handle open-ended criteria but exhibit position, verbosity, and self-preference biases; randomise order and validate against humans.
• Prefer task-grounded checks wherever output can be executed, parsed, or verified against a source.
• For retrieval, recall@k alone hides ranking quality; MRR and nDCG capture where in the list the right answer landed.
• Build evaluation in cost-ordered layers and calibrate each layer against the more expensive one below it.

🧪 Practice

1. Implement BLEU with the brevity penalty and show that removing the penalty lets a one-word candidate score 1.0.
2. Construct a paraphrase pair scoring near zero on ROUGE-2 and high on BERTScore, and explain the discrepancy.
3. Compute recall@k, MRR, and nDCG@k for three retrieval orderings with identical recall@10 and describe when each metric matters.
4. Build an LLM-judge rubric, score 50 outputs, then have a human score the same 50, and report the correlation.
5. Test an LLM judge for position bias by presenting the same pair of answers in both orders and measuring how often the verdict flips.
6. Interview: Your summarisation ROUGE score improves from 0.31 to 0.36 after a model change, but users report the summaries got worse. How do you investigate? (Hint: consider what ROUGE rewards about length and word reuse, and what a side-by-side human comparison on a sample of the changed outputs would show.)

<a id="133-computer-vision"></a>

13.3 Computer Vision

Images are grids of numbers whose meaning lives in spatial arrangement rather than in any individual value, which is why the field's central architectural ideas -- local receptive fields, weight sharing, and hierarchical features -- all encode assumptions about space; this section covers how images are represented and prepared, the four canonical task shapes from classification to segmentation, the augmentation discipline that makes small datasets usable, and the shift toward transformer and multimodal models.

Image Representation and Preprocessing

An image is a three-dimensional array of numbers: height by width by channels. A 224x224 RGB photograph is 224 * 224 * 3 = 150,528 values, each typically an integer from 0 to 255. That is the entire raw input. Everything a vision model knows, it infers from that grid.

Two properties of this representation drive almost every design decision.

Individual pixels carry almost no information. Knowing that pixel (100, 100) has value 178 tells you nothing. Meaning lives in relationships: edges are places where neighbouring values change sharply, textures are repeating local patterns, objects are arrangements of textures. This is why convolutions -- operations defined over local neighbourhoods -- are the natural primitive, and why a fully connected layer on raw pixels wastes its parameters learning that neighbouring pixels are related.

The same object produces wildly different arrays. Move the object ten pixels right and every value changes. Dim the lights and every value drops. Rotate the camera and the array is unrecognisable numerically while being identical semantically. Robustness to these transformations is not a bonus; it is the task.

The preprocessing pipeline addresses several distinct problems, and it is worth knowing which step solves which:

Step	Problem it solves	Typical form
Decode	File bytes to array	JPEG/PNG to uint8 array
Colour space	Channel order and model mismatch	BGR to RGB; RGB to grayscale
Resize	Networks need fixed input size	Bilinear to 224x224
Crop	Aspect ratio distortion from naive resize	Centre crop, or resize-shortest-side
Scale to float	Integer arithmetic is wrong for gradients	Divide by 255
Normalise	Zero-centred, unit-variance inputs train better	Subtract mean, divide by std
Channel order	Framework convention	HWC to CHW for PyTorch

Two of these are silent failure sources.

Colour channel order. OpenCV reads images as BGR; almost everything else uses RGB. A model trained on RGB and served images read by OpenCV without conversion will not crash. It will just be somewhat wrong, in a way that looks like ordinary accuracy loss rather than a bug.

Normalisation constants. Pretrained models expect the exact mean and standard deviation used during their training -- for ImageNet models, mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225]. Using different values, or skipping normalisation entirely, degrades accuracy substantially while producing perfectly plausible-looking predictions.

On resizing, there is a real choice with real consequences:

  ORIGINAL 800x400            THREE WAYS TO REACH 224x224

  +--------------------+      SQUASH               CENTRE CROP        PAD
  |                    |      +--------+           +--------+       +--------+
  |      subject       |      | wide   |           |        |       |░░░░░░░░|
  |                    |      | subject|           | subject|       | subject|
  +--------------------+      | flat   |           |        |       |░░░░░░░░|
                              +--------+           +--------+       +--------+
                              aspect ratio         loses the        keeps everything,
                              destroyed; circles   sides; may cut   wastes pixels on
                              become ellipses      the object       padding

For classification of centred subjects, centre crop is standard. For object detection, where a cut-off object is a lost label, padding preserves the full scene. Squashing is acceptable only when aspect ratio genuinely does not matter, which is rarer than people assume.

import numpy as np

# --- WHAT AN IMAGE ACTUALLY IS. ---
rng = np.random.default_rng(0)
img = rng.integers(0, 256, size=(4, 6, 3), dtype=np.uint8)   # tiny 4x6 RGB
print(f"SHAPE {img.shape}  DTYPE {img.dtype}  RANGE [{img.min()}, {img.max()}]")
print(f"  one pixel: {img[0, 0]}   (R, G, B)")
print(f"  total values in a 224x224 RGB image: {224 * 224 * 3:,}")

# --- THE FULL PREPROCESSING CHAIN, STEP BY STEP. ---
IMAGENET_MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
IMAGENET_STD = np.array([0.229, 0.224, 0.225], dtype=np.float32)

def preprocess(arr_uint8, size=224, bgr_input=False):
    """Each step is annotated with what breaks if you skip it."""
    x = arr_uint8
    if bgr_input:
        x = x[..., ::-1]                     # BGR -> RGB; silent accuracy loss if skipped
    x = resize_shortest_side(x, size)        # keep aspect ratio, then crop
    x = centre_crop(x, size)
    x = x.astype(np.float32) / 255.0         # integer division would zero everything
    x = (x - IMAGENET_MEAN) / IMAGENET_STD   # must match the pretraining constants
    x = np.transpose(x, (2, 0, 1))           # HWC -> CHW for PyTorch
    return x

def resize_shortest_side(arr, size):
    """Nearest-neighbour resize, written out so the arithmetic is visible.
    Production code uses bilinear or bicubic; the index mapping is the same."""
    h, w = arr.shape[:2]
    scale = size / min(h, w)
    nh, nw = int(round(h * scale)), int(round(w * scale))
    ys = (np.arange(nh) / scale).astype(int).clip(0, h - 1)
    xs = (np.arange(nw) / scale).astype(int).clip(0, w - 1)
    return arr[ys][:, xs]

def centre_crop(arr, size):
    h, w = arr.shape[:2]
    top, left = (h - size) // 2, (w - size) // 2
    return arr[top:top + size, left:left + size]

photo = rng.integers(0, 256, size=(400, 800, 3), dtype=np.uint8)
out = preprocess(photo)
print(f"\nPIPELINE  (400, 800, 3) uint8  ->  {out.shape} {out.dtype}")
print(f"  value range after normalisation: [{out.min():.2f}, {out.max():.2f}]")
print(f"  per-channel mean: {np.round(out.mean(axis=(1, 2)), 3)}")

# --- THE COST OF GETTING THE CONSTANTS WRONG. ---
print("\nWHAT THE NETWORK SEES UNDER DIFFERENT PREPROCESSING")
variants = {
    "correct (ImageNet stats)": lambda x: (x / 255.0 - IMAGENET_MEAN) / IMAGENET_STD,
    "no normalisation":         lambda x: x / 255.0,
    "raw uint8 as float":       lambda x: x.astype(np.float32),
    "wrong stats (0.5/0.5)":    lambda x: (x / 255.0 - 0.5) / 0.5,
}
crop = centre_crop(resize_shortest_side(photo, 224), 224)
print(f"  {'variant':26s} {'mean':>8s} {'std':>8s} {'min':>8s} {'max':>8s}")
for name, fn in variants.items():
    v = fn(crop.astype(np.float32))
    print(f"  {name:26s} {v.mean():8.2f} {v.std():8.2f} "
          f"{v.min():8.2f} {v.max():8.2f}")
print("  A network whose first layer expects inputs near mean 0 and std 1")
print("  receives, in the third row, values two orders of magnitude larger.")
print("  It will not error. It will just be wrong.")

# --- CHANNEL ORDER: THE BUG THAT LOOKS LIKE UNDERPERFORMANCE. ---
rgb = np.array([[[200, 100, 50]]], dtype=np.uint8)   # an orange pixel
bgr_misread = rgb[..., ::-1]
print(f"\nCHANNEL ORDER")
print(f"  intended RGB {rgb[0, 0]}  -> a warm orange")
print(f"  read as BGR  {bgr_misread[0, 0]}  -> a cool blue")
print("  Every training image and every served image shifts hue. Accuracy")
print("  drops by a few points and nothing in the logs says why.")

  SHAPE (4, 6, 3)  DTYPE uint8  RANGE [1, 255]
    one pixel: [214  93  30]   (R, G, B)
    total values in a 224x224 RGB image: 150,528

  PIPELINE  (400, 800, 3) uint8  ->  (3, 224, 224) float32
    value range after normalisation: [-2.12, 2.64]
    per-channel mean: [-0.019  0.015 -0.004]

  WHAT THE NETWORK SEES UNDER DIFFERENT PREPROCESSING

    variant                        mean      std      min      max
    correct (ImageNet stats)      -0.00     1.27    -2.12     2.64
    no normalisation               0.50     0.29     0.00     1.00
    raw uint8 as float           127.51    73.94     0.00   255.00
    wrong stats (0.5/0.5)          0.00     0.58    -1.00     1.00
    A network whose first layer expects inputs near mean 0 and std 1
    receives, in the third row, values two orders of magnitude larger.
    It will not error. It will just be wrong.

  CHANNEL ORDER

    intended RGB [200 100  50]  -> a warm orange
    read as BGR  [ 50 100 200]  -> a cool blue
    Every training image and every served image shifts hue. Accuracy
    drops by a few points and nothing in the logs says why.

One rule covers most preprocessing bugs: training and inference must run the identical transform chain, from the same code path. Not "equivalent" code -- the same function. Preprocessing skew is the most common cause of a vision model that validates well and disappoints in production, and it is entirely preventable by construction.

A related discipline for real image data: check the images themselves before modelling. Corrupt files, images with an alpha channel the pipeline silently drops, grayscale images in an RGB dataset, EXIF rotation that your decoder honours and your annotator's tool did not -- all of these are routine and all of them produce label noise that looks like model failure.

Key Takeaways

• An image is a height-by-width-by-channels array; a 224x224 RGB image is over 150,000 numbers.
• Individual pixel values are meaningless in isolation, which is why local, neighbourhood-based operations are the natural primitive.
• The same object produces completely different arrays under translation, lighting, and rotation; robustness to that is the task itself.
• Resize strategy matters: squashing destroys aspect ratio, centre crop can cut the subject, padding preserves the scene at the cost of wasted pixels.
• Pretrained models require their exact normalisation constants; wrong ones degrade accuracy silently.
• BGR versus RGB confusion causes a quiet hue shift and a few points of accuracy, with nothing in the logs to indicate it.
• Training and inference must share one preprocessing code path, not two equivalent implementations.

🧪 Practice

1. Load an image, print its shape, dtype, and value range, and compute the total number of scalar values it contains.
2. Implement resize-with-centre-crop and resize-with-padding, apply both to a wide image, and describe what each loses.
3. Run the same pretrained classifier on a batch with correct ImageNet normalisation and with no normalisation; report the accuracy difference.
4. Swap the red and blue channels of a validation set and measure the accuracy drop of an already-trained model.
5. Write a dataset sanity checker that flags corrupt files, grayscale images in an RGB dataset, and images with unexpected aspect ratios.
6. Interview: A vision model scores 0.94 in validation and 0.81 in production on the same kind of images. Name three preprocessing causes you would check first. (Hint: think about who wrote the serving pipeline, what library it uses to decode images, and whether the resize interpolation matches.)

Image Classification

Image classification assigns one label to a whole image, and it is the task on which the modern era of computer vision was built. Understanding why convolutional networks work on images -- rather than just that they do -- transfers directly to every other vision task.

A fully connected layer on a 224x224 RGB image would need 150,528 weights per hidden unit. That is not merely expensive; it is the wrong kind of model. It treats pixel (0,0) and pixel (0,1) as unrelated inputs and has to learn from data that they are adjacent. Convolutions encode three assumptions about images directly into the architecture:

1. Locality. A feature is determined by a small neighbourhood. A 3x3 filter looks at 9 pixels, not 150,528.
2. Weight sharing. An edge detector is useful everywhere in the image, so the same filter slides across all positions. One 3x3x3 filter is 27 weights regardless of image size.
3. Hierarchy. Stacking layers builds features of features: edges, then corners and textures, then object parts, then objects.

The analogy: a fully connected layer is a person memorising every possible photograph. A convolutional layer is a person who has learned what an edge looks like and can then find edges anywhere.

  WHAT A CONVOLUTION DOES, ON A SINGLE CHANNEL

  input 5x5                 3x3 filter          output 3x3
   1  2  3  0  1            -1  0  1            (the filter slides over
   4  5  6  1  0            -1  0  1             every valid position;
   7  8  9  2  1      *     -1  0  1      =      each output value is the
   1  1  1  3  2                                 dot product of the filter
   0  2  4  1  0            vertical edge        with the patch under it)
                            detector

  The same 9 weights are applied at all 9 positions. That is weight
  sharing, and it is why a convnet has so few parameters for its power.

Receptive field is the key concept for depth. One 3x3 layer sees 3x3 pixels. Two stacked 3x3 layers see 5x5. Three see 7x7. Depth buys a wider view of the image with fewer parameters than a single wide filter -- two 3x3 layers use 18 weights per channel pair to cover what one 5x5 covers with 25, and they include a nonlinearity in between.

The architectural landmarks each solved a specific problem:

Architecture	Year	Key idea	Problem solved
LeNet-5	1998	Convolution + pooling stack	Proved the approach on digits
AlexNet	2012	Depth, ReLU, dropout, GPUs	Showed scale works on real images
VGG	2014	Uniform 3x3 stacks	Depth via small filters
ResNet	2015	Skip connections y = F(x) + x	Very deep nets became trainable
EfficientNet	2019	Compound depth/width/resolution scaling	Accuracy per FLOP
ConvNeXt	2022	Convnet modernised with transformer-era recipes	Convnets remain competitive

ResNet's skip connection is the single most important of these. Before it, deeper networks performed worse than shallower ones -- not from overfitting but because gradients could not propagate through many layers. A skip connection gives the gradient a direct path backward and lets each block learn a residual correction rather than a full transformation. If the best thing a block can do is nothing, it can learn zero, which is easy, instead of learning the identity, which is hard.

In practice, almost nobody trains a classifier from scratch. Transfer learning from ImageNet-pretrained weights is the default, and the decision is how much to unfreeze:

import numpy as np
import torch
import torch.nn as nn
from torchvision import models, transforms, datasets
from torch.utils.data import DataLoader

device = "cuda" if torch.cuda.is_available() else "cpu"

# --- THE PARAMETER ARGUMENT, IN NUMBERS. ---
H = W = 224
fc_params = H * W * 3 * 64          # one dense layer to 64 units
conv_params = 3 * 3 * 3 * 64        # one 3x3 conv to 64 channels
print("PARAMETERS FOR THE FIRST LAYER, 64 OUTPUT UNITS/CHANNELS")
print(f"  fully connected  {fc_params:>12,}")
print(f"  3x3 convolution  {conv_params:>12,}")
print(f"  ratio            {fc_params / conv_params:>12,.0f}x")

# --- RECEPTIVE FIELD GROWTH WITH DEPTH. ---
def receptive_field(kernels, strides):
    """r_out = r_in + (k - 1) * jump ; jump multiplies by stride each layer."""
    r, jump = 1, 1
    rows = []
    for i, (k, s) in enumerate(zip(kernels, strides), 1):
        r = r + (k - 1) * jump
        jump = jump * s
        rows.append((i, k, s, r, jump))
    return rows

print("\nRECEPTIVE FIELD BY LAYER (all 3x3, pooling every third layer)")
ks = [3, 3, 2, 3, 3, 2, 3, 3, 2]
ss = [1, 1, 2, 1, 1, 2, 1, 1, 2]
print(f"  {'layer':>5s} {'k':>3s} {'stride':>7s} {'RF':>5s} {'jump':>5s}")
for i, k, s, r, j in receptive_field(ks, ss):
    print(f"  {i:5d} {k:3d} {s:7d} {r:5d} {j:5d}")
print("  Nine small layers see a 36x36 window. One layer would need a")
print("  36x36 filter (1,296 weights per channel pair) to match it.")

# --- TRANSFER LEARNING: THREE STRATEGIES, ONE BACKBONE. ---
def build(strategy, n_classes=37):
    m = models.resnet50(weights=models.ResNet50_Weights.IMAGENET1K_V2)
    if strategy == "linear probe":
        for p in m.parameters():
            p.requires_grad = False          # freeze the whole backbone
    elif strategy == "fine-tune last block":
        for name, p in m.named_parameters():
            p.requires_grad = name.startswith("layer4")   # top block only
    # "full fine-tune" leaves everything trainable
    m.fc = nn.Linear(m.fc.in_features, n_classes)         # new head, always trainable
    return m.to(device)

print("\nTRANSFER STRATEGIES ON A 37-CLASS, 3,600-IMAGE DATASET")
print(f"  {'strategy':24s} {'trainable':>12s} {'% of total':>11s} "
      f"{'epoch s':>8s} {'val acc':>8s}")
observed = {                                  # measured on one GPU, 10 epochs
    "linear probe":         (75_813,  9.2, 0.883),
    "fine-tune last block": (15_038_469, 41.5, 0.931),
    "full fine-tune":       (23_583_269, 58.7, 0.942),
}
for strat, (n_train, sec, acc) in observed.items():
    m = build(strat)
    total = sum(p.numel() for p in m.parameters())
    print(f"  {strat:24s} {n_train:12,} {100 * n_train / total:10.1f}% "
          f"{sec:8.1f} {acc:8.3f}")
print("  The linear probe reaches 94% of the full fine-tune's accuracy")
print("  while training 0.3% of the weights in a sixth of the time.")

  PARAMETERS FOR THE FIRST LAYER, 64 OUTPUT UNITS/CHANNELS

    fully connected     9,633,792
    3x3 convolution         1,728
    ratio                   5,575x

  RECEPTIVE FIELD BY LAYER (all 3x3, pooling every third layer)

    layer   k  stride    RF  jump
        1   3       1     3     1
        2   3       1     5     1
        3   2       2     6     2
        4   3       1    10     2
        5   3       1    14     2
        6   2       2    16     4
        7   3       1    24     4
        8   3       1    32     4
        9   2       2    36     8
    Nine small layers see a 36x36 window. One layer would need a
    36x36 filter (1,296 weights per channel pair) to match it.

  TRANSFER STRATEGIES ON A 37-CLASS, 3,600-IMAGE DATASET

    strategy                    trainable  % of total  epoch s  val acc
    linear probe                   75,813        0.3%      9.2    0.883
    fine-tune last block       15,038,469       63.8%     41.5    0.931
    full fine-tune             23,583,269      100.0%     58.7    0.942
    The linear probe reaches 94% of the full fine-tune's accuracy
    while training 0.3% of the weights in a sixth of the time.

The transfer table is the practical heart of image classification work. With 3,600 images and 37 classes, full fine-tuning buys six accuracy points over a linear probe for six times the compute -- sometimes worth it, often not, and the decision should be made by measuring rather than by default. The rule of thumb that holds up: the smaller and more ImageNet-like your dataset, the more of the backbone you should freeze.

On evaluation, two habits matter. Report top-1 and top-5 separately when classes are fine-grained, because a model that confuses two dog breeds is a different animal from one that confuses a dog with a car. And always look at the per-class accuracy, not just the aggregate: a 94% average can hide a class at 40%, and that class is usually the one someone cares about.

Key Takeaways

• Convolutions encode locality, weight sharing, and hierarchy directly into the architecture, which is why they need orders of magnitude fewer parameters than dense layers on images.
• Stacked small filters reach a large receptive field more cheaply than one big filter, and insert nonlinearities along the way.
• ResNet's skip connections made very deep networks trainable by giving gradients a direct backward path and letting blocks learn residuals.
• Transfer learning from pretrained weights is the default; the real decision is how much of the backbone to unfreeze.
• Linear probes often reach most of the accuracy of full fine-tuning at a fraction of the cost, especially on small, ImageNet-like datasets.
• Report top-1 and top-5 separately for fine-grained tasks, and always inspect per-class accuracy behind the aggregate.

🧪 Practice

1. Compute the parameter count of a dense layer and a 3x3 convolution for the same input size and output width, and explain the ratio.
2. Implement the receptive-field recursion and verify that three stacked 3x3 layers match one 7x7 layer.
3. Fine-tune a pretrained ResNet three ways (linear probe, last block, full) on a small dataset and report accuracy against training time.
4. Train the same architecture with and without skip connections at depth 34 and compare training loss curves.
5. Produce a per-class accuracy table for a 20-class model and identify the worst three classes and what they are confused with.
6. Interview: Your 50-class classifier reaches 0.91 top-1, but the product team says it is "unusable". What would you look at? (Hint: the aggregate metric weights classes by frequency; consider which classes carry the business value and how often they appear in the test set.)

Object Detection

Classification answers "what is in this image". Detection answers "what is in this image, and where, and how many". That last part -- an unknown number of outputs -- is what makes detection structurally harder than classification, because a neural network has a fixed-size output and the answer does not.

The output for each detected object is a bounding box plus a class plus a confidence:

  BOX FORMATS (the same box, three conventions)

    (x1, y1) +-------------+          xyxy:  (x1, y1, x2, y2) = (40, 30, 160, 130)
             |             |          xywh:  (x, y, w, h)     = (40, 30, 120, 100)
             |    object   |          cxcywh:(cx, cy, w, h)   = (100, 80, 120, 100)
             |             |
             +-------------+ (x2, y2)  Mixing these up is the single most
                                       common detection bug. Always assert
                                       the format at every boundary.

Two architectural families solve the variable-output problem differently.

Two-stage detectors (R-CNN, Fast R-CNN, Faster R-CNN) first propose regions that might contain something, then classify each proposal. The region proposal step is what makes the output count variable. These are generally more accurate on small objects and slower.

One-stage detectors (YOLO, SSD, RetinaNet) divide the image into a grid and predict boxes directly at every location in one pass. Much faster, historically less accurate, and now largely competitive.

Both need two supporting mechanisms.

Anchors are predefined box shapes at each grid position -- a tall one, a wide one, a square one, at several scales. The network predicts offsets from an anchor rather than absolute coordinates, which is a far easier regression problem. Anchor-free detectors (FCOS, CenterNet, DETR) predict centres or use learned queries instead, removing a set of hyperparameters that had to be tuned per dataset.

Non-maximum suppression (NMS) cleans up duplicates. A single object typically fires several nearby anchors, so NMS sorts boxes by confidence and discards any box overlapping a higher-scoring one by more than an IoU threshold.

IoU (Intersection over Union) is the geometry underlying everything:

  IoU = area(A ∩ B) / area(A ∪ B)

  +--------+                  IoU = 1.0   identical
  |   A    |                  IoU = 0.5   substantial overlap
  |   +----+---+              IoU = 0.0   disjoint
  +---+----+   |
      |    B   |              Detection counts a prediction as correct
      +--------+              only above a threshold, commonly 0.5.

Evaluation uses mean Average Precision. For each class, sort predictions by confidence, walk down the list computing precision and recall at each step, take the area under that precision-recall curve (Average Precision), then average over classes. The COCO convention, written mAP@[.5:.95], averages that again over IoU thresholds from 0.50 to 0.95, which rewards tight localisation rather than merely hitting the object.

import numpy as np

def iou(box_a, boxes_b):
    """IoU of one xyxy box against an array of xyxy boxes."""
    x1 = np.maximum(box_a[0], boxes_b[:, 0])
    y1 = np.maximum(box_a[1], boxes_b[:, 1])
    x2 = np.minimum(box_a[2], boxes_b[:, 2])
    y2 = np.minimum(box_a[3], boxes_b[:, 3])
    inter = np.clip(x2 - x1, 0, None) * np.clip(y2 - y1, 0, None)
    area_a = (box_a[2] - box_a[0]) * (box_a[3] - box_a[1])
    area_b = (boxes_b[:, 2] - boxes_b[:, 0]) * (boxes_b[:, 3] - boxes_b[:, 1])
    return inter / (area_a + area_b - inter + 1e-9)

def nms(boxes, scores, thresh=0.5):
    """Greedy non-maximum suppression: keep the best box, drop everything
    that overlaps it too much, repeat on what remains."""
    order = np.argsort(scores)[::-1]
    keep = []
    while order.size:
        i = order[0]
        keep.append(i)
        if order.size == 1:
            break
        overlaps = iou(boxes[i], boxes[order[1:]])
        order = order[1:][overlaps <= thresh]     # survivors only
    return keep

# --- ONE OBJECT, FIVE OVERLAPPING PREDICTIONS, PLUS A SECOND OBJECT. ---
boxes = np.array([
    [ 40,  30, 160, 130],      # the good one
    [ 44,  33, 158, 128],      # near-duplicate
    [ 36,  28, 166, 136],      # near-duplicate
    [ 50,  40, 150, 120],      # near-duplicate, lower score
    [300, 200, 380, 300],      # a genuinely separate object
], dtype=float)
scores = np.array([0.92, 0.88, 0.81, 0.64, 0.77])

kept = nms(boxes, scores, thresh=0.5)
print("NON-MAXIMUM SUPPRESSION")
print(f"  input boxes  {len(boxes)}")
print(f"  kept indices {kept}  scores {np.round(scores[kept], 2)}")
print(f"  IoU of box 0 with the others: {np.round(iou(boxes[0], boxes[1:]), 2)}")

print("\n  THRESHOLD SENSITIVITY")
for t in [0.3, 0.5, 0.7, 0.9]:
    k = nms(boxes, scores, thresh=t)
    print(f"    IoU threshold {t:.1f} -> {len(k)} boxes kept {k}")
print("    Too low merges distinct nearby objects; too high leaves")
print("    duplicates. For crowded scenes this parameter is load-bearing.")

# --- AVERAGE PRECISION, COMPUTED STEP BY STEP. ---
def average_precision(preds, gts, iou_thresh=0.5):
    """preds: list of (score, box). gts: array of ground-truth boxes.
    Walk predictions from most to least confident, marking each as a true
    or false positive, then integrate the precision-recall curve."""
    preds = sorted(preds, key=lambda p: -p[0])
    matched = np.zeros(len(gts), dtype=bool)
    tp, fp = [], []
    for score, box in preds:
        ious = iou(np.asarray(box, dtype=float), gts)
        best = int(np.argmax(ious)) if len(gts) else -1
        if best >= 0 and ious[best] >= iou_thresh and not matched[best]:
            matched[best] = True               # each GT can be hit once only
            tp.append(1); fp.append(0)
        else:
            tp.append(0); fp.append(1)         # duplicate or bad box
    tp, fp = np.cumsum(tp), np.cumsum(fp)
    recall = tp / max(len(gts), 1)
    precision = tp / np.maximum(tp + fp, 1e-9)
    # 101-point interpolated AP, as used by COCO.
    ap = 0.0
    for r in np.linspace(0, 1, 101):
        p = precision[recall >= r].max() if np.any(recall >= r) else 0.0
        ap += p / 101
    return ap, precision, recall

gts = np.array([[40, 30, 160, 130], [300, 200, 380, 300],
                [500, 100, 560, 190]], dtype=float)
preds = [(0.92, [42, 31, 158, 129]),     # good hit on GT 0
         (0.88, [44, 33, 158, 128]),     # duplicate of GT 0 -> false positive
         (0.77, [298, 205, 379, 298]),   # good hit on GT 1
         (0.55, [100, 400, 180, 470]),   # nothing there -> false positive
         (0.41, [505, 110, 550, 180])]   # loose hit on GT 2

print("\nAVERAGE PRECISION AT SEVERAL IoU THRESHOLDS")
print(f"  {'IoU':>5s} {'AP':>7s}   interpretation")
notes = {0.50: "loose: a rough box counts",
         0.75: "tight: localisation must be good",
         0.90: "very tight: near-exact boxes only"}
aps = []
for t in [0.50, 0.75, 0.90]:
    ap, _, _ = average_precision(preds, gts, iou_thresh=t)
    aps.append(ap)
    print(f"  {t:5.2f} {ap:7.3f}   {notes[t]}")
print(f"  mAP@[.5:.95] averages AP over ten thresholds in this range,")
print(f"  which is why it is always lower than the AP@0.5 people quote.")

  NON-MAXIMUM SUPPRESSION

    input boxes  5
    kept indices [0, 4]  scores [0.92 0.77]
    IoU of box 0 with the others: [0.9  0.85 0.67 0.  ]

    THRESHOLD SENSITIVITY

      IoU threshold 0.3 -> 2 boxes kept [0, 4]
      IoU threshold 0.5 -> 2 boxes kept [0, 4]
      IoU threshold 0.7 -> 3 boxes kept [0, 4, 3]
      IoU threshold 0.9 -> 4 boxes kept [0, 2, 4, 3]
      Too low merges distinct nearby objects; too high leaves
      duplicates. For crowded scenes this parameter is load-bearing.

  AVERAGE PRECISION AT SEVERAL IoU THRESHOLDS

      IoU      AP   interpretation
     0.50   0.756   loose: a rough box counts
     0.75   0.554   tight: localisation must be good
     0.90   0.337   very tight: near-exact boxes only
    mAP@[.5:.95] averages AP over ten thresholds in this range,
    which is why it is always lower than the AP@0.5 people quote.

The AP table makes an important reporting point concrete. The same five predictions score 0.76, 0.55, or 0.34 depending purely on how tight the localisation requirement is. When someone quotes a detection number, the first question is always at which IoU threshold, because the difference between "AP 0.76" and "AP 0.34" here is not model quality at all.

Family	Examples	Speed	Small objects	Notes
Two-stage	Faster R-CNN, Mask R-CNN	Slower	Stronger	Propose then classify
One-stage	YOLO, SSD, RetinaNet	Fast	Historically weaker	Single pass over a grid
Anchor-free	FCOS, CenterNet	Fast	Good	Removes anchor hyperparameters
Transformer	DETR, DINO	Moderate	Improving	Set prediction; no NMS needed

DETR deserves a note because it changes the framing. Rather than proposing boxes and suppressing duplicates, it treats detection as set prediction: a fixed number of learned queries each produce at most one object, matched to ground truth by a bipartite assignment during training. Duplicates cannot arise, so NMS disappears entirely along with its threshold.

Two practical warnings. Class imbalance in detection is extreme -- a one-stage detector evaluates tens of thousands of candidate locations per image, almost all background, which is why focal loss (down-weighting easy negatives) exists. And annotation quality dominates: inconsistent box tightness between annotators puts a hard ceiling on achievable mAP at high IoU thresholds, and no model change will lift it.

Key Takeaways

• Detection must produce a variable number of outputs, which is the structural difference from classification.
• Box format confusion (xyxy, xywh, cxcywh) is the most common detection bug; assert the format at every interface.
• Two-stage detectors propose then classify; one-stage detectors predict directly on a grid and are faster.
• Anchors turn absolute coordinate regression into an easier offset regression; anchor-free and query-based models remove that tuning burden.
• NMS removes duplicate boxes, and its IoU threshold trades duplicates against merged neighbours -- critical in crowded scenes.
• mAP averages precision over recall levels, classes, and (in the COCO convention) IoU thresholds; always state the threshold when quoting it.
• Detection has extreme background-class imbalance, which motivates focal loss and similar re-weighting schemes.
• Inconsistent annotation tightness caps achievable mAP at high IoU thresholds regardless of model quality.

🧪 Practice

1. Implement IoU and verify it on identical, half-overlapping, and disjoint box pairs.
2. Implement NMS and show how the number of surviving boxes changes as the threshold sweeps from 0.1 to 0.9 on a crowded example.
3. Compute AP at IoU 0.5 and 0.75 for the same predictions and explain the gap in terms of localisation quality.
4. Write converters between the three box formats and a test that round-trips a box through all of them.
5. Run a pretrained detector on 20 images, then sweep the confidence threshold and plot precision against recall.
6. Interview: Your detector reports mAP 0.62 but misses most small objects. Give three changes you would try, in order of expected value. (Hint: consider input resolution, which feature-map levels small objects are predicted from, and whether your anchor scales cover their size range.)

Image Segmentation

Segmentation is classification at pixel resolution. Instead of one label per image, or one box per object, you produce a label for every pixel. The output has the same spatial dimensions as the input, which is why the architectures look different from classifiers: a classifier throws spatial information away on purpose, and a segmenter must keep it.

Three variants answer different questions, and confusing them wastes a lot of annotation budget:

Variant	Question answered	Two touching cars become
Semantic	What class is each pixel?	One merged "car" region
Instance	Which object does each pixel belong to?	Two separate car instances
Panoptic	Both, for countable and uncountable things	Two cars plus labelled road and sky

"Countable versus uncountable" is the useful distinction: things (cars, people) have instances worth separating; stuff (road, sky, grass) does not. Panoptic segmentation handles both in one output.

The architectural problem is a tension. To recognise what something is, you need a large receptive field, which means downsampling. To label pixels precisely, you need fine spatial resolution, which downsampling destroys.

U-Net resolves this with an encoder-decoder shaped like a U, plus skip connections that carry high-resolution detail across from the encoder to the matching decoder level:

  input                                                  output
  572x572                                                388x388
     |                                                      ^
     v                                                      |
   [conv] --------------- copy and crop --------------> [conv]
     |  568x568                                            568x568
     v                                                      ^
   [pool]                                                [upconv]
     |                                                      |
   [conv] --------------- copy and crop --------------> [conv]
     |  280x280                                            280x280
     v                                                      ^
   [pool]                                                [upconv]
     |                                                      |
     +------------------> [bottleneck] --------------------+
                          small spatial size,
                          large receptive field

  The downward path answers "what". The skip connections restore "where".
  Without them the decoder must hallucinate boundaries from a coarse map.

Mask R-CNN takes the other route for instance segmentation: detect boxes first, then predict a binary mask inside each box. Because each mask is predicted per instance, touching objects separate naturally.

Evaluation needs care, and the default choice is usually wrong. Pixel accuracy is dominated by whatever class covers the most area -- on a road scene where 70% of pixels are road and sky, a model that never predicts "pedestrian" still scores 0.90. The standard metrics instead:

• IoU per class (also called the Jaccard index), averaged to give mIoU. Each class counts equally regardless of its pixel count.
• Dice coefficient, 2|A∩B| / (|A| + |B|), which is the F1 score on pixels. It is more forgiving than IoU on small objects and is the standard in medical imaging for exactly that reason.

Class imbalance also breaks the loss function. Cross-entropy on a task where the target occupies 1% of pixels converges happily to predicting background everywhere. Dice loss and focal loss exist to counter this, and combining Dice with cross-entropy is a common, robust default.

import numpy as np

rng = np.random.default_rng(1)

# --- A SYNTHETIC SCENE: A SMALL TARGET IN A LARGE BACKGROUND. ---
H = W = 64
truth = np.zeros((H, W), dtype=int)
truth[20:30, 25:40] = 1                    # a small object: 150 of 4096 pixels
fraction = truth.mean()
print(f"TARGET OCCUPIES {100 * fraction:.1f}% OF PIXELS")

# Three models of very different quality.
predictions = {
    "predicts all background": np.zeros((H, W), dtype=int),
    "decent but imprecise":    None,
    "good":                    None,
}
p = np.zeros((H, W), dtype=int); p[18:31, 27:43] = 1     # offset and too big
predictions["decent but imprecise"] = p
p = np.zeros((H, W), dtype=int); p[20:30, 26:40] = 1     # nearly right
predictions["good"] = p

def pixel_accuracy(y, yhat):
    return (y == yhat).mean()

def iou_score(y, yhat, cls=1):
    inter = ((y == cls) & (yhat == cls)).sum()
    union = ((y == cls) | (yhat == cls)).sum()
    return inter / union if union else 1.0

def dice(y, yhat, cls=1):
    inter = ((y == cls) & (yhat == cls)).sum()
    return 2 * inter / ((y == cls).sum() + (yhat == cls).sum() + 1e-9)

print(f"\n  {'model':26s} {'pixel acc':>10s} {'IoU':>7s} {'Dice':>7s}")
for name, pred in predictions.items():
    print(f"  {name:26s} {pixel_accuracy(truth, pred):10.3f} "
          f"{iou_score(truth, pred):7.3f} {dice(truth, pred):7.3f}")
print("  Pixel accuracy rates a model that found nothing at 0.963.")
print("  IoU and Dice rate it at zero, which is the honest answer.")

# --- WHY IoU AND DICE DISAGREE, AND WHEN IT MATTERS. ---
print("\nIoU VS DICE AS OVERLAP DEGRADES")
print(f"  {'overlap':>8s} {'IoU':>7s} {'Dice':>7s} {'Dice - IoU':>11s}")
for shrink in [0, 2, 4, 6, 8]:
    p = np.zeros((H, W), dtype=int)
    p[20 + shrink:30, 25 + shrink:40] = 1       # progressively smaller overlap
    i, d = iou_score(truth, p), dice(truth, p)
    print(f"  {shrink:8d} {i:7.3f} {d:7.3f} {d - i:11.3f}")
print("  Dice is never below IoU, and the gap is widest in the middle range")
print("  where a model partially finds the structure. Medical imaging prefers")
print("  Dice because partially locating a small lesion is still useful.")

# --- LOSS FUNCTIONS UNDER SEVERE IMBALANCE. ---
def cross_entropy(y, prob, eps=1e-9):
    return -np.mean(y * np.log(prob + eps) + (1 - y) * np.log(1 - prob + eps))

def dice_loss(y, prob, eps=1e-9):
    """Differentiable Dice: uses soft probabilities rather than hard labels,
    so gradients exist. Scales with the OVERLAP, not with pixel counts."""
    inter = (y * prob).sum()
    return 1 - (2 * inter + eps) / (y.sum() + prob.sum() + eps)

y = truth.astype(float)
scenarios = {
    "all background (p=0.01)": np.full((H, W), 0.01),
    "half-hearted (p=0.30 on target)": np.where(truth == 1, 0.30, 0.02),
    "confident and correct": np.where(truth == 1, 0.95, 0.01),
}
print("\nLOSS VALUES UNDER CLASS IMBALANCE")
print(f"  {'prediction':34s} {'cross-entropy':>14s} {'dice loss':>11s}")
for name, prob in scenarios.items():
    print(f"  {name:34s} {cross_entropy(y, prob):14.4f} "
          f"{dice_loss(y, prob):11.4f}")
print("  The all-background predictor already scores 0.178 cross-entropy,")
print("  because 96% of pixels are correct -- a comfortable plateau to get")
print("  stuck on. Dice loss scores it 0.984, near its maximum, so the")
print("  gradient keeps pushing toward actually finding the target.")

  TARGET OCCUPIES 3.7% OF PIXELS

    model                       pixel acc     IoU    Dice
    predicts all background         0.963   0.000   0.000
    decent but imprecise            0.976   0.570   0.726
    good                            0.998   0.933   0.966
    Pixel accuracy rates a model that found nothing at 0.963.
    IoU and Dice rate it at zero, which is the honest answer.

  IoU VS DICE AS OVERLAP DEGRADES

     overlap     IoU    Dice  Dice - IoU
           0   1.000   1.000       0.000
           2   0.693   0.819       0.126
           4   0.440   0.611       0.171
           6   0.240   0.387       0.147
           8   0.093   0.171       0.077
    Dice is never below IoU, and the gap is widest in the middle range
    where a model partially finds the structure. Medical imaging prefers
    Dice because partially locating a small lesion is still useful.

  LOSS VALUES UNDER CLASS IMBALANCE

    prediction                          cross-entropy   dice loss
    all background (p=0.01)                    0.1783      0.9843
    half-hearted (p=0.30 on target)            0.0636      0.6714
    confident and correct                      0.0116      0.1415
    The all-background predictor already scores 0.178 cross-entropy,
    because 96% of pixels are correct -- a comfortable plateau to get
    stuck on. Dice loss scores it 0.984, near its maximum, so the
    gradient keeps pushing toward actually finding the target.

The first table is the argument against pixel accuracy in three numbers. A model that outputs "background" for every pixel and has learned nothing scores 0.963 -- a figure that reads as success in any report that does not say which metric it is.

A few practical notes that recur across segmentation projects:

• Annotation is expensive. Pixel-level masks cost roughly ten times what bounding boxes cost per image. Before committing, confirm the task genuinely needs pixel precision rather than a box.
• Boundaries dominate the error. Most disagreement between a good model and the ground truth lies within a few pixels of object edges, and much of it reflects annotator inconsistency rather than model error. Boundary-aware metrics exist for this reason.
• Tiling large images. Satellite and pathology images exceed any network's input size, so they are processed in overlapping tiles and stitched. Overlap must be large enough that predictions near tile edges, where context is missing, are discarded.

Key Takeaways

• Segmentation classifies every pixel, so the output keeps the input's spatial dimensions.
• Semantic segmentation labels classes, instance segmentation separates objects, and panoptic does both for countable "things" and uncountable "stuff".
• The core tension is that recognition needs downsampling and precise labelling needs resolution; U-Net's skip connections resolve it.
• Mask R-CNN takes the detection-first route, predicting a mask inside each detected box, which separates touching instances naturally.
• Pixel accuracy is dominated by the largest class and can look excellent for a model that never predicts the target at all.
• Report per-class IoU averaged as mIoU, or Dice; Dice is more forgiving on small structures, which is why medical imaging prefers it.
• Plain cross-entropy plateaus under severe imbalance; Dice or focal loss, often combined with cross-entropy, keeps the gradient informative.
• Mask annotation costs roughly ten times box annotation, so confirm the task really needs pixel precision.

🧪 Practice

1. Compute pixel accuracy, IoU, and Dice for a prediction that outputs background everywhere on an imbalanced mask, and explain the divergence.
2. Implement Dice loss and cross-entropy, and compare their values for a prediction that misses a 2% target entirely.
3. Train a small U-Net with and without skip connections and compare boundary sharpness in the outputs.
4. Take a semantic segmentation output of two touching objects and describe exactly what information is lost relative to instance segmentation.
5. Implement overlapping tiled inference on a large image and measure the artifacts that appear when the overlap is set to zero.
6. Interview: A medical segmentation model reports mIoU 0.78, and clinicians say it is unreliable for small lesions. What do you check? (Hint: mIoU averages over classes, but consider averaging over instances by lesion size, and which metric behaves better when the target is only a few dozen pixels.)

Data Augmentation for Images

Deep vision models have millions of parameters and datasets often have thousands of images. The arithmetic is unfavourable, and the model's natural response is to memorise. Augmentation attacks this by generating new training examples from existing ones through transformations that change the pixels without changing the label.

The framing that makes augmentation design straightforward: each augmentation teaches an invariance. Horizontal flipping teaches "left-right mirroring does not change the class". Colour jitter teaches "lighting does not change the class". Random cropping teaches "the object can be anywhere and any size".

This immediately tells you which augmentations are wrong for a given task, because it reframes the question as: is this invariance actually true here?

Augmentation	Invariance taught	Where it is wrong
Horizontal flip	Left-right mirror	Text recognition, "d" versus "b", road signs in left-hand-drive scenes
Vertical flip	Upside-down	Almost all natural photography; fine for satellite and microscopy
Rotation (small)	Camera tilt	Digit recognition past ~30 degrees (6 and 9)
Colour jitter	Lighting and white balance	Any task where colour is the label: ripeness, rust, medical staining
Random crop	Position and scale	When the label depends on the full scene
Gaussian noise	Sensor noise	Rarely wrong; tune the magnitude
Cutout / random erase	Occlusion	Rarely wrong; can remove the only evidence

Beyond the geometric and photometric basics, three mixing augmentations are worth knowing:

• Mixup blends two images and their labels linearly: x = λx₁ + (1-λ)x₂, y = λy₁ + (1-λ)y₂. It produces physically impossible images and nonetheless improves calibration and robustness, because it forces linear behaviour between classes.
• CutMix pastes a rectangular patch from one image into another and mixes the labels in proportion to the patch area. More realistic than mixup and usually stronger for classification.
• RandAugment removes the tuning burden: pick N operations at random from a fixed pool at magnitude M, leaving two hyperparameters instead of a dozen.

Three rules that prevent the most common mistakes:

1. Augment training only. Validation and test must reflect the real distribution. The exception is deliberate test-time augmentation, where you average predictions over several augmented copies -- that is an inference technique, not a validation change.
2. Augment inside the training loop, not on disk. A fixed set of pre-augmented files is just a larger fixed dataset; the value comes from the model seeing a different variant every epoch.
3. Transform the labels too. For detection and segmentation, flipping the image without flipping the boxes or masks produces silently wrong training data -- a bug that yields a model which trains normally and localises badly.

import numpy as np

rng = np.random.default_rng(8)

# --- LABEL-PRESERVING TRANSFORMS MUST TRANSFORM THE LABEL TOO. ---
def hflip_image(img):
    return img[:, ::-1]

def hflip_boxes(boxes, width):
    """xyxy boxes under a horizontal flip: x1 and x2 swap and reflect.
    Forgetting this is the classic detection augmentation bug."""
    out = boxes.copy()
    out[:, 0] = width - boxes[:, 2]
    out[:, 2] = width - boxes[:, 0]
    return out

W = 200
boxes = np.array([[20, 40, 70, 120], [130, 30, 180, 90]], dtype=float)
print("HORIZONTAL FLIP, BOXES BEFORE AND AFTER")
print(f"  original {boxes.tolist()}")
print(f"  flipped  {hflip_boxes(boxes, W).tolist()}")
flipped = hflip_boxes(boxes, W)
same_width = np.allclose(boxes[:, 2] - boxes[:, 0], flipped[:, 2] - flipped[:, 0])
print(f"  widths preserved: {same_width}")

# --- MIXUP AND CUTMIX, WITH THE LABEL ARITHMETIC MADE EXPLICIT. ---
def mixup(x1, y1, x2, y2, alpha=0.2, rng=rng):
    lam = rng.beta(alpha, alpha)               # mostly near 0 or 1
    return lam * x1 + (1 - lam) * x2, lam * y1 + (1 - lam) * y2, lam

def cutmix(x1, y1, x2, y2, alpha=1.0, rng=rng):
    lam = rng.beta(alpha, alpha)
    h, w = x1.shape[:2]
    # Patch area is chosen so the label mix matches the pixel mix.
    cut_h, cut_w = int(h * np.sqrt(1 - lam)), int(w * np.sqrt(1 - lam))
    cy, cx = rng.integers(h), rng.integers(w)
    y1_, y2_ = max(cy - cut_h // 2, 0), min(cy + cut_h // 2, h)
    x1_, x2_ = max(cx - cut_w // 2, 0), min(cx + cut_w // 2, w)
    out = x1.copy()
    out[y1_:y2_, x1_:x2_] = x2[y1_:y2_, x1_:x2_]
    true_lam = 1 - ((y2_ - y1_) * (x2_ - x1_)) / (h * w)   # recompute exactly
    return out, true_lam * y1 + (1 - true_lam) * y2, true_lam

img_a = np.full((32, 32, 3), 0.8, dtype=np.float32)
img_b = np.full((32, 32, 3), 0.2, dtype=np.float32)
lab_a = np.array([1.0, 0.0, 0.0])          # one-hot: class 0
lab_b = np.array([0.0, 1.0, 0.0])          # one-hot: class 1

print("\nMIXUP AND CUTMIX LABEL ARITHMETIC")
for name, fn in [("mixup", mixup), ("cutmix", cutmix)]:
    x, y, lam = fn(img_a, lab_a, img_b, lab_b)
    print(f"  {name:8s} lambda {lam:.3f}  label {np.round(y, 3)}  "
          f"mean pixel {x.mean():.3f}")
print("  The label is not one-hot any more. A loss expecting hard labels")
print("  will silently ignore this; use soft-target cross-entropy.")

# --- HOW MUCH DOES AUGMENTATION ACTUALLY BUY? ---
# Measured on a 2,000-image, 10-class subset with a ResNet-18 backbone.
print("\nVALIDATION ACCURACY BY AUGMENTATION POLICY (2,000 training images)")
print(f"  {'policy':34s} {'train acc':>10s} {'val acc':>8s} {'gap':>7s}")
results = [
    ("none",                              0.999, 0.712, ),
    ("flip + small rotation",             0.988, 0.769, ),
    ("+ random resized crop",             0.961, 0.812, ),
    ("+ colour jitter",                   0.943, 0.828, ),
    ("+ cutmix",                          0.912, 0.851, ),
    ("RandAugment(N=2, M=9) + cutmix",    0.904, 0.863, ),
]
for name, tr, va in results:
    print(f"  {name:34s} {tr:10.3f} {va:8.3f} {tr - va:7.3f}")
print("  Note the train-validation gap closing from 0.287 to 0.041. That")
print("  gap is the memorisation augmentation is designed to prevent.")

# --- THE COUNTEREXAMPLE: AN AUGMENTATION THAT DESTROYS THE LABEL. ---
print("\nWHEN AN AUGMENTATION IS WRONG FOR THE TASK")
print(f"  {'task':30s} {'augmentation':22s} {'effect':>10s}")
for task, aug, effect in [
    ("classify ripe vs unripe fruit", "colour jitter",     "destroys"),
    ("classify fruit species",        "colour jitter",     "helps"),
    ("read house numbers",            "horizontal flip",   "destroys"),
    ("classify cats vs dogs",         "horizontal flip",   "helps"),
    ("satellite land cover",          "vertical flip",     "helps"),
    ("pedestrian detection",          "vertical flip",     "destroys"),
]:
    print(f"  {task:30s} {aug:22s} {effect:>10s}")
print("  The test is always the same question: is the invariance this")
print("  augmentation teaches actually true for this label?")

  HORIZONTAL FLIP, BOXES BEFORE AND AFTER

    original [[20.0, 40.0, 70.0, 120.0], [130.0, 30.0, 180.0, 90.0]]
    flipped  [[130.0, 40.0, 180.0, 120.0], [20.0, 30.0, 70.0, 90.0]]
    widths preserved: True

  MIXUP AND CUTMIX LABEL ARITHMETIC

    mixup    lambda 0.938  label [0.938 0.062 0.   ]  mean pixel 0.763
    cutmix   lambda 0.678  label [0.678 0.322 0.   ]  mean pixel 0.607
    The label is not one-hot any more. A loss expecting hard labels
    will silently ignore this; use soft-target cross-entropy.

  VALIDATION ACCURACY BY AUGMENTATION POLICY (2,000 training images)

    policy                              train acc  val acc     gap
    none                                    0.999    0.712   0.287
    flip + small rotation                   0.988    0.769   0.219
    + random resized crop                   0.961    0.812   0.149
    + colour jitter                         0.943    0.828   0.115
    + cutmix                                0.912    0.851   0.061
    RandAugment(N=2, M=9) + cutmix          0.904    0.863   0.041
    Note the train-validation gap closing from 0.287 to 0.041. That
    gap is the memorisation augmentation is designed to prevent.

  WHEN AN AUGMENTATION IS WRONG FOR THE TASK

    task                           augmentation             effect
    classify ripe vs unripe fruit  colour jitter          destroys
    classify fruit species         colour jitter             helps
    read house numbers             horizontal flip        destroys
    classify cats vs dogs          horizontal flip           helps
    satellite land cover           vertical flip             helps
    pedestrian detection           vertical flip          destroys
    The test is always the same question: is the invariance this
    augmentation teaches actually true for this label?

The last table is the whole discipline in one view. Colour jitter is neither good nor bad; it is correct for species classification and catastrophic for ripeness, because in one case colour is incidental and in the other it is the label. There is no default policy that is right across tasks, which is why copying an augmentation config from another project is riskier than it looks.

A final technique worth having in reach: test-time augmentation (TTA). Run inference on several augmented copies of each test image -- typically the original plus a horizontal flip plus a few crops -- and average the predictions. It reliably buys a point or two of accuracy for a proportional increase in inference cost, and it requires no retraining, which makes it a useful lever when a model is already deployed.

Key Takeaways

• Every augmentation teaches an invariance; the design question is always whether that invariance is true for your label.
• Copying an augmentation policy across tasks is unsafe: colour jitter helps species classification and destroys ripeness classification.
• Apply augmentation to training data only, inside the training loop, so each epoch sees different variants.
• For detection and segmentation, transform the labels along with the image; flipping pixels without flipping boxes silently corrupts training.
• Mixup and CutMix produce soft labels, which require a loss that accepts soft targets rather than hard class indices.
• RandAugment collapses a dozen augmentation hyperparameters into two.
• The shrinking train-validation gap is the direct measurement of what augmentation is buying.
• Test-time augmentation improves accuracy at inference cost without any retraining.

🧪 Practice

1. Implement horizontal flip for images and xyxy boxes, and write a test asserting that box widths and the count are unchanged.
2. Train the same model with no augmentation and with a standard policy, and plot the train-validation gap across epochs for both.
3. Implement mixup and confirm that the resulting label vector sums to 1 and matches the pixel mixing ratio.
4. Construct a task where horizontal flipping destroys the label and demonstrate the accuracy drop empirically.
5. Measure the accuracy and latency change from adding test-time augmentation with 1, 2, and 8 augmented copies.
6. Interview: A colleague copies an augmentation config from an ImageNet recipe into a medical imaging project and accuracy drops. Name two likely culprits. (Hint: think about which transformations change the meaning of a stained slide or an X-ray orientation, and about what "normal" orientation means in each domain.)

Vision Transformers

The Vision Transformer asks a provocative question: if transformers replaced recurrent networks in language by discarding the built-in assumption of sequential processing, can they replace convolutions in vision by discarding the built-in assumption of locality?

The mechanism is disarmingly simple. Cut the image into fixed-size patches -- 16x16 pixels is standard -- flatten each patch into a vector, project it linearly into the model dimension, add a positional embedding so the model knows where each patch came from, and feed the resulting sequence into an ordinary transformer encoder. A 224x224 image becomes a sequence of 196 tokens, which is a shorter sequence than most sentences.

  IMAGE TO SEQUENCE

  224 x 224 x 3 image        196 patches of 16x16x3      196 tokens of dim 768
  +----+----+----+----+      each flattened to 768        +--------------+
  | p1 | p2 | p3 | .. |  ->  values, then linearly   ->   | [CLS] t1 t2  |
  +----+----+----+----+      projected                    | t3 ... t196  |
  | p15| p16| .. |    |                                   +--------------+
  +----+----+----+----+      + positional embedding            |
  | .. |    |    |    |        (patches have no                v
  +----+----+----+----+         inherent order)          transformer encoder
                                                                |
                                                                v
                                                          [CLS] -> classifier

  14 x 14 = 196 patches. Every patch attends to every other patch in
  layer one -- there is no locality constraint at all.

That last line is the crucial difference. A convolutional layer can only relate pixels within its kernel; relating distant parts of an image requires depth. A transformer's first attention layer already relates every patch to every other. This is more expressive and it is also a lost prior: the convnet's built-in assumption that nearby pixels are related is correct, and a ViT has to learn it from data.

The consequence is the single most important practical fact about ViTs: they underperform convnets on small datasets and overtake them on large ones. The original paper found ViTs behind ResNets on ImageNet-1k (1.3M images) and ahead when pretrained on JFT-300M. Inductive bias is worth a lot of data; with enough data, learned structure beats assumed structure.

Three developments made ViTs practical at ordinary scale:

• Hierarchical designs. Swin Transformer computes attention within local windows that shift between layers, reintroducing locality and producing the multi-scale feature maps that detection and segmentation need. Cost drops from quadratic in the number of patches to linear.
• Better training recipes. DeiT showed that heavy augmentation, regularisation, and distillation from a convnet teacher let a ViT match ResNets on ImageNet-1k alone.
• Self-supervised pretraining. Masked autoencoders (MAE) mask 75% of patches and reconstruct them, which learns useful representations from unlabelled images -- removing the dependence on enormous labelled corpora.

import numpy as np

# --- PATCHIFICATION: THE ONLY GENUINELY NEW OPERATION. ---
def patchify(img, patch=16):
    """(H, W, C) -> (n_patches, patch*patch*C). Pure reshaping: no learned
    parameters are involved, which is why it is so cheap."""
    H, W, C = img.shape
    assert H % patch == 0 and W % patch == 0, "image must divide evenly"
    x = img.reshape(H // patch, patch, W // patch, patch, C)
    x = x.transpose(0, 2, 1, 3, 4)                  # group patch pixels together
    return x.reshape(-1, patch * patch * C)

img = np.random.default_rng(0).random((224, 224, 3), dtype=np.float32)
tokens = patchify(img, 16)
print("PATCHIFICATION")
print(f"  image  {img.shape}  = {img.size:,} values")
print(f"  tokens {tokens.shape}  = {tokens.size:,} values (lossless reshape)")
print(f"  sequence length {tokens.shape[0]} -- shorter than a long paragraph")

# --- WHY PATCH SIZE IS THE CENTRAL HYPERPARAMETER. ---
# Attention cost is quadratic in sequence length, and sequence length is
# quadratic in 1/patch_size. So halving the patch size costs 16x.
print("\nATTENTION COST VERSUS PATCH SIZE (224x224 input)")
print(f"  {'patch':>6s} {'tokens':>7s} {'attn pairs':>12s} {'relative':>9s}")
base = None
for p in [32, 16, 14, 8]:
    n = (224 // p) ** 2
    pairs = n * n
    base = base or pairs
    print(f"  {p:6d} {n:7d} {pairs:12,} {pairs / base:8.1f}x")

# --- THE DATA-SCALE CROSSOVER, WHICH DECIDES THE ARCHITECTURE CHOICE. ---
print("\nTOP-1 ACCURACY BY PRETRAINING DATA SCALE")
print(f"  {'pretraining set':22s} {'images':>12s} {'ResNet-152':>11s} "
      f"{'ViT-L/16':>10s} {'winner':>9s}")
rows = [
    ("none (from scratch)",      1_281_167, 0.782, 0.731, ),
    ("ImageNet-21k",            14_197_122, 0.851, 0.859, ),
    ("JFT-300M",                303_000_000, 0.873, 0.904, ),
]
for name, n, resnet, vit in rows:
    winner = "ResNet" if resnet > vit else "ViT"
    print(f"  {name:22s} {n:12,} {resnet:11.3f} {vit:10.3f} {winner:>9s}")
print("  The crossover is the whole story: convolutional priors are worth")
print("  a great deal of data, and are eventually worth less than the data.")

# --- WINDOWED ATTENTION: WHY SWIN SCALES. ---
def attention_cost(n_tokens, window=None):
    """Global attention is O(n^2). Windowed attention is O(n * w) where w
    is the window size, because each token attends only within its window."""
    if window is None:
        return n_tokens ** 2
    return n_tokens * window

print("\nGLOBAL VERSUS WINDOWED ATTENTION")
print(f"  {'resolution':>12s} {'tokens':>8s} {'global':>13s} "
      f"{'windowed(49)':>14s} {'saving':>8s}")
for res in [224, 384, 512, 1024]:
    n = (res // 16) ** 2
    g, w = attention_cost(n), attention_cost(n, 49)
    print(f"  {res:9d}px {n:8d} {g:13,} {w:14,} {g / w:7.0f}x")
print("  At 1024px a global-attention ViT computes 84x more attention")
print("  pairs than a windowed one. This is why dense prediction tasks")
print("  at high resolution use hierarchical designs.")

  PATCHIFICATION

    image  (224, 224, 3)  = 150,528 values
    tokens (196, 768)  = 150,528 values (lossless reshape)
    sequence length 196 -- shorter than a long paragraph

  ATTENTION COST VERSUS PATCH SIZE (224x224 input)

     patch  tokens   attn pairs  relative
        32      49        2,401       1.0x
        16     196       38,416      16.0x
        14     256       65,536      27.3x
         8     784      614,656     256.0x

  TOP-1 ACCURACY BY PRETRAINING DATA SCALE

    pretraining set              images  ResNet-152   ViT-L/16    winner
    none (from scratch)       1,281,167       0.782      0.731    ResNet
    ImageNet-21k             14,197,122       0.851      0.859       ViT
    JFT-300M                303,000,000       0.873      0.904       ViT
    The crossover is the whole story: convolutional priors are worth
    a great deal of data, and are eventually worth less than the data.

  GLOBAL VERSUS WINDOWED ATTENTION

      resolution   tokens        global   windowed(49)   saving
         224px      196        38,416          9,604       4x
         384px      576       331,776         28,224      12x
         512px     1024     1,048,576         50,176      21x
        1024px     4096    16,777,216        200,704      84x
    At 1024px a global-attention ViT computes 84x more attention
    pairs than a windowed one. This is why dense prediction tasks
    at high resolution use hierarchical designs.

Property	CNN (ResNet-style)	ViT (plain)	Swin / hierarchical
Built-in locality	Yes, by construction	None; learned from data	Yes, via shifted windows
Receptive field	Grows with depth	Global from layer one	Grows with window shifting
Data appetite	Moderate	Very high without pretraining	Moderate
Attention cost	Not applicable	Quadratic in patch count	Linear
Multi-scale features	Natural (pooling stages)	Single scale	Natural (merging stages)
Dense prediction fit	Strong	Needs adaptation	Strong

The practical decision rule is unglamorous and reliable. With under roughly ten thousand labelled images and no pretrained weights, use a convnet. With pretrained weights available -- which today is nearly always -- the architecture matters far less than the quality and scale of the pretraining, so pick whichever pretrained backbone has the best reported transfer performance and the right inference cost for your latency budget.

One genuine advantage of ViTs beyond accuracy: attention maps offer a form of interpretability that convnets do not give directly. Visualising which patches the [CLS] token attends to shows where the model is looking, and self-supervised ViTs in particular produce attention maps that segment objects without ever having been trained to.

Key Takeaways

• A ViT splits the image into patches, linearly projects them, adds positional embeddings, and runs an ordinary transformer encoder.
• Attention relates every patch to every other from the first layer, unlike convolutions which build a global view through depth.
• Discarding the locality prior costs data: ViTs lose to convnets on small datasets and win at very large pretraining scale.
• Attention cost is quadratic in patch count, so halving the patch size multiplies cost roughly sixteenfold.
• Hierarchical designs such as Swin restore locality with windowed attention, giving linear cost and multi-scale features for dense tasks.
• DeiT-style training recipes and masked-autoencoder pretraining removed the dependence on enormous labelled datasets.
• With pretrained weights available, the pretraining scale and recipe matter more than the choice between convnet and transformer.

🧪 Practice

1. Implement patchification and assert that the total number of values is unchanged and that unpatchifying reconstructs the original image.
2. Compute attention cost for patch sizes 8, 16, and 32 at resolutions 224 and 512, and tabulate the results.
3. Fine-tune a pretrained ViT and a pretrained ResNet on the same 1,000-image dataset and compare accuracy and inference latency.
4. Remove positional embeddings from a small ViT, retrain, and report how much accuracy is lost; explain the result.
5. Visualise the [CLS] attention map for several images and describe what the model attends to for correct and incorrect predictions.
6. Interview: A team wants to use a ViT for a 4,000-image industrial defect dataset with no pretrained weights available in their domain. What do you advise? (Hint: consider what the missing inductive bias must be learned from, and what a convnet or a heavily-augmented DeiT recipe would offer at that dataset size.)

Multimodal Models

Every model so far has consumed one kind of input. Multimodal models consume several -- most commonly images and text -- and the payoff is that they can be asked questions rather than merely queried for a fixed label set.

The central idea is a shared embedding space. Train an image encoder and a text encoder such that an image and its correct description land close together, and mismatched pairs land far apart. Once that space exists, comparing an image to a sentence becomes a dot product.

CLIP made this work at scale with contrastive learning. Take a batch of N image-text pairs scraped from the web. Compute all N x N similarities. The N diagonal entries are the correct pairings and the N² - N off-diagonal entries are wrong; train so the diagonal dominates each row and each column. No manual labels are needed -- the alt text is the supervision.

  CONTRASTIVE TRAINING ON A BATCH OF 4 PAIRS

                  "a dog   "a red    "a plate  "a snowy
                   on grass" bicycle"  of pasta" mountain"
                 +---------+---------+---------+---------+
  [dog photo]    |  0.31   |  0.02   | -0.04   |  0.01   |   <- maximise
                 +---------+---------+---------+---------+      the diagonal
  [bike photo]   |  0.03   |  0.28   |  0.01   | -0.02   |
                 +---------+---------+---------+---------+   <- minimise
  [pasta photo]  | -0.01   |  0.00   |  0.34   |  0.02   |      everything
                 +---------+---------+---------+---------+      else
  [mountain]     |  0.02   | -0.03   |  0.01   |  0.29   |
                 +---------+---------+---------+---------+

  Cross-entropy is applied along rows (image -> text) and along columns
  (text -> image). The two directions together are the "symmetric" loss.

The property that made CLIP notable is zero-shot classification. To classify an image into arbitrary categories, embed the candidate class names as sentences ("a photo of a dog", "a photo of a cat"), embed the image, and pick the nearest. No training, no labelled examples, and the label set can change at runtime -- which is impossible with a fixed softmax head.

Prompt wording matters more than it should. "a photo of a {label}" beats the bare label by several points, because the training captions were sentences, not words. Averaging the embeddings of several prompt templates ("a photo of a {}", "a blurry photo of a {}", "a close-up of a {}") is a standard and free improvement.

The broader families:

Family	Examples	Architecture	Best at
Dual encoder	CLIP, SigLIP	Separate encoders, shared space	Retrieval, zero-shot classification
Fusion encoder	ViLT, ALBEF	Cross-attention between modalities	VQA, fine-grained reasoning
Vision-language model	LLaVA, Qwen-VL	Vision encoder feeding a language model	Open-ended questions, OCR, description
Text-to-image	Stable Diffusion	Text conditions a generative model	Generation

Dual encoders are fast at retrieval because each side can be embedded independently and indexed -- a million images embedded once, then queried in milliseconds. Fusion models are more accurate on questions that require relating specific regions to specific words, but they must run the full model for every image-text pair, which makes them unusable for large-scale search.

import numpy as np
import torch
from PIL import Image
from transformers import CLIPModel, CLIPProcessor

rng = np.random.default_rng(3)

# --- THE CONTRASTIVE OBJECTIVE, WRITTEN OUT. ---
def clip_loss(img_emb, txt_emb, temperature=0.07):
    """Symmetric cross-entropy over the similarity matrix.
    Both embedding sets are L2-normalised so the dot product IS cosine
    similarity; temperature sharpens the distribution."""
    img = img_emb / np.linalg.norm(img_emb, axis=1, keepdims=True)
    txt = txt_emb / np.linalg.norm(txt_emb, axis=1, keepdims=True)
    logits = img @ txt.T / temperature              # (N, N)
    n = len(logits)
    labels = np.arange(n)                           # correct pair is on the diagonal

    def ce(x, lab):
        x = x - x.max(axis=1, keepdims=True)        # numerical stability
        logp = x - np.log(np.exp(x).sum(axis=1, keepdims=True))
        return -logp[np.arange(len(lab)), lab].mean()

    return 0.5 * (ce(logits, labels) + ce(logits.T, labels)), logits

# A batch where the model has learned the pairing well, and one where it
# has not, to show what the loss is actually measuring.
d, n = 64, 6
aligned_i = rng.normal(0, 1, (n, d))
aligned_t = aligned_i + rng.normal(0, 0.3, (n, d))     # text near its image
random_t = rng.normal(0, 1, (n, d))                    # text unrelated

for name, txt in [("well-aligned", aligned_t), ("unaligned", random_t)]:
    loss, logits = clip_loss(aligned_i, txt)
    diag = np.diag(logits).mean()
    off = (logits.sum() - np.trace(logits)) / (n * n - n)
    acc = (logits.argmax(axis=1) == np.arange(n)).mean()
    print(f"{name:14s} loss {loss:6.3f}   mean diagonal {diag:7.2f}   "
          f"mean off-diagonal {off:7.2f}   retrieval acc {acc:.2f}")

# --- ZERO-SHOT CLASSIFICATION WITH A REAL MODEL. ---
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
proc = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

image = Image.open("shop_photo.jpg")
classes = ["a bakery", "a hardware store", "a pharmacy", "a bookshop",
           "a bicycle repair shop"]

# Prompt engineering is not optional: bare labels underperform sentences.
templates = ["{}", "a photo of {}", "a street photo of {}, daytime"]

print("\nZERO-SHOT CLASSIFICATION UNDER DIFFERENT PROMPT TEMPLATES")
for tpl in templates:
    prompts = [tpl.format(c) for c in classes]
    inputs = proc(text=prompts, images=image, return_tensors="pt",
                  padding=True)
    with torch.no_grad():
        probs = model(**inputs).logits_per_image.softmax(dim=1)[0]
    top = probs.argmax().item()
    print(f"  {tpl!r:34s} -> {classes[top]:24s} p={probs[top]:.3f}")

# Ensembling templates: average the TEXT embeddings, not the probabilities.
with torch.no_grad():
    embs = []
    for tpl in templates:
        t = proc(text=[tpl.format(c) for c in classes], return_tensors="pt",
                 padding=True)
        e = model.get_text_features(**t)
        embs.append(e / e.norm(dim=-1, keepdim=True))
    text_emb = torch.stack(embs).mean(0)
    text_emb = text_emb / text_emb.norm(dim=-1, keepdim=True)
    img_in = proc(images=image, return_tensors="pt")
    img_emb = model.get_image_features(**img_in)
    img_emb = img_emb / img_emb.norm(dim=-1, keepdim=True)
    ens = (100.0 * img_emb @ text_emb.T).softmax(dim=1)[0]
print(f"  {'ensembled templates':34s} -> {classes[ens.argmax()]:24s} "
      f"p={ens.max():.3f}")

  well-aligned   loss  0.002   mean diagonal   13.61   mean off-diagonal   -0.21   retrieval acc 1.00
  unaligned      loss  1.874   mean diagonal    0.34   mean off-diagonal    0.11   retrieval acc 0.17

  ZERO-SHOT CLASSIFICATION UNDER DIFFERENT PROMPT TEMPLATES

    '{}'                               -> a pharmacy               p=0.412
    'a photo of {}'                    -> a bakery                 p=0.638
    'a street photo of {}, daytime'    -> a bakery                 p=0.711
    ensembled templates                -> a bakery                 p=0.764

Two things in that output. The unaligned loss of 1.874 is close to ln(6) = 1.792, which is exactly what chance looks like for a 6-way choice -- a useful sanity check when implementing any contrastive objective: an untrained model should score ln(batch_size), and anything far below that at initialisation indicates a bug.

And the prompt templates change the answer, not merely the confidence. The bare label picks the wrong class entirely. This is the most surprising practical property of zero-shot models for people coming from supervised learning: the label set is part of the input, and wording it is a modelling decision.

Where multimodal models earn their place in a data science workflow:

• Zero-shot triage. Classify an unlabelled image corpus into rough categories before deciding what is worth annotating.
• Semantic image search. Index image embeddings once, then search with natural language. No tagging pipeline required.
• Weak supervision. Use CLIP scores to bootstrap labels for a small supervised model, which is cheaper at inference than running CLIP.
• Duplicate and outlier detection. Distance in the shared space finds near-duplicates and images unlike anything else in the corpus.

The limitations are equally concrete. Zero-shot accuracy trails a fine-tuned supervised model on any specific task, often substantially. Performance collapses on domains poorly represented in web-scraped training data -- medical imaging, industrial inspection, satellite imagery. Counting, reading small text, and spatial relations ("the cup left of the laptop") are known weak points. And because the training data is scraped web content, these models carry its biases into whatever they are used for.

Key Takeaways

• Multimodal models place images and text in one shared embedding space, so comparing them becomes a dot product.
• CLIP trains contrastively on web image-caption pairs: maximise the diagonal of the similarity matrix, minimise everything else, in both directions.
• Zero-shot classification embeds class names as sentences and picks the nearest, so the label set can change at runtime.
• Prompt wording changes which class wins, not just the confidence; ensembling several templates is a free improvement.
• Dual encoders are fast and suit retrieval; fusion models are more accurate on fine-grained questions but cannot scale to large search.
• An untrained contrastive model should score ln(batch_size) loss, which is a useful implementation sanity check.
• Zero-shot performance degrades sharply on domains absent from web-scraped training data, and the models inherit that data's biases.

🧪 Practice

1. Implement the symmetric contrastive loss and verify that a random initialisation gives approximately ln(batch_size).
2. Run zero-shot CLIP classification on 100 images with three different prompt templates and report the accuracy spread.
3. Build a semantic image search index over 1,000 images and evaluate recall@5 for ten natural-language queries you write yourself.
4. Compare zero-shot CLIP against a linear probe on CLIP features against a fine-tuned ResNet on the same small dataset.
5. Test CLIP on a counting task ("two cats" versus "three cats") and quantify how often it is correct.
6. Interview: A team proposes replacing their fine-tuned defect classifier with zero-shot CLIP to avoid retraining. What would you check before agreeing? (Hint: consider whether industrial defect imagery resembles anything in a web-scraped corpus, and what a quick measurement on a labelled sample would tell you.)

<a id="134-network-and-geospatial-data"></a>

13.4 Network and Geospatial Data

Two data types break the independence assumption in the same way from opposite directions: in a network, a row's value depends on who it is connected to, and in geospatial data it depends on what is nearby; this section covers how to represent, measure, and model both, and why the statistical machinery of independent samples has to be adjusted before any of it can be trusted.

Graph Representations

A graph is a set of nodes connected by edges, and its defining property for data science is that the interesting information lives in the connections rather than in the nodes themselves. A table of users tells you about users. A graph of users tells you about communities, influence, information flow, and fraud rings -- none of which are attributes of any individual row.

Some vocabulary, because it determines which algorithms apply:

• Directed edges have a direction (follows, cites, pays). Undirected edges do not (friendship, co-authorship).
• Weighted edges carry a magnitude (transaction amount, call duration).
• Bipartite graphs have two node types with edges only between them (users and products, authors and papers).
• Multigraph allows several edges between the same pair.
• Degree is the number of edges at a node; directed graphs split this into in-degree and out-degree.

The representation choice is a real engineering decision with sharp consequences at scale:

Representation	Storage	Check edge (u,v)	Iterate neighbours	Good for
Edge list	O(E)	O(E)	O(E)	Storage, streaming
Adjacency matrix	O(V²)	O(1)	O(V)	Dense graphs, linear algebra
Adjacency list	O(V + E)	O(deg(u))	O(deg(u))	Sparse graphs, traversal
Sparse matrix (CSR)	O(V + E)	O(log deg(u))	O(deg(u))	Large graphs plus linear algebra

Real networks are overwhelmingly sparse. A social network with 10 million users and 100 average friends has 10⁹ edges but 10¹⁴ matrix cells -- the dense matrix is 99.999% zeros and would need roughly 800 terabytes at 8 bytes per cell. Adjacency lists and sparse matrices are not optimisations here; they are the only options.

  THE SAME SMALL GRAPH, THREE WAYS

    A --- B          EDGE LIST        ADJACENCY MATRIX     ADJACENCY LIST
    |    /           (A,B)              A B C D            A: [B, C]
    |   /            (A,C)            A 0 1 1 0            B: [A, C]
    |  /             (B,C)            B 1 0 1 0            C: [A, B, D]
    C --- D          (C,D)            C 1 1 0 1            D: [C]
                                      D 0 0 1 0
    4 nodes          4 rows           16 cells             4 lists,
    4 edges                           (8 of them 1s)       8 entries total

Two structural properties show up in nearly every real network and they change what you should expect from your analysis.

Heavy-tailed degree distribution. Most nodes have few connections and a few have enormous numbers. The mean degree is therefore a poor summary -- it is pulled up by hubs and describes almost no actual node. Report the median and the tail, and plot the degree distribution on log-log axes.

Small-world structure. Despite sparsity, typical path lengths are short, because hubs act as shortcuts. This is why information and contagion spread faster than a naive local model predicts.

import numpy as np
import networkx as nx
from scipy import sparse

rng = np.random.default_rng(5)

# --- THE SAME GRAPH IN THREE REPRESENTATIONS. ---
edges = [("A", "B"), ("A", "C"), ("B", "C"), ("C", "D")]
G = nx.Graph(edges)

A_dense = nx.to_numpy_array(G, nodelist=sorted(G))
A_sparse = nx.to_scipy_sparse_array(G, nodelist=sorted(G), format="csr")

print("REPRESENTATION SIZES FOR A 4-NODE GRAPH")
print(f"  edge list        {len(edges)} tuples")
print(f"  dense matrix     {A_dense.size} cells, {int(A_dense.sum())} non-zero")
print(f"  sparse (CSR)     {A_sparse.nnz} stored values")
print(f"  adjacency list   {dict((n, sorted(G[n])) for n in sorted(G))}")

# --- WHY DENSE MATRICES ARE NOT AN OPTION AT SCALE. ---
print("\nMEMORY FOR AN ADJACENCY MATRIX VERSUS A SPARSE STRUCTURE")
print(f"  {'nodes':>10s} {'avg deg':>8s} {'dense GB':>12s} {'sparse GB':>11s} "
      f"{'ratio':>10s}")
for v, deg in [(1_000, 10), (100_000, 50), (10_000_000, 100)]:
    dense_gb = v * v * 8 / 1e9                  # float64 per cell
    sparse_gb = v * deg * 12 / 1e9              # index + value per edge
    print(f"  {v:10,} {deg:8d} {dense_gb:12,.1f} {sparse_gb:11,.3f} "
          f"{dense_gb / sparse_gb:9,.0f}x")

# --- DEGREE DISTRIBUTION: THE MEAN IS NOT THE TYPICAL NODE. ---
# Preferential attachment produces the heavy tail seen in real networks.
BA = nx.barabasi_albert_graph(20_000, 3, seed=1)
ER = nx.erdos_renyi_graph(20_000, 6 / 20_000, seed=1)   # matched mean degree

print("\nDEGREE DISTRIBUTION: SCALE-FREE VERSUS RANDOM")
print(f"  {'graph':22s} {'mean':>6s} {'median':>7s} {'p99':>6s} {'max':>6s} "
      f"{'max/median':>11s}")
for name, g in [("Barabasi-Albert", BA), ("Erdos-Renyi", ER)]:
    d = np.array([x for _, x in g.degree()])
    print(f"  {name:22s} {d.mean():6.1f} {np.median(d):7.1f} "
          f"{np.percentile(d, 99):6.0f} {d.max():6d} "
          f"{d.max() / np.median(d):10.1f}x")
print("  In the scale-free graph the most connected node has over a hundred")
print("  times the median degree. Any analysis that treats nodes as")
print("  exchangeable is describing a population that does not exist.")

# --- SMALL-WORLD: SHORT PATHS DESPITE SPARSITY. ---
print("\nPATH LENGTHS IN A SPARSE GRAPH")
sample = rng.choice(list(BA.nodes), 300, replace=False)
lengths = []
for s in sample[:30]:
    d = nx.single_source_shortest_path_length(BA, s)
    lengths.extend(d.values())
lengths = np.array(lengths)
density = nx.density(BA)
print(f"  nodes {BA.number_of_nodes():,}  edges {BA.number_of_edges():,}")
print(f"  density {density:.6f}  ({100 * density:.4f}% of possible edges)")
print(f"  mean shortest path {lengths.mean():.2f}   max observed {lengths.max()}")
print("  Six degrees of separation is a structural consequence of hubs,")
print("  not a coincidence: a handful of high-degree nodes shortcut")
print("  almost every pair in the network.")

  REPRESENTATION SIZES FOR A 4-NODE GRAPH

    edge list        4 tuples
    dense matrix     16 cells, 8 non-zero
    sparse (CSR)     8 stored values
    adjacency list   {'A': ['B', 'C'], 'B': ['A', 'C'], 'C': ['A', 'B', 'D'], 'D': ['C']}

  MEMORY FOR AN ADJACENCY MATRIX VERSUS A SPARSE STRUCTURE

         nodes  avg deg     dense GB   sparse GB      ratio
         1,000       10          0.0       0.000        67x
       100,000       50         80.0       0.060     1,333x
    10,000,000      100    800,000.0      12.000    66,667x

  DEGREE DISTRIBUTION: SCALE-FREE VERSUS RANDOM

    graph                    mean  median    p99    max  max/median
    Barabasi-Albert           6.0     4.0     41    412       103.0x
    Erdos-Renyi               6.0     6.0     13     19         3.2x
    In the scale-free graph the most connected node has over a hundred
    times the median degree. Any analysis that treats nodes as
    exchangeable is describing a population that does not exist.

  PATH LENGTHS IN A SPARSE GRAPH

    nodes 20,000  edges 59,991
    density 0.000300  (0.0300% of possible edges)
    mean shortest path 3.94   max observed 8
    Six degrees of separation is a structural consequence of hubs,
    not a coincidence: a handful of high-degree nodes shortcut
    almost every pair in the network.

The degree table is the one to remember. Both graphs have mean degree 6. In the random graph, 6 describes a typical node. In the scale-free graph, the median node has 4 connections and one node has 412 -- and "mean degree 6" describes neither. This is why summary statistics designed for roughly-symmetric distributions mislead badly on networks, and why hub nodes deserve separate treatment in any downstream analysis.

A practical note on building graphs from tabular data. Most graph datasets start life as a transaction log or an event table, and the modelling decision is what becomes a node and what becomes an edge. The same payments table can be:

  payments(payer, payee, amount, timestamp)

  as a DIRECTED WEIGHTED graph of accounts:
      node = account, edge = payment, weight = amount
      -> good for flow analysis, fraud rings, money paths

  as a BIPARTITE graph of accounts and merchants:
      nodes = {accounts} + {merchants}, edges = transactions
      -> good for recommendation, co-purchase similarity

  as a PROJECTED graph of accounts only:
      edge between two accounts that paid the same merchant
      -> good for community detection, but the projection
         creates a clique per merchant and inflates density

That last warning matters. Projecting a bipartite graph onto one side turns every shared neighbour into a complete subgraph, so a single popular merchant with 10,000 customers produces roughly 50 million edges. Projections need weighting or thresholding, or they become computationally and statistically useless.

Key Takeaways

• In graph data the signal lives in the connections, not in node attributes; that is what distinguishes it from a table.
• Choose the representation by operation: edge lists for storage, adjacency lists for traversal, sparse matrices for linear algebra.
• Real networks are extremely sparse, so dense adjacency matrices are infeasible past a few tens of thousands of nodes.
• Degree distributions are heavy-tailed; the mean degree describes almost no actual node, so report the median and the tail.
• Small-world structure means short paths despite sparsity, because hubs act as shortcuts for nearly every pair.
• Deciding what becomes a node and what becomes an edge is a modelling choice, and the same source table supports several different graphs.
• Projecting a bipartite graph creates a clique per shared neighbour and can explode edge counts; weight or threshold the projection.

🧪 Practice

1. Build the same small graph as an edge list, adjacency matrix, and adjacency list, and implement an edge-existence check for each.
2. Compute the memory required for dense and sparse representations of a graph with 1 million nodes and average degree 20.
3. Generate scale-free and random graphs with matched mean degree and plot both degree distributions on log-log axes.
4. Take a bipartite user-product dataset, project it onto users, and report the edge count before and after applying a minimum-shared-items threshold.
5. Measure the mean shortest path in a sparse graph before and after deleting the top 1% of nodes by degree.
6. Interview: A colleague reports "our network has an average of 40 connections per user, so engagement is healthy". What is your response? (Hint: consider what the distribution behind that mean probably looks like, and what a median and a top-decile share would reveal instead.)

Centrality and Community Detection

Two questions dominate network analysis: who matters and who belongs together. Centrality answers the first, community detection the second, and both come with a family of definitions rather than a single answer.

Centrality measures importance, but "important" means different things, and choosing the wrong measure gives a confidently wrong answer:

Measure	Definition	"Important" means	Example use
Degree	Number of connections	Well connected locally	Most-followed accounts
Betweenness	Fraction of shortest paths passing through	Controls flow between groups	Bridge accounts, chokepoints
Closeness	Inverse mean distance to all others	Can reach everyone quickly	Fast information spread
Eigenvector	Connected to other well-connected nodes	Influential by association	Prestige, endorsement
PageRank	Eigenvector with damping and teleportation	Robust influence in directed graphs	Web ranking, citation

The distinction between degree and betweenness is the one worth internalising. A node with 500 connections all inside one tight group has high degree and low betweenness -- remove it and nothing disconnects. A node with 4 connections, two to each of two otherwise separate communities, has low degree and very high betweenness -- remove it and the network splits. For fraud detection, infrastructure resilience, or identifying key intermediaries, betweenness is usually the right question and degree is the one people compute by mistake.

PageRank fixes two failure modes of plain eigenvector centrality on directed graphs: nodes with no outgoing edges absorb all the score ("rank sinks"), and disconnected components cannot be compared. The damping factor (conventionally 0.85) means a random surfer follows a link 85% of the time and teleports to a random node 15% of the time, which guarantees a unique, well-behaved solution.

Community detection partitions nodes into groups with dense internal and sparse external connections. The standard objective is modularity: the fraction of edges inside communities, minus what you would expect if edges were rewired at random while preserving degrees. High modularity means the grouping explains more structure than chance.

The main algorithms:

• Louvain: greedily moves nodes between communities to increase modularity, then collapses communities into super-nodes and repeats. Fast and widely used. Can produce internally disconnected communities.
• Leiden: refines Louvain with a guarantee that communities are connected. Strictly better; use it where available.
• Label propagation: each node adopts the most common label among its neighbours until stable. Near-linear time, but non-deterministic and unstable.
• Spectral clustering: k-means on the eigenvectors of the graph Laplacian. Principled, requires knowing k, and does not scale past moderate sizes.

Modularity has a known weakness worth stating: the resolution limit. It cannot detect communities smaller than roughly sqrt(2E) edges, so in a large network genuinely distinct small groups get merged. The resolution parameter in modern implementations exists to work around this.

import numpy as np
import networkx as nx
from networkx.algorithms.community import (louvain_communities,
                                           label_propagation_communities,
                                           modularity)
from sklearn.metrics import adjusted_rand_score, normalized_mutual_info_score

# --- A GRAPH BUILT TO SEPARATE DEGREE FROM BETWEENNESS. ---
# Two dense clusters joined by a single low-degree bridge node.
G = nx.Graph()
cluster_a = [f"a{i}" for i in range(12)]
cluster_b = [f"b{i}" for i in range(12)]
G.add_edges_from((u, v) for i, u in enumerate(cluster_a)
                 for v in cluster_a[i + 1:])          # a-cluster is a clique
G.add_edges_from((u, v) for i, u in enumerate(cluster_b)
                 for v in cluster_b[i + 1:])          # so is b
G.add_edges_from([("bridge", "a0"), ("bridge", "b0")])  # the ONLY link

deg = nx.degree_centrality(G)
btw = nx.betweenness_centrality(G)
clo = nx.closeness_centrality(G)
eig = nx.eigenvector_centrality(G, max_iter=1000)

print("FOUR CENTRALITIES DISAGREE ABOUT WHO MATTERS")
print(f"  {'node':8s} {'degree':>8s} {'between':>9s} {'close':>8s} "
      f"{'eigen':>8s}")
for n in ["a0", "a5", "bridge", "b0"]:
    print(f"  {n:8s} {deg[n]:8.3f} {btw[n]:9.3f} {clo[n]:8.3f} {eig[n]:8.3f}")
print(f"\n  highest degree      {max(deg, key=deg.get)}")
print(f"  highest betweenness {max(btw, key=btw.get)}")
print("  The bridge has the LOWEST degree and the HIGHEST betweenness.")
print("  a5, with nearly the highest degree, has betweenness of exactly 0:")
print("  no shortest path anywhere in the graph goes through it.")

# Confirm the structural claim rather than asserting it.
H = G.copy(); H.remove_node("bridge")
print(f"  components after removing the bridge: "
      f"{nx.number_connected_components(H)}")
H2 = G.copy(); H2.remove_node("a5")
print(f"  components after removing a5 (high degree): "
      f"{nx.number_connected_components(H2)}")

# --- COMMUNITY DETECTION ON A GRAPH WITH KNOWN GROUPS. ---
sizes = [40, 60, 30, 70]
p_in, p_out = 0.25, 0.008
probs = [[p_in if i == j else p_out for j in range(4)] for i in range(4)]
SBM = nx.stochastic_block_model(sizes, probs, seed=7)
truth = {n: SBM.nodes[n]["block"] for n in SBM}

def to_labels(communities, g):
    lab = {}
    for i, c in enumerate(communities):
        for n in c:
            lab[n] = i
    return [lab[n] for n in g.nodes]

y_true = [truth[n] for n in SBM.nodes]
methods = {
    "Louvain (res=1.0)":  louvain_communities(SBM, seed=1),
    "Louvain (res=2.0)":  louvain_communities(SBM, resolution=2.0, seed=1),
    "label propagation":  list(label_propagation_communities(SBM)),
}

print("\nCOMMUNITY DETECTION AGAINST KNOWN GROUND TRUTH (4 blocks)")
print(f"  {'method':22s} {'found':>6s} {'modularity':>11s} {'ARI':>6s} "
      f"{'NMI':>6s}")
for name, comms in methods.items():
    y_pred = to_labels(comms, SBM)
    print(f"  {name:22s} {len(comms):6d} "
          f"{modularity(SBM, comms):11.3f} "
          f"{adjusted_rand_score(y_true, y_pred):6.3f} "
          f"{normalized_mutual_info_score(y_true, y_pred):6.3f}")

# --- MODULARITY OF A RANDOM PARTITION: THE BASELINE NOBODY REPORTS. ---
rng = np.random.default_rng(0)
random_labels = rng.integers(0, 4, len(SBM))
random_comms = [set(np.array(SBM.nodes)[random_labels == k]) for k in range(4)]
print(f"\n  random 4-way partition modularity "
      f"{modularity(SBM, random_comms):.3f}")
print("  Any modularity score needs this comparison. A number near zero")
print("  means the partition explains nothing the degree sequence did not.")

  FOUR CENTRALITIES DISAGREE ABOUT WHO MATTERS

    node       degree   between    close    eigen
    a0          0.500     0.518    0.511    0.207
    a5          0.458     0.000    0.400    0.204
    bridge      0.083     0.522    0.522    0.038
    b0          0.500     0.518    0.511    0.207

    highest degree      a0
    highest betweenness bridge
    The bridge has the LOWEST degree and the HIGHEST betweenness.
    a5, with nearly the highest degree, has betweenness of exactly 0:
    no shortest path anywhere in the graph goes through it.
    components after removing the bridge: 2
    components after removing a5 (high degree): 1

  COMMUNITY DETECTION AGAINST KNOWN GROUND TRUTH (4 blocks)

    method                  found  modularity    ARI    NMI
    Louvain (res=1.0)           4       0.627  0.981  0.971
    Louvain (res=2.0)           7       0.548  0.694  0.812
    label propagation           4       0.625  0.962  0.949

    random 4-way partition modularity -0.003
    Any modularity score needs this comparison. A number near zero
    means the partition explains nothing the degree sequence did not.

The centrality table earns its space. Four reasonable definitions of "important" produce four different rankings on the same 25-node graph, and the a5 row is the sharpest illustration: a node with the second-highest degree in the network that lies on precisely zero shortest paths. Deleting it changes nothing; deleting the barely-connected bridge splits the graph in two. If your analysis is about resilience, influence propagation, or identifying intermediaries, degree is answering a different question than the one you asked.

A useful diagram of the same point:

     CLUSTER A                          CLUSTER B
   a1---a2---a3                       b1---b2---b3
    \   |   /                          \   |   /
     \  |  /                            \  |  /
      a0 ------- bridge --------------- b0
     /  |  \                            /  |  \
   a4---a5---a6                       b4---b5---b6

   a5:      degree 11, betweenness 0     -- locally central, structurally inert
   a0:      degree 12, betweenness high  -- the gateway into cluster A
   bridge:  degree 2,  betweenness max   -- the single point of failure

On community detection, three habits separate a defensible analysis from a decorative one:

1. Always compare against a null. A random partition of this graph scores modularity near zero. Report your score next to it.
2. Check stability. Louvain is order-dependent and stochastic. Run it with several seeds and measure how often node pairs land together; pairs that co-occur 55% of the time are not a community.
3. Validate against something external. If you have any node attribute not used in detection -- department, country, product category -- check whether the communities align with it. Communities that correspond to nothing observable are usually an artifact of the algorithm's resolution parameter.

Key Takeaways

• Centrality is a family of definitions, not one measure; pick the one that matches the question being asked.
• Degree measures local connectedness; betweenness measures control of flow between groups, and the two can be nearly opposite.
• A node can have high degree and zero betweenness (inside a clique) or low degree and maximum betweenness (a bridge).
• PageRank adds damping and teleportation to eigenvector centrality, fixing rank sinks and disconnected components in directed graphs.
• Modularity compares within-community edge density against a degree-preserving random baseline; always report the random-partition score alongside it.
• Louvain is fast but can produce disconnected communities; Leiden fixes this and should be preferred where available.
• Modularity has a resolution limit that merges genuinely small communities in large graphs; the resolution parameter counteracts it.
• Validate communities against an external node attribute and check their stability across random seeds.

🧪 Practice

1. Construct a graph where the highest-degree node and the highest-betweenness node are different, and verify by deleting each and counting connected components.
2. Compute all five centrality measures on a real network and report the rank correlation between each pair.
3. Run Louvain with resolutions 0.5, 1.0, and 2.0 on the same graph and report the number of communities and the modularity of each.
4. Run Louvain 20 times with different seeds and compute the fraction of runs in which each node pair shares a community.
5. Generate a stochastic block model with known communities and measure ARI as the between-block edge probability increases.
6. Interview: A fraud team wants "the most important accounts" in a payment network. What would you ask before choosing a centrality measure? (Hint: importance for what -- moving money between rings, receiving from many sources, or being endorsed by other suspicious accounts? Each maps to a different measure.)

Link Prediction

Link prediction asks which edges should exist but do not yet: which two users will become friends, which researchers will co-author, which account pair will transact, which protein pair interacts. It is the recommendation problem stated in graph terms, and it is also how missing data in networks gets filled.

The task has an unusual structure that dictates everything about how it is set up. In a graph with V nodes there are V(V-1)/2 possible edges and typically only O(V) actual ones, so the positive class is a vanishingly small fraction of the candidate space. A network with 10,000 nodes has 50 million candidate pairs and perhaps 50,000 edges -- one positive per thousand candidates.

This makes two things true. Accuracy is meaningless. And you cannot score every pair at scale, so candidate generation -- usually restricting to pairs within two or three hops -- is part of the method, not a preprocessing detail.

Heuristic scores exploit the observation that most new edges close triangles: people befriend friends of friends. The classic family, in increasing sophistication:

Heuristic	Score for pair (u, v)	Intuition
Common neighbours	`	N(u) ∩ N(v)
Jaccard	`	N(u) ∩ N(v)
Adamic-Adar	`Σ 1/log	N(w)
Resource allocation	`Σ 1/	N(w)
Preferential attachment	`	N(u)

Adamic-Adar captures a genuinely useful idea: sharing a mutual friend who has 8 friends is far more informative than sharing one who has 8,000. The first implies a real social context; the second implies almost nothing.

Embedding methods (node2vec, DeepWalk) learn a vector per node from random walks and score pairs by similarity. Graph neural networks go further, learning representations that aggregate neighbour features, and generally win when nodes carry attributes.

The part practitioners get wrong is evaluation. Two specific traps:

1. Temporal leakage. If edges have timestamps, split by time. Training on the full graph and testing on randomly held-out edges lets the model see the future structure around the very edge it must predict.
2. Negative sampling bias. The chosen negatives determine the score. If you sample negatives uniformly at random, most are pairs of unrelated nodes in distant parts of the graph, which is trivially easy and inflates AUC. Realistic negatives are pairs that are plausible but did not connect.

import numpy as np
import networkx as nx
from sklearn.metrics import roc_auc_score, average_precision_score

rng = np.random.default_rng(11)

# --- BUILD A GRAPH AND HOLD OUT EDGES PROPERLY. ---
G_full = nx.barabasi_albert_graph(1200, 4, seed=3)
edges = list(G_full.edges())
rng.shuffle(edges)
n_test = len(edges) // 10

# Removing edges can disconnect nodes, which silently changes the problem.
G_train = G_full.copy()
test_pos = []
for u, v in edges:
    if len(test_pos) >= n_test:
        break
    if G_train.degree(u) > 1 and G_train.degree(v) > 1:   # keep both connected
        G_train.remove_edge(u, v)
        test_pos.append((u, v))

print(f"GRAPH  {G_full.number_of_nodes()} nodes, "
      f"{G_full.number_of_edges()} edges")
print(f"  held out {len(test_pos)} positive edges")
print(f"  candidate pairs in total: "
      f"{G_full.number_of_nodes() * (G_full.number_of_nodes() - 1) // 2:,}")
print(f"  positive rate among all pairs: "
      f"{G_full.number_of_edges() / (G_full.number_of_nodes() ** 2 / 2):.5f}")

# --- TWO WAYS TO SAMPLE NEGATIVES, WITH VERY DIFFERENT DIFFICULTY. ---
nodes = list(G_train.nodes)
def sample_random_negatives(g, k):
    """Any non-edge. Most will be distant, unrelated pairs: easy."""
    out = set()
    while len(out) < k:
        u, v = rng.choice(nodes, 2, replace=False)
        if not g.has_edge(u, v):
            out.add((int(u), int(v)))
    return list(out)

def sample_hard_negatives(g, k):
    """Non-edges between nodes two hops apart: plausible but absent.
    This is what the model will actually face in deployment."""
    out = set()
    attempts = 0
    while len(out) < k and attempts < k * 200:
        attempts += 1
        u = int(rng.choice(nodes))
        nbrs = list(g[u])
        if not nbrs:
            continue
        w = int(rng.choice(nbrs))
        two_hop = [x for x in g[w] if x != u and not g.has_edge(u, x)]
        if two_hop:
            out.add((u, int(rng.choice(two_hop))))
    return list(out)

# --- HEURISTIC SCORERS. ---
def common_neighbours(g, u, v):
    return len(set(g[u]) & set(g[v]))

def jaccard(g, u, v):
    a, b = set(g[u]), set(g[v])
    return len(a & b) / len(a | b) if a | b else 0.0

def adamic_adar(g, u, v):
    """Rare shared neighbours count for more: 1/log(degree)."""
    return sum(1 / np.log(g.degree(w))
               for w in set(g[u]) & set(g[v]) if g.degree(w) > 1)

def resource_allocation(g, u, v):
    return sum(1 / g.degree(w) for w in set(g[u]) & set(g[v]) if g.degree(w))

def preferential_attachment(g, u, v):
    return g.degree(u) * g.degree(v)

scorers = {
    "common neighbours":      common_neighbours,
    "Jaccard":                jaccard,
    "Adamic-Adar":            adamic_adar,
    "resource allocation":    resource_allocation,
    "preferential attachment": preferential_attachment,
}

for neg_name, sampler in [("random negatives", sample_random_negatives),
                          ("hard (2-hop) negatives", sample_hard_negatives)]:
    test_neg = sampler(G_train, len(test_pos))
    pairs = test_pos + test_neg
    y = np.r_[np.ones(len(test_pos)), np.zeros(len(test_neg))]
    print(f"\n{neg_name.upper()}")
    print(f"  {'scorer':26s} {'AUC':>7s} {'AP':>7s}")
    for name, fn in scorers.items():
        s = np.array([fn(G_train, u, v) for u, v in pairs])
        print(f"  {name:26s} {roc_auc_score(y, s):7.3f} "
              f"{average_precision_score(y, s):7.3f}")

print("\n  The same scorers look excellent against random negatives and")
print("  merely useful against realistic ones. Only the second column")
print("  describes deployment, where every candidate is already plausible.")

  GRAPH  1200 nodes, 4784 edges
    held out 478 positive edges
    candidate pairs in total: 719,400
    positive rate among all pairs: 0.00664

  RANDOM NEGATIVES

    scorer                         AUC      AP
    common neighbours            0.906   0.882
    Jaccard                      0.904   0.879
    Adamic-Adar                  0.938   0.923
    resource allocation          0.941   0.928
    preferential attachment      0.917   0.895

  HARD (2-HOP) NEGATIVES

    scorer                         AUC      AP
    common neighbours            0.671   0.643
    Jaccard                      0.652   0.628
    Adamic-Adar                  0.714   0.688
    resource allocation          0.723   0.697
    preferential attachment      0.608   0.579

    The same scorers look excellent against random negatives and
    merely useful against realistic ones. Only the second column
    describes deployment, where every candidate is already plausible.

That drop -- Adamic-Adar from 0.938 to 0.714 -- is not noise and it is not a worse model. It is the difference between a benchmark that asks "can you distinguish a real edge from two random strangers" and one that asks "can you distinguish a real edge from a plausible non-edge". Only the second question is ever asked in production, because a deployed system only ever scores candidates that passed some plausibility filter in the first place.

The correct evaluation setup, in order:

  1. SPLIT BY TIME if edges are timestamped
       train on edges before T, test on edges formed after T
       never sample test edges at random from a static graph

  2. REMOVE test edges from the training graph
       a heuristic that can see the edge it predicts scores perfectly
       and teaches you nothing

  3. GENERATE CANDIDATES the way production will
       usually 2-hop neighbourhoods, not all pairs

  4. SAMPLE NEGATIVES from that same candidate set
       matching the deployment distribution, not uniform pairs

  5. REPORT AP or precision@k, not AUC alone
       with a 1-in-1000 base rate, AUC flatters everything

Step 5 deserves emphasis. AUC treats the ranking of a random positive above a random negative as the objective, which is forgiving when negatives vastly outnumber positives. Average precision and precision@k reflect what a user of the system experiences: of the twenty suggestions shown, how many were right.

One further consideration specific to social and recommendation settings: link prediction systems are self-fulfilling. Suggesting a connection makes it more likely, so the next training set contains edges the previous model created. Without care this collapses into recommending the already popular -- exactly what preferential attachment scores -- and the network's structure is shaped by the model rather than measured by it.

Key Takeaways

• Link prediction scores candidate node pairs for edges that do not yet exist; the positive class is a tiny fraction of all pairs.
• Candidate generation, typically 2-hop neighbourhoods, is part of the method because scoring all pairs is infeasible.
• Most heuristics exploit triangle closure; Adamic-Adar and resource allocation weight rare shared neighbours more heavily than common ones.
• Test edges must be removed from the training graph, or the heuristic sees the answer it is being asked to predict.
• Split by time when timestamps exist; random splits leak future structure.
• Negative sampling strategy determines the reported score; uniform random negatives are too easy and inflate AUC substantially.
• Report average precision or precision@k rather than AUC alone under extreme class imbalance.
• Deployed link prediction changes the graph it learns from, which can drive a popularity feedback loop.

🧪 Practice

1. Implement common neighbours, Jaccard, and Adamic-Adar, and rank the same candidate pairs with each; report where the rankings disagree most.
2. Evaluate a link predictor with uniformly random negatives and with 2-hop negatives, and report both AUC values.
3. Hold out edges randomly versus by timestamp on a temporal graph and compare the resulting scores.
4. Compute precision@10 and average precision for a predictor with a 1-in-1000 positive rate, and compare them against AUC.
5. Train node2vec embeddings and compare their link-prediction AP against Adamic-Adar on the same split.
6. Interview: Your friend-suggestion model has AUC 0.95 offline and a 2% acceptance rate in production. What explains the gap? (Hint: think about which pairs the offline negatives were drawn from versus which pairs actually reach a user's screen.)

Coordinate Systems and Projections

Geospatial data has a problem no other data type has: the Earth is a curved, slightly lumpy ellipsoid, and every map, every screen, and every distance calculation is flat. Projection is the act of flattening, and it is mathematically impossible to do without distortion. Every projection preserves some properties and destroys others, and choosing wrongly produces numbers that look reasonable and are wrong by kilometres.

Three concepts, often confused:

• Datum: the reference ellipsoid modelling the Earth's shape, plus its positioning. WGS84 is the GPS standard; NAD83 and ETRS89 are regional. The same latitude and longitude in different datums can differ by tens of metres -- irrelevant for a country-level map, fatal for surveying.
• Coordinate Reference System (CRS): datum plus a coordinate system. Identified by an EPSG code. EPSG:4326 is WGS84 latitude/longitude in degrees.
• Projection: the mathematical transformation from the curved surface to a plane.

The projections you will actually meet:

CRS / Projection	EPSG	Units	Preserves	Use
WGS84 lat/lon	4326	Degrees	Nothing (unprojected)	Storage, GPS, data interchange
Web Mercator	3857	Metres	Local shape/angle	Web maps; badly distorts area
UTM zone N	326NN	Metres	Local distance and area	Accurate local measurement
Albers Equal Area	varies	Metres	Area	Choropleths, density maps
Azimuthal Equidistant	varies	Metres	Distance from one centre	Range rings, travel time

The single most common geospatial bug is computing distances or areas on EPSG:4326 as though degrees were a linear unit. A degree of latitude is about 111 km everywhere. A degree of longitude is 111 km at the equator, 78 km at 45 degrees north, and 0 km at the pole. Euclidean distance on lat/lon is therefore wrong everywhere except along a meridian, and the error grows with latitude.

  WHY DEGREES ARE NOT A DISTANCE UNIT

   equator (0 deg)        |<---- 1 degree lon = 111 km ---->|
   45 deg north           |<-- 1 deg lon = 78.6 km -->|
   60 deg north           |<- 1 deg = 55.8 km ->|
   80 deg north           |<- 19 km ->|
   pole (90 deg)          | 0 km

   The same numeric difference in longitude means a different physical
   distance at every latitude. sqrt(dlat^2 + dlon^2) is meaningless.

Web Mercator's area distortion is the other famous one: it preserves angles, which is why it is used for interactive maps where shapes must look right when you zoom, and it inflates area by 1/cos²(latitude). Greenland appears roughly the size of Africa and is fourteen times smaller. Any choropleth of density on Web Mercator overstates high-latitude regions dramatically.

import numpy as np
import geopandas as gpd
from shapely.geometry import Point, box

R = 6_371_008.8            # mean Earth radius in metres

def haversine(lat1, lon1, lat2, lon2):
    """Great-circle distance on a sphere. Correct to ~0.5% everywhere,
    which is far better than anything Euclidean-on-degrees achieves."""
    p1, p2 = np.radians(lat1), np.radians(lat2)
    dp, dl = np.radians(lat2 - lat1), np.radians(lon2 - lon1)
    a = np.sin(dp / 2) ** 2 + np.cos(p1) * np.cos(p2) * np.sin(dl / 2) ** 2
    return 2 * R * np.arcsin(np.sqrt(a))

def naive_degree_distance(lat1, lon1, lat2, lon2, km_per_deg=111.32):
    """The bug: treat degrees as a flat metric unit."""
    return np.hypot(lat2 - lat1, lon2 - lon1) * km_per_deg * 1000

# --- THE SAME ONE-DEGREE LONGITUDE STEP AT DIFFERENT LATITUDES. ---
print("ONE DEGREE OF LONGITUDE, MEASURED CORRECTLY AND NAIVELY")
print(f"  {'latitude':>9s} {'true (km)':>11s} {'naive (km)':>11s} "
      f"{'error':>9s}")
for lat in [0, 30, 45, 60, 75, 85]:
    true_km = haversine(lat, 0, lat, 1) / 1000
    naive_km = naive_degree_distance(lat, 0, lat, 1) / 1000
    print(f"  {lat:8d}° {true_km:11.1f} {naive_km:11.1f} "
          f"{100 * (naive_km - true_km) / true_km:8.1f}%")

# --- A REAL PAIR OF CITIES. ---
oslo = (59.9139, 10.7522)
stockholm = (59.3293, 18.0686)
print(f"\nOSLO TO STOCKHOLM")
print(f"  haversine          {haversine(*oslo, *stockholm) / 1000:8.1f} km")
print(f"  naive on degrees   "
      f"{naive_degree_distance(*oslo, *stockholm) / 1000:8.1f} km")
print(f"  published road/air distance is about 417 km great-circle.")

# --- PROJECTING PROPERLY, WITH A LIBRARY. ---
pts = gpd.GeoDataFrame(
    {"city": ["Oslo", "Stockholm"]},
    geometry=[Point(lon, lat) for lat, lon in [oslo, stockholm]],
    crs="EPSG:4326",            # ALWAYS declare the CRS on creation
)

# UTM zone 33N covers southern Norway and Sweden; metres, minimal distortion.
utm = pts.to_crs("EPSG:32633")
d = utm.geometry.iloc[0].distance(utm.geometry.iloc[1])
print(f"  projected to UTM33N {d / 1000:8.1f} km   <- agrees with haversine")

# --- WEB MERCATOR AREA INFLATION. ---
def mercator_area_factor(lat):
    """Web Mercator inflates area by 1/cos^2(lat). This is why Greenland
    looks continental."""
    return 1 / np.cos(np.radians(lat)) ** 2

print("\nWEB MERCATOR AREA INFLATION BY LATITUDE")
print(f"  {'latitude':>9s} {'inflation':>10s}  example")
examples = {0: "Congo basin", 30: "Egypt", 45: "northern Italy",
            60: "southern Norway", 70: "northern Sweden", 80: "Svalbard"}
for lat, place in examples.items():
    print(f"  {lat:8d}° {mercator_area_factor(lat):9.1f}x  {place}")
print("  A density choropleth drawn on Web Mercator makes Arctic regions")
print("  look 30 times larger than they are, which visually dominates the")
print("  map regardless of the underlying values.")

# --- THE CRS MISMATCH BUG, MADE VISIBLE. ---
# Two datasets, same place, different CRS. Joining them without
# reprojecting produces zero matches and no error.
poly_4326 = gpd.GeoDataFrame(
    geometry=[box(10.6, 59.85, 10.9, 60.0)], crs="EPSG:4326")
poly_3857 = poly_4326.to_crs("EPSG:3857")
print(f"\nSAME POLYGON, TWO COORDINATE SYSTEMS")
print(f"  EPSG:4326 bounds {np.round(poly_4326.total_bounds, 3)}")
print(f"  EPSG:3857 bounds {np.round(poly_3857.total_bounds, 0)}")
print("  The numbers are not comparable at all. A spatial join across")
print("  these returns an empty result, which reads as 'no matches found'")
print("  rather than 'you forgot to reproject'.")

  ONE DEGREE OF LONGITUDE, MEASURED CORRECTLY AND NAIVELY

     latitude   true (km)  naive (km)     error
           0°       111.2       111.3      0.1%
          30°        96.3       111.3     15.6%
          45°        78.6       111.3     41.6%
          60°        55.6       111.3    100.2%
          75°        28.8       111.3    286.8%
          85°         9.7       111.3   1048.8%

  OSLO TO STOCKHOLM

    haversine             416.4 km
    naive on degrees      817.0 km
    published road/air distance is about 417 km great-circle.
    projected to UTM33N    416.6 km   <- agrees with haversine

  WEB MERCATOR AREA INFLATION BY LATITUDE

     latitude  inflation  example
           0°        1.0x  Congo basin
          30°        1.3x  Egypt
          45°        2.0x  northern Italy
          60°        4.0x  southern Norway
          70°        8.5x  northern Sweden
          80°       33.2x  Svalbard
    A density choropleth drawn on Web Mercator makes Arctic regions
    look 30 times larger than they are, which visually dominates the
    map regardless of the underlying values.

  SAME POLYGON, TWO COORDINATE SYSTEMS

    EPSG:4326 bounds [ 10.6   59.85  10.9   60.  ]
    EPSG:3857 bounds [1180003. 8384608. 1213398. 8399737.]
    The numbers are not comparable at all. A spatial join across
    these returns an empty result, which reads as 'no matches found'
    rather than 'you forgot to reproject'.

The Oslo-Stockholm line is the practical warning in one number: treating degrees as kilometres nearly doubles the answer, and at 60 degrees north the error on an east-west leg is over 100%. Nothing about the output signals a problem -- it is a plausible-looking distance in plausible-looking units.

A short working discipline that prevents nearly all of this:

1. Store in EPSG:4326. It is the interchange standard and what every GPS and API produces.
2. Reproject to a metric CRS before measuring. UTM for the local zone, or an equal-area projection for national and continental extents.
3. Declare the CRS at creation and assert it at every boundary. Never let a geometry exist without a known CRS; libraries will happily compute with mismatched ones.
4. Match the projection to the operation. Equal-area for anything involving area or density, equidistant for buffers and range rings, conformal for display.

The recurring failure mode is silent rather than loud. A spatial join across mismatched CRSs returns no matches, a buffer in degrees produces an ellipse that is far too wide at high latitude, and an area calculation on lat/lon returns a number in square degrees that nobody notices is not square kilometres.

Key Takeaways

• Flattening a curved Earth always distorts something; every projection trades one property for another.
• Datum, CRS, and projection are distinct concepts; EPSG codes name the combination unambiguously.
• EPSG:4326 is degrees, not a distance unit; Euclidean distance on lat/lon is wrong everywhere except along a meridian.
• A degree of longitude shrinks with latitude, so the error from treating degrees as linear grows from negligible at the equator to enormous in the Arctic.
• Use haversine for spherical distance, or reproject to a metric CRS such as the local UTM zone before measuring.
• Web Mercator preserves angles and inflates area by 1/cos²(latitude), which badly distorts density maps.
• Choose the projection by operation: equal-area for density, equidistant for buffers, conformal for display.
• CRS mismatches fail silently, usually as an empty spatial join rather than an error.

🧪 Practice

1. Compute the distance between two cities with haversine and with Euclidean distance on degrees, and report the percentage error.
2. Tabulate the true length of one degree of longitude at latitudes 0, 30, 60, and 80, and explain the pattern.
3. Take a polygon in EPSG:4326, compute its area before and after projecting to an equal-area CRS, and explain the difference.
4. Attempt a spatial join between two datasets in different CRSs, observe the empty result, then fix it by reprojecting.
5. Create a 5 km buffer around a point in EPSG:4326 and in a local UTM zone, and compare the resulting shapes.
6. Interview: A dashboard shows population density by region on a Web Mercator basemap, and northern regions look dominant. What would you check? (Hint: consider what the projection does to the visual area of each region, and whether the colour scale or the geometry is doing the misleading.)

Spatial Joins and Indexing

A spatial join combines two datasets by geometric relationship rather than by a shared key. Which sales region contains this customer? Which customers fall within 2 km of this store? Which census tracts does this flood zone overlap? There is no join key -- the relationship is the geometry.

The relationships, drawn from the standard DE-9IM predicates:

Predicate	True when	Typical use
contains	A fully encloses B	Region contains point
within	A is fully inside B	Point within region (the inverse)
intersects	Any shared point at all	Broadest catch-all
touches	Share a boundary but no interior	Adjacent parcels
overlaps	Partial overlap, same dimension	Competing service areas
crosses	Pass through each other	Road crossing a river
dwithin	Within a given distance	Catchment areas

The computational problem is that geometric predicates are expensive. Testing whether a point is inside a polygon with 5,000 vertices is not a comparison; it is a ray-casting loop over every edge. Joining 1 million points against 3,000 such polygons naively means 3 billion of those tests, which will not finish.

Spatial indexing solves this with a two-phase strategy that is worth understanding because the same pattern appears throughout computational geometry:

  TWO-PHASE SPATIAL JOIN

  PHASE 1 -- FILTER (cheap, approximate)
    Compare bounding boxes only. A bounding box test is four numeric
    comparisons. This eliminates the overwhelming majority of pairs.

       +-------------------+          bounding box of the polygon
       |    ___            |
       |   /   \___        |   <- the real geometry, expensive to test
       |  |  A      \      |
       |   \________/      |
       +-------------------+
          .   x       .        x = candidate point
                               . = points rejected by the box test alone

  PHASE 2 -- REFINE (expensive, exact)
    Run the true predicate only on survivors of phase 1. A point inside
    the box may still be outside the polygon.

The index itself is usually an R-tree: a hierarchy of nested bounding boxes that lets a query skip entire subtrees whose box does not intersect the query box. Query cost drops from O(n) to roughly O(log n).

Other indexing schemes trade exactness for speed and simplicity:

• Grid / quadtree: divide space into cells, bucket geometries by cell. Simple, and degrades when data is highly clustered.
• Geohash: encode a lat/lon into a string where shared prefixes mean spatial proximity. Convenient in key-value stores; suffers at cell boundaries, where two adjacent points can have completely different prefixes.
• H3 / S2: hierarchical cell systems (hexagons and spherical quads). Uniform cell area, well-defined neighbours, and increasingly the default for large-scale spatial aggregation.

import numpy as np
import geopandas as gpd
from shapely.geometry import Point, Polygon
from shapely.strtree import STRtree
import time

rng = np.random.default_rng(2)

# --- BUILD A REALISTIC WORKLOAD: MANY POINTS, MODERATE POLYGONS. ---
N_POINTS, N_POLYS = 200_000, 400

def random_polygon(cx, cy, r=0.02, n_vertices=60):
    """A wiggly polygon: vertex count is what makes the exact test costly."""
    ang = np.sort(rng.uniform(0, 2 * np.pi, n_vertices))
    rad = r * rng.uniform(0.6, 1.4, n_vertices)
    return Polygon(np.c_[cx + rad * np.cos(ang), cy + rad * np.sin(ang)])

polys = [random_polygon(*rng.uniform(0, 1, 2)) for _ in range(N_POLYS)]
pts = [Point(x, y) for x, y in rng.uniform(0, 1, (N_POINTS, 2))]

print(f"WORKLOAD  {N_POINTS:,} points x {N_POLYS} polygons")
print(f"  naive pair count: {N_POINTS * N_POLYS:,}")
print(f"  mean polygon vertices: "
      f"{np.mean([len(p.exterior.coords) for p in polys]):.0f}")

# --- NAIVE: EVERY PAIR, EXACT PREDICATE. TIMED ON A SUBSET. ---
subset = pts[:2_000]
t0 = time.perf_counter()
naive_hits = sum(1 for p in subset for poly in polys if poly.contains(p))
naive_s = time.perf_counter() - t0
print(f"\n  naive on {len(subset):,} points: {naive_s:.2f}s  "
      f"-> {naive_s * N_POINTS / len(subset):,.0f}s for the full set")

# --- INDEXED: R-TREE FILTER, THEN EXACT REFINE. ---
tree = STRtree(polys)
t0 = time.perf_counter()
hits = 0
candidates_tested = 0
for p in pts:
    for idx in tree.query(p):            # PHASE 1: bounding-box candidates
        candidates_tested += 1
        if polys[idx].contains(p):       # PHASE 2: exact predicate
            hits += 1
indexed_s = time.perf_counter() - t0
print(f"  indexed on {N_POINTS:,} points: {indexed_s:.2f}s")
print(f"  exact predicate evaluations: {candidates_tested:,} "
      f"({100 * candidates_tested / (N_POINTS * N_POLYS):.4f}% of naive)")
print(f"  speedup: {(naive_s * N_POINTS / len(subset)) / indexed_s:,.0f}x")

# --- THE FILTER IS APPROXIMATE, WHICH IS THE WHOLE POINT. ---
false_positives = candidates_tested - hits
print(f"\n  phase 1 returned {candidates_tested:,} candidates")
print(f"  phase 2 confirmed {hits:,} of them")
print(f"  {100 * false_positives / max(candidates_tested, 1):.1f}% were inside")
print(f"  the bounding box but outside the polygon -- which is exactly why")
print(f"  the exact test cannot be skipped.")

# --- GEOPANDAS DOES ALL OF THIS FOR YOU, IF YOU LET IT. ---
gdf_pts = gpd.GeoDataFrame(geometry=pts, crs="EPSG:4326")
gdf_poly = gpd.GeoDataFrame({"zone": range(N_POLYS)}, geometry=polys,
                            crs="EPSG:4326")
t0 = time.perf_counter()
joined = gpd.sjoin(gdf_pts, gdf_poly, predicate="within", how="inner")
sjoin_s = time.perf_counter() - t0
print(f"\n  gpd.sjoin: {sjoin_s:.2f}s, {len(joined):,} matches")
print("  sjoin builds the spatial index internally. Writing the nested")
print("  loop by hand is the most common performance mistake in")
print("  geospatial code, and it is invisible until the data grows.")

# --- BUFFER-BASED JOINS: THE UNITS TRAP AGAIN. ---
print("\n  A '2 km catchment' join needs a metric CRS:")
print("    WRONG:  gdf.to_crs(4326).buffer(2000)   -> 2000 DEGREES")
print("    RIGHT:  gdf.to_crs(utm_epsg).buffer(2000)  -> 2000 metres")
print("  The wrong version does not error. It produces a buffer larger")
print("  than the planet and joins everything to everything.")

  WORKLOAD  200,000 points x 400 polygons
    naive pair count: 80,000,000
    mean polygon vertices: 61

    naive on 2,000 points: 3.41s  -> 341s for the full set
    indexed on 200,000 points: 1.91s
    exact predicate evaluations: 224,118 (0.2801% of naive)
    speedup: 179x

    phase 1 returned 224,118 candidates
    phase 2 confirmed 94,836 of them
    57.7% were inside the bounding box but outside the polygon -- which
    is exactly why the exact test cannot be skipped.

    gpd.sjoin: 0.88s, 94,836 matches
    sjoin builds the spatial index internally. Writing the nested
    loop by hand is the most common performance mistake in
    geospatial code, and it is invisible until the data grows.

    A '2 km catchment' join needs a metric CRS:
      WRONG:  gdf.to_crs(4326).buffer(2000)   -> 2000 DEGREES
      RIGHT:  gdf.to_crs(utm_epsg).buffer(2000)  -> 2000 metres
    The wrong version does not error. It produces a buffer larger
    than the planet and joins everything to everything.

Two numbers carry the lesson. The index reduced 80 million exact predicate evaluations to 224,000 -- a factor of 357 -- which is where the speedup comes from. And 57.7% of those survivors still failed the exact test, which is why the filter phase cannot be the answer on its own: a bounding box is a generous approximation of a wiggly shape.

A few operational notes that matter on real data:

• Invalid geometries break predicates. Self-intersecting polygons ("bowtie" geometries), unclosed rings, and duplicate vertices are common in real-world shapefiles. Run a validity check and repair before joining, or the join will fail or, worse, silently return wrong results for those rows.
• Points exactly on boundaries are ambiguous. within excludes the boundary, intersects includes it, and a customer on a region border can join to zero regions or two. Decide the convention explicitly.
• Many-to-many joins explode. Overlapping polygons mean one point can match several rows. Check the output row count against the input before aggregating anything, or a duplicate join will inflate every downstream sum.
• H3 or geohash aggregation is often better than polygon joins for large-scale analytics: convert points to cell IDs once, then aggregate with an ordinary group-by, which distributes trivially and needs no geometry engine.

Key Takeaways

• Spatial joins combine datasets by geometric relationship, not by a shared key, so the predicate choice is the join condition.
• Exact geometric predicates are expensive, especially on polygons with many vertices; naive all-pairs joins do not scale.
• Spatial indexes use a cheap bounding-box filter followed by an exact refine, which is where nearly all the speedup comes from.
• The filter is approximate by design: a large fraction of box-level candidates fail the exact test.
• R-trees are the general-purpose index; grids, geohash, and H3 trade exactness or generality for simplicity and distributability.
• Buffer distances need a metric CRS; buffering in degrees produces nonsense without raising an error.
• Invalid geometries, boundary cases, and many-to-many matches are routine in real data and need explicit handling.

🧪 Practice

1. Implement a naive point-in-polygon join and an R-tree-indexed version, and compare runtimes as the point count grows from 1,000 to 100,000.
2. Count how many bounding-box candidates fail the exact predicate for a set of irregular polygons, and explain the ratio.
3. Perform a spatial join with within and with intersects on points that lie exactly on polygon boundaries, and compare the row counts.
4. Buffer a set of points by 5 km in EPSG:4326 and in a local UTM zone, and compare the resulting areas.
5. Aggregate 1 million points into H3 cells at two resolutions and compare the runtime against an equivalent polygon join.
6. Interview: A spatial join of 500,000 customers against 200 sales regions returns 640,000 rows. Is that a bug? (Hint: think about what geometric situation produces more output rows than input rows, and what you would check about the region polygons to confirm it.)

Spatial Autocorrelation

"Everything is related to everything else, but near things are more related than distant things." That sentence -- Tobler's first law of geography -- is the reason spatial data cannot be handed to standard statistical tools unmodified.

Ordinary regression and hypothesis testing assume observations are independent. Spatial data violates this systematically: house prices, disease rates, income, temperature, and vegetation are all more similar between neighbouring locations than between distant ones. The consequences are precise and severe:

• Standard errors are understated. If 500 observations are really 60 independent clusters, the effective sample size is far smaller than n, and every confidence interval is too narrow.
• Significance is overstated. Consequently, p-values are too small, and relationships that are artifacts of shared location look real.
• Residuals carry structure. If your model's errors are spatially clustered, a variable that varies over space is missing.

Moran's I is the standard global measure of spatial autocorrelation. It is essentially a correlation coefficient between each location's value and the weighted average of its neighbours' values:

            n       Σᵢ Σⱼ wᵢⱼ (xᵢ - x̄)(xⱼ - x̄)
  I  =  --------- * -------------------------------
          Σᵢ Σⱼ wᵢⱼ        Σᵢ (xᵢ - x̄)²

  Interpretation:
    I near  +1   strong clustering  (high values beside high values)
    I near   0   spatial randomness (the null)
    I near  -1   dispersion / checkerboard (high beside low)

  The expected value under the null is not 0 but -1/(n-1), which
  matters for small n.

Everything hinges on the spatial weights matrix W, which encodes what "neighbour" means. This is a modelling decision, not a technical detail, and different definitions give different answers on the same data:

Weights scheme	Definition	Suits
Queen contiguity	Shares any boundary point	Polygons, administrative areas
Rook contiguity	Shares an edge, not just a corner	Grids, raster cells
K-nearest neighbours	The k closest, whatever the distance	Points, uneven density
Distance band	Everything within radius d	Points with meaningful scale
Inverse distance	Weight falls off as 1/d or 1/d²	Continuous decay processes

Local Moran's I (a LISA statistic) decomposes the global figure into a per-location value, which is far more actionable. It classifies each location into one of four regimes:

                  neighbours HIGH        neighbours LOW
                +----------------------+----------------------+
   value HIGH   |  HH  -- hot spot     |  HL  -- high outlier |
                |  (cluster of high)   |  (a peak in a valley)|
                +----------------------+----------------------+
   value LOW    |  LH  -- low outlier  |  LL  -- cold spot    |
                |  (a dip on a plateau)|  (cluster of low)    |
                +----------------------+----------------------+

  HH and LL are clusters. HL and LH are spatial outliers, and they are
  often the more interesting finding: a single high-crime block inside a
  low-crime district is a different phenomenon from a high-crime district.

import numpy as np

rng = np.random.default_rng(6)

# --- THREE 12x12 GRIDS WITH KNOWN SPATIAL STRUCTURE. ---
n = 12
coords = np.array([(i, j) for i in range(n) for j in range(n)], dtype=float)

random_field = rng.normal(0, 1, n * n)                      # I should be ~0
gradient = coords[:, 0] + rng.normal(0, 0.4, n * n)         # strong clustering
checker = np.array([(-1) ** (i + j) for i, j in coords.astype(int)]) \
          + rng.normal(0, 0.2, n * n)                       # dispersion

# --- BUILD ROOK-CONTIGUITY WEIGHTS. ---
def rook_weights(coords, row_standardise=True):
    """Two cells are neighbours if they share an edge: distance exactly 1."""
    d = np.linalg.norm(coords[:, None, :] - coords[None, :, :], axis=-1)
    W = ((d > 0) & (d <= 1.0)).astype(float)
    if row_standardise:                     # each row sums to 1, so the lag
        s = W.sum(axis=1, keepdims=True)    # is a neighbour AVERAGE
        W = np.divide(W, s, out=np.zeros_like(W), where=s > 0)
    return W

W = rook_weights(coords)

def morans_i(x, W):
    z = x - x.mean()
    lag = W @ z                                   # weighted neighbour average
    num = (z * lag).sum()
    return (len(x) / W.sum()) * num / (z ** 2).sum()

def morans_i_pvalue(x, W, n_perm=999, rng=rng):
    """Permutation inference: reshuffle the values across locations and see
    how often chance produces an I this extreme. No distributional
    assumption is required, which is why this is the standard approach."""
    obs = morans_i(x, W)
    null = np.array([morans_i(rng.permutation(x), W) for _ in range(n_perm)])
    p = (np.sum(np.abs(null) >= abs(obs)) + 1) / (n_perm + 1)
    return obs, null.mean(), null.std(), p

print("GLOBAL MORAN'S I ON THREE FIELDS (rook contiguity, 999 permutations)")
print(f"  expected under the null: {-1 / (n * n - 1):+.4f}")
print(f"  {'field':20s} {'I':>8s} {'null mean':>10s} {'null sd':>8s} "
      f"{'p':>7s}")
for name, x in [("random", random_field), ("gradient", gradient),
                ("checkerboard", checker)]:
    i, m, s, p = morans_i_pvalue(x, W)
    print(f"  {name:20s} {i:+8.3f} {m:+10.3f} {s:8.3f} {p:7.3f}")

# --- THE WEIGHTS DEFINITION CHANGES THE ANSWER. ---
def knn_weights(coords, k):
    d = np.linalg.norm(coords[:, None, :] - coords[None, :, :], axis=-1)
    np.fill_diagonal(d, np.inf)
    W = np.zeros_like(d)
    idx = np.argsort(d, axis=1)[:, :k]
    np.put_along_axis(W, idx, 1.0, axis=1)
    return W / W.sum(axis=1, keepdims=True)

print("\nSAME DATA, DIFFERENT NEIGHBOUR DEFINITIONS")
print(f"  {'weights':22s} {'I (gradient)':>13s} {'I (checker)':>12s}")
for label, Wx in [("rook (edge-sharing)", W),
                  ("4 nearest neighbours", knn_weights(coords, 4)),
                  ("8 nearest neighbours", knn_weights(coords, 8)),
                  ("20 nearest neighbours", knn_weights(coords, 20))]:
    print(f"  {label:22s} {morans_i(gradient, Wx):+13.3f} "
          f"{morans_i(checker, Wx):+12.3f}")
print("  Widening the neighbourhood dilutes the checkerboard signal to")
print("  nearly zero. W is a hypothesis about the scale of the process,")
print("  and reporting I without reporting W is meaningless.")

# --- LOCAL MORAN'S I: WHERE, NOT JUST WHETHER. ---
def local_morans(x, W):
    z = (x - x.mean()) / x.std()
    lag = W @ z
    return z * lag, z, lag                    # Ii, standardised value, lag

# A field with one genuine hot spot and one isolated high outlier.
field = rng.normal(0, 0.5, n * n)
hot = [(i, j) for i in range(2, 5) for j in range(2, 5)]
for i, j in hot:
    field[i * n + j] += 3.0                   # a cluster of high values
field[9 * n + 9] += 4.0                       # a lone spike

Ii, z, lag = local_morans(field, W)
labels = np.where((z > 0) & (lag > 0), "HH",
          np.where((z < 0) & (lag < 0), "LL",
           np.where((z > 0) & (lag < 0), "HL", "LH")))

print("\nLOCAL MORAN CLASSIFICATION")
print(f"  {'cell':>8s} {'value z':>9s} {'lag z':>8s} {'Ii':>8s}  type")
strongest = list(np.argsort(-np.abs(Ii))[:4])          # the cluster core
outliers = [i for i in np.argsort(-z) if labels[i] == "HL"][:1]   # the spike
for idx in strongest + [o for o in outliers if o not in strongest]:
    r, c = divmod(idx, n)
    print(f"  ({r:2d},{c:2d}) {z[idx]:+9.2f} {lag[idx]:+8.2f} "
          f"{Ii[idx]:+8.2f}  {labels[idx]}")
print("  The 3x3 block shows as HH -- a real cluster. Cell (9,9) shows as")
print("  HL: a high value surrounded by low ones, which is a different")
print("  finding entirely and would be invisible in the global statistic.")

  GLOBAL MORAN'S I ON THREE FIELDS (rook contiguity, 999 permutations)

    expected under the null: -0.0070
    field                       I  null mean  null sd       p
    random                 -0.014     -0.007    0.052   0.618
    gradient               +0.921     -0.007    0.051   0.001
    checkerboard           -0.832     -0.006    0.052   0.001

  SAME DATA, DIFFERENT NEIGHBOUR DEFINITIONS

    weights                 I (gradient)  I (checker)
    rook (edge-sharing)           +0.921       -0.832
    4 nearest neighbours          +0.901       -0.775
    8 nearest neighbours          +0.874       -0.012
    20 nearest neighbours         +0.856       +0.061
    Widening the neighbourhood dilutes the checkerboard signal to
    nearly zero. W is a hypothesis about the scale of the process,
    and reporting I without reporting W is meaningless.

  LOCAL MORAN CLASSIFICATION

        cell   value z    lag z       Ii  type
     ( 3, 3)     +3.08    +3.08    +9.49  HH
     ( 2, 3)     +3.11    +2.25    +7.00  HH
     ( 3, 4)     +3.05    +2.24    +6.83  HH
     ( 4, 3)     +3.02    +2.21    +6.67  HH
     ( 9, 9)     +4.19    -0.24    -1.01  HL
    The 3x3 block shows as HH -- a real cluster. Cell (9,9) shows as
    HL: a high value surrounded by low ones, which is a different
    finding entirely and would be invisible in the global statistic.

The weights table is the point most often skipped in applied work. The checkerboard field goes from I = -0.83 to I = -0.01 purely by changing what counts as a neighbour, with no change to the data. Widening the neighbourhood to eight nearest cells brings in the diagonals, which share the centre cell's parity and cancel the alternation exactly. W encodes an assumption about the spatial scale at which the process operates, and two analysts using different W will reach different conclusions from the same map.

What to do once autocorrelation is confirmed:

Situation	Response
Autocorrelation in the outcome only	May vanish once covariates are added; check residuals
Autocorrelation in the residuals	The model is misspecified; a spatial term is needed
Neighbouring outcomes influence each other	Spatial lag model (SAR): include W y as a predictor
A shared unobserved regional driver	Spatial error model (SEM): W enters the error term
Relationships vary across space	Geographically weighted regression
Just need honest inference	Cluster-robust or spatial HAC standard errors

The diagnostic order matters. Test Moran's I on the residuals, not on the raw outcome. Raw house prices are obviously spatially clustered; the question is whether anything remains clustered after accounting for size, age, and neighbourhood type. If it does, the model is missing something that varies over space.

Finally, a caution that applies to any spatial aggregation: the modifiable areal unit problem. Results computed over census tracts can differ substantially from the same analysis over postcodes or grid cells, because the boundaries are arbitrary and different aggregations mix different populations. Where feasible, report the analysis at more than one spatial resolution and state whether the conclusion is stable.

Key Takeaways

• Spatial data violates the independence assumption, which understates standard errors and overstates significance.
• Moran's I measures whether similar values cluster in space; its null expectation is -1/(n-1), not zero.
• The spatial weights matrix W defines "neighbour" and is a substantive modelling choice; the same data gives different I under different W.
• Permutation tests provide inference for Moran's I without distributional assumptions.
• Local Moran's I identifies where clustering occurs and distinguishes clusters (HH, LL) from spatial outliers (HL, LH).
• Spatial outliers are often the more actionable finding and are invisible in the global statistic.
• Test autocorrelation on model residuals, not the raw outcome; residual clustering means a spatially varying predictor is missing.
• The modifiable areal unit problem means conclusions can depend on the chosen boundaries; check stability across resolutions.

🧪 Practice

1. Generate random, clustered, and checkerboard spatial fields and compute Moran's I for each, confirming the expected signs.
2. Implement a permutation test for Moran's I and report the p-value along with the null distribution's mean and standard deviation.
3. Compute Moran's I for the same dataset under rook, queen, and k-nearest weights for three values of k, and tabulate the results.
4. Fit an ordinary regression to spatial data, test the residuals for autocorrelation, and describe what a significant result implies.
5. Compute local Moran's I and map the HH, LL, HL, and LH classifications; identify at least one spatial outlier.
6. Aggregate the same point data to two different boundary sets and compare the conclusions, as a demonstration of the modifiable areal unit problem.
7. Interview: A colleague reports that a neighbourhood programme significantly reduced crime, using a regression over 200 city blocks. What would you check first? (Hint: consider whether adjacent blocks are independent observations, and what that implies about the effective sample size behind the p-value.)

<a id="135-recommender-systems"></a>

13.5 Recommender Systems

Recommendation is the domain where the gap between offline metrics and real outcomes is widest, because the system changes the behaviour it is trained on: users can only click what they were shown, so every log is a record of the previous model's decisions as much as of user preference; this section covers the two foundational approaches, what changes when feedback is implicit rather than rated, how to handle new users and items, and how to evaluate something that shapes its own training data.

Content-Based Filtering

Content-based filtering recommends items similar to the ones a user already liked, where "similar" is measured using the items' own attributes: genre, brand, price, text description, image, category. The user is represented by a profile built from the items they engaged with, and recommendations are the nearest unseen items to that profile.

The appeal is that it needs no other users. A brand-new platform with one customer can make sensible recommendations on day one, which is impossible for any approach that relies on patterns across a user base.

The pipeline is short:

1. Represent items as feature vectors: TF-IDF over descriptions, one-hot categories, numeric attributes, or learned embeddings.
2. Build a user profile by aggregating the items they liked -- typically a weighted mean, with weights from rating, recency, or engagement depth.
3. Score candidates by similarity to the profile, usually cosine.
4. Filter and rank: remove already-seen items, apply business rules, return the top k.

Two details make the difference between a working system and a naive one.

Feature weighting matters more than the similarity function. If item vectors concatenate a 5,000-dimensional TF-IDF block and three numeric attributes, the text will dominate cosine similarity completely, no matter how important price actually is. Normalise each block, then weight blocks deliberately.

Recency and negative signal. A profile that averages everything a user ever touched represents a person who no longer exists. Weight recent interactions more heavily, and use explicit dislikes if you have them -- subtracting disliked item vectors is crude but effective.

The characteristic failure is over-specialisation, sometimes called the filter bubble. A user who watched three documentaries receives only documentaries, because by construction the system recommends the neighbourhood of what they already consumed. It can never surface the comedy they would have loved, because nothing in their history points there.

import numpy as np
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.preprocessing import StandardScaler, normalize
from sklearn.metrics.pairwise import cosine_similarity

items = pd.DataFrame([
    ("The Deep Field",   "documentary space science telescope cosmology", 2019, 118, 8.1),
    ("Orbital Decay",    "documentary space engineering satellites failure", 2021, 96, 7.4),
    ("Salt Marsh",       "documentary nature wetlands birds conservation", 2020, 88, 7.9),
    ("Night Shift",      "comedy workplace friendship hospital",           2018, 102, 6.8),
    ("Two Left Feet",    "comedy romance dance competition",               2022, 95, 7.1),
    ("The Long Descent", "thriller mountaineering survival isolation",     2021, 130, 7.6),
    ("Signal Lost",      "thriller space survival isolation rescue",       2023, 124, 8.3),
    ("Quiet Harbour",    "drama family fishing village grief",             2017, 110, 7.7),
], columns=["title", "tags", "year", "minutes", "rating"])

# --- BUILD ITEM VECTORS FROM TWO VERY DIFFERENT FEATURE BLOCKS. ---
text = TfidfVectorizer().fit_transform(items["tags"]).toarray()
numeric = StandardScaler().fit_transform(items[["year", "minutes", "rating"]])

print(f"FEATURE BLOCKS  text {text.shape}  numeric {numeric.shape}")
print(f"  text block norm per item   {np.round(np.linalg.norm(text, axis=1), 2)}")
print(f"  numeric block norm per item"
      f" {np.round(np.linalg.norm(numeric, axis=1), 2)}")
print("  Concatenating these directly lets whichever block has the larger")
print("  norm dominate cosine similarity, regardless of its importance.")

def build_item_matrix(text_w=1.0, num_w=1.0):
    """Normalise each block to unit length, THEN weight it. This makes the
    weights mean what you think they mean."""
    return np.hstack([text_w * normalize(text),
                      num_w * normalize(numeric)])

# --- USER PROFILE: A RECENCY-WEIGHTED MEAN OF LIKED ITEMS. ---
def user_profile(item_matrix, liked_idx, ratings=None, half_life=3):
    """liked_idx is ordered oldest to newest. Older interactions decay,
    because a profile averaging five years of history describes nobody."""
    w = np.array([0.5 ** ((len(liked_idx) - 1 - i) / half_life)
                  for i in range(len(liked_idx))])
    if ratings is not None:
        w = w * np.asarray(ratings)
    p = (item_matrix[liked_idx] * w[:, None]).sum(axis=0) / w.sum()
    return p / (np.linalg.norm(p) + 1e-12)

def recommend(item_matrix, liked_idx, k=3, **kw):
    p = user_profile(item_matrix, liked_idx, **kw)
    sims = cosine_similarity(p.reshape(1, -1), item_matrix)[0]
    sims[liked_idx] = -np.inf                    # never re-recommend
    order = np.argsort(-sims)[:k]
    return [(items.title[i], round(float(sims[i]), 3)) for i in order]

# This user watched two space documentaries, then a space thriller.
watched = [0, 1, 6]

print("\nRECOMMENDATIONS UNDER DIFFERENT BLOCK WEIGHTS")
for tw, nw in [(1.0, 0.0), (1.0, 1.0), (0.2, 1.0), (0.0, 1.0)]:
    M = build_item_matrix(tw, nw)
    recs = recommend(M, watched, k=3)
    print(f"  text={tw:.1f} numeric={nw:.1f} -> "
          + ", ".join(f"{t} ({s})" for t, s in recs))
print("  The winner is stable here, but the REASON changes completely:")
print("  with text it matched 'thriller survival isolation'; with numeric")
print("  alone it matched a 2021 film of 130 minutes. On a real catalogue")
print("  those two reasons diverge fast, and only one is about content.")

# --- OVER-SPECIALISATION, MEASURED. ---
M = build_item_matrix(1.0, 0.3)
def catalogue_coverage(item_matrix, n_users=200, k=3, rng=np.random.default_rng(0)):
    """What fraction of the catalogue ever gets recommended to anyone?
    Content-based systems concentrate on whatever is near existing tastes."""
    shown = set()
    intra_sims = []
    for _ in range(n_users):
        liked = list(rng.choice(len(items), 2, replace=False))
        recs = recommend(item_matrix, liked, k=k)
        idx = [items.index[items.title == t][0] for t, _ in recs]
        shown.update(idx)
        S = cosine_similarity(item_matrix[idx])
        intra_sims.append(S[np.triu_indices(len(idx), 1)].mean())
    return len(shown) / len(items), float(np.mean(intra_sims))

cov, intra = catalogue_coverage(M)
print(f"\nDIVERSITY DIAGNOSTICS OVER 200 SIMULATED USERS")
print(f"  catalogue coverage          {cov:.2f}")
print(f"  mean intra-list similarity  {intra:.3f}")
print("  High intra-list similarity means the three recommendations are")
print("  near-duplicates of each other. Coverage is 1.0 on a catalogue of")
print("  eight; on a real catalogue of 50,000 items a content-based system")
print("  typically reaches 10-30% of it, and never shows the rest.")

  FEATURE BLOCKS  text (8, 31)  numeric (8, 3)

    text block norm per item   [1.   1.   1.   1.   1.   1.   1.   1.  ]
    numeric block norm per item [1.39 1.06 1.54 2.1  1.73 1.64 2.39 1.66]
    Concatenating these directly lets whichever block has the larger
    norm dominate cosine similarity, regardless of its importance.

  RECOMMENDATIONS UNDER DIFFERENT BLOCK WEIGHTS

    text=1.0 numeric=0.0 -> The Long Descent (0.372), Salt Marsh (0.198), Night Shift (0.0)
    text=1.0 numeric=1.0 -> The Long Descent (0.355), Salt Marsh (0.152), Two Left Feet (-0.073)
    text=0.2 numeric=1.0 -> The Long Descent (0.338), Salt Marsh (0.105), Two Left Feet (-0.14)
    text=0.0 numeric=1.0 -> The Long Descent (0.337), Salt Marsh (0.105), Two Left Feet (-0.146)
    The winner is stable here, but the REASON changes completely:
    with text it matched 'thriller survival isolation'; with numeric
    alone it matched a 2021 film of 130 minutes. On a real catalogue
    those two reasons diverge fast, and only one is about content.

  DIVERSITY DIAGNOSTICS OVER 200 SIMULATED USERS

    catalogue coverage          1.00
    mean intra-list similarity  0.313
    High intra-list similarity means the three recommendations are
    near-duplicates of each other. Coverage is 1.0 on a catalogue of
    eight; on a real catalogue of 50,000 items a content-based system
    typically reaches 10-30% of it, and never shows the rest.

Property	Content-based
Needs other users	No
New item (cold start)	Handled immediately from its features
New user (cold start)	Needs at least one interaction
Explainability	Strong: "because you watched X"
Serendipity	Poor: recommends the neighbourhood of known taste
Depends on metadata quality	Completely

That last row is the practical constraint. A content-based system is only as good as its item features, and in most real catalogues those features are sparse, inconsistent, or entered by whoever uploaded the item. Before investing in the model, audit the metadata: how many items have a description, how many categories are used by exactly one item, how many fields are filled by default.

The natural complement is collaborative filtering, which ignores item attributes entirely and learns from co-occurrence in user behaviour. Content handles new items; collaboration finds non-obvious connections. Nearly all production systems combine them, which is the subject of a later topic.

Key Takeaways

• Content-based filtering recommends items similar to what a user already engaged with, using the items' own attributes.
• It works with a single user and handles brand-new items immediately, which no collaborative method can.
• Normalise each feature block before weighting, or the highest-dimensional block silently dominates the similarity.
• Weight interactions by recency; a profile averaging years of history describes a user who no longer exists.
• Over-specialisation is structural, not a tuning problem: the method can only recommend near what the user already consumed.
• Track catalogue coverage and intra-list similarity alongside accuracy, or diversity failures go unnoticed.
• The system's ceiling is set by metadata quality; audit it before improving the model.

🧪 Practice

1. Build TF-IDF item vectors for a small catalogue and produce recommendations for a user profile built from three liked items.
2. Concatenate a text block and a numeric block without normalising, then with normalising, and compare the resulting recommendations.
3. Add recency weighting with half-lives of 1, 5, and 20 interactions, and report how the recommendations change for a user whose taste shifted.
4. Compute catalogue coverage and mean intra-list similarity for your recommender over 500 simulated users.
5. Introduce a new item with no interaction history and confirm the system can recommend it; then explain why a collaborative method could not.
6. Interview: A music service using content-based recommendations finds that users stop discovering new genres after a few weeks. Explain the mechanism and propose one mitigation. (Hint: think about what the profile is a weighted average of, and what fraction of recommendations you might deliberately draw from outside the profile's neighbourhood.)

Collaborative Filtering

Collaborative filtering makes a bet that sounds almost too simple: if two people agreed about many items in the past, they will agree about the next one. It uses nothing about the items themselves -- no genre, no description, no price. It only uses the pattern of who interacted with what.

This is why it can find connections no content-based system ever will. If buyers of a particular hiking boot also buy a specific brand of thermal flask, collaborative filtering surfaces that immediately. No attribute shared by the two products explains it; the connection exists only in behaviour.

The data is a user-item matrix, and its defining property is extreme sparsity. A service with 1 million users and 100,000 items where each user rates 50 items has a matrix that is 99.95% empty. Everything about the methods follows from working around that.

  THE USER-ITEM MATRIX (? = unobserved, which is nearly everything)

           item1 item2 item3 item4 item5
    Ana      5     ?     4     ?     1
    Ben      ?     3     ?     5     ?
    Cleo     4     ?     5     ?     2
    Dani     1     5     ?     4     ?

    Ana and Cleo agree on items 1, 3, 5 -> they are neighbours.
    Cleo has not rated item 2; Ana has not either, so neighbours
    cannot help there. Dani rated it 5, but Dani disagrees with Ana
    everywhere else, so that is weak evidence.

Two families:

Neighbourhood methods compute similarities directly. User-based: find users similar to the target, recommend what they liked. Item-based: find items similar to ones the user liked, where similarity is measured by co-rating patterns rather than attributes. Item-based is usually preferred in production because item-item similarities are more stable over time than user-user ones and can be precomputed.

Matrix factorisation learns latent factors. Approximate the m x n rating matrix as R ≈ P Qᵀ, where P is m x k user factors and Q is n x k item factors. Each user and item becomes a vector in a shared k-dimensional space, and a predicted rating is a dot product. The factors are learned, not designed, and often turn out to correspond to interpretable axes -- serious versus light, mainstream versus niche -- though there is no guarantee of that.

The critical implementation detail is that factorisation must be fitted only on observed entries. Treating unobserved cells as zeros teaches the model that every unrated item was disliked, which is false: most unrated items were simply never seen.

Baselines matter enormously and are usually omitted. Much of any rating's value is explained by two offsets: some users rate everything generously, and some items are simply better. Modelling r̂ = μ + b_user + b_item + p_u · q_i rather than the dot product alone typically captures most of the achievable improvement before any latent factor is learned.

import numpy as np

rng = np.random.default_rng(4)

# --- A SYNTHETIC RATING MATRIX WITH KNOWN LATENT STRUCTURE. ---
n_users, n_items, k_true = 600, 300, 4
P_true = rng.normal(0, 1, (n_users, k_true))
Q_true = rng.normal(0, 1, (n_items, k_true))
user_bias = rng.normal(0, 0.6, n_users)      # some people rate generously
item_bias = rng.normal(0, 0.8, n_items)      # some items are just better
mu = 3.6

full = (mu + user_bias[:, None] + item_bias[None, :]
        + 0.4 * P_true @ Q_true.T + rng.normal(0, 0.3, (n_users, n_items)))
full = np.clip(full, 1, 5)

# Observe only 4% of cells -- realistic sparsity.
mask = rng.random((n_users, n_items)) < 0.04
rows, cols = np.nonzero(mask)
vals = full[rows, cols]
print(f"MATRIX {n_users} x {n_items} = {n_users * n_items:,} cells")
print(f"  observed {len(vals):,} ({100 * mask.mean():.1f}%)")

# Hold out 20% of the observed ratings.
perm = rng.permutation(len(vals))
cut = int(0.8 * len(vals))
tr, te = perm[:cut], perm[cut:]

# --- BASELINES BEFORE ANY MODEL. ---
mu_hat = vals[tr].mean()
def rmse(pred, true):
    return float(np.sqrt(np.mean((pred - true) ** 2)))

print(f"\n  {'model':34s} {'params':>9s} {'test RMSE':>10s}")
print(f"  {'global mean':34s} {1:9d} "
      f"{rmse(mu_hat, vals[te]):10.4f}")

# Bias-only model, fitted by alternating least squares with regularisation.
def fit_biases(rows, cols, vals, n_users, n_items, lam=5.0, iters=20):
    mu = vals.mean()
    bu, bi = np.zeros(n_users), np.zeros(n_items)
    for _ in range(iters):
        resid = vals - mu - bi[cols]
        bu = np.bincount(rows, resid, n_users) / (np.bincount(rows, None, n_users) + lam)
        resid = vals - mu - bu[rows]
        bi = np.bincount(cols, resid, n_items) / (np.bincount(cols, None, n_items) + lam)
    return mu, bu, bi

mu_b, bu, bi = fit_biases(rows[tr], cols[tr], vals[tr], n_users, n_items)
pred_bias = mu_b + bu[rows[te]] + bi[cols[te]]
print(f"  {'+ user and item biases':34s} {n_users + n_items + 1:9d} "
      f"{rmse(pred_bias, vals[te]):10.4f}")

# --- MATRIX FACTORISATION BY SGD, ON OBSERVED ENTRIES ONLY. ---
def fit_mf(rows, cols, vals, n_users, n_items, k=8, lr=0.01, lam=0.05,
           epochs=40, rng=rng):
    """Note what is NOT here: any loop over unobserved cells. Treating
    them as zeros would teach the model that unseen means disliked."""
    mu = vals.mean()
    bu, bi = np.zeros(n_users), np.zeros(n_items)
    P = rng.normal(0, 0.1, (n_users, k))
    Q = rng.normal(0, 0.1, (n_items, k))
    idx = np.arange(len(vals))
    for _ in range(epochs):
        rng.shuffle(idx)
        for t in idx:
            u, i, r = rows[t], cols[t], vals[t]
            pred = mu + bu[u] + bi[i] + P[u] @ Q[i]
            e = r - pred
            bu[u] += lr * (e - lam * bu[u])
            bi[i] += lr * (e - lam * bi[i])
            pu = P[u].copy()
            P[u] += lr * (e * Q[i] - lam * P[u])       # update together,
            Q[i] += lr * (e * pu - lam * Q[i])         # using the OLD P[u]
    return mu, bu, bi, P, Q

for k in [2, 8, 32]:
    mu_m, bu_m, bi_m, P, Q = fit_mf(rows[tr], cols[tr], vals[tr],
                                    n_users, n_items, k=k)
    pred = (mu_m + bu_m[rows[te]] + bi_m[cols[te]]
            + np.sum(P[rows[te]] * Q[cols[te]], axis=1))
    n_par = n_users + n_items + 1 + k * (n_users + n_items)
    print(f"  {f'+ {k} latent factors':34s} {n_par:9,} "
          f"{rmse(pred, vals[te]):10.4f}")

# --- THE ZERO-IMPUTATION MISTAKE, QUANTIFIED. ---
dense = np.zeros((n_users, n_items))
dense[rows[tr], cols[tr]] = vals[tr]                 # unobserved -> 0
U, S, Vt = np.linalg.svd(dense, full_matrices=False)
approx = (U[:, :8] * S[:8]) @ Vt[:8]
print(f"  {'SVD with zeros imputed (k=8)':34s} {'--':>9s} "
      f"{rmse(approx[rows[te], cols[te]], vals[te]):10.4f}")
print("\n  The last row is the trap: filling unobserved cells with zeros")
print("  makes the model reconstruct mostly-zero rows, so its predictions")
print("  collapse toward zero on a 1-to-5 scale.")

  MATRIX 600 x 300 = 180,000 cells
    observed 7,213 (4.0%)

    model                                params  test RMSE
    global mean                               1     1.1642
    + user and item biases                  901     0.9431
    + 2 latent factors                    2,701     0.8977
    + 8 latent factors                    8,101     0.8614
    + 32 latent factors                  29,701     0.8792
    SVD with zeros imputed (k=8)             --     3.3812

    The last row is the trap: filling unobserved cells with zeros
    makes the model reconstruct mostly-zero rows, so its predictions
    collapse toward zero on a 1-to-5 scale.

Three things to take from that table.

Biases alone close most of the gap -- from 1.164 to 0.943, while eight latent factors only reach 0.861. In production systems the proportions are similar, which is why a bias model is the baseline any factorisation must beat before it is worth deploying.

Thirty-two factors is worse than eight. With 5,770 training ratings and 29,701 parameters the model has more capacity than data, and regularisation only partly compensates. Latent dimensionality is a capacity knob and must be tuned against the amount of observed data, not set to a round number.

And zero imputation produces an RMSE of 3.38 on a 1-to-5 scale, which is worse than guessing the global mean by a factor of three. "Missing" and "zero" are different statements, and conflating them is the most common implementation error in collaborative filtering.

Method	Scales to millions	Cold start	Explainable	Captures non-obvious links
User-based neighbourhood	Poorly (user-user is O(m²))	No	Yes: "people like you"	Yes
Item-based neighbourhood	Well (precomputable)	No	Yes: "similar to X"	Yes
Matrix factorisation	Well	No	Weakly	Yes
Content-based	Well	Items yes	Strongly	No

Every collaborative row has "No" in the cold-start column, and that is the defining weakness: a new item with no interactions has no row in the matrix and cannot be recommended at all, no matter how good it is.

Two further practical notes. Popularity bias is built in: popular items appear in more co-occurrence patterns and are therefore recommended more, which makes them more popular. And the matrix is not missing at random -- users choose what to rate, and they mostly rate things they expected to like, so the observed entries are a biased sample of preferences.

Key Takeaways

• Collaborative filtering uses only interaction patterns, so it finds connections that no item attribute would reveal.
• The user-item matrix is extremely sparse; every method is an answer to that sparsity.
• Item-based neighbourhoods are usually preferred over user-based because item similarities are stable and precomputable.
• Matrix factorisation learns latent user and item vectors whose dot product predicts the rating.
• Fit only on observed entries; imputing zeros teaches the model that unrated means disliked and destroys prediction quality.
• User and item bias terms capture most of the achievable improvement before any latent factor is learned -- always fit them as the baseline.
• Latent dimensionality is a capacity parameter and can overfit; tune it against the volume of observed ratings.
• Collaborative methods cannot recommend items with no interaction history, and they amplify popularity.

🧪 Practice

1. Build a user-item matrix from a ratings dataset and report its sparsity and the distribution of ratings per user.
2. Implement a bias-only model and a matrix factorisation model, and report how much of the total RMSE improvement the biases alone provide.
3. Fit factorisation with zeros imputed and with observed entries only, and compare the test RMSE.
4. Sweep the number of latent factors from 1 to 64 and plot train and test RMSE to locate the overfitting point.
5. Implement item-based neighbourhood recommendation and compare its top-10 lists against those from factorisation for the same users.
6. Interview: A colleague reports RMSE 0.82 for their recommender and wants to ship it. What do you ask? (Hint: what does a bias-only baseline score on the same split, and does RMSE on held-out ratings measure what the product actually does, which is choose which ten items to display?)

Implicit Feedback Models

Explicit ratings are rare. Most systems never see a star rating; they see clicks, views, plays, purchases, dwell time, and skips. This is implicit feedback, and it differs from explicit ratings in ways that break the standard factorisation recipe.

Three differences matter.

There are no negatives. A user who did not click an item might dislike it, or might never have seen it, or might have scrolled past while distracted. The absence of an interaction is genuinely ambiguous, whereas a one-star rating is unambiguous. This is the central problem.

Signal strength varies. Watching a film once is weak evidence; watching it four times is strong. A purchase outweighs a page view. Implicit data carries a magnitude that is not a preference score but is correlated with confidence.

The data is skewed and noisy. Accidental clicks, autoplay, shared accounts, and bots all produce interactions that reflect no preference at all.

The standard solution, from the implicit-ALS formulation, separates preference from confidence:

  p_ui = 1 if the user interacted with the item, else 0     (preference)

  c_ui = 1 + alpha * r_ui                                   (confidence)
         where r_ui is the raw count (plays, clicks, minutes)

  minimise:  Σ over ALL (u,i)  c_ui * (p_ui - x_u · y_i)²  + regularisation
                    ^^^^^^^^
             including unobserved pairs -- but they carry confidence 1,
             the lowest possible, so the model is only weakly pushed to
             predict 0 for them.

This is the key idea. Every unobserved pair is treated as a zero preference, but with minimal confidence, so a strong signal elsewhere easily overrides it. An item played fifty times gets confidence 1 + 50α and dominates the loss. The formulation says "I believe this user likes this item, strongly" and "I have no idea about this pair, weakly" in the same objective.

Summing over all m x n pairs sounds infeasible, and the reason it is not is a linear-algebra identity: the normal equations decompose into a term computed once from all items plus a correction over only the observed ones. That makes each ALS iteration O(observed * k² + (m+n) * k³) rather than O(m * n * k).

Bayesian Personalised Ranking (BPR) takes a different route. Rather than predicting a value per pair, it optimises the ordering: for each observed item i and a sampled unobserved item j, push the score of i above the score of j. This matches the actual task -- ranking a list -- more directly than any pointwise loss.

import numpy as np
from scipy import sparse

rng = np.random.default_rng(9)

# --- SIMULATE IMPLICIT FEEDBACK: PLAY COUNTS, HEAVILY SKEWED. ---
n_users, n_items, k_true = 800, 400, 5
P_true = rng.normal(0, 1, (n_users, k_true))
Q_true = rng.normal(0, 1, (n_items, k_true))
affinity = 1 / (1 + np.exp(-(P_true @ Q_true.T)))        # true preference
popularity = rng.pareto(1.2, n_items) + 1                # a few huge hits
popularity /= popularity.max()

# Exposure is not random: users mostly see popular items. This is the
# feedback loop that makes implicit data biased before any model runs.
exposure = 0.02 + 0.25 * popularity[None, :]
seen = rng.random((n_users, n_items)) < exposure
plays = np.where(seen & (rng.random((n_users, n_items)) < affinity),
                 rng.geometric(0.4, (n_users, n_items)), 0)

R = sparse.csr_matrix(plays)
print(f"IMPLICIT MATRIX {n_users} x {n_items}")
print(f"  non-zero entries {R.nnz:,} ({100 * R.nnz / (n_users * n_items):.2f}%)")
print(f"  play counts: median {np.median(R.data):.0f}  "
      f"p95 {np.percentile(R.data, 95):.0f}  max {R.data.max()}")
print(f"  top 5% of items account for "
      f"{100 * np.sort(np.asarray(R.sum(0)).ravel())[-20:].sum() / R.sum():.0f}%"
      f" of all plays")

# --- IMPLICIT ALS: PREFERENCE AND CONFIDENCE, SEPARATED. ---
def implicit_als(R, k=16, alpha=40.0, lam=0.1, iters=15):
    """R holds raw counts. p = 1 where count > 0; c = 1 + alpha * count.
    The trick that makes this tractable: YtY is computed ONCE per
    iteration over all items, then corrected per user over their few
    observed items."""
    m, n = R.shape
    X = rng.normal(0, 0.01, (m, k))
    Y = rng.normal(0, 0.01, (n, k))
    Rt = R.T.tocsr()
    I = np.eye(k)

    def solve_side(A, Bmat, Rside):
        BtB = Bmat.T @ Bmat                       # the "all pairs" term
        for u in range(A.shape[0]):
            s, e = Rside.indptr[u], Rside.indptr[u + 1]
            idx, cnt = Rside.indices[s:e], Rside.data[s:e]
            if len(idx) == 0:
                continue
            Bu = Bmat[idx]
            cu = alpha * cnt                      # confidence MINUS the base 1
            # (BtB + Bu^T diag(cu) Bu + lam I) x = Bu^T (1 + cu)
            Aleft = BtB + Bu.T @ (Bu * cu[:, None]) + lam * I
            Aright = Bu.T @ (1.0 + cu)
            A[u] = np.linalg.solve(Aleft, Aright)
        return A

    for _ in range(iters):
        X = solve_side(X, Y, R)
        Y = solve_side(Y, X, Rt)
    return X, Y

# --- EVALUATE AS A RANKING PROBLEM, NOT A REGRESSION. ---
# Hold out one played item per user: the "leave-one-out" protocol.
held = {}
R_train = R.tolil(copy=True)
for u in range(n_users):
    items_u = R.indices[R.indptr[u]:R.indptr[u + 1]]
    if len(items_u) >= 3:
        pick = int(rng.choice(items_u))
        held[u] = pick
        R_train[u, pick] = 0
R_train = R_train.tocsr()
R_train.eliminate_zeros()

def evaluate(score_fn, k=10):
    hits, rr, ndcg = 0, 0.0, 0.0
    for u, true_item in held.items():
        seen_tr = set(R_train.indices[R_train.indptr[u]:R_train.indptr[u + 1]])
        scores = score_fn(u)
        scores[list(seen_tr)] = -np.inf           # never re-recommend
        top = np.argsort(-scores)[:k]
        if true_item in top:
            hits += 1
            rank = int(np.where(top == true_item)[0][0]) + 1
            rr += 1 / rank
            ndcg += 1 / np.log2(rank + 1)
    n = len(held)
    return hits / n, rr / n, ndcg / n

pop_scores = np.asarray(R_train.sum(0)).ravel().astype(float)
X, Y = implicit_als(R_train, k=16, alpha=40.0)

print(f"\nRANKING METRICS, LEAVE-ONE-OUT OVER {len(held)} USERS, k=10")
print(f"  {'model':24s} {'HR@10':>7s} {'MRR@10':>8s} {'nDCG@10':>9s}")
for name, fn in [("popularity", lambda u: pop_scores.copy()),
                 ("implicit ALS (a=40)", lambda u: (Y @ X[u]).copy())]:
    hr, mrr, nd = evaluate(fn)
    print(f"  {name:24s} {hr:7.3f} {mrr:8.3f} {nd:9.3f}")

# --- ALPHA CONTROLS HOW MUCH THE COUNTS MATTER. ---
print("\nEFFECT OF THE CONFIDENCE SCALING alpha")
print(f"  {'alpha':>7s} {'HR@10':>7s} {'nDCG@10':>9s}  interpretation")
notes = {1: "counts barely matter", 10: "mild weighting",
         40: "standard starting point", 200: "a single replay dominates"}
for a in [1, 10, 40, 200]:
    Xa, Ya = implicit_als(R_train, k=16, alpha=float(a))
    hr, _, nd = evaluate(lambda u: (Ya @ Xa[u]).copy())
    print(f"  {a:7d} {hr:7.3f} {nd:9.3f}  {notes[a]}")

  IMPLICIT MATRIX 800 x 400
    non-zero entries 4,118 (1.29%)
    play counts: median 2  p95 6  max 14
    top 5% of items account for 54% of all plays

  RANKING METRICS, LEAVE-ONE-OUT OVER 521 USERS, k=10

    model                      HR@10   MRR@10   nDCG@10
    popularity                 0.180    0.072     0.097
    implicit ALS (a=40)        0.310    0.139     0.178

  EFFECT OF THE CONFIDENCE SCALING alpha

      alpha   HR@10   nDCG@10  interpretation
          1   0.246     0.139  counts barely matter
         10   0.292     0.167  mild weighting
         40   0.310     0.178  standard starting point
        200   0.283     0.161  a single replay dominates

The popularity baseline is the one that keeps people honest. Recommending the ten most-played items to everybody, with no personalisation at all, achieves HR@10 of 0.180 here -- seven times better than random. Any implicit model must be measured against it, because a substantial share of what looks like learned preference is actually learned popularity. A reported HR@10 of 0.20 sounds respectable and would be worse than doing nothing.

The alpha sweep shows the shape of the trade. Too low and the model ignores how strongly a user engaged; too high and a single accidental replay carries as much weight as a genuine favourite. Values between 10 and 50 are the usual operating range, and the parameter should be tuned, not copied.

Aspect	Explicit ratings	Implicit feedback
Negative signal	Present and unambiguous	Absent; non-interaction is ambiguous
Volume	Scarce	Abundant
Noise	Lower	High: accidents, autoplay, bots
Target	Predict a value	Rank a list
Loss	Squared error on observed cells	Confidence-weighted, or pairwise ranking
Metric	RMSE, MAE	HR@k, MRR, nDCG@k, coverage
Evaluation split	Random held-out ratings	Leave-one-out or time-based

Two operational cautions.

Deduplicate and cap. Raw counts contain autoplay loops and bot traffic. Capping counts at a percentile, or using log(1 + count) instead of the raw value, removes most of the damage and usually improves results.

Distinguish the interaction types. A purchase, a wishlist add, a click, and a three-second view are not the same evidence. Either model them separately or map them to explicit confidence weights, rather than pooling them into one count.

Key Takeaways

• Implicit feedback has no true negatives: a missing interaction may mean dislike or may mean the item was never seen.
• The implicit-ALS formulation separates binary preference from a confidence weight derived from interaction counts.
• Unobserved pairs are included in the loss but with minimal confidence, so observed signal easily overrides them.
• An algebraic decomposition makes summing over all pairs tractable, which is why implicit ALS scales.
• BPR optimises pairwise ordering instead of pointwise values, matching the ranking task more directly.
• Evaluate with ranking metrics (HR@k, MRR, nDCG@k), never RMSE.
• Always report a popularity baseline; much of an implicit model's apparent skill is learned popularity.
• Cap or log-transform raw counts, and treat different interaction types as different evidence rather than pooling them.

🧪 Practice

1. Convert an explicit ratings dataset to implicit by thresholding, and compare what a pointwise squared-error model and a ranking model learn.
2. Implement the confidence formula c = 1 + alpha * r and sweep alpha, reporting HR@10 and nDCG@10 for each value.
3. Build a popularity-only recommender and report how much of your model's HR@10 it already accounts for.
4. Compare leave-one-out and time-based evaluation splits on the same implicit dataset and explain any difference.
5. Cap play counts at the 95th percentile and compare ranking metrics before and after.
6. Interview: Your implicit recommender has HR@10 of 0.22 and the team is pleased. What single number would change your assessment? (Hint: think about what a completely unpersonalised system would score on the same evaluation, and what that implies about how much personalisation is actually happening.)

Hybrid Recommenders

Content-based and collaborative filtering fail in complementary ways. Content-based cannot surface anything outside a user's existing taste neighbourhood but handles new items instantly. Collaborative finds non-obvious connections but is helpless about items nobody has touched. Hybrids combine them so that each covers the other's gap.

The combination strategies, roughly in order of engineering cost:

Strategy	Mechanism	When it fits
Weighted	score = w₁·content + w₂·collab	Both signals always available
Switching	Pick one model per request based on a rule	Cold-start users or items
Cascade	One model generates candidates, another reranks	Large catalogues; the standard shape
Feature-augmented	One model's output is a feature in the other	When you already have a ranker
Mixed	Show results from both in one list	Explicit diversity requirements
Meta-level	One model's learned representation feeds the other	Rich content signals

The cascade pattern -- usually called retrieval-then-ranking -- is what virtually every large production system actually uses, for a reason that is purely computational:

  THE TWO-STAGE ARCHITECTURE

  CATALOGUE (10^6 - 10^8 items)
        |
        v
  RETRIEVAL / CANDIDATE GENERATION        several cheap sources in parallel:
    ~500-2000 candidates                    - collaborative nearest neighbours
    must be FAST: sub-linear,                - content similarity
    approximate is fine                      - recently popular
        |                                    - editorially curated
        v                                    - the user's own history
  RANKING
    ~50-200 items scored                    an expensive model with rich
    an accurate model is affordable          features: context, time of day,
    here because the set is small            device, session, cross-features
        |
        v
  RE-RANKING / POLICY
    final 10-20 shown                       diversity, freshness, business
                                             rules, exploration slots

  Each stage trades recall for precision. Anything the retrieval stage
  misses can never be recommended, no matter how good the ranker is.

That last line is the single most important operational fact about recommender architecture. Ranking quality is capped by retrieval recall, and teams routinely spend months tuning the ranker while the retrieval stage quietly discards the right answer.

Two-tower models are the modern retrieval workhorse. A user tower consumes user features and context; an item tower consumes item features. Both output vectors in one shared space, trained so that positives have high dot product. Because the towers are independent, all item vectors can be precomputed and indexed for approximate nearest-neighbour search, and only the user tower runs at request time. This is what makes personalised retrieval over a hundred million items possible in a few milliseconds.

import time
import numpy as np

rng = np.random.default_rng(12)

# --- A HYBRID SCORER WITH FALLBACK LOGIC. ---
class HybridRecommender:
    """Weighted hybrid with switching for cold start. The weights are not
    fixed: they depend on how much evidence each component has."""

    def __init__(self, min_interactions=5, min_item_events=20):
        self.min_interactions = min_interactions
        self.min_item_events = min_item_events

    def component_weights(self, n_user_events, n_item_events):
        """Confidence in the collaborative signal grows with evidence.
        Below the thresholds, fall back to content and popularity."""
        collab_conf = min(n_user_events / self.min_interactions, 1.0)
        item_conf = min(n_item_events / self.min_item_events, 1.0)
        w_collab = collab_conf * item_conf
        w_content = 1.0 - w_collab
        return w_collab, w_content

    def score(self, collab_score, content_score, popularity, n_user_events,
              n_item_events, pop_weight=0.1):
        wc, wt = self.component_weights(n_user_events, n_item_events)
        return (wc * collab_score + wt * content_score
                + pop_weight * popularity)

rec = HybridRecommender()
print("HOW THE BLEND SHIFTS WITH AVAILABLE EVIDENCE")
print(f"  {'user events':>12s} {'item events':>12s} {'w_collab':>9s} "
      f"{'w_content':>10s}  situation")
cases = [(0, 500, "brand new user"), (2, 500, "two clicks in"),
         (5, 500, "established user"), (50, 3, "new item"),
         (50, 500, "both well known"), (1, 1, "cold start both sides")]
for nu, ni, label in cases:
    wc, wt = rec.component_weights(nu, ni)
    print(f"  {nu:12d} {ni:12d} {wc:9.2f} {wt:10.2f}  {label}")

# --- WHY RETRIEVAL RECALL CAPS EVERYTHING. ---
# Simulate: a perfect ranker cannot recover what retrieval never returned.
n_items = 50_000

def retrieval_recall(n_candidates, quality):
    """quality in [0,1]: how well the retriever concentrates probability
    mass on the true item. 0 = uniform random candidates."""
    p_random = n_candidates / n_items
    return min(quality + (1 - quality) * p_random, 1.0)

print("\nEND-TO-END RECALL@10 UNDER A PERFECT RANKER")
print(f"  {'candidates':>11s} {'retrieval recall':>17s} "
      f"{'ranker top-10 recall':>21s} {'end-to-end':>11s}")
for n_cand, q in [(50, 0.31), (200, 0.52), (1000, 0.71), (5000, 0.83)]:
    r = retrieval_recall(n_cand, q)
    # An oracle ranker puts the true item first whenever it is present.
    print(f"  {n_cand:11,} {r:17.3f} {1.000:21.3f} {r:11.3f}")
print("  The ranker column is 1.000 -- a perfect model -- and end-to-end")
print("  recall still tops out at 0.847. Everything above that line is")
print("  a retrieval problem, and no amount of ranker work touches it.")

# --- A TWO-TOWER RETRIEVER, IN OUTLINE. ---
def two_tower_scores(user_feats, item_feats, Wu, Wi):
    """Each tower maps its own features into a shared space. The item
    tower output is precomputed and indexed; only the user tower runs
    at request time, which is what makes this fast enough to deploy."""
    u = np.tanh(user_feats @ Wu)
    v = np.tanh(item_feats @ Wi)
    u /= np.linalg.norm(u, axis=-1, keepdims=True) + 1e-9
    v /= np.linalg.norm(v, axis=-1, keepdims=True) + 1e-9
    return u @ v.T

d_u, d_i, d_shared = 24, 32, 16
Wu = rng.normal(0, 0.3, (d_u, d_shared))
Wi = rng.normal(0, 0.3, (d_i, d_shared))
users = rng.normal(0, 1, (5, d_u))
items = rng.normal(0, 1, (100_000, d_i))

t0 = time.perf_counter()
item_index = np.tanh(items @ Wi)                       # done OFFLINE, once
item_index /= np.linalg.norm(item_index, axis=1, keepdims=True) + 1e-9
offline_s = time.perf_counter() - t0

t0 = time.perf_counter()
u = np.tanh(users @ Wu)
u /= np.linalg.norm(u, axis=1, keepdims=True) + 1e-9
top = np.argpartition(-(u @ item_index.T), 200, axis=1)[:, :200]
online_s = time.perf_counter() - t0

print(f"\nTWO-TOWER RETRIEVAL OVER {len(items):,} ITEMS")
print(f"  offline item indexing  {offline_s * 1000:8.1f} ms  (once, then cached)")
print(f"  online for 5 users     {online_s * 1000:8.1f} ms  "
      f"({online_s * 1000 / 5:.2f} ms each)")
print("  The item tower never runs at request time. That separation is")
print("  the entire reason two-tower models are deployable.")

  HOW THE BLEND SHIFTS WITH AVAILABLE EVIDENCE

     user events  item events  w_collab  w_content  situation
               0          500      0.00       1.00  brand new user
               2          500      0.40       0.60  two clicks in
               5          500      1.00       0.00  established user
              50            3      0.15       0.85  new item
              50          500      1.00       0.00  both well known
               1            1      0.01       0.99  cold start both sides

  END-TO-END RECALL@10 UNDER A PERFECT RANKER

     candidates  retrieval recall  ranker top-10 recall  end-to-end
             50             0.311                 1.000       0.311
            200             0.522                 1.000       0.522
          1,000             0.716                 1.000       0.716
          5,000             0.847                 1.000       0.847
    The ranker column is 1.000 -- a perfect model -- and end-to-end
    recall still tops out at 0.847. Everything above that line is
    a retrieval problem, and no amount of ranker work touches it.

  TWO-TOWER RETRIEVAL OVER 100,000 ITEMS

    offline item indexing      31.4 ms  (once, then cached)
    online for 5 users          7.2 ms  (1.44 ms each)
    The item tower never runs at request time. That separation is
    the entire reason two-tower models are deployable.

The middle table is the one to internalise. Given a perfect ranker -- one that always places the correct item first -- end-to-end recall is exactly retrieval recall. Improving the ranker from good to excellent moves nothing if the right item was never among the 200 candidates. Before a ranking project, measure retrieval recall at your candidate count; if it is 0.5, half your possible wins are inaccessible until retrieval improves.

The weighting table shows the other half of the hybrid design: the blend should depend on evidence, not be a fixed constant. A user with zero history gets pure content and popularity; a user with fifty interactions on a well-known item gets pure collaborative. The transition is continuous, which avoids the discontinuity a hard switching rule would produce at the threshold.

A few design cautions:

• Do not tune the weights on offline metrics alone. Hybrid weights change the distribution of what gets shown, and offline evaluation is scored against logs generated by the previous distribution. Weights that look optimal offline routinely underperform online.
• Keep the sources separable. When recommendations go wrong, you need to know which retrieval source produced the bad item. Tag every candidate with its source and log it.
• Reserve an exploration slot. If every slot goes to the highest-scoring candidates, the system never collects evidence about anything it does not already favour, and the training data narrows over time.

Key Takeaways

• Hybrids exist because content-based and collaborative methods fail in complementary ways.
• Retrieval-then-ranking is the standard production architecture: cheap approximate candidate generation, then an expensive accurate model on a small set.
• End-to-end quality is capped by retrieval recall; a perfect ranker cannot recover an item retrieval never returned.
• Measure retrieval recall at your candidate count before investing in ranking improvements.
• Two-tower models allow item vectors to be precomputed and indexed, so only the user tower runs per request.
• Blend weights should depend on how much evidence each component has, giving a continuous transition rather than a hard switch.
• Tag candidates with their source so failures can be attributed.
• Reserve exploration slots, or the system's training data narrows to what it already believes.

🧪 Practice

1. Implement a weighted hybrid whose weights depend on interaction counts, and tabulate the blend across cold-start and established cases.
2. Measure retrieval recall at candidate counts of 50, 200, and 1,000 on a real dataset, and compute the end-to-end ceiling for each.
3. Build a two-stage system with a cheap retriever and an expensive ranker, and report latency and recall at each stage.
4. Implement a two-tower scorer, precompute the item index, and measure request-time latency with and without the precomputation.
5. Compare a hard switching rule against a continuous weighting at the cold-start threshold, and describe the discontinuity.
6. Interview: A team has spent a quarter improving their ranking model with no measurable product impact. What would you look at first? (Hint: think about which stage of the pipeline determines the set of items the ranker ever sees, and what metric would reveal whether that stage is the bottleneck.)

Cold Start Problem

Cold start is the recommender's bootstrapping problem: the system needs interaction history to make good recommendations, and it needs good recommendations to generate interaction history. Three distinct versions exist, and conflating them leads to solving the wrong one.

Variant	Situation	Why it is hard
New user	A person with no history	No signal about their preferences
New item	An item nobody has interacted with	No row in the co-occurrence matrix
New system	A launch with neither users nor items	No collaborative signal exists at all

New item is the most tractable and the most commercially urgent -- a retailer adding 500 products a week cannot wait for each to accumulate interactions. The fix is to place new items in the same latent space as existing ones using their content features, so they are recommendable immediately.

The clean formulation: learn a mapping f: content features -> latent factors, trained on items that do have learned factors. For a new item, predict its factors from its metadata. This gives a usable vector on day one, which is gradually replaced by the learned one as interactions arrive.

New user requires acquiring information, and the options form a ladder of increasing intrusiveness:

1. Popularity and recency. Recommend what is broadly liked. Weak, free, and better than nothing.
2. Context. Device, locale, time of day, referral source. Available before any interaction and surprisingly predictive.
3. Onboarding. Ask directly: pick three genres, follow five accounts. Effective and costly in conversion -- every question loses some users.
4. Active learning. Ask about the items that are most informative, not the most popular. An item everyone likes tells you nothing; an item that splits the population tells you a great deal.
5. Fast adaptation. Update the user's representation after every interaction rather than in the nightly batch.

The active-learning idea deserves emphasis because it is the one people miss. Asking a new user whether they like the single most popular film is close to useless, because almost everyone says yes. Asking about a divisive film partitions the user space in half. The right criterion is variance of opinion, not mean.

Exploration is the systematic answer, and it is a multi-armed bandit problem: every recommendation is a choice between exploiting what you already believe and exploring to learn more. Thompson sampling and UCB give principled ways to allocate that budget, and the practical version is simpler -- reserve a fraction of slots for items the system is uncertain about.

import numpy as np
from sklearn.linear_model import Ridge

rng = np.random.default_rng(15)

# --- ACTIVE LEARNING: WHICH QUESTION TO ASK A NEW USER. ---
# Simulated population: each user has a hidden 2-D taste vector.
n_pop = 4000
taste = rng.normal(0, 1, (n_pop, 2))
items = {
    "blockbuster":   (np.array([0.1, 0.1]), 2.2),   # (direction, intercept)
    "divisive cult": (np.array([1.8, -0.4]), 0.0),
    "niche art":     (np.array([0.9, 1.5]), -1.6),
    "universal hit": (np.array([0.05, 0.0]), 3.0),
}

def like_probability(taste, direction, intercept):
    return 1 / (1 + np.exp(-(taste @ direction + intercept)))

print("WHICH ITEM SHOULD YOU ASK A NEW USER ABOUT?")
print(f"  {'item':16s} {'P(like)':>9s} {'variance':>9s} "
      f"{'bits of information':>20s}")
for name, (d, b) in items.items():
    p = like_probability(taste, d, b)
    mean_p = p.mean()
    # Entropy of the ANSWER is how much you learn from asking.
    H = -(mean_p * np.log2(mean_p + 1e-12)
          + (1 - mean_p) * np.log2(1 - mean_p + 1e-12))
    print(f"  {name:16s} {mean_p:9.3f} {p.var():9.3f} {H:20.3f}")
print("  The universal hit is the worst question: everyone says yes, so")
print("  the answer carries almost no information. The divisive item")
print("  splits the population and is worth a full bit.")

# --- HOW FAST DOES A PROFILE CONVERGE? ---
def posterior_after_answers(taste_pop, asked, answers):
    """Keep the population members whose predicted answers match, and use
    their mean taste as the new user's estimate. A crude but faithful
    picture of what any cold-start onboarding is doing."""
    keep = np.ones(len(taste_pop), dtype=bool)
    for (d, b), ans in zip(asked, answers):
        p = like_probability(taste_pop, d, b)
        keep &= (p > 0.5) == bool(ans)
    if keep.sum() == 0:
        return taste_pop.mean(0), 0
    return taste_pop[keep].mean(0), int(keep.sum())

true_user = np.array([1.4, -0.9])
order_informative = ["divisive cult", "niche art", "blockbuster", "universal hit"]
order_popular = ["universal hit", "blockbuster", "niche art", "divisive cult"]

print("\nPROFILE ERROR AFTER EACH ONBOARDING QUESTION")
print(f"  {'questions':>10s} {'informative-first':>19s} {'popular-first':>15s}")
for n_q in range(0, 5):
    row = []
    for order in (order_informative, order_popular):
        asked = [items[k] for k in order[:n_q]]
        answers = [int(like_probability(true_user[None, :], d, b)[0] > 0.5)
                   for d, b in asked]
        est, n_kept = posterior_after_answers(taste, asked, answers)
        row.append(np.linalg.norm(est - true_user))
    print(f"  {n_q:10d} {row[0]:19.3f} {row[1]:15.3f}")
print("  Asking the informative question first nearly halves the error in")
print("  one step. Question ORDER matters as much as question count, and")
print("  every extra question costs conversion.")

# --- NEW ITEM: PREDICT LATENT FACTORS FROM CONTENT. ---
n_items, k, d_content = 2000, 12, 40
content = rng.normal(0, 1, (n_items, d_content))
true_map = rng.normal(0, 0.4, (d_content, k))
factors = content @ true_map + rng.normal(0, 0.5, (n_items, k))  # learned factors

warm = np.arange(1600)           # items with enough interactions
cold = np.arange(1600, 2000)     # brand new items

mapper = Ridge(alpha=1.0).fit(content[warm], factors[warm])
pred_cold = mapper.predict(content[cold])

def cosine_rows(A, B):
    A = A / (np.linalg.norm(A, axis=1, keepdims=True) + 1e-9)
    B = B / (np.linalg.norm(B, axis=1, keepdims=True) + 1e-9)
    return np.sum(A * B, axis=1)

mean_vec = np.repeat(factors[warm].mean(0)[None, :], len(cold), axis=0)
print("\nNEW-ITEM FACTORS PREDICTED FROM CONTENT")
print(f"  {'strategy':30s} {'cosine to true factors':>24s}")
print(f"  {'global mean vector':30s} "
      f"{cosine_rows(mean_vec, factors[cold]).mean():24.3f}")
print(f"  {'content -> factor regression':30s} "
      f"{cosine_rows(pred_cold, factors[cold]).mean():24.3f}")
print("  A cold item goes from 'indistinguishable from average' to a")
print("  usable position in the latent space, purely from metadata.")

  WHICH ITEM SHOULD YOU ASK A NEW USER ABOUT?

    item              P(like)  variance   bits of information
    blockbuster         0.900     0.000                 0.469
    divisive cult       0.500     0.106                 1.000
    niche art           0.254     0.061                 0.818
    universal hit       0.953     0.000                 0.274
    The universal hit is the worst question: everyone says yes, so
    the answer carries almost no information. The divisive item
    splits the population and is worth a full bit.

  PROFILE ERROR AFTER EACH ONBOARDING QUESTION

     questions   informative-first   popular-first
             0               1.664           1.664
             1               0.956           1.663
             2               0.781           1.662
             3               0.779           0.954
             4               0.779           0.779
    Asking the informative question first nearly halves the error in
    one step. Question ORDER matters as much as question count, and
    every extra question costs conversion.

  NEW-ITEM FACTORS PREDICTED FROM CONTENT

    strategy                         cosine to true factors
    global mean vector                               -0.004
    content -> factor regression                      0.981
    A cold item goes from 'indistinguishable from average' to a
    usable position in the latent space, purely from metadata.

Two results worth carrying forward.

Asking about the universal hit is worth 0.274 bits and asking about the divisive item is worth a full bit -- and the profile-error table shows what that means in practice: the popular-first ordering wastes two questions learning nothing. Since every onboarding question costs some fraction of users who abandon, the order is not a refinement, it is most of the value.

And the content-to-factor mapping takes a new item from cosine -0.004 (indistinguishable from the catalogue average) to 0.981 against its eventual learned factors, before a single interaction exists. That is the standard production answer to new-item cold start, and it is a small regression rather than a new architecture.

  COLD-START REMEDIES BY VARIANT

  NEW ITEM                        NEW USER                  NEW SYSTEM
  -----------------------------   ------------------------  -------------------
  content -> factor mapping       popularity + recency      content-based only
  category/brand priors           context (device, locale)  editorial curation
  editorial boost for a window    onboarding questions      manual rules
  exploration slots in results    active learning           import external data
  decay the prior as data arrives fast per-event updates    buy or bootstrap logs

One more consideration that is easy to miss: cold start is not only a launch-day problem. It recurs continuously. Every new product, every returning user whose taste has drifted, every expansion into a new market re-creates it. Treat cold-start handling as a permanent component of the system, with its own monitoring -- what fraction of the catalogue has fewer than n interactions, and how does recommendation quality vary with item age?

Key Takeaways

• Cold start has three distinct variants -- new user, new item, new system -- with different remedies.
• New items are handled by mapping content features into the learned latent space, giving a usable vector before any interaction.
• New users are handled by a ladder from popularity, through context and onboarding, to active learning and fast adaptation.
• The informativeness of an onboarding question is the entropy of its answer: divisive items teach far more than universally liked ones.
• Question order matters as much as question count, because every extra question costs conversion.
• Exploration is the systematic treatment: reserve slots for items the system is uncertain about, or it never learns beyond its current beliefs.
• Cold start recurs continuously with new items, drifting users, and new markets; monitor it rather than treating it as a launch problem.

🧪 Practice

1. Compute the entropy of the answer for several candidate onboarding items and rank them by informativeness.
2. Simulate onboarding with informative-first and popular-first orderings and plot profile error against the number of questions.
3. Train a regression from item content features to learned latent factors and evaluate it on held-out items by cosine similarity.
4. Implement an epsilon-greedy exploration slot in a recommender and measure its effect on catalogue coverage over simulated sessions.
5. Report the distribution of interactions per item in a real dataset and the share of the catalogue below a cold-start threshold.
6. Interview: A marketplace adds 2,000 listings a day, and sellers complain that new listings get no traffic. Describe a solution and its cost. (Hint: think about what signal a brand-new listing does have, and what you would have to give up in the results page to guarantee it some exposure.)

Offline and Online Evaluation

Recommender evaluation is harder than it looks, for one structural reason: the system generates the data it is evaluated on. A user can only click what they were shown, so every log records the previous model's decisions as much as the user's preferences. Offline evaluation on those logs asks "would this new model have agreed with the old one", which is not the question anyone cares about.

That does not make offline evaluation useless. It makes it a filter rather than a verdict: cheap enough to run on every change, good enough to eliminate clearly worse models, and never sufficient to declare a winner.

Offline metrics, and what each actually measures:

Metric	Measures	Blind spot
RMSE / MAE	Rating prediction error	Nobody sees ratings; they see a list
Precision@k	Share of the top k that were relevant	Ignores order within the k
Recall@k	Share of relevant items captured in top k	Ignores order; unfair to short lists
MAP	Precision averaged over recall levels	Binary relevance only
nDCG@k	Gain discounted by position	Needs graded relevance to shine
MRR	Position of the first relevant item	Ignores everything after the first
Coverage	Fraction of catalogue ever recommended	Says nothing about quality
Intra-list similarity	Redundancy within one list	Says nothing about relevance
Novelty	Inverse popularity of recommended items	Can be gamed by recommending junk

The split matters as much as the metric. A random split of interactions leaks the future: the model sees a user's later behaviour while predicting their earlier behaviour. Use a temporal split -- train on everything before time T, test after -- or leave-one-out on each user's most recent interaction.

Position bias is the specific mechanism that corrupts offline numbers. Items shown at the top of a list get clicked more regardless of relevance. An item at position 1 might get five times the clicks of the same item at position 10. Training on raw click logs therefore teaches the model to reproduce the previous ranking, not to find better items.

The standard correction is inverse propensity scoring: weight each observed click by 1 / P(the item was shown at that position). Clicks on items that were unlikely to be seen count for more.

Online evaluation is the verdict. A/B tests measure what actually changed, and the metric hierarchy matters:

  METRIC LADDER, SHALLOW TO DEEP

  CTR                   easy to move, easy to game with clickbait
     |
  session engagement    time spent, items consumed per session
     |
  retention             do users come back next week, next month
     |
  business outcome      revenue, subscription renewal, supply-side health

  A change that lifts CTR by 8% and reduces 30-day retention is a
  losing change. Optimise the deepest metric you can measure reliably
  within the time you have.

import numpy as np
from scipy.stats import norm, spearmanr

rng = np.random.default_rng(21)

# --- POSITION BIAS: WHY RAW LOGS MISLEAD. ---
# Examination probability falls sharply with position.
def examination_prob(position, decay=0.65):
    return decay ** (position - 1)

n_positions = 10
print("POSITION BIAS IN A LOGGED RANKING")
print(f"  {'position':>9s} {'P(examined)':>12s} "
      f"{'clicks if relevance=0.30':>26s}")
for pos in range(1, n_positions + 1):
    e = examination_prob(pos)
    print(f"  {pos:9d} {e:12.3f} {e * 0.30:26.3f}")
print("  The same item with identical relevance collects 48 times fewer")
print("  clicks at position 10 than at position 1. Raw CTR by position")
print("  measures the layout, not the items.")

# --- INVERSE PROPENSITY SCORING RECOVERS THE TRUE RANKING. ---
n_items = 10
true_relevance = rng.uniform(0.1, 0.6, n_items)
logged_order = np.argsort(-rng.random(n_items))      # last model's arbitrary order

n_sessions = 200_000
clicks = np.zeros(n_items)
impressions = np.zeros(n_items)
propensity = np.zeros(n_items)
for pos, item in enumerate(logged_order, start=1):
    e = examination_prob(pos)
    impressions[item] = n_sessions
    propensity[item] = e
    clicks[item] = rng.binomial(n_sessions, e * true_relevance[item])

naive_ctr = clicks / impressions                      # biased by position
ips_ctr = (clicks / impressions) / propensity         # corrected

print("\nESTIMATING RELEVANCE FROM LOGGED CLICKS")
print(f"  {'item':>5s} {'shown at':>9s} {'true rel':>9s} {'naive CTR':>10s} "
      f"{'IPS est':>9s}")
for item in np.argsort(-true_relevance):
    pos = int(np.where(logged_order == item)[0][0]) + 1
    print(f"  {item:5d} {pos:9d} {true_relevance[item]:9.3f} "
          f"{naive_ctr[item]:10.3f} {ips_ctr[item]:9.3f}")
print(f"\n  rank correlation with truth -- naive "
      f"{spearmanr(naive_ctr, true_relevance).statistic:.3f}"
      f"   IPS {spearmanr(ips_ctr, true_relevance).statistic:.3f}")

# --- A/B TEST SIZING FOR A RECOMMENDER CHANGE. ---
def sample_size_per_arm(baseline, mde_rel, alpha=0.05, power=0.8):
    """Two-proportion test. mde_rel is the relative lift you want to detect,
    so 0.02 means a 2% relative improvement."""
    p1 = baseline
    p2 = baseline * (1 + mde_rel)
    pbar = (p1 + p2) / 2
    za, zb = norm.ppf(1 - alpha / 2), norm.ppf(power)
    num = (za * np.sqrt(2 * pbar * (1 - pbar))
           + zb * np.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
    return int(np.ceil(num / (p2 - p1) ** 2))

print("\nUSERS PER ARM TO DETECT A CTR LIFT (alpha=0.05, power=0.80)")
print(f"  {'baseline CTR':>13s} " +
      " ".join(f"{f'+{int(m*100)}%':>10s}" for m in [0.01, 0.02, 0.05, 0.10]))
for base in [0.02, 0.05, 0.15]:
    row = " ".join(f"{sample_size_per_arm(base, m):10,}"
                   for m in [0.01, 0.02, 0.05, 0.10])
    print(f"  {base:13.2f} {row}")
print("  Detecting a 1% relative lift on a 2% baseline needs millions of")
print("  users per arm. Most recommender changes are simply not")
print("  measurable at the precision teams assume they are.")

# --- OFFLINE AND ONLINE DISAGREE: THE CENTRAL PROBLEM. ---
print("\nWHY OFFLINE RANKINGS DO NOT TRANSFER")
print(f"  {'model':26s} {'offline nDCG@10':>16s} {'online CTR lift':>16s}")
observed = [("baseline",                 0.284, "--"),
            ("deeper ranker",            0.311, "+1.2%"),
            ("more candidates",          0.296, "+4.7%"),
            ("popularity-boosted",       0.342, "-2.1%"),
            ("diversity re-ranked",      0.271, "+3.4%")]
for name, off, on in observed:
    print(f"  {name:26s} {off:16.3f} {on:>16s}")
print("  The best offline model lost online, and the second-worst won.")
print("  Offline metrics reward agreeing with the logged ranking; the")
print("  popularity-boosted model does that better than anything and")
print("  shows users nothing they had not already found.")

  POSITION BIAS IN A LOGGED RANKING

     position  P(examined)   clicks if relevance=0.30
            1        1.000                      0.300
            2        0.650                      0.195
            3        0.423                      0.127
            4        0.275                      0.082
            5        0.179                      0.054
            6        0.116                      0.035
            7        0.075                      0.023
            8        0.049                      0.015
            9        0.032                      0.010
           10        0.021                      0.006
    The same item with identical relevance collects 48 times fewer
    clicks at position 10 than at position 1. Raw CTR by position
    measures the layout, not the items.

  ESTIMATING RELEVANCE FROM LOGGED CLICKS

     item  shown at  true rel  naive CTR   IPS est
        4         7     0.581      0.044     0.580
        1         2     0.547      0.356     0.548
        8        10     0.512      0.011     0.519
        0         5     0.463      0.083     0.464
        6         1     0.401      0.401     0.401
        3         9     0.344      0.011     0.346
        9         3     0.298      0.126     0.298
        2         8     0.219      0.011     0.218
        5         6     0.176      0.020     0.176
        7         4     0.128      0.035     0.128

    rank correlation with truth -- naive 0.261   IPS 1.000

  USERS PER ARM TO DETECT A CTR LIFT (alpha=0.05, power=0.80)

     baseline CTR        +1%        +2%        +5%       +10%
             0.02  7,729,300  1,941,813    315,207     80,682
             0.05  2,993,904    752,692    122,123     31,234
             0.15    893,193    224,212     36,309      9,257
    Detecting a 1% relative lift on a 2% baseline needs millions of
    users per arm. Most recommender changes are simply not
    measurable at the precision teams assume they are.

  WHY OFFLINE RANKINGS DO NOT TRANSFER

    model                       offline nDCG@10  online CTR lift
    baseline                              0.284               --
    deeper ranker                         0.311            +1.2%
    more candidates                       0.296            +4.7%
    popularity-boosted                    0.342            -2.1%
    diversity re-ranked                   0.271            +3.4%
    The best offline model lost online, and the second-worst won.
    Offline metrics reward agreeing with the logged ranking; the
    popularity-boosted model does that better than anything and
    shows users nothing they had not already found.

The IPS block shows the correction working exactly as intended. Item 4 is the most relevant item in the catalogue and, because it happened to sit at position 7, its raw click rate is 0.044 -- eighth out of ten. Weighting by the inverse examination probability recovers 0.580 against a true relevance of 0.581, and the rank correlation with truth goes from 0.261 to 1.000. Nothing about the items changed; only the accounting did.

The sample-size table is the number most teams have never computed. At a 2% baseline CTR, detecting a 1% relative improvement needs 7.7 million users per arm. If your product has 200,000 weekly users, the smallest effect you can reliably detect is far larger than most model changes produce -- which means a long series of "no significant difference" results that are correctly reported and tell you nothing about whether the changes helped.

A workable evaluation discipline:

  1. OFFLINE, on every change
       temporal or leave-one-out split, ranking metrics, IPS-corrected
       where possible. Purpose: reject clearly worse models cheaply.

  2. GUARDRAILS, before any launch
       coverage, intra-list similarity, novelty, latency, and per-segment
       performance. Purpose: catch changes that improve the average and
       damage a group or the catalogue.

  3. ONLINE A/B, powered for a realistic effect size
       randomise by USER, not by session or request, or the same person
       sees both variants and the effect washes out.

  4. LONG-HORIZON CHECK
       retention and repeat usage over weeks. Purpose: catch the
       engagement trap where CTR rises and users leave sooner.

A final caution specific to recommenders: interference. Standard A/B tests assume the treatment given to one user does not affect another. In a marketplace or social network this is false -- promoting an item to the treatment group changes its popularity, its inventory, and therefore what the control group sees. Cluster-randomised or switchback designs exist for this reason, and ignoring it biases the measured effect in ways that are not small.

Key Takeaways

• Recommenders generate their own training data, so offline evaluation measures agreement with the previous model rather than user preference.
• Use offline metrics as a cheap filter to reject worse models, never as the final verdict.
• Rating-error metrics like RMSE do not match the task; use ranking metrics such as nDCG@k, MAP, and MRR.
• Split temporally or leave-one-out; random splits let the model see a user's future.
• Position bias means logged clicks reflect where items were shown; inverse propensity scoring corrects for it and can recover the true ordering.
• Compute the required sample size before running an A/B test; small relative lifts on low base rates need millions of users per arm.
• Randomise by user, and track a metric ladder from CTR through retention to business outcome, because shallow metrics can rise while deep ones fall.
• Marketplace and social recommenders violate the no-interference assumption, which biases standard A/B estimates.

🧪 Practice

1. Compute precision@k, recall@k, MAP, and nDCG@k for the same ranked list and explain where they disagree.
2. Simulate position-biased click logs and show that naive CTR ranks items incorrectly while IPS-corrected estimates do not.
3. Compare a random split and a temporal split for the same recommender and report both sets of metrics.
4. Compute the required sample size per arm to detect a 3% relative CTR lift at your product's baseline rate.
5. Track coverage and intra-list similarity alongside nDCG across several model variants and identify one that improves accuracy and harms diversity.
6. Design a switchback or cluster-randomised experiment for a marketplace recommender and state what interference it is intended to handle.
7. Interview: Your new model improves offline nDCG@10 by 12% and reduces CTR by 3% in an A/B test. Which do you trust, and what do you do next? (Hint: consider what the offline labels were generated by, and whether a 3% CTR drop with a coverage or novelty increase might be the intended trade rather than a regression.)

---

<a id="14-big-data-and-distributed-computing"></a>