10. Supervised Learning
In supervised learning, the correct answer is given to the model during training. Every supervised problem is one of two kinds:
- Regression: the target is a number (a price, a delivery time, a count).
- Classification: the target is a label (fraud or not fraud, which of six products).
This choice comes first, before you pick any algorithm. It decides the loss function, the metrics, what the model outputs, and which models can fit the problem at all. This chapter has one section for each kind. Each section starts by defining the task and then walks through the algorithms that are best explained through it. Many algorithms (trees, ensembles, nearest neighbours) work for both tasks. They appear in whichever section their examples use, and the last topic of the chapter shows how little really changes between the two versions.
Every algorithm makes an assumption about how the features relate to the target. The practical skill is spotting which assumption your data supports. Chapter 9 covered how to evaluate and compare models honestly. This chapter assumes you follow those rules and focuses on how each algorithm works.
<a id="101-regression"></a>
10.1 Regression
In regression, the target is a number on a number line. That one fact shapes everything else: what counts as a mistake, which loss you minimise, which metrics make sense, and which models can produce the answer. This section covers:
- what makes a problem regression, and why the loss decides what the model actually predicts;
- the main families of regression models and the assumption each one makes;
- linear models: least squares, checking assumptions, ridge and lasso, generalized linear models, and reading coefficients;
- tree-based models: single trees, random forests, gradient boosting and feature importance;
- Gaussian processes, which return a prediction together with how sure they are.
What Makes a Problem Regression
People often say "regression means the target is continuous". That is close, but it can mislead. A better test asks two questions about the target values:
- Is the difference between two values meaningful?
- Is being wrong by 6 twice as bad as being wrong by 3?
If both answers are yes, the problem is regression. A postcode stored as an integer fails the first question: the difference between two postcodes means nothing. A satisfaction score from 1 to 5 probably fails the second. Chapter 1.3 introduced these measurement scales; here is where getting them wrong starts to cost you.
The most important idea in this topic is that a regression model does not
predict "the" target value. For any input x, the target has a whole range
of possible values, described by the distribution P(y | x). The model
predicts one summary of that distribution, and the loss you train with
decides which summary:
| Loss | What it predicts | Use it when |
|---|---|---|
| squared error (L2) | the mean | errors are symmetric and outliers are real |
| absolute error (L1) | the median | the target is skewed or has bad values |
| Huber | a mean that resists outliers | you want L2's precision without outlier damage |
| pinball / quantile, level q | the q-th quantile | you need a guarantee, not an average |
| Poisson / Tweedie deviance | a mean on a count-like scale | the target is a count, a rate, or spend with many zeros |
These are not tuning options. They answer different questions, and on a skewed target they give different answers from the same data. Take a courier quoting delivery times. Traffic can only make trips longer, so delivery time is skewed to the right. If the courier quotes the mean delivery time, it is late on about a third of deliveries. Quoting the 90th percentile instead is a different model of the same data, and the only change is the loss.
Because the target is a number, three more things follow.
The model's output has no limits, but your target usually does. A
least-squares model can predict a negative count, a negative price, or a
probability of 1.3, because nothing in the loss stops it. Clipping the
prediction afterwards is the wrong fix. The right fix is to build the limit
into the model with a link function, which is what generalized linear models
do (covered later in this section). Taking the log of the target does
something similar, but has its own trap: exp(mean of log y) is the geometric
mean, which is smaller than the ordinary mean.
Some error can never be removed. Part of the variation in y is pure
noise, so the error has a floor above zero. RMSE only makes sense compared with
the target's own scale and spread. R² tries to fix that, but it is measured
against the variance of this particular test set, so the same model scores
differently on a narrow test set than on a wide one. Chapter 9.3 covers these
metrics in detail.
Decide whether you need to extrapolate before choosing a model. If new data can fall outside the training range (a hotter summer, a bigger house, a higher spend), the model must be able to continue a trend. This is the clearest difference between the model families in the next topic.
import numpy as np
import pandas as pd
from sklearn.linear_model import (LinearRegression, QuantileRegressor,
HuberRegressor, PoissonRegressor)
rng = np.random.default_rng(7)
n = 4000
# Delivery time in minutes against distance. The noise is RIGHT-SKEWED,
# because traffic can only ever make a trip longer, never shorter.
dist = rng.uniform(1, 12, n)
noise = rng.lognormal(0.0, 0.9, n) # median 1, mean ~1.50
y = 20 + 2.5 * dist + noise
X = dist.reshape(-1, 1)
# The truth, available only because we built it: three different "typical"
# delivery times for the SAME trip.
off_mean = np.exp(0.5 * 0.9 ** 2)
off_med = 1.0
off_q90 = np.exp(0.9 * 1.2816)
print("THE TARGET HAS NO SINGLE 'TYPICAL' VALUE")
print(f" for a 6 km trip the true conditional mean = "
f"{20 + 2.5*6 + off_mean:.2f} min")
print(f" true conditional median = "
f"{20 + 2.5*6 + off_med:.2f} min")
print(f" true conditional q90 = "
f"{20 + 2.5*6 + off_q90:.2f} min")
# --- The LOSS chooses which of those three the model chases. ---
fits = {
"squared (MSE)": LinearRegression().fit(X, y),
"absolute (MAE)": QuantileRegressor(quantile=0.5, alpha=0).fit(X, y),
"pinball (q90)": QuantileRegressor(quantile=0.9, alpha=0).fit(X, y),
"Huber ": HuberRegressor(epsilon=1.35).fit(X, y),
}
print("\nSAME DATA, SAME MODEL CLASS, FOUR LOSSES")
print(f" {'loss':16s} {'slope':>7s} {'pred @ 6 km':>12s} estimates")
targets = {"squared (MSE)": "the conditional MEAN",
"absolute (MAE)": "the conditional MEDIAN",
"pinball (q90)": "the conditional 90th PERCENTILE",
"Huber ": "mean, with outliers down-weighted"}
for name, m in fits.items():
p = m.predict([[6.0]])[0]
print(f" {name:16s} {float(np.ravel(m.coef_)[0]):7.3f} {p:12.2f}"
f" {targets[name]}")
# --- Why this is a decision and not a detail. ---
late_mean = (y > 20 + 2.5 * dist + off_mean).mean()
late_q90 = (y > 20 + 2.5 * dist + off_q90).mean()
print("\n A quoted delivery time is a PROMISE, so the summary is a choice:")
print(f" late rate when the courier quotes the mean: {late_mean:.1%}")
print(f" late rate when the courier quotes the q90 : {late_q90:.1%}")
# --- Robustness: 1% of rows get a logging error, and they happen to be long
# trips, where a distorted point also has leverage on the slope.
y_bad = y.copy()
far = np.flatnonzero(dist > 9)
hit = rng.choice(far, size=n // 100, replace=False)
y_bad[hit] += 400
print("\nAFTER CORRUPTING 1% OF ROWS (+400 min, all on long trips)")
print(f" {'loss':16s} {'slope clean':>11s} {'slope dirty':>11s} {'shift':>7s}")
for name, m in fits.items():
clean = float(np.ravel(m.coef_)[0])
if "squared" in name:
d = LinearRegression().fit(X, y_bad)
elif "absolute" in name:
d = QuantileRegressor(quantile=0.5, alpha=0).fit(X, y_bad)
elif "pinball" in name:
d = QuantileRegressor(quantile=0.9, alpha=0).fit(X, y_bad)
else:
d = HuberRegressor(epsilon=1.35).fit(X, y_bad)
dirty = float(np.ravel(d.coef_)[0])
print(f" {name:16s} {clean:11.3f} {dirty:11.3f} {dirty-clean:+7.3f}")
# --- A target that is a COUNT is still regression, but squared loss on the
# raw scale predicts impossible values. ---
n2 = 3000
ad_spend = rng.uniform(0, 3, n2)
lam = np.exp(-0.4 + 0.9 * ad_spend) # signups per day
counts = rng.poisson(lam)
Xc = ad_spend.reshape(-1, 1)
ols = LinearRegression().fit(Xc, counts)
pois = PoissonRegressor(alpha=0).fit(Xc, counts)
grid = np.array([[0.0], [0.5], [1.5], [3.0]])
print("\nCOUNTS: THE LINK FUNCTION IS PART OF THE TASK DEFINITION")
print(f" {'ad spend':>9s} {'true mean':>10s} {'OLS':>8s} {'Poisson GLM':>12s}")
for g in grid:
t = np.exp(-0.4 + 0.9 * g[0])
print(f" {g[0]:9.1f} {t:10.2f} {ols.predict([g])[0]:8.2f}"
f" {pois.predict([g])[0]:12.2f}")
print(f" OLS predicts {ols.predict([[0.0]])[0]:.2f} signups at zero spend; "
"a count cannot be negative.")
THE TARGET HAS NO SINGLE 'TYPICAL' VALUE
for a 6 km trip the true conditional mean = 36.50 min
true conditional median = 36.00 min
true conditional q90 = 38.17 min
SAME DATA, SAME MODEL CLASS, FOUR LOSSES
loss slope pred @ 6 km estimates
squared (MSE) 2.516 36.44 the conditional MEAN
absolute (MAE) 2.499 35.99 the conditional MEDIAN
pinball (q90) 2.536 38.05 the conditional 90th PERCENTILE
Huber 2.502 36.08 mean, with outliers down-weighted
A quoted delivery time is a PROMISE, so the summary is a choice:
late rate when the courier quotes the mean: 32.0%
late rate when the courier quotes the q90 : 9.3%
AFTER CORRUPTING 1% OF ROWS (+400 min, all on long trips)
loss slope clean slope dirty shift
squared (MSE) 2.516 4.097 +1.581
absolute (MAE) 2.499 2.504 +0.005
pinball (q90) 2.536 2.602 +0.066
Huber 2.502 2.506 +0.004
COUNTS: THE LINK FUNCTION IS PART OF THE TASK DEFINITION
ad spend true mean OLS Poisson GLM
0.0 0.67 -0.69 0.69
0.5 1.05 0.70 1.07
1.5 2.59 3.49 2.61
3.0 9.97 7.67 9.92
OLS predicts -0.69 signups at zero spend; a count cannot be negative.
Key Takeaways
- A problem is regression when differences between target values are meaningful and proportional, not just because the target is stored as a number.
- A regression model predicts a summary of
P(y | x), and the loss picks the summary: squared error gives the mean, absolute error the median, pinball loss a quantile, and Huber a mean that resists outliers.- On a skewed target these summaries are quite different, so choosing the loss is a business decision about what the number promises.
- With squared error, a few bad rows far from the centre can pull the whole fit. Absolute and Huber losses barely notice them.
- If the target is dirty and you cannot clean it first, use L1 or Huber loss.
- For limited targets such as counts, build the limit into the model with a link function and a matching loss. Do not clip a least-squares output.
- RMSE has a floor set by the noise, and
R²depends on the spread of the test set, so neither means much on its own.
🧪 Practice
- Simulate a right-skewed target, fit squared, absolute and pinball losses, and check that each one recovers the summary it should.
- For each of those fits, report the share of held-out rows that are above the prediction. Which one would you quote to a customer?
- Add a large error to 1% of the rows and record how far each loss moves. Repeat with the bad rows placed near the centre of the data and explain the difference.
- Fit a least-squares model to a count target and count the negative test predictions. Refit as a Poisson GLM and compare.
- Fit a model on
log(y), exponentiate the predictions, and show that they are too low on average compared withy. Measure the gap.- Compute
R²for one fixed model on a narrow and a wide slice of the same test set. Explain in one sentence why the two numbers differ.- Interview: A pricing model has a good RMSE, but the business says it under-quotes expensive jobs. What do you check first? (Hint: which summary of the target did the loss estimate, and which one did the business want?)
Regression Algorithms and the Bets They Make
Every regression model family makes a guess about the shape of the relationship between features and target. That guess is what makes the family strong, and it is also how the family fails.
| Family | The guess | Extrapolates | Needs scaling | Interactions | Covered in |
|---|---|---|---|---|---|
| linear / GLM | a weighted sum, possibly through a link | yes | only for penalties | by hand | 10.1 |
| ridge / lasso / elastic net | the same, with most coefficients small or zero | yes | yes | by hand | 10.1 |
| polynomial and splines | a smooth curve of limited flexibility | badly | for penalties | by hand | 10.1 |
| decision tree | the target is constant inside boxes | no | no | automatic | 10.1 |
| random forest | many noisy box surfaces, averaged | no | no | automatic | 10.1 |
| gradient boosting | a sum of many small box-shaped corrections | no | no | automatic | 10.1 |
| k-nearest neighbours | nearby rows have similar targets | no | yes | automatic | 10.2 |
| SVR / kernel methods | smooth, under a similarity measure you choose | weakly | yes | automatic | 10.2 |
| gaussian process | a distribution over smooth functions | weakly, and says it is unsure | yes | automatic | 10.1 |
The extrapolation column surprises people. Trees, forests, boosted ensembles and k-NN all predict by averaging training targets. So their predictions can never go outside the range of targets seen in training. This is simple arithmetic, not a quirk of any library. When the input moves past the edge of the training data, the prediction becomes a flat line at whatever value the last leaf held. A linear model keeps going instead. That is either exactly right or confidently wrong, but at least it is a line you can question.
The scaling column describes the same idea as a preprocessing rule. Any model that computes distances or dot products compares features with each other: k-NN, SVR, kernel methods, Gaussian processes, and every penalised linear model. A column measured in square centimetres then outweighs one measured in years, without any warning. Trees split on one feature at a time, so rescaling a feature does not affect them.
For tabular data, gradient boosting is the usual default. The useful question is not "can anything beat it?" but "what do I need that it cannot give me?":
- Coefficients you can explain in a meeting: use a linear model.
- Predictions outside the training range: use a parametric model.
- Reliable uncertainty from a few hundred rows: use a Gaussian process.
import numpy as np
from sklearn.ensemble import RandomForestRegressor, HistGradientBoostingRegressor
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF, WhiteKernel
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.metrics import mean_absolute_error, root_mean_squared_error
from sklearn.neighbors import KNeighborsRegressor
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.svm import SVR
from sklearn.tree import DecisionTreeRegressor
rng = np.random.default_rng(3)
# Energy use against outdoor temperature and floor area. The temperature
# response is U-shaped -- heating below, cooling above -- and the two
# predictors interact, which is the bet each family gets right or wrong.
def truth(temp, area):
return 40 + 0.9 * (temp - 18) ** 2 + 0.06 * area + 0.004 * area * np.abs(temp - 18)
n = 1200
temp = rng.uniform(2, 30, n)
area = rng.uniform(50, 400, n)
y = truth(temp, area) + rng.normal(0, 8, n)
X = np.column_stack([temp, area])
# Test sets: one INSIDE the training range, one OUTSIDE it. The second is
# where the families stop agreeing.
t_in = rng.uniform(2, 30, 600)
a_in = rng.uniform(50, 400, 600)
X_in = np.column_stack([t_in, a_in])
y_in = truth(t_in, a_in) + rng.normal(0, 8, 600)
t_out = rng.uniform(30, 38, 400) # hotter than anything seen
a_out = rng.uniform(50, 400, 400)
X_out = np.column_stack([t_out, a_out])
y_out = truth(t_out, a_out) + rng.normal(0, 8, 400)
models = {
"linear (OLS)": LinearRegression(),
"polynomial deg 2": make_pipeline(PolynomialFeatures(2), Ridge(alpha=1.0)),
"decision tree": DecisionTreeRegressor(min_samples_leaf=15, random_state=0),
"random forest": RandomForestRegressor(n_estimators=300, min_samples_leaf=2,
random_state=0),
"gradient boosting": HistGradientBoostingRegressor(max_iter=400, random_state=0),
"k-NN (k=10)": make_pipeline(StandardScaler(), KNeighborsRegressor(10)),
"SVR (RBF)": make_pipeline(StandardScaler(), SVR(C=300, epsilon=2.0)),
"gaussian process": make_pipeline(
StandardScaler(),
GaussianProcessRegressor(kernel=RBF(2.0) + WhiteKernel(1.0),
normalize_y=True, random_state=0)),
}
print("IN-RANGE ACCURACY vs EXTRAPOLATION (irreducible noise sd = 8.0)")
print(f" {'family':19s} {'RMSE in':>8s} {'RMSE out':>9s} {'pred @ 35C/200sqm':>18s}")
probe = np.array([[35.0, 200.0]])
true_probe = truth(35.0, 200.0)
for name, m in models.items():
m.fit(X, y)
r_in = root_mean_squared_error(y_in, m.predict(X_in))
r_out = root_mean_squared_error(y_out, m.predict(X_out))
p = m.predict(probe)[0]
print(f" {name:19s} {r_in:8.2f} {r_out:9.2f} {p:18.1f}")
print(f" {'TRUTH':19s} {'':8s} {'':9s} {true_probe:18.1f}")
# --- What the tree families actually do beyond the edge of the data. ---
print("\nWHY THE TREES FAIL OUTSIDE THE RANGE (area fixed at 200 sqm)")
print(f" {'temp':>5s} {'truth':>8s} {'forest':>8s} {'poly deg 2':>11s}")
rf = models["random forest"]
poly = models["polynomial deg 2"]
for t in [26, 29, 30, 32, 35, 38]:
q = np.array([[float(t), 200.0]])
flag = "" if t <= 30 else " <- outside training range"
print(f" {t:5d} {truth(t, 200.0):8.1f} {rf.predict(q)[0]:8.1f}"
f" {poly.predict(q)[0]:11.1f}{flag}")
print(" A tree's prediction is an average of training rows, so it is bounded")
print(" by what it has already seen. It cannot rise past its largest leaf.")
# --- Scaling: the same bet, stated as a preprocessing requirement. ---
print("\nSENSITIVITY TO FEATURE SCALING (area in sqm vs in sq cm)")
X_cm = np.column_stack([temp, area * 10000])
X_in_cm = np.column_stack([t_in, a_in * 10000])
for name in ["random forest", "k-NN (k=10)", "SVR (RBF)"]:
base = models[name]
a = root_mean_squared_error(y_in, base.predict(X_in))
if name == "random forest":
alt = RandomForestRegressor(n_estimators=300, min_samples_leaf=2,
random_state=0)
elif name.startswith("k-NN"):
alt = KNeighborsRegressor(10) # deliberately unscaled
else:
alt = SVR(C=300, epsilon=2.0) # deliberately unscaled
alt.fit(X_cm, y)
b = root_mean_squared_error(y_in, alt.predict(X_in_cm))
note = "scale-free" if abs(b - a) < 1.0 else "SCALE-DEPENDENT"
print(f" {name:19s} RMSE {a:7.2f} -> {b:7.2f} {note}")
IN-RANGE ACCURACY vs EXTRAPOLATION (irreducible noise sd = 8.0)
family RMSE in RMSE out pred @ 35C/200sqm
linear (OLS) 57.60 269.81 44.2
polynomial deg 2 8.13 10.12 329.8
decision tree 11.24 138.53 188.4
random forest 9.05 136.68 187.6
gradient boosting 9.62 140.24 186.5
k-NN (k=10) 10.50 146.26 180.2
SVR (RBF) 8.28 67.19 280.1
gaussian process 8.26 61.78 282.2
TRUTH 325.7
WHY THE TREES FAIL OUTSIDE THE RANGE (area fixed at 200 sqm)
temp truth forest poly deg 2
26 116.0 119.8 114.6
29 169.7 172.2 169.1
30 191.2 187.6 191.1
32 239.6 187.6 240.8 <- outside training range
35 325.7 187.6 329.8 <- outside training range
38 428.0 187.6 435.9 <- outside training range
A tree's prediction is an average of training rows, so it is bounded
by what it has already seen. It cannot rise past its largest leaf.
SENSITIVITY TO FEATURE SCALING (area in sqm vs in sq cm)
random forest RMSE 9.05 -> 9.05 scale-free
k-NN (k=10) RMSE 10.50 -> 68.40 SCALE-DEPENDENT
SVR (RBF) RMSE 8.28 -> 69.07 SCALE-DEPENDENT
Key Takeaways
- Each regression family assumes a shape for the relationship. That assumption is both its strength and how it fails.
- Trees, forests, boosting and k-NN predict by averaging training targets, so their predictions stay inside the training target range and go flat outside the training inputs.
- Linear and polynomial models always extrapolate. That helps when the trend is real and hurts when it is not.
- Distance-based models, kernel models and penalised linear models need features on similar scales. Trees do not, because they split one feature at a time.
- A linear model can lose clearly inside the training range and still be the right choice if new data goes beyond it.
- Gradient boosting is a good default for tabular data. Move away from it when you need something specific: interpretability, extrapolation or uncertainty.
🧪 Practice
- Fit a forest and a quadratic model on a limited input range, predict well beyond it, and plot both. Find the exact value where the forest stops changing.
- Show that a forest never predicts outside
[min(y), max(y)]of the training set, and explain why in one sentence about leaves.- Multiply one feature by 10,000 and refit k-NN, SVR and a random forest. Report which RMSEs change.
- On data with a known interaction, compare a linear model without an interaction term against a depth-3 tree. Then add the interaction by hand and compare again.
- Fit a Gaussian process on 200 rows and plot its prediction interval inside and outside the training range. Describe what happens at the edge.
- Build a time-ordered dataset with a trend, fit a forest and a linear model, and test on a later time window. Explain the result to someone who thinks boosting always wins.
- Interview: A demand model does very well in backtests but under-predicts every month of an unusually hot summer. It is a gradient-boosted ensemble. What is happening, and what is the smallest fix? (Hint: what can a boosted tree output when the input is beyond every split it learned?)
Simple and Multiple Linear Regression
Say you want to predict house prices from floor area. You plot the points, see a general upward trend, and want the line that best summarises it. "Best" needs a definition. The standard one, used for two centuries, is least squares: pick the line that makes the sum of squared vertical distances from the points to the line as small as possible.
Why squared distances rather than plain distances? Three reasons:
- Squared error is smooth everywhere, so the best line can be found with a formula instead of a search.
- It is the maximum-likelihood answer when the errors are normally distributed.
- It predicts the mean of the target for each input, which is usually what people want.
The downside is sensitivity to outliers. Squaring means one point 10 units away counts as much as a hundred points 1 unit away.
Simple regression has one predictor: y = b0 + b1*x. Multiple
regression has several: y = b0 + b1*x1 + b2*x2 + ... + bp*xp. "Linear" means
linear in the coefficients, not in the features. Adding x^2 as a column
keeps the model linear, because the prediction is still a weighted sum. What
you cannot do and stay linear is something like x^b1.
The key change happens when you go from one predictor to several. In simple
regression, b1 is the average change in y for one more unit of x. In
multiple regression, b1 is the average change in y for one more unit of
x1 while all the other predictors stay the same. That condition is the
whole difference, and it is why the same predictor can get very different
coefficients in the two models.
Example: bigger houses tend to sit on bigger plots. If you regress price on floor area alone, the slope also includes the value of the extra land, because land comes along with floor area. Add plot size to the model, and floor area keeps only the part of the effect that really belongs to it.
There is also a geometric view: fitting is projection. Treat the target
y as one vector with one entry per row. All the predictions the model could
possibly make form a flat subspace spanned by the predictor columns. Least
squares picks the point in that subspace closest to y, which is the
perpendicular projection of y onto it. The residuals are what is left over,
and they are always at right angles (orthogonal) to every predictor. You do
not need to check this; it is what "closest" means.
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(0)
# A market where BIGGER houses sit on BIGGER plots: the two predictors are
# correlated, which is what makes simple and multiple regression disagree.
n = 400
plot = rng.normal(8.0, 2.0, n) # plot size, thousands sqft
sqft = 900 + 180 * plot + rng.normal(0, 260, n) # house size, sqft
price = 40_000 + 120 * sqft + 9_000 * plot + rng.normal(0, 25_000, n)
df = pd.DataFrame({"sqft": sqft, "plot": plot, "price": price})
# --- LEAST SQUARES BY HAND: the normal equations, one line of algebra. ---
X = np.column_stack([np.ones(n), df["sqft"], df["plot"]]) # design matrix
y = df["price"].to_numpy()
beta = np.linalg.solve(X.T @ X, X.T @ y) # solve, never invert (ch. 2.1)
print("normal equations :", np.round(beta, 2))
fitted = LinearRegression().fit(df[["sqft", "plot"]], y)
print("sklearn :", np.round(
np.r_[fitted.intercept_, fitted.coef_], 2))
# --- SIMPLE VS MULTIPLE: the same data, two different questions. ---
simple = LinearRegression().fit(df[["sqft"]], y)
print(f"\nsimple price ~ sqft : {simple.coef_[0]:8.2f} per sqft")
print(f"multiple price ~ sqft + plot : {fitted.coef_[0]:8.2f} per sqft")
print(f"true generating coefficient : {120.0:8.2f} per sqft")
print("The simple slope absorbs plot size, because plot rides along with")
print("sqft. The multiple slope holds plot FIXED, which is the partial effect.")
# The gap is exactly the omitted path, and it is an identity, not an estimate:
# simple slope = partial slope + (effect of plot) * (slope of plot on sqft)
aux = LinearRegression().fit(df[["sqft"]], df["plot"]) # plot regressed on sqft
print(f"\nomitted path = {fitted.coef_[1]:.2f} * {aux.coef_[0]:.5f}"
f" = {fitted.coef_[1] * aux.coef_[0]:.2f}")
print(f"observed gap = {simple.coef_[0]:.2f} - {fitted.coef_[0]:.2f}"
f" = {simple.coef_[0] - fitted.coef_[0]:.2f}")
# --- RESIDUALS ARE ORTHOGONAL TO THE PREDICTORS, BY CONSTRUCTION. ---
resid = y - X @ beta
print("\nX' r (should be ~0):", np.round(X.T @ resid, 6))
# --- R^2 IS A RATIO OF SUMS OF SQUARES, NOT A GRADE. ---
ss_res = (resid ** 2).sum()
ss_tot = ((y - y.mean()) ** 2).sum()
r2 = 1 - ss_res / ss_tot
p = X.shape[1] - 1
r2_adj = 1 - (1 - r2) * (n - 1) / (n - p - 1)
print(f"\nR^2 {r2:.4f} adjusted R^2 {r2_adj:.4f}")
# Adding pure noise never lowers R^2, which is why it cannot select models.
noise = rng.normal(0, 1, (n, 12))
Xn = np.column_stack([df[["sqft", "plot"]], noise])
r2_noise = LinearRegression().fit(Xn, y).score(Xn, y)
print(f"R^2 after adding 12 columns of pure noise: {r2_noise:.4f}"
f" (was {r2:.4f})")
print("R^2 went UP. It is monotone in the number of predictors, so it is a")
print("description of the fit, never evidence that the model generalises.")
# --- FITTING IS PROJECTION: y-hat lives in the column space of X. ---
Q, _ = np.linalg.qr(X)
print(f"\n||y_hat|| via beta {np.linalg.norm(X @ beta):.3f}"
f" via projection {np.linalg.norm(Q @ (Q.T @ y)):.3f}")
SIMPLE VS MULTIPLE: THE SAME DATA, TWO DIFFERENT QUESTIONS
price ~ sqft slope 153.53 "houses with more floor space
sell for more"
price ~ sqft + plot slope 118.82 "the same house with more floor
space sells for more"
true generating value slope 120.00
the gap is an identity, not an accident:
153.53 - 118.82 = 34.70
(plot effect) x (plot per sqft) = 9744.89 x 0.00356 = 34.70
PROJECTION: WHAT LEAST SQUARES ACTUALLY DOES
y (the observed target)
|\
| \
residual r | \ r is PERPENDICULAR to the subspace,
| \ so X'r = 0 by construction
| \
-------+----------------- column space of X
y_hat (every prediction the model can make)
X' r = [-0.0e+00 -2.1e-05 0.0e+00] <- zero, up to floating point
R^2 IS A DESCRIPTION OF FIT, NOT EVIDENCE OF GENERALISATION
R^2 0.9047
adjusted R^2 0.9042
R^2 after 12 columns of pure noise 0.9071 <- went UP
R^2 is non-decreasing in the number of predictors, so it cannot
choose between models. Use held-out error for that (chapter 9.3).
Key Takeaways
- Least squares minimises squared vertical distance. This has a direct formula and predicts the mean, but it is sensitive to outliers.
- "Linear" means linear in the coefficients, so columns like
x^2andlog(x)are allowed.- A simple-regression slope answers "how much more do bigger houses cost?". A multiple-regression slope answers "how much does extra space cost when everything else stays the same?". These are different questions with different answers.
- The gap between the two slopes is exactly the effect that flows through the variable you left out, so it can be predicted.
- Fitting is a perpendicular projection onto the space of the predictor columns, so residuals are always orthogonal to the predictors.
- R^2 never goes down when you add predictors, even pure noise, so it describes fit and cannot be used to choose a model.
🧪 Practice
- Fit a two-predictor regression with the normal equations using
np.linalg.solveand check that the coefficients matchLinearRegression.- Generate two correlated predictors and check numerically that the simple slope equals the multiple-regression slope plus the effect through the left-out variable.
- Add ten columns of pure noise and report R^2, adjusted R^2 and cross-validated R^2 before and after. Explain why only one of the three behaves sensibly.
- Check that the residuals are orthogonal to every column of the design matrix, and explain what breaks if you forget the intercept column.
- Interview: A colleague finds that floor area has a coefficient of 153 in one model and 119 in another, on the same data, and asks which one is the bug. What do you say? (Hint: neither is a bug. Ask what else is in each model and what question each coefficient answers.)
Assumptions and Diagnostics
Least squares returns coefficients for any data you give it. Whether they mean anything depends on assumptions that the fitting never checks. The key point is that each assumption protects something different, and breaking it costs you something specific. People often memorise the list and then treat all five as equally fatal. That is wrong, and it stops them from making progress.
| Assumption | What it gives you | What breaking it costs |
|---|---|---|
| Linearity in coefficients | The right shape of model | Wrong coefficients and wrong predictions |
| Independent errors | Correct standard errors | Intervals far too narrow; the estimates are still fine |
| Constant error variance | Correct standard errors | Intervals wrong in either direction; the estimates are still fine |
| Normally distributed errors | Exact results on small samples | Almost nothing once n is large |
| No perfect collinearity | One unique answer | No answer at all; near-collinearity makes coefficients unstable |
Look at the last column. Only linearity makes the coefficients wrong. Independence and constant variance damage the uncertainty around the coefficients. That matters a lot if you want confidence intervals and p-values, and not at all if you only need predictions. Normality is the assumption people worry about most and the one that matters least in practice.
Linearity: plot residuals against fitted values. If the model has the right shape, the residuals form a flat band around zero with no pattern. A U-shape means a squared term is missing. A fan shape points to a different problem (see constant variance below). Always look at this plot first.
Independence usually breaks because of structure you already know about: rows ordered in time, the same person measured several times, or rows grouped by store or school. The Durbin-Watson statistic detects correlation between neighbouring errors, but the better check is to ask how the rows were produced. If the same customer appears eleven times, the errors are not independent, whatever the statistic says.
Constant variance (homoscedasticity) means the errors have the same spread everywhere. Spending data often breaks it: rich customers vary more, in absolute terms, than poor ones. The Breusch-Pagan test checks the fan shape formally by regressing squared residuals on the predictors. The usual fix is not to drop the model but to use heteroscedasticity-robust standard errors. They keep the coefficients and only correct the uncertainty.
Normal errors are needed for exact t and F tests on small samples. With a few hundred rows, the central limit theorem takes over and the intervals are about right anyway. Check it with a Q-Q plot against a normal distribution, and only worry when n is small or the tails are extreme.
Multicollinearity is the one that surprises people. If two predictors are
perfectly correlated, there is no unique answer. If they are nearly
correlated, it is worse, because you get an answer with no warning. The fit is
good and the predictions are good, but the individual coefficients jump around
with tiny changes in the data. The variance inflation factor (VIF) measures
this: regress one predictor on all the others, and VIF = 1/(1-R^2). Above 5
is worth a look; above 10 is usually a problem. Collinearity hurts
interpretation, not prediction.
Finally, influence. One row can move a coefficient more than a thousand others put together. Leverage measures how unusual a row's predictor values are. Cook's distance combines leverage with the size of the residual, and estimates how much the fit would change if that row were removed. High leverage alone is harmless if the point lies on the trend. High leverage plus a big residual is the dangerous combination.
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(7)
n = 500
# --- A DATASET THAT VIOLATES THREE ASSUMPTIONS ON PURPOSE. ---
x1 = rng.uniform(0, 10, n)
x2 = rng.normal(0, 1, n)
x3 = 3 * x1 + rng.normal(0, 0.05, n) # near-duplicate of x1
curved = 2 * x1 + 0.45 * x1 ** 2 # non-linear in x1
spread = 0.35 + 0.45 * x1 # error grows with x1
y = 5 + curved + 1.5 * x2 + rng.normal(0, spread, n)
X = pd.DataFrame({"x1": x1, "x2": x2, "x3": x3})
model = LinearRegression().fit(X[["x1", "x2"]], y)
resid = y - model.predict(X[["x1", "x2"]])
# --- 1. LINEARITY: residuals vs fitted should show no pattern. ---
# Bin the fitted values and print the mean residual per bin: a curve here is
# the plot's signal, made numeric so it survives a text document.
fit = model.predict(X[["x1", "x2"]])
bins = pd.qcut(fit, 8, labels=False)
print("LINEARITY mean residual by fitted-value octile")
for b in range(8):
m = resid[bins == b].mean()
bar = "#" * int(abs(m) * 2)
print(f" octile {b}: {m:+7.2f} {bar}")
print("A smile shape (positive at both ends, negative in the middle) is the")
print("signature of a missing quadratic term, not of random noise.")
# --- 2. HOMOSCEDASTICITY: residual spread should not track the fitted value. ---
print("\nHOMOSCEDASTICITY residual SD by fitted-value octile")
for b in range(8):
print(f" octile {b}: sd {resid[bins == b].std():6.2f}")
# Breusch-Pagan: regress squared residuals on the predictors and test the fit.
aux = LinearRegression().fit(X[["x1", "x2"]], resid ** 2)
r2_aux = aux.score(X[["x1", "x2"]], resid ** 2)
lm_stat = n * r2_aux # chi-square with p degrees of freedom
from scipy import stats
print(f" Breusch-Pagan LM = {lm_stat:.2f}, "
f"p = {stats.chi2.sf(lm_stat, 2):.2e} -> non-constant variance")
# --- 3. MULTICOLLINEARITY: variance inflation factors. ---
def vif(frame):
out = {}
for col in frame.columns:
others = frame.drop(columns=[col])
r2 = LinearRegression().fit(others, frame[col]).score(others, frame[col])
out[col] = 1.0 / max(1e-12, 1 - r2) # 1/(1-R^2) of col on the rest
return pd.Series(out)
print("\nMULTICOLLINEARITY variance inflation factors")
print(vif(X[["x1", "x2"]]).round(2).to_string())
print(vif(X[["x1", "x2", "x3"]]).round(1).to_string(), " <- x3 = 3*x1 + noise")
# What collinearity does to the coefficients: the FIT is fine, the SPLIT is not.
for label, cols in [("x1 + x2", ["x1", "x2"]), ("x1 + x2 + x3", ["x1", "x2", "x3"])]:
m = LinearRegression().fit(X[cols], y)
coefs = ", ".join(f"{c}={v:+9.3f}" for c, v in zip(cols, m.coef_))
print(f" {label:14s} R^2 {m.score(X[cols], y):.4f} {coefs}")
# --- 4. INFLUENCE: leverage and Cook's distance find the rows that matter. ---
Xd = np.column_stack([np.ones(n), x1, x2])
H_diag = np.einsum("ij,jk,ik->i", Xd, np.linalg.inv(Xd.T @ Xd), Xd) # hat diagonal
p = Xd.shape[1]
mse = (resid ** 2).sum() / (n - p)
cooks = resid ** 2 / (p * mse) * H_diag / (1 - H_diag) ** 2
top = np.argsort(cooks)[-5:][::-1]
print(f"\nINFLUENCE mean leverage {H_diag.mean():.4f} (= p/n = {p/n:.4f})")
for i in top:
print(f" row {i:3d}: leverage {H_diag[i]:.4f} residual {resid[i]:+8.2f}"
f" Cook's D {cooks[i]:.4f}")
# --- 5. NORMALITY OF ERRORS: needed for exact inference, not for fitting. ---
std_resid = resid / resid.std()
qs = [0.01, 0.05, 0.25, 0.50, 0.75, 0.95, 0.99]
print("\nNORMALITY standardised residual quantiles vs normal")
for q in qs:
print(f" q={q:4.2f}: observed {np.quantile(std_resid, q):+6.2f}"
f" normal {stats.norm.ppf(q):+6.2f}")
# --- THE FIX, AND WHAT IT BUYS ---
X2 = X[["x1", "x2"]].assign(x1_sq=x1 ** 2)
fixed = LinearRegression().fit(X2, y)
print(f"\nadding x1^2: R^2 {fixed.score(X2, y):.4f} "
f"(was {model.score(X[['x1','x2']], y):.4f})")
r2 = y - fixed.predict(X2)
print("mean residual by octile after the fix:")
b2 = pd.qcut(fixed.predict(X2), 8, labels=False)
print(" ", " ".join(f"{r2[b2 == b].mean():+5.2f}" for b in range(8)))
# Heteroscedasticity-robust standard errors: keep the coefficients, fix the SEs.
def ols_se(Xd, r, robust):
XtX_inv = np.linalg.inv(Xd.T @ Xd)
if robust: # White / HC0 sandwich estimator
meat = Xd.T @ np.diag(r ** 2) @ Xd
return np.sqrt(np.diag(XtX_inv @ meat @ XtX_inv))
s2 = (r ** 2).sum() / (len(r) - Xd.shape[1])
return np.sqrt(np.diag(s2 * XtX_inv))
Xf = np.column_stack([np.ones(n), x1, x2, x1 ** 2])
rf = y - Xf @ np.linalg.solve(Xf.T @ Xf, Xf.T @ y)
print("\nstandard errors on the corrected model")
for name, s_cl, s_rb in zip(["const", "x1", "x2", "x1^2"],
ols_se(Xf, rf, False), ols_se(Xf, rf, True)):
print(f" {name:6s} classical {s_cl:.4f} robust {s_rb:.4f}"
f" ratio {s_rb/s_cl:.2f}")
1. LINEARITY: MEAN RESIDUAL BY FITTED-VALUE OCTILE
octile 0: +4.94 ######### a SMILE. the fitted line is too low
octile 1: +0.55 # at the extremes and too high in the
octile 2: -2.57 ##### middle, which is what a missing
octile 3: -3.56 ####### quadratic term looks like.
octile 4: -2.48 ####
octile 5: -2.97 ##### after adding x1^2:
octile 6: +0.73 # -0.06 +0.14 -0.22 +0.09 +0.97
octile 7: +5.28 ########## -1.06 -0.28 +0.42 <- no pattern
2. HOMOSCEDASTICITY: RESIDUAL SD BY OCTILE
1.66 1.40 2.17 2.13 3.16 3.07 3.56 5.14
| |
+----------- spread TRIPLES ---------------+
Breusch-Pagan LM = 25.81, p = 2.5e-06
residual
| . . . : .:: .:|:. a FAN. constant-variance
| . : . : .:. ::.::.:|::.::.:|:: inference is invalid here,
0 +----:-:---:--:--:--:---:-------:---- but the coefficients are
| . : . .: .: .:: .:.:|.::.:.:|:. still unbiased.
| . . .: . .:: .:|
+------------------------------------> fitted
3. MULTICOLLINEARITY: THE FIT IS FINE, THE SPLIT IS NOT
model R^2 x1 x2 x3
x1 + x2 0.9454 + 6.499 +1.442 -
x1 + x2 + x3 0.9459 +31.315 +1.428 -8.270
^ ^ ^
barely moves explodes large and negative
VIF: x1 26401 x2 1.0 x3 26401 (x3 = 3*x1 + noise)
The two collinear columns cancel each other. Predictions are
unaffected; the individual coefficients are now meaningless.
4. INFLUENCE
mean leverage 0.0060 (= p/n)
row 19: leverage 0.0081 residual +20.10 Cook's D 0.0547
row 483: leverage 0.0195 residual +11.90 Cook's D 0.0474
leverage answers "is this row unusual in X"
Cook's D answers "would deleting it move the model"
5. NORMALITY: STANDARDISED RESIDUAL QUANTILES
q observed normal
0.01 -1.91 -2.33 tails slightly off, which at n=500
0.50 -0.17 +0.00 changes essentially nothing about
0.99 +2.66 +2.33 the intervals
THE FIX: ROBUST STANDARD ERRORS KEEP THE COEFFICIENTS
term classical robust ratio
const 0.4014 0.2218 0.55
x1 0.1837 0.1655 0.90
x2 0.1355 0.1355 1.00
x1^2 0.0178 0.0191 1.08
Robust SEs can be smaller OR larger. "Robust" means correct under
heteroscedasticity, not conservative.
Key Takeaways
- Only the linearity assumption makes coefficients wrong. Independence and constant variance break the standard errors but leave the estimates usable.
- Plot residuals against fitted values first. A curve in that plot means the model has the wrong shape.
- Fix non-constant variance with robust standard errors, not by dropping the model. Robust errors can be smaller as well as larger.
- Near-collinearity gives no warning: the fit stays good while individual coefficients become unstable and meaningless.
- Collinearity hurts interpretation, not prediction, so it only matters if you plan to read the coefficients.
- High leverage only matters together with a large residual. Cook's distance combines the two into one number.
- Normal errors are the least important assumption at realistic sample sizes, and the one people test most.
🧪 Practice
- Simulate data with a real quadratic relationship, fit a straight line, and build the binned residual table that shows the curve.
- Compute VIFs for a design where one column is almost a copy of another. Show that predictions are fine while the two coefficients blow up in opposite directions.
- Implement HC0 robust standard errors from the sandwich formula and compare them with the classical ones on data with non-constant variance.
- Compute leverage and Cook's distance by hand, remove the most influential row, and report how far each coefficient moves.
- Interview: Your model has an R^2 of 0.94 and a residual plot that fans out to the right. Your manager asks whether the 95% confidence interval on the key coefficient can be trusted. What do you say? (Hint: separate what the problem does to the estimate from what it does to the interval.)
Ridge and Lasso Regression
Least squares has one goal: make the training residuals small. With sixty features and only a hundred and twenty rows, it can make them very small by bending itself to fit the noise. The model has more freedom than the data can control.
Regularisation adds a second term that penalises large coefficients. Now every bit of coefficient size has to be paid for with a real drop in error. Features that only help by fitting noise cannot pay. This is the bias-variance trade-off from chapter 9.2, made explicit and adjustable: you accept a little bias to get a big drop in variance.
There are two main penalties. The only difference between them is their shape.
Ridge (L2) penalises the sum of squared coefficients:
minimise ||y - Xb||^2 + alpha * sum(b_j^2). The pull towards zero is
proportional to the coefficient's size, so it gets weaker as the coefficient
shrinks and never reaches zero. Ridge shrinks all coefficients smoothly and
keeps every feature.
Lasso (L1) penalises the sum of absolute values:
minimise ||y - Xb||^2 + alpha * sum(|b_j|). The pull towards zero has the
same strength no matter how small the coefficient is. So coefficients reach
exactly zero and stay there. As a side effect, lasso does feature
selection.
A picture helps. Draw the squared-error surface as ellipses around the unpenalised answer, and draw the region the penalty allows. For L2, that region is a circle. For L1, it is a diamond with its corners on the axes. The penalised answer is where the growing ellipse first touches the region. A circle has no corners, so the touch point usually has every coordinate non-zero. A diamond's corners lie on the axes, and touching a corner means some coefficient is exactly zero.
Three practical rules:
Always standardise first. The penalty is measured in coefficient units. A feature in cents has coefficients a hundred times smaller than the same feature in dollars, so it gets penalised a hundred times less. Put the scaler inside your pipeline so it is refitted on every cross-validation fold.
Do not penalise the intercept. It shifts every prediction, and shrinking it towards zero just pulls all predictions towards zero.
Lasso is unstable when features are correlated. If two predictors are near copies that share one effect, lasso picks one of them more or less at random, and which one changes from sample to sample. Ridge splits the effect between them and gives the same split each time. So a lasso's chosen features are not a discovery about the world. Elastic net exists for this reason: it mixes both penalties, so correlated features enter or leave the model together.
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet
from sklearn.linear_model import RidgeCV, LassoCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split, cross_val_score
rng = np.random.default_rng(3)
# 60 features, only 5 of which matter, and 120 training rows: the regime where
# unregularised least squares has more freedom than the data can support.
n, p, k = 160, 60, 5
X = rng.normal(0, 1, (n, p))
X[:, 1] = X[:, 0] + rng.normal(0, 0.1, n) # a correlated pair, on purpose
true = np.zeros(p)
true[:k] = [1.5, 1.5, -2.5, 1.8, 2.2] # x0 and x1 SHARE one effect
y = X @ true + rng.normal(0, 1.5, n)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=40, random_state=0)
def rmse(m):
return float(np.sqrt(np.mean((yte - m.predict(Xte)) ** 2)))
# --- WHY PENALISE AT ALL: the unregularised fit memorises. ---
ols = LinearRegression().fit(Xtr, ytr)
print(f"{'model':22s} {'train RMSE':>10s} {'test RMSE':>10s} {'nonzero':>8s}")
tr = float(np.sqrt(np.mean((ytr - ols.predict(Xtr)) ** 2)))
print(f"{'least squares':22s} {tr:10.3f} {rmse(ols):10.3f} {p:8d}")
# ALWAYS standardise before penalising: the penalty is in coefficient units, so
# a feature measured in cents would be penalised 100x less than one in dollars.
for name, est in [("ridge (alpha=10)", Ridge(alpha=10)),
("lasso (alpha=0.1)", Lasso(alpha=0.1, max_iter=10000)),
("elastic net (0.5)", ElasticNet(alpha=0.1, l1_ratio=0.5,
max_iter=10000))]:
pipe = make_pipeline(StandardScaler(), est).fit(Xtr, ytr)
coef = pipe[-1].coef_
tr = float(np.sqrt(np.mean((ytr - pipe.predict(Xtr)) ** 2)))
print(f"{name:22s} {tr:10.3f} {rmse(pipe):10.3f} "
f"{int((np.abs(coef) > 1e-8).sum()):8d}")
# --- THE PATH: how coefficients move as the penalty grows. ---
print("\nRIDGE PATH (coefficients shrink together, none reach zero)")
print(f" {'alpha':>8s} " + " ".join(f"x{i}".rjust(7) for i in range(6))
+ " ||b||")
for a in [0.01, 1, 10, 100, 1000]:
m = make_pipeline(StandardScaler(), Ridge(alpha=a)).fit(Xtr, ytr)
c = m[-1].coef_
print(f" {a:8.2f} " + " ".join(f"{v:+7.3f}" for v in c[:6])
+ f" {np.linalg.norm(c):6.2f}")
print("\nLASSO PATH (coefficients hit zero exactly, and stay there)")
print(f" {'alpha':>8s} " + " ".join(f"x{i}".rjust(7) for i in range(6))
+ " nonzero")
for a in [0.01, 0.1, 0.5, 1.0, 2.0]:
m = make_pipeline(StandardScaler(),
Lasso(alpha=a, max_iter=10000)).fit(Xtr, ytr)
c = m[-1].coef_
print(f" {a:8.2f} " + " ".join(f"{v:+7.3f}" for v in c[:6])
+ f" {int((np.abs(c) > 1e-8).sum()):7d}")
# --- LASSO IS UNSTABLE WHEN FEATURES ARE CORRELATED. ---
# x0 and x1 are near-duplicates. Lasso picks ONE of them, and which one it
# picks flips with the sample. Ridge splits the credit and is stable.
print("\nSTABILITY UNDER RESAMPLING (x0 and x1 are 0.99-correlated)")
picks = {"lasso": [], "ridge": []}
for seed in range(12):
idx = np.random.default_rng(seed).choice(len(Xtr), len(Xtr), replace=True)
for name, est in [("lasso", Lasso(alpha=0.3, max_iter=10000)),
("ridge", Ridge(alpha=10))]:
c = make_pipeline(StandardScaler(), est).fit(Xtr[idx], ytr[idx])[-1].coef_
picks[name].append((c[0], c[1]))
for name in ("lasso", "ridge"):
arr = np.array(picks[name])
zeroed = int(np.sum(np.abs(arr) < 1e-8))
print(f" {name}: x0 {arr[:,0].mean():+.2f} +/- {arr[:,0].std():.2f} "
f"x1 {arr[:,1].mean():+.2f} +/- {arr[:,1].std():.2f} "
f"zeros across 24 slots: {zeroed}")
# --- CHOOSING ALPHA IS A CROSS-VALIDATION PROBLEM, NOT A GUESS. ---
rcv = make_pipeline(StandardScaler(),
RidgeCV(alphas=np.logspace(-3, 3, 40), cv=5)).fit(Xtr, ytr)
lcv = make_pipeline(StandardScaler(),
LassoCV(cv=5, random_state=0, max_iter=20000)).fit(Xtr, ytr)
print(f"\nRidgeCV alpha {rcv[-1].alpha_:.3f} test RMSE {rmse(rcv):.3f}")
print(f"LassoCV alpha {lcv[-1].alpha_:.3f} test RMSE {rmse(lcv):.3f}"
f" kept {int((np.abs(lcv[-1].coef_) > 1e-8).sum())}/{p} features")
print("\ntrue nonzero features: 0, 1, 2, 3, 4")
print("lasso kept:", [int(i) for i in np.flatnonzero(np.abs(lcv[-1].coef_) > 1e-8)])
WHY PENALISE: 60 FEATURES, 120 TRAINING ROWS, 5 THAT MATTER
model train RMSE test RMSE nonzero
least squares 0.986 2.132 60
ridge (alpha=10) 1.109 1.901 60
lasso (alpha=0.1) 1.261 1.650 23
elastic net (0.5) 1.202 1.704 43
^ ^
worse on train, BETTER on test.
that trade IS regularisation.
THE TWO PENALTY SHAPES
L2 (ridge): a circle L1 (lasso): a diamond
b2 b2
| ___ | /\
| / \ | / \
----+---( * )----- b1 ----+---< * >----- b1
| \ ___ / | \ /
| | \/
no corners, so contact at a corners ON the axes, so contact
point with both coords nonzero often means b2 = 0 exactly
THE PATHS
ridge: everything shrinks together, nothing vanishes
alpha x0 x1 x2 ||b||
0.01 +5.327 -2.176 -2.853 7.17
1.00 +2.331 +0.796 -2.790 4.82
10.00 +1.534 +1.354 -2.463 4.16
1000.00 +0.267 +0.265 -0.252 0.59
lasso: coefficients hit zero exactly and stay there
alpha x0 x1 x2 nonzero
0.01 +3.106 +0.000 -2.794 51
0.10 +2.901 +0.000 -2.541 23
0.50 +2.476 +0.000 -2.097 4
2.00 +0.889 +0.000 -0.445 3
STABILITY: x0 AND x1 ARE 0.99-CORRELATED AND SHARE ONE TRUE EFFECT
(truth: x0 = +1.5, x1 = +1.5)
lasso x0 +2.50 +/- 0.35 x1 +0.22 +/- 0.28 5 of 24 slots zeroed
ridge x0 +1.48 +/- 0.10 x1 +1.39 +/- 0.11 0 of 24 slots zeroed
Lasso assigns the shared effect to ONE feature and drops the other,
recovering neither true value. Ridge splits it and recovers both.
A lasso's selected set is a modelling artefact, not a finding.
CHOOSING ALPHA IS CROSS-VALIDATION, NOT INTUITION
RidgeCV alpha 4.924 test RMSE 1.984
LassoCV alpha 0.164 test RMSE 1.689 kept 11 of 60 features
lasso kept: 0, 2, 3, 4, 12, 19, 28, 33, 35, 41, 56
^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^
4 of the 5 true 7 false positives, and x1 missed
| Ridge (L2) | Lasso (L1) | Elastic net | |
|---|---|---|---|
| Coefficients reach zero | Never | Yes | Yes |
| Selects features | No | Yes | Yes, in groups |
| Works with p > n | Yes | Yes, keeps at most n | Yes |
| Correlated predictors | Splits the effect | Picks one at random | Keeps them together |
| Stable selection | N/A | No | Better |
| Direct formula | Yes | No, needs coordinate descent | No |
| Use when | Many small effects, collinearity | Few real effects, want sparsity | Correlated groups, want sparsity |
Key Takeaways
- Regularisation trades a small increase in bias for a large drop in variance. That is why a penalised model can fit the training set worse and the test set better.
- Ridge shrinks coefficients in proportion and never reaches zero. Lasso's constant pull sets some coefficients to exactly zero, which selects features.
- The L1 region has corners on the axes and the L2 region does not. That is the whole geometric reason lasso produces zeros.
- Standardise inside the pipeline before penalising, because the penalty depends on the units of each feature.
- Among correlated predictors, lasso's choice is random and changes with the sample, so never present lasso's selected features as a discovery.
- Elastic net keeps correlated features together. Always choose alpha by cross-validation, not by rule of thumb.
🧪 Practice
- Fit ridge with alphas across six orders of magnitude and plot or tabulate the coefficient paths. Confirm that none reaches zero.
- Do the same for lasso and report the alpha at which each feature drops out.
- Fit lasso with and without a
StandardScaleron data where one feature is multiplied by a thousand. Explain why the selected features differ.- Build two features with correlation 0.99 that share one true effect. Bootstrap the fit fifty times and report how often lasso picks each one.
- Compare
RidgeCVwith 5-fold cross-validation against its default leave-one-out on a small noisy dataset. Explain why they can disagree.- Interview: A stakeholder wants "the ten features that drive churn", and your lasso selected exactly ten. What warning do you give before they act on it? (Hint: refit on a bootstrap sample and see whether the same ten come back.)
Generalized Linear Models
Linear regression, logistic regression and Poisson regression look like three different methods. They are really one method with three settings. Seeing this makes each one easier to use, and tells you what to do when the target does not fit the usual shapes. (Logistic regression is covered in detail in 10.2.)
A generalized linear model (GLM) has three parts:
- A linear predictor:
z = b0 + b1*x1 + ... + bp*xp. This is the same in every GLM, and it is the part you fit. - A link function
gthat connects the target's mean to the linear predictor:g(mu) = z. Its job is to stretch the allowed range ofmuover the whole number line, so the linear part has no limits. - A distribution that describes how the target varies around its mean. It sets the variance function, which decides how much weight each row gets.
Fill in the three parts and you get the familiar models:
- Normal distribution + identity link = linear regression.
- Binomial distribution + logit link = logistic regression.
- Poisson distribution + log link = count regression.
Nothing else changes. The same algorithm (iteratively reweighted least squares) fits all of them.
Choose the link to match the target's range. Counts cannot be negative, so
a log link keeps predictions positive. It also makes coefficients
multiplicative, so exp(b) is a rate ratio. Probabilities lie between 0 and 1,
so a logit link maps them onto the whole number line. Positive, skewed amounts
(such as spend) also use a log link, for the same reason as counts.
The variance function is the part people skip, and should not. It is what makes a GLM different from transforming the target and running least squares. In a Poisson model, the variance equals the mean. So a row expected to have 10 events should vary much more than a row expected to have 0.5. The GLM knows this and weights rows to match. Ordinary least squares assumes every row has the same variance and gets the weights wrong.
Two extras come up all the time.
Offsets handle exposure. Suppose insurance policies are watched for
different lengths of time. The number of claims grows with the length of the
window, and you want coefficients that describe the claim rate. Add
log(exposure) to the linear predictor with its coefficient fixed at 1. That
is an offset, not a feature. If you add it as a feature, the model estimates
its own slope for exposure, which answers a different (and usually wrong)
question.
Overdispersion is the common way Poisson models fail. "Variance equals
mean" is a strong assumption. Real count data usually varies more, because of
differences between rows that you did not measure. To check, divide the Pearson
chi-squared statistic by the residual degrees of freedom; the result should be
close to 1. If it is 2, your standard errors are too small by about sqrt(2),
so a coefficient that looks significant at p = 0.01 may not be. The
coefficients themselves stay roughly correct. Fix it with a quasi-Poisson
correction or, better, a negative binomial model, which adds a parameter
for the extra variation.
import numpy as np
import pandas as pd
from sklearn.linear_model import (LinearRegression, PoissonRegressor,
GammaRegressor, TweedieRegressor)
rng = np.random.default_rng(13)
# --- COUNTS: claims per policy, with an EXPOSURE that varies by row. ---
n = 4000
age = rng.integers(18, 75, n)
urban = rng.integers(0, 2, n)
years = rng.uniform(0.25, 3.0, n) # exposure: policy-years at risk
rate = np.exp(-0.4 - 0.015 * age + 0.55 * urban) # claims PER YEAR
claims = rng.poisson(rate * years) # counts scale with exposure
X = np.column_stack([age, urban])
print(f"claims: mean {claims.mean():.3f} variance {claims.var():.3f}"
f" max {claims.max()} zeros {np.mean(claims == 0):.1%}")
# --- WHY NOT ORDINARY LEAST SQUARES ON A COUNT. ---
ols = LinearRegression().fit(X, claims)
print(f"\nleast squares on the count: min prediction {ols.predict(X).min():+.3f}"
f", and at age 100 rural it predicts {ols.predict([[100, 0]])[0]:+.3f}")
print("Nothing stops the fitted plane from going below zero -- an impossible")
print("count -- because an identity link puts no constraint on the range.")
# The deeper problem: a Poisson's variance EQUALS its mean, so the constant-
# variance assumption is wrong in a way that misweights every row.
res = claims - ols.predict(X)
dec = pd.qcut(ols.predict(X), 5, labels=False, duplicates="drop")
print(" fitted quintile: " + " ".join(f"{ols.predict(X)[dec==d].mean():5.2f}"
for d in range(5)))
print(" residual variance:" + " ".join(f"{res[dec==d].var():6.2f}"
for d in range(5)))
print(" Variance tracks the mean, which is exactly what OLS assumes away.")
# --- THE GLM: log link, and exposure enters as an OFFSET, not a feature. ---
# log(mu) = log(exposure) + b0 + b1*age + ... so the coefficients describe the
# RATE. Passing exposure as a feature would estimate a slope for it instead of
# fixing it at 1, which is a different and usually wrong model.
pois = PoissonRegressor(alpha=1e-8, max_iter=2000).fit(
X, claims / years, sample_weight=years) # equivalent to an offset
print(f"\nPoisson GLM (log link) intercept {pois.intercept_:+.4f}"
f" age {pois.coef_[0]:+.5f} urban {pois.coef_[1]:+.4f}")
print(f"truth intercept {-0.4:+.4f}"
f" age {-0.015:+.5f} urban {0.55:+.4f}")
print(f"\nexp(urban) = {np.exp(pois.coef_[1]):.3f}: urban policies claim"
f" {(np.exp(pois.coef_[1]) - 1) * 100:.0f}% more often per policy-year.")
print("A log link makes coefficients MULTIPLICATIVE on the rate.")
# --- OVERDISPERSION: the assumption variance == mean is testable. ---
mu = pois.predict(X) * years
pearson = ((claims - mu) ** 2 / np.maximum(mu, 1e-9)).sum()
dispersion = pearson / (n - 3) # ~1 if Poisson is adequate
print(f"\ndispersion (Pearson chi2 / df) = {dispersion:.3f}")
# Inject extra variation and watch the diagnostic catch it.
gamma_noise = rng.gamma(0.7, 1 / 0.7, n) # unobserved heterogeneity
claims_od = rng.poisson(rate * years * gamma_noise)
pois_od = PoissonRegressor(alpha=1e-8, max_iter=2000).fit(
X, claims_od / years, sample_weight=years)
mu_od = pois_od.predict(X) * years
disp_od = ((claims_od - mu_od) ** 2 / np.maximum(mu_od, 1e-9)).sum() / (n - 3)
print(f"dispersion with unobserved heterogeneity = {disp_od:.3f} -> > 1")
print("Coefficients stay roughly unbiased; the STANDARD ERRORS are too small by")
print(f"about sqrt({disp_od:.2f}) = {np.sqrt(disp_od):.2f}x. Fix with a")
print("quasi-Poisson scale factor or a negative binomial model.")
# --- POSITIVE, RIGHT-SKEWED, MULTIPLICATIVE: claim SEVERITY wants Gamma. ---
sev_mu = np.exp(6.0 + 0.02 * age + 0.3 * urban)
severity = rng.gamma(shape=2.0, scale=sev_mu / 2.0) # variance grows as mu^2
print(f"\nseverity: mean {severity.mean():,.0f} median"
f" {np.median(severity):,.0f} skew present")
gam = GammaRegressor(alpha=1e-8, max_iter=2000).fit(X, severity)
lin_log = LinearRegression().fit(X, np.log(severity))
print(f" Gamma GLM (log link) age {gam.coef_[0]:+.5f}"
f" urban {gam.coef_[1]:+.4f}")
print(f" OLS on log(y) age {lin_log.coef_[0]:+.5f}"
f" urban {lin_log.coef_[1]:+.4f}")
print(f" truth age {0.02:+.5f} urban {0.3:+.4f}")
print("Both handle skew, but they target different things: the Gamma GLM models")
print("E[y] directly, while OLS on log(y) models E[log y], whose exponential is")
print("the GEOMETRIC mean and is biased low for E[y].")
gm = np.exp(lin_log.predict(X)).mean()
print(f" mean of exp(OLS fit) {gm:,.0f} Gamma GLM mean prediction"
f" {gam.predict(X).mean():,.0f} actual mean {severity.mean():,.0f}")
# --- PURE PREMIUM: zero-inflated and continuous at once -> Tweedie. ---
pure = claims * severity / years # loss cost per policy-year
tw = TweedieRegressor(power=1.5, alpha=1e-8, max_iter=5000).fit(X, pure)
print(f"\npure premium: {np.mean(pure == 0):.0%} exact zeros,"
f" continuous above 0")
print(f" Tweedie(p=1.5) age {tw.coef_[0]:+.5f} urban {tw.coef_[1]:+.4f}"
f" (truth {-0.015+0.02:+.5f} / {0.55+0.3:+.4f})")
# --- THE FAMILY TABLE ---
table = pd.DataFrame([
("continuous, symmetric", "Gaussian", "identity", "variance constant"),
("binary / proportion", "Binomial", "logit", "variance mu(1-mu)"),
("count", "Poisson", "log", "variance = mu"),
("count, overdispersed", "Negative binomial", "log", "variance > mu"),
("positive, skewed", "Gamma", "log", "variance ~ mu^2"),
("positive with exact zeros", "Tweedie 1<p<2", "log", "mass at 0 + skew"),
("time to event", "Gamma / Weibull", "log", "censoring needs survival"),
], columns=["target looks like", "family", "usual link", "variance story"])
print("\n", table.to_string(index=False))
THE THREE SLOTS
z = b0 + b1*x1 + ... (identical in every GLM)
|
| inverse link
v
mu = E[y | x] (range enforced by the link)
|
| variance function
v
how y scatters around mu (sets the weights during fitting)
WHY NOT LEAST SQUARES ON A COUNT
min prediction +0.208, and at age 100 rural it predicts -0.113
-> an identity link puts no constraint on the range
fitted quintile: 0.36 0.61 0.75 0.90 1.13
residual variance: 0.43 0.67 0.88 1.03 1.58
| |
+--- variance tracks the mean
That is the Poisson property (variance = mean), and it is exactly
what OLS assumes away.
POISSON GLM WITH EXPOSURE AS AN OFFSET
intercept age urban
fitted -0.3292 -0.01695 +0.5062
truth -0.4000 -0.01500 +0.5500
exp(+0.5062) = 1.659: urban policies claim 66% more per policy-year.
A log link makes coefficients MULTIPLICATIVE on the rate.
OVERDISPERSION IS TESTABLE
dispersion (Pearson chi2 / df), clean Poisson data 1.004
dispersion with unobserved heterogeneity added 2.060
standard errors are too small by about sqrt(2.06) = 1.44x
coefficients stay roughly unbiased; the INFERENCE is broken
GAMMA VS OLS-ON-LOG: THEY TARGET DIFFERENT THINGS
severity: mean 1,242 median 971 (right-skewed)
Gamma GLM (log link) age +0.01885 urban +0.3699
OLS on log(y) age +0.01885 urban +0.3624
truth age +0.02000 urban +0.3000
Coefficients agree, but the PREDICTIONS do not:
mean of exp(OLS fit) 941
Gamma GLM mean prediction 1,242
actual mean 1,242
exp(E[log y]) is the GEOMETRIC mean, biased low for E[y]. If you
need an expected amount, model E[y] directly.
THE FAMILY TABLE
target looks like family link variance
continuous, symmetric Gaussian identity constant
binary / proportion Binomial logit mu(1-mu)
count Poisson log = mu
count, overdispersed Negative binomial log > mu
positive, skewed Gamma log ~ mu^2
positive with exact zeros Tweedie 1<p<2 log mass at 0
time to event Gamma / Weibull log censoring
Key Takeaways
- A GLM is a linear predictor, a link function that sets the range of the mean, and a distribution that sets the variance. Linear, logistic and Poisson regression are three choices of those parts.
- Choose the link to match the target's range. A log link also makes coefficients multiplicative, so
exp(b)is a ratio.- The variance function is what separates a GLM from transforming the target, because it decides how much weight each row gets.
- Put exposure in the model as an offset with a coefficient fixed at 1, not as a feature with its own slope.
- Poisson assumes variance equals mean. Check the dispersion, and if it is above 1, correct the standard errors or switch to negative binomial.
- Overdispersion breaks p-values but leaves coefficients roughly correct.
- A Gamma GLM models
E[y], while least squares onlog(y)modelsE[log y]. Exponentiating that gives the geometric mean, which is too low as an estimate of the mean.
🧪 Practice
- Fit a Poisson GLM with varying exposure, once with exposure as an offset and once as a normal feature. Compare the coefficients with the true values.
- Compute the dispersion statistic on clean Poisson data and on data with added variation between rows. Report how much the standard errors need to be corrected.
- Simulate right-skewed positive amounts, fit a Gamma GLM and least squares on the log, and compare each model's average prediction with the true average.
- Fit least squares to a count target with low rates. Report the share of negative predictions and the residual variance in each fifth of the fitted values.
- Fit a Tweedie model to a target that is exactly zero for most rows and positive otherwise. Explain why neither Poisson nor Gamma alone would fit.
- Interview: Your Poisson model of support tickets per account says a feature is significant at p = 0.001, and the dispersion statistic is 3.4. What do you do before reporting it? (Hint: what does dispersion do to the standard error, and what does that do to the p-value?)
Interpreting Coefficients
A fitted linear model gives you a table of numbers, and there are many ways to read that table wrongly. This topic lists the common mistakes. Interpretation is the main reason to use a linear model, and also where it is most often misused.
Start with the units. A coefficient is the change in y for one more unit
of x. Measure education in months instead of years and the coefficient
becomes twelve times smaller, even though nothing about the model changed. So
the size of a coefficient tells you nothing about how important a feature is
until you know the unit and the spread of that column.
Logs change how you read coefficients. This matters most, because log targets and log features are everywhere.
| Form | Regression | Read the coefficient as | Meaning |
|---|---|---|---|
| level-level | y on x | b | units of y per unit of x |
| log-level | log y on x | 100*b percent | percent change in y per unit of x |
| level-log | y on log x | b/100 | units of y per 1% change in x |
| log-log | log y on log x | b | percent per percent (an elasticity) |
The log-level percentage is an approximation. The exact figure is
exp(b) - 1. The two agree closely when b is below about 0.10 and differ a
lot above that: a coefficient of 0.50 means a 65% increase, not 50%.
Standardising makes sizes comparable. Divide each predictor by its standard deviation, and each coefficient becomes "the effect of a one-standard-deviation change". Predictors in dollars and in years can then be compared. Two warnings. Comparable does not mean causal: a bigger standardised coefficient is not a stronger cause. And do not standardise binary predictors, because "a one-SD change in being female" cannot happen to anyone. Leave 0/1 columns as they are.
Every categorical coefficient is a comparison. One level is dropped as the baseline, and each dummy coefficient is the difference from that level. The baseline has no coefficient because it is the zero point. Drop a different level and every number in the table changes, while the predictions stay the same. Always say which level is the baseline.
Interactions and polynomials do not have a single coefficient. If the model
has exper and exper^2, the effect of one more year of experience is
b_exper + 2*b_exper_sq*exper, which depends on how much experience someone
already has. The exper coefficient alone is the slope at zero experience,
usually the least interesting point. Report the effect at meaningful values
instead.
The Table 2 fallacy is the most damaging mistake and the least known. It means fitting one regression and reading every row as a causal effect. That cannot work, because the controls needed to measure one variable's effect correctly often break the measurement of another. Suppose education affects wages partly through occupation. With occupation in the model, the education coefficient is the direct effect. Without it, it is the total effect. Both are valid, but neither is "the" effect. A regression table is designed to answer one causal question; the other rows are adjustments, not findings. Chapter 3.5 covers the theory. The practical rule here: name the one coefficient you are reading causally, and treat the rest as controls.
A coefficient without an interval is not a finding. Report standard errors and confidence intervals. Remember that with nearly collinear predictors, the interval can be huge even when the model predicts well.
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression, LogisticRegression, Ridge
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(21)
n = 2500
# A wage model, because wage data makes every interpretation trap concrete.
educ = rng.integers(8, 21, n) # years of schooling
exper = rng.integers(0, 40, n) # years of experience
female = rng.integers(0, 2, n)
region = rng.choice(["north", "south", "west"], n, p=[0.4, 0.35, 0.25])
# Log wage is linear, so wage itself is MULTIPLICATIVE in the predictors.
log_wage = (1.6 + 0.085 * educ + 0.022 * exper - 0.0004 * exper ** 2
- 0.12 * female + 0.05 * (region == "west")
- 0.04 * (region == "south") + rng.normal(0, 0.30, n))
wage = np.exp(log_wage)
df = pd.DataFrame({"educ": educ, "exper": exper, "female": female,
"region": region, "wage": wage, "log_wage": log_wage})
# --- 1. UNITS. A coefficient is "change in y per one unit of x", and the unit
# is whatever the column happens to be measured in.
X = df[["educ", "exper"]]
raw = LinearRegression().fit(X, df["wage"])
print("LEVEL-LEVEL y in dollars, x in years")
print(f" educ {raw.coef_[0]:+.3f} dollars/hour per extra year of schooling")
print(f" exper {raw.coef_[1]:+.3f} dollars/hour per extra year of experience")
# Rescale one column and the coefficient changes by exactly that factor. The
# model is identical; only the units of the answer moved.
Xc = X.assign(educ_months=X["educ"] * 12).drop(columns="educ")
rescaled = LinearRegression().fit(Xc, df["wage"])
print(f" same model, educ in MONTHS: {rescaled.coef_[0]:+.4f}"
f" (= {raw.coef_[0]:+.3f} / 12)")
print(" Neither number is 'bigger'. Size of a coefficient says nothing until")
print(" you know the unit and the spread of the column.")
# --- 2. LOG TRANSFORMS change the reading to a PERCENTAGE. ---
logmod = LinearRegression().fit(X, np.log(df["wage"]))
b = logmod.coef_[0]
print(f"\nLOG-LEVEL log(y) on x: coefficient {b:+.4f}")
print(f" approximate reading: +{b*100:.1f}% per extra year of schooling")
print(f" exact reading: +{(np.exp(b) - 1)*100:.1f}% (exp(b) - 1)")
print(" The approximation is fine below ~0.10 and drifts above it:")
for c in [0.05, 0.10, 0.25, 0.50, 1.00]:
print(f" b={c:.2f}: approx {c*100:5.1f}% exact {(np.exp(c)-1)*100:6.1f}%")
# log-log gives an ELASTICITY: a percentage per percentage, unit-free.
df["firm_size"] = rng.lognormal(4, 1, n)
df["log_wage2"] = df["log_wage"] + 0.06 * np.log(df["firm_size"])
ll = LinearRegression().fit(np.log(df[["firm_size"]]), df["log_wage2"])
print(f"\nLOG-LOG elasticity {ll.coef_[0]:+.4f}: a firm 1% larger pays"
f" {ll.coef_[0]:.3f}% more, and one 10% larger pays about"
f" {ll.coef_[0] * 10:.2f}% more")
table = pd.DataFrame([
("level-level", "y on x", "b", "units of y per unit of x"),
("log-level", "log y on x", "100*b", "percent change in y per unit of x"),
("level-log", "y on log x", "b/100", "units of y per 1% change in x"),
("log-log", "log y on log x", "b", "percent per percent (elasticity)"),
], columns=["form", "regression", "read as", "meaning"])
print("\n", table.to_string(index=False))
# --- 3. CATEGORICAL BASELINES. Every dummy coefficient is a COMPARISON. ---
D = pd.get_dummies(df[["educ", "exper", "region"]], columns=["region"],
drop_first=True, dtype=float)
cat = LinearRegression().fit(D, df["log_wage"])
print("\nCATEGORICAL region, with 'north' dropped as the baseline")
for name, c in zip(D.columns, cat.coef_):
if name.startswith("region"):
print(f" {name:16s} {c:+.4f} = {name.split('_')[1]} vs NORTH")
print(" There is no 'north' coefficient because north is the zero point. Drop")
print(" a different level and every number changes while the model does not.")
# --- 4. STANDARDISED COEFFICIENTS make magnitudes comparable, not causal. ---
cols = ["educ", "exper", "female"]
Z = StandardScaler().fit_transform(df[cols])
std = LinearRegression().fit(Z, df["log_wage"])
print(f"\nSTANDARDISED effect of a one-SD move in each predictor")
for c, raw_b, z_b, sd in zip(cols,
LinearRegression().fit(df[cols], df["log_wage"]).coef_,
std.coef_, df[cols].std()):
print(f" {c:8s} raw {raw_b:+.4f} (per unit, sd {sd:5.2f})"
f" standardised {z_b:+.4f} (per sd)")
print(" Note 'female' is binary: a 'one-SD change' in it is not a thing that")
print(" can happen to a person, so leave binary predictors unstandardised.")
# --- 5. INTERACTIONS AND POLYNOMIALS: there is no single coefficient. ---
df["exper_sq"] = df["exper"] ** 2
quad = LinearRegression().fit(df[["educ", "exper", "exper_sq", "female"]],
df["log_wage"])
b_e, b_e2 = quad.coef_[1], quad.coef_[2]
print(f"\nPOLYNOMIAL exper {b_e:+.4f}, exper^2 {b_e2:+.6f}")
print(" The effect of one more year depends on where you are:")
for yrs in [0, 5, 15, 25, 35]:
marginal = b_e + 2 * b_e2 * yrs # d/dexper of the fitted curve
print(f" at {yrs:2d} years: {marginal*100:+6.2f}% per extra year")
print(f" peak wage at {-b_e / (2 * b_e2):.1f} years of experience")
print(" Reporting 'the' experience coefficient here would be meaningless.")
# --- 6. THE TABLE 2 FALLACY: adjusted coefficients are not all causal. ---
# Suppose occupation is a MEDIATOR of the education effect. Controlling for it
# does not make the education coefficient 'better'; it answers a different
# question, and the two numbers cannot both be the total effect.
occ_score = 0.7 * df["educ"] + rng.normal(0, 1.5, n)
df["log_wage3"] = df["log_wage"] + 0.09 * occ_score
tot = LinearRegression().fit(df[["educ"]], df["log_wage3"]).coef_[0]
dir_ = LinearRegression().fit(
df[["educ"]].assign(occ=occ_score), df["log_wage3"]).coef_[0]
print(f"\nMEDIATION educ coefficient WITHOUT occupation {tot:+.4f} (total)")
print(f" educ coefficient WITH occupation {dir_:+.4f} (direct)")
print(" Both are correct answers to different questions. Reading every row of")
print(" one regression table as a causal effect is the Table 2 fallacy: the")
print(" covariate set that identifies one effect generally ruins another.")
# --- 7. UNCERTAINTY. A coefficient without an interval is not a finding. ---
Xd = np.column_stack([np.ones(n), df["educ"], df["exper"], df["female"]])
beta = np.linalg.solve(Xd.T @ Xd, Xd.T @ df["log_wage"])
r = df["log_wage"] - Xd @ beta
s2 = (r ** 2).sum() / (n - Xd.shape[1])
se = np.sqrt(np.diag(s2 * np.linalg.inv(Xd.T @ Xd)))
print("\nUNCERTAINTY")
print(f" {'term':10s} {'estimate':>9s} {'se':>7s} {'95% interval':>20s}")
for name, bb, ss in zip(["const", "educ", "exper", "female"], beta, se):
print(f" {name:10s} {bb:+9.4f} {ss:7.4f}"
f" [{bb - 1.96*ss:+.4f}, {bb + 1.96*ss:+.4f}]")
1. UNITS: THE SAME MODEL, TWO DIFFERENT-LOOKING ANSWERS
educ in YEARS +1.772 dollars/hour per year
educ in MONTHS +0.1028 dollars/hour per month (= 1.772 / 12)
Coefficient size is meaningless without the unit and the spread.
2. LOG TRANSFORMS: THE APPROXIMATION AND THE EXACT FIGURE
log(wage) on educ: b = +0.0862
approximate +8.6% exact exp(b)-1 = +9.0%
b approx exact
0.05 5.0% 5.1% fine
0.10 10.0% 10.5% fine
0.25 25.0% 28.4% drifting
0.50 50.0% 64.9% do not use the approximation
1.00 100.0% 171.8% seriously, do not
log-log elasticity +0.0793: a firm 1% larger pays 0.079% more
3. CATEGORICAL: EVERY DUMMY IS A COMPARISON
region_south -0.0265 = south vs NORTH
region_west +0.0334 = west vs NORTH
There is no 'north' row because north IS the zero point.
4. STANDARDISED: COMPARABLE, NOT CAUSAL
feature raw coef column sd per-sd coef
educ +0.0863 3.75 +0.3237
exper +0.0054 11.55 +0.0624
female -0.1077 0.50 -0.0538 <- binary: do NOT
standardise this
5. POLYNOMIAL: THERE IS NO SINGLE EFFECT
(exper +0.0214, exper^2 -0.000408)
marginal effect = b_exper + 2 * b_exper_sq * exper
at 0 years +2.14% per extra year
at 5 years +1.73%
at 15 years +0.91% %/yr
at 25 years +0.10% | \
at 35 years -0.72% | \___
| \___
peak wage at 26.2 years +-------------\--> experience
26.2
Reporting "the" experience coefficient (+2.14%) reports the slope at
zero experience, the least interesting point on the curve.
6. THE TABLE 2 FALLACY
educ coefficient WITHOUT occupation +0.1485 <- TOTAL effect
educ coefficient WITH occupation +0.0883 <- DIRECT effect
educ ------------------------------> wage (direct)
\ /
\--> occupation ---------------> (mediated)
Both numbers are correct answers to DIFFERENT questions. One
regression table identifies one causal effect; the other rows are
adjustments, not findings.
7. UNCERTAINTY
term estimate se 95% interval
const +1.6927 0.0264 [+1.6409, +1.7445]
educ +0.0863 0.0016 [+0.0831, +0.0895]
exper +0.0054 0.0005 [+0.0044, +0.0064]
female -0.1077 0.0123 [-0.1317, -0.0837]
Key Takeaways
- The size of a coefficient means nothing without the column's unit and spread, because rescaling a predictor rescales its coefficient.
- With a log target,
100*bis roughly a percentage change. The exact figure isexp(b)-1, and the two differ noticeably above about 0.10.- A log-log coefficient is an elasticity: percent change per percent change, with no units.
- Standardised coefficients make sizes comparable but say nothing about cause. Leave binary predictors unstandardised.
- Every dummy coefficient is a difference from the dropped baseline level, and you must state which level that is.
- With polynomials or interactions, the effect of a predictor changes with its value, so report effects at meaningful values rather than one number.
- The Table 2 fallacy is reading every row of one regression as causal. The controls that are right for one effect are usually wrong for another.
- Controlling for a mediator turns a total effect into a direct effect. Both are correct answers to different questions.
🧪 Practice
- Fit the same model with one predictor in two different units and check that the coefficient changes by exactly the conversion factor.
- Tabulate the approximate and exact percentage readings of a log-level coefficient for
bfrom 0.01 to 1.0. Where would you stop using the approximation?- Fit a model with a categorical predictor twice, with a different baseline each time. Check that the predictions are identical while every coefficient differs.
- Fit a quadratic in one predictor and report its effect at the 10th, 50th and 90th percentiles of that predictor.
- Build a dataset with a known mediator, fit the model with and without it, and write two sentences saying what each education coefficient means.
- Compute classical standard errors by hand from the design matrix and check them against
statsmodelsor a similar library.- Interview: A report says "controlling for hospital, the treatment reduces mortality by 8%", but which hospital a patient went to depended on how sick they were. What is your concern? (Hint: does the control come before or after the treatment in the causal chain?)
Decision Tree Construction
Linear models use one weighted sum for the whole dataset. Trees work differently: they ask a series of yes-or-no questions, each about one feature. Trees work for both regression and classification. This topic and the next use a classification example because it is the easiest to picture. The tree ensembles after them return to regression.
A decision tree is a chain of questions. Is income above 60? If yes, is the debt ratio below 0.45? If yes, approve. Each question splits the data in two, and the tree keeps splitting until it decides to stop. Each final region is called a leaf. The prediction in a leaf is the most common class (for classification) or the average target (for regression) of the training rows that ended up there.
Trees are popular because this is how people already reason, and you can print the fitted model and read it. The structure also gives you two things for free.
Interactions are automatic. In a linear model, "high income helps unless debt is also high" must be added by hand as a product term. A tree splits on debt ratio and then splits on income inside each branch, so it captures the interaction without being told. Every path from the root to a leaf is a combination of conditions, which is exactly what an interaction is.
Rescaling features does not matter. A split at income > 60 and a split at log(income) > log(60) divide the data in exactly the same way. Any transformation that keeps the order of values (standardising, taking logs, clipping at percentiles) changes nothing. After all the care that penalised linear models need, this is a relief.
How does the tree pick a split? It tries every option at each step and keeps the best. For every feature and every possible threshold, it measures how much the split reduces impurity (how mixed the classes are), and keeps the best one. Then it repeats inside each child. Within one step the search is complete, but across steps it is greedy: it never goes back to change an earlier split based on what it found later. Finding the best possible tree overall is computationally infeasible (NP-hard), so every library uses this greedy approach.
The structure also has costs.
Boundaries are made of straight, axis-aligned cuts. Every split has the form "feature <= threshold", so the regions are rectangles. A diagonal boundary has to be approximated by a staircase. A line that logistic regression describes with two parameters may take a tree dozens of leaves to copy, and still poorly.
Trees are unstable (high variance). Because each split is chosen greedily, a small change in the data can change an early split and rearrange everything below it. Two samples from the same data source can produce completely different-looking trees with similar accuracy. Ensembles exist to fix this weakness, which is why a single tree is rarely the final model.
import numpy as np
import pandas as pd
from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(2)
# A loan-approval problem with an INTERACTION: high income is fine unless the
# debt ratio is also high. A linear model needs that term written by hand; a
# tree discovers it by splitting twice.
n = 2000
income = rng.uniform(20, 140, n) # thousands
debt_ratio = rng.uniform(0.0, 0.8, n)
approve = ((income > 60) & (debt_ratio < 0.45)).astype(int)
flip = rng.random(n) < 0.06 # 6% label noise, so nothing is perfect
approve = np.abs(approve - flip.astype(int))
X = pd.DataFrame({"income": income, "debt_ratio": debt_ratio})
Xtr, Xte, ytr, yte = train_test_split(X, approve, test_size=0.3, random_state=0)
# --- HOW ONE SPLIT IS CHOSEN: try every threshold, keep the purest. ---
def gini(y):
if len(y) == 0:
return 0.0
p = np.mean(y)
return 2 * p * (1 - p) # 1 - p^2 - (1-p)^2, for two classes
def best_split(Xa, ya):
"""Exhaustive greedy search: every feature, every midpoint between values."""
parent, best = gini(ya), None
for col in Xa.columns:
vals = np.unique(Xa[col])
for thr in (vals[:-1] + vals[1:]) / 2: # candidate midpoints
mask = Xa[col] <= thr
nl, nr = mask.sum(), (~mask).sum()
if nl == 0 or nr == 0:
continue
# weighted child impurity: the WEIGHTING is the whole point
child = (nl * gini(ya[mask.values])
+ nr * gini(ya[~mask.values])) / len(ya)
gain = parent - child
if best is None or gain > best[0]:
best = (gain, col, thr, nl, nr)
return best
sub = Xtr.iloc[:400].reset_index(drop=True)
suby = pd.Series(ytr[:400])
gain, col, thr, nl, nr = best_split(sub, suby)
print(f"parent gini {gini(suby):.4f} on {len(suby)} rows")
print(f"best first split: {col} <= {thr:.3f} gain {gain:.4f}"
f" ({nl} left / {nr} right)")
# Show the search surface: gain as a function of the threshold, per feature.
print("\nGAIN AS THE THRESHOLD SWEEPS (the greedy search, made visible)")
for c in X.columns:
print(f" {c}")
lo, hi = sub[c].min(), sub[c].max()
for t in np.linspace(lo, hi, 11)[1:-1]:
mask = sub[c] <= t
if mask.sum() in (0, len(sub)):
continue
g = gini(suby) - (mask.sum() * gini(suby[mask.values])
+ (~mask).sum() * gini(suby[~mask.values])) / len(sub)
print(f" thr {t:7.3f} gain {g:.4f} {'#' * int(g * 120)}")
# --- THE FULL TREE: recursion, and a stopping rule. ---
tree = DecisionTreeClassifier(max_depth=2, random_state=0).fit(Xtr, ytr)
print("\n" + export_text(tree, feature_names=list(X.columns), decimals=2))
print(f"depth-2 tree: train {tree.score(Xtr, ytr):.4f}"
f" test {tree.score(Xte, yte):.4f}")
print("The tree recovered both thresholds (60 and 0.45) without being told")
print("that an interaction existed. That is what recursive partitioning buys.")
# --- DEPTH IS THE COMPLEXITY DIAL, AND IT OVERFITS FAST. ---
print(f"\n{'depth':>6s} {'leaves':>7s} {'train':>7s} {'test':>7s}")
for d in [1, 2, 3, 5, 8, 12, None]:
t = DecisionTreeClassifier(max_depth=d, random_state=0).fit(Xtr, ytr)
label = "full" if d is None else str(d)
print(f"{label:>6s} {t.get_n_leaves():7d} {t.score(Xtr, ytr):7.4f}"
f" {t.score(Xte, yte):7.4f}")
print("A fully grown tree reaches 1.0 on train by isolating every noisy row in")
print("its own leaf. Perfect training accuracy is the SYMPTOM, not the goal.")
# --- WHAT A TREE CANNOT DO: a diagonal boundary needs many axis-aligned steps.
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
diag = (X["income"] / 140 + X["debt_ratio"] / 0.8 > 1.0).astype(int)
print("\nAN AXIS-ALIGNED MODEL ON A DIAGONAL BOUNDARY")
Xa, Xb, ya, yb = train_test_split(X, diag, test_size=0.3, random_state=0)
for d in [2, 4, 8, None]:
t = DecisionTreeClassifier(max_depth=d, random_state=0).fit(Xa, ya)
lbl = "full" if d is None else f"depth {d}"
print(f" tree {lbl:8s} leaves {t.get_n_leaves():4d}"
f" test {t.score(Xb, yb):.4f}")
lg = make_pipeline(StandardScaler(), LogisticRegression()).fit(Xa, ya)
print(f" logistic regression (2 parameters) test {lg.score(Xb, yb):.4f}")
print("The tree needs dozens of leaves to approximate what one line does")
print("exactly. Trees are not universally better; they are differently biased.")
HOW ONE SPLIT IS CHOSEN: SWEEP EVERY THRESHOLD, KEEP THE BEST GAIN
parent gini 0.4790 on 400 rows
best first split: debt_ratio <= 0.446 gain 0.1614 (234 left / 166 right)
income debt_ratio
thr 32.2 gain 0.029 ### thr 0.081 gain 0.012 #
thr 44.2 gain 0.069 #### thr 0.240 gain 0.061 #######
thr 56.1 gain 0.105 ###### thr 0.400 gain 0.121 ##############
thr 68.1 gain 0.082 ##### thr 0.480 gain 0.150 #############
thr 80.0 gain 0.039 ## thr 0.560 gain 0.112 #############
thr 103.9 gain 0.028 # thr 0.719 gain 0.038 ####
^ ^
best income split a BETTER split, so it wins the root
THE FITTED TREE RECOVERS AN INTERACTION NOBODY DECLARED
|--- debt_ratio <= 0.45
| |--- income <= 59.85 -> class 0
| |--- income > 59.85 -> class 1
|--- debt_ratio > 0.45
| |--- debt_ratio <= 0.57 -> class 0
| |--- debt_ratio > 0.57 -> class 0
true rule: approve if income > 60 AND debt_ratio < 0.45
train 0.9421 test 0.9450 (6% of labels were flipped as noise)
THE PARTITION IS A SET OF RECTANGLES
debt 0.8 +------------+---------------------+
ratio | reject | reject |
0.45+------------+---------------------+
| reject | APPROVE |
0.0 +------------+---------------------+
0 60 140 income
DEPTH IS THE COMPLEXITY DIAL AND IT OVERFITS FAST
depth leaves train test
1 2 0.7614 0.7700
2 4 0.9414 0.9450 <- the true structure, found
3 8 0.9421 0.9450
5 26 0.9486 0.9400
8 63 0.9636 0.9283
12 116 0.9871 0.9067
full 142 1.0000 0.8900 <- memorised the label noise
WHAT AN AXIS-ALIGNED MODEL CANNOT DO
a diagonal boundary, approximated by a staircase:
| \ | +--+
| \ the truth | | +--+ what a tree can express
| \ | | +--+
| \ | | +--+
+----------\ +--+-----------+
tree depth 2 leaves 4 test 0.8467
tree depth 4 leaves 15 test 0.9483
tree full leaves 37 test 0.9683
logistic regression, 2 parameters, test 0.9933
Trees are not better than linear models. They are differently biased.
Key Takeaways
- A tree is a series of splits on one feature at a time. A leaf predicts the most common class or the average target of the training rows that reach it.
- Interactions come for free, because every path from root to leaf is a combination of conditions.
- Trees are not affected by any order-preserving change to a feature, so scaling and log transforms make no difference.
- At each step the tree tries every split and keeps the best, but it never revisits earlier steps, because finding the best tree overall is infeasible.
- Rectangular regions approximate a diagonal boundary badly. Here a two-parameter linear model can beat a forty-leaf tree.
- A fully grown tree reaches perfect training accuracy by isolating noisy rows, so its depth must be limited.
- Greedy splitting makes trees unstable, and that is exactly what ensembles fix.
🧪 Practice
- Implement the full split search for one node using Gini impurity and check that it picks the same first split as sklearn's
DecisionTreeClassifier.- Print the gain for every candidate threshold on two features and explain why one feature wins the root.
- Generate data whose true rule combines two conditions and check that a depth-2 tree finds both thresholds.
- Multiply one feature by 1000 and take the log of another. Check that the tree does not change while a penalised linear model does.
- Build data with a diagonal boundary and report how many leaves a tree needs to match logistic regression's test accuracy.
- Interview: A colleague says trees do not need feature scaling, so they are always the safer default on unfamiliar tabular data. Where does that argument fail? (Hint: what shapes can the boundary take, and what happens outside the training range?)
Splitting Criteria and Pruning
A tree algorithm is defined by two choices: how it scores a possible split, and when it stops splitting. The first matters less than people expect. The second matters a lot.
Impurity measures score how mixed a node is. For classification there are
three options, where p_k is the share of class k in the node:
- Gini impurity,
1 - sum(p_k^2): the chance that two rows picked at random from the node have different labels. - Entropy,
-sum(p_k * log2(p_k)): the number of bits needed to describe a label picked from the node. - Misclassification rate,
1 - max(p_k): the error you make by always predicting the majority class.
Gini and entropy give almost the same trees in practice. Entropy reacts a little more near the extremes, and Gini is slightly faster because it needs no logarithms. The difference in accuracy is almost always smaller than the noise in your evaluation. This is not a setting worth tuning.
Misclassification rate is different, and the reason is useful to understand. It is the metric you actually care about, yet it is a poor way to choose splits. Gini and entropy are curved (strictly concave), so any split that makes the children purer earns some credit. Misclassification rate is made of straight pieces, so it cannot tell apart a split that makes one child perfectly pure from one that only shifts the majorities a little. The greedy search gets stuck where the other two keep improving. So: split using Gini or entropy, and evaluate using the metric you care about.
For regression, the tree uses variance reduction: pick the split that minimises the sum of squared differences from the mean inside each child. It is the same greedy idea with squared error in place of impurity. Splitting on absolute error also exists; it gives leaves that predict the median, but it is slower.
Stopping is what makes or breaks a tree. There are two approaches.
Pre-pruning (early stopping) prevents splits from happening:
max_depthlimits the number of questions in a row;min_samples_leafrefuses leaves below a minimum size;min_samples_splitrefuses to split small nodes;max_leaf_nodeslimits the total number of leaves;min_impurity_decreaserequires each split to improve impurity by a minimum amount.
Pre-pruning is cheap and is what people try first. Its weakness is the horizon effect: a split that looks useless on its own may set up a very useful split just below it, and pre-pruning cuts it off before the benefit is visible. The classic example is an exclusive-or (XOR) pattern, where neither feature on its own reduces impurity at all.
Post-pruning grows the full tree first, then removes branches that are not
worth keeping. Cost-complexity pruning makes "worth keeping" precise: it
minimises error + alpha * (number of leaves). As alpha grows from zero,
the tree shrinks in a fixed sequence of steps, and only a limited number of
alpha values give different trees. Compute that sequence, cross-validate
over it, and pick the best. Post-pruning avoids the horizon effect completely,
because it has already seen what lies below every split. The price is growing
a large tree first.
In practice, trees inside boosted ensembles are pre-pruned to a shallow depth
and do not need post-pruning, because the ensemble controls complexity. For a
single tree that people will read, use cost-complexity pruning with alpha
chosen by cross-validation.
import numpy as np
import pandas as pd
from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor
from sklearn.model_selection import train_test_split, cross_val_score
rng = np.random.default_rng(4)
# --- THE THREE CLASSIFICATION IMPURITY MEASURES, SIDE BY SIDE. ---
def gini(p):
return 1 - p ** 2 - (1 - p) ** 2
def entropy(p):
p = np.clip(p, 1e-12, 1 - 1e-12)
return -(p * np.log2(p) + (1 - p) * np.log2(1 - p))
def misclass(p):
return np.minimum(p, 1 - p)
print(f"{'p(class 1)':>11s} {'gini':>7s} {'entropy':>8s} {'misclass':>9s}")
for p in [0.0, 0.1, 0.25, 0.5, 0.75, 0.9, 1.0]:
print(f"{p:11.2f} {gini(p):7.4f} {entropy(p):8.4f} {misclass(p):9.4f}")
# Why misclassification rate is a bad SPLITTING criterion even though it is the
# metric you care about: it is not strictly concave, so a split that improves
# purity can register zero gain and the greedy search stalls.
def weighted(fn, counts):
tot = sum(a + b for a, b in counts)
return sum((a + b) / tot * fn(a / (a + b)) for a, b in counts)
parent = [(400, 400)]
split_a = [(300, 100), (100, 300)] # both children still predict a majority
split_b = [(400, 200), (0, 200)] # one child made PURE
print("\nA SPLIT THAT MISCLASSIFICATION CANNOT SEE")
print(f" {'criterion':12s} {'parent':>8s} {'split A':>9s} {'split B':>9s}")
for name, fn in [("gini", gini), ("entropy", entropy), ("misclass", misclass)]:
p0, a, b = weighted(fn, parent), weighted(fn, split_a), weighted(fn, split_b)
print(f" {name:12s} {p0:8.4f} {a:9.4f} {b:9.4f}"
f" gains: A {p0-a:+.4f} B {p0-b:+.4f}")
print(" Misclassification scores A and B IDENTICALLY (+0.2500 each), even")
print(" though B makes one child perfectly pure and therefore final. Gini and")
print(" entropy are strictly concave, so both prefer B. Split on gini or")
print(" entropy; EVALUATE with misclassification or your real metric.")
# --- REGRESSION TREES SPLIT ON VARIANCE, NOT PURITY. ---
# The reduction in sum of squared error is the same greedy idea; the leaf
# prediction is the mean of the rows that land there.
x = rng.uniform(0, 10, 300)
y = np.where(x < 4, 2.0, 8.0) + rng.normal(0, 1, 300)
best = None
for thr in np.linspace(0.5, 9.5, 91):
l, r = y[x <= thr], y[x > thr]
if len(l) < 5 or len(r) < 5:
continue
sse = ((l - l.mean()) ** 2).sum() + ((r - r.mean()) ** 2).sum()
if best is None or sse < best[0]:
best = (sse, thr)
print(f"\nREGRESSION SPLIT parent SSE {((y - y.mean())**2).sum():.1f}"
f" best threshold x <= {best[1]:.2f} child SSE {best[0]:.1f}")
print(f" variance reduction {1 - best[0]/((y - y.mean())**2).sum():.1%}"
" (the true breakpoint is 4.00)")
# --- PRE-PRUNING VS POST-PRUNING ON A REAL-ISH PROBLEM. ---
n = 1500
Z = rng.normal(0, 1, (n, 8))
logit = 1.4 * Z[:, 0] - 1.1 * Z[:, 1] + 0.9 * Z[:, 0] * Z[:, 1]
yc = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
Xtr, Xte, ytr, yte = train_test_split(Z, yc, test_size=0.3, random_state=0)
print("\nPRE-PRUNING: stop growing early, using hyperparameters")
print(f" {'setting':32s} {'leaves':>7s} {'train':>7s} {'test':>7s}")
settings = [
("no limits (grow to purity)", {}),
("max_depth=3", {"max_depth": 3}),
("min_samples_leaf=30", {"min_samples_leaf": 30}),
("min_samples_split=50", {"min_samples_split": 50}),
("max_leaf_nodes=12", {"max_leaf_nodes": 12}),
("min_impurity_decrease=0.005", {"min_impurity_decrease": 0.005}),
]
for label, kw in settings:
t = DecisionTreeClassifier(random_state=0, **kw).fit(Xtr, ytr)
print(f" {label:32s} {t.get_n_leaves():7d} {t.score(Xtr, ytr):7.4f}"
f" {t.score(Xte, yte):7.4f}")
# --- POST-PRUNING: grow fully, then remove the splits that do not pay. ---
# Cost-complexity pruning minimises error + ccp_alpha * (number of leaves).
# sklearn computes the alpha values at which each collapse becomes worthwhile.
full = DecisionTreeClassifier(random_state=0).fit(Xtr, ytr)
path = full.cost_complexity_pruning_path(Xtr, ytr)
alphas = path.ccp_alphas[:-1] # drop the root-only tree
print(f"\nPOST-PRUNING: {len(alphas)} candidate alphas from"
f" {alphas.min():.5f} to {alphas.max():.5f}")
print(f" {'ccp_alpha':>10s} {'leaves':>7s} {'train':>7s} {'test':>7s} {'cv':>7s}")
rows = []
for a in [0.0, 0.0005, 0.002, 0.005, 0.01, 0.02, 0.05]:
t = DecisionTreeClassifier(random_state=0, ccp_alpha=a).fit(Xtr, ytr)
cv = cross_val_score(DecisionTreeClassifier(random_state=0, ccp_alpha=a),
Xtr, ytr, cv=5).mean()
rows.append((a, t.get_n_leaves(), t.score(Xtr, ytr), t.score(Xte, yte), cv))
print(f" {a:10.4f} {t.get_n_leaves():7d} {t.score(Xtr, ytr):7.4f}"
f" {t.score(Xte, yte):7.4f} {cv:7.4f}")
best_alpha = max(rows, key=lambda r: r[4])
print(f" cross-validation picks alpha {best_alpha[0]:.4f}"
f" -> {best_alpha[1]} leaves, test {best_alpha[3]:.4f}")
print("\nPost-pruning is the more principled procedure: it lets a weak split")
print("stand if a strong split sits underneath it. Pre-pruning would have cut")
print("the weak split and never discovered the strong one -- the horizon effect.")
THE THREE IMPURITY MEASURES
p(class 1) gini entropy misclass
0.00 0.0000 0.0000 0.0000
0.25 0.3750 0.8113 0.2500
0.50 0.5000 1.0000 0.5000 <- all peak at 0.5
0.75 0.3750 0.8113 0.2500
1.00 0.0000 0.0000 0.0000
impurity
| _--- entropy (scaled)
| / \
| / _-_ \ gini entropy and gini are strictly CONCAVE
| / / \ \ misclass is two straight lines, so it
| | / /\ \ | misclass has no curvature to reward purity
+-+/--/--\-\+--------> p
0 0.5 1
WHY MISCLASSIFICATION IS A BAD SPLITTING CRITERION
parent: 400 / 400
split A: (300/100) and (100/300) both children still mixed
split B: (400/200) and ( 0/200) one child PERFECTLY PURE
criterion parent split A split B gain A gain B
gini 0.5000 0.3750 0.3333 +0.1250 +0.1667 B wins
entropy 1.0000 0.8113 0.6887 +0.1887 +0.3113 B wins
misclass 0.5000 0.2500 0.2500 +0.2500 +0.2500 TIE
Misclassification cannot see that B produced a finished leaf.
REGRESSION SPLITS MINIMISE SQUARED ERROR
parent SSE 2832.2 -> best split x <= 4.00 -> child SSE 300.8
variance reduction 89.4% (the true breakpoint is 4.00)
PRE-PRUNING: SIX WAYS TO STOP EARLY
setting leaves train test
no limits (grow to purity) 181 1.0000 0.6911
max_depth=3 8 0.7781 0.7644
min_samples_leaf=30 25 0.7952 0.7756
min_samples_split=50 46 0.8343 0.7333
max_leaf_nodes=12 12 0.7905 0.7667
min_impurity_decrease=0.005 9 0.7790 0.7622
POST-PRUNING: COST-COMPLEXITY, CROSS-VALIDATED
(84 candidate alphas, from 0.00000 to 0.06666)
ccp_alpha leaves train test cv
0.0000 181 1.0000 0.6911 0.6924
0.0020 50 0.8838 0.7289 0.7095
0.0050 9 0.7790 0.7622 0.7543
0.0100 5 0.7781 0.7644 0.7752 <- cv picks this
0.0200 3 0.7781 0.7644 0.7752
0.0500 3 0.7781 0.7644 0.7752
181 leaves -> 5 leaves, and the held-out score IMPROVES by 7 points.
| Pre-pruning | Post-pruning | |
|---|---|---|
| When it acts | Before a split is made | After the tree is grown |
| Cost | Cheap; never builds the big tree | Must grow the full tree first |
| Horizon effect | Affected | Not affected |
| Main settings | max_depth, min_samples_leaf | ccp_alpha |
| How to choose | Grid search the settings | Cross-validate the alpha path |
| Typical use | Trees inside ensembles | A single readable tree |
Key Takeaways
- Gini and entropy give almost the same trees, so choosing between them is not a meaningful tuning decision.
- Misclassification rate is the metric you care about but a poor splitting rule, because it is not curved and cannot reward a split that purifies one child.
- Regression trees split to minimise squared error inside the children, using the same greedy method.
- Pre-pruning is cheap but suffers from the horizon effect: it can block a useless-looking split that sets up a valuable one.
- Cost-complexity post-pruning grows the tree first, so the horizon effect cannot fool it.
- Only a limited number of alpha values give different pruned trees, so you can list them all and cross-validate.
- Trees inside an ensemble are kept shallow with pre-pruning. A single readable tree should use cost-complexity pruning.
🧪 Practice
- Tabulate Gini, entropy and misclassification rate across class shares. Check that all three peak at 0.5 but only two are strictly curved.
- Rebuild the pair of splits from the example above and check that misclassification rate scores them the same while Gini prefers one.
- Fit trees with
criterion="gini"andcriterion="entropy"on one dataset and compare test accuracy over ten random seeds. Is the difference bigger than the seed-to-seed noise?- Build an XOR dataset and show that setting
min_impurity_decreasetoo high stops the tree from learning it at all.- Compute the full cost-complexity pruning path, cross-validate over it, and plot the number of leaves against training and validation score.
- Interview: Your single decision tree has 400 leaves and 100% training accuracy. Your manager asks whether to cap the depth at 5 or prune afterwards. Which do you recommend and why? (Hint: what happens to a valuable split that sits below a useless-looking one?)
Bagging and Random Forests
A single deep tree has low bias and high variance. It can fit almost any shape, but it fits a slightly different one every time the data changes. Averaging is the fix for exactly this situation.
Here is why. If you average k independent models that each have variance
v, the average has variance v/k. The bias stays the same, because the
average of unbiased things is unbiased. So averaging many low-bias,
high-variance models gives a model with low bias and low variance, as long as
the models are independent.
You cannot get truly independent models from one dataset, but you can get close. Bagging (bootstrap aggregating) works like this:
- Draw a bootstrap sample:
nrows picked at random with replacement. - Fit a tree on it.
- Repeat many times.
- Average the trees' predictions (regression) or take a vote (classification).
Each tree sees a slightly different dataset, so each makes different mistakes, and the mistakes partly cancel out.
Bagging comes with a handy bonus. Sampling n rows with replacement leaves out
a fraction (1 - 1/n)^n of them, which approaches 1/e, about 36.8%. That
tree never saw these out-of-bag (OOB) rows, so they can be used to test
it. Collect the OOB predictions across all trees and you get a validation
score without holding out any data.
The weakness of bagging is that the trees are not different enough. They all
see roughly the same data, so they all pick the same strong split at the root,
and their errors are correlated. If k models each have variance v and
pairwise correlation rho, the average has variance rho*v + (1-rho)*v/k,
not v/k. The second term shrinks as you add trees; the first term does
not. Correlation sets a floor on how much averaging can help.
Random forests lower that floor. On top of bootstrapping the rows, at every split each tree may only look at a random subset of the features. If the strongest feature is not in the subset, the tree has to find a different split, so the trees become less alike.
This is a trade, not a free gain, and that is often missed. Limiting the
features makes each single tree worse. You pay some bias to buy less
correlation. The best setting is somewhere in the middle and depends on the
problem. That is why max_features is a hyperparameter, and why scikit-learn's
default for regression is all features (plain bagging) while its default for
classification is sqrt(p). If you believe "random forests always beat
bagging", read the example below carefully.
Extremely randomised trees (Extra Trees) go further: they pick thresholds at random instead of searching for the best one. This makes the trees even less alike and much faster to fit.
Two properties of forests are worth stressing:
- Adding trees almost never causes overfitting. More trees only reduce
variance; they never increase it. So
n_estimatorsis a question of compute time, not of risk. - Forests cannot extrapolate. A prediction is an average of leaf values, and a leaf can only report the average of rows it has seen. Beyond the training range the prediction goes flat.
import numpy as np
import pandas as pd
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import (BaggingRegressor, RandomForestRegressor,
ExtraTreesRegressor)
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(6)
# --- WHY AVERAGING WORKS: variance falls, bias does not. ---
# A deep tree is a low-bias, high-variance estimator. Fit it on 40 different
# samples from the same process and watch how far apart the fits are.
def make(n, seed):
r = np.random.default_rng(seed)
X = r.uniform(-3, 3, (n, 1))
y = np.sin(X[:, 0] * 1.5) + r.normal(0, 0.35, n)
return X, y
grid = np.linspace(-3, 3, 200).reshape(-1, 1)
preds = np.array([DecisionTreeRegressor(random_state=0).fit(*make(120, s))
.predict(grid) for s in range(40)])
truth = np.sin(grid[:, 0] * 1.5)
print("ONE DEEP TREE, 40 INDEPENDENT TRAINING SAMPLES")
print(f" pointwise SD across the 40 fits : {preds.std(axis=0).mean():.4f}"
" <- the variance term")
print(f" MSE of a typical single tree : {np.mean((preds - truth)**2):.4f}")
print(f" MSE of the average of the 40 trees :"
f" {np.mean((preds.mean(axis=0) - truth)**2):.4f} <- bias^2 only")
print("Averaging removes the variance term and leaves the bias untouched. That")
print("is the entire theory of bagging, in two numbers.")
# --- BAGGING: bootstrap the rows, fit a tree on each, average. ---
X, y = make(600, 100)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
def rmse(m, Xa, ya):
return float(np.sqrt(np.mean((ya - m.predict(Xa)) ** 2)))
base = DecisionTreeRegressor(random_state=0).fit(Xtr, ytr)
print(f"\nsingle unpruned tree test RMSE {rmse(base, Xte, yte):.4f}")
print(f"{'n_estimators':>13s} {'bagged RMSE':>12s}")
for k in [1, 3, 10, 30, 100, 300]:
bag = BaggingRegressor(DecisionTreeRegressor(random_state=0),
n_estimators=k, random_state=0).fit(Xtr, ytr)
print(f"{k:13d} {rmse(bag, Xte, yte):12.4f}")
print("Error drops steeply then flattens. More trees never HURT accuracy, they")
print("just stop helping, so n_estimators is a compute budget, not a risk.")
# --- OUT-OF-BAG ROWS: free validation data, from the bootstrap itself. ---
print(f"\neach bootstrap sample omits (1 - 1/n)^n of the rows -> 1/e ="
f" {1/np.e:.4f}")
for n in [10, 100, 1000, 10000]:
print(f" n={n:5d}: {(1 - 1/n)**n:.4f}")
# --- RANDOM FORESTS: bagging PLUS a random feature subset at every split. ---
# The setup that makes the mechanism visible: 8 latent signals, each duplicated
# five times with a little noise. Every bagged tree keeps finding the SAME
# dominant column first, so the trees stay correlated and averaging saturates.
m = 1200
S = rng.normal(0, 1, (m, 8)) # the 8 real signals
Z = np.column_stack([S[:, j] + rng.normal(0, 0.15, m)
for j in range(8) for _ in range(5)]) # 40 columns
w = np.array([3.0, 2.2, 1.8, 1.5, 1.2, 1.0, 0.8, 0.6])
t = S @ w + rng.normal(0, 1.2, m)
Ztr, Zte, ttr, tte = train_test_split(Z, t, test_size=0.3, random_state=0)
def tree_corr(est, Xa):
"""Mean pairwise correlation between the member trees' predictions."""
P = np.array([tr.predict(Xa) for tr in est.estimators_])
C = np.corrcoef(P)
return float(C[np.triu_indices_from(C, 1)].mean())
print(f"\n40 columns carrying only 8 distinct signals")
print(f" {'max_features':>13s} {'test RMSE':>10s} {'tree corr':>10s}")
for mf in [None, 20, 13, 6, 3]:
est = RandomForestRegressor(n_estimators=300, max_features=mf,
random_state=0).fit(Ztr, ttr)
label = "all 40" if mf is None else str(mf)
print(f" {label:>13s} {rmse(est, Zte, tte):10.4f} {tree_corr(est, Zte):10.3f}")
xt = ExtraTreesRegressor(n_estimators=300, max_features=13,
random_state=0).fit(Ztr, ttr)
print(f" {'extra trees':>13s} {rmse(xt, Zte, tte):10.4f}"
f" {tree_corr(xt, Zte):10.3f} <- random thresholds too")
print("\nCorrelation falls monotonically as max_features shrinks, but RMSE has an")
print("INTERIOR optimum: subsampling also makes each tree worse. It is a")
print("bias-for-decorrelation trade, and the right amount is a tuned value --")
print("which is why sklearn's regressor default is all features, not sqrt(p).")
# The algebra behind the trade-off: averaging k estimators with pairwise
# correlation rho leaves a floor of rho*var that more trees cannot remove.
print("\nvariance of an average of k=200 estimators, each with variance 1:")
for rho in [1.0, 0.8, 0.5, 0.2, 0.0]:
print(f" rho={rho:.1f} -> rho*var + (1-rho)*var/k ="
f" {rho + (1 - rho) / 200:.4f}")
# --- OOB VERSUS A HELD-OUT SET. ---
rf = RandomForestRegressor(n_estimators=300, max_features=13, oob_score=True,
random_state=0).fit(Ztr, ttr)
oob = float(np.sqrt(np.mean((ttr - rf.oob_prediction_) ** 2)))
print(f"\nOOB RMSE {oob:.4f} held-out test RMSE {rmse(rf, Zte, tte):.4f}")
print("Each row is scored by only the ~37% of trees that excluded it, which")
print("biases OOB pessimistic in EXPECTATION -- but that bias is small next to")
print("the variation between one test split and another, which is why OOB came")
print("out lower here. Good enough to tune on when data is scarce; not a")
print("substitute for a held-out set when you are reporting a final number.")
# --- WHAT A FOREST CANNOT DO: extrapolate. ---
xr = np.linspace(0, 10, 300).reshape(-1, 1)
yr = 2.0 * xr[:, 0] + rng.normal(0, 0.5, 300)
rf1 = RandomForestRegressor(n_estimators=200, random_state=0).fit(xr, yr)
print(f"\nEXTRAPOLATION (trained on x in [0, 10], max observed y"
f" {yr.max():.1f})")
for v in [5.0, 10.0, 15.0, 50.0]:
print(f" x={v:5.1f} forest {rf1.predict([[v]])[0]:7.3f} true {2*v:7.3f}")
print(" Past the training range the prediction is a constant, because a leaf")
print(" can only report the mean of rows it actually saw. Linear models")
print(" extrapolate (sometimes wrongly); trees refuse to.")
WHY AVERAGING WORKS, IN TWO NUMBERS
one deep tree, 40 independent training samples:
pointwise SD across the 40 fits 0.3453 <- the variance term
MSE of a typical single tree 0.1238
MSE of the average of the 40 0.0027 <- bias^2 only
45x lower error, and nothing about the individual trees changed.
BAGGING: MORE TREES HELP, THEN STOP HELPING
single unpruned tree 0.4879
n_estimators bagged test RMSE
1 0.4824
3 0.4443
10 0.4435
30 0.4357
100 0.4349
300 0.4338
RMSE
| *
| \
| *___
| \____*_______*_________* a floor, not a turn upward
+------------------------------> n_estimators
OUT-OF-BAG ROWS ARE FREE VALIDATION DATA
a bootstrap sample of n rows omits (1 - 1/n)^n of them:
n=10 0.3487
n=100 0.3660
n=1000 0.3677 -> 1/e = 0.3679
n=10000 0.3679
row: 1 2 3 4 5 6 7 8
tree 1: x x x x x (rows 3,5,8 are OUT OF BAG)
tree 2: x x x x x (rows 1,6,7 are OUT OF BAG)
tree 3: x x x x x x (rows 2,4 are OUT OF BAG)
score each row with only the trees that never saw it.
FEATURE SUBSAMPLING IS A TRADE, NOT A GIFT
(40 columns carrying only 8 distinct signals)
max_features test RMSE tree corr
all 40 2.3334 0.629 <- plain bagging
20 2.3243 0.627
13 2.2923 0.617
6 2.2756 0.587 <- best RMSE
3 2.3025 0.500 <- lowest correlation, worse RMSE
extra trees 2.1776 0.608
Correlation falls monotonically; RMSE has an INTERIOR optimum.
Fewer candidate features = more decorrelation AND a worse tree.
THE ALGEBRA THAT EXPLAINS THE FLOOR
variance of an average of k=200 estimators, each with variance 1:
rho=1.0 -> 1.0000 averaging does NOTHING when trees agree
rho=0.8 -> 0.8010
rho=0.5 -> 0.5025
rho=0.2 -> 0.2040
rho=0.0 -> 0.0050 the ideal, unreachable in practice
FORESTS DO NOT EXTRAPOLATE
(trained on x in [0, 10], max observed y = 20.6)
x= 5.0 forest 10.003 true 10.000
x=10.0 forest 20.430 true 20.000
x=15.0 forest 20.430 true 30.000 <- flat
x=50.0 forest 20.430 true 100.000 <- still flat
y | true /
| /
| /___________ forest: constant past the last leaf
| /
+-----------------------> x
10
Key Takeaways
- Averaging
kindependent models divides the variance bykand leaves the bias unchanged. That is why averaging rescues unstable trees.- Bagging imitates independence with bootstrap samples. Each sample leaves about 36.8% of rows out of bag, which gives free validation.
- Correlated trees leave a floor of
rho * variancethat more trees cannot remove.- Random forests lower that floor by limiting the features each split may use. The trees become less alike, but each one gets worse.
- So
max_featureshas a best value in the middle and must be tuned. Forests do not always beat bagging.- More trees never hurt a bagged model or a forest, so
n_estimatorsis a compute budget, not an overfitting risk.- Forests cannot extrapolate, because a leaf can only report the average of rows it saw in training.
🧪 Practice
- Fit one deep tree on forty independent samples from the same source. Compare the mean squared error of a single fit with that of the average fit.
- Check numerically that
(1 - 1/n)^napproaches1/e. Then compare a forest's out-of-bag score with a held-out test score.- Vary
max_featuresfrom 1 to all features. Record test error and the average correlation between the trees' predictions.- Find a dataset where plain bagging beats the default random forest, and explain which feature structure causes it.
- Train a forest on
xin [0, 10] and predict atx = 20. Do the same with linear regression. Which kind of failure would you rather have?- Compare
ExtraTreesRegressorwithRandomForestRegressoron fit time and accuracy using the samen_estimators.- Interview: You raise
n_estimatorsfrom 100 to 2000 in a random forest and the test score does not move. What does that tell you, and what would you change next? (Hint: which of the two variance terms is already used up, and which one is left?)
Gradient Boosted Trees
Bagging starts with models that are too flexible and averages away their variance. Boosting starts at the other end. It begins with a model that is far too simple, then keeps adding small corrections aimed at whatever the current model gets wrong. Bagging fits its trees separately and in parallel. Boosting fits them one after another, and each new tree looks at the mistakes of the ones before it.
The easiest way to see it is with squared-error regression:
- Start by predicting the mean.
- Compute the residuals (actual minus predicted).
- Fit a small tree to the residuals.
- Add a small fraction of that tree's predictions to the model.
- Recompute the residuals and repeat.
Each tree patches the current model's mistakes. Adding up many small patches gives a flexible model built from simple trees.
It is called gradient boosting because fitting the residual is a special
case of a more general idea. For squared error 0.5*(y - F)^2, the negative
gradient with respect to the prediction F is exactly the residual y - F.
So "fit the residual" is the same as "fit the negative gradient". Once you see
it that way, you can boost any loss that has a gradient: log loss for
classification, Poisson deviance for counts, quantile loss for uneven costs.
Each round fits a tree to the negative gradient and takes a step. It is
gradient descent (chapter 2.2), but each step adds a tree instead of changing
a parameter.
Three hyperparameters do most of the work.
Learning rate (shrinkage) scales how much each tree contributes. Smaller steps mean no single tree dominates, and the final fit is smoother. This is the most important regularisation setting.
Number of rounds works together with the learning rate. Halve the rate and you need about twice as many rounds for the same fit. Tune them together, or fix a small rate and let early stopping choose the number of rounds.
Tree depth sets the most complex interaction the model can learn. A stump (depth 1) cannot capture interactions at all; depth 3 can capture three-way ones. Depths of 3 to 8 are typical, much shallower than the fully grown trees in a forest.
The key practical difference from bagging: boosting can overfit when you add more trees. In a forest, extra trees only average. In boosting, extra trees keep chasing the remaining residuals, including residuals that are just noise. So the held-out error can start rising, and the number of trees needs early stopping rather than "as many as I can afford".
That said, the classic U-shaped curve needs some nuance. With a small learning rate, boosting overfits slowly. The held-out error usually reaches a shallow minimum and then stays almost flat for a long time, and training error reaches zero long before held-out error gets noticeably worse. With a large learning rate, it overfits within a dozen rounds. The example below shows both cases. Knowing which case you are in tells you how careful you need to be with early stopping.
Stochastic gradient boosting fits each tree on a random subset of the rows. This adds regularisation and speed at no cost, and is usually worth turning on.
import numpy as np
import pandas as pd
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import (GradientBoostingRegressor, RandomForestRegressor,
HistGradientBoostingRegressor)
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(8)
# --- BOOSTING BY HAND: fit the RESIDUALS, over and over. ---
# Squared-error boosting is the clearest case, because the negative gradient of
# 0.5*(y - F)^2 with respect to F is exactly the residual (y - F).
n = 500
x = np.sort(rng.uniform(0, 10, n)).reshape(-1, 1)
y = np.sin(x[:, 0]) + 0.35 * x[:, 0] + rng.normal(0, 0.4, n)
F = np.full(n, y.mean()) # stage 0: the best constant
LR, STUMPS = 0.1, 200 # learning rate and number of rounds
trace = []
for stage in range(STUMPS):
residual = y - F # = negative gradient for squared error
h = DecisionTreeRegressor(max_depth=2).fit(x, residual) # a WEAK learner
F = F + LR * h.predict(x) # shrunken step, not a full one
if stage in (0, 1, 4, 19, 49, 99, 199):
trace.append((stage + 1, float(np.sqrt(np.mean((y - F) ** 2)))))
print("BOOSTING IS GRADIENT DESCENT IN FUNCTION SPACE")
print(f" stage 0 (predict the mean): train RMSE"
f" {np.sqrt(np.mean((y - y.mean())**2)):.4f}")
for stage, r in trace:
print(f" after {stage:3d} stumps : train RMSE {r:.4f}")
sk = GradientBoostingRegressor(n_estimators=200, learning_rate=0.1, max_depth=2,
random_state=0).fit(x, y)
print(f" sklearn, same settings : train RMSE"
f" {np.sqrt(np.mean((y - sk.predict(x))**2)):.4f}")
# --- BAGGING VS BOOSTING: parallel and low-variance vs sequential and low-bias.
m = 4000
Z = rng.normal(0, 1, (m, 10))
t = (2.0 * Z[:, 0] + 1.5 * np.abs(Z[:, 1]) - 1.2 * Z[:, 2] * Z[:, 3]
+ 0.8 * np.sin(3 * Z[:, 4]) + rng.normal(0, 0.8, m))
Ztr, Zte, ttr, tte = train_test_split(Z, t, test_size=0.3, random_state=0)
def rmse(m_, Xa, ya):
return float(np.sqrt(np.mean((ya - m_.predict(Xa)) ** 2)))
print(f"\n{'model':44s} {'train':>7s} {'test':>7s}")
for label, est in [
("single stump (depth 2)", DecisionTreeRegressor(max_depth=2, random_state=0)),
("random forest, 300 deep trees",
RandomForestRegressor(n_estimators=300, random_state=0)),
("boosted, 300 stumps, lr=0.1",
GradientBoostingRegressor(n_estimators=300, learning_rate=0.1, max_depth=2,
random_state=0)),
("boosted, 300 depth-3 trees, lr=0.1",
GradientBoostingRegressor(n_estimators=300, learning_rate=0.1, max_depth=3,
random_state=0)),
]:
est.fit(Ztr, ttr)
print(f" {label:42s} {rmse(est, Ztr, ttr):7.4f} {rmse(est, Zte, tte):7.4f}")
print("A forest averages LOW-BIAS, high-variance trees. Boosting sums")
print("HIGH-BIAS, low-variance trees. Opposite starting points, both effective.")
# --- THE LEARNING RATE / N_ESTIMATORS TRADE-OFF. ---
# They multiply: halving the learning rate needs roughly twice the rounds.
print(f"\n{'lr':>6s} {'rounds':>7s} {'train':>7s} {'test':>7s} note")
for lr, k, note in [(1.0, 300, "overshoots, memorises fast"),
(0.3, 300, "good here, but fragile"),
(0.1, 300, "the usual default"),
(0.1, 1000, "more rounds at the same lr"),
(0.03, 1000, "slower, and not yet caught up"),
(0.03, 300, "UNDERTRAINED: stopped too early")]:
g = GradientBoostingRegressor(n_estimators=k, learning_rate=lr, max_depth=3,
random_state=0).fit(Ztr, ttr)
print(f"{lr:6.2f} {k:7d} {rmse(g, Ztr, ttr):7.4f} {rmse(g, Zte, tte):7.4f}"
f" {note}")
print("Shrinkage is the main regulariser: small steps mean each tree corrects a")
print("little, so no single tree's mistakes dominate. Lower lr needs more rounds.")
# --- BOOSTING CAN OVERFIT, UNLIKE BAGGING. EARLY STOPPING IS NOT OPTIONAL. ---
# Overfitting is a small-data, high-noise, large-step phenomenon, so build a
# dataset that has all three and watch the held-out curve at two learning rates.
mo = 700
Zo = rng.normal(0, 1, (mo, 10))
sig = (2.0 * Zo[:, 0] + 1.5 * np.abs(Zo[:, 1]) - 1.2 * Zo[:, 2] * Zo[:, 3]
+ 0.8 * np.sin(3 * Zo[:, 4]))
to = sig + rng.normal(0, 1.5, mo) # noise as large as the signal
Otr, Ote, otr, ote = train_test_split(Zo, to, test_size=0.4, random_state=0)
print("\nTHE HELD-OUT CURVE, AT TWO LEARNING RATES (700 rows, heavy noise)")
for lr in (0.1, 0.5):
gb = GradientBoostingRegressor(n_estimators=800, learning_rate=lr,
max_depth=5, random_state=0).fit(Otr, otr)
# staged_predict replays the ensemble one round at a time -- the cheapest
# way to see the whole learning curve from a single fit.
st = np.array([np.sqrt(np.mean((ote - p) ** 2))
for p in gb.staged_predict(Ote)])
tr = np.array([np.sqrt(np.mean((otr - p) ** 2))
for p in gb.staged_predict(Otr)])
best = int(np.argmin(st))
print(f" learning rate {lr}")
print(f" {'round':>6s} {'train':>7s} {'test':>7s}")
for k in sorted({10, 50, best + 1, 200, 400, 800}):
mark = " <- best" if k == best + 1 else ""
print(f" {k:6d} {tr[k-1]:7.4f} {st[k-1]:7.4f}{mark}")
print(f" best {st[best]:.4f} at round {best+1}, final {st[-1]:.4f}"
f" (degraded {100*(st[-1]/st[best]-1):.1f}%)")
print("At lr=0.1 the curve has a shallow minimum and then PLATEAUS: boosting")
print("with small steps overfits slowly, which is why it is forgiving in")
print("practice. At lr=0.5 it overfits within a dozen rounds. In both cases")
print("training error goes to zero, so it tells you nothing.")
# Built-in early stopping: hold out a slice and stop when it stops improving.
es = GradientBoostingRegressor(n_estimators=3000, learning_rate=0.1, max_depth=5,
validation_fraction=0.15, n_iter_no_change=25,
random_state=0).fit(Otr, otr)
print(f"\nearly stopping fired at {es.n_estimators_} rounds, test RMSE"
f" {rmse(es, Ote, ote):.4f}")
print("n_estimators is not a hyperparameter to tune by hand when the library")
print("can find it for you from a validation slice.")
# --- SUBSAMPLING ROWS: 'stochastic gradient boosting'. ---
print(f"\n{'subsample':>10s} {'test RMSE':>10s}")
for ss in [1.0, 0.8, 0.5]:
g = GradientBoostingRegressor(n_estimators=400, learning_rate=0.1,
max_depth=4, subsample=ss,
random_state=0).fit(Ztr, ttr)
print(f"{ss:10.1f} {rmse(g, Zte, tte):10.4f}")
BOOSTING IS GRADIENT DESCENT IN FUNCTION SPACE
F_0 = mean(y)
repeat: r = y - F (the negative gradient of squared error)
h = tree.fit(X, r)
F = F + 0.1 * h(X)
stage 0 (predict the mean) train RMSE 1.2198
after 1 stumps train RMSE 1.1208
after 2 stumps train RMSE 1.0335
after 5 stumps train RMSE 0.8279
after 20 stumps train RMSE 0.4628
after 50 stumps train RMSE 0.3777
after 200 stumps train RMSE 0.3349
sklearn, same settings train RMSE 0.3349 <- identical
BAGGING AND BOOSTING START FROM OPPOSITE ENDS
model train test
single stump (depth 2) 1.8518 1.8147
random forest, 300 deep trees 0.5356 1.3391
boosted, 300 stumps, lr=0.1 1.1430 1.2600
boosted, 300 depth-3 trees, lr=0.1 0.7954 1.0575
forest: average LOW-bias, HIGH-variance trees -> kill variance
boosting: sum HIGH-bias, LOW-variance trees -> kill bias
LEARNING RATE AND ROUNDS MULTIPLY
lr rounds train test note
1.00 300 0.1714 1.3399 overshoots, memorises fast
0.30 300 0.4446 1.0010 good here, but fragile
0.10 300 0.7954 1.0575 the usual default
0.10 1000 0.4177 0.9347 more rounds at the same lr
0.03 1000 0.8843 1.1651 slower, and not yet caught up
0.03 300 1.2279 1.3058 UNDERTRAINED: stopped too early
THE HELD-OUT CURVE, AT TWO LEARNING RATES
(700 rows, noise as large as the signal)
lr = 0.1 lr = 0.5
round train test round train test
10 1.7112 2.2575 10 0.6237 2.1434
50 0.6027 1.9974 11 0.6011 2.1311 <- best
150 0.1654 1.9690 <- best 50 0.0280 2.1688
200 0.0855 1.9714 200 0.0000 2.1691
400 0.0075 1.9738 400 0.0000 2.1691
800 0.0001 1.9736 800 0.0000 2.1691
degraded 0.2% from the best degraded 1.8% from the best
error
| train \ small steps: shallow minimum,
| \___ then a long plateau
| \_____________
| test \__/----------------- large steps: minimum in a
+---------------------------> round dozen rounds, then flat
In BOTH cases training error reaches zero, so it tells you nothing.
early stopping fired at 82 rounds, test RMSE 2.0939
STOCHASTIC GRADIENT BOOSTING: SUBSAMPLE THE ROWS
subsample test RMSE
1.0 0.9942
0.8 0.9002 <- cheaper AND better
0.5 0.9207
| Random forest | Gradient boosting | |
|---|---|---|
| How trees are fitted | Separately, in parallel | One after another, on residuals |
| Kind of tree | Deep, low bias | Shallow, high bias |
| What combining fixes | Variance | Bias |
| More trees | Never hurts | Can overfit |
| Needs early stopping | No | Yes |
| Main settings | max_features, n_estimators | learning_rate, depth, rounds |
| Sensitivity to tuning | Low; good out of the box | High; rewards careful tuning |
| Free validation | Out-of-bag score | None; hold out a slice |
Key Takeaways
- Boosting fits each new tree to the current model's errors and adds a small fraction of it, so the final model is a sum of small corrections.
- Fitting the residual is the same as fitting the negative gradient of squared error, which is why boosting works with any loss that has a gradient.
- Learning rate and number of rounds trade off against each other. The learning rate is the main regularisation setting.
- Tree depth limits the most complex interaction the model can learn, and boosted trees are much shallower than forest trees.
- Unlike bagging, adding rounds can make held-out results worse, so early stopping on a validation slice is standard.
- With a small learning rate, overfitting is slow and the curve flattens. With a large one, it shows up within a dozen rounds.
- Zero training error does not tell you when to stop. Only held-out data can.
- Fitting each tree on a sample of rows usually improves both speed and accuracy, so turn it on by default.
🧪 Practice
- Implement squared-error boosting by hand with depth-2 trees and a learning rate of 0.1, and match
GradientBoostingRegressoron training RMSE.- Show that halving the learning rate needs roughly twice the rounds to reach the same training error.
- Use
staged_predictto plot training and held-out error against the number of rounds at learning rates of 0.03, 0.1 and 0.5. Describe how the shape changes.- Turn on
validation_fractionandn_iter_no_change, and compare where early stopping fires with the true held-out minimum.- Compare
subsamplevalues of 1.0, 0.8 and 0.5 on accuracy and fit time.- Boost with a quantile loss at the 0.9 quantile and check that about 90% of held-out targets fall below the prediction.
- Interview: Your boosted model's training error is 0.001, and a colleague says this proves it has converged and you should stop adding trees. What is wrong with this reasoning? (Hint: which curve decides when to stop, and what does the other curve actually measure?)
XGBoost, LightGBM, CatBoost
Gradient boosting as described so far is slow, because finding the best split means sorting every feature at every node. Three libraries rebuilt it to be fast and to handle real tabular data well. Together they power most production models on tabular data. The example in this topic is a classification task, but everything here applies to regression too.
The idea all three share is histogram-based splitting. Each continuous feature is grouped into about 256 bins once, at the start. The split search then only checks bin boundaries instead of every distinct value. It only needs the sum of gradients in each bin, so instead of sorting at every node it does a quick scan over the bins.
XGBoost came first and made two more ideas popular:
- It adds L1 and L2 penalties on the leaf values directly into the boosting objective, so regularisation is part of how splits are scored.
- It handles missing values by learning a default direction for each split. Rows with a missing value go left or right, whichever reduces the loss more. Missingness becomes something the model learns from, instead of something you have to fill in.
XGBoost grows trees level by level: it splits every node at the current
depth before going deeper. This gives balanced trees and makes max_depth a
meaningful limit.
LightGBM grows trees leaf by leaf instead: at each step it splits
whichever leaf would reduce the loss the most, anywhere in the tree. The trees
become deep and lopsided, spending their complexity where the errors are. This
is usually faster and more accurate for the same number of leaves, but it
overfits sooner, so num_leaves must be paired with min_child_samples. One
trap: with leaf-wise growth, num_leaves = 2^max_depth is not the same as a
level-wise tree of that depth, because the leaves are spent unevenly. LightGBM
also has two more tricks (gradient-based one-side sampling and exclusive
feature bundling) that mainly help on very wide, sparse data.
CatBoost focuses on categorical features. Instead of one-hot encoding a column with 300 levels into 300 sparse columns, it replaces each level with a number computed from the target. This is called target encoding. Done naively, it leaks: a row's own target goes into its own encoding, and the model learns that leak. CatBoost uses ordered target statistics: rows are put in a random order, and each row's encoding only uses rows that come before it. That removes the leak. CatBoost also uses symmetric (oblivious) trees, where every node at the same depth uses the same split. The tree then works like a lookup table, which makes prediction very fast.
Scikit-learn's HistGradientBoosting* uses the same histogram method,
needs no extra library, and is a perfectly good default. It has real limits,
though: native categorical features can have at most 255 levels, which the
example below runs into.
In honest terms: on most tabular problems the accuracy differences between these libraries are small. The choice comes down to fit time, how categorical features are handled, and which library your team already knows. The one clear exception is categorical features with many levels, where CatBoost's ordered target statistics give a real advantage.
import time
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
import xgboost as xgb
import lightgbm as lgb
from catboost import CatBoostClassifier
rng = np.random.default_rng(17)
n = 40_000
# A realistic mix: numeric features, a HIGH-CARDINALITY categorical, and
# missing values -- the three things these libraries handle differently.
city = rng.choice([f"city_{i:03d}" for i in range(300)], n) # 300 levels
city_effect = {c: v for c, v in zip(np.unique(city),
rng.normal(0, 0.7, len(np.unique(city))))}
num = rng.normal(0, 1, (n, 12))
logit = (1.1 * num[:, 0] - 0.9 * num[:, 1] + 0.7 * num[:, 2] * num[:, 3]
+ np.array([city_effect[c] for c in city]) + rng.normal(0, 0.4, n))
y = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
df = pd.DataFrame(num, columns=[f"f{i}" for i in range(12)])
df["city"] = pd.Categorical(city)
# Punch holes in one column: each library has its own missing-value policy.
mask = rng.random(n) < 0.15
df.loc[mask, "f0"] = np.nan # f0 is the STRONGEST predictor
Xtr, Xte, ytr, yte = train_test_split(df, y, test_size=0.25, random_state=0,
stratify=y)
print(f"{n:,} rows, 12 numeric + 1 categorical with 300 levels,"
f" 15% missing in f0, base rate {y.mean():.3f}")
results = []
# --- XGBOOST: histogram splits, and a LEVEL-WISE tree that grows depth by depth.
t0 = time.perf_counter()
xg = xgb.XGBClassifier(
n_estimators=400, learning_rate=0.08, max_depth=6,
subsample=0.8, colsample_bytree=0.8,
reg_lambda=1.0, # L2 on leaf weights, on by default
enable_categorical=True, tree_method="hist", # needed for pandas category
eval_metric="auc", random_state=0)
xg.fit(Xtr, ytr)
results.append(("XGBoost", time.perf_counter() - t0,
roc_auc_score(yte, xg.predict_proba(Xte)[:, 1])))
# --- LIGHTGBM: LEAF-WISE growth. It splits whichever leaf reduces loss most,
# so trees are deeper and narrower. Fast, and easier to overfit.
t0 = time.perf_counter()
lg = lgb.LGBMClassifier(
n_estimators=400, learning_rate=0.08, num_leaves=63,
min_child_samples=40, # the main guard against leaf-wise overfit
subsample=0.8, subsample_freq=1, colsample_bytree=0.8,
verbose=-1, random_state=0)
lg.fit(Xtr, ytr, categorical_feature=["city"])
results.append(("LightGBM", time.perf_counter() - t0,
roc_auc_score(yte, lg.predict_proba(Xte)[:, 1])))
# --- CATBOOST: ordered target statistics for categoricals, plus symmetric
# (oblivious) trees where every node at a depth shares one split.
t0 = time.perf_counter()
Xtr_cb = Xtr.copy(); Xte_cb = Xte.copy()
for d in (Xtr_cb, Xte_cb):
d["city"] = d["city"].astype(str) # CatBoost wants strings, not NaN cats
cb = CatBoostClassifier(iterations=400, learning_rate=0.08, depth=6,
cat_features=["city"], verbose=0, random_seed=0)
cb.fit(Xtr_cb, ytr)
results.append(("CatBoost", time.perf_counter() - t0,
roc_auc_score(yte, cb.predict_proba(Xte_cb)[:, 1])))
# --- SKLEARN'S HISTOGRAM BOOSTER: the same algorithm, no extra dependency. ---
t0 = time.perf_counter()
try:
hg = HistGradientBoostingClassifier(
max_iter=400, learning_rate=0.08, max_leaf_nodes=63,
categorical_features=["city"], random_state=0).fit(Xtr, ytr)
results.append(("sklearn HistGB", time.perf_counter() - t0,
roc_auc_score(yte, hg.predict_proba(Xte)[:, 1])))
except ValueError as e:
# A real, concrete limitation worth knowing before you choose a library.
print(f"\nsklearn HistGB refused: {e}")
# Fall back to ordinal codes, which is what you would have to do by hand.
Xtr_o = Xtr.assign(city=Xtr["city"].cat.codes)
Xte_o = Xte.assign(city=Xte["city"].cat.codes)
t0 = time.perf_counter()
hg = HistGradientBoostingClassifier(
max_iter=400, learning_rate=0.08, max_leaf_nodes=63,
random_state=0).fit(Xtr_o, ytr)
results.append(("sklearn HistGB (ordinal)", time.perf_counter() - t0,
roc_auc_score(yte, hg.predict_proba(Xte_o)[:, 1])))
print(f"\n{'library':26s} {'fit seconds':>12s} {'test AUC':>9s}")
for name, secs, auc in results:
print(f"{name:26s} {secs:12.2f} {auc:9.4f}")
print("The accuracy spread is small. The differences that matter in practice are")
print("fit time, categorical handling, and how each one wants to be tuned.")
# --- LEVEL-WISE VS LEAF-WISE, MADE CONCRETE. ---
print("\nGROWTH STRATEGY: same leaf budget, very different trees")
for nl in [15, 63, 255]:
l = lgb.LGBMClassifier(n_estimators=150, num_leaves=nl, learning_rate=0.08,
min_child_samples=40, verbose=-1,
random_state=0).fit(Xtr, ytr,
categorical_feature=["city"])
depth_equiv = int(np.log2(nl + 1))
x2 = xgb.XGBClassifier(n_estimators=150, max_depth=depth_equiv,
learning_rate=0.08, enable_categorical=True,
tree_method="hist", random_state=0).fit(Xtr, ytr)
print(f" num_leaves {nl:4d} (LightGBM) AUC"
f" {roc_auc_score(yte, l.predict_proba(Xte)[:, 1]):.4f} vs"
f" max_depth {depth_equiv} (XGBoost) AUC"
f" {roc_auc_score(yte, x2.predict_proba(Xte)[:, 1]):.4f}")
print(" LightGBM's num_leaves is NOT 2^max_depth in effect: leaf-wise growth")
print(" spends its budget where the loss is, so the same count is more")
print(" expressive and overfits sooner. Tune num_leaves with min_child_samples.")
# --- MISSING VALUES: learned, not imputed. ---
print("\nMISSING VALUES f0 has 15% NaN and none of these models needed an imputer")
print(" Each split learns a DEFAULT DIRECTION for missing rows, chosen by")
print(" whichever side reduces the loss more. That is strictly more expressive")
print(" than mean imputation, which asserts the missing value was average.")
Xte_nan = Xte.copy(); Xte_nan["f0"] = np.nan
print(f" test AUC with f0 entirely missing at predict time:"
f" {roc_auc_score(yte, xg.predict_proba(Xte_nan)[:, 1]):.4f}"
f" (was {results[0][2]:.4f})")
# --- HIGH-CARDINALITY CATEGORICALS: the reason CatBoost exists. ---
print("\nHIGH-CARDINALITY CATEGORICAL: 300 cities")
one_hot = pd.get_dummies(Xtr.drop(columns="city"), dtype=float)
oh_tr = pd.concat([one_hot, pd.get_dummies(Xtr["city"], dtype=float)], axis=1)
oh_te = pd.concat([Xte.drop(columns="city"),
pd.get_dummies(Xte["city"], dtype=float)], axis=1)
oh_te = oh_te.reindex(columns=oh_tr.columns, fill_value=0.0)
t0 = time.perf_counter()
xo = xgb.XGBClassifier(n_estimators=400, learning_rate=0.08, max_depth=6,
tree_method="hist", random_state=0).fit(oh_tr, ytr)
print(f" one-hot, {oh_tr.shape[1]} columns : AUC"
f" {roc_auc_score(yte, xo.predict_proba(oh_te)[:, 1]):.4f}"
f" fit {time.perf_counter()-t0:.2f}s")
print(f" native categorical handling : AUC {results[0][2]:.4f}"
f" fit {results[0][1]:.2f}s")
print(f" CatBoost target statistics : AUC {results[2][2]:.4f}"
f" fit {results[2][1]:.2f}s")
print(" One-hot forces the tree to isolate cities one binary split at a time,")
print(" and here it actually edged out XGBoost's native handling -- the native")
print(" path is faster and less memory-hungry, not automatically more accurate.")
print(" CatBoost's ordered target statistics won clearly, which is the one")
print(" place where the library choice made a real difference on this data.")
40,000 rows, 12 numeric + one 300-level categorical, 15% missing in f0
sklearn HistGB refused:
"Categorical feature 'city' is expected to have a cardinality
<= 255 but actually has a cardinality of 300"
library fit seconds test AUC
XGBoost 3.36 0.7748
LightGBM 1.66 0.7878
CatBoost 8.10 0.8035
sklearn HistGB (ordinal codes) 1.02 0.7766
LEVEL-WISE (XGBoost) VS LEAF-WISE (LightGBM)
level-wise: finish a depth before descending
o every node at depth d is
/ \ considered before depth d+1
o o -> balanced, max_depth is a
/ \ / \ real budget
o o o o
leaf-wise: always split the most promising leaf
o complexity goes where the loss
/ \ is -> deep, lopsided, more
o o expressive per leaf, overfits
/ \ sooner
o o
/ \
o o
num_leaves 15 (LightGBM) AUC 0.7802 vs max_depth 4 (XGB) AUC 0.7701
num_leaves 63 (LightGBM) AUC 0.7830 vs max_depth 6 (XGB) AUC 0.7670
num_leaves 255 (LightGBM) AUC 0.7821 vs max_depth 8 (XGB) AUC 0.7565
num_leaves = 2^max_depth is NOT an equivalence.
MISSING VALUES ARE LEARNED, NOT IMPUTED
each split stores a DEFAULT DIRECTION for missing rows:
f0 <= 0.3 ?
|-- yes -> left
|-- no -> right
|-- NaN -> whichever side reduced the loss more
test AUC with f0 entirely missing at predict time: 0.6977
test AUC with f0 present: 0.7748
The model degrades gracefully instead of erroring, and it never had
to assert (as mean imputation does) that the missing value was average.
HIGH-CARDINALITY CATEGORICAL: 300 CITIES
one-hot, 312 columns AUC 0.7818 fit 4.85s
XGBoost native categorical AUC 0.7748 fit 3.36s
CatBoost target statistics AUC 0.8035 fit 8.10s
One-hot actually edged out XGBoost's native path here -- native
handling is faster and leaner, not automatically more accurate.
CatBoost's ordered target statistics won clearly, which is the one
place the library choice mattered on this data.
| XGBoost | LightGBM | CatBoost | sklearn HistGB | |
|---|---|---|---|---|
| Tree growth | Level by level | Leaf by leaf | Symmetric / oblivious | Leaf by leaf |
| Main complexity setting | max_depth | num_leaves | depth | max_leaf_nodes |
| Categorical features | Native, optional | Native, by index | Ordered target statistics | Native, max 255 levels |
| Missing values | Learned default direction | Learned | Learned | Learned |
| Relative fit speed | Medium | Fastest | Slowest here | Fast |
| Best for | A safe, well-documented default | Large data, tight budgets | Many high-cardinality categoricals | No extra library |
Key Takeaways
- All three libraries group continuous features into bins once, which replaces sorting at every node with a quick scan over bins.
- XGBoost puts L1 and L2 penalties on leaf values into the split scoring and grows trees level by level.
- LightGBM grows trees leaf by leaf. This is more powerful per leaf but overfits sooner, so pair
num_leaveswithmin_child_samples.num_leaves = 2^max_depthdoes not make the two growth styles equivalent, because leaf-wise growth spends its leaves unevenly.- CatBoost's ordered target statistics encode categorical features with many levels without the leak that naive target encoding causes.
- Every library learns where to send missing values, which is more flexible than filling them with the mean.
- Accuracy differences between libraries are usually small. Fit time, categorical handling and team experience decide the choice.
- scikit-learn's histogram booster needs no extra library but allows at most 255 levels per native categorical feature.
🧪 Practice
- Fit XGBoost, LightGBM and CatBoost on the same dataset with the same learning rate and number of rounds. Report fit time and test AUC.
- Compare LightGBM with
num_leaves = 2^dagainst XGBoost withmax_depth = dfor d = 3, 5 and 7. Explain why they are not equivalent.- Make 20% of your strongest feature missing, fit without filling the gaps, and compare test performance with mean imputation.
- Build a categorical feature with 500 levels and compare one-hot encoding, XGBoost's native handling and CatBoost on accuracy and memory.
- Implement a naive target encoding that uses each row's own target, and show the leak by comparing training and held-out scores.
- Interview: A teammate says that switching from XGBoost to LightGBM raised validation AUC from 0.81 to 0.86 with default settings. What is your first question? (Hint: which defaults differ, and did the comparison keep the model's effective size the same?)
Feature Importance from Trees
Every tree library has a feature_importances_ attribute. It takes one line to
print, and it is one of the most misread numbers in applied machine learning.
It is not meaningless. The problem is that it answers a narrower question than
people think, and it has specific, predictable biases.
Impurity importance (also called Gini importance or mean decrease in impurity, MDI) adds up, over every split on a feature, how much that split reduced impurity, weighted by how many rows passed through it. It costs nothing, because these numbers were already computed during training. It has two serious flaws:
- It is computed on training data, so a feature that only helped by fitting noise still gets credit.
- It favours features with many distinct values. A continuous feature with three thousand distinct values offers three thousand possible thresholds. With that many tries, one of them will reduce impurity a bit even if the feature is pure noise. A binary feature gets only one try. In the example below, a useless continuous feature scores more than six times higher than a useless binary one, just because it has more possible split points.
Permutation importance fixes both flaws. Take the fitted model, shuffle one column of the held-out data, and measure how much your chosen metric gets worse. Shuffling breaks the link between that column and the target but keeps the column's values, so the drop shows how much the model relied on it. It uses data the model never saw, uses the metric you care about, and works for any model, not only trees. The costs: it needs one prediction pass per feature per repeat, and it must be repeated to get an error bar.
Drop-column importance refits the model without each feature in turn. This answers "how much does this feature add?" most directly, but it is expensive: one full refit per feature.
All three share one problem that no trick can remove: correlated features split their credit. If two near-identical columns carry one signal, each one looks about half as important as the signal really is, because the model can use either. Shuffle one and the other covers for it. Drop one and the other takes over its whole role. In the example below, one feature's permutation importance almost quadruples when its twin is removed, while held-out accuracy barely changes. This does not mean importance is broken. It means importance describes the fitted model, not the world. To measure what an underlying signal is worth, shuffle or drop the whole correlated group.
One habit worth adopting: compare against a noise baseline. Shuffle the target, refit, and look at the importance scores that pure noise produces. In the example below, the largest impurity importance with a shuffled target is 0.27, higher than several "real" features in the real model. "This feature has non-zero importance" means nothing until you know what zero looks like.
Finally, none of these measure cause. A feature can be important to a model because it is a proxy, a leak, or a consequence of the target. Importance shows what the model relies on, not how the world works.
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score
rng = np.random.default_rng(23)
n = 3000
# Four features engineered to expose the three classic importance failures:
# signal -- genuinely predictive, low cardinality
# high_card -- genuinely predictive, but continuous (many split points)
# noise_card -- USELESS, and continuous with many split points
# dup_a/dup_b -- two near-copies of one real signal
signal = rng.integers(0, 3, n).astype(float) # 3 distinct values
high_card = rng.normal(0, 1, n) # ~3000 distinct values
noise_card = rng.normal(0, 1, n) # useless, continuous
noise_bin = rng.integers(0, 2, n).astype(float) # useless, binary
shared = rng.normal(0, 1, n)
dup_a = shared + rng.normal(0, 0.05, n) # 0.999-correlated pair
dup_b = shared + rng.normal(0, 0.05, n)
y = (2.5 * signal + 1.5 * high_card + 2.0 * shared + rng.normal(0, 1.0, n))
X = pd.DataFrame({"signal": signal, "high_card": high_card,
"noise_card": noise_card, "noise_bin": noise_bin,
"dup_a": dup_a, "dup_b": dup_b})
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
rf = RandomForestRegressor(n_estimators=300, random_state=0).fit(Xtr, ytr)
print(f"test R^2 {r2_score(yte, rf.predict(Xte)):.4f}")
# --- 1. IMPURITY IMPORTANCE (Gini / MDI): fast, built in, and BIASED. ---
mdi = pd.Series(rf.feature_importances_, index=X.columns)
print("\n1. IMPURITY IMPORTANCE (model.feature_importances_)")
for f, v in mdi.sort_values(ascending=False).items():
print(f" {f:12s} {v:.4f} {'#' * int(v * 100)}")
print(" Computed on TRAINING data by summing impurity reduction. A useless")
print(" continuous feature still gets credit, because with 3000 candidate")
print(" thresholds one of them always fits some noise.")
print(f" noise_card ({mdi['noise_card']:.4f}) scores"
f" {mdi['noise_card'] / mdi['noise_bin']:.1f}x the equally useless"
f" noise_bin ({mdi['noise_bin']:.4f}) purely for having more split points.")
# --- 2. PERMUTATION IMPORTANCE: measured on HELD-OUT data, against a metric. ---
perm = permutation_importance(rf, Xte, yte, n_repeats=20, random_state=0,
scoring="r2")
pi = pd.Series(perm.importances_mean, index=X.columns)
pi_sd = pd.Series(perm.importances_std, index=X.columns)
print("\n2. PERMUTATION IMPORTANCE on the test set (drop in R^2 when shuffled)")
for f in pi.sort_values(ascending=False).index:
print(f" {f:12s} {pi[f]:+.4f} +/- {pi_sd[f]:.4f}"
f" {'#' * max(0, int(pi[f] * 100))}")
print(" The useless features now sit at roughly zero, which is the correct")
print(" answer. Permuting a column breaks its link to the target while")
print(" keeping the model fixed, so the score drop measures actual reliance.")
# --- 3. BOTH METHODS SPLIT CREDIT BETWEEN CORRELATED FEATURES. ---
print("\n3. THE CORRELATED-PAIR TRAP (dup_a and dup_b are 0.999 correlated)")
print(f" impurity: dup_a {mdi['dup_a']:.4f} dup_b {mdi['dup_b']:.4f}"
f" sum {mdi['dup_a'] + mdi['dup_b']:.4f}")
print(f" permutation: dup_a {pi['dup_a']:+.4f} dup_b {pi['dup_b']:+.4f}"
f" sum {pi['dup_a'] + pi['dup_b']:+.4f}")
# Drop one and refit: the survivor absorbs the whole effect, proving neither
# individual number was a measure of the underlying signal's importance.
X2 = X.drop(columns="dup_b")
rf2 = RandomForestRegressor(n_estimators=300, random_state=0).fit(
Xtr.drop(columns="dup_b"), ytr)
pi2 = pd.Series(permutation_importance(rf2, Xte.drop(columns="dup_b"), yte,
n_repeats=20, random_state=0,
scoring="r2").importances_mean,
index=X2.columns)
print(f" after DROPPING dup_b: dup_a permutation importance"
f" {pi2['dup_a']:+.4f} (was {pi['dup_a']:+.4f})")
print(f" test R^2 barely moves: {r2_score(yte, rf.predict(Xte)):.4f} ->"
f" {r2_score(yte, rf2.predict(Xte.drop(columns='dup_b'))):.4f}")
print(" Each twin got a FRACTION of the signal's real importance, and the")
print(" survivor's importance nearly quadruples once the other is gone, while")
print(" held-out R^2 barely moves. Permutation importance measures how much")
print(" the MODEL relies on a column, not how much the SIGNAL matters.")
# --- 4. DROP-COLUMN IMPORTANCE: the honest but expensive answer. ---
print("\n4. DROP-COLUMN IMPORTANCE (refit without the column)")
full = r2_score(yte, rf.predict(Xte))
for f in X.columns:
m = RandomForestRegressor(n_estimators=150, random_state=0).fit(
Xtr.drop(columns=f), ytr)
drop = full - r2_score(yte, m.predict(Xte.drop(columns=f)))
print(f" without {f:12s} R^2 falls by {drop:+.4f}")
print(" Costs one refit per feature, and STILL reports ~0 for each half of a")
print(" redundant pair. To value the underlying signal, drop the GROUP.")
pair = ["dup_a", "dup_b"]
both = RandomForestRegressor(n_estimators=150, random_state=0).fit(
Xtr.drop(columns=pair), ytr)
both_drop = full - r2_score(yte, both.predict(Xte.drop(columns=pair)))
print(f" without BOTH dup_a and dup_b: R^2 falls by {both_drop:+.4f}")
# --- 5. A NULL CHECK: how big is 'importance' when nothing is there? ---
# Shuffle the target, refit, and read off the importance scale produced by
# pure noise. Anything below that band is not a finding.
ysh = rng.permutation(ytr)
rfn = RandomForestRegressor(n_estimators=300, random_state=0).fit(Xtr, ysh)
null_mdi = pd.Series(rfn.feature_importances_, index=X.columns)
print("\n5. NULL CALIBRATION: impurity importances with a SHUFFLED target")
print(" " + " ".join(f"{f}={v:.3f}" for f, v in null_mdi.items()))
print(f" Largest importance from pure noise: {null_mdi.max():.4f}. Every")
print(" feature gets a non-zero score, so 'it has non-zero importance' is")
print(" not evidence of anything. Compare against this band.")
THE SETUP (test R^2 0.8963)
signal genuinely predictive, only 3 distinct values
high_card genuinely predictive, ~3000 distinct values
noise_card USELESS, ~3000 distinct values
noise_bin USELESS, 2 distinct values
dup_a/dup_b two 0.999-correlated copies of one real signal
1. IMPURITY IMPORTANCE: FREE, AND BIASED TOWARD MANY SPLIT POINTS
signal 0.3443 ##################################
high_card 0.2240 ######################
dup_b 0.2101 #####################
dup_a 0.1963 ###################
noise_card 0.0219 ## <- USELESS, but continuous
noise_bin 0.0035 <- USELESS, and binary
noise_card scores 6.3x noise_bin. Both contribute nothing. The only
difference is 3000 candidate thresholds versus one.
2. PERMUTATION IMPORTANCE: HELD-OUT, METRIC-BASED, MODEL-AGNOSTIC
signal +0.7030 +/- 0.0196
high_card +0.3389 +/- 0.0160
dup_a +0.1772 +/- 0.0060
dup_b +0.1770 +/- 0.0071
noise_bin -0.0004 +/- 0.0004 <- correctly ~zero
noise_card -0.0029 +/- 0.0010 <- correctly ~zero
original column: [3.1, 0.4, 2.2, 1.8, 0.9] R^2 = 0.896
shuffled column: [0.9, 2.2, 3.1, 0.4, 1.8] R^2 = 0.193
^ same values, broken link -> drop of 0.703
3. THE CORRELATED-PAIR TRAP
impurity: dup_a 0.1963 dup_b 0.2101
permutation: dup_a +0.1772 dup_b +0.1770
after DROPPING dup_b:
dup_a permutation importance +0.6680 (was +0.1772, nearly 4x)
test R^2 0.8941 (was 0.8963, unchanged)
Neither number measured the signal. Each measured how much THIS
model leaned on THAT column, given a substitute was available.
4. DROP-COLUMN IMPORTANCE: HONEST, AND EXPENSIVE
without signal R^2 falls by +0.3878
without high_card R^2 falls by +0.2151
without noise_card R^2 falls by -0.0026
without noise_bin R^2 falls by +0.0003
without dup_a R^2 falls by +0.0023 <- looks worthless
without dup_b R^2 falls by +0.0024 <- looks worthless
without BOTH R^2 falls by +0.4118 <- the real value
To value a signal carried by several columns, drop the GROUP.
5. NULL CALIBRATION: IMPORTANCES FROM A SHUFFLED TARGET
signal 0.048 high_card 0.273 noise_card 0.261
noise_bin 0.027 dup_a 0.194 dup_b 0.197
Largest importance from pure noise: 0.2725 -- higher than several
features scored in the real model. Non-zero importance is not
evidence of anything until you know what zero looks like.
| Impurity (MDI) | Permutation | Drop-column | |
|---|---|---|---|
| Data used | Training | Held-out | Held-out, after refit |
| Cost | Free | One pass per feature per repeat | One refit per feature |
| Favours many distinct values | Yes, strongly | No | No |
| Rewards overfitting | Yes | No | No |
| Works for any model | No, trees only | Yes | Yes |
| Handles correlated features | No | No | No |
| Measures cause | No | No | No |
Key Takeaways
- Impurity importance is free, but it uses training data and favours features with many distinct values, so a useless continuous column beats a useless binary one.
- Permutation importance measures how much your real metric drops on held-out data and works for any model. It costs one prediction pass per feature per repeat.
- Drop-column importance is the most direct measure and needs one refit per feature.
- All three split credit between correlated features, so a column carrying an important signal can score close to zero.
- Importance describes what the fitted model relies on, not how the world works. To value a signal spread across columns, shuffle or drop the whole group.
- Compare against a shuffled-target baseline before treating any non-zero importance as a finding.
- No importance measure shows cause. Proxies, leaks and consequences of the target can all score highly.
🧪 Practice
- Add a pure-noise continuous column and a pure-noise binary column to a dataset, fit a forest, and compare their impurity importances.
- Compute permutation importance on the training set and on the test set. Explain any features whose rank changes.
- Copy your strongest feature with a little noise added, then report permutation importance for both copies before and after dropping one.
- Implement drop-column importance and compare its ranking with permutation importance. Note where they disagree.
- Shuffle the target, refit, and report the largest importance any feature gets. Use it as a threshold for your real model.
- Interview: A stakeholder wants to remove the two least important features to simplify a pipeline, and those two features are highly correlated with each other. What do you check first? (Hint: what happens to each one's score when the other is present?)
Gaussian Processes
Every model so far returns a single number. A Gaussian process (GP) returns a number and how sure it is about it. That second output is the reason to care about a method that cannot handle more than a few thousand rows.
The key idea is a change in what the uncertainty is about. A parametric model puts a probability distribution on its parameters; for example, ridge regression can be seen as a normal prior on the coefficients. A Gaussian process puts a distribution directly on functions. Before seeing any data, you have a prior over every possible curve. After seeing data, you have a posterior: curves that pass close to the observations and spread out where there are none. To predict, you read off the mean and variance of that posterior at the new input.
The distribution is defined by a kernel. A kernel is a similarity function:
K(x, x') says how strongly the function's values at x and x' are linked.
If K(x, x') is large, the function must take similar values at the two
inputs. If it is near zero, the two values are free to differ. So the kernel
is the prior over functions, and its settings describe what likely functions
look like. (Kernels are also central to SVMs, covered in 10.2.)
- RBF with a short length scale gives wiggly functions; with a long length scale, smooth and slowly changing ones.
- Matern with
nu = 0.5gives rough, jagged paths; withnu = 2.5, smoother ones. It is the usual choice when RBF's perfect smoothness is too strong an assumption. - ExpSineSquared gives periodic functions, for a known cycle.
- DotProduct gives linear trends.
- WhiteKernel adds independent observation noise.
Kernels can be added and multiplied and still be valid kernels, so you can
combine ideas about the data's structure. "A yearly pattern on top of a
linear trend, measured with noise" becomes ExpSineSquared + DotProduct + WhiteKernel. You state the idea before fitting, and the log marginal
likelihood then scores how well it fits. That is a clearer workflow than
tuning a black box.
The uncertainty is the real benefit, and it behaves as it should: narrow where there is a lot of data, wider in the gaps, and wide outside the observed range, where the model falls back to the prior. A random forest, by contrast, returns a confident flat value outside its training range with no hint that it is guessing. This is why GPs are the standard engine for Bayesian optimisation, including hyperparameter search: the model's own uncertainty tells you where the next expensive test is most worth running.
Two honest warnings. First, the intervals are a model output, not a guarantee.
The kernel settings, including the noise level, are fitted by maximising the
marginal likelihood. On small samples the noise estimate can come out too low,
which makes the intervals too narrow. In practice, set a lower limit on the
WhiteKernel noise level, and check the coverage of the intervals instead of
assuming it.
Second, the cost. Fitting means solving a system with the n by n kernel
matrix, which takes time proportional to n³ and memory proportional to n².
Doubling the data makes it about eight times slower. Beyond a few thousand
rows you need sparse or inducing-point approximations. That is why GPs are used
in small-data settings: physical experiments, engineering simulations, Bayesian
optimisation, and anywhere each observation is expensive to get.
import numpy as np
import pandas as pd
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import (RBF, Matern, WhiteKernel,
ConstantKernel, ExpSineSquared,
RationalQuadratic, DotProduct)
rng = np.random.default_rng(47)
# --- THE IDEA: put a distribution over FUNCTIONS, not over parameters. ---
# A GP is defined by a mean (usually 0) and a kernel that says how correlated
# the function values at two inputs are. Sampling from it gives whole curves.
grid = np.linspace(0, 10, 200).reshape(-1, 1)
print("SAMPLING FUNCTIONS FROM THE PRIOR")
for label, k in [("RBF, length_scale=0.3", RBF(0.3)),
("RBF, length_scale=2.0", RBF(2.0)),
("Matern nu=0.5 (rough)", Matern(1.0, nu=0.5)),
("Matern nu=2.5 (smoother)", Matern(1.0, nu=2.5))]:
K = k(grid) + 1e-10 * np.eye(len(grid)) # jitter for numerical safety
draws = np.linalg.cholesky(K) @ rng.normal(0, 1, (len(grid), 200))
# A useful summary of "wiggliness": average absolute first difference.
rough = float(np.abs(np.diff(draws, axis=0)).mean())
print(f" {label:26s} mean |f(x+h) - f(x)| = {rough:.4f}")
print(" The kernel IS the prior: it decides how wiggly a plausible function")
print(" is before you have seen any data. Short length scale, wiggly.")
# --- CONDITIONING ON DATA: the posterior mean and its UNCERTAINTY. ---
def truth(x):
return np.sin(x) + 0.3 * x
NOISE = 0.25
# 30 observations with a deliberate GAP between 4.5 and 6.5, so you can watch
# what the model says about a region it has no information about.
Xobs = np.sort(np.concatenate([rng.uniform(0, 4.5, 20),
rng.uniform(6.5, 9.0, 10)])).reshape(-1, 1)
yobs = truth(Xobs[:, 0]) + rng.normal(0, NOISE, len(Xobs))
# WhiteKernel models observation noise, and its bounds matter: left unbounded on
# few points the optimiser will happily conclude the data is noise-free.
kernel = ConstantKernel(1.0) * RBF(1.0) + WhiteKernel(0.05, (1e-3, 1.0))
gp = GaussianProcessRegressor(kernel=kernel, normalize_y=True,
random_state=0).fit(Xobs, yobs)
wide = np.linspace(0, 11, 200).reshape(-1, 1) # extends past the data
mu, sd = gp.predict(wide, return_std=True)
print(f"\nfitted kernel: {gp.kernel_}")
print(f"true noise variance {NOISE**2:.4f}")
print(f"\n{'x':>5s} {'truth':>7s} {'mean':>7s} {'sd':>6s} region")
for xq, region in [(1.0, "dense data"), (3.0, "dense data"),
(5.5, "THE GAP"), (7.5, "sparse data"),
(10.5, "BEYOND the data")]:
i = int(np.argmin(np.abs(wide[:, 0] - xq)))
print(f"{xq:5.1f} {truth(xq):7.3f} {mu[i]:7.3f} {sd[i]:6.3f} {region}")
print("Uncertainty is smallest where data is dense, larger in the gap, and")
print("largest outside the observed range. No other model in this chapter")
print("reports that distinction for free.")
nearest = np.min(np.abs(wide[:, 0][:, None] - Xobs[:, 0][None, :]), axis=1)
print(f"\ncorrelation(distance to nearest observation, predicted sd) ="
f" {np.corrcoef(nearest, sd)[0, 1]:.4f}")
# --- CALIBRATION: are the intervals honest? Check, do not assume. ---
Xt = rng.uniform(0, 9, 600).reshape(-1, 1)
yt = truth(Xt[:, 0]) + rng.normal(0, NOISE, 600)
mt, st = gp.predict(Xt, return_std=True)
z = (yt - mt) / st
print("\nINTERVAL COVERAGE on 600 held-out points inside the observed range")
for level, zc in [(0.50, 0.674), (0.80, 1.282), (0.95, 1.960)]:
print(f" nominal {level:.0%} actual {np.mean(np.abs(z) < zc):.1%}")
print(f" Close to nominal, and consistent with a fitted noise level of")
print(f" {gp.kernel_.k2.noise_level:.4f} against a true {NOISE**2:.4f}. But note")
print(" WHY it worked: the WhiteKernel bounds kept the optimiser from")
print(" concluding the data was noise-free, which it will do on a handful of")
print(" points. Coverage is a modelling output to VALIDATE, not a guarantee.")
# --- KERNEL CHOICE ENCODES STRUCTURE YOU ALREADY KNOW. ---
t = np.linspace(0, 6, 220).reshape(-1, 1)
seasonal = (np.sin(2 * np.pi * t[:, 0]) + 0.35 * t[:, 0]
+ rng.normal(0, 0.12, len(t)))
tr_i = np.sort(rng.choice(len(t), 70, replace=False))
te_i = np.setdiff1d(np.arange(len(t)), tr_i)
print("\nKERNEL CHOICE ON PERIODIC-PLUS-TREND DATA")
print(f" {'kernel':44s} {'test RMSE':>10s} {'log ML':>9s}")
for label, k in [
("RBF only", ConstantKernel() * RBF(1.0) + WhiteKernel(0.05)),
("Matern nu=1.5", ConstantKernel() * Matern(1.0, nu=1.5)
+ WhiteKernel(0.05)),
("linear (DotProduct)", ConstantKernel() * DotProduct() + WhiteKernel(0.05)),
("periodic + linear", ConstantKernel() * ExpSineSquared(1.0, 1.0)
+ ConstantKernel() * DotProduct() + WhiteKernel(0.05)),
("rational quadratic", ConstantKernel() * RationalQuadratic()
+ WhiteKernel(0.05)),
]:
g = GaussianProcessRegressor(kernel=k, normalize_y=True,
random_state=0).fit(t[tr_i], seasonal[tr_i])
pred = g.predict(t[te_i])
r = float(np.sqrt(np.mean((seasonal[te_i] - pred) ** 2)))
print(f" {label:42s} {r:10.4f} {g.log_marginal_likelihood_value_:9.2f}")
print(" Kernels ADD and MULTIPLY: a periodic kernel plus a linear one says")
print(" 'a repeating pattern on top of a trend', which is a hypothesis you can")
print(" state before fitting. The log marginal likelihood then scores it.")
# --- THE COST: O(n^3) to fit, because a GP inverts an n-by-n matrix. ---
import time
print("\nWHY GPs DO NOT SCALE (fitting inverts an n x n kernel matrix)")
for n in [200, 800, 3200]:
Xn = rng.uniform(0, 10, (n, 1))
yn = truth(Xn[:, 0]) + rng.normal(0, 0.1, n)
t0 = time.perf_counter()
GaussianProcessRegressor(kernel=ConstantKernel() * RBF(1.0)
+ WhiteKernel(0.01),
optimizer=None).fit(Xn, yn)
print(f" n={n:5d}: fit {time.perf_counter()-t0:7.3f}s"
f" kernel matrix {n*n:,} entries"
f" {n*n*8/1e6:.1f} MB")
print(" Doubling n multiplies the work by about 8. Past a few thousand rows")
print(" you need sparse or inducing-point approximations, which is why GPs")
print(" live in small-data settings: physical experiments, hyperparameter")
print(" search, and anywhere each observation is expensive to obtain.")
THE KERNEL IS THE PRIOR OVER FUNCTIONS
(mean |f(x+h) - f(x)| across 200 sampled curves)
RBF, length_scale=0.3 0.1325 wiggly
RBF, length_scale=2.0 0.0203 smooth
Matern nu=0.5 (rough) 0.2503 roughest
Matern nu=2.5 (smoother) 0.0523
length_scale=0.3 length_scale=2.0
/\ /\ /\ ____
/ \/ \ / \ / / \____
/ \/ \/ / \___
CONDITIONING ON DATA: THE MEAN AND THE UNCERTAINTY
fitted kernel: 1.67^2 * RBF(length_scale=2.13) + WhiteKernel(0.0629)
(true noise variance 0.0625, and observations have a GAP in 4.5 to 6.5)
x truth mean sd region
1.0 1.141 1.325 0.255 dense data
3.0 1.041 1.167 0.260 dense data
5.5 0.944 1.004 0.326 THE GAP
7.5 3.188 2.932 0.267 sparse data
10.5 2.270 3.068 0.890 BEYOND the data
correlation(distance to nearest observation, predicted sd) = 0.9315
y | . . ,-~-, .
| . . ,-' `-, .
| o o o o ( wide ) o o <- bands widen in the gap
| . . . `-, ,-' . `~-,_ and beyond the data
+--------------------------------------> x
dense data GAP sparse none
ARE THE INTERVALS HONEST? CHECK, DO NOT ASSUME.
nominal 50% actual 47.8%
nominal 80% actual 76.3%
nominal 95% actual 93.2%
Close to nominal here -- but only because the WhiteKernel bounds
stopped the optimiser concluding the data was noise-free, which it
will happily do on a handful of points.
KERNEL COMPOSITION EXPRESSES A HYPOTHESIS
(periodic-plus-trend data)
kernel test RMSE log ML
RBF only 0.1298 -10.38
Matern nu=1.5 0.1646 -19.40
linear (DotProduct) 0.7122 -87.47
periodic + linear 0.1156 31.04 <- best on BOTH
rational quadratic 0.1298 -10.38
The log marginal likelihood scores the structural hypothesis, so it
can be compared across kernels without a validation split.
WHY GPs DO NOT SCALE
n= 200: fit 0.034s kernel matrix 40,000 entries 0.3 MB
n= 800: fit 0.064s kernel matrix 640,000 entries 5.1 MB
n= 3200: fit 1.166s kernel matrix 10,240,000 entries 81.9 MB
Doubling n multiplies the work by about 8.
| Gaussian process | Random forest | RBF SVM | |
|---|---|---|---|
| Built-in uncertainty | Yes, for every prediction | No | No |
| Outside the training range | Falls back to prior, wide bands | Flat, confident | Flat, confident |
| Encodes prior knowledge | Yes, through the kernel | No | Only smoothness |
Fit cost in n | Cubic | Near-linear | Quadratic to cubic |
| Practical row limit | A few thousand | Millions | ~100,000 |
| How to compare models | Log marginal likelihood | Held-out score | Held-out score |
Key Takeaways
- A Gaussian process puts a distribution over functions instead of parameters. Predicting means reading the posterior's mean and variance.
- The kernel is the prior: its form and length scale say what likely functions look like before you see any data.
- Kernels can be added and multiplied, so ideas such as "periodic plus trend" can be written down and then scored with the log marginal likelihood.
- The predicted uncertainty is narrow where data is dense, wider in gaps and wide outside the observed range. This makes GPs a good fit for Bayesian optimisation.
- The intervals are estimated, not guaranteed. Keep the noise level above zero and check coverage on real data.
- Fitting time grows with the cube of the number of rows and memory with the square, so GPs need approximations beyond small datasets.
- GPs fit best where observations are expensive: experiments, simulations and hyperparameter search.
🧪 Practice
- Draw sample curves from RBF priors with length scales 0.1, 1 and 10, and measure how wiggly each is with the average absolute difference between neighbouring points.
- Fit a GP to ten points with a deliberate gap. Report the predicted standard deviation inside the data, inside the gap and beyond the range.
- Over a dense grid, correlate the distance to the nearest observation with the predicted standard deviation.
- Check interval coverage at the 50%, 80% and 95% levels on held-out data. Repeat with the
WhiteKernelbounds removed.- Fit periodic-plus-trend data with four different kernels. Rank them by held-out RMSE and by log marginal likelihood. Do the rankings agree?
- Time GP fitting at 250, 500, 1000 and 2000 rows and estimate how fast the time grows.
- Interview: You must choose the next five experiments, each costing a week of lab time, and you have twelve results so far. Why might a Gaussian process beat a gradient-boosted tree here? (Hint: what does the model need to tell you besides its best guess, and how many rows do you have?)
<a id="102-classification"></a>
10.2 Classification
In classification, the target is one label from a fixed set, and labels cannot be added, subtracted or averaged. Losing that arithmetic changes how everything works. The model really outputs a score, not a label. The label only appears when you apply a threshold to the score, and the right threshold is a business decision, not a model setting. This section covers:
- what makes a problem classification, and how scores, thresholds and costs fit together;
- what kind of boundary each model family can draw;
- logistic regression and its multi-class and ordered versions;
- generative classifiers: Naive Bayes and discriminant analysis;
- nearest neighbours and how to choose a distance metric;
- support vector machines and the kernel trick;
- ensembles that combine different models: voting, stacking, and why diversity matters;
- how regression and classification share almost all their machinery.
What Makes a Problem Classification
A classification target is a set of names. card, wallet and transfer
might be stored as 0, 1 and 2 because some system needed an integer. But
wallet is not halfway between card and transfer, and no row can be predicted
as 1.07. If you number the classes in a different order, any model that treats
the codes as numbers gives different answers. That alone proves the numbers
never meant anything.
Classification comes in four types, and people often mix them up:
| Type | Target | Example |
|---|---|---|
| binary | one of two labels | fraud / not fraud |
| multiclass | exactly one of k labels | which of six failure modes |
| multilabel | any number of k labels | tags on a support ticket |
| ordinal | one of k ordered labels | risk rated low / medium / high |
Multilabel is not the same as multiclass: "exactly one" and "any number" need different model outputs and different metrics. Ordinal is the tricky one. Its labels have an order, but the gaps between them have no fixed size. A plain classifier ignores the order, and a regression on the codes invents gap sizes that do not exist.
The most important idea in this topic is: the model produces a score; a
threshold turns the score into a decision; the costs of mistakes set the
threshold. Calling .predict() quietly uses a threshold of 0.5, and 0.5 is
almost never right. If a false positive costs C_fp and a false negative
costs C_fn, the threshold that minimises expected cost on a calibrated
probability is
t* = C_fp / (C_fp + C_fn)
This equals 0.5 only when both mistakes cost the same, which is rare. Chapter 8.4 covers decision analysis in detail. The point here is that the threshold is not a hyperparameter to tune with cross-validation on accuracy. It is a business input, and you can change it after the model ships without retraining.
Two things follow from this.
Accuracy must be compared with a do-nothing model. If 7% of transactions are fraud, a model that always says "not fraud" scores 93% accuracy. An accuracy number means nothing without the base rate next to it. With strong imbalance, a genuinely useful model can even score lower than the do-nothing one. Chapter 9.3 covers metrics that work with imbalanced classes.
Ranking well and being correct are different things. Ranking metrics such as AUC, precision@k and lift only look at the order of the scores. So a model can rank almost perfectly while its probabilities are all too high. That is fine if you only use the top 500 cases. It is not fine if you multiply the probability by a cost. Log loss and the Brier score do notice wrong probabilities, and calibration (chapter 9.3) fixes them.
One more practical point: you almost never train on the metric you care about. Precision, recall, F1 and accuracy all jump in steps as the score crosses the threshold, so their gradient is zero almost everywhere and useless for training. Classifiers instead minimise a smooth stand-in loss, such as cross-entropy or hinge loss. You then get the metric you care about by choosing the threshold afterwards. Class weights and resampling change which mistakes the stand-in loss cares about, but this two-step process stays the same.
import numpy as np
from sklearn.calibration import CalibratedClassifierCV
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.metrics import (accuracy_score, brier_score_loss, log_loss,
roc_auc_score)
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
rng = np.random.default_rng(11)
# --- 1. THE LABEL HAS NO ARITHMETIC. Three payment methods, encoded 0/1/2
# because some pipeline needed a number.
methods = np.array(["card", "wallet", "transfer"])
codes = rng.integers(0, 3, 600)
feat = rng.normal(0, 1, 600)
lin = LinearRegression().fit(feat.reshape(-1, 1), codes)
pred_code = lin.predict([[0.4]])[0]
print("1. A CATEGORY IS NOT A NUMBER")
print(f" regressing the code 0/1/2 predicts {pred_code:.2f} for a new row.")
print(f" There is no payment method at {pred_code:.2f}, and the model has")
print(" quietly asserted that wallet sits exactly between card and")
print(" transfer, and that transfer is twice wallet. Relabel the classes")
print(" in a different order and every prediction changes.")
# --- 2. IMBALANCE: accuracy is scored against a trivial competitor. ---
n = 20000
amount = rng.lognormal(3.2, 1.0, n)
age_days = rng.uniform(0, 900, n)
foreign = rng.random(n) < 0.18
logit = (-3.9 + 0.55 * np.log(amount) - 0.0022 * age_days + 1.1 * foreign)
p_true = 1 / (1 + np.exp(-logit))
fraud = (rng.random(n) < p_true).astype(int)
X = np.column_stack([np.log(amount), age_days, foreign.astype(float)])
Xtr, Xte, ytr, yte = train_test_split(X, fraud, test_size=0.4, random_state=0,
stratify=fraud)
clf = LogisticRegression(max_iter=1000).fit(Xtr, ytr)
proba = clf.predict_proba(Xte)[:, 1]
base = yte.mean()
print(f"\n2. BASE RATE {base:.2%} FRAUD")
print(f" 'never fraud' classifier accuracy : {1 - base:.3%}")
print(f" logistic regression accuracy : "
f"{accuracy_score(yte, proba >= 0.5):.3%}")
print(f" logistic regression ROC AUC : {roc_auc_score(yte, proba):.3f}")
print(" Accuracy makes the two look comparable. AUC shows they are not.")
# --- 3. THE MODEL PRODUCES A SCORE; THE THRESHOLD PRODUCES THE DECISION. ---
cost_fn, cost_fp = 240.0, 9.0 # a missed fraud vs a review of a good order
print(f"\n3. ONE SCORE, MANY DECISIONS (miss costs {cost_fn:.0f}, "
f"review costs {cost_fp:.0f})")
print(f" {'threshold':>9s} {'flagged':>8s} {'caught':>7s} {'missed':>7s}"
f" {'false alarms':>13s} {'cost':>10s}")
best = None
for t in [0.02, 0.04, 0.05, 0.10, 0.20, 0.35, 0.50]:
pred = proba >= t
tp = int(((pred == 1) & (yte == 1)).sum())
fp = int(((pred == 1) & (yte == 0)).sum())
fn = int(((pred == 0) & (yte == 1)).sum())
cost = cost_fn * fn + cost_fp * fp
if best is None or cost < best[1]:
best = (t, cost)
print(f" {t:9.2f} {int(pred.sum()):8d} {tp:7d} {fn:7d} {fp:13d}"
f" {cost:10,.0f}")
print(f" cheapest threshold on this grid: {best[0]:.2f}, not 0.50.")
print(f" the theoretical optimum is cost_fp/(cost_fp+cost_fn) = "
f"{cost_fp/(cost_fp+cost_fn):.3f}")
# --- 4. RANKING WELL AND BEING RIGHT ARE DIFFERENT PROPERTIES.
# The amount column arrives three times under three names, which is
# an ordinary data-plumbing accident.
dup = [0, 0, 0, 1, 2]
Xtr_d, Xte_d = Xtr[:, dup], Xte[:, dup]
p_lr = LogisticRegression(max_iter=1000).fit(Xtr_d, ytr).predict_proba(Xte_d)[:, 1]
p_nb = GaussianNB().fit(Xtr_d, ytr).predict_proba(Xte_d)[:, 1]
cal = CalibratedClassifierCV(GaussianNB(), method="isotonic", cv=5)
p_cal = cal.fit(Xtr_d, ytr).predict_proba(Xte_d)[:, 1]
print("\n4. A GOOD RANKER CAN STILL BE A BAD PROBABILITY")
print(f" {'model':24s} {'AUC':>6s} {'log loss':>9s} {'Brier':>8s}"
f" {'mean p':>7s} {'actual':>7s}")
for name, p in [("logistic regression", p_lr), ("naive Bayes", p_nb),
("naive Bayes, calibrated", p_cal)]:
print(f" {name:24s} {roc_auc_score(yte, p):6.3f} {log_loss(yte, p):9.4f}"
f" {brier_score_loss(yte, p):8.5f} {p.mean():7.3f} {base:7.3f}")
print(" Naive Bayes counts the duplicated evidence three times, so it")
print(" ORDERS the cases about as well and still reports probabilities")
print(" that are not probabilities. AUC cannot see the difference; a cost")
print(" calculation fed by those numbers can.")
1. A CATEGORY IS NOT A NUMBER
regressing the code 0/1/2 predicts 0.98 for a new row.
There is no payment method at 0.98, and the model has
quietly asserted that wallet sits exactly between card and
transfer, and that transfer is twice wallet. Relabel the classes
in a different order and every prediction changes.
2. BASE RATE 6.95% FRAUD
'never fraud' classifier accuracy : 93.050%
logistic regression accuracy : 93.037%
logistic regression ROC AUC : 0.747
Accuracy makes the two look comparable. AUC shows they are not.
3. ONE SCORE, MANY DECISIONS (miss costs 240, review costs 9)
threshold flagged caught missed false alarms cost
0.02 6927 545 11 6382 60,078
0.04 4795 488 68 4307 55,083
0.05 3988 447 109 3541 58,029
0.10 1658 288 268 1370 76,650
0.20 346 104 452 242 110,658
0.35 36 18 538 18 129,282
0.50 1 0 556 1 133,449
cheapest threshold on this grid: 0.04, not 0.50.
the theoretical optimum is cost_fp/(cost_fp+cost_fn) = 0.036
4. A GOOD RANKER CAN STILL BE A BAD PROBABILITY
model AUC log loss Brier mean p actual
logistic regression 0.747 0.2249 0.06021 0.069 0.070
naive Bayes 0.732 0.2524 0.06887 0.103 0.070
naive Bayes, calibrated 0.732 0.2293 0.06105 0.069 0.070
Naive Bayes counts the duplicated evidence three times, so it
ORDERS the cases about as well and still reports probabilities
that are not probabilities. AUC cannot see the difference; a cost
calculation fed by those numbers can.
Key Takeaways
- Classification targets are names, not numbers. An integer code is just a storage format, never a quantity to regress on.
- Binary, multiclass, multilabel and ordinal are four different tasks. Ordinal is the one that both classification and regression handle badly.
- The model gives a score, a threshold turns it into a decision, and the cost of each type of mistake sets the threshold.
- For calibrated probabilities, the cost-minimising threshold is
C_fp / (C_fp + C_fn). The 0.5 inside.predict()assumes both mistakes cost the same.- The threshold is a business input that can change after deployment without retraining, so do not fix it inside the model.
- With imbalanced classes, accuracy is compared against a model that does nothing, so always quote it with the base rate.
- Ranking metrics only see the order of the scores, so a model can have a strong AUC and wrong probabilities.
- Thresholded metrics have no useful gradient, so training uses a smooth stand-in loss and the target metric is reached by choosing the threshold.
🧪 Practice
- Encode a three-class target as 0/1/2, fit a linear regression, and show that reordering the class labels changes every prediction.
- On an imbalanced dataset, report the accuracy of a constant classifier and of your model. How big must the gap be before you would present the model?
- Try many thresholds, compute the total expected cost at each, and compare the best one with
C_fp / (C_fp + C_fn).- Copy one predictive column three times, refit Naive Bayes, and compare AUC, log loss and the average predicted probability before and after.
- Calibrate that model with isotonic regression and show which of the three numbers change and which do not.
- Build a multilabel problem, fit it as multiclass, and describe exactly what information is lost.
- Interview: A team says their classifier has an AUC of 0.91 and wants to use its output probability to decide how much money to reserve for each claim. What do you check before agreeing? (Hint: which property of the scores does AUC ignore?)
Classification Algorithms and Their Decision Boundaries
A classifier divides the feature space into regions, one per label. The model family decides what shape those regions can have, and that shape is the model's assumption. So "which classifier is best?" really means "what shape is the boundary?". You can often see the answer by plotting two features.
Model families also differ in a second way.
- Discriminative models learn the boundary, or
P(y | x), directly: logistic regression, SVMs, trees, boosting, neural networks. - Generative models learn how the data in each class is distributed,
P(x | y), then use Bayes' rule to getP(y | x): Naive Bayes, LDA and QDA, and Gaussian mixtures.
In practice, generative models can work with very little data, because their assumptions do much of the work. When those assumptions are wrong, they fail in predictable ways: Naive Bayes counts correlated evidence more than once, and LDA draws a straight line whatever the data looks like.
| Family | Boundary it can draw | Gives probabilities | Multiclass | Needs scaling |
|---|---|---|---|---|
| logistic regression | one flat boundary (hyperplane) | yes | softmax or OvR | for penalties |
| LDA | one flat boundary (hyperplane) | yes | built in | no |
| QDA | one curved (quadratic) surface | yes | built in | no |
| naive Bayes | shaped by the assumed density | yes, often poor | built in | no |
| decision tree | axis-aligned boxes | leaf frequencies | built in | no |
| random forest | many boxes, averaged | averaged leaves | built in | no |
| gradient boosting | many boxes, added up | yes, via summed logits | built in or OvR | no |
| k-NN | local, follows the data | neighbour votes | built in | yes |
| SVM with RBF kernel | smooth, kernel-shaped | no, needs Platt | OvO | yes |
Two rows in that table cause real problems in production.
- An SVM does not give probabilities. Its output is a signed distance
from the boundary. Setting
probability=Truefits a separate calibration model on top. That costs an extra round of cross-validation, and near the boundary its probabilities can disagree with the SVM's own predicted labels. - The multiclass strategy matters. One-vs-rest (OvR) fits k separate models whose scores were never trained to be compared with each other. One-vs-one (OvO) fits k(k-1)/2 of them. Only softmax and models that are multiclass by design give class probabilities that always sum to one.
You should still fit a linear model first, but not because it will win. It tells you whether the boundary is roughly straight, and when it fails, that failure is useful information. If logistic regression and LDA both barely beat the base rate while k-NN on the same features scores in the nineties, that gap is the finding: the boundary is curved.
import numpy as np
from sklearn.discriminant_analysis import (LinearDiscriminantAnalysis,
QuadraticDiscriminantAnalysis)
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier
rng = np.random.default_rng(5)
# Two features, one curved boundary: applicants are approved when a smooth
# combination of income and debt clears a bar, which no straight line matches.
n = 1500
x1 = rng.uniform(-3, 3, n)
x2 = rng.uniform(-3, 3, n)
margin = 1.6 - (x1 ** 2 + x2 ** 2) / 3.0 + 0.5 * x1
y = (margin + rng.normal(0, 0.35, n) > 0).astype(int)
X = np.column_stack([x1, x2])
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.4, random_state=0)
models = {
"logistic regression": LogisticRegression(),
"LDA": LinearDiscriminantAnalysis(),
"QDA": QuadraticDiscriminantAnalysis(),
"decision tree": DecisionTreeClassifier(min_samples_leaf=20, random_state=0),
"k-NN (k=15)": make_pipeline(StandardScaler(), KNeighborsClassifier(15)),
"SVM (RBF)": make_pipeline(StandardScaler(), SVC(C=5)),
"random forest": RandomForestClassifier(n_estimators=300, random_state=0),
}
W, H = 33, 15
gx = np.linspace(-3, 3, W)
gy = np.linspace(3, -3, H)
grid = np.array([[a, b] for b in gy for a in gx])
true_region = ((1.6 - (grid[:, 0] ** 2 + grid[:, 1] ** 2) / 3.0
+ 0.5 * grid[:, 0]) > 0).astype(int)
def render(labels):
rows = []
for r in range(H):
rows.append("".join("#" if v else "." for v in labels[r * W:(r + 1) * W]))
return rows
panels = {"TRUE BOUNDARY": render(true_region)}
scores = {}
for name, m in models.items():
m.fit(Xtr, ytr)
scores[name] = accuracy_score(yte, m.predict(Xte))
panels[name] = render(m.predict(grid))
print("THE SHAPE OF THE BOUNDARY IS THE ALGORITHM'S ASSUMPTION")
print("('#' = predicted approve, '.' = predicted decline, over the x1/x2 plane)")
names = list(panels)
for i in range(0, len(names), 2):
pair = names[i:i + 2]
print()
print(" " + " ".join(f"{p:{W}s}" for p in pair))
for r in range(H):
print(" " + " ".join(f"{panels[p][r]:{W}s}" for p in pair))
print("\nWHAT EACH FAMILY IS ACTUALLY DRAWING")
shape = {
"logistic regression": ("one hyperplane", "yes, native", "softmax or OvR"),
"LDA": ("one hyperplane", "yes, native", "native k-class"),
"QDA": ("one quadric", "yes, native", "native k-class"),
"decision tree": ("axis-parallel boxes", "leaf frequencies", "native k-class"),
"k-NN (k=15)": ("local, data-shaped", "neighbour votes", "native k-class"),
"SVM (RBF)": ("smooth, kernel-shaped", "no, needs Platt", "OvO"),
"random forest": ("many boxes, averaged", "averaged leaves", "native k-class"),
}
print(f" {'family':20s} {'boundary':22s} {'probabilities':17s} "
f"{'multiclass':15s} {'acc':>6s}")
for name in models:
b, p, mc = shape[name]
print(f" {name:20s} {b:22s} {p:17s} {mc:15s} {scores[name]:6.3f}")
THE SHAPE OF THE BOUNDARY IS THE ALGORITHM'S ASSUMPTION
('#' = predicted approve, '.' = predicted decline, over the x1/x2 plane)
TRUE BOUNDARY logistic regression
................................. ...................##############
................................. ..................###############
................#########........ ..................###############
............#################.... ..................###############
..........#####################.. ..................###############
.........#######################. ..................###############
........######################### ..................###############
........######################### ..................###############
........######################### ..................###############
.........#######################. ..................###############
..........#####################.. ..................###############
............#################.... ..................###############
................#########........ ..................###############
................................. ..................###############
................................. ..................###############
LDA QDA
..................############### .................................
..................############### .................................
..................############### ....................#########....
..................############### ................#################
..................############### ..............###################
..................############### .............####################
..................############### ............#####################
..................############### ............#####################
..................############### ............#####################
..................############### .............####################
..................############### ..............###################
..................############### ................#################
..................############### ....................###########..
..................############### .................................
..................############### .................................
decision tree k-NN (k=15)
................................. .................................
................................. .................................
................................. ..............############.......
...........###################.## .............################....
...........###################.## ...........###################...
.......#######################.## .........########################
.......#######################.## ........#########################
.......#######################.## ........#########################
.......#######################.## ........#########################
...........###################.## .........########################
...........###################.## ...........###################...
...........###################.## ..............###############....
................................. .................########........
................................. .................................
................................. .................................
SVM (RBF) random forest
................................. .................................
................................. .................................
...............##########........ ................#########........
............#################.... ............##################...
...........####################.. .............#################...
.........######################## .........########################
.........######################## ........#########################
........######################### .......##########################
.........######################## ........#########################
..........####################### .........########################
...........####################.. ...........###################...
.............################.... .............################....
................##########....... ...............###########.......
................................. .......................#.........
................................. .................................
WHAT EACH FAMILY IS ACTUALLY DRAWING
family boundary probabilities multiclass acc
logistic regression one hyperplane yes, native softmax or OvR 0.645
LDA one hyperplane yes, native native k-class 0.645
QDA one quadric yes, native native k-class 0.862
decision tree axis-parallel boxes leaf frequencies native k-class 0.893
k-NN (k=15) local, data-shaped neighbour votes native k-class 0.933
SVM (RBF) smooth, kernel-shaped no, needs Platt OvO 0.940
random forest many boxes, averaged averaged leaves native k-class 0.935
Key Takeaways
- A classifier divides the feature space into regions, and the model family decides what shape those regions can have. That shape is the assumption.
- No amount of tuning lets a linear classifier follow a curved boundary. The gap to a flexible model shows how curved the boundary is.
- Discriminative models learn the boundary directly. Generative models learn each class's distribution and use Bayes' rule; they need less data and fail in more predictable ways.
- Trees and forests draw axis-aligned boxes, so they copy a diagonal boundary with a staircase instead of learning it.
- k-NN and kernel SVMs measure distances, so they need scaled features. Tree ensembles do not.
- SVMs output a signed distance, not a probability. Turning it into a probability needs a separate calibration step.
- One-vs-rest and one-vs-one scores were never trained to be compared across classes, unlike softmax or a built-in multiclass model.
🧪 Practice
- Generate a circular boundary and plot the predicted regions of logistic regression, QDA, a tree and an RBF SVM on the same grid.
- Rotate that dataset by 45 degrees and refit a shallow decision tree. Explain the change in accuracy in terms of axis-aligned splits.
- Fit an SVM with and without a calibration wrapper and compare the predicted labels near the boundary.
- Fit a 4-class problem with OvR and with softmax. For each, check whether the class scores sum to one.
- Train Naive Bayes on features you have made correlated on purpose, and link the drop in quality to the independence assumption.
- Take a dataset where logistic regression barely beats the base rate and work out whether the cause is a curved boundary or useless features.
- Interview: A colleague finds that a decision tree beats logistic regression by 25 accuracy points on two features and concludes that trees are better. What other explanation would you suggest, and what single plot settles it? (Hint: what does a curved true boundary do to a model that can only draw straight ones?)
Logistic Regression
Now the target is yes or no: did the customer leave, is the transaction fraud,
did the patient relapse. The obvious idea is to code it as 0 and 1 and run
least squares. That fails, for a clear reason. A straight line has no limits,
so it will predict 1.3 and -0.2, which are not probabilities. Clipping them
does not help either. A clipped line still assumes the same effect for every
unit of x, and that is impossible near the edges: you cannot add 0.1 to a
probability of 0.98.
What you need is a curve that is roughly straight in the middle and flattens out as it approaches 0 and 1. Logistic regression gets this by changing what is modelled as linear. Instead of the probability itself, it models the log-odds:
odds = p / (1 - p) ranges over (0, infinity)
logit = log(p / (1 - p)) ranges over (-infinity, +infinity)
The logit takes a probability, which must lie between 0 and 1, and stretches it
over the whole number line, where a linear function can move freely. So the
model is logit(p) = b0 + b1*x1 + ... + bp*xp. Undoing the logit gives the
prediction: p = 1 / (1 + exp(-z)), the sigmoid function.
This choice has several consequences.
Coefficients multiply the odds. A coefficient of 0.56 means one more unit
multiplies the odds by exp(0.56) = 1.76, a 76% increase in the odds. It
does not mean a 76% increase in probability, and it is not a fixed number of
percentage points either. The same coefficient moves the probability by 14
points when starting from 0.50, and by only 1.5 points when starting from 0.02,
because the sigmoid is steep in the middle and flat at the ends. People get
this wrong all the time.
There is no direct formula. Least squares has one because its objective is quadratic. The logistic objective is not, so it is fitted step by step using Newton's method, here called iteratively reweighted least squares (IRLS). The good news: the log-likelihood has a single peak (it is concave), so there is one best answer and the fit converges reliably.
The decision boundary is a straight line (or flat plane). The model is
linear in the log-odds. The boundary is where p = 0.5, which means z = 0,
and that is a hyperplane in feature space. So logistic regression cannot
separate classes that are not linearly separable with the features you give
it. If the true boundary is a circle, you need to add x1^2 + x2^2 as a
column.
Perfect separation breaks the fit. If some feature separates the classes with no overlap, the likelihood keeps rising as the coefficient grows: a steeper curve is always better, forever. There is no maximum, so the unpenalised estimate does not exist. In practice you see huge coefficients and a convergence warning. Regularisation fixes this by keeping the objective bounded, which is one reason sklearn regularises logistic regression by default.
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression, LogisticRegression
rng = np.random.default_rng(11)
# --- WHY NOT LINEAR REGRESSION ON A 0/1 TARGET. ---
x = np.r_[rng.normal(2, 1, 200), rng.normal(6, 1, 200)]
y = np.r_[np.zeros(200), np.ones(200)].astype(int)
lin = LinearRegression().fit(x.reshape(-1, 1), y)
print("LINEAR REGRESSION ON A BINARY TARGET")
for v in [-2, 0, 4, 8, 12]:
print(f" x={v:3d} -> predicted 'probability' {lin.predict([[v]])[0]:+.3f}")
print("Values outside [0, 1] are not probabilities, and no amount of clipping")
print("makes the fitted line a model of a probability.")
# --- THE LOGISTIC MODEL: linear in the LOG-ODDS, not in the probability. ---
def sigmoid(z):
return 1.0 / (1.0 + np.exp(-z))
print("\nTHE LINK logit(p) = log(p / (1-p)) is linear; p is not")
print(f" {'log-odds':>9s} {'odds':>8s} {'p':>7s}")
for z in [-3, -1, 0, 1, 3]:
print(f" {z:9.1f} {np.exp(z):8.3f} {sigmoid(z):7.3f}")
# --- FITTING BY MAXIMUM LIKELIHOOD: no closed form, so iterate. ---
# Newton / IRLS by hand, to show there is no mystery inside `.fit()`.
Xd = np.column_stack([np.ones(len(x)), x])
beta = np.zeros(2)
for it in range(8):
p = sigmoid(Xd @ beta)
W = p * (1 - p) # variance of each Bernoulli
grad = Xd.T @ (y - p) # score
H = -(Xd * W[:, None]).T @ Xd # Hessian, negative definite
beta = beta - np.linalg.solve(H, grad) # Newton step
ll = np.sum(y * np.log(p + 1e-12) + (1 - y) * np.log(1 - p + 1e-12))
print(f" iter {it}: beta {np.round(beta, 4)} log-likelihood {ll:9.4f}")
sk = LogisticRegression(C=np.inf, max_iter=2000).fit(x.reshape(-1, 1), y)
print(f"\nhand-rolled IRLS : {np.round(beta, 4)}")
print(f"sklearn : {np.round(np.r_[sk.intercept_, sk.coef_[0]], 4)}")
# --- INTERPRETATION: exp(coefficient) is an ODDS RATIO, not a probability. ---
df = pd.DataFrame({
"tenure_months": rng.integers(1, 60, 3000),
"monthly_spend": rng.gamma(4, 15, 3000),
"support_tickets": rng.poisson(1.2, 3000),
})
logit = (-1.0 - 0.045 * df["tenure_months"] + 0.010 * df["monthly_spend"]
+ 0.55 * df["support_tickets"])
df["churn"] = (rng.random(3000) < sigmoid(logit)).astype(int)
feats = ["tenure_months", "monthly_spend", "support_tickets"]
m = LogisticRegression(C=np.inf, max_iter=4000).fit(df[feats], df["churn"])
print(f"\nchurn base rate {df['churn'].mean():.3f}")
print(f"{'feature':18s} {'coef':>8s} {'odds ratio':>11s} reading")
for f, c in zip(feats, m.coef_[0]):
direction = "increases" if c > 0 else "decreases"
print(f" {f:16s} {c:+8.4f} {np.exp(c):11.3f} one more unit {direction}"
f" the odds by {abs(np.exp(c) - 1) * 100:.1f}%")
# The same coefficient means different things in probability terms depending on
# where you start, which is why "marginal effect" is not one number.
print("\nTHE SAME COEFFICIENT, DIFFERENT PROBABILITY EFFECTS")
coef = m.coef_[0][2] # support_tickets
for base_p in [0.02, 0.10, 0.50, 0.90]:
base_logit = np.log(base_p / (1 - base_p))
new_p = sigmoid(base_logit + coef)
print(f" p={base_p:.2f} -> {new_p:.3f} (+{(new_p-base_p)*100:5.2f} pp)")
print("Constant on the log-odds scale is NOT constant on the probability")
print("scale. The effect is largest near p = 0.5 and vanishes at the extremes.")
# --- PERFECT SEPARATION: the likelihood has no maximum. ---
xs = np.array([1., 2., 3., 4., 6., 7., 8., 9.])
ys = np.array([0, 0, 0, 0, 1, 1, 1, 1])
print("\nPERFECT SEPARATION (classes never overlap)")
for C in [0.1, 1.0, 1e4, 1e6]:
ms = LogisticRegression(C=C, max_iter=200000).fit(xs.reshape(-1, 1), ys)
print(f" C={C:<8g} slope {ms.coef_[0][0]:12.4f}"
f" P(y=1 | x=5) {ms.predict_proba([[5.]])[0,1]:.4f}")
print("With no penalty the slope diverges: pushing it higher always improves")
print("the likelihood. Regularisation is what makes the estimate exist.")
WHY NOT A STRAIGHT LINE ON A 0/1 TARGET
x=-2 -> "probability" -0.716 Values outside [0,1] are not
x= 0 -> "probability" -0.314 probabilities, and clipping them
x= 4 -> "probability" +0.491 does not repair the constant-effect
x= 8 -> "probability" +1.296 assumption that produced them.
x=12 -> "probability" +2.100
THE SIGMOID: LINEAR IN LOG-ODDS, SATURATING IN PROBABILITY
p
1.0 | _____---------
| ___--
| _-- flat: a big change in z
0.5 +- - - - - - -*- - - - - - - - - - - barely moves p
| _-- steep: small changes
| ___-- in z move p a lot
0.0 |---- flat again
+----------------------------------> z = b0 + b1*x
-6 -3 0 3 6
log-odds odds p
-3.0 0.050 0.047
-1.0 0.368 0.269
0.0 1.000 0.500
+1.0 2.718 0.731
+3.0 20.086 0.953
FITTING: NEWTON / IRLS CONVERGES IN A HANDFUL OF STEPS
iter 0: beta [ -3.25 0.80] log-likelihood -277.2589
iter 3: beta [-10.35 2.61] log-likelihood -32.6223
iter 6: beta [-14.60 3.73] log-likelihood -23.7191
iter 7: beta [-14.63 3.73] log-likelihood -23.6975
hand-rolled IRLS [-14.6308 3.7335]
sklearn [-14.6326 3.7339]
READING THE COEFFICIENTS (churn base rate 0.299)
feature coef odds ratio reading
tenure_months -0.0405 0.960 4.0% lower odds per month
monthly_spend +0.0078 1.008 0.8% higher odds per dollar
support_tickets +0.5645 1.759 75.9% higher odds per ticket
THE TRAP: CONSTANT ON THE ODDS SCALE IS NOT CONSTANT IN PROBABILITY
(one extra support ticket, coefficient +0.5645)
starting p new p change
0.02 0.035 + 1.46 pp
0.10 0.163 + 6.35 pp
0.50 0.637 +13.75 pp <- largest effect near 0.5
0.90 0.941 + 4.06 pp
PERFECT SEPARATION: THE LIKELIHOOD HAS NO MAXIMUM
C=0.1 slope 0.4398
C=1 slope 1.1364
C=1e4 slope 6.9256 weaker penalty -> steeper slope,
C=1e6 slope 7.8370 with no limit. the unpenalised
estimate does not exist.
Key Takeaways
- Linear regression on a 0/1 target predicts impossible values, because a line with no limits cannot model a probability.
- Logistic regression is linear in the log-odds, and the sigmoid simply undoes the logit.
exp(coefficient)is an odds ratio: it multiplies the odds. It is never a fixed change in probability.- The same coefficient changes the probability most near p = 0.5 and hardly at all near 0 or 1, so its effect on probability is not one number.
- Fitting is iterative because the objective is not quadratic, but it has a single peak, so the answer is unique.
- The decision boundary is a hyperplane, so curved boundaries need extra, hand-made features.
- With perfect separation the unpenalised coefficients grow without limit. Regularisation makes a finite answer exist.
🧪 Practice
- Fit least squares to a 0/1 target and count how many predictions fall outside [0, 1].
- Implement IRLS from the gradient and Hessian and check that your coefficients match
LogisticRegressionto four decimals.- Take a fitted coefficient and compute the change in probability it causes starting from base rates of 0.01, 0.1, 0.5 and 0.9. Explain the pattern in one sentence.
- Build a dataset with a circular true boundary. Show that logistic regression fails on the raw features, then add squared terms and show it succeeds.
- Build perfectly separable data and report the fitted slope at C = 0.01, 1, 100 and 10000.
- Interview: A model reports an odds ratio of 3.0 for a rare outcome with a 1% base rate, and a product manager says the feature "triples the risk". Are they right? (Hint: work out the probability after tripling the odds when p = 0.01, then try p = 0.4.)
Multinomial and Ordinal Regression
Binary logistic regression handles two outcomes. Real targets often have more: which of three plans a visitor picks, how many stars out of five a reviewer gives, which of forty products a customer buys. The key question is whether the classes have an order. Order is information, and ignoring it costs accuracy.
Multinomial (softmax) regression is for classes with no order. Each class gets its own linear score, and the scores are turned into probabilities that sum to one:
P(y = k | x) = exp(z_k) / sum over all j of exp(z_j) where z_k = b_k . x
Only the differences between class scores matter: add the same number to
every z_k and the probabilities do not change. So one class is chosen as the
reference, and every other class is compared against it. For example, a
coefficient of +0.80 for "referred" in the "pro" row means being referred
multiplies the odds of choosing pro rather than free by exp(0.80). A class
coefficient on its own, without a reference, does not exist.
The alternative is one-vs-rest: train K separate binary models, each asking "class k or not?". It is simpler, easy to run in parallel, and works well in practice. But its probabilities come from separate models and do not sum to one, so they must be rescaled afterwards. Softmax fits all classes together and always gives consistent probabilities. Use softmax when you need reliable multi-class probabilities. One-vs-rest is fine when you only need a ranking or the top label, and for some base models it is the only option.
Ordinal regression is for ordered classes, and this is where the interesting choice is. With satisfaction ratings from 1 to 5, you have three options, and two of them are wrong.
- Treat the ratings as unordered classes. This works but wastes information. You estimate four separate sets of coefficients when one would do. Nothing stops the model from saying that higher quality raises P(rating=5) and P(rating=2) while lowering P(rating=4), a pattern that makes no sense for ordered ratings.
- Treat the ratings as numbers and use linear regression. This assumes the step from 1 to 2 is the same size as the step from 4 to 5, and you have no reason to believe that about how people answer. It also predicts values like 4.3 and 5.7, which are not ratings.
- Use the proportional odds model (ordered logit). This is the right one.
The proportional odds model imagines a hidden continuous score
z = b.x + error and a set of cut-off points. You see rating k when z
falls between cut-off k-1 and cut-off k. This leads to cumulative logits
with one shared slope and one intercept per cut-off:
logit P(Y <= k | x) = alpha_k - b . x for k = 1 .. K-1
The shared b is what captures the order. A feature that raises the hidden
score moves probability towards higher ratings at every cut-off at once. The
name "proportional odds" comes from this sharing: a feature's odds ratio is the
same at every cut-off. That is a real assumption, and you can check it. Fit the
K-1 cumulative logits separately and see whether their slopes agree. If they
differ a lot, use a partial proportional odds model or go back to the
unordered (nominal) model.
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.multiclass import OneVsRestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix
rng = np.random.default_rng(5)
# --- THREE UNORDERED CLASSES: which plan does a visitor pick? ---
n = 3000
visits = rng.gamma(3, 4, n)
referred = rng.integers(0, 2, n)
# Utility of each class; the chosen class is the argmax plus noise (a logit model
# in disguise -- this is exactly the story softmax regression tells).
util = np.column_stack([
np.zeros(n), # free, the reference class
-2.0 + 0.16 * visits + 0.8 * referred, # pro
-4.5 + 0.22 * visits + 0.3 * referred, # premium
])
plan = np.argmax(util + rng.gumbel(0, 1, util.shape), axis=1) # Gumbel -> softmax
X = np.column_stack([visits, referred])
names = ["free", "pro", "premium"]
print("class counts:", {k: int(v) for k, v in zip(names, np.bincount(plan))})
Xtr, Xte, ytr, yte = train_test_split(X, plan, test_size=0.3, random_state=0,
stratify=plan)
# --- MULTINOMIAL (softmax) VS ONE-VS-REST. ---
multi = LogisticRegression(C=np.inf, max_iter=5000).fit(Xtr, ytr)
ovr = OneVsRestClassifier(LogisticRegression(C=np.inf, max_iter=5000)).fit(Xtr, ytr)
print(f"\nmultinomial accuracy {accuracy_score(yte, multi.predict(Xte)):.4f}")
print(f"one-vs-rest accuracy {accuracy_score(yte, ovr.predict(Xte)):.4f}")
# The structural difference: softmax probabilities sum to 1 by construction,
# OvR probabilities do not, so they must be renormalised after the fact.
raw_ovr = np.column_stack([e.predict_proba(Xte[:3])[:, 1]
for e in ovr.estimators_])
print("\nprobabilities for three test rows")
print(" multinomial (sums to 1 by construction)")
for row in multi.predict_proba(Xte[:3]):
print(f" {np.round(row, 3)} sum {row.sum():.3f}")
print(" one-vs-rest (three independent fits, sum is whatever it is)")
for row in raw_ovr:
print(f" {np.round(row, 3)} sum {row.sum():.3f}")
# Coefficients in a softmax model are read RELATIVE TO A REFERENCE class.
# sklearn returns one row per class, which is over-parameterised: only the
# DIFFERENCES between rows are identified. Subtract the reference row to read.
coef = multi.coef_ - multi.coef_[0]
print("\nsoftmax coefficients, expressed against 'free' as the reference")
print(f" {'class':10s} {'visits':>9s} {'referred':>10s}")
for i, nm in enumerate(names):
print(f" {nm:10s} {coef[i][0]:+9.4f} {coef[i][1]:+10.4f}")
print(" true values: pro +0.160 / +0.800, premium +0.220 / +0.300")
# --- ORDERED CLASSES: satisfaction 1..5. Order is information. ---
m2 = 4000
quality = rng.normal(0, 1, m2)
latent = 1.1 * quality + rng.logistic(0, 1, m2) # a latent continuous score
cuts = np.array([-2.0, -0.7, 0.6, 2.1]) # four thresholds -> five levels
rating = np.searchsorted(cuts, latent) # 0..4, an ORDERED outcome
print("\nrating counts:", np.bincount(rating))
# WRONG OPTION 1: treat it as nominal. Throws the ordering away, and pays for it
# by estimating 4 slopes where the ordinal model estimates 1.
nominal = LogisticRegression(C=np.inf, max_iter=5000).fit(
quality.reshape(-1, 1), rating)
print("\nnominal multinomial: one slope per class, order ignored")
print(" slopes:", np.round(nominal.coef_.ravel(), 3),
"<- monotone here only by luck, nothing enforces it")
# WRONG OPTION 2: treat it as a number. Assumes the gap 1->2 equals 4->5.
from sklearn.linear_model import LinearRegression
lin = LinearRegression().fit(quality.reshape(-1, 1), rating)
print(f"\nlinear on the codes: slope {lin.coef_[0]:.3f}")
print(f" predicts {lin.predict([[3.0]])[0]:.2f} for quality=3.0,"
" but the scale stops at 4")
# RIGHT: PROPORTIONAL ODDS. Model P(Y <= k) with a SHARED slope and one
# intercept per threshold. Fit it as K-1 binary logits, tied by construction.
print("\nPROPORTIONAL ODDS fit K-1 cumulative logits, check the shared slope")
print(f" {'cut':14s} {'intercept':>10s} {'slope':>8s}")
slopes = []
for k in range(4):
binary = (rating > k).astype(int) # P(Y > k)
b = LogisticRegression(C=np.inf, max_iter=5000).fit(
quality.reshape(-1, 1), binary)
slopes.append(b.coef_[0][0])
print(f" P(Y > {k}) {b.intercept_[0]:+10.4f} {b.coef_[0][0]:+8.4f}")
print(f" slopes are {np.round(slopes, 3)}: close together, which is the")
print(f" proportional-odds assumption HOLDING. Pooled slope {np.mean(slopes):.3f}"
f" vs true 1.100")
print(" If these slopes differed sharply, the assumption would be violated and")
print(" a partial-proportional-odds or nominal model would be needed.")
UNORDERED CLASSES: SOFTMAX VS ONE-VS-REST
class counts: free 1254, pro 1516, premium 230
multinomial accuracy 0.6467
one-vs-rest accuracy 0.6500 accuracy is essentially tied
probabilities for three test rows
multinomial one-vs-rest
[0.634 0.317 0.049] sum 1.000 [0.633 0.324 0.057] sum 1.013
[0.567 0.401 0.033] sum 1.000 [0.569 0.431 0.035] sum 1.035
[0.100 0.699 0.202] sum 1.000 [0.102 0.721 0.185] sum 1.008
^^^^^^^^^ ^^^^^^^^^
coherent by construction needs renormalising
SOFTMAX COEFFICIENTS ARE ONLY MEANINGFUL AGAINST A REFERENCE
class visits referred
free +0.0000 +0.0000 <- the reference, fixed at zero
pro +0.1559 +0.7965 (true +0.160 / +0.800)
premium +0.1926 +0.2548 (true +0.220 / +0.300)
ORDERED CLASSES: THE LATENT-SCORE PICTURE
latent z = b.x + error, cut by thresholds into observed categories
rating 1 | rating 2 | rating 3 | rating 4 | rating 5
------------+------------+------------+------------+------------> z
a_1 a_2 a_3 a_4
Raising b.x slides the whole distribution RIGHT, so probability
moves toward higher ratings at every cut at once. One slope, four
intercepts. That is the entire model.
THE THREE OPTIONS, ON THE SAME DATA (true slope 1.100)
nominal multinomial slopes [-1.064 -0.597 -0.035 +0.618 +1.079]
4 free slope vectors; monotone here by luck only
linear on the codes slope 0.674, predicts 4.03 at quality=3.0
but the scale stops at 4, and 1->2 is assumed
to equal 4->5
proportional odds cut intercept slope
P(Y > 0) +1.9677 +1.0713
P(Y > 1) +0.7052 +1.1586
P(Y > 2) -0.6073 +1.1809
P(Y > 3) -2.0902 +1.0432
slopes agree (1.04 to 1.18) -> the proportional
odds assumption HOLDS. pooled 1.114 vs true 1.100
| Nominal multinomial | Proportional odds | Linear on codes | |
|---|---|---|---|
| Uses the order | No | Yes | Yes, and assumes more |
| Coefficients per feature | K-1 | 1 | 1 |
| Assumes equal steps | No | No | Yes |
| Predictions are real categories | Yes | Yes | No |
| Extra assumption to check | None | Proportional odds | Equal steps between levels |
| Use when | Classes truly have no order | Ordered, with a shared effect | Rarely; only as a rough baseline |
Key Takeaways
- First ask whether the classes are ordered. Getting this wrong either wastes information or adds a false assumption.
- Softmax probabilities always sum to one. One-vs-rest probabilities come from separate models and must be rescaled.
- Only differences between softmax class scores matter, so every multi-class coefficient is a comparison with a reference class.
- Treating ordered ratings as unordered needs K-1 times as many coefficients and allows patterns that make no sense.
- Treating ordered ratings as numbers assumes equal steps between levels and predicts values that are not categories.
- Proportional odds uses one shared slope and one intercept per cut-off, and that sharing is what captures the order.
- You can test the proportional odds assumption by fitting the cumulative logits separately and comparing their slopes.
🧪 Practice
- Fit softmax and one-vs-rest to the same three-class problem. Report accuracy and the row sums of the predicted probabilities.
- Re-express a fitted softmax model's coefficients against a different reference class and check that the predicted probabilities do not change.
- Simulate ordinal data from a hidden score with known cut-offs, fit the K-1 cumulative logits, and check whether the slopes agree.
- Simulate ordinal data that breaks proportional odds by giving one cut-off a different slope, and show that your check catches it.
- Compare a nominal model with a proportional odds model on ordinal data with only 300 rows. Explain why the gap grows as n shrinks.
- Interview: A survey team wants to predict Net Promoter Score answers on a 0-10 scale and suggests linear regression because "it is just a number". What do you propose instead and why? (Hint: does the step from 6 to 7 mean the same as the step from 9 to 10 in how NPS is actually used?)
Naive Bayes Classifiers
Bayes' rule says P(y|x) is proportional to P(y) * P(x|y). The prior P(y)
is just how common each class is, which is easy to estimate. The hard part is
the likelihood P(x|y): the probability of seeing this whole combination of
feature values in a given class. With 30 binary features there are over a
billion possible combinations, and you cannot estimate a probability for each.
Naive Bayes gets around this with one assumption: within each class, the
features are independent of each other. Then the joint likelihood becomes a
product of one-feature likelihoods, and each of those can be estimated just by
counting. The number of parameters drops from 2^p - 1 to 2p. Thirty
features go from a billion parameters to sixty.
This is a deliberate trade, not laziness. The assumption is almost always false (in text, "machine" and "learning" clearly appear together), but without it the model could not be fitted at all. In return you get three real benefits.
Fitting is a single counting pass. No optimisation, no iterations, nothing to converge. The formula is exact and works on any amount of data.
It works with very little data. Because it estimates so few parameters, it has very low variance (and high bias). Naive Bayes can be useful with only a couple of dozen labelled examples, which is why it has been the standard baseline for text classification for decades.
It handles many features well. Ten thousand features just means ten thousand separate counts. That is not a problem for Naive Bayes the way it is for distance-based methods.
Two implementation details are required.
Work with logs. Multiplying ten thousand probabilities gives a number too small for the computer to store (it becomes zero). Add the logs instead.
Smooth the counts. If a word never appeared with a class in training, its estimated probability is zero. Zero times anything is zero, so one unseen word cancels all the other evidence in the document and sends the log-likelihood to minus infinity. Laplace (add-alpha) smoothing adds a small extra count to every cell, so no probability is ever exactly zero.
Now the cost, which is the thing most people get wrong. Naive Bayes gives badly calibrated probabilities. When features are correlated, it treats each one as separate evidence, so the same signal is counted several times and the probabilities are pushed towards 0 and 1. The example below uses a set of documents where words come in groups of five near-synonyms. Naive Bayes matches logistic regression on accuracy and AUC, but its log loss is twice as high, and two thirds of its predictions are above 0.99 or below 0.01. The reliability table shows it predicting 0.018 where the true rate is 0.161.
The practical rule: Naive Bayes ranks well but its probabilities are wrong. Use it to pick a label or rank candidates. Do not use its probabilities for a cost-based threshold, an expected-value calculation, or anything else that uses the number itself, unless you calibrate it first with isotonic regression or Platt scaling.
The variants differ only in what distribution they assume for each feature:
| Variant | Feature type | Assumption for each feature |
|---|---|---|
| GaussianNB | Continuous | One normal distribution per feature per class |
| MultinomialNB | Counts | Word counts or frequencies, never negative |
| BernoulliNB | Binary | Present or absent, and counts absence as evidence |
| ComplementNB | Counts | MultinomialNB adjusted for imbalanced classes |
| CategoricalNB | Categorical | One categorical distribution per feature |
import numpy as np
import pandas as pd
from sklearn.naive_bayes import GaussianNB, MultinomialNB, BernoulliNB
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, log_loss, roc_auc_score
rng = np.random.default_rng(53)
# --- THE DERIVATION, IN THREE LINES OF ARITHMETIC. ---
# Bayes: P(y|x) is proportional to P(y) * P(x|y). The hard part is P(x|y) for a
# whole feature vector. Naive Bayes ASSUMES the features are independent given
# the class, so that joint factorises into a product of one-dimensional terms.
print("WHY THE 'NAIVE' ASSUMPTION IS NECESSARY, NOT LAZY")
for p_ in [3, 10, 30]:
joint = 2 ** p_ - 1 # parameters for a full joint over binary x
naive = 2 * p_ # parameters under conditional independence
print(f" {p_:3d} binary features: full joint needs {joint:>12,d}"
f" parameters, naive needs {naive:>3d}")
print(" The full joint is unestimable and the factorisation is what makes the")
print(" model fittable at all. It is a deliberate trade, not an oversight.")
# --- COUNTING IS THE WHOLE FIT: one pass, closed form, no optimisation. ---
n = 4000
spam = (rng.random(n) < 0.35).astype(int)
# Three binary "word present" features with class-dependent rates.
rates = {0: [0.10, 0.30, 0.05], 1: [0.70, 0.35, 0.60]}
Xb = np.column_stack([
(rng.random(n) < np.where(spam == 1, rates[1][j], rates[0][j])).astype(int)
for j in range(3)])
Btr, Bte, str_, ste = train_test_split(Xb, spam, test_size=0.3, random_state=0,
stratify=spam)
# Fit by hand: class priors and per-feature conditional probabilities, with
# Laplace smoothing so an unseen combination does not zero out the product.
ALPHA = 1.0
prior = np.array([np.mean(str_ == c) for c in (0, 1)])
theta = np.array([[(Btr[str_ == c, j].sum() + ALPHA)
/ ((str_ == c).sum() + 2 * ALPHA) for j in range(3)]
for c in (0, 1)])
print(f"\nfitted by counting: prior {np.round(prior, 4)}")
print(f" P(word_j = 1 | ham ) {np.round(theta[0], 4)} (true {rates[0]})")
print(f" P(word_j = 1 | spam) {np.round(theta[1], 4)} (true {rates[1]})")
def nb_predict(Z):
"""Work in LOG space: multiplying 10,000 probabilities underflows to 0."""
ll = np.log(prior)[None, :] + np.stack([
(Z * np.log(theta[c]) + (1 - Z) * np.log(1 - theta[c])).sum(1)
for c in (0, 1)], axis=1)
return ll.argmax(1)
lib = BernoulliNB(alpha=ALPHA).fit(Btr, str_)
print(f"\nhand-rolled accuracy {accuracy_score(ste, nb_predict(Bte)):.4f}"
f" BernoulliNB {lib.score(Bte, ste):.4f}")
# --- WHY LAPLACE SMOOTHING IS NOT OPTIONAL. ---
# A single zero count makes the entire product zero, whatever the other
# evidence says. One unseen word vetoes every other word in the document.
print("\nZERO COUNTS VETO EVERYTHING (one word never seen with class 'spam')")
for a in [0.0, 1e-10, 0.01, 1.0]:
t = (Btr[str_ == 1, 0].sum() * 0 + a) / ((str_ == 1).sum() + 2 * a)
contribution = -np.inf if t == 0 else np.log(t)
print(f" alpha={a:<7g} P(word|spam) = {t:.3e}"
f" log contribution {contribution:>9.3f}")
print(" With alpha=0 the log-likelihood is -inf: the class is ruled out")
print(" entirely on the strength of one absent word.")
# --- A CORPUS THAT VIOLATES THE ASSUMPTION ON PURPOSE. ---
# Real text does not have independent words: synonyms and phrases travel
# together. Model that directly -- 6 "phrases" of 5 near-synonymous words, each
# phrase either present or absent as a unit -- and the independence assumption
# is violated in exactly the way real text violates it.
NPH, WPP = 6, 5
P = NPH * WPP
m = 5000
lab = (rng.random(m) < 0.5).astype(int)
phrase_prob = np.array([[0.15, 0.20, 0.25, 0.50, 0.55, 0.60], # class 0
[0.50, 0.55, 0.60, 0.15, 0.20, 0.25]]) # class 1
W = np.zeros((m, P))
for j in range(NPH):
present = rng.random(m) < phrase_prob[lab, j]
for w in range(WPP):
W[:, j * WPP + w] = present * rng.poisson(1.5, m) # all 5 co-occur
W += rng.poisson(0.15, (m, P)) # background noise
Wtr, Wte, ltr, lte = train_test_split(W, lab, test_size=0.3, random_state=0,
stratify=lab)
# --- WHERE NAIVE BAYES WINS: VERY LITTLE TRAINING DATA. ---
print("\nSMALL-DATA BEHAVIOUR: Naive Bayes vs logistic regression")
print(f" {'train rows':>11s} {'NB acc':>8s} {'LR acc':>8s} {'NB wins by':>11s}")
for k in [20, 50, 150, 600, 3500]:
nb_k = MultinomialNB().fit(Wtr[:k], ltr[:k])
lr_k = LogisticRegression(max_iter=3000).fit(Wtr[:k], ltr[:k])
a_nb, a_lr = nb_k.score(Wte, lte), lr_k.score(Wte, lte)
print(f" {k:11d} {a_nb:8.4f} {a_lr:8.4f} {a_nb - a_lr:+11.4f}")
print(" Naive Bayes has high bias and very low variance, so it is useful from")
print(" a couple of dozen rows. Logistic regression catches up given data.")
# --- THE COST: PROBABILITIES ARE BADLY CALIBRATED. ---
nb = MultinomialNB().fit(Wtr, ltr)
lr = LogisticRegression(max_iter=3000).fit(Wtr, ltr)
print("\nCALIBRATION: correlated features make NB overconfident")
print(f" {'model':22s} {'accuracy':>9s} {'AUC':>7s} {'log loss':>9s}"
f" {'extreme':>8s}")
for label, mm in [("MultinomialNB", nb), ("logistic regression", lr)]:
pp = mm.predict_proba(Wte)[:, 1]
extreme = np.mean((pp > 0.99) | (pp < 0.01))
print(f" {label:22s} {mm.score(Wte, lte):9.4f}"
f" {roc_auc_score(lte, pp):7.4f} {log_loss(lte, pp):9.4f}"
f" {extreme:8.1%}")
print(" Identical accuracy and AUC. Log loss more than DOUBLE, because NB")
print(" multiplies five copies of the same phrase as five independent votes,")
print(" so one piece of evidence is counted five times and the probability is")
print(" driven to the extremes. RANKING survives; CALIBRATION does not, which")
print(" is why NB is usable for classification and unusable for thresholds,")
print(" expected-value calculations, or anything that reads the number itself.")
# Reliability check: bucket the predictions and compare with observed frequency.
print("\n RELIABILITY (predicted bucket vs observed rate)")
for label, mm in [("MultinomialNB", nb), ("logistic regression", lr)]:
pp = mm.predict_proba(Wte)[:, 1]
print(f" {label}")
for lo, hi in [(0.0, 0.2), (0.2, 0.4), (0.4, 0.6), (0.6, 0.8), (0.8, 1.0)]:
sel = (pp >= lo) & (pp < hi)
if sel.sum() < 10:
print(f" [{lo:.1f}, {hi:.1f}): too few rows to judge")
continue
print(f" [{lo:.1f}, {hi:.1f}): predicted {pp[sel].mean():.3f}"
f" observed {lte[sel].mean():.3f} n={sel.sum()}")
# --- THE VARIANTS, AND WHICH DATA EACH ASSUMES. ---
table = pd.DataFrame([
("GaussianNB", "continuous", "one Gaussian per feature per class"),
("MultinomialNB", "counts", "word counts or frequencies, non-negative"),
("BernoulliNB", "binary", "presence/absence, and penalises absence"),
("ComplementNB", "counts", "MultinomialNB fixed for imbalanced text"),
("CategoricalNB", "categorical", "one categorical distribution per feature"),
], columns=["variant", "feature type", "what it assumes"])
print("\n", table.to_string(index=False))
# GaussianNB on continuous data, and what its assumption costs.
Z = rng.normal(0, 1, (3000, 6))
Z[:, 1] = Z[:, 0] * 0.95 + rng.normal(0, 0.3, 3000) # a correlated pair
cls = (Z[:, 0] + Z[:, 1] - 0.8 * Z[:, 2] + rng.normal(0, 0.6, 3000) > 0.5
).astype(int)
Ztr, Zte, ctr, cte = train_test_split(Z, cls, test_size=0.3, random_state=0)
print("\nGAUSSIAN NB WITH A CORRELATED FEATURE PAIR")
for label, m in [("GaussianNB", GaussianNB()),
("logistic regression", LogisticRegression(max_iter=2000))]:
m.fit(Ztr, ctr)
p = m.predict_proba(Zte)[:, 1]
print(f" {label:22s} acc {m.score(Zte, cte):.4f}"
f" AUC {roc_auc_score(cte, p):.4f} log loss {log_loss(cte, p):.4f}")
print(" GaussianNB treats the 0.95-correlated pair as two independent votes,")
print(" so it double-counts that evidence. Accuracy survives; log loss does not.")
WHY THE 'NAIVE' ASSUMPTION IS NECESSARY, NOT LAZY
binary features full joint needs naive needs
3 7 6
10 1,023 20
30 1,073,741,823 60
the full joint is unestimable; the factorisation is what makes the
model fittable at all.
full joint: P(x1,x2,x3 | y) <- one parameter per combination
naive: P(x1|y) P(x2|y) P(x3|y) <- one parameter per feature
THE FIT IS COUNTING (no optimiser, no iterations)
prior [0.6286 0.3714]
P(word_j=1 | ham ) [0.1005 0.3087 0.0522] true [0.10 0.30 0.05]
P(word_j=1 | spam) [0.6977 0.3522 0.6094] true [0.70 0.35 0.60]
hand-rolled accuracy 0.8717 BernoulliNB 0.8717
ZERO COUNTS VETO EVERYTHING
alpha=0 P(word|spam) = 0.000e+00 log contribution -inf
alpha=1e-10 P(word|spam) = 9.615e-14 log contribution -29.973
alpha=0.01 P(word|spam) = 9.615e-06 log contribution -11.552
alpha=1 P(word|spam) = 9.597e-04 log contribution -6.949
With alpha=0 one absent word rules out the class entirely, however
strong the other evidence is.
SMALL-DATA BEHAVIOUR
train rows NB acc LR acc NB wins by
20 0.7780 0.7467 +0.0313
50 0.7813 0.7227 +0.0587
150 0.7973 0.7533 +0.0440
600 0.7947 0.7833 +0.0113
3500 0.8027 0.7987 +0.0040
High bias, very low variance: useful from two dozen rows, and caught
up with once there is real data.
THE COST: CALIBRATION
(words arrive in phrases of 5 near-synonyms, so features are correlated)
model accuracy AUC log loss extreme
MultinomialNB 0.8027 0.8861 0.8399 65.8%
logistic regression 0.7987 0.8829 0.4292 2.1%
^^^^^^ ^^^^^^ ^^^^^^
identical identical DOUBLE
RELIABILITY (predicted bucket vs observed rate)
MultinomialNB logistic regression
bucket pred observed bucket pred observed
[0, .2) 0.018 0.161 [0, .2) 0.090 0.087
[.2,.4) 0.304 0.443 [.2,.4) 0.292 0.258
[.4,.6) 0.490 0.571 [.4,.6) 0.501 0.519
[.6,.8) 0.699 0.577 [.6,.8) 0.703 0.730
[.8, 1] 0.983 0.842 [.8, 1] 0.912 0.914
NB says 0.018 where the truth is 0.161, and 0.983 where it is 0.842.
It counts one phrase as five independent votes.
GAUSSIAN NB WITH A 0.95-CORRELATED FEATURE PAIR
GaussianNB acc 0.9011 AUC 0.9619 log loss 0.2505
logistic regression acc 0.9200 AUC 0.9785 log loss 0.1855
Same story on continuous features: accuracy survives, log loss does not.
Key Takeaways
- The independence assumption cuts the number of parameters from exponential to linear in the number of features, which is what makes the model possible to fit.
- Fitting is a single counting pass with an exact formula, so there is no optimiser and nothing to converge.
- High bias and very low variance make Naive Bayes useful with a few dozen rows and a strong baseline with many features.
- Always work with logs, because multiplying thousands of probabilities rounds down to zero.
- Laplace smoothing is required: a single zero count sends the log-likelihood to minus infinity and cancels all other evidence.
- Correlated features make the model count the same evidence several times, which pushes probabilities towards 0 and 1.
- Ranking and AUC survive this, but calibration does not. Do not use Naive Bayes probabilities for thresholds or expected values without calibrating them.
- The variants differ only in the distribution assumed for each feature, so pick the one that matches your feature type.
🧪 Practice
- Implement Bernoulli Naive Bayes by counting, with Laplace smoothing, and match
BernoulliNBon accuracy.- Compute the number of parameters for the full joint distribution and for Naive Bayes at 5, 20 and 50 binary features.
- Set alpha to zero on data with a feature-class combination never seen in training, and show what happens to the predicted log-likelihood.
- Compare Naive Bayes with logistic regression at 20, 100, 1000 and 10,000 training rows, and find where logistic regression overtakes it.
- Build a set of documents where words appear in correlated groups. Compare accuracy, AUC and log loss for both models, plus a reliability table.
- Calibrate a Naive Bayes model with
CalibratedClassifierCVusing isotonic regression and report the improvement in log loss and Brier score.- Interview: Your spam filter uses Naive Bayes, and you must set a threshold so that at most 0.1% of real mail is blocked. Why is the raw score a problem, and what do you do? (Hint: what would need to be true about the number 0.999 for that threshold to mean what you want?)
Discriminant Analysis
Discriminant analysis is a generative model like Naive Bayes, with one assumption relaxed. Naive Bayes treats each feature separately within a class. Discriminant analysis models each class as a full multivariate normal (Gaussian) distribution, so it can capture correlations between features.
There are two versions, and they differ in one choice:
Linear discriminant analysis (LDA) assumes every class is normal and all classes share the same covariance matrix (the same spread and correlations). With this assumption, the squared terms in the Bayes decision rule cancel out, and the boundary is a straight line (or flat plane). That is where the name comes from: the boundary is linear because the covariance is shared, not as a separate design choice.
Quadratic discriminant analysis (QDA) gives each class its own covariance matrix. The squared terms no longer cancel, so the boundary is curved. It can handle classes with genuinely different shapes or spreads.
The number of parameters shows the trade-off clearly. With K classes and p
features, LDA estimates K mean vectors and one covariance matrix:
K*p + p(p+1)/2. QDA estimates K covariance matrices:
K*p + K*p(p+1)/2. With 200 features and 5 classes, that is 21,100 against
100,500. QDA's extra flexibility is real, and so is its need for data. In the
example below, with 40 features and nine rows per feature, QDA reaches 0.97
training accuracy and still loses to LDA on held-out data.
It helps to see the whole family as a range of covariance assumptions:
| Model | Covariance assumption | Boundary shape |
|---|---|---|
| Gaussian Naive Bayes | Diagonal, one per class | Curved, axis-aligned |
| LDA | Full, shared by all classes | Straight |
| QDA | Full, one per class | Curved |
| Regularised LDA | Shared, pulled towards identity | Straight, more stable |
Shrinkage is the practical middle ground. It pulls the estimated covariance
towards a simple scaled identity matrix. This keeps the matrix inverse stable
when there is not enough data to estimate all p(p+1)/2 entries well. Be
precise about when it helps: with 40 features and 360 rows it adds almost
nothing, but with 250 features and 180 rows the unshrunk model is no better
than guessing and shrinkage raises it to 0.77. Shrinkage is not a free
improvement. It fixes an unstable covariance estimate, and it pays off when p
gets close to n.
The comparison with logistic regression is the important one, because it is
the generative versus discriminative distinction in action. LDA models
P(x|y) and P(y) and then uses Bayes' rule to get P(y|x). Logistic
regression models P(y|x) directly and never describes how x was produced.
Both draw a straight boundary on the same data, so why choose one over the
other?
Because the normal-distribution assumption is extra information. When it is true, LDA learns faster: it needs less data, as the example shows. When it is false (skewed features, heavy tails, outliers, categorical predictors), that information is wrong, and logistic regression is safer because it assumes less. With lots of data the two perform about the same, so the choice matters most when data is scarce.
LDA can also do something unique: supervised dimensionality reduction. It
finds the directions that best separate the classes compared with the spread
inside each class, and projects to at most K-1 dimensions. PCA, by contrast,
looks for directions with the most variance and never sees the labels. In the
example below, PCA spends all three of its components on noisy high-variance
columns and scores 0.28. LDA's three components score 0.70, almost matching
all twenty original features. High variance and usefulness for separating
classes are not the same thing.
import numpy as np
import pandas as pd
from sklearn.discriminant_analysis import (LinearDiscriminantAnalysis,
QuadraticDiscriminantAnalysis)
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, log_loss
rng = np.random.default_rng(59)
# --- WHAT DISCRIMINANT ANALYSIS ASSUMES, AND WHAT IT BUYS. ---
# LDA: each class is a Gaussian, all classes SHARE one covariance matrix.
# QDA: each class is a Gaussian with its OWN covariance matrix.
# The shared-covariance assumption is what makes LDA's boundary linear.
def make_data(n, shared_cov, seed):
r = np.random.default_rng(seed)
S0 = np.array([[1.0, 0.75], [0.75, 1.0]])
S1 = S0 if shared_cov else np.array([[2.5, -0.9], [-0.9, 0.8]])
a = r.multivariate_normal([0, 0], S0, n // 2)
b = r.multivariate_normal([1.6, 1.6], S1, n // 2)
return (np.vstack([a, b]),
np.r_[np.zeros(n // 2), np.ones(n // 2)].astype(int))
print("THE ASSUMPTION DECIDES WHICH MODEL WINS")
print(f" {'data':26s} {'LDA':>7s} {'QDA':>7s} {'logistic':>9s} {'GaussianNB':>11s}")
for label, shared in [("shared covariance", True),
("different covariances", False)]:
X, y = make_data(4000, shared, 1)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
stratify=y)
scores = []
for m in [LinearDiscriminantAnalysis(), QuadraticDiscriminantAnalysis(),
LogisticRegression(max_iter=2000), GaussianNB()]:
scores.append(m.fit(Xtr, ytr).score(Xte, yte))
print(f" {label:26s} {scores[0]:7.4f} {scores[1]:7.4f}"
f" {scores[2]:9.4f} {scores[3]:11.4f}")
print(" With a shared covariance LDA's extra assumption is FREE and it matches")
print(" logistic regression. With different covariances only QDA can express")
print(" the curved boundary, and LDA is structurally unable to catch up.")
# --- THE PARAMETER COUNT IS THE WHOLE STORY OF THE TRADE-OFF. ---
print("\nPARAMETERS TO ESTIMATE, FOR K CLASSES AND p FEATURES")
print(f" {'p':>4s} {'K':>3s} {'LDA':>8s} {'QDA':>10s} {'Gaussian NB':>12s}")
for p_, K in [(2, 2), (10, 2), (10, 5), (50, 5), (200, 5)]:
lda = K * p_ + p_ * (p_ + 1) // 2 # K means + ONE covariance
qda = K * p_ + K * p_ * (p_ + 1) // 2 # K means + K covariances
gnb = K * p_ + K * p_ # K means + K diagonal variances
print(f" {p_:4d} {K:3d} {lda:8,d} {qda:10,d} {gnb:12,d}")
print(" QDA's cost grows with K*p^2. At 200 features and 5 classes it needs")
print(" 100,500 parameters, so it needs a great deal of data or it overfits.")
print(" Gaussian NB is LDA with the covariance forced DIAGONAL: the same family")
print(" with the strongest possible assumption about feature independence.")
# --- WATCH QDA OVERFIT AS p GROWS RELATIVE TO n. ---
# The truth here is a SHARED covariance, so LDA's assumption is correct and
# QDA's extra freedom buys nothing except variance.
print("\nQDA NEEDS DATA: fixed n, growing p (truth is a SHARED covariance)")
print(f" {'p':>4s} {'n/p':>6s} {'LDA':>8s} {'QDA':>8s} {'QDA train':>10s}"
f" {'LDA+shrink':>11s}")
n = 600
for p_ in [2, 5, 15, 40]:
A = rng.normal(0, 1, (p_, p_))
S = (A @ A.T) / p_ + np.eye(p_) # scale-stable as p grows
mu = np.full(p_, 2.0 / np.sqrt(p_)) # constant total separation
X = np.vstack([rng.multivariate_normal(np.zeros(p_), S, n // 2),
rng.multivariate_normal(mu, S, n // 2)])
y = np.r_[np.zeros(n // 2), np.ones(n // 2)].astype(int)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.4, random_state=0,
stratify=y)
l = LinearDiscriminantAnalysis().fit(Xtr, ytr)
q = QuadraticDiscriminantAnalysis().fit(Xtr, ytr)
sh = LinearDiscriminantAnalysis(solver="lsqr",
shrinkage="auto").fit(Xtr, ytr)
print(f" {p_:4d} {len(Xtr)/p_:6.1f} {l.score(Xte, yte):8.4f}"
f" {q.score(Xte, yte):8.4f} {q.score(Xtr, ytr):10.4f}"
f" {sh.score(Xte, yte):11.4f}")
print(" At p=40 with 9 rows per feature, QDA's TRAINING accuracy is 0.97 while")
print(" its test accuracy has collapsed below LDA's. Estimating K separate")
print(" p-by-p covariance matrices is simply not possible at this n.")
# --- SHRINKAGE: THE MIDDLE GROUND, AND WHEN IT ACTUALLY PAYS. ---
# Shrinkage matters once p is comparable to n, where the sample covariance is
# near-singular and inverting it amplifies noise. Below that it costs a little.
print("\nSHRINKAGE pulls the covariance toward a scaled identity")
print(f" {'n_train':>8s} {'p':>5s} " +
" ".join(f"{lbl:>7s}" for lbl in ["none", "0.1", "0.5", "0.9", "auto"]))
for nn, pp in [(600, 40), (400, 200), (300, 250)]:
A = rng.normal(0, 1, (pp, pp))
S = (A @ A.T) / pp + np.eye(pp)
mu = np.full(pp, 2.5 / np.sqrt(pp))
X = np.vstack([rng.multivariate_normal(np.zeros(pp), S, nn // 2),
rng.multivariate_normal(mu, S, nn // 2)])
y = np.r_[np.zeros(nn // 2), np.ones(nn // 2)].astype(int)
Xa, Xb, ya, yb = train_test_split(X, y, test_size=0.4, random_state=0,
stratify=y)
cells = []
for sh in [None, 0.1, 0.5, 0.9, "auto"]:
m = LinearDiscriminantAnalysis(solver="lsqr", shrinkage=sh).fit(Xa, ya)
cells.append(f"{m.score(Xb, yb):7.4f}")
print(f" {len(Xa):8d} {pp:5d} " + " ".join(cells))
print(" At p=40 with 360 rows, shrinkage buys essentially nothing. At p=250")
print(" with 180 rows, the unshrunk estimate is AT CHANCE and shrinkage lifts")
print(" it to 0.77. Shrinkage is not a free improvement; it is the fix for a")
print(" near-singular covariance estimate, so it pays exactly when p nears n.")
# --- GENERATIVE VS DISCRIMINATIVE: THE REAL DISTINCTION. ---
# LDA models P(x|y) and P(y), then applies Bayes' rule. Logistic regression
# models P(y|x) directly. Same linear boundary, different route, and the
# difference shows up at small n and under outliers.
print("\nGENERATIVE (LDA) VS DISCRIMINATIVE (LOGISTIC) AT SMALL n")
Xb, yb = make_data(4000, True, 7)
Btr, Bte, ytr2, yte2 = train_test_split(Xb, yb, test_size=0.5, random_state=0,
stratify=yb)
print(f" {'train rows':>11s} {'LDA':>8s} {'logistic':>9s}")
for k in [10, 20, 50, 200, 2000]:
l = LinearDiscriminantAnalysis().fit(Btr[:k], ytr2[:k])
g = LogisticRegression(max_iter=3000).fit(Btr[:k], ytr2[:k])
print(f" {k:11d} {l.score(Bte, yte2):8.4f} {g.score(Bte, yte2):9.4f}")
print(" LDA converges faster because the Gaussian assumption is extra")
print(" information, and when it is true that information is free. When it is")
print(" false -- skewed features, outliers, categorical predictors -- logistic")
print(" regression is the safer choice because it assumes less.")
# LDA also does supervised dimensionality reduction, which PCA does not.
print("\nLDA AS SUPERVISED DIMENSIONALITY REDUCTION")
nm = 2000
cls = rng.integers(0, 4, nm)
Xm = rng.normal(0, 1, (nm, 20))
Xm[:, :2] += cls[:, None] * 1.2 # the class signal, modest variance
Xm[:, 2:8] *= 12 # high-variance columns with NO signal
Mtr, Mte, mtr, mte = train_test_split(Xm, cls, test_size=0.3, random_state=0,
stratify=cls)
from sklearn.decomposition import PCA
print(f" LDA can project to at most K-1 = {len(np.unique(cls)) - 1} dimensions")
for label, tr in [("LDA (uses the labels)",
LinearDiscriminantAnalysis(n_components=3).fit(Mtr, mtr)),
("PCA (ignores the labels)",
PCA(n_components=3).fit(Mtr))]:
clf = LogisticRegression(max_iter=3000).fit(tr.transform(Mtr), mtr)
print(f" {label:26s} 3-component test accuracy"
f" {clf.score(tr.transform(Mte), mte):.4f}")
full = LogisticRegression(max_iter=3000).fit(Mtr, mtr)
print(f" {'all 20 features':26s} test accuracy {full.score(Mte, mte):.4f}")
print(" PCA spent all three components on the high-variance noise columns and")
print(" threw the signal away. LDA kept nearly all of it in three dimensions,")
print(" because it is the only one of the two that saw the labels. Variance and")
print(" discriminative power are different things.")
THE ASSUMPTION DECIDES WHICH MODEL WINS
data LDA QDA logistic GaussianNB
shared covariance 0.7975 0.7975 0.7967 0.7942
different covariances 0.8700 0.9025 0.8750 0.8867
^^^^^^
Only QDA can express a curved boundary. When the covariances really
are shared, LDA's extra assumption is FREE.
LDA: shared covariance QDA: per-class covariance
___ ___ ___ ______
/ \ / \ / A \ / \
| A | | B | | | | B |
\___/ \___/ \___/ \________/
| \ /
a LINE separates them a CURVE is needed
PARAMETERS TO ESTIMATE
p K LDA QDA Gaussian NB
2 2 7 10 8
10 2 75 130 40
10 5 105 325 100
50 5 1,525 6,625 500
200 5 21,100 101,500 2,000
QDA grows as K*p^2. Gaussian NB is LDA with the covariance forced
DIAGONAL -- the same family, strongest possible assumption.
QDA NEEDS DATA (truth is a SHARED covariance)
p n/p LDA QDA QDA train LDA+shrink
2 180.0 0.8042 0.8125 0.7917 0.8042
5 72.0 0.8167 0.8000 0.8167 0.8292
15 24.0 0.7792 0.6833 0.8250 0.7833
40 9.0 0.7667 0.6125 0.9667 0.7333
^^^^^^
Training accuracy 0.97 while test accuracy collapsed below LDA's.
SHRINKAGE PAYS WHEN p APPROACHES n
n_train p none 0.1 0.5 0.9 auto
360 40 0.8083 0.8167 0.8083 0.7958 0.8042
240 200 0.6813 0.7063 0.7312 0.7438 0.7438
180 250 0.5083 0.7083 0.7583 0.7667 0.7583
^^^^^^ ^^^^^^
at chance rescued
GENERATIVE (LDA) VS DISCRIMINATIVE (LOGISTIC) AT SMALL n
train rows LDA logistic
10 0.6305 0.5665 <- the Gaussian assumption is
20 0.7750 0.7825 extra information, and it
50 0.7995 0.7975 pays when data is scarce
200 0.8055 0.8045
2000 0.8060 0.8060 <- they converge
LDA AS SUPERVISED DIMENSIONALITY REDUCTION
(20 features: 2 carry the class signal, 6 are high-variance noise)
LDA (uses the labels) 3 components test accuracy 0.7017
PCA (ignores the labels) 3 components test accuracy 0.2817
all 20 features test accuracy 0.7000
PCA maximises VARIANCE and spent every component on the noise
columns. LDA maximises SEPARATION and kept essentially all of the
signal in three dimensions.
Key Takeaways
- LDA and QDA model each class as a full multivariate normal distribution, so unlike Naive Bayes they can capture correlations between features.
- LDA's boundary is straight because it assumes one shared covariance matrix, not because of a separate design choice.
- QDA gives each class its own covariance and so a curved boundary, but its parameter count grows like
K*p^2.- Gaussian Naive Bayes, LDA and QDA are one family that differs only in how much covariance structure each one estimates.
- Shrinkage stabilises an unreliable covariance estimate. It helps when
pgets close tonand adds nothing when data is plentiful.- LDA is generative and logistic regression is discriminative. The normal assumption makes LDA need less data when it is true, and biased when it is false.
- LDA also works as supervised dimensionality reduction to at most
K-1dimensions, which PCA cannot do because it never sees the labels.
🧪 Practice
- Generate two classes with a shared covariance and two with different covariances. Compare LDA, QDA, logistic regression and GaussianNB on both.
- Tabulate the number of parameters for LDA, QDA and GaussianNB for several values of
pandK.- Keep
nfixed and increasep, recording QDA's training and test accuracy until it falls behind LDA.- Vary
shrinkagefrom 0 to 1 atp/nratios of 0.1, 0.5 and 0.9, and describe when shrinkage starts to matter.- Compare LDA with logistic regression at 10, 30, 100 and 1000 training rows on normal data. Repeat with one strongly skewed feature.
- Project a multi-class dataset to
K-1dimensions with LDA and with PCA, and compare a classifier's accuracy on each.- Interview: You have 90 labelled samples, 400 gene-expression features and three classes. Which member of this family would you use, and how would you set it up? (Hint: count the parameters each option needs and compare with the 90 rows you have.)
k-Nearest Neighbors
Most models learn a set of parameters from the training data and then throw the data away. k-nearest neighbours (k-NN) does the opposite. It keeps every row. To predict for a new point, it finds the k closest stored rows and lets them vote. That is the whole algorithm. There is no training step: "fitting" just means storing the data. k-NN works for regression too, as described at the end of this topic.
Do not dismiss it as a toy. k-NN is non-parametric: it assumes nothing about the shape of the relationship. With enough data, it gets close to the best possible boundary, whatever shape that boundary has. It handles many classes with no changes, and it is easy to explain: "we predicted this customer will leave because the fifteen most similar customers left" is a better explanation than most models can give.
k controls the bias-variance trade-off directly. At k=1, each training point is its own nearest neighbour, so training accuracy is always exactly 1.0, and the boundary bends around every noisy label: highest variance, lowest bias. As k grows, each prediction averages over a wider area and becomes smoother. At k=n, every prediction is the overall majority class: highest bias, zero variance. One direct consequence: training accuracy is useless for k-NN, so choose k with cross-validation.
Three practical facts decide whether k-NN will work at all.
You must scale the features. Distance is computed in each column's raw units. If income ranges over 100,000 and age over 50, income contributes about two thousand times more to the squared distance, and age is effectively ignored. The example below shows unscaled k-NN scoring worse than guessing on a problem where the scaled version reaches 98%.
Too many dimensions ruin it (the curse of dimensionality). In high dimensions, all distances between points become nearly the same. With 500 features, the farthest of a thousand random points is only about 1.2 times as far away as the nearest. When every point is about equally far away, "nearest" means almost nothing. Another way to see it: to capture 10% of the data in 10 dimensions, you need a box that covers 79% of each axis, which is not a "neighbourhood" in any useful sense. Reduce the number of dimensions first, or use a model that does not rely on distance.
The work happens at prediction time. Fitting just stores the data, but every prediction compares the new point with every stored row. Spatial indexes such as k-d trees and ball trees help in low dimensions, but become no faster than brute force in high dimensions. This is the reverse of parametric models, and it makes k-NN awkward in systems that need fast responses. The same nearest-neighbour idea, with an approximate index, is what powers modern vector search.
Two refinements are worth knowing. Distance weighting gives closer neighbours a bigger say. This helps most with a large k, where you want smoothing without distant points counting as much as close ones. For regression, the prediction is the average (or weighted average) of the neighbours' targets instead of a vote.
import numpy as np
import pandas as pd
from sklearn.neighbors import KNeighborsClassifier, KNeighborsRegressor
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.datasets import make_moons
rng = np.random.default_rng(31)
# --- THE ALGORITHM: there is no training. Store the data, and at predict time
# find the k closest rows and vote.
X, y = make_moons(n_samples=1200, noise=0.28, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
stratify=y)
def knn_predict(Xq, Xtr, ytr, k):
"""k-NN from scratch, to show there is nothing hidden in the library."""
d = np.sqrt(((Xq[:, None, :] - Xtr[None, :, :]) ** 2).sum(-1)) # all pairs
idx = np.argsort(d, axis=1)[:, :k] # k closest
return (ytr[idx].mean(axis=1) > 0.5).astype(int) # majority
hand = knn_predict(Xte, Xtr, ytr, 15)
lib = KNeighborsClassifier(n_neighbors=15).fit(Xtr, ytr).predict(Xte)
print(f"hand-rolled agrees with sklearn on {np.mean(hand == lib):.1%} of rows")
print(f"test accuracy {np.mean(hand == yte):.4f}")
# --- K IS THE SMOOTHNESS DIAL: it IS the bias-variance trade-off, visibly. ---
print(f"\n{'k':>5s} {'train':>7s} {'test':>7s} {'5-fold cv':>10s} behaviour")
notes = {1: "memorises every point, incl. noise", 5: "still jagged",
15: "about right here", 51: "smooth", 201: "over-smoothed",
len(Xtr): "predicts the global majority, always"}
for k in [1, 5, 15, 51, 201, len(Xtr)]:
m = KNeighborsClassifier(n_neighbors=k).fit(Xtr, ytr)
# k larger than a CV fold's training size cannot be cross-validated at all,
# which is itself a hint that k=n is a degenerate setting.
cv = (cross_val_score(KNeighborsClassifier(n_neighbors=k), Xtr, ytr,
cv=5).mean() if k <= 4 * len(Xtr) // 5 else float("nan"))
cv_s = f"{cv:10.4f}" if cv == cv else f"{'n/a':>10s}"
print(f"{k:5d} {m.score(Xtr, ytr):7.4f} {m.score(Xte, yte):7.4f}"
f" {cv_s} {notes[k]}")
print("k=1 has training accuracy 1.0 BY CONSTRUCTION: every point is its own")
print("nearest neighbour. Training accuracy is meaningless for k-NN.")
# --- SCALING IS NOT OPTIONAL: distance is in the units of the columns. ---
df = pd.DataFrame({
"age": rng.integers(20, 70, 2000).astype(float), # range ~50
"income": rng.normal(60_000, 20_000, 2000), # range ~100,000
})
target = ((df["age"] > 45) ^ (df["income"] > 60_000)).astype(int)
Atr, Ate, btr, bte = train_test_split(df, target, test_size=0.3, random_state=0)
raw = KNeighborsClassifier(15).fit(Atr, btr)
scaled = make_pipeline(StandardScaler(), KNeighborsClassifier(15)).fit(Atr, btr)
print(f"\nSCALING age spans ~50 units, income spans ~100,000")
print(f" unscaled k-NN test accuracy {raw.score(Ate, bte):.4f}")
print(f" scaled k-NN test accuracy {scaled.score(Ate, bte):.4f}")
print(" Unscaled, a 1-year age difference contributes 1 to squared distance")
print(" while a $1 income difference contributes 1 as well -- so income, with")
print(" its far larger numeric range, dominates and age is invisible.")
# --- THE CURSE OF DIMENSIONALITY: 'nearest' stops meaning anything. ---
print("\nTHE CURSE OF DIMENSIONALITY")
print(f" {'dims':>5s} {'min dist':>9s} {'max dist':>9s} {'ratio':>7s}"
f" {'mean dist':>13s}")
for d in [1, 2, 5, 10, 50, 500]:
P = rng.normal(0, 1, (1000, d))
q = rng.normal(0, 1, (1, d))
dist = np.sqrt(((P - q) ** 2).sum(1))
# the contrast between nearest and farthest is what k-NN relies on
print(f" {d:5d} {dist.min():9.3f} {dist.max():9.3f}"
f" {dist.max()/dist.min():7.2f} {dist.mean():13.3f}")
print(" In 500 dimensions the farthest point is barely 1.5x the distance of")
print(" the nearest. When all distances are nearly equal, 'nearest neighbour'")
print(" carries almost no information, and every distance-based method suffers.")
# How many points you need to keep a neighbourhood locally dense:
print("\n to capture 10% of the data range in each dimension you need a cube")
print(" of side 0.1^(1/d) in a unit hypercube:")
for d in [1, 2, 10, 100]:
print(f" d={d:4d}: side length {0.1 ** (1/d):.4f}"
f" ({'a true neighbourhood' if d < 5 else 'nearly the whole range'})")
# --- REGRESSION AND THE WEIGHTING CHOICE. ---
xr = rng.uniform(0, 10, 800).reshape(-1, 1)
yr = np.sin(xr[:, 0]) + rng.normal(0, 0.3, 800)
# A RANDOM split, not a positional one: xr sorted plus a positional split would
# put the whole test set outside the training range, which k-NN cannot do.
Rtr, Rte, str_, ste = train_test_split(xr, yr, test_size=0.3, random_state=0)
print(f"\n{'k':>4s} {'uniform RMSE':>13s} {'distance RMSE':>14s}")
for k in [1, 5, 20, 60, 200]:
a = KNeighborsRegressor(k, weights="uniform").fit(Rtr, str_)
b = KNeighborsRegressor(k, weights="distance").fit(Rtr, str_)
ra = np.sqrt(np.mean((ste - a.predict(Rte)) ** 2))
rb = np.sqrt(np.mean((ste - b.predict(Rte)) ** 2))
print(f"{k:4d} {ra:13.4f} {rb:14.4f}")
print("Distance weighting matters most at large k, where it lets you smooth")
print("without letting far-away points vote as loudly as close ones.")
# --- THE REAL COST IS AT PREDICT TIME, NOT FIT TIME. ---
import time
for n in [2000, 20_000, 100_000]:
Z = rng.normal(0, 1, (n, 20))
t = (rng.random(n) < 0.5).astype(int)
m = KNeighborsClassifier(15)
t0 = time.perf_counter(); m.fit(Z, t); fit = time.perf_counter() - t0
q = rng.normal(0, 1, (500, 20))
t0 = time.perf_counter(); m.predict(q); pred = time.perf_counter() - t0
print(f" n={n:7,d}: fit {fit*1000:7.2f} ms predict 500 rows"
f" {pred*1000:8.2f} ms")
print(" Fitting is just storing the data. The cost moved to inference, which")
print(" is the opposite of every model that trains parameters.")
THE ALGORITHM
new point ? find the k=5 nearest stored rows and vote
o o x neighbours: o o x o o -> predict o
o ? x
o x x no parameters were fitted. the data IS the model.
k IS THE BIAS-VARIANCE DIAL, WITH NOTHING HIDDEN
k train test 5-fold cv behaviour
1 1.0000 0.9250 0.9024 memorises every point, incl. noise
5 0.9524 0.9333 0.9250 still jagged
15 0.9405 0.9361 0.9345 about right here
51 0.9369 0.9417 0.9381 smooth
201 0.9274 0.9194 0.9071 over-smoothed
840 0.5000 0.5000 n/a predicts the global majority, always
k=1 train accuracy is 1.0 BY CONSTRUCTION -- every point is its own
nearest neighbour. Training accuracy tells you nothing here.
k=1 boundary k=51 boundary
_/\_/\__/\_ ___________
/ \ / \
| jagged | | smooth |
\_/\__/\_/\_/ \___________/
SCALING IS NOT OPTIONAL
(age spans ~50 units, income spans ~100,000)
unscaled k-NN test accuracy 0.4867 <- WORSE THAN CHANCE
scaled k-NN test accuracy 0.9783
Unscaled, one dollar of income counts the same as one year of age,
so income's larger numeric range swamps the distance entirely.
THE CURSE OF DIMENSIONALITY
dims min dist max dist ratio mean dist
1 0.001 4.447 5347.14 0.986
2 0.029 4.584 158.55 1.664
5 0.399 6.712 16.82 2.683
10 1.503 6.274 4.18 3.776
50 6.861 11.708 1.71 9.213
500 29.464 34.540 1.17 32.086
^^^^
farthest point is 1.17x the nearest.
"nearest neighbour" has stopped meaning anything.
side of a cube capturing 10% of the range per dimension:
d=1 0.1000 a true neighbourhood
d=2 0.3162 a true neighbourhood
d=10 0.7943 nearly the whole range
d=100 0.9772 nearly the whole range
DISTANCE WEIGHTING MATTERS AT LARGE k
k uniform RMSE distance RMSE
1 0.4308 0.4308
5 0.3491 0.3638
20 0.3212 0.3462
60 0.3262 0.3390
200 0.4728 0.3455 <- weighting rescues heavy smoothing
THE COST IS AT INFERENCE, NOT TRAINING
n= 2,000: fit 0.81 ms predict 500 rows 8.29 ms
n= 20,000: fit 3.40 ms predict 500 rows 33.63 ms
n=100,000: fit 6.01 ms predict 500 rows 113.64 ms
Key Takeaways
- k-NN has no training step. Fitting stores the data and all the work happens at prediction time, the reverse of parametric models.
- It is truly non-parametric: with enough data it gets close to the best boundary of any shape, and its predictions are easy to explain.
- k directly controls the bias-variance trade-off. Training accuracy is always 1.0 at k=1, so it cannot be used to choose k.
- Features must be scaled, because distance uses raw column units and a column with a large range dominates.
- In high dimensions all distances become similar, so "nearest" loses its meaning and k-NN performs badly. Reduce dimensions first.
- Distance weighting lets you use a large k for smoothness without giving distant points an equal vote.
- Prediction cost grows with the size of the training set, which makes plain k-NN awkward in fast systems and is why approximate indexes exist.
🧪 Practice
- Implement k-NN classification from scratch with a full distance matrix and check that it matches
KNeighborsClassifierexactly.- Vary k from 1 to n and plot training, test and cross-validated accuracy on one chart. Explain why one of the three lines is flat.
- Build a dataset with two features on very different scales and report k-NN accuracy with and without standardisation.
- For 1, 5, 20, 100 and 500 dimensions, sample 1000 points and report the ratio of the farthest to the nearest distance from a query point.
- Compare
weights="uniform"andweights="distance"for k from 1 to 200 on a regression problem.- Time prediction for 1000 query rows against training sets of 1000, 10,000 and 100,000 rows, and fit a curve to the growth.
- Interview: A team wants to use k-NN on 300-dimensional text embeddings and asks whether to standardise the features first. What do you tell them, and what else worries you? (Hint: what does a single embedding dimension mean on its own, and which distance metric were the embeddings trained for?)
Distance Metric Selection
Nearest-neighbour methods, kernel methods and clustering algorithms all depend on a definition of "close". That definition is a modelling assumption, just as important as choosing the shape of a model, yet it is almost always left at the default. This topic explains what the common metrics assume and when they are wrong.
Euclidean distance (L2) is the straight-line distance:
sqrt(sum((a-b)^2)). Because differences are squared, one large difference on
a single axis outweighs many small ones. It is the right default for truly
geometric data, where all dimensions are measured in comparable units.
Manhattan distance (L1) is sum(|a-b|): the distance you walk along the
axes, like city blocks. It spreads the blame evenly across dimensions, so it is
more robust when one feature has an outlier. It also holds up better in high
dimensions.
Chebyshev distance (L-infinity) is max(|a-b|): only the single biggest
difference counts. Use it when a large difference on any one dimension rules a
match out, no matter what the others say.
These three are all special cases of the Minkowski distance, with p = 1,
2 and infinity. You can set p anywhere in between. A small p spreads the
blame across axes; a large p focuses on the worst axis. Choosing p says
which kind of difference you think matters more.
Cosine distance is 1 - (a.b)/(|a||b|): it measures the angle between two
vectors and ignores their lengths completely. Use it whenever the size of a
vector is noise rather than signal. A long document and a short one on the
same topic point in a similar direction but have very different lengths.
Euclidean distance says the long one is farther from the short one than an
unrelated document is, while cosine distance says they are almost the same.
Text, TF-IDF vectors and embeddings are the standard cases. A useful fact: if
you first scale every row to length 1 (L2 normalisation), Euclidean distance
gives the same ranking as cosine distance. So normalising the rows and keeping
the default metric expresses the same idea more clearly.
Mahalanobis distance takes correlation between features into account. Euclidean distance treats every axis as independent, so when two features are highly correlated, it counts their shared direction twice. Mahalanobis distance divides out the covariance matrix. This is the same as "whitening" the data (removing the correlations) and then using Euclidean distance. The effect is concrete: in a tightly correlated cloud of points, two points at the same Euclidean distance from the centre can be very differently unusual, depending on whether they lie along the correlation or across it.
Mixed data types need a metric that can handle them. You cannot subtract
"Germany" from "Brazil". One-hot encoding plus Euclidean distance quietly says
that every category mismatch costs sqrt(2), a number nobody chose and one
that changes as you add levels. Gower distance scales each numeric column
by its range, scores each categorical column as 0 (same) or 1 (different), and
averages them, so every column gets equal weight by design.
The rule behind all of this: the metric states what you believe "similar" means. Treat it as a hyperparameter to cross-validate, not a default to accept.
import numpy as np
import pandas as pd
from scipy.spatial.distance import cdist
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(37)
# --- THE SAME TWO POINTS, FIVE METRICS, FIVE DIFFERENT ANSWERS. ---
a = np.array([1.0, 2.0, 3.0, 0.0])
b = np.array([4.0, 0.0, 3.0, 1.0])
print("a =", a, "\nb =", b)
print(f" euclidean (L2) {np.sqrt(((a-b)**2).sum()):.4f}"
" straight line; dominated by the largest gap")
print(f" manhattan (L1) {np.abs(a-b).sum():.4f}"
" sum of per-axis gaps; robust to one big one")
print(f" chebyshev (Linf) {np.abs(a-b).max():.4f}"
" the single worst axis")
cos = 1 - a @ b / (np.linalg.norm(a) * np.linalg.norm(b))
print(f" cosine distance {cos:.4f}"
" angle only; magnitude is discarded entirely")
# --- WHY COSINE DOMINATES TEXT AND EMBEDDINGS: length should not matter. ---
short_doc = np.array([2.0, 1.0, 0.0, 3.0]) # same topic mix...
long_doc = short_doc * 10 # ...ten times as long
other = np.array([0.0, 3.0, 4.0, 0.0])
print("\nA SHORT AND A LONG DOCUMENT ON THE SAME TOPIC")
for name, v in [("long version of the same doc", long_doc),
("a genuinely different doc", other)]:
e = np.sqrt(((short_doc - v) ** 2).sum())
c = 1 - short_doc @ v / (np.linalg.norm(short_doc) * np.linalg.norm(v))
print(f" {name:30s} euclidean {e:7.3f} cosine {c:.4f}")
print(" Euclidean calls the long version FARTHER than an unrelated document.")
print(" Cosine calls it identical, which is the right answer for topic.")
# --- MINKOWSKI p INTERPOLATES, AND p CHANGES WHICH ERRORS DOMINATE. ---
print("\nMINKOWSKI DISTANCE AS p VARIES (one big gap vs many small ones)")
spread = np.array([1.0, 1.0, 1.0, 1.0, 1.0]) # five small differences
spike = np.array([2.3, 0.0, 0.0, 0.0, 0.0]) # one large difference
print(f" {'p':>5s} {'five gaps of 1':>15s} {'one gap of 2.3':>16s} verdict")
for p in [1, 2, 3, 10, np.inf]:
ds = np.linalg.norm(spread, p)
dk = np.linalg.norm(spike, p)
v = "spread is farther" if ds > dk else "spike is farther"
plabel = "inf" if p == np.inf else str(p)
print(f" {plabel:>5s} {ds:15.4f} {dk:16.4f} {v}")
print(" Small p spreads blame across axes; large p is dominated by the worst")
print(" axis. Your choice of p encodes which kind of difference you consider")
print(" more significant, which is a modelling decision, not a default.")
# --- MAHALANOBIS: the metric that accounts for CORRELATION. ---
# When features are correlated, Euclidean distance double-counts the shared
# direction. Mahalanobis divides out the covariance first.
cov = np.array([[1.0, 0.92], [0.92, 1.0]])
L = np.linalg.cholesky(cov)
pts = rng.normal(0, 1, (800, 2)) @ L.T # a tight diagonal cloud
centre = np.zeros(2)
inv = np.linalg.inv(np.cov(pts.T))
cand = np.array([[2.0, 2.0], # ALONG the correlation -> ordinary
[2.0, -2.0]]) # ACROSS it -> genuinely unusual
print("\nMAHALANOBIS a 0.92-correlated cloud, two equally-Euclidean points")
for pt in cand:
e = np.sqrt((pt ** 2).sum())
mh = float(np.sqrt(pt @ inv @ pt))
# empirical check: how many cloud points are at least this extreme?
md_all = np.sqrt(np.einsum("ij,jk,ik->i", pts, inv, pts))
print(f" point {str(pt):12s} euclidean {e:.4f} mahalanobis {mh:6.3f}"
f" more extreme than {np.mean(md_all < mh):6.1%} of the cloud")
print(" Identical Euclidean distance, very different meaning. Mahalanobis is")
print(" Euclidean distance after whitening, which is also exactly what")
print(" standardising does when the features are uncorrelated.")
# --- MIXED TYPES: you cannot average a metre and a category. ---
df = pd.DataFrame({
"age": [25, 40, 41, 62],
"plan": ["free", "pro", "pro", "free"],
"region": ["eu", "us", "eu", "us"],
})
def gower(frame):
"""Gower distance: per-column normalised distance, then averaged.
Numeric columns use |a-b| / range; categorical columns use 0 or 1."""
n = len(frame)
D = np.zeros((n, n))
for col in frame.columns:
v = frame[col]
if pd.api.types.is_numeric_dtype(v):
rng_ = v.max() - v.min()
d = np.abs(v.to_numpy()[:, None] - v.to_numpy()[None, :]) / rng_
else:
d = (v.to_numpy()[:, None] != v.to_numpy()[None, :]).astype(float)
D += d
return D / len(frame.columns)
print("\nGOWER DISTANCE ON MIXED TYPES")
print(df.to_string(index=False))
print("\n", pd.DataFrame(np.round(gower(df), 3)).to_string())
print("Rows 1 and 2 (ages 40 and 41, both pro, different region) are closest.")
print("One-hot encoding plus Euclidean distance would instead make the region")
print("mismatch cost sqrt(2), an arbitrary number nobody chose deliberately.")
# --- CHOOSING A METRIC IS A TUNING DECISION, WITH CONSEQUENCES. ---
n = 2500
# Sparse, high-dimensional, count-like data whose MAGNITUDE varies a lot:
# the shape of text retrieval, where document length is a nuisance.
V = rng.poisson(0.5, (n, 60)).astype(float)
topic = rng.integers(0, 3, n)
for t in range(3):
V[topic == t, t * 20:(t + 1) * 20] += rng.poisson(0.3,
((topic == t).sum(), 20))
length = rng.gamma(1.0, 8.0, n)[:, None] # documents vary in length
V = V * length # magnitude now dominates
Vtr, Vte, ttr, tte = train_test_split(V, topic, test_size=0.3, random_state=0,
stratify=topic)
print("\nSPARSE COUNT DATA WITH VARYING MAGNITUDE: the metric decides")
for metric in ["euclidean", "manhattan", "chebyshev", "cosine"]:
m = KNeighborsClassifier(15, metric=metric).fit(Vtr, ttr)
print(f" {metric:12s} test accuracy {m.score(Vte, tte):.4f}")
# L2-normalising the rows makes Euclidean distance a monotone function of
# cosine distance, so the two become the same ranking.
Vn = V / np.linalg.norm(V, axis=1, keepdims=True)
Ntr, Nte, ntr, nte = train_test_split(Vn, topic, test_size=0.3, random_state=0,
stratify=topic)
m = KNeighborsClassifier(15).fit(Ntr, ntr)
print(f" {'euclidean':12s} after L2-normalising rows: {m.score(Nte, nte):.4f}")
print(" Cosine wins because it discards the length nuisance. Normalising the")
print(" rows first gets you the same thing with the default metric, which is")
print(" usually the cleaner way to express the intent in a pipeline.")
THE SAME TWO POINTS, FOUR METRICS
a = [1, 2, 3, 0] b = [4, 0, 3, 1]
euclidean (L2) 3.7417 straight line
manhattan (L1) 6.0000 sum of per-axis gaps
chebyshev (Linf) 3.0000 the single worst axis
cosine 0.3186 angle only, magnitude discarded
L1 vs L2 vs Linf, as unit balls:
L1: a diamond L2: a circle Linf: a square
/\ ___ +----+
/ \ / \ | |
\ / \___/ +----+
\/
WHY COSINE DOMINATES TEXT AND EMBEDDINGS
short_doc = [2, 1, 0, 3]
long_doc = short_doc * 10 same topic mix, ten times as long
other = [0, 3, 4, 0] a genuinely different document
euclidean cosine
long version of the same doc 33.675 0.0000
a genuinely different doc 5.745 0.8396
^^^^^^
Euclidean calls the long version FARTHER than an unrelated document.
MINKOWSKI p DECIDES WHICH DIFFERENCES MATTER
p five gaps of 1 one gap of 2.3 verdict
1 5.0000 2.3000 spread is farther
2 2.2361 2.3000 spike is farther
3 1.7100 2.3000 spike is farther
10 1.1746 2.3000 spike is farther
inf 1.0000 2.3000 spike is farther
MAHALANOBIS: DISTANCE THAT KNOWS ABOUT CORRELATION
(a 0.92-correlated cloud; both points are 2.83 away in Euclidean terms)
point [ 2, 2] mahalanobis 2.025 more extreme than 87.0% of cloud
point [ 2, -2] mahalanobis 9.903 more extreme than 100.0% of cloud
x2 | . ::. [2,2] lies ALONG the cloud:
| .:::::. ordinary.
| .::::. * [2,2]
--+--:::.--------------- x1 [2,-2] lies ACROSS it:
| ::. genuinely unusual, though
| * [2,-2] equally far in Euclidean terms.
GOWER DISTANCE ON MIXED TYPES
age plan region
25 free eu
40 pro us
41 pro eu
62 free us
0 1 2 3
0 0.000 0.802 0.477 0.667
1 0.802 0.000 0.342 0.532 rows 1 and 2 are closest:
2 0.477 0.342 0.000 0.856 ages 40 and 41, both pro
3 0.667 0.532 0.856 0.000
Each column contributes at most 1, by construction. One-hot plus
Euclidean would instead price a region mismatch at sqrt(2).
THE METRIC CHANGES THE ANSWER
(sparse count data whose magnitude varies a lot)
euclidean 0.6987
manhattan 0.6187
chebyshev 0.5693
cosine 0.7387 <- length is a nuisance here
euclidean after L2-normalising 0.7400 <- the same idea, stated better
| Metric | Formula | Ignores size | Handles correlation | Typical use |
|---|---|---|---|---|
| Euclidean | sqrt(sum((a-b)^2)) | No | No | Geometric data, comparable units |
| Manhattan | sum(abs(a-b)) | No | No | Outliers, many dimensions |
| Chebyshev | max(abs(a-b)) | No | No | Any one large difference rules out a match |
| Cosine | 1 - a.b/(norm a * norm b) | Yes | No | Text, TF-IDF, embeddings |
| Mahalanobis | sqrt(d' S^-1 d) | No | Yes | Correlated features, outlier scoring |
| Gower | Per column, scaled, averaged | Per column | No | Mixed numeric and categorical |
Key Takeaways
- The distance metric is an assumption about what "similar" means, not a default to accept.
- Euclidean distance is dominated by the largest single difference. Manhattan spreads blame evenly. Chebyshev looks only at the worst axis.
- Minkowski
pmoves between them, and choosing it says which kind of difference matters more.- Cosine distance ignores vector length, which is right whenever length is noise, as with documents and embeddings.
- After scaling rows to length 1, Euclidean distance ranks points the same way as cosine distance, so normalising is often the clearer approach.
- Mahalanobis distance removes correlation first, so it correctly treats a point across a correlated cloud as more unusual than one along it.
- One-hot encoding plus Euclidean distance quietly prices every category mismatch at
sqrt(2). Gower distance gives each column equal weight on purpose.
🧪 Practice
- Compute Euclidean, Manhattan, Chebyshev and cosine distance for the same pair of vectors, and explain why the rankings differ.
- Multiply one vector by ten and report which metrics change.
- Vary Minkowski
pfrom 1 to 20 for a pair that differs a lot on one axis and a pair that differs a little on many axes. Find thepwhere their order flips.- Generate a strongly correlated two-dimensional cloud. Compare Euclidean and Mahalanobis distance for a point along the correlation and a point across it.
- Implement Gower distance for a table with numeric and categorical columns, and compare its nearest-neighbour rankings with one-hot plus Euclidean.
- Cross-validate
metricas a k-NN hyperparameter on sparse count data and report how much accuracy the choice is worth.- Interview: Your recommender uses Euclidean distance on user-item interaction counts, and heavy users always come back as everyone's nearest neighbour. What is happening and what would you change? (Hint: how does a heavy user's vector compare with others in length versus direction?)
Support Vector Machines
Take two classes that a straight line can separate perfectly. There are infinitely many such lines. Logistic regression picks one based on its likelihood, with no special preference for a line that sits comfortably between the classes over one that just grazes a point. A support vector machine (SVM) does have a preference: it picks the line with the widest margin, meaning the biggest possible gap between the line and the nearest point of either class.
Why is that a good idea? Robustness. A boundary that only just clears a training point will put a similar future point on the wrong side. A boundary in the middle of the gap leaves the most room for error on both sides. There is also a formal theory of why the margin helps, but the geometric picture is the useful one.
This leads to the method's defining feature: only the points on the margin decide the answer. These are the support vectors. You could delete every other point and the fitted model would not change at all. In the example below, 4 rows out of 200 define the whole model.
Why does this happen? Because of the loss function. An SVM minimises hinge
loss, max(0, 1 - y*f(x)), which is exactly zero for any point correctly
classified and outside the margin. A point with zero loss has zero gradient,
so it has no influence. Compare log loss, which gets small but never reaches
zero, so logistic regression always uses every row. The trade is direct: hinge
loss gives you a sparse model, but you lose calibrated probabilities, because
no likelihood is being maximised. predict_proba on an SVM comes from a
separate logistic fit on the SVM's outputs (Platt scaling), not from the SVM
itself.
Real data is rarely perfectly separable, so the soft margin allows some
points on the wrong side, at a price. C is that price:
- Small
C: mistakes are cheap, so the model prefers a wide margin and accepts some misclassified points. Strong regularisation, many support vectors. - Large
C: mistakes are expensive, so the margin narrows and bends to fit outliers. Weak regularisation, few support vectors.
Note the direction: C is the inverse of regularisation strength. That is
the opposite of alpha in ridge and lasso, and a frequent source of confusion.
Two practical points. You must scale the features, for the same reason as
k-NN: the margin and the RBF kernel both depend on distance, so an unscaled
column dominates them. And kernel SVMs do not scale to large data: fitting
time grows between n² and n³ with the number of rows, because the algorithm
works with a matrix of all pairs of rows. Beyond about a hundred thousand rows,
a kernel SVM is impractical. That is the main reason gradient-boosted trees
replaced SVMs as the default for tabular data. LinearSVC handles the linear
case directly and scales to millions of rows.
SVMs are still a strong choice in one situation: a moderate number of rows and many features. There the margin's robustness pays off and the cost is still affordable. Text classification, with tens of thousands of features and a few thousand documents, is the classic example.
import numpy as np
import pandas as pd
from sklearn.svm import SVC, LinearSVC, SVR
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_blobs, make_classification
rng = np.random.default_rng(41)
# --- THE IDEA: of all separating lines, prefer the one with the widest MARGIN.
X, y = make_blobs(n_samples=200, centers=2, cluster_std=1.1, random_state=6)
svm = make_pipeline(StandardScaler(), SVC(kernel="linear", C=1e6)).fit(X, y)
sv = svm[-1].support_vectors_
w = svm[-1].coef_[0]
margin = 2.0 / np.linalg.norm(w)
print("MAXIMUM MARGIN ON SEPARABLE DATA")
print(f" training rows {len(X)}")
print(f" support vectors {len(sv)} ({len(sv)/len(X):.1%} of the data)")
print(f" margin width {margin:.4f} (= 2 / ||w||)")
print(" Only the support vectors matter: delete every other row and refit.")
mask = np.zeros(len(X), bool)
mask[svm[-1].support_] = True
svm2 = make_pipeline(StandardScaler(),
SVC(kernel="linear", C=1e6)).fit(X[mask], y[mask])
print(f" refit on support vectors ONLY: predictions identical on"
f" {np.mean(svm.predict(X) == svm2.predict(X)):.1%} of rows")
# --- C IS THE SOFT-MARGIN DIAL: how much violation you will tolerate. ---
Xo, yo = make_classification(n_samples=600, n_features=2, n_redundant=0,
n_informative=2, n_clusters_per_class=1,
class_sep=0.9, flip_y=0.08, random_state=3)
Atr, Ate, btr, bte = train_test_split(Xo, yo, test_size=0.3, random_state=0,
stratify=yo)
print(f"\n{'C':>8s} {'support vecs':>13s} {'margin':>8s} {'train':>7s} {'test':>7s}")
for C in [0.01, 0.1, 1.0, 10.0, 1000.0]:
m = make_pipeline(StandardScaler(), SVC(kernel="linear", C=C)).fit(Atr, btr)
wv = m[-1].coef_[0]
print(f"{C:8.2f} {len(m[-1].support_):13d} {2/np.linalg.norm(wv):8.4f}"
f" {m.score(Atr, btr):7.4f} {m.score(Ate, bte):7.4f}")
print("Small C: a wide margin, many violations tolerated, many support vectors.")
print("Large C: a narrow margin that bends to fit outliers. C is INVERSE")
print("regularisation strength -- large C means LESS regularisation.")
# --- HINGE LOSS IS WHAT MAKES AN SVM AN SVM. ---
# It is exactly zero for points correctly classified beyond the margin, which is
# why most rows have no influence at all on the solution.
print("\nHINGE LOSS max(0, 1 - y*f(x)) vs LOG LOSS")
print(f" {'y*f(x)':>8s} {'hinge':>7s} {'log loss':>9s} reading")
for margin_val in [-2.0, -0.5, 0.0, 0.5, 1.0, 2.0, 5.0]:
hinge = max(0.0, 1 - margin_val)
ll = np.log1p(np.exp(-margin_val))
note = ("inside the margin or wrong" if margin_val < 1
else "outside the margin: ZERO influence")
print(f" {margin_val:8.1f} {hinge:7.3f} {ll:9.3f} {note}")
print(" Log loss is never exactly zero, so logistic regression uses EVERY row.")
print(" Hinge loss is, so an SVM's solution depends only on the support vectors.")
print(" That also means an SVM gives you no calibrated probability: there is no")
print(" likelihood being maximised.")
# --- SVM VS LOGISTIC REGRESSION, ON THE SAME DATA. ---
print(f"\n{'model':34s} {'test acc':>9s} {'probabilities':>14s}")
for label, est in [
("logistic regression", LogisticRegression(max_iter=2000)),
("linear SVM (hinge loss)", SVC(kernel="linear", C=1.0)),
("RBF SVM", SVC(kernel="rbf", C=1.0, gamma="scale")),
]:
m = make_pipeline(StandardScaler(), est).fit(Atr, btr)
has_proba = hasattr(est, "predict_proba") and not isinstance(est, SVC)
print(f" {label:32s} {m.score(Ate, bte):9.4f}"
f" {'yes, calibrated' if has_proba else 'no, needs Platt':>14s}")
# --- SCALING IS MANDATORY, FOR THE SAME REASON AS k-NN. ---
df = pd.DataFrame({"a": rng.normal(0, 1, 800),
"b": rng.normal(0, 1, 800) * 1000})
lbl = ((df["a"] + df["b"] / 1000) > 0).astype(int)
Ctr, Cte, dtr, dte = train_test_split(df, lbl, test_size=0.3, random_state=0)
raw = SVC(kernel="rbf").fit(Ctr, dtr)
sc = make_pipeline(StandardScaler(), SVC(kernel="rbf")).fit(Ctr, dtr)
print(f"\nSCALING column b is 1000x the scale of column a")
print(f" unscaled RBF SVM test accuracy {raw.score(Cte, dte):.4f}"
f" ({len(raw.support_)} support vectors)")
print(f" scaled RBF SVM test accuracy {sc.score(Cte, dte):.4f}"
f" ({len(sc[-1].support_)} support vectors)")
print(" The RBF kernel is a function of squared distance, so an unscaled column")
print(" swamps it exactly as it swamps k-NN.")
# --- THE COST: SVMs DO NOT SCALE TO LARGE n. ---
import time
print("\nFIT TIME GROWS SUPERLINEARLY IN n (kernel SVMs are O(n^2) to O(n^3))")
for n in [1000, 4000, 16_000]:
Zn = rng.normal(0, 1, (n, 20))
tn = (Zn[:, 0] + Zn[:, 1] ** 2 + rng.normal(0, 0.5, n) > 1).astype(int)
t0 = time.perf_counter(); SVC(kernel="rbf").fit(Zn, tn)
kt = time.perf_counter() - t0
t0 = time.perf_counter(); LinearSVC(max_iter=5000, dual="auto").fit(Zn, tn)
lt = time.perf_counter() - t0
print(f" n={n:6,d}: kernel SVM {kt:7.3f}s LinearSVC {lt:7.3f}s"
f" ratio {kt/lt:6.1f}x")
print(" Past roughly 100,000 rows a kernel SVM becomes impractical, which is")
print(" the main reason boosted trees displaced it for tabular problems.")
THE MAXIMUM MARGIN
class A o o both lines separate perfectly,
o o \ | but only one is centred in the gap
o o \ |
------------------ | <- SVM: the widest margin
x x \|
x x x \
class B x x \ <- a line that barely clears a point
training rows 200
support vectors 4 (2.0% of the data)
margin width 0.5418 (= 2 / ||w||)
refit on the 4 support vectors ONLY: identical on 100.0% of rows
C IS THE SOFT-MARGIN PRICE (and INVERSE regularisation strength)
C support vecs margin train test
0.01 276 1.9872 0.8881 0.8556
0.10 172 1.2534 0.8786 0.8722
1.00 147 1.0224 0.8786 0.8722
10.00 144 0.9977 0.8810 0.8722
1000.00 143 0.9899 0.8810 0.8722
small C -> wide margin, violations tolerated, MANY support vectors
large C -> narrow margin that bends to outliers, FEW support vectors
HINGE LOSS IS WHY ONLY SOME POINTS MATTER
y*f(x) hinge log loss reading
-2.0 3.000 2.127 wrong side, heavily penalised
0.0 1.000 0.693 on the boundary
0.5 0.500 0.474 inside the margin
1.0 0.000 0.313 ON the margin
2.0 0.000 0.127 outside: ZERO influence
5.0 0.000 0.007 outside: still zero, log loss still isn't
loss
|\ hinge log loss never reaches zero, so
| \ logistic regression uses EVERY
| \ row. hinge loss does, so an SVM
| \____________ 0 depends only on support vectors.
+----+-----------------> y*f(x)
1
SVM VS LOGISTIC REGRESSION
model test acc probabilities
logistic regression 0.8667 yes, calibrated
linear SVM (hinge loss) 0.8722 no, needs Platt scaling
RBF SVM 0.8722 no, needs Platt scaling
SCALING IS MANDATORY (column b is 1000x the scale of column a)
unscaled RBF SVM 0.7375 (330 support vectors)
scaled RBF SVM 0.9875 ( 97 support vectors)
KERNEL SVMs DO NOT SCALE
n= 1,000: kernel SVM 0.031s LinearSVC 0.003s ratio 10.6x
n= 4,000: kernel SVM 0.404s LinearSVC 0.007s ratio 59.5x
n=16,000: kernel SVM 5.899s LinearSVC 0.026s ratio 226.1x
4x the rows -> roughly 13x the time. This is the reason boosted
trees replaced SVMs as the tabular default.
Key Takeaways
- An SVM picks the separating boundary with the widest margin, which leaves the most room for error on both sides.
- Only the support vectors decide the answer. Every other training row could be deleted without changing the model.
- This comes from hinge loss being exactly zero outside the margin. Log loss is never zero, so logistic regression uses every row.
- The same property means no calibrated probabilities: no likelihood is maximised, and
predict_probais a separate Platt fit.Cis the price of margin violations and the inverse of regularisation strength, the opposite of ridge'salpha.- Features must be scaled, because both the margin and the RBF kernel depend on distance.
- Kernel SVM fitting time grows between n² and n³, which rules them out beyond about a hundred thousand rows.
LinearSVCscales much further.- SVMs still shine with a moderate number of rows and many features, such as in text classification.
🧪 Practice
- Fit a linear SVM on separable data, refit using only the support vectors, and check that the predictions are identical.
- Vary
Cover six orders of magnitude and tabulate the margin width, the number of support vectors, and training and test accuracy.- Plot hinge loss and log loss against the margin
y*f(x)and mark where hinge loss reaches zero.- Compare an SVM's
predict_proba(withprobability=True) with logistic regression's on a reliability curve. Which one is calibrated?- Fit an RBF SVM with and without standardisation on data with one column multiplied by 1000. Report accuracy and the number of support vectors.
- Time
SVCagainstLinearSVCat 1000, 10,000 and 50,000 rows and estimate how fast the time grows withn.- Interview: Your SVM has 4800 support vectors out of 5000 training rows. What does that tell you, and what would you change? (Hint: what value of
Cpushes almost every point inside the margin, and what does that say about how well the current boundary separates anything?)
Kernel Trick and Kernel Choice
Two circles, one inside the other, cannot be separated by a straight line. The inner disc is one class, the ring around it is the other, and no line can divide them. A linear SVM scores barely better than guessing.
Now change the coordinates. Map each point (x1, x2) to
(x1^2, x2^2, sqrt(2)*x1*x2). The circle x1^2 + x2^2 = r^2 becomes the plane
u + v = r^2 in the new coordinates, and a plane is linear. The data could
always be separated; just not in the coordinates you started with. Fit a linear
SVM in the new space and accuracy jumps from 57% to 99%.
The obvious problem is cost. Explicit feature maps get huge. The number of
degree-d terms built from p features is comb(d+p-1, d). For degree 5 and
100 features that is about 92 million columns; for degree 10 and 1000 features
it is more than 10^23. You cannot build such matrices.
Here is the trick, one of the most elegant ideas in machine learning. The SVM
never needs the new coordinates themselves, only the dot products between
mapped points. For many maps, that dot product can be computed directly from
the original points. For the map above, phi(a).phi(b) = (a.b)^2 exactly: two
multiplications instead of building three new coordinates. In the degree-5
case, it is one dot product instead of 92 million features. A function
K(a,b) that equals a dot product in some feature space is called a
kernel, and replacing dot products with a kernel is the kernel trick.
It works for any algorithm that can be written using only dot products between rows. That includes SVMs, ridge regression, PCA and k-means.
The standard kernels and what each one assumes:
| Kernel | Formula | Hidden feature space | Assumes |
|---|---|---|---|
| Linear | a.b | The original features | The boundary is already linear |
| Polynomial | (gamma*a.b + coef0)^d | All products up to degree d | Interactions up to order d |
| RBF / Gaussian | exp(-gamma*‖a-b‖^2) | Infinite-dimensional | A smooth boundary that varies locally |
| Sigmoid | tanh(gamma*a.b + coef0) | Not always a valid space | Mostly historical; rarely a good choice |
Some parameters need special attention.
gamma in the RBF kernel sets how far each training point reaches. The
kernel is exp(-gamma * squared distance). A large gamma makes it fall off
quickly, so each training point only affects its immediate surroundings. Set it
high enough and every point only explains itself: training accuracy gets close
to 1.0 while test accuracy collapses. Near-perfect training accuracy, many
support vectors and falling test accuracy together are the clear sign that
gamma is too high.
coef0 in the polynomial kernel matters. With coef0 = 0, the kernel only
contains terms of exactly degree d, with no linear or constant part. On data
that needs a linear part, a degree-2 kernel like this can score worse than
guessing, as the example shows. Setting coef0 = 1 adds all the lower-degree
terms back.
C and gamma affect each other, so tune them together. The good settings
form a diagonal band in the (C, gamma) plane (a bigger C can partly make up
for a smaller gamma), and tuning one at a time will miss it. Use a
two-dimensional grid with log-spaced values.
Finally, what makes a function a valid kernel? Mercer's condition: the matrix of kernel values for any set of points (the Gram matrix) must be positive semi-definite. This guarantees that a hidden feature space really exists and that the optimisation has one best answer (it stays convex). The sigmoid kernel breaks this condition for many settings, which is the concrete reason to avoid it. Valid kernels can also be added and multiplied to make new valid kernels, so you can combine them to build models.
import numpy as np
import pandas as pd
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler, PolynomialFeatures
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.datasets import make_circles, make_moons
rng = np.random.default_rng(43)
# --- THE PROBLEM: a circular boundary is not linearly separable in (x1, x2). ---
X, y = make_circles(n_samples=600, noise=0.10, factor=0.45, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
stratify=y)
lin = make_pipeline(StandardScaler(), SVC(kernel="linear")).fit(Xtr, ytr)
print(f"linear SVM on concentric circles: test accuracy {lin.score(Xte, yte):.4f}")
print("A straight line cannot separate an inner disc from an outer ring.")
# --- THE EXPLICIT FIX: map into a space where it IS linear. ---
def phi(Z):
"""A hand-built feature map: (x1, x2) -> (x1^2, x2^2, sqrt(2)*x1*x2).
This is the degree-2 polynomial map, written out."""
return np.column_stack([Z[:, 0] ** 2, Z[:, 1] ** 2,
np.sqrt(2) * Z[:, 0] * Z[:, 1]])
explicit = SVC(kernel="linear").fit(phi(Xtr), ytr)
print(f"\nafter mapping to (x1^2, x2^2, sqrt2*x1*x2): test accuracy"
f" {explicit.score(phi(Xte), yte):.4f}")
print("The circle x1^2 + x2^2 = r^2 is a straight line in the FIRST TWO of")
print("those new coordinates. Same model, different space.")
# --- THE TRICK: you never need the map, only the inner products. ---
# For the map above, phi(a).phi(b) = (a.b)^2 exactly. Verify it.
a, b = Xtr[0], Xtr[1]
print(f"\nphi(a) . phi(b) = {phi(a[None]) @ phi(b[None]).T}")
print(f"(a . b)^2 = {(a @ b) ** 2:.10f}")
from math import comb
print("Identical. Now count what the explicit map would cost: the number of")
print("degree-d monomials over p features is comb(d + p - 1, d).")
print(f" {'degree':>6s} {'features':>9s} {'explicit map size':>26s}")
for d, p_ in [(2, 2), (2, 100), (5, 100), (10, 1000)]:
print(f" {d:6d} {p_:9d} {comb(d + p_ - 1, d):26,d}")
print("Every row above costs ONE dot product as a kernel. That is the trick.")
# Verify the equivalence across the whole matrix, then confirm the SVM agrees.
K_explicit = phi(Xtr) @ phi(Xtr).T
K_kernel = (Xtr @ Xtr.T) ** 2
print(f"max |K_explicit - K_kernel| = {np.abs(K_explicit - K_kernel).max():.2e}")
kern = SVC(kernel="poly", degree=2, gamma=1.0, coef0=0.0).fit(Xtr, ytr)
print(f"SVC(kernel='poly', degree=2) test accuracy {kern.score(Xte, yte):.4f}")
# --- THE COMMON KERNELS, AND WHAT EACH ASSUMES. ---
Xm, ym = make_moons(n_samples=900, noise=0.22, random_state=1)
Mtr, Mte, ntr, nte = train_test_split(Xm, ym, test_size=0.3, random_state=0,
stratify=ym)
print(f"\n{'kernel':30s} {'test acc':>9s} {'support vecs':>13s}")
for label, kw in [
("linear K = a.b", dict(kernel="linear", C=1.0)),
("poly d=2, coef0=0", dict(kernel="poly", degree=2, coef0=0.0, C=1.0)),
("poly d=2, coef0=1", dict(kernel="poly", degree=2, coef0=1.0, C=1.0)),
("poly d=5, coef0=1", dict(kernel="poly", degree=5, coef0=1.0, C=1.0)),
("rbf K = exp(-g|a-b|^2)", dict(kernel="rbf", C=1.0, gamma="scale")),
("sigmoid K = tanh(g a.b + c)", dict(kernel="sigmoid", C=1.0)),
]:
m = make_pipeline(StandardScaler(), SVC(**kw)).fit(Mtr, ntr)
print(f" {label:28s} {m.score(Mte, nte):9.4f} {len(m[-1].support_):13d}")
print(" coef0=0 gives a HOMOGENEOUS kernel: only pure degree-d terms, with no")
print(" linear or constant part, which is why it collapses here. coef0=1 adds")
print(" all the lower-degree terms back. It is not a cosmetic default.")
# --- GAMMA IN THE RBF KERNEL IS THE REACH OF EACH TRAINING POINT. ---
# K(a,b) = exp(-gamma * |a-b|^2). Large gamma -> the kernel decays fast ->
# each point influences only its immediate surroundings -> overfitting.
print("\nRBF GAMMA: how far a single training point's influence reaches")
print(f" {'gamma':>8s} {'K at d=0.5':>11s} {'K at d=2':>9s} {'train':>7s}"
f" {'test':>7s} {'SVs':>5s}")
for g in [0.01, 0.1, 1.0, 10.0, 100.0]:
m = make_pipeline(StandardScaler(), SVC(kernel="rbf", gamma=g, C=1.0)
).fit(Mtr, ntr)
print(f" {g:8.2f} {np.exp(-g * 0.25):11.4f} {np.exp(-g * 4):9.2e}"
f" {m.score(Mtr, ntr):7.4f} {m.score(Mte, nte):7.4f}"
f" {len(m[-1].support_):5d}")
print(" gamma=100: K at distance 2 is 1e-174, so each point only explains")
print(" itself. Training accuracy near 1.0 and test accuracy collapsing is")
print(" the unmistakable signature of gamma set too high.")
# --- C AND GAMMA INTERACT, SO TUNE THEM TOGETHER ON A GRID. ---
grid = GridSearchCV(make_pipeline(StandardScaler(), SVC(kernel="rbf")),
{"svc__C": [0.1, 1, 10, 100],
"svc__gamma": [0.01, 0.1, 1.0, 10.0]},
cv=5, n_jobs=-1).fit(Mtr, ntr)
res = pd.DataFrame(grid.cv_results_)
pivot = res.pivot_table(index="param_svc__gamma", columns="param_svc__C",
values="mean_test_score")
print("\nCROSS-VALIDATED ACCURACY: rows are gamma, columns are C")
print(pivot.round(4).to_string())
print(f"best {grid.best_params_} cv {grid.best_score_:.4f}"
f" test {grid.score(Mte, nte):.4f}")
print("The good region is a diagonal band, not a point: a larger C can")
print("compensate for a smaller gamma. Searching one at a time misses it.")
# --- WHAT MAKES A VALID KERNEL. ---
# Mercer's condition: the Gram matrix must be positive semi-definite for every
# finite sample, which is what guarantees an implicit feature space exists.
print("\nMERCER'S CONDITION: the Gram matrix must be positive semi-definite")
S = Xtr[:200]
for name, K in [("rbf", np.exp(-1.0 * ((S[:, None] - S[None]) ** 2).sum(-1))),
("linear", S @ S.T),
("poly d=3", (1 + S @ S.T) ** 3),
("sigmoid (NOT always valid)", np.tanh(1.0 * S @ S.T + 1.0))]:
ev = np.linalg.eigvalsh((K + K.T) / 2)
print(f" {name:28s} min eigenvalue {ev.min():+.4e}"
f" {'valid' if ev.min() > -1e-8 else 'INVALID here'}")
print(" The sigmoid kernel has negative eigenvalues, so no feature space")
print(" corresponds to it and the optimisation is no longer convex. It is")
print(" kept for historical reasons; there is rarely a reason to choose it.")
THE PROBLEM AND THE MAP
in (x1, x2): no line separates in (x1^2, x2^2): a line does
o o o o o x2^2 |
o x x o | o o o o outer ring
o x x o |
o x x o | x x inner disc
o o o o o +----------- x1^2
linear SVM on the circles test accuracy 0.5667
linear SVM after the map test accuracy 0.9889
THE TRICK: YOU NEVER NEED THE MAP
phi(a) . phi(b) = 0.0573861946
(a . b)^2 = 0.0573861946 identical, to 16 digits
max |K_explicit - K_kernel| = 8.88e-16
degree features explicit map size
2 2 3
2 100 5,050
5 100 91,962,520
10 1000 288,216,356,245,328,994,082,600
Every row above costs ONE dot product as a kernel.
KERNELS ON THE SAME DATA
kernel test acc support vecs
linear K = a.b 0.8815 210
poly d=2, coef0=0 0.4593 625 <- see below
poly d=2, coef0=1 0.8852 211
poly d=5, coef0=1 0.9593 95
rbf K = exp(-g|a-b|^2) 0.9556 138
sigmoid K = tanh(g a.b + c) 0.7259 230
coef0=0 is a HOMOGENEOUS kernel: pure degree-d terms only, with no
linear or constant part, which is why it collapses here.
GAMMA IS THE REACH OF A TRAINING POINT
gamma K at d=0.5 K at d=2 train test SVs
0.01 0.9975 9.61e-01 0.8587 0.8926 288 too smooth
0.10 0.9753 6.70e-01 0.8889 0.9037 222
1.00 0.7788 1.83e-02 0.9524 0.9519 113 about right
10.00 0.0821 4.25e-18 0.9635 0.9444 232
100.00 0.0000 1.92e-174 0.9841 0.9333 548 memorising
K(d)
1 |* small gamma: a point's influence
| \___ reaches far -> smooth boundary
| \______ small gamma
0 |*_____________________ large gamma: influence dies within
+-------------------> d a hair's breadth -> memorisation
C AND GAMMA INTERACT: TUNE THEM TOGETHER
(cross-validated accuracy; rows gamma, columns C)
C=0.1 C=1 C=10 C=100
g=0.01 0.8429 0.8556 0.8651 0.8683
g=0.10 0.8603 0.8794 0.9222 0.9429
g=1.00 0.9413 0.9476 0.9524 0.9556
g=10.0 0.9460 0.9524 0.9444 0.9317
The good region is a DIAGONAL BAND, not a point: larger C partly
compensates for smaller gamma. Tuning one at a time misses it.
best C=100, gamma=1.0, cv 0.9556, test 0.9556
MERCER'S CONDITION: THE GRAM MATRIX MUST BE POSITIVE SEMI-DEFINITE
rbf min eigenvalue -6.05e-15 valid
linear min eigenvalue -3.22e-14 valid
poly d=3 min eigenvalue -1.08e-13 valid
sigmoid min eigenvalue -7.77e+00 INVALID
A negative eigenvalue means no feature space corresponds to the
kernel and the optimisation is no longer convex.
Key Takeaways
- Data that a straight line cannot separate in its original coordinates can often be separated after a feature map. The problem is the coordinates, not the model.
- Explicit polynomial feature maps grow so fast that they cannot be built beyond small degrees.
- Kernel methods only need dot products between mapped points, and for many maps these can be computed directly from the original points.
- Any algorithm that uses only dot products can use kernels, including ridge regression, PCA and k-means.
gammain the RBF kernel sets how far each training point's influence reaches. Too high agammagives perfect training accuracy and collapsing test accuracy.- With
coef0 = 0, a polynomial kernel drops all lower-degree terms, which can be disastrous.Candgammaaffect each other, so the good region is a diagonal band and they must be tuned on a joint grid.- Mercer's condition requires a positive semi-definite Gram matrix. The sigmoid kernel breaks it for many settings.
🧪 Practice
- Generate two circles, one inside the other. Confirm that a linear SVM fails, then apply the degree-2 map by hand and confirm that a linear SVM succeeds.
- Check numerically that
phi(a).phi(b)equals(a.b)^2for the map above, across a whole Gram matrix.- Count the size of the explicit feature map with
comb(d+p-1, d)for severaldandp, and note where it becomes impossible to build.- Vary RBF
gammaover five orders of magnitude. Record training accuracy, test accuracy and the number of support vectors, and spot the overfitting pattern.- Compare a degree-3 polynomial kernel with
coef0=0andcoef0=1, and explain the difference in terms of which terms are present.- Run a two-dimensional grid search over
Candgammaand describe the shape of the high-scoring region.- Compute the smallest eigenvalue of the Gram matrix for the RBF, polynomial and sigmoid kernels, and find which one is not a valid kernel on your data.
- Interview: An RBF SVM has 99% training accuracy, 62% test accuracy, and almost every training row is a support vector. Which hyperparameter do you suspect, and which way do you move it? (Hint: what does it mean that almost every point is needed to define the boundary?)
Voting and Averaging Ensembles
Random forests and boosting build ensembles from one kind of model. Voting and averaging ensembles do something simpler and more general: take several models you already have, from any families, and combine their outputs. The examples here are mostly classification, but the same ideas apply to averaging regression models.
Why would that help? The classic argument is Condorcet's jury theorem. Suppose
k voters are each right with probability q, and their mistakes are
independent. Then the chance that the majority is right rises towards 1 as k
grows, as long as q > 0.5. With q = 0.55, eleven voters are right 63% of
the time and 201 voters 92% of the time.
Both parts of that condition matter. If q < 0.5, voting makes things
worse and moves towards being certainly wrong: at q = 0.45, 201 voters
are right only 7.7% of the time. And "independent mistakes" is the hard part.
Five models trained on the same data are never independent, and how far they
are from independent decides how much voting helps.
Hard voting takes the most common predicted label. Soft voting averages the predicted probabilities and then picks the most likely class. Prefer soft voting whenever the models give usable probabilities. Hard voting ignores how confident each model is: a model that is 51% sure counts the same as one that is 99% sure. Soft voting also gives the ensemble a probability, which hard voting cannot.
For regression, the equivalent is simply averaging the predictions. Here the maths is exact and worth knowing. The ambiguity decomposition says:
ensemble error = average member error - diversity
"Diversity" is how far, on average, the members' predictions are from the ensemble's prediction (measured as squared difference). Both terms are zero or positive, so the ensemble is always at least as good as the average member, and the difference is exactly how much the members disagree. This is a theorem, not a rule of thumb.
But read it carefully. It compares the ensemble with the average member, not the best one. In the example below, a simple average of ridge, a tree and a forest scores 2.49, while the forest alone scores 2.08. Averaging a strong model with two weak ones drags it down. The theorem does not promise that ensembling helps.
Weights are the obvious next step, and the obvious next mistake. Weights are extra parameters fitted on the same data, and with only a few models it is easy to overfit them. Equal weights are a strong default. If you do tune weights, use cross-validation and expect only a small gain. The example shows four weighting schemes whose cross-validated AUCs differ by only 0.004, which is within the noise.
What you really need is diversity. Five nearly identical random forests average to almost exactly one random forest: their average correlation is 0.984, and the gain is 0.002 AUC. Five different model families gain 0.021, ten times as much, even though each of them is weaker on its own. Combine models with different inductive biases (different built-in assumptions): a linear model, a distance-based model, a tree ensemble, a kernel method. Combining five versions of the same idea mostly just costs compute.
import numpy as np
import pandas as pd
from sklearn.ensemble import (VotingClassifier, RandomForestClassifier,
RandomForestRegressor)
from sklearn.linear_model import LogisticRegression, Ridge
from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeRegressor
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import accuracy_score, log_loss, roc_auc_score
from math import comb
rng = np.random.default_rng(61)
# --- THE CONDORCET ARGUMENT: why majority voting works at all. ---
# k independent voters, each right with probability q. The majority is right with
# probability sum over m > k/2 of binom(k, m) q^m (1-q)^(k-m).
def majority_accuracy(k, q):
return sum(comb(k, m) * q ** m * (1 - q) ** (k - m)
for m in range(k // 2 + 1, k + 1))
print("MAJORITY VOTE OF k INDEPENDENT VOTERS, EACH RIGHT WITH PROBABILITY q")
print(f" {'q':>5s} " + " ".join(f"k={k}".rjust(8) for k in [1, 3, 11, 51, 201]))
for q in [0.45, 0.51, 0.55, 0.65, 0.80]:
cells = " ".join(f"{majority_accuracy(k, q):8.4f}"
for k in [1, 3, 11, 51, 201])
print(f" {q:5.2f} " + cells)
print(" Above q=0.5 the vote converges to certainty. BELOW it, voting makes")
print(" things WORSE, and converges to certainty of being wrong. Voting")
print(" amplifies whatever the members are, so weak members must still be")
print(" better than chance -- and independent, which is the hard part.")
# --- HARD VS SOFT VOTING: discarding confidence costs you. ---
n = 6000
Z = rng.normal(0, 1, (n, 12))
logit = (1.3 * Z[:, 0] - 1.1 * Z[:, 1] + 0.9 * Z[:, 2] * Z[:, 3]
+ 0.7 * np.abs(Z[:, 4]) - 0.6 * Z[:, 5] ** 2)
y = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
Xtr, Xte, ytr, yte = train_test_split(Z, y, test_size=0.3, random_state=0,
stratify=y)
members = [
("logistic", make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))),
("knn", make_pipeline(StandardScaler(), KNeighborsClassifier(25))),
("naive bayes", GaussianNB()),
("random forest", RandomForestClassifier(n_estimators=200, random_state=0)),
("svm rbf", make_pipeline(StandardScaler(),
SVC(kernel="rbf", probability=True, random_state=0))),
]
print(f"\n{'member':16s} {'accuracy':>9s} {'AUC':>7s} {'log loss':>9s}")
fitted = {}
for name, est in members:
est.fit(Xtr, ytr)
fitted[name] = est
p = est.predict_proba(Xte)[:, 1]
print(f" {name:14s} {est.score(Xte, yte):9.4f} {roc_auc_score(yte, p):7.4f}"
f" {log_loss(yte, p):9.4f}")
hard = VotingClassifier(members, voting="hard").fit(Xtr, ytr)
soft = VotingClassifier(members, voting="soft").fit(Xtr, ytr)
ps = soft.predict_proba(Xte)[:, 1]
print(f"\n {'hard vote (majority label)':32s} accuracy"
f" {hard.score(Xte, yte):.4f}")
print(f" {'soft vote (mean probability)':32s} accuracy"
f" {soft.score(Xte, yte):.4f}"
f" AUC {roc_auc_score(yte, ps):.4f} log loss {log_loss(yte, ps):.4f}")
print(" Hard voting throws away every member's confidence and keeps only its")
print(" argmax, so a model that is 51% sure counts the same as one that is 99%")
print(" sure. Prefer soft voting whenever the members give usable probabilities.")
# --- WEIGHTED SOFT VOTING, AND WHY EQUAL WEIGHTS ARE A REASONABLE DEFAULT. ---
print("\nWEIGHTS (tuned by cross-validation, not by eyeballing test scores)")
for label, w in [("equal", [1, 1, 1, 1, 1]),
("by rough skill", [2, 1, 1, 3, 2]),
("forest-heavy", [1, 1, 1, 6, 1]),
("drop naive bayes", [1, 1, 0, 2, 1])]:
v = VotingClassifier(members, voting="soft", weights=w)
cv = cross_val_score(v, Xtr, ytr, cv=5, scoring="roc_auc").mean()
v.fit(Xtr, ytr)
p = v.predict_proba(Xte)[:, 1]
print(f" {label:18s} weights {str(w):18s} cv AUC {cv:.4f}"
f" test AUC {roc_auc_score(yte, p):.4f}")
print(" Equal weights are a strong default because weights are p more")
print(" parameters fitted on the same data, and it is easy to overfit them.")
# --- WHEN AVERAGING FAILS: MEMBERS THAT AGREE. ---
print("\nAVERAGING NEEDS DISAGREEMENT")
similar = [(f"forest_{i}", RandomForestClassifier(n_estimators=200,
random_state=i))
for i in range(5)]
for label, group in [("5 near-identical forests", similar),
("5 different model families", members)]:
v = VotingClassifier(group, voting="soft").fit(Xtr, ytr)
P = np.array([est.predict_proba(Xte)[:, 1]
for est in v.named_estimators_.values()])
C = np.corrcoef(P)
solo = np.mean([roc_auc_score(yte, row) for row in P])
ens = roc_auc_score(yte, v.predict_proba(Xte)[:, 1])
print(f" {label:26s} member AUC {solo:.4f} -> ensemble {ens:.4f}"
f" gain {ens - solo:+.4f}")
print(f" {'':26s} mean pairwise correlation"
f" {C[np.triu_indices_from(C, 1)].mean():.3f}")
print(" Averaging five copies of the same idea gains almost nothing. Averaging")
print(" five different ideas gains real accuracy, because their errors differ.")
# --- REGRESSION: AVERAGING IS EVEN SIMPLER, AND THE ALGEBRA IS EXACT. ---
t = (2.0 * Z[:, 0] + 1.5 * np.abs(Z[:, 1]) - 1.2 * Z[:, 2] * Z[:, 3]
+ rng.normal(0, 1.0, n))
Rtr, Rte, ttr, tte = train_test_split(Z, t, test_size=0.3, random_state=0)
regs = [("ridge", make_pipeline(StandardScaler(), Ridge(alpha=1.0))),
("tree", DecisionTreeRegressor(max_depth=6, random_state=0)),
("forest", RandomForestRegressor(n_estimators=200, random_state=0))]
print(f"\n{'model':16s} {'test MSE':>9s}")
preds = {}
for name, est in regs:
est.fit(Rtr, ttr)
preds[name] = est.predict(Rte)
print(f" {name:14s} {np.mean((tte - preds[name]) ** 2):9.4f}")
avg = np.mean(list(preds.values()), axis=0)
print(f" {'simple average':14s} {np.mean((tte - avg) ** 2):9.4f}")
# The exact decomposition: ensemble error = average member error - diversity.
mean_member_err = np.mean([np.mean((tte - p) ** 2) for p in preds.values()])
diversity = np.mean([np.mean((p - avg) ** 2) for p in preds.values()])
print(f"\n AMBIGUITY DECOMPOSITION (exact for squared error)")
print(f" average member error {mean_member_err:.4f}")
print(f" minus diversity -{diversity:.4f}")
print(f" = ensemble error {mean_member_err - diversity:.4f}"
f" (measured {np.mean((tte - avg) ** 2):.4f})")
print(" The ensemble is ALWAYS at least as good as the average member, and the")
print(" gap is exactly how much the members disagree. That is a theorem, not a")
print(" heuristic -- but note it compares against the AVERAGE member, not the")
print(" best one, so an ensemble can still lose to its strongest member.")
CONDORCET: MAJORITY VOTE OF k INDEPENDENT VOTERS
q k=1 k=3 k=11 k=51 k=201
0.45 0.4500 0.4253 0.3669 0.2359 0.0774 <- WORSE, fast
0.51 0.5100 0.5150 0.5271 0.5571 0.6118
0.55 0.5500 0.5748 0.6331 0.7641 0.9226
0.65 0.6500 0.7183 0.8513 0.9862 1.0000
0.80 0.8000 0.8960 0.9883 1.0000 1.0000
Voting AMPLIFIES whatever the members are. Above 0.5 it converges to
certainty of being right; below it, to certainty of being wrong.
HARD VS SOFT VOTING
member accuracy AUC log loss
logistic 0.7078 0.7779 0.5662
knn 0.7228 0.7970 0.5559
naive bayes 0.7394 0.8231 0.5203
random forest 0.7444 0.8221 0.5228
svm rbf 0.7644 0.8478 0.4837
hard vote (majority label) accuracy 0.7472
soft vote (mean probability) accuracy 0.7578 AUC 0.8341
three members predicting class 1 with p = 0.51, 0.52, 0.99
and two predicting class 0 with p = 0.05, 0.02:
hard: 3 votes for 1, 2 for 0 -> class 1
soft: mean p = (0.51+0.52+0.99+0.05+0.02)/5 = 0.418 -> class 0
Hard voting cannot tell a 51% opinion from a 99% one.
WEIGHTS ARE MORE PARAMETERS, SO EQUAL WEIGHTS ARE A STRONG DEFAULT
scheme weights cv AUC test AUC
equal [1, 1, 1, 1, 1] 0.8435 0.8341
by rough skill [2, 1, 1, 3, 2] 0.8429 0.8345
forest-heavy [1, 1, 1, 6, 1] 0.8393 0.8310
drop naive bayes [1, 1, 0, 2, 1] 0.8417 0.8332
The whole range is 0.004 of cross-validated AUC: inside the noise.
AVERAGING NEEDS DISAGREEMENT
5 near-identical forests member AUC 0.8219 -> ensemble 0.8238
mean pairwise correlation 0.984
5 different model families member AUC 0.8136 -> ensemble 0.8341
mean pairwise correlation 0.863
WORSE individual members, TEN TIMES the gain.
THE AMBIGUITY DECOMPOSITION IS EXACT FOR SQUARED ERROR
model test MSE
ridge 3.3010
tree 2.8801
forest 2.0831 <- the best single model
simple average 2.4937
average member error 2.7548
minus diversity -0.2611
= ensemble error 2.4937 (measured 2.4937)
The ensemble beats the AVERAGE member by exactly the diversity. It
still LOSES to the forest alone, because averaging a strong model
with two weak ones drags it down. The theorem guarantees nothing
about beating your best model.
| Hard voting | Soft voting | Weighted averaging | |
|---|---|---|---|
| What is combined | Predicted labels | Predicted probabilities | Weighted probabilities |
Needs predict_proba | No | Yes | Yes |
| Uses model confidence | No | Yes | Yes |
| Outputs a probability | No | Yes | Yes |
| Extra parameters | None | None | One per model |
| Risk | Throws away information | None to speak of | Overfitting the weights |
Key Takeaways
- Majority voting moves towards certainty only when each model is better than chance. Below 0.5 it makes the error worse.
- Independent mistakes are what really limit voting, and models trained on the same data are never independent.
- Soft voting averages probabilities and keeps each model's confidence, which hard voting throws away.
- For squared error, the ambiguity decomposition is exact: ensemble error equals average member error minus diversity.
- That only guarantees beating the average member, so an ensemble can still lose to its best member.
- Weights are extra parameters fitted on the same data, so equal weights are a strong default and tuned weights usually gain little.
- Different built-in assumptions are what pay off: five different families beat five versions of one family, even when each one is weaker.
🧪 Practice
- Implement Condorcet's formula and tabulate majority accuracy for
qof 0.45, 0.52 and 0.7 with 1 to 501 voters.- Build a soft-voting ensemble from five different model families and compare it with hard voting on accuracy, AUC and log loss.
- Build a case where hard and soft voting disagree. Explain which one is right and why.
- Check the ambiguity decomposition numerically on a regression ensemble, and find a case where the ensemble loses to its best member.
- Cross-validate four weighting schemes and report whether any beats equal weights by more than the fold-to-fold standard error.
- Compare an ensemble of five identical forests with different seeds against an ensemble of five different families. Report the average pairwise correlation for both.
- Interview: You average five models and the ensemble is worse than your best single model. A colleague says ensembling is overrated. What is the real explanation? (Hint: what exactly does averaging guarantee to beat?)
Stacking and Blending
Voting treats every model the same, or weights them by numbers you picked. Stacking replaces the guessing with a fitted model. You train a second-level model, a meta-model, whose inputs are the base models' predictions and whose target is the real label. The meta-model learns how to combine them. It can even learn that one model is reliable in some region and another model elsewhere, which no fixed weighting can do.
The whole difficulty comes down to one question: what data do you train the meta-model on?
The obvious approach is badly broken. Fit the base models on the training set, have them predict that same training set, and give those predictions to the meta-model. The problem is that base models are overconfident on rows they have effectively memorised. A random forest grown to full depth predicts its own training rows almost perfectly. The meta-model sees this, decides the forest is almost never wrong, and gives it a huge weight. That weight reflects memorisation, not skill. In the example below, this leaky meta-model reaches a training AUC of exactly 1.0000 and gives the forest a coefficient of 10.45, compared with 0.47 when done correctly.
The fix is out-of-fold predictions:
- Split the training data into
kfolds. - For each fold, train the base models on the other
k-1folds and predict the held-out fold. - Put those predictions together into the meta-model's input table. Every row's inputs now come from models that never saw that row.
- Fit the meta-model on this table.
- Separately, refit each base model on all the training data, for use at prediction time.
cross_val_predict does steps 1 to 3, and StackingClassifier does all of
them.
Three practical rules:
Keep the meta-model simple. With five inputs you do not need a forest. Regularised logistic or linear regression is the standard choice, and the example shows why. A random-forest meta-model reaches a training AUC of 1.0 and loses 0.02 of test AUC: a gap of 0.19 between training and test, against 0.014 for logistic regression. The meta-model's job is to weight the base models, not to find new patterns.
Diversity of the base models still matters more than their individual strength, just as with voting.
passthrough lets the meta-model see the original features as well as the
base predictions. This can help when one model is reliable only in a region
you can identify from the features. It also gives the meta-model much more room
to overfit, so test it rather than assuming it helps. In the example, it
changed nothing.
Blending is stacking with one holdout set instead of k folds. Set aside
20-30% of the training data, fit the base models on the rest, predict the
holdout, and fit the meta-model there. It needs one fit per base model instead
of k+1, which matters when base models are expensive. The downside is that
both levels see less data and the meta-model's inputs are noisier, so it
usually does slightly worse: 0.8203 against 0.8283 below.
Be realistic about the gain. Here, stacking beat the best single model by 0.006 AUC, which is typical. Stacking wins competitions, where the fourth decimal place matters. In production, maintaining five models for a 0.6% relative gain is often not worth it. Decide which situation you are in before you build one.
import numpy as np
import pandas as pd
from sklearn.ensemble import (StackingClassifier, RandomForestClassifier,
HistGradientBoostingClassifier)
from sklearn.linear_model import LogisticRegression, LogisticRegressionCV
from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import (train_test_split, cross_val_predict,
StratifiedKFold)
from sklearn.metrics import roc_auc_score, log_loss
rng = np.random.default_rng(67)
n = 8000
Z = rng.normal(0, 1, (n, 14))
logit = (1.2 * Z[:, 0] - 1.0 * Z[:, 1] + 1.1 * Z[:, 2] * Z[:, 3]
+ 0.8 * np.abs(Z[:, 4]) - 0.7 * Z[:, 5] ** 2
+ 0.6 * np.sin(2 * Z[:, 6]))
y = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
Xtr, Xte, ytr, yte = train_test_split(Z, y, test_size=0.3, random_state=0,
stratify=y)
base = [
("logistic", make_pipeline(StandardScaler(),
LogisticRegression(max_iter=2000))),
("knn", make_pipeline(StandardScaler(), KNeighborsClassifier(35))),
("nb", GaussianNB()),
("forest", RandomForestClassifier(n_estimators=250, random_state=0)),
("boost", HistGradientBoostingClassifier(max_iter=200, random_state=0)),
]
print(f"{'base model':14s} {'test AUC':>9s}")
for name, est in base:
est.fit(Xtr, ytr)
print(f" {name:12s} {roc_auc_score(yte, est.predict_proba(Xte)[:, 1]):9.4f}")
# --- THE LEAKAGE TRAP: why you must use OUT-OF-FOLD predictions. ---
# WRONG: fit the base models on all of the training data, then use their
# IN-SAMPLE predictions as features for the meta-model. The base models are
# overconfident on rows they memorised, so the meta-model learns to trust
# whichever model overfits hardest.
naive_feats = np.column_stack([est.predict_proba(Xtr)[:, 1] for _, est in base])
naive_meta = LogisticRegression(max_iter=2000).fit(naive_feats, ytr)
test_feats = np.column_stack([est.predict_proba(Xte)[:, 1] for _, est in base])
# RIGHT: generate each base model's predictions for a row using only folds that
# EXCLUDED that row. cross_val_predict does exactly this.
cv = StratifiedKFold(5, shuffle=True, random_state=0)
oof = np.column_stack([
cross_val_predict(est, Xtr, ytr, cv=cv, method="predict_proba")[:, 1]
for _, est in base])
proper_meta = LogisticRegression(max_iter=2000).fit(oof, ytr)
print("\nIN-SAMPLE VS OUT-OF-FOLD META-FEATURES")
print(f" {'':22s} {'train AUC':>10s} {'test AUC':>9s}")
print(f" {'in-sample (LEAKY)':22s}"
f" {roc_auc_score(ytr, naive_meta.predict_proba(naive_feats)[:, 1]):10.4f}"
f" {roc_auc_score(yte, naive_meta.predict_proba(test_feats)[:, 1]):9.4f}")
print(f" {'out-of-fold (correct)':22s}"
f" {roc_auc_score(ytr, proper_meta.predict_proba(oof)[:, 1]):10.4f}"
f" {roc_auc_score(yte, proper_meta.predict_proba(test_feats)[:, 1]):9.4f}")
print("\nWHAT THE META-MODEL LEARNED (coefficients on each base model)")
print(f" {'base model':12s} {'leaky':>9s} {'out-of-fold':>12s}")
for name, wl, wp in zip([b[0] for b in base], naive_meta.coef_[0],
proper_meta.coef_[0]):
print(f" {name:12s} {wl:9.3f} {wp:12.3f}")
print(" The leaky version's weights are distorted toward whichever model has")
print(" the most memorised training predictions. Out-of-fold weights reflect")
print(" each model's GENERALISATION, which is the thing you wanted to combine.")
# --- THE LIBRARY DOES THIS CORRECTLY: StackingClassifier uses OOF internally. ---
stack = StackingClassifier(base, final_estimator=LogisticRegression(max_iter=2000),
cv=5, n_jobs=-1).fit(Xtr, ytr)
best_single = max(roc_auc_score(yte, est.predict_proba(Xte)[:, 1])
for _, est in base)
print(f"\nbest single base model test AUC {best_single:.4f}")
print(f"StackingClassifier test AUC"
f" {roc_auc_score(yte, stack.predict_proba(Xte)[:, 1]):.4f}")
simple_avg = test_feats.mean(1)
print(f"simple average test AUC {roc_auc_score(yte, simple_avg):.4f}")
# --- PASSTHROUGH: let the meta-model see the original features too. ---
stack_pt = StackingClassifier(base,
final_estimator=LogisticRegression(max_iter=3000),
cv=5, passthrough=True, n_jobs=-1).fit(Xtr, ytr)
print(f"stacking + passthrough test AUC"
f" {roc_auc_score(yte, stack_pt.predict_proba(Xte)[:, 1]):.4f}")
# --- THE META-MODEL SHOULD BE SIMPLE. ---
print("\nCHOICE OF META-MODEL (fitted on the same out-of-fold features)")
for label, meta in [
("logistic regression", LogisticRegression(max_iter=2000)),
("logistic, L2 tuned", LogisticRegressionCV(cv=5, max_iter=3000)),
("random forest", RandomForestClassifier(n_estimators=200, random_state=0)),
("boosted trees", HistGradientBoostingClassifier(max_iter=200,
random_state=0)),
]:
meta.fit(oof, ytr)
tr_auc = roc_auc_score(ytr, meta.predict_proba(oof)[:, 1])
te_auc = roc_auc_score(yte, meta.predict_proba(test_feats)[:, 1])
print(f" {label:22s} train {tr_auc:.4f} test {te_auc:.4f}"
f" gap {tr_auc - te_auc:+.4f}")
print(" A flexible meta-model overfits the base models' quirks: five inputs do")
print(" not need a forest. Regularised linear regression is the standard and")
print(" usually the best choice.")
# --- BLENDING: one holdout split instead of k folds. ---
Btr, Bhold, ybt, ybh = train_test_split(Xtr, ytr, test_size=0.25, random_state=1,
stratify=ytr)
from sklearn.base import clone
blend_feats, blend_test = [], []
for name, est in base:
e = clone(est).fit(Btr, ybt) # fit on the smaller slice only
blend_feats.append(e.predict_proba(Bhold)[:, 1])
blend_test.append(e.predict_proba(Xte)[:, 1])
blend_meta = LogisticRegression(max_iter=2000).fit(
np.column_stack(blend_feats), ybh)
blend_p = blend_meta.predict_proba(np.column_stack(blend_test))[:, 1]
print(f"\nblending (single 25% holdout) test AUC"
f" {roc_auc_score(yte, blend_p):.4f}")
print(f"stacking (5-fold OOF) test AUC"
f" {roc_auc_score(yte, stack.predict_proba(Xte)[:, 1]):.4f}")
print("Blending is cheaper -- one fit per base model instead of k+1 -- and uses")
print("less data for both stages, so it is noisier. Use it when base models are")
print("expensive; use stacking when you can afford the folds.")
THE ARCHITECTURE
LEVEL 0 (base models) LEVEL 1 (meta-model)
logistic ---\
knn ----\
nb -----+--> out-of-fold --> logistic ---> prediction
forest ----/ predictions regression
boost ---/ as features
THE LEAKAGE TRAP
WRONG: in-sample predictions as meta-features
fold-free: train base on ALL of X_train, predict on ALL of X_train
-> a purity-grown forest predicts its own rows almost perfectly
-> the meta-model learns "always trust the forest"
RIGHT: out-of-fold predictions
fold 1: train on folds 2-5, predict fold 1
fold 2: train on folds 1,3-5, predict fold 2 every row's
fold 3: train on folds 1-2,4-5, predict fold 3 features come
fold 4: train on folds 1-3,5, predict fold 4 from models that
fold 5: train on folds 1-4, predict fold 5 never saw it
train AUC test AUC
in-sample (LEAKY) 1.0000 0.8226
out-of-fold (correct) 0.8422 0.8281
^^^^^^
a perfect score that means nothing
WHAT THE META-MODEL LEARNED
base model leaky out-of-fold
logistic -1.210 -0.983
knn -0.707 2.638
nb -1.250 2.262
forest 10.451 0.466 <- 22x too large
boost 6.446 2.205
The leaky weights describe which model memorises hardest. The
out-of-fold weights describe which model GENERALISES.
WHAT STACKING ACTUALLY BUYS
best single base model 0.8225
simple average 0.8208 <- worse than the best member
StackingClassifier 0.8283 <- +0.0058 over the best member
stacking + passthrough 0.8280 <- no help here
THE META-MODEL MUST BE SIMPLE
meta-model train test gap
logistic regression 0.8422 0.8281 +0.0140
logistic, L2 tuned 0.8376 0.8247 +0.0129
random forest 1.0000 0.8069 +0.1931 <- overfits
boosted trees 0.9714 0.8052 +0.1662 <- overfits
Five inputs do not need a forest. The meta-model weights; it does
not discover.
STACKING VS BLENDING
blending (single 25% holdout) 0.8203 1 fit per base model
stacking (5-fold OOF) 0.8283 k+1 fits per base model
Blending is cheaper and noisier, because both stages see less data.
| Voting / averaging | Stacking | Blending | |
|---|---|---|---|
| How models are combined | Fixed weights | A fitted meta-model | A fitted meta-model |
| Meta-model inputs come from | Not applicable | k-fold out-of-fold predictions | A single holdout |
| Fits per base model | 1 | k+1 | 1 |
| Can trust models by region | No | Yes | Yes |
| Leakage risk | None | High if in-sample predictions are used | Lower, but the holdout is smaller |
| Typical gain over best model | None to small | Small but real | Slightly less than stacking |
Key Takeaways
- Stacking replaces hand-picked weights with a fitted meta-model whose inputs are the base models' predictions.
- The meta-model's inputs must be out-of-fold predictions. In-sample predictions leak and teach the meta-model to trust whichever model memorises most.
- The sign of leakage is a meta-model with a near-perfect training score and strangely distorted coefficients.
StackingClassifierbuilds the out-of-fold inputs for you, so prefer it to a hand-written version.- Keep the meta-model simple and regularised. A flexible meta-model overfits the base models' quirks instead of weighting them.
passthroughalso gives the meta-model the original features. This adds both power and overfitting risk, so test it.- Blending uses one holdout instead of k folds: cheaper, noisier and usually a little worse.
- Realistic gains over the best single model are fractions of a percent, so weigh them against the cost of running several models in production.
🧪 Practice
- Build a stacking ensemble by hand with
cross_val_predictand check that it closely matchesStackingClassifier.- Build the leaky version on purpose with in-sample predictions, and report the meta-model's training score and coefficients.
- Compare the meta-model's coefficients from leaky and out-of-fold inputs, and find which base model is distorted most.
- Try logistic regression, a shallow tree and a random forest as meta-models, and report the training-to-test gap for each.
- Turn on
passthroughand measure the change. Explain in one sentence why it might help or hurt on your data.- Implement blending with a 25% holdout and compare accuracy and total fit time with 5-fold stacking.
- Add a deliberately terrible base model to a working stack and report the coefficient the meta-model gives it.
- Interview: Your stacked ensemble scores 0.94 on the data used to build the meta-model and 0.86 on a fresh test set, while your best base model scores 0.88 on both. What went wrong? (Hint: where did the meta-model's training inputs come from?)
Ensemble Diversity
Every ensemble method in this chapter depends on the same thing: the members must make different mistakes. Averaging a model with itself achieves nothing, however good the model is. This topic shows how to measure "different mistakes", and why you cannot simply push diversity as high as possible.
There are three standard measures, each answering a slightly different question.
Disagreement is the share of rows where two classifiers predict different labels. It is simple and easy to read, but it counts cases where both models are wrong the same as cases where exactly one is right.
Yule's Q is computed from a table of both-right, both-wrong and one-right cases. It runs from -1 to +1. +1 means the two models fail on exactly the same rows; 0 means their mistakes are independent. Q is the most useful of the three for ensembles, because it measures how correlated the mistakes are, not just the outputs.
Cohen's kappa measures how much two models agree beyond what chance would produce. Close to 1 means almost identical outputs; close to 0 means agreement no better than chance. The kappa-error diagram plots pairwise kappa against average pairwise error, and is the standard way to look at a pool of candidate models.
Notice what the example shows: five genuinely different model families still have Q values between 0.87 and 0.97. Even very different algorithms fail on mostly the same rows, because some rows are inherently hard. Label noise and overlapping classes affect every model in the same way. This sets a limit on what diversity can achieve.
The oracle analysis measures that limit directly. Compute the share of rows where at least one model is right. That is what a perfect way of combining the models could reach. Below, the best single model scores 0.745, all five models are right together on 0.552 of the rows, and at least one is right on 0.852. So there is 0.107 of room for improvement. A majority vote captures none of it and scores 0.739, below the best single model, because a majority vote needs most models to be right, not just one. Stacking tries to close this gap by learning which model to trust where. This is why oracle analysis is worth doing before you build a complex ensemble: if the oracle is barely above your best model, no way of combining them will help.
There are three ways to create diversity, and you can use them together:
- Vary the data: bootstrap the rows (bagging), reweight them (boosting), or take subsamples.
- Vary the features: random feature subsets per split (random forests) or per model (random subspaces).
- Vary the algorithm: different model families, different hyperparameters, or added randomness such as Extra Trees' random thresholds.
The third is by far the most effective for the effort, because it changes the inductive bias (the model's built-in assumptions), not just the sample. A linear model and a tree are wrong about different things in a structural way, which two bootstrapped trees never are.
Now the key warning, which is the real point of this topic. Diversity is a
means, not a goal. The members must be accurate on their own and diverse,
and these two pull against each other: every method that makes members less
alike also makes them weaker. In the example, using only feature subsets gives
the lowest kappa of any setup (0.082), but not the most accurate ensemble,
because its members are the weakest. Similarly, as random forest
max_features drops from 14 to 1, the average pairwise kappa falls steadily
from 0.276 to 0.091, while ensemble accuracy peaks somewhere in the middle.
The goal is not maximum diversity but the best balance between diversity and
accuracy.
import numpy as np
import pandas as pd
from sklearn.ensemble import (RandomForestClassifier, BaggingClassifier,
VotingClassifier, ExtraTreesClassifier)
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, roc_auc_score
rng = np.random.default_rng(71)
n = 6000
Z = rng.normal(0, 1, (n, 14))
logit = (1.2 * Z[:, 0] - 1.0 * Z[:, 1] + 1.0 * Z[:, 2] * Z[:, 3]
+ 0.8 * np.abs(Z[:, 4]) - 0.7 * Z[:, 5] ** 2)
y = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
Xtr, Xte, ytr, yte = train_test_split(Z, y, test_size=0.3, random_state=0,
stratify=y)
# --- MEASURING DIVERSITY: three standard statistics. ---
def disagreement(a, b):
"""Fraction of rows on which two classifiers give different labels."""
return float(np.mean(a != b))
def q_statistic(a, b, truth):
"""Yule's Q on the both-right / both-wrong / one-right table.
+1 means the two make identical mistakes, 0 means independent errors."""
ca, cb = a == truth, b == truth
n11 = np.sum(ca & cb); n00 = np.sum(~ca & ~cb)
n10 = np.sum(ca & ~cb); n01 = np.sum(~ca & cb)
denom = n11 * n00 + n01 * n10
return float((n11 * n00 - n01 * n10) / denom) if denom else 0.0
def kappa(a, b):
"""Cohen's kappa between two classifiers' outputs: agreement beyond chance."""
obs = np.mean(a == b)
pa = np.mean(a); pb = np.mean(b)
exp = pa * pb + (1 - pa) * (1 - pb)
return float((obs - exp) / (1 - exp)) if exp < 1 else 1.0
members = {
"logistic": make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000)),
"knn": make_pipeline(StandardScaler(), KNeighborsClassifier(35)),
"naive_bayes": GaussianNB(),
"forest": RandomForestClassifier(n_estimators=200, random_state=0),
"svm_rbf": make_pipeline(StandardScaler(), SVC(kernel="rbf", random_state=0)),
}
pred = {}
for name, est in members.items():
pred[name] = est.fit(Xtr, ytr).predict(Xte)
names = list(members)
print("PAIRWISE DIVERSITY (disagreement / Q statistic / kappa)")
print(f" {'pair':26s} {'disagree':>9s} {'Q':>7s} {'kappa':>7s}")
for i in range(len(names)):
for j in range(i + 1, len(names)):
a, b = pred[names[i]], pred[names[j]]
print(f" {names[i] + ' vs ' + names[j]:26s}"
f" {disagreement(a, b):9.4f} {q_statistic(a, b, yte):7.3f}"
f" {kappa(a, b):7.3f}")
print(" High Q means the two fail on the SAME rows, so combining them buys")
print(" nothing. Low disagreement means they are near-copies. You want pairs")
print(" that are individually accurate and disagree about WHICH rows are hard.")
# --- THE ORACLE BOUND: what diversity makes POSSIBLE. ---
# If some member is right, an ideal combiner could get the row right. The gap
# between that oracle and the best single model is the headroom diversity gives.
P = np.array([pred[k] for k in names])
any_right = np.mean((P == yte[None, :]).any(axis=0))
all_right = np.mean((P == yte[None, :]).all(axis=0))
best = max(accuracy_score(yte, pred[k]) for k in names)
print(f"\nORACLE ANALYSIS")
print(f" best single member accuracy {best:.4f}")
print(f" rows where ALL members are right {all_right:.4f}")
print(f" rows where ANY member is right {any_right:.4f} <- the oracle")
print(f" headroom available to a combiner {any_right - best:+.4f}")
vote = VotingClassifier(list(members.items()), voting="hard").fit(Xtr, ytr)
print(f" majority vote actually achieves {vote.score(Xte, yte):.4f}")
print(" A majority vote captures only a fraction of the oracle's headroom,")
print(" because it needs MOST members right, not just one. Stacking can do")
print(" better precisely by learning WHICH member to trust where.")
# --- IDENTICAL MEMBERS ARE ONE MEMBER. ---
# The degenerate case first, so the rest has a floor to compare against.
same = [(f"lr_{i}", make_pipeline(StandardScaler(),
LogisticRegression(max_iter=2000)))
for i in range(5)]
v_same = VotingClassifier(same, voting="hard").fit(Xtr, ytr)
solo = make_pipeline(StandardScaler(),
LogisticRegression(max_iter=2000)).fit(Xtr, ytr)
print("\nFIVE IDENTICAL MEMBERS")
print(f" single logistic regression {solo.score(Xte, yte):.4f}")
print(f" vote of five copies of it {v_same.score(Xte, yte):.4f}"
f" kappa between any two: 1.000")
print(" Averaging a model with itself is averaging nothing. Diversity is not a")
print(" nice-to-have; without it an ensemble is a slower copy of one model.")
# --- THE THREE WAYS TO MANUFACTURE DIVERSITY. ---
def member_preds(ens, Xa):
"""Each member's predictions, respecting per-member feature subsets."""
feats = getattr(ens, "estimators_features_", None)
if feats is None:
return np.array([e.predict(Xa) for e in ens.estimators_])
return np.array([e.predict(Xa[:, f]) for e, f in zip(ens.estimators_, feats)])
tree = DecisionTreeClassifier(random_state=0)
configs = [
("rows only (bootstrap)",
BaggingClassifier(tree, n_estimators=25, bootstrap=True,
max_features=1.0, random_state=0)),
("features only (50% subsets)",
BaggingClassifier(tree, n_estimators=25, bootstrap=False,
max_features=0.5, random_state=0)),
("rows and features",
BaggingClassifier(tree, n_estimators=25, bootstrap=True,
max_features=0.5, random_state=0)),
("rows, features, and split choice",
RandomForestClassifier(n_estimators=25, max_features=4, random_state=0)),
("plus random thresholds",
ExtraTreesClassifier(n_estimators=25, max_features=4, random_state=0)),
]
print("\nSOURCES OF DIVERSITY, ALL BUILT FROM THE SAME DECISION TREE")
print(f" {'randomisation':34s} {'member acc':>11s} {'ensemble':>9s}"
f" {'mean kappa':>11s}")
for label, ens in configs:
ens.fit(Xtr, ytr)
Pm = member_preds(ens, Xte)
ks = [kappa(Pm[i], Pm[j]) for i in range(len(Pm))
for j in range(i + 1, len(Pm))]
macc = np.mean([accuracy_score(yte, row) for row in Pm])
print(f" {label:34s} {macc:11.4f} {ens.score(Xte, yte):9.4f}"
f" {np.mean(ks):11.3f}")
print(" Each added source of randomness lowers kappa. Note that lower kappa")
print(" does not automatically mean a better ensemble: feature-only subsetting")
print(" is the most diverse here and not the most accurate, because its members")
print(" are the weakest. The two effects must be traded off.")
# --- THE TRADE-OFF: DIVERSITY IS NOT FREE. ---
print("\nDIVERSITY VS INDIVIDUAL STRENGTH (random forest max_features)")
print(f" {'max_features':>13s} {'member acc':>11s} {'ensemble':>9s}"
f" {'mean kappa':>11s}")
for mf in [14, 8, 4, 2, 1]:
rf = RandomForestClassifier(n_estimators=100, max_features=mf,
random_state=0).fit(Xtr, ytr)
Pm = np.array([t.predict(Xte) for t in rf.estimators_])
member_acc = np.mean([accuracy_score(yte, row) for row in Pm])
ks = [kappa(Pm[i], Pm[j]) for i in range(0, 40) for j in range(i + 1, 40)]
print(f" {mf:13d} {member_acc:11.4f} {rf.score(Xte, yte):9.4f}"
f" {np.mean(ks):11.3f}")
print(" Individual members get WORSE and less correlated as max_features falls.")
print(" The ensemble tracks the product of those two effects, which is why the")
print(" best setting is interior. Diversity is a means, never the objective.")
THREE WAYS TO MEASURE DIVERSITY
Q statistic works on the error agreement table:
model B right model B wrong
model A right n11 n10
model A wrong n01 n00
Q = (n11*n00 - n01*n10) / (n11*n00 + n01*n10)
Q = +1 they fail on exactly the same rows -> combining is useless
Q = 0 their errors are independent -> combining pays
PAIRWISE DIVERSITY ACROSS FIVE MODEL FAMILIES
pair disagree Q kappa
logistic vs knn 0.1728 0.897 0.655
logistic vs naive_bayes 0.1428 0.936 0.714
logistic vs svm_rbf 0.1917 0.871 0.617
knn vs forest 0.1411 0.928 0.718
naive_bayes vs forest 0.1011 0.967 0.798
forest vs svm_rbf 0.1200 0.950 0.760
Every Q is between 0.87 and 0.97. Even completely different
algorithms fail on largely the SAME rows, because some rows are
inherently hard. That is the ceiling on what diversity can buy.
ORACLE ANALYSIS: HOW MUCH HEADROOM EXISTS
best single member accuracy 0.7450
rows where ALL members are right 0.5517
rows where ANY member is right 0.8517 <- a perfect combiner
headroom available +0.1067
majority vote actually achieves 0.7389 <- BELOW the best member
row: 1 2 3 4 5 6
logistic: Y Y N Y N N
knn: Y N Y Y N N any-right: rows 1,2,3,4 = 4/6
forest: Y Y Y N N N majority: rows 1,2,3 = 3/6
row 4 is lost: only 1 of 3 right
A majority vote needs MOST members right. The oracle needs ONE.
Stacking exists to close part of that gap.
IDENTICAL MEMBERS ARE ONE MEMBER
single logistic regression 0.6811
vote of five copies of it 0.6811 kappa between any two: 1.000
SOURCES OF DIVERSITY, ALL FROM THE SAME DECISION TREE
randomisation member acc ensemble mean kappa
rows only (bootstrap) 0.6369 0.7117 0.280
features only (50% subsets) 0.5744 0.6811 0.082
rows and features 0.5706 0.6917 0.065
rows, features, and split choice 0.6291 0.7117 0.226
plus random thresholds 0.6134 0.7211 0.164
^^^^^^ ^^^^^
The MOST diverse configuration (kappa 0.082) is NOT the most
accurate, because its members are the weakest.
THE TRADE-OFF, IN ONE TABLE
max_features member acc ensemble mean kappa
14 0.6346 0.7239 0.276
8 0.6352 0.7278 0.268
4 0.6246 0.7322 0.224 <- best ensemble
2 0.6069 0.7339 0.154
1 0.5826 0.7217 0.091 <- most diverse
member accuracy kappa ensemble
falls \ falls \ peaks in the MIDDLE
\ \ _--*--_
\ \ _- -_
Diversity is a means. The objective is ensemble accuracy.
| Measure | Range | What +1 / high means | Best use |
|---|---|---|---|
| Disagreement | 0 to 1 | Outputs often differ | Quick, easy-to-read check |
| Yule's Q | -1 to +1 | Identical mistakes | Judging ensemble potential |
| Cohen's kappa | -1 to +1 | Identical outputs | Kappa-error diagrams |
| Oracle accuracy | 0 to 1 | Some model is always right | Upper limit on possible gain |
Key Takeaways
- Every ensemble method depends on members making different mistakes. Identical members give an ensemble identical to one member.
- Yule's Q measures how correlated the mistakes are, not just the outputs, which makes it the most useful diversity measure for ensembles.
- Even completely different algorithms fail on mostly the same rows, because some rows are inherently hard. That limits the possible gain.
- Oracle analysis (the share of rows where any model is right) sets an upper limit for any way of combining models. Compute it before building a complex ensemble.
- A majority vote needs most models to be right, not just one, so it captures only part of the oracle's room and can score below the best model.
- Diversity comes from varying the data, the features or the algorithm. Varying the algorithm changes the built-in assumptions and pays off most.
- Every method that makes members less alike also makes them weaker, so the most diverse setup is often not the most accurate.
- Diversity is a means; the goal is ensemble accuracy, and the best setting is in the middle.
🧪 Practice
- Implement disagreement, Yule's Q and Cohen's kappa, and compute all pairwise values for five different classifiers.
- Compute the oracle accuracy for your pool and compare it with the best single model and the majority vote.
- Build a three-model example where the oracle is right on a row and the majority vote is wrong, and explain why.
- Compare an ensemble of five models with the same seed against five with different seeds, and report the kappa of each pool.
- Isolate each source of diversity (rows only, features only, both) with
BaggingClassifier. Report member accuracy, ensemble accuracy and average kappa for each.- Vary
max_featuresin a random forest from 1 top, record all three numbers, and find where the ensemble is best.- Draw a kappa-error diagram for ten candidate models and use it to pick five.
- Interview: Your oracle analysis says at least one of your six models is right on 96% of rows, but your best single model gets 81% and your stacked ensemble 82%. What does that gap tell you, and what would you try? (Hint: the information is in the pool. What is the combiner failing to learn about when to trust each model?)
One Algorithm, Two Tasks
Most model families in this chapter appear twice in a library: once with a
Regressor suffix and once with a Classifier suffix. The difference is much
smaller than the names suggest. Grow a decision tree on a 0/1 column with the
regression criterion, and grow another with the classification criterion, and
you get the same thing: a set of regions with a number in each leaf. The
average of a 0/1 column in a leaf is the share of class 1 in that leaf. Only
three things really differ between the two versions: the splitting criterion,
how the output is transformed, and the loss. Everything else is shared.
| Family | As regression | As classification | What changes |
|---|---|---|---|
| linear model | identity link, squared error | logit link, cross-entropy | link + loss |
| decision tree | leaf mean, variance splits | leaf class shares, Gini or entropy | criterion + aggregation |
| random forest | average the leaf means | average the leaf class shares | aggregation |
| gradient boosting | fit residuals of squared loss | fit gradients of log loss, add up in logit space | loss + output transform |
| k-NN | mean of the k neighbours | vote of the k neighbours | aggregation |
| SVM | epsilon-insensitive loss | hinge loss | loss |
| neural network | linear output, MSE | softmax output, cross-entropy | output layer + loss |
A few methods really belong to only one side. Logistic regression, LDA, QDA and Naive Bayes are classification-only, because each one is built around a model of class membership. Quantile and isotonic regression are regression-only. But the model type (trees, ensembles, kernels, neighbours, networks) is almost always shared. That is why "which task, and which loss?" matters far more than "which algorithm?".
This leads to a question that comes up all the time: should you convert one task into the other? It comes up because the business often wants a decision while the data records a quantity.
The more common mistake is turning a number into categories to build a
classifier. Say you record repair costs, and the business asks whether a
claim will exceed a limit of 500. So you create the label cost > 500 and
train a classifier. It works. The problem is what you lost. A claim of 505 and
a claim of 2,400 now look exactly the same to the model, which no longer knows
that one of them is only just over the line. With plenty of data around that
exact threshold, this costs almost nothing, which is why people keep doing it.
It costs you in two specific cases:
- When labels are scarce. The distance from the threshold that you threw away was the cheapest extra information you had.
- When the threshold changes, as limits, budgets and SLAs always do. A regression model answers the new question for free. The classifier learned one cut-off and must be retrained.
So the rule is: if the numeric value is recorded, model it and apply the threshold afterwards. Classify the binned label only when the underlying quantity is truly not observed (you know the customer left, but not how close they were to leaving), when the cut-off is a fixed physical or legal boundary, or when behaviour really changes at the cut-off instead of passing smoothly through it.
The opposite mistake is regressing on ordered class codes, and it has the same shape. Encoding low/medium/high as 0/1/2 and using squared loss assumes the step from low to medium equals the step from medium to high. Sometimes this approximation is harmless, and keeping the order helps the model. The proper options for ordered targets are ordinal regression, or, if you have the underlying quantity, regressing on it and grouping the predictions into bands afterwards. That keeps the order without inventing exact distances.
import numpy as np
from sklearn.ensemble import (HistGradientBoostingClassifier,
HistGradientBoostingRegressor)
from sklearn.metrics import accuracy_score, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier, KNeighborsRegressor
from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor
rng = np.random.default_rng(13)
# --- 1. THE SAME MACHINE, RETARGETED. A tree grown on a 0/1 column with the
# regression criterion and a tree grown with the classification
# criterion produce the same object: a leaf, and the frequency in it.
n = 2000
f1 = rng.normal(0, 1, n)
f2 = rng.normal(0, 1, n)
p = 1 / (1 + np.exp(-(0.8 * f1 - 1.2 * f2 + 0.4)))
lab = (rng.random(n) < p).astype(int)
Xa = np.column_stack([f1, f2])
probe = Xa[:6]
reg = DecisionTreeRegressor(min_samples_leaf=40, random_state=0).fit(Xa, lab)
cls = DecisionTreeClassifier(min_samples_leaf=40, random_state=0).fit(Xa, lab)
kr = KNeighborsRegressor(25).fit(Xa, lab)
kc = KNeighborsClassifier(25).fit(Xa, lab)
print("1. REGRESSION AND CLASSIFICATION SHARE THE MACHINERY")
print(f" {'row':>4s} {'tree regressor':>15s} {'tree classifier P(1)':>21s}"
f" {'kNN reg':>9s} {'kNN clf P(1)':>13s}")
for i in range(6):
print(f" {i:4d} {reg.predict(probe[i:i+1])[0]:15.4f}"
f" {cls.predict_proba(probe[i:i+1])[0, 1]:21.4f}"
f" {kr.predict(probe[i:i+1])[0]:9.4f}"
f" {kc.predict_proba(probe[i:i+1])[0, 1]:13.4f}")
print(" The leaf mean of a 0/1 column IS the class frequency in that leaf.")
print(" What changes between the two APIs is the split criterion, the")
print(" output transform, and the loss -- not the model class.")
# --- 2. THE BINNING TRAP. A repair-cost model where the business question is
# 'will this claim exceed the 500 fast-track limit?'
m = 12000
age = rng.uniform(0, 15, m)
mileage = rng.uniform(0, 200, m)
severity = rng.uniform(0, 1, m)
cost = (80 + 22 * age + 1.4 * mileage + 900 * severity ** 2
+ rng.normal(0, 220, m)).clip(10, None)
Xb = np.column_stack([age, mileage, severity])
Xtr, Xte, ctr, cte = train_test_split(Xb, cost, test_size=0.4, random_state=0)
LIMIT = 500.0
ytr, yte = (ctr > LIMIT).astype(int), (cte > LIMIT).astype(int)
reg_m = HistGradientBoostingRegressor(random_state=0).fit(Xtr, ctr)
clf_m = HistGradientBoostingClassifier(random_state=0).fit(Xtr, ytr)
reg_pred = reg_m.predict(Xte)
print(f"\n2. THE SAME DECISION, TWO FRAMINGS (limit = {LIMIT:.0f})")
print(f" {'framing':38s} {'accuracy':>9s} {'AUC':>7s}")
print(f" {'classify the binned label directly':38s} "
f"{accuracy_score(yte, clf_m.predict(Xte)):9.3f}"
f" {roc_auc_score(yte, clf_m.predict_proba(Xte)[:, 1]):7.3f}")
print(f" {'regress the cost, then threshold it':38s} "
f"{accuracy_score(yte, reg_pred > LIMIT):9.3f}"
f" {roc_auc_score(yte, reg_pred):7.3f}")
print(" At the threshold it was trained on, with plenty of data, the")
print(" classifier gives up almost nothing. The cost of binning shows up")
print(" in two other places.")
# --- (a) When labels are scarce, the discarded information is missed. ---
print("\n (a) SMALL SAMPLES, where the discarded distance would have helped:")
print(f" {'training rows':>14s} {'classifier AUC':>15s} {'regressor AUC':>14s}")
for k in [150, 400, 1500, 7200]:
r = HistGradientBoostingRegressor(random_state=0).fit(Xtr[:k], ctr[:k])
c = HistGradientBoostingClassifier(random_state=0).fit(Xtr[:k], ytr[:k])
print(f" {k:14d} {roc_auc_score(yte, c.predict_proba(Xte)[:, 1]):15.3f}"
f" {roc_auc_score(yte, r.predict(Xte)):14.3f}")
# --- The regression model answers questions nobody asked it at fit time. ---
print("\n (b) AND THE LIMIT MOVES. Finance revises the fast-track cap:")
print(f" {'new limit':>10s} {'regressor accuracy':>19s} {'classifier':>12s}")
for new in [300.0, 500.0, 800.0, 1200.0]:
yt = (cte > new).astype(int)
a_r = accuracy_score(yt, reg_pred > new)
a_c = accuracy_score(yt, clf_m.predict(Xte))
note = "" if new == LIMIT else " <- classifier must be refit"
print(f" {new:10.0f} {a_r:19.3f} {a_c:12.3f}{note}")
# --- 3. GOING THE OTHER WAY: ordinal labels handed to a plain classifier. ---
edges = [0, 300, 700, np.inf]
band_tr = np.digitize(ctr, edges[1:-1])
band_te = np.digitize(cte, edges[1:-1])
multi = HistGradientBoostingClassifier(random_state=0).fit(Xtr, band_tr)
pred_band = multi.predict(Xte)
reg_band = np.digitize(reg_pred, edges[1:-1])
print("\n3. THREE ORDERED BANDS: low / mid / high")
print(f" {'approach':34s} {'exact':>7s} {'off by 1':>9s} {'off by 2':>9s}")
for name, pb in [("multiclass classifier", pred_band),
("regress, then band the prediction", reg_band)]:
d = np.abs(pb - band_te)
print(f" {name:34s} {(d == 0).mean():7.3f} {int((d == 1).sum()):9d}"
f" {int((d == 2).sum()):9d}")
print(" A plain multiclass loss scores low-called-high exactly as badly as")
print(" low-called-mid, because to cross-entropy the labels are unordered.")
print(" The regression framing cannot make that mistake as easily: to call")
print(" a low claim 'high' it has to be wrong by the whole mid band.")
1. REGRESSION AND CLASSIFICATION SHARE THE MACHINERY
row tree regressor tree classifier P(1) kNN reg kNN clf P(1)
0 0.7750 0.7750 0.5600 0.5600
1 0.5918 0.5918 0.5200 0.5200
2 0.4222 0.4222 0.4000 0.4000
3 0.5000 0.5000 0.3200 0.3200
4 0.8889 0.8889 0.9600 0.9600
5 0.9750 0.9750 0.9200 0.9200
The leaf mean of a 0/1 column IS the class frequency in that leaf.
What changes between the two APIs is the split criterion, the
output transform, and the loss -- not the model class.
2. THE SAME DECISION, TWO FRAMINGS (limit = 500)
framing accuracy AUC
classify the binned label directly 0.791 0.870
regress the cost, then threshold it 0.793 0.875
At the threshold it was trained on, with plenty of data, the
classifier gives up almost nothing. The cost of binning shows up
in two other places.
(a) SMALL SAMPLES, where the discarded distance would have helped:
training rows classifier AUC regressor AUC
150 0.848 0.861
400 0.832 0.850
1500 0.851 0.855
7200 0.870 0.875
(b) AND THE LIMIT MOVES. Finance revises the fast-track cap:
new limit regressor accuracy classifier
300 0.856 0.744 <- classifier must be refit
500 0.793 0.791
800 0.839 0.675 <- classifier must be refit
1200 0.927 0.443 <- classifier must be refit
3. THREE ORDERED BANDS: low / mid / high
approach exact off by 1 off by 2
multiclass classifier 0.659 1588 49
regress, then band the prediction 0.683 1493 27
A plain multiclass loss scores low-called-high exactly as badly as
low-called-mid, because to cross-entropy the labels are unordered.
The regression framing cannot make that mistake as easily: to call
a low claim 'high' it has to be wrong by the whole mid band.
Key Takeaways
- Regression and classification share almost all their machinery. What differs is the splitting criterion, the output transform and the loss.
- A regression tree's leaf mean on a 0/1 column equals a classification tree's predicted probability in that leaf, and the same holds for k-NN.
- Only a few methods truly belong to one side: logistic regression, LDA, QDA and Naive Bayes only classify; quantile and isotonic regression only predict numbers.
- Turning a recorded number into categories throws away the distance to the cut-off. That costs little at the trained threshold when data is plentiful.
- The cost shows up when labels are scarce, and most of all when the threshold moves: a regression model answers at any cut-off, but a classifier must be retrained.
- Whenever the numeric value is actually recorded, model it and apply the threshold afterwards.
- Classify the binned label when the quantity is truly unobserved, when the cut-off is fixed by law or physics, or when behaviour changes at the cut-off.
- For ordered targets, regressing the codes invents distances and plain multiclass ignores the order. Prefer ordinal regression, or regress the underlying quantity and group it into bands.
🧪 Practice
- Fit a tree regressor and a tree classifier on the same 0/1 target with the same settings, and check that the predictions match exactly.
- Repeat with k-NN, then explain in one sentence why they are equal.
- Take a numeric target, train a classifier on a binned version and a regressor on the raw values, and compare AUC at the training threshold for sample sizes from 100 to 10,000.
- Move the threshold after training and re-score both models. Estimate what retraining would cost in your own workflow.
- Group an ordered target into three levels, fit a multiclass classifier, and count the errors that are off by two levels, compared with a regress-then-band approach.
- Find a case in your own work where a binary label was built from a numeric field, and check whether the raw value is still available.
- Interview: A churn model is trained on the label "cancelled within 30 days". Product now wants to know who will cancel within 90 days. What can you reuse, and what would you have built differently? (Hint: what was thrown away when the time until cancelling became a yes/no label?)
<a id="11-unsupervised-learning"></a>