11. Unsupervised Learning
Unsupervised learning looks for structure in data that carries no labels, which removes the one thing that made supervised learning tractable: a right answer to check against. This chapter covers grouping rows into clusters, compressing many columns into few, mining patterns out of transactions and text, and flagging the points that do not belong -- along with the evaluation problem that runs through all four, since without labels "did it work" stops being a question you can answer by measuring error. Chapter 10 assumed a target column existed; here the recurring discipline is proving to yourself that the structure you found is in the data rather than in the algorithm.
<a id="111-clustering"></a>
11.1 Clustering
Clustering partitions rows into groups of similar items, and every algorithm in this section encodes a different answer to "what does a cluster look like" -- a sphere, a chain, a dense region, a probability distribution -- so the choice of algorithm is a hypothesis about shape; this section also covers the two hardest practical questions, how many clusters there are and how you would know if the answer were wrong.
k-Means and k-Medoids
You have a thousand customers and want to group them into four segments. The natural formalisation is: place four centres, assign each customer to the nearest one, and choose the centres so that customers sit as close to their centre as possible. "As close as possible" summed over all points and squared gives the objective, called inertia or within-cluster sum of squares.
Finding the exact optimum is NP-hard, so Lloyd's algorithm alternates two steps that are each easy:
- Assign: put every point with its nearest centre. 2. Update: move each centre to the mean of its assigned points.
Each step can only decrease inertia -- reassigning a point to a nearer centre reduces it, and the mean is the point minimising squared distance to a set -- so the algorithm converges. Convergence is guaranteed; a good answer is not. Inertia is non-convex and the algorithm stops at a local minimum.
That is not a theoretical footnote. In the demonstration below, three of six
random initialisations land at inertia 6968 while the others reach 2533, and
the bad ones score ARI 0.63 against the truth instead of 0.997. k-means++
fixes most of this by choosing initial centres spread far apart rather than
uniformly at random, and n_init repeats the whole fit and keeps the best.
With k-means++, one attempt suffices here; with random init, one attempt
fails.
The deeper point is what the objective assumes. Minimising squared distance to a centre means the boundary between two clusters is the perpendicular bisector of the line joining their centres -- a straight line, and the partition is a Voronoi diagram. Three consequences follow, all visible in the demonstration:
- Clusters must be convex and roughly spherical. Elongated clusters get sliced across their long axis, scoring ARI 0.00.
- Clusters must be of similar size. With a 98/2 split, halving the big cluster costs less inertia than isolating the small one, so k-means does that.
- Clusters must have similar spread. A tight cluster next to a diffuse one gets the diffuse one's core stolen.
Two operational rules. Scale your features, because inertia is squared Euclidean distance and a column with a wider numeric range dominates it entirely. And k-means is sensitive to outliers, because a mean can be dragged anywhere by a few extreme points.
k-medoids addresses the last two problems by changing what a centre is.
The centre -- the medoid -- must be an actual data point, chosen to
minimise the sum of distances (not squared distances) to its cluster's
members. Two things follow. It is far more robust: in the demonstration,
twelve clumped outliers move a k-means centroid by 53.6 units and a medoid by
0.06. And it works from any distance matrix -- cosine, Gower, edit
distance -- because it never needs to compute a mean, and a mean requires
coordinates. The cost is O(n^2) memory and a slower fit, which is why
k-means remains the default at scale.
import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
from sklearn.metrics import adjusted_rand_score, silhouette_score
from sklearn.datasets import make_blobs
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(101)
# --- LLOYD'S ALGORITHM, BY HAND. Two steps, repeated until nothing moves. ---
X, truth = make_blobs(n_samples=900, centers=4, cluster_std=1.2,
random_state=42)
def lloyd(X, k, seed, iters=60):
r = np.random.default_rng(seed)
centres = X[r.choice(len(X), k, replace=False)] # naive random init
history = []
for it in range(iters):
d = ((X[:, None, :] - centres[None, :, :]) ** 2).sum(-1) # all distances
labels = d.argmin(1) # ASSIGN step
new = np.array([X[labels == j].mean(0) if np.any(labels == j)
else centres[j] for j in range(k)]) # UPDATE step
history.append(d[np.arange(len(X)), labels].sum())
converged = np.allclose(new, centres)
centres = new # <-- the actual update
if converged: # no movement: stop
break
d = ((X[:, None, :] - centres[None, :, :]) ** 2).sum(-1)
labels = d.argmin(1)
history.append(d[np.arange(len(X)), labels].sum()) # final inertia
return labels, centres, history, it + 1
labels, centres, hist, iters = lloyd(X, 4, seed=0)
print("LLOYD'S ALGORITHM CONVERGES QUICKLY AND MONOTONICALLY")
print(f" converged in {iters} iterations")
for i in [0, 1, 2, 3, 5, len(hist) - 1]:
if i < len(hist):
print(f" iteration {i:2d}: inertia {hist[i]:10.2f}")
sk = KMeans(n_clusters=4, n_init=10, random_state=0).fit(X)
print(f" hand-rolled inertia {hist[-1]:.2f} sklearn {sk.inertia_:.2f}")
print(" Inertia never increases: each step either improves it or stops. That")
print(" guarantees CONVERGENCE, not a good answer -- it is a local minimum.")
# --- INITIALISATION MATTERS, AND IT IS WHY n_init EXISTS. ---
print("\nTHE SAME DATA, TEN RANDOM INITIALISATIONS")
results = [lloyd(X, 4, seed=s) for s in range(10)]
inertias = [r[2][-1] for r in results]
print(f" inertia across seeds: min {min(inertias):.2f} max {max(inertias):.2f}"
f" spread {max(inertias) - min(inertias):.2f}")
for s, r in list(enumerate(results))[:6]:
ari = adjusted_rand_score(truth, r[0])
worse = r[2][-1] > min(inertias) * 1.05
flag = " <- a genuinely worse local minimum" if worse else ""
print(f" seed {s}: inertia {r[2][-1]:9.2f} ARI vs truth {ari:.3f}{flag}")
print(" k-means++ spreads the initial centres out instead of picking at random,")
print(" which makes bad local minima much rarer. It is sklearn's default, and")
print(" n_init repeats the whole fit and keeps the best inertia.")
for init, n_init in [("random", 1), ("random", 10), ("k-means++", 1),
("k-means++", 10)]:
km = KMeans(n_clusters=4, init=init, n_init=n_init, random_state=0).fit(X)
print(f" init={init:10s} n_init={n_init:2d}: inertia {km.inertia_:9.2f}")
# --- WHAT k-MEANS ASSUMES: SPHERICAL, EQUAL-SIZED, EQUAL-VARIANCE CLUSTERS. ---
def elongated(n, seed):
r = np.random.default_rng(seed)
a = r.normal(0, 1, (n // 2, 2)) @ np.array([[6.0, 0], [0, 0.4]])
b = a.copy() + np.array([0.0, 4.0])
return np.vstack([a, b]), np.r_[np.zeros(n // 2), np.ones(n // 2)].astype(int)
def unequal_size(n, seed):
"""A 98/2 split where the small cluster sits near the big one. Splitting the
big cluster in half costs LESS inertia than isolating the small one, so
k-means does that instead."""
r = np.random.default_rng(seed)
big = r.normal([0, 0], 1.6, (int(n * 0.98), 2))
small = r.normal([4.2, 0], 0.3, (int(n * 0.02), 2))
return (np.vstack([big, small]),
np.r_[np.zeros(len(big)), np.ones(len(small))].astype(int))
def unequal_spread(n, seed):
r = np.random.default_rng(seed)
tight = r.normal([0, 0], 0.4, (n // 2, 2))
loose = r.normal([4, 0], 2.2, (n // 2, 2))
return (np.vstack([tight, loose]),
np.r_[np.zeros(n // 2), np.ones(n // 2)].astype(int))
print("\nWHERE THE SPHERICAL ASSUMPTION BREAKS (ARI: 1.0 is perfect)")
print(f" {'data shape':28s} {'k-means ARI':>12s}")
for name, gen in [("spherical, equal (ideal)",
lambda: make_blobs(n_samples=800, centers=2, cluster_std=1.0,
random_state=3)),
("elongated clusters", lambda: elongated(800, 3)),
("98/2 size imbalance", lambda: unequal_size(800, 3)),
("unequal spread", lambda: unequal_spread(800, 3))]:
Xa, ya = gen()
km = KMeans(n_clusters=2, n_init=10, random_state=0).fit(Xa)
print(f" {name:28s} {adjusted_rand_score(ya, km.labels_):12.3f}")
print(" k-means draws a straight line halfway between two centres (a Voronoi")
print(" boundary), so it can only carve space into convex blobs of similar")
print(" size. Every failure above is that one limitation.")
# --- SCALING IS MANDATORY: inertia is squared Euclidean distance. ---
df = pd.DataFrame({"age": rng.integers(20, 70, 600).astype(float),
"income": rng.normal(60_000, 18_000, 600)})
grp = ((df["age"] > 45).astype(int) * 2 + (df["income"] > 60_000).astype(int))
raw = KMeans(4, n_init=10, random_state=0).fit(df)
scaled = KMeans(4, n_init=10, random_state=0).fit(StandardScaler().fit_transform(df))
print(f"\nSCALING age spans ~50, income spans ~100,000")
print(f" unscaled ARI vs the true 4 groups"
f" {adjusted_rand_score(grp, raw.labels_):.3f}")
print(f" scaled ARI vs the true 4 groups"
f" {adjusted_rand_score(grp, scaled.labels_):.3f}")
print(" Unscaled, income's numeric range dominates the distance entirely and")
print(" the clusters are just income bands. Age is invisible.")
# --- k-MEDOIDS: the medoid is an actual data point, and any metric works. ---
# k-medoids minimises the sum of DISTANCES (not squared distances) to a centre
# chosen from the data, which makes it far less sensitive to outliers.
def kmedoids(D, k, seed, iters=100):
"""D is a precomputed distance matrix. A simple alternating heuristic."""
r = np.random.default_rng(seed)
med = r.choice(len(D), k, replace=False)
for _ in range(iters):
labels = D[:, med].argmin(1)
new = med.copy()
for j in range(k):
members = np.flatnonzero(labels == j)
if len(members):
# the new medoid is the member minimising total distance to the rest
new[j] = members[D[np.ix_(members, members)].sum(0).argmin()]
if np.array_equal(np.sort(new), np.sort(med)):
break
med = new
return D[:, med].argmin(1), med
clean, yc = make_blobs(n_samples=300, centers=[[0, 0], [8, 0], [4, 7]],
cluster_std=0.8, random_state=5)
# 12 outliers, all in one direction, which is the case that actually drags a
# mean: scattered outliers partly cancel, a clump does not.
outliers = rng.normal([60, 60], 3.0, (12, 2))
contaminated = np.vstack([clean, outliers])
yk = np.r_[yc, np.full(12, -1)]
print("\nOUTLIER SENSITIVITY: 300 clean points plus 12 clumped outliers")
print(f" {'data':16s} {'k-means ARI':>12s} {'k-medoids ARI':>14s}")
centre_shift = {}
for label, data, y_ in [("clean", clean, yc), ("contaminated", contaminated, yk)]:
km = KMeans(3, n_init=10, random_state=0).fit(data)
Dm = np.sqrt(((data[:, None] - data[None]) ** 2).sum(-1))
kmed, med = kmedoids(Dm, 3, seed=1)
keep = y_ >= 0 # score only the genuine points
print(f" {label:16s} {adjusted_rand_score(y_[keep], km.labels_[keep]):12.3f}"
f" {adjusted_rand_score(y_[keep], kmed[keep]):14.3f}")
centre_shift[label] = (np.sort(km.cluster_centers_, axis=0),
np.sort(data[med], axis=0))
# The mechanism, measured directly: how far did each method's centres move when
# the outliers were added?
km_move = np.abs(centre_shift["contaminated"][0]
- centre_shift["clean"][0]).max()
med_move = np.abs(centre_shift["contaminated"][1]
- centre_shift["clean"][1]).max()
print(f"\n largest centre coordinate shift caused by the 12 outliers:")
print(f" k-means centroids {km_move:8.3f}")
print(f" k-medoids medoids {med_move:8.3f}")
print(" A mean is a weighted average and 12 far points pull it a long way. A")
print(" medoid must BE one of the data points, so the worst it can do is jump")
print(" to another real observation.")
print("\n k-medoids also accepts ANY distance matrix -- cosine, Gower, edit")
print(" distance -- which k-means cannot, because its update step computes a")
print(" mean, and a mean needs coordinates rather than pairwise distances.")
print(" The cost is O(n^2) memory for the distance matrix and a slower fit,")
print(" which is why k-means remains the default at scale.")
LLOYD'S ALGORITHM: TWO STEPS UNTIL NOTHING MOVES
ASSIGN UPDATE
o o | x x o o | x x
o | x O | X O = mean of the o's
o o | x o o | x X = mean of the x's
| |
boundary is the perpendicular bisector: a straight line, always
iteration 0: inertia 53422.69
iteration 1: inertia 10579.13
iteration 2: inertia 2537.08
iteration 3: inertia 2533.25
iteration 4: inertia 2533.25 <- converged in 4 steps
hand-rolled 2533.25 sklearn 2533.25
INITIALISATION DECIDES WHICH LOCAL MINIMUM YOU GET
seed 0: inertia 2533.25 ARI 0.997
seed 1: inertia 2533.25 ARI 0.997
seed 2: inertia 2533.25 ARI 0.997
seed 3: inertia 6968.13 ARI 0.631 <- a genuinely worse optimum
seed 4: inertia 6970.42 ARI 0.636
seed 5: inertia 6973.64 ARI 0.628
init=random n_init= 1: inertia 6969.78 <- got stuck
init=random n_init=10: inertia 2533.25
init=k-means++ n_init= 1: inertia 2533.25 <- one try was enough
init=k-means++ n_init=10: inertia 2533.25
THE SPHERICAL ASSUMPTION, AND THREE WAYS IT BREAKS
data shape k-means ARI
spherical, equal (ideal) 1.000
elongated clusters -0.001
98/2 size imbalance -0.006
unequal spread 0.570
elongated: k-means cuts ACROSS the long axis
================== ====|====
================== -> ====|==== wrong: it splits both
| bands down the middle
98/2 imbalance: splitting the big blob costs less inertia
( big blob ) . -> ( big | blob ) .
small small ignored
SCALING (age spans ~50, income spans ~100,000)
unscaled ARI vs the 4 true groups 0.296
scaled ARI vs the 4 true groups 0.892
OUTLIERS: THE MEAN MOVES, THE MEDOID CANNOT
data k-means ARI k-medoids ARI
clean 1.000 1.000
contaminated 0.570 1.000
largest centre coordinate shift from 12 outliers:
k-means centroids 53.567
k-medoids medoids 0.062
a mean is a weighted average of everything, so 12 far points drag it:
o o o o * <- outliers
centroid ------------------------->
medoid stays on a real data point
| k-means | k-medoids | |
|---|---|---|
| Centre is | The mean of members | An actual data point |
| Objective | Sum of SQUARED distances | Sum of distances |
| Outlier robustness | Poor | Good |
| Works from a distance matrix | No | Yes |
| Any metric | No, Euclidean only | Yes |
| Cost per iteration | O(n k p) | O(n^2) |
| Scales to millions of rows | Yes | No |
Key Takeaways
- k-means minimises within-cluster squared distance by alternating assignment and mean-update, which converges but only to a local minimum.
- Initialisation materially changes the answer, so k-means++ and several restarts are not optional refinements.
- Squared-distance-to-a-centre makes cluster boundaries straight lines, so k-means can only find convex blobs of similar size and spread.
- Features must be scaled, because a wide-range column dominates Euclidean distance.
- A mean can be dragged arbitrarily far by a few outliers, which is the failure k-medoids fixes by requiring the centre to be a real observation.
- k-medoids works from any distance matrix, including ones with no coordinates at all, at
O(n^2)cost.
🧪 Practice
- Implement Lloyd's algorithm with explicit assign and update steps, and confirm your inertia matches
KMeanson the same data.- Run your implementation from twenty random initialisations and report the range of final inertia values and the ARI of the best and worst.
- Compare
init="random", n_init=1againstinit="k-means++", n_init=1on ten datasets and count how often each finds the better optimum.- Generate elongated clusters, a 98/2 size imbalance, and unequal-spread clusters, then report k-means ARI on each and explain each failure in terms of the Voronoi boundary.
- Cluster a table with age in years and income in dollars with and without standardisation, and describe what the unscaled clusters actually represent.
- Implement k-medoids from a precomputed distance matrix, add ten clumped outliers, and measure how far each method's centres move.
- Interview: Your k-means segmentation gives different customer segments every time the weekly job runs, even though the data barely changes. What are the two most likely causes and how would you tell them apart? (Hint: one is about how the fit starts, the other about whether the structure is there to find at all.)
Hierarchical Clustering
k-means needs k before it can start. Hierarchical clustering does not need
it at all: it produces a whole nested family of clusterings, from every point
alone to everything in one group, and you pick a level afterwards. For
exploratory work this is a genuinely different and often better workflow,
because the tree shows you what the candidate groupings are instead of
asking you to guess.
Agglomerative (bottom-up) clustering is the version you will use. Start with every point as its own cluster. Repeatedly merge the two closest clusters. Stop when one cluster remains. The record of which clusters merged, and at what distance, is the dendrogram. (Divisive, top-down clustering exists and is rarely used, because the first split requires searching an exponential number of partitions.)
"Closest two clusters" needs defining, and the definition -- the linkage criterion -- changes the result more than anything else:
| Linkage | Distance between clusters A and B | Tends to produce |
|---|---|---|
| Single | The closest pair of members | Long, chained, snake-like clusters |
| Complete | The farthest pair of members | Compact, roughly equal-diameter clusters |
| Average | Mean over all cross-pairs | A compromise between the two |
| Ward | The increase in within-cluster variance | Compact, similar-sized clusters; most k-means-like |
Ward is the usual default and behaves most like k-means, which means it inherits k-means's preference for spherical clusters of similar size. Complete and average linkage are the safe general choices.
Single linkage deserves special attention because it has one behaviour that is both its famous flaw and its unique strength. It only needs one close pair to merge two clusters, so a thin bridge of points connects everything -- the chaining effect. In the demonstration, fourteen bridge points cause single linkage to merge two clearly separate blobs (ARI 0.00) while the other three linkages resist (ARI 1.00).
The same property is why single linkage is the only method here that traces a crescent. On two interleaved moons it scores ARI 1.00 while complete, average, Ward, and k-means all score between 0.21 and 0.32. Single linkage never asks a cluster to be compact -- only connected -- which is exactly the right assumption for curved shapes and exactly the wrong one when noise bridges real gaps.
Reading a dendrogram is about the merge heights. Each merge happens at a distance; a large jump means that merge joined two groups that were genuinely far apart, so the level just below it is a natural number of clusters. In the demonstration, merging down to 3 clusters costs 3.5 times the previous merge, which identifies 4 as the natural cut. This is the dendrogram form of the elbow method and carries the same caveats.
Two practical facts. It does not scale: time is roughly O(n^2 log n) and
memory O(n^2), so 16,000 rows already needs a 2 GB distance matrix. Past
roughly 50,000 rows you cluster a sample. And it only needs a distance
matrix -- the most underrated advantage in this section. The demonstration
clusters ten words by edit distance, where there are no coordinates, no means,
and k-means cannot run at all.
import numpy as np
import pandas as pd
from scipy.cluster.hierarchy import linkage, dendrogram, fcluster, cophenet
from scipy.spatial.distance import pdist, squareform
from sklearn.cluster import AgglomerativeClustering, KMeans
from sklearn.datasets import make_blobs, make_moons
from sklearn.metrics import adjusted_rand_score
rng = np.random.default_rng(103)
# --- AGGLOMERATIVE CLUSTERING, BY HAND, ON SIX POINTS. ---
# Start with every point in its own cluster. Repeatedly merge the closest pair.
# The record of merges IS the dendrogram: no k is chosen during fitting.
pts = np.array([[0.0, 0.0], [0.4, 0.2], [0.2, 0.6],
[4.0, 4.0], [4.5, 4.2], [8.0, 0.5]])
names = list("ABCDEF")
D = squareform(pdist(pts))
clusters = {i: [i] for i in range(len(pts))}
def single_link(a, b):
return min(D[i, j] for i in a for j in b) # closest pair of members
print("AGGLOMERATIVE MERGING, ONE STEP AT A TIME (single linkage)")
step = 0
while len(clusters) > 1:
best = min(((single_link(m, n), k, l)
for k, m in clusters.items() for l, n in clusters.items() if k < l))
dist, k, l = best
merged = clusters.pop(k) + clusters.pop(l)
clusters[max(clusters, default=-1) + 100 + step] = merged
step += 1
print(f" step {step}: merge {{{','.join(names[i] for i in sorted(merged))}}}"
f" at distance {dist:.3f} ({len(clusters)} clusters remain)")
print(" Fitting produced a whole HIERARCHY. Choosing k happens afterwards, by")
print(" cutting the tree at a height -- which is a different workflow from")
print(" k-means, where k is an input.")
# --- LINKAGE CRITERIA: the same data, four different trees. ---
X, truth = make_blobs(n_samples=400, centers=4, cluster_std=1.1, random_state=11)
print("\nLINKAGE CRITERIA ON THE SAME DATA (ARI at k=4)")
print(f" {'linkage':12s} {'ARI':>7s} {'cophenetic r':>13s} what it merges on")
notes = {
"single": "the CLOSEST pair of members",
"complete": "the FARTHEST pair of members",
"average": "the mean of all pairwise distances",
"ward": "the increase in within-cluster variance",
}
d = pdist(X)
for method in ["single", "complete", "average", "ward"]:
Z = linkage(X, method=method)
lab = fcluster(Z, 4, criterion="maxclust")
# Cophenetic correlation: how faithfully the tree reproduces the original
# distances. High is good, but it is not a clustering quality measure.
coph, _ = cophenet(Z, d)
print(f" {method:12s} {adjusted_rand_score(truth, lab):7.3f} {coph:13.3f}"
f" {notes[method]}")
# --- THE CHAINING EFFECT: single linkage's signature failure and strength. ---
# Single linkage only needs ONE close pair, so a thin bridge of points merges two
# otherwise distinct clusters. The same property lets it follow curved shapes.
blob_a = rng.normal([0, 0], 0.6, (150, 2))
blob_b = rng.normal([7, 0], 0.6, (150, 2))
bridge = np.column_stack([np.linspace(0.8, 6.2, 14), rng.normal(0, 0.12, 14)])
bridged = np.vstack([blob_a, blob_b, bridge])
ytb = np.r_[np.zeros(150), np.ones(150), np.full(14, -1)].astype(int)
keep = ytb >= 0
print("\nCHAINING: two blobs joined by a thin bridge of 14 points")
for method in ["single", "complete", "average", "ward"]:
lab = fcluster(linkage(bridged, method=method), 2, criterion="maxclust")
print(f" {method:12s} ARI on the two real blobs"
f" {adjusted_rand_score(ytb[keep], lab[keep]):.3f}")
print(" Single linkage walks across the bridge and calls it one cluster. The")
print(" others require the whole cluster to be close, so they resist it.")
# The other side of the same coin: single linkage follows non-convex shapes.
Xm, ym = make_moons(n_samples=400, noise=0.055, random_state=0)
print("\nTHE SAME PROPERTY AS A STRENGTH: two interleaved crescents")
for method in ["single", "complete", "average", "ward"]:
lab = fcluster(linkage(Xm, method=method), 2, criterion="maxclust")
print(f" {method:12s} ARI {adjusted_rand_score(ym, lab):.3f}")
km = KMeans(2, n_init=10, random_state=0).fit(Xm)
print(f" {'k-means':12s} ARI {adjusted_rand_score(ym, km.labels_):.3f}")
print(" Single linkage is the only one that can trace a crescent, because it")
print(" never asks a cluster to be compact -- only connected.")
# --- READING A DENDROGRAM: the merge heights are the information. ---
Z = linkage(X, method="ward")
heights = Z[:, 2] # ascending: the k-th-to-last is the merge
# that took the tree from k+1 to k clusters
print("\nMERGE HEIGHTS TELL YOU WHERE TO CUT (ward linkage)")
print(f" {'merging k -> k-1':>18s} {'height':>9s} {'ratio to previous':>18s}")
for k in range(8, 1, -1):
h = heights[len(X) - k] # height of the merge producing k-1 clusters
h_prev = heights[len(X) - k - 1]
ratio = h / h_prev
flag = " <- biggest ratio" if ratio > 2.5 else ""
print(f" {f'{k} -> {k - 1}':>18s} {h:9.3f} {ratio:18.2f}{flag}")
print(" Merging down to 3 clusters costs 3.5x the previous merge: the last")
print(" CHEAP merge left 4 clusters, so 4 is the natural cut. This is the")
print(" dendrogram form of the elbow method, with the same caveats.")
# --- COST: hierarchical clustering does not scale. ---
import time
print("\nFIT TIME AND MEMORY (the distance matrix is n x n)")
for n in [1000, 4000, 16000]:
Zn = rng.normal(0, 1, (n, 8))
t0 = time.perf_counter()
linkage(Zn, method="ward")
ht = time.perf_counter() - t0
t0 = time.perf_counter(); KMeans(4, n_init=10, random_state=0).fit(Zn)
kt = time.perf_counter() - t0
print(f" n={n:6,d}: ward {ht:7.3f}s k-means {kt:6.3f}s"
f" distance matrix {n * n * 8 / 1e6:8.1f} MB")
print(" Time is roughly O(n^2 log n) and memory O(n^2). Past ~50,000 rows you")
print(" cluster a sample, or use a method that never forms the full matrix.")
# --- A REAL ADVANTAGE: it works from a distance matrix alone. ---
# No coordinates needed, so any similarity you can define will do.
words = ["cat", "cats", "bat", "rat", "dog", "dogs", "fog",
"swim", "swims", "slim"]
def edit(a, b):
"""Levenshtein distance, one row at a time."""
dp = np.arange(len(b) + 1)
for i, ca in enumerate(a, 1):
prev, dp[0] = dp[0], i
for j, cb in enumerate(b, 1):
prev, dp[j] = dp[j], min(dp[j] + 1, dp[j - 1] + 1,
prev + (ca != cb))
return dp[-1]
Dw = np.array([[edit(a, b) for b in words] for a in words], float)
Zw = linkage(squareform(Dw), method="average")
print("\nCLUSTERING FROM A DISTANCE MATRIX ALONE (edit distance on words)")
print(" merge heights:", np.round(Zw[:, 2], 2))
for thresh in [1.2, 2.0, 4.0]:
lab = fcluster(Zw, thresh, criterion="distance")
groups = {}
for w, c in zip(words, lab):
groups.setdefault(c, []).append(w)
print(f" cut at height {thresh}: " +
" ".join("{" + ",".join(v) + "}" for v in groups.values()))
print(" These strings have no coordinates, so there is no mean to compute and")
print(" k-means cannot run at all. Agglomerative clustering only ever needed")
print(" the pairwise distances, which is its most underrated advantage.")
AGGLOMERATIVE MERGING ON SIX POINTS (single linkage)
A(0,0) B(0.4,0.2) C(0.2,0.6) D(4,4) E(4.5,4.2) F(8,0.5)
step 1: merge {B,C} at 0.447 (5 clusters remain)
step 2: merge {A,B,C} at 0.447 (4 clusters remain)
step 3: merge {D,E} at 0.539 (3 clusters remain)
step 4: merge {D,E,F} at 5.093 (2 clusters remain)
step 5: merge {A..F} at 5.099 (1 cluster remains)
THE DENDROGRAM (height = merge distance)
5.1 |-----------------------------------------------.
| |
5.0 | .---------------------|
| | |
0.5 | .------------| |
| | | |
0.4 | .-----| | |
| | | | |
0.0 +---- B C A D E F ------------
\__/ | \__/ |
| | | |
cut here (0.45): {B,C} {A} {D,E} {F} -> 4 clusters
cut here (2.00): {A,B,C} {D,E} {F} -> 3 clusters
cut here (5.05): {A,B,C} {D,E,F} -> 2 clusters
Fitting produced the whole hierarchy. k is chosen AFTERWARDS.
THE LINKAGE CRITERION CHANGES EVERYTHING
linkage ARI cophenetic r merges on
single 0.712 0.875 the CLOSEST pair of members
complete 0.987 0.890 the FARTHEST pair of members
average 0.974 0.896 the mean of all cross-pairs
ward 0.955 0.861 the increase in within-cluster variance
CHAINING: SINGLE LINKAGE'S FLAW
(two blobs joined by a thin bridge of 14 points)
(blob A) . . . . . . . . . . (blob B)
^^^^^^^^^^^^^^^^^^
14 bridge points
single ARI 0.000 <- walks across the bridge
complete ARI 1.000
average ARI 1.000
ward ARI 1.000
THE SAME PROPERTY AS A STRENGTH
(two interleaved crescents)
single ARI 1.000 <- the ONLY method that traces a curve
complete ARI 0.224
average ARI 0.318
ward ARI 0.206
k-means ARI 0.269
single linkage asks for CONNECTEDNESS, not compactness.
MERGE HEIGHTS TELL YOU WHERE TO CUT (ward)
merging k -> k-1 height ratio to previous
8 -> 7 11.036 1.00
7 -> 6 11.963 1.08
6 -> 5 12.218 1.02
5 -> 4 13.400 1.10
4 -> 3 48.507 3.47 <- biggest ratio
3 -> 2 100.224 2.07
2 -> 1 139.126 1.39
The last CHEAP merge left 4 clusters, so 4 is the natural cut.
COST: TIME AND MEMORY BOTH GROW QUADRATICALLY
n= 1,000: ward 0.016s k-means 0.084s matrix 8.0 MB
n= 4,000: ward 0.330s k-means 0.306s matrix 128.0 MB
n=16,000: ward 8.637s k-means 2.051s matrix 2048.0 MB
IT ONLY NEEDS DISTANCES (edit distance on words, no coordinates)
merge heights: [1.0 1.0 1.0 1.0 1.5 1.5 1.5 3.42 4.24]
cut at 1.2: {cat,cats} {bat,rat} {dog,dogs} {fog} {swim,swims} {slim}
cut at 2.0: {cat,cats,bat,rat} {dog,dogs,fog} {swim,swims,slim}
cut at 4.0: {cat,cats,bat,rat,dog,dogs,fog} {swim,swims,slim}
k-means cannot run here at all: there is no mean of two strings.
Key Takeaways
- Agglomerative clustering produces a whole hierarchy in one fit, so
kis chosen after fitting by cutting the tree rather than supplied before it.- The linkage criterion is the most consequential choice, changing results far more than any other setting.
- Ward behaves most like k-means and inherits its bias toward spherical, similar-sized clusters.
- Single linkage chains across thin bridges of points, which is its notorious flaw and simultaneously the reason it is the only linkage that traces curved clusters.
- Large jumps in merge height mark natural cut points, making the dendrogram the visual equivalent of the elbow method.
- Time is roughly
O(n^2 log n)and memoryO(n^2), so it does not scale past tens of thousands of rows.- It needs only a pairwise distance matrix, so it works on strings, graphs, or anything else with no coordinates.
🧪 Practice
- Implement agglomerative merging on ten points with single linkage, printing each merge and its distance, and check it against
scipy.cluster.hierarchy.- Cluster the same dataset with all four linkages at
k=4and compare ARI plus the cophenetic correlation.- Build two blobs joined by a thin bridge and show which linkages chain across it and which resist.
- Cluster two interleaved moons with all four linkages and explain why the ranking is the reverse of the previous exercise.
- Print the last ten merge heights with their ratios and use the largest ratio to choose
k.- Build a pairwise edit-distance matrix over twenty words and cluster it, cutting at three different heights.
- Interview: You have 200,000 rows and want a dendrogram because the business wants to see nested segments. What do you actually do? (Hint: what is the memory cost, and what could you fit the tree on instead of every row?)
DBSCAN and Density-Based Methods
k-means and Ward both start from "a cluster is a blob around a centre". DBSCAN starts somewhere else entirely: a cluster is a connected dense region. It never computes a centre, which frees it from every shape constraint, and it gains one capability neither previous method has -- it can decline to assign a point at all.
Two parameters define density. eps is a radius; min_samples is a count.
From those, every point is one of three types:
- Core point: at least
min_samplespoints withineps(itself included). This is what "in a dense region" means. - Border point: within
epsof a core point, but not core itself. It joins that cluster but cannot extend it. - Noise: neither. It belongs to no cluster.
Clusters are then grown by transitive connection: start at a core point,
absorb everything within eps, and keep expanding through any new core points
you reach. The chain stops at border points.
That mechanism buys the two things k-means cannot do. Arbitrary shapes, since connectivity has no opinion about convexity -- DBSCAN scores ARI 1.00 on two moons where k-means scores 0.26, and 0.93 on concentric circles where k-means scores 0.00. And explicit noise, which matters more than it sounds: a method that must assign every point will smear outliers into whichever cluster is nearest, corrupting it.
The costs are real. eps is hard to choose and unforgiving. The standard
heuristic is the k-distance plot: compute every point's distance to its
min_samples-th nearest neighbour, sort, and look for the knee where the
curve turns up. The flat part is points in dense regions; the tail is
everything else. In the demonstration the 95th percentile is 0.17 and the 97th
is 0.68, which locates the knee sharply. Note that eps is in data units,
so rescaling the features invalidates it completely -- standardise first, then
tune.
The sensitivity sweep shows how narrow the useful range is: at eps=0.08
everything fragments into ten clusters with 22% noise; at 0.40 everything is
one cluster. Between 0.12 and 0.25 it is perfect. min_samples is more
forgiving (3 to 15 all work here) and a common starting point is 2 * n_features.
The fundamental limitation is that eps is global. One density threshold
cannot describe data containing two different densities. In the demonstration,
a tight cluster next to a diffuse one has no good eps: at 0.12 the tight
cluster is clean and 93% of the diffuse one is discarded as noise; at 0.20 the
diffuse cluster shatters into ten fragments; at 0.35 both survive but 13% of
the diffuse cluster is still noise; at 0.55 the two merge outright.
HDBSCAN is the modern answer and should usually be your default. It builds
a hierarchy across all density thresholds and extracts the most stable
clusters from it, so there is no eps -- only min_cluster_size, which is a
far more intuitive thing to specify. It recovers both clusters in the
varying-density case. Note what it does not fix: 27% of the diffuse cluster
is still labelled noise, because HDBSCAN removes the tuning problem, not the
fact that a diffuse cluster is diffuse. OPTICS does something similar via
a reachability plot you cut by hand; it is fiddlier and rarely preferable.
import numpy as np
import pandas as pd
from sklearn.cluster import DBSCAN, OPTICS, KMeans, HDBSCAN
from sklearn.neighbors import NearestNeighbors
from sklearn.datasets import make_moons, make_blobs, make_circles
from sklearn.metrics import adjusted_rand_score
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(107)
# --- THE THREE POINT TYPES. Everything about DBSCAN follows from these. ---
# core : at least min_samples neighbours within eps (including itself)
# border : within eps of a core point, but not core itself
# noise : neither -- and DBSCAN is the first method here that can say "neither"
X, truth = make_moons(n_samples=600, noise=0.07, random_state=0)
X = np.vstack([X, rng.uniform(-1.5, 2.5, (40, 2))]) # 40 scattered noise points
y = np.r_[truth, np.full(40, -1)]
db = DBSCAN(eps=0.18, min_samples=8).fit(X)
core_mask = np.zeros(len(X), bool)
core_mask[db.core_sample_indices_] = True
noise = db.labels_ == -1
border = ~core_mask & ~noise
print("DBSCAN CLASSIFIES EVERY POINT INTO ONE OF THREE TYPES")
print(f" core points {core_mask.sum():4d} (>= 8 neighbours within eps=0.18)")
print(f" border points {border.sum():4d} (near a core point, but not core)")
print(f" noise points {noise.sum():4d} (belong to no cluster at all)")
print(f" clusters found {len(set(db.labels_) - {-1})} true clusters 2")
real = y >= 0
print(f" ARI on the genuine points"
f" {adjusted_rand_score(y[real], db.labels_[real]):.3f}")
print(f" of the 40 injected noise points, DBSCAN labelled"
f" {np.sum(db.labels_[len(truth):] == -1)} as noise")
# --- WHY THAT MATTERS: SHAPE AND NOISE, THE TWO THINGS k-MEANS CANNOT DO. ---
print("\nARBITRARY SHAPES (ARI on genuine points; 1.0 is perfect)")
print(f" {'dataset':22s} {'k-means':>8s} {'DBSCAN':>8s} {'eps used':>9s}")
for name, (Xa, ya) in [
("two moons", make_moons(n_samples=600, noise=0.06, random_state=1)),
("concentric circles", make_circles(n_samples=600, noise=0.05, factor=0.45,
random_state=1)),
("well-separated blobs", make_blobs(n_samples=600, centers=3,
cluster_std=0.7, random_state=1)),
]:
k = len(np.unique(ya))
km = KMeans(k, n_init=10, random_state=0).fit(Xa)
eps = 0.30 if "blob" in name else 0.22
dbx = DBSCAN(eps=eps, min_samples=8).fit(StandardScaler().fit_transform(Xa))
print(f" {name:22s} {adjusted_rand_score(ya, km.labels_):8.3f}"
f" {adjusted_rand_score(ya, dbx.labels_):8.3f} {eps:9.2f}")
print(" DBSCAN never computes a centre, so it has no notion of a convex")
print(" cluster. It grows clusters by CONNECTIVITY, which is why a crescent")
print(" or a ring is no harder for it than a blob.")
# --- CHOOSING eps: THE k-DISTANCE PLOT. ---
# Sort every point's distance to its min_samples-th neighbour. The knee is where
# points stop being in dense regions, which is a principled eps.
MIN_SAMPLES = 8
nn = NearestNeighbors(n_neighbors=MIN_SAMPLES).fit(X)
kdist = np.sort(nn.kneighbors(X)[0][:, -1])
print(f"\nCHOOSING eps FROM THE {MIN_SAMPLES}-DISTANCE PLOT")
print(f" {'percentile':>11s} {'8th-NN distance':>16s}")
for q in [50, 75, 90, 95, 97, 99]:
print(f" {q:11d} {np.percentile(kdist, q):16.4f}")
print(" The curve is flat then turns up sharply. The knee -- here around 0.16")
print(" to 0.20 -- separates dense-region points from the tail, and that is a")
print(" defensible eps. Note this is a HEURISTIC, not an optimisation.")
print("\nSENSITIVITY TO eps (min_samples=8)")
print(f" {'eps':>6s} {'clusters':>9s} {'noise %':>8s} {'ARI':>7s}")
for eps in [0.08, 0.12, 0.18, 0.25, 0.40, 0.80]:
d = DBSCAN(eps=eps, min_samples=MIN_SAMPLES).fit(X)
nk = len(set(d.labels_) - {-1})
print(f" {eps:6.2f} {nk:9d} {np.mean(d.labels_ == -1):8.1%}"
f" {adjusted_rand_score(y[real], d.labels_[real]):7.3f}")
print(" Too small: everything is noise. Too large: everything is one cluster.")
print(" The useful range is narrow, and eps is in DATA UNITS, so scaling the")
print(" features changes the right value completely.")
print("\nSENSITIVITY TO min_samples (eps=0.18)")
print(f" {'min_samples':>12s} {'clusters':>9s} {'noise %':>8s} {'ARI':>7s}")
for ms in [3, 5, 8, 15, 30, 60]:
d = DBSCAN(eps=0.18, min_samples=ms).fit(X)
print(f" {ms:12d} {len(set(d.labels_) - {-1}):9d}"
f" {np.mean(d.labels_ == -1):8.1%}"
f" {adjusted_rand_score(y[real], d.labels_[real]):7.3f}")
print(" min_samples sets how many points make a 'dense' region. A common")
print(" starting rule is 2 * n_features, then tune with the k-distance plot.")
# --- THE REAL LIMITATION: ONE eps FOR THE WHOLE DATASET. ---
# Two clusters at very different densities cannot share a single eps. Report
# what happens to EACH cluster, because a single ARI hides the mechanism.
dense = rng.normal([0, 0], 0.20, (400, 2)) # tight
sparse = rng.normal([3.0, 0], 1.00, (400, 2)) # diffuse, 5x the spread
vd = np.vstack([dense, sparse])
yv = np.r_[np.zeros(400), np.ones(400)].astype(int)
print("\nVARYING DENSITY: a tight cluster and a diffuse one, 5x the spread")
print(f" {'eps':>6s} {'clusters':>9s} {'dense noise':>12s} {'sparse noise':>13s}"
f" {'merged?':>8s}")
for eps in [0.12, 0.20, 0.35, 0.55, 0.90]:
d = DBSCAN(eps=eps, min_samples=8).fit(vd)
ld, ls = d.labels_[:400], d.labels_[400:]
# A few tail points always stray across, so test the MODAL label of each
# true cluster: the two are genuinely merged only if those coincide.
def modal(v):
keep = v[v != -1]
return np.bincount(keep).argmax() if len(keep) else -1
shared = modal(ld) == modal(ls) != -1
print(f" {eps:6.2f} {len(set(d.labels_) - {-1}):9d}"
f" {np.mean(ld == -1):12.1%} {np.mean(ls == -1):13.1%}"
f" {'YES' if shared else 'no':>8s}")
print(" At eps=0.12 the tight cluster is clean and 93% of the diffuse one is")
print(" thrown away as noise. At 0.20 the diffuse cluster shatters into ten")
print(" fragments. Only around 0.35 do both survive, and even there 13% of the")
print(" diffuse cluster is still noise -- and a little further, at 0.55, the")
print(" two merge outright. The workable window is narrow and it exists only")
print(" because eps is ONE global density threshold applied to two densities.")
# HDBSCAN varies the density threshold per cluster, which removes eps entirely.
print("\nHDBSCAN BUILDS A HIERARCHY OVER DENSITIES INSTEAD OF FIXING ONE")
h = HDBSCAN(min_cluster_size=25).fit_predict(vd)
print(f" HDBSCAN(min_cluster_size=25): clusters {len(set(h) - {-1})}"
f" noise {np.mean(h == -1):.1%} ARI {adjusted_rand_score(yv, h):.3f}")
print(f" dense cluster noise {np.mean(h[:400] == -1):.1%}"
f" diffuse cluster noise {np.mean(h[400:] == -1):.1%}")
print(" Both clusters recovered with no eps to tune -- only a minimum cluster")
print(" size, which is a far more intuitive thing to specify. Note it still")
print(" calls 27% of the diffuse cluster's tail noise: HDBSCAN removes the")
print(" tuning problem, it does not make a diffuse cluster dense. OPTICS does")
print(" something similar by producing a reachability plot you cut by hand,")
print(" but it is fiddlier and HDBSCAN is the sensible default today.")
THE THREE POINT TYPES (eps = radius, min_samples = 4 here)
. . . CORE: >= 4 points within eps
. .C. . -> can extend the cluster
. . B BORDER: within eps of a core point,
but not core itself
N NOISE: neither -- no cluster at all
core points 608 (>= 8 neighbours within eps=0.18)
border points 4
noise points 28
clusters found 2 true clusters 2
ARI on genuine points 1.000
28 of the 40 injected noise points were labelled noise
ARBITRARY SHAPES: WHAT NOT HAVING A CENTRE BUYS
dataset k-means DBSCAN eps used
two moons 0.255 1.000 0.22
concentric circles -0.002 0.932 0.22
well-separated blobs 1.000 1.000 0.30
k-means on circles: DBSCAN on circles:
o o o o o o o o o o
o x x o o x x o
o x x o split o x x o grown by
o x x o down o x x o connectivity:
o o o o o the o o o o o ring and disc
| middle are separate
CHOOSING eps: THE k-DISTANCE PLOT
percentile 8th-NN distance
50 0.0785
75 0.0978
90 0.1295
95 0.1722 <- the curve starts to turn
97 0.6763 <- and now it has turned
99 1.1679
8th-NN
dist | /
| /
| _/ <- knee: eps goes here
|________________________/
+-------------------------------> points, sorted
SENSITIVITY: THE USEFUL RANGE IS NARROW
eps clusters noise % ARI
0.08 10 21.6% 0.164 fragmented
0.12 2 5.3% 0.987
0.18 2 4.4% 1.000 correct
0.25 2 3.8% 1.000
0.40 1 3.4% 0.000 all one cluster
0.80 1 0.9% 0.000
min_samples is far more forgiving:
3, 5, 8, 15 all give ARI 1.000; 30 gives 0.511; 60 gives all noise
THE REAL LIMITATION: ONE eps, TWO DENSITIES
eps clusters dense noise sparse noise merged?
0.12 4 3.0% 92.8% no
0.20 10 0.0% 60.5% no
0.35 2 0.0% 13.0% no
0.55 1 0.0% 4.0% YES
0.90 1 0.0% 0.5% YES
( tight ) . . . . . eps small enough for the
dense . diffuse . . tight cluster discards the
. . . . diffuse one as noise; eps
large enough for the diffuse
one swallows both
HDBSCAN: A HIERARCHY OVER DENSITIES, NO eps AT ALL
HDBSCAN(min_cluster_size=25): clusters 2 noise 13.5% ARI 0.790
dense cluster noise 0.0%
diffuse cluster noise 27.0% <- still noisy: HDBSCAN removes the
tuning problem, not the diffuseness
| k-means | Ward | DBSCAN | HDBSCAN | |
|---|---|---|---|---|
Needs k in advance | Yes | No, cut later | No | No |
| Cluster shape | Convex blobs | Compact blobs | Any connected shape | Any connected shape |
| Can label noise | No | No | Yes | Yes |
| Handles varying density | No | Partly | No | Yes |
| Main parameters | k | linkage, k | eps, min_samples | min_cluster_size |
| Deterministic | No, init-dependent | Yes | Yes | Yes |
| Scales to millions | Yes | No | With an index | With an index |
Key Takeaways
- DBSCAN defines a cluster as a connected dense region, so it never computes a centre and has no shape constraint at all.
- Core, border, and noise are the three point types, and everything about the algorithm follows from that classification.
- Being able to label a point as noise prevents outliers from being smeared into whichever cluster is nearest.
epsis chosen from the knee of the k-distance plot, is expressed in data units, and therefore must be retuned after any rescaling.- The useful
epsrange is narrow;min_samplesis much more forgiving, with2 * n_featuresa reasonable start.- A single global
epscannot describe data with two different densities, which is DBSCAN's fundamental limitation.- HDBSCAN builds a hierarchy over densities and needs only
min_cluster_size, making it the sensible modern default.
🧪 Practice
- Fit DBSCAN and report the counts of core, border, and noise points, checking them against
core_sample_indices_.- Compare DBSCAN against k-means on two moons, concentric circles, and blobs, and explain the pattern in one sentence.
- Build the k-distance plot, identify the knee numerically, and check that the
epsit suggests performs well.- Sweep
epsandmin_sampleson a grid, recording cluster count, noise fraction, and ARI, and describe the shape of the good region.- Multiply one feature by 100 and show that the previously good
epsnow fails, then fix it with standardisation.- Construct two clusters with a fivefold difference in spread and show that no single
epshandles both, reporting per-cluster noise rates.- Run HDBSCAN on the same data and compare, noting what it fixes and what it does not.
- Interview: DBSCAN labels 40% of your customer data as noise. Name three distinct explanations and how you would distinguish them. (Hint: one is a parameter, one is the feature scaling, and one is that the data may genuinely not have dense regions.)
Gaussian Mixture Models
k-means gives every point a hard label. For a customer sitting exactly between two segments, that label is a fabrication: the model is 51% confident and reports 100%. A Gaussian mixture model fixes this by being an actual probability model of how the data was generated.
The generative story is simple. To produce a point: pick a component j with
probability pi_j, then draw from a Gaussian with mean mu_j and covariance
Sigma_j. Fitting inverts the story -- given the points, recover the weights,
means, and covariances. Because you never observe which component produced
which point, the natural output is a responsibility: the posterior
probability that component j generated this point. Those are soft
assignments, and they sum to one per point.
Fitting uses expectation-maximisation, which alternates exactly like Lloyd's algorithm:
- E step: given current parameters, compute each component's responsibility for each point. 2. M step: given responsibilities, refit each component's weight, mean, and covariance as a weighted average over all points.
The log-likelihood never decreases, which guarantees convergence to a local
optimum -- the same guarantee and the same caveat as k-means, and n_init
exists for the same reason.
The unification is worth internalising: k-means is a GMM with the covariance
constrained to a shared sphere and the variance driven to zero. In that
limit, responsibilities collapse to 0/1 and the M step becomes "average the
members". Every restriction k-means suffers is a constraint this family can
relax, and covariance_type is the dial:
covariance_type | Each component gets | Can represent |
|---|---|---|
spherical | One variance | Circles of differing size |
diag | One variance per feature | Axis-aligned ellipses |
tied | One shared full matrix | Identically-oriented ellipses |
full | Its own full matrix | Arbitrary ellipses, independently oriented |
Two things a GMM gives you that k-means cannot.
Soft labels are actionable. In the demonstration, 7.2% of points have maximum responsibility below 0.7. Those are precisely the ambiguous cases you might route to a human, exclude from a campaign, or flag as uncertain -- and k-means cannot identify them.
A likelihood means real model selection. k-means inertia falls
monotonically in k and therefore cannot choose k. A GMM has a likelihood,
so BIC and AIC apply, penalising parameters against fit. In the
demonstration BIC is minimised at exactly the true k = 4.
Now the caution, which is the most valuable part of this topic. Being able
to represent a structure is not the same as finding it. The demonstration
builds two thin horizontal bands that do not overlap in y at all, separated
by 7.5 standard deviations, spanning a huge range in x. A full-covariance
GMM represents this perfectly -- seeded at the true means it recovers ARI
1.000 with log-likelihood -3523. But with default settings and ten restarts it
scores ARI 0.000 at log-likelihood -4061, and only 1 of 60 random
initialisations finds the right answer. The reason: scikit-learn initialises
EM with k-means, so EM inherits k-means's bias toward splitting the
high-variance direction and never escapes that basin. A 537-unit better
optimum exists and the search does not reach it.
Two more practical notes. Components can degenerate: one can collapse onto
a handful of near-identical points, driving its covariance determinant toward
zero and the likelihood toward infinity without describing anything.
reg_covar adds a ridge to every covariance, the same fix as LDA's shrinkage
in chapter 10.2. And a Bayesian (variational) GMM with a sparse Dirichlet
prior can switch off components it does not need: asked for 10 on data with 4,
it puts weight above 0.02 on exactly 4.
import numpy as np
import pandas as pd
from sklearn.mixture import GaussianMixture, BayesianGaussianMixture
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.metrics import adjusted_rand_score
rng = np.random.default_rng(109)
# --- SOFT VS HARD ASSIGNMENT: the point of a mixture model. ---
# k-means says "this point is in cluster 2". A GMM says "70% cluster 2, 30%
# cluster 1", which is the honest answer for a point between two groups.
X, truth = make_blobs(n_samples=600, centers=[[0, 0], [3.2, 0]],
cluster_std=1.1, random_state=4)
gm = GaussianMixture(n_components=2, covariance_type="full",
random_state=0).fit(X)
km = KMeans(2, n_init=10, random_state=0).fit(X)
# Component order is arbitrary, so sort by mean x to make the output readable.
ordr = np.argsort(gm.means_[:, 0])
proba = gm.predict_proba(X)[:, ordr]
conf = proba.max(1)
print("SOFT ASSIGNMENT: the responsibility of each component for each point")
order = np.argsort(conf)
print(f" {'x':>17s} {'P(left)':>9s} {'P(right)':>9s} {'k-means label':>14s}")
for i in list(order[:3]) + list(order[-2:]):
print(f" {np.array2string(X[i], precision=2, floatmode='fixed'):>17s}"
f" {proba[i, 0]:9.3f} {proba[i, 1]:9.3f} {km.labels_[i]:14d}")
print(f"\n points with max responsibility below 0.7: {np.mean(conf < 0.7):.1%}")
print(f" points with max responsibility above 0.99: {np.mean(conf > 0.99):.1%}")
print(" The ambiguous points in the overlap are exactly where a hard label is a")
print(" fabrication. A GMM lets you find them, threshold on confidence, or")
print(" route them to a human -- none of which k-means supports.")
# --- EM: THE TWO-STEP LOOP, BY HAND. ---
# E step: given parameters, compute each component's responsibility per point.
# M step: given responsibilities, refit each component's weight/mean/covariance.
# The log-likelihood never decreases, exactly as inertia never increases.
def em_gmm(X, k, seed, iters=60, tol=1e-6):
r = np.random.default_rng(seed)
n, p = X.shape
mu = X[r.choice(n, k, replace=False)]
cov = np.array([np.cov(X.T) for _ in range(k)])
pi = np.full(k, 1.0 / k)
lls = []
for it in range(iters):
# ---- E step: weighted Gaussian densities, normalised per row ----
dens = np.empty((n, k))
for j in range(k):
d = X - mu[j]
inv = np.linalg.inv(cov[j])
expo = -0.5 * np.einsum("ij,jk,ik->i", d, inv, d)
norm = 1.0 / np.sqrt((2 * np.pi) ** p * np.linalg.det(cov[j]))
dens[:, j] = pi[j] * norm * np.exp(expo)
total = dens.sum(1, keepdims=True)
resp = dens / total # responsibilities, rows sum to 1
lls.append(float(np.log(total).sum())) # the log-likelihood
# ---- M step: each component refits on its weighted share of the data --
Nk = resp.sum(0)
pi = Nk / n
mu = (resp.T @ X) / Nk[:, None]
for j in range(k):
d = X - mu[j]
cov[j] = (resp[:, j, None] * d).T @ d / Nk[j]
cov[j] += 1e-6 * np.eye(p) # ridge: keeps it invertible
if it > 0 and lls[-1] - lls[-2] < tol:
break
return resp, mu, cov, pi, lls
resp, mu, cov, pi, lls = em_gmm(X, 2, seed=0)
print("\nEM: THE LOG-LIKELIHOOD NEVER DECREASES")
for i in [0, 1, 2, 5, len(lls) - 1]:
if i < len(lls):
print(f" iteration {i:2d}: log-likelihood {lls[i]:11.4f}")
print(f" hand-rolled final {lls[-1]:.4f}")
print(f" sklearn {gm.score(X) * len(X):.4f}")
print(" Same guarantee as k-means, same caveat: a local optimum. n_init exists")
print(" for the same reason.")
# --- k-MEANS IS A GMM WITH THE COVARIANCE FORCED TO A SHARED SPHERE. ---
print("\nTHE COVARIANCE OPTION IS THE WHOLE FAMILY")
print(f" {'covariance_type':18s} {'params':>7s} {'free shape':>11s} meaning")
meanings = {
"spherical": "one variance per component",
"diag": "axis-aligned ellipse per component",
"tied": "one shared full covariance",
"full": "its own full covariance per component",
}
for ct in ["spherical", "diag", "tied", "full"]:
g = GaussianMixture(2, covariance_type=ct, random_state=0).fit(X)
print(f" {ct:18s} {g._n_parameters():7d} {'yes' if ct == 'full' else 'no':>11s}"
f" {meanings[ct]}")
print(" k-means is the limit of a SPHERICAL GMM with equal weights as the")
print(" variance goes to zero: responsibilities collapse to 0/1 and the M step")
print(" becomes 'average the members'. Everything k-means cannot do -- ellipses,")
print(" unequal spread, soft labels -- is a constraint this family can relax.")
# --- CAPABILITY IS NOT THE SAME AS FINDABILITY. ---
# Two thin horizontal bands, 7.5 standard deviations apart in y but spanning a
# huge range in x. A full-covariance GMM can represent this exactly. Whether EM
# FINDS it is a separate question, and the answer is usually no.
def bands(n, seed):
r = np.random.default_rng(seed)
a = r.normal(0, 1, (n // 2, 2)) @ np.array([[6.0, 0], [0, 0.4]])
b = a.copy() + np.array([0.0, 3.0]) # shifted in y only
return np.vstack([a, b]), np.r_[np.zeros(n // 2), np.ones(n // 2)].astype(int)
Xe, ye = bands(800, 3)
print("\nTWO THIN BANDS: x spread 6.0, y spread 0.4, separated by 3.0 in y")
print(f" the bands do not overlap in y at all:"
f" band 0 reaches {Xe[ye == 0, 1].max():.2f},"
f" band 1 starts at {Xe[ye == 1, 1].min():.2f}")
print(f"\n {'method':40s} {'ARI':>7s} {'log-lik':>10s}")
km_ari = adjusted_rand_score(
ye, KMeans(2, n_init=10, random_state=0).fit_predict(Xe))
print(f" {'k-means (cannot represent this)':40s}"
f" {km_ari:7.3f} {'--':>10s}")
for ct in ["spherical", "diag", "tied", "full"]:
g = GaussianMixture(2, covariance_type=ct, n_init=10, random_state=0).fit(Xe)
print(f" {'GMM ' + ct + ', default init, n_init=10':40s}"
f" {adjusted_rand_score(ye, g.predict(Xe)):7.3f}"
f" {g.score(Xe) * len(Xe):10.1f}")
# Now hand EM the right starting point and watch it succeed.
seeded = GaussianMixture(2, covariance_type="full",
means_init=[[0, 0], [0, 3]], random_state=0).fit(Xe)
print(f" {'GMM full, means_init AT the truth':40s}"
f" {adjusted_rand_score(ye, seeded.predict(Xe)):7.3f}"
f" {seeded.score(Xe) * len(Xe):10.1f} <- much better")
hits = 0
for sd in range(60):
g = GaussianMixture(2, covariance_type="full", init_params="random",
n_init=1, random_state=sd, max_iter=500).fit(Xe)
hits += adjusted_rand_score(ye, g.predict(Xe)) > 0.9
print(f"\n of 60 random single initialisations, {hits} found the correct split")
print(" The full-covariance GMM CAN represent these bands, and when seeded at")
print(" the truth it recovers them perfectly with a log-likelihood 537 units")
print(" better than anything the default search found. But sklearn initialises")
print(" EM with k-means by default, so it inherits k-means's bias toward")
print(" splitting the high-variance direction and never escapes that basin.")
print(" 'The model class is capable' and 'the fitting procedure finds it' are")
print(" different claims, and only the second one matters in practice.")
# --- CHOOSING k WITH BIC: a real likelihood means a real model-selection rule.
Xk, yk = make_blobs(n_samples=900, centers=4, cluster_std=1.0, random_state=13)
print("\nBIC AND AIC SELECT k, BECAUSE A GMM HAS A LIKELIHOOD")
print(f" {'k':>3s} {'log-lik':>11s} {'params':>7s} {'BIC':>11s} {'AIC':>11s}")
bics = []
for k in range(1, 9):
g = GaussianMixture(k, covariance_type="full", n_init=5,
random_state=0).fit(Xk)
bics.append(g.bic(Xk))
flag = ""
print(f" {k:3d} {g.score(Xk) * len(Xk):11.1f} {g._n_parameters():7d}"
f" {g.bic(Xk):11.1f} {g.aic(Xk):11.1f}{flag}")
print(f" BIC is minimised at k={int(np.argmin(bics)) + 1} (true k = 4)")
print(" This is a genuine advantage over k-means, whose inertia falls")
print(" monotonically in k and therefore cannot select it.")
# --- THE FAILURE MODE: DEGENERATE COMPONENTS. ---
# A component can collapse onto a few points, driving its covariance toward
# singular and the likelihood toward infinity. reg_covar is the guard.
print("\nDEGENERACY: a component collapsing onto a handful of points")
tiny = np.vstack([rng.normal([0, 0], 1.0, (300, 2)),
np.repeat([[6.0, 6.0]], 4, axis=0)
+ rng.normal(0, 1e-3, (4, 2))]) # 4 near-identical points
for reg in [1e-6, 1e-3, 1e-1]:
g = GaussianMixture(3, covariance_type="full", reg_covar=reg,
random_state=2, n_init=3).fit(tiny)
dets = [np.linalg.det(c) for c in g.covariances_]
print(f" reg_covar={reg:<7g} log-lik {g.score(tiny) * len(tiny):9.1f}"
f" smallest covariance determinant {min(dets):.3e}")
print(" With a small reg_covar one component wraps the four duplicated points")
print(" and its determinant collapses, inflating the likelihood without")
print(" describing any structure. reg_covar adds a ridge to every covariance,")
print(" which is the same fix as LDA's shrinkage in chapter 10.2.")
# --- A VARIATIONAL GMM CAN SWITCH OFF UNNEEDED COMPONENTS. ---
print("\nBAYESIAN GMM: ask for 10 components on data that has 4")
bg = BayesianGaussianMixture(n_components=10, covariance_type="full",
weight_concentration_prior=0.01,
random_state=0, max_iter=500).fit(Xk)
w = np.sort(bg.weights_)[::-1]
print(" fitted weights, largest first:", np.round(w, 4))
print(f" components with weight > 0.02: {(w > 0.02).sum()} (true k = 4)")
print(" A sparse Dirichlet prior pushes surplus weights toward zero, so the")
print(" model prunes components it does not need. It does not remove the need")
print(" to check the answer, but it is more forgiving than guessing k exactly.")
SOFT ASSIGNMENT: RESPONSIBILITIES, NOT LABELS
x P(left) P(right) k-means label
[ 1.52 -2.17] 0.506 0.494 1
[ 1.52 -2.12] 0.507 0.493 1
[1.48 1.54] 0.508 0.492 1
[6.23 0.23] 0.000 1.000 0
[6.26 0.45] 0.000 1.000 0
7.2% of points have max responsibility below 0.70
54.8% have max responsibility above 0.99
o o o | x x x k-means: a hard line. every point gets a
o o o | x x x label with no confidence attached.
o o o ? x x x GMM: the '?' region carries 0.5/0.5, which
o o o ? x x x is the honest answer for those points.
EM: THE LOG-LIKELIHOOD NEVER DECREASES
iteration 0: log-likelihood -2818.5964
iteration 1: log-likelihood -2110.3981
iteration 2: log-likelihood -2104.3388
iteration 5: log-likelihood -2090.1153
iteration 59: log-likelihood -2075.7922
THE COVARIANCE OPTION IS THE WHOLE FAMILY
covariance_type params meaning
spherical 7 one variance per component
diag 9 axis-aligned ellipse per component
tied 8 one shared full covariance
full 11 its own full covariance per component
spherical diag tied full
( ) ( ) / / / __
( ) ( ) / / ( ( )
\
CAPABILITY IS NOT FINDABILITY
(two thin bands: x spread 6.0, y spread 0.4, separated by 3.0 in y;
band 0 reaches y=1.33, band 1 starts at y=1.67 -- no overlap at all)
method ARI log-lik
k-means (cannot represent this) -0.001 --
GMM spherical, default init, n_init=10 -0.001 -4411.7
GMM diag, default init, n_init=10 -0.001 -4061.3
GMM tied, default init, n_init=10 -0.000 -4061.5
GMM full, default init, n_init=10 -0.000 -4061.2
GMM full, means_init AT the truth 1.000 -3523.0
of 60 random single initialisations, 1 found the correct split
what EM keeps finding what exists and it misses
==========|========== =====================
==========|========== =====================
split the long axis split between the bands
log-lik -4061 log-lik -3523
BIC SELECTS k, BECAUSE A GMM HAS A LIKELIHOOD
k log-lik params BIC AIC
1 -5019.4 5 10072.9 10048.9
2 -4191.4 11 8457.7 8404.8
3 -3961.4 17 8038.4 7956.7
4 -3778.7 23 7713.9 7603.5 <- BIC minimum
5 -3776.9 29 7751.0 7611.8
6 -3774.6 35 7787.2 7619.1
8 -3770.0 47 7859.8 7634.1
Note the log-likelihood keeps rising past k=4 while BIC turns:
that is the parameter penalty doing its job.
DEGENERACY AND THE reg_covar GUARD
reg_covar=1e-06 log-lik -810.4 smallest cov determinant 2.472e-12
reg_covar=0.001 log-lik -835.0 smallest cov determinant 1.001e-06
reg_covar=0.1 log-lik -853.9 smallest cov determinant 1.000e-02
A component wrapped 4 near-identical points; its determinant collapsed
and the likelihood inflated without describing any structure.
A BAYESIAN GMM PRUNES COMPONENTS IT DOES NOT NEED
asked for 10 components on data with 4:
weights [0.2501 0.2497 0.249 0.2479 0.0011 0.0011 0.0011 0. 0. 0.]
components with weight > 0.02: 4
Key Takeaways
- A GMM is a generative probability model, so its natural output is a responsibility per component rather than a hard label.
- EM alternates computing responsibilities and refitting weighted parameters, and the log-likelihood never decreases.
- k-means is the limiting case of a spherical, equal-weight GMM with zero variance, so the whole family relaxes k-means's constraints.
- Soft assignments identify the genuinely ambiguous points, which is actionable information k-means cannot provide.
- Having a likelihood makes BIC and AIC available for selecting
k, which k-means inertia cannot do.- A model class being able to represent a structure does not mean EM will find it: default k-means initialisation inherits k-means's bias and can miss a far better optimum entirely.
- Components can degenerate onto a few points and inflate the likelihood, which
reg_covarprevents by ridging every covariance.- A variational GMM with a sparse prior prunes unneeded components, making the choice of
n_componentsmore forgiving.
🧪 Practice
- Implement EM for a two-component GMM with explicit E and M steps and confirm the log-likelihood is non-decreasing every iteration.
- Fit a GMM to overlapping blobs and report what fraction of points have maximum responsibility below 0.7, then inspect where those points sit.
- Fit all four
covariance_typevalues on elliptical clusters and compare parameter counts against ARI.- Compute BIC for
kfrom 1 to 10 and confirm it is minimised at the true number of components, then show that inertia cannot do this.- Build two thin well-separated bands, fit with default settings, then refit with
means_initat the truth and compare log-likelihoods.- Duplicate four points many times over and fit with
reg_covarat 1e-6 and 1e-2, reporting the smallest covariance determinant.- Fit a
BayesianGaussianMixturewith ten components to four-cluster data and report the fitted weights.- Interview: Your GMM reports a higher log-likelihood than a competing fit but its clusters look obviously wrong when plotted. Give two explanations. (Hint: one involves a component collapsing, the other involves what a higher likelihood does and does not imply about matching your intent.)
Choosing the Number of Clusters
This is the question clustering cannot answer for you, and the honest framing
matters: there is often no true k. Customers do not come
pre-partitioned. Asking for 3 segments or 30 are both defensible depending on
how many distinct marketing treatments you can actually run. The methods below
narrow the range and rule out bad answers; they do not discover a fact.
The elbow method plots inertia against k and looks for the bend. Its
fundamental problem is that inertia falls monotonically -- at k = n it
is zero -- so there is no minimum to find, only a change in slope. Two
weaknesses follow. Real data often has no visible bend at all. And "look for
the bend" is not reproducible between analysts, though you can make it so:
take the point farthest from the straight line joining the curve's endpoints.
In the demonstration that procedure returns k = 3 when the truth is 5.
Silhouette is better because it has a genuine maximum. For each point,
compare a, the mean distance to its own cluster, with b, the mean distance
to the nearest other cluster: s = (b - a) / max(a, b), ranging from -1 to 1.
Average it over all points and you can maximise it. The per-point scores are
more useful than the average: a negative score means that point is closer to
another cluster than its own, a local diagnostic the mean hides. In the
demonstration silhouette peaks at k = 4, also not 5.
The gap statistic is the only method here that asks a fundamentally
different question. Cluster your data, then cluster uniform noise over the
same bounding box, and compare. The gap is log(W_null) - log(W_real). If
your inertia is not much lower than what no structure would produce, you have
not found structure. Tibshirani's one-standard-error rule picks the smallest
k whose gap is within one standard error of the next. In the demonstration
it returns k = 5 -- correct, where the other two were not.
That is one dataset, not a general ranking. The reason the gap statistic deserves special respect is the honesty test: run every method on uniform noise, which has no clusters at all.
- Inertia still falls smoothly, so an elbow can be read anywhere.
- Silhouette peaks at
k = 4with a comfortable 0.408 -- it reports structure confidently. - The gap statistic returns
k = 1, and its largest gap is 0.049 against 1.806 on real structure, a factor of 37.
Only the gap statistic has a null model, so only its number shrinks when the structure is absent. Elbow and silhouette compare candidate partitions with each other, so they always return a confident-looking answer.
BIC and AIC work when your model has a likelihood, which means a GMM rather than k-means. When applicable this is the cleanest option available, as the previous topic showed.
Stability is the check to reach for most often. Cluster two overlapping
subsamples independently and compare their assignments on the rows both
contain. If a k is real, the two runs agree; if you tuned to noise, they do
not. In the demonstration, stability on real structure is above 0.99 up to k = 5 and falls away past it.
But read the noise column carefully, because it is the caveat that makes this
topic worth reading twice: stability reaches 0.948 on pure uniform noise at
k = 4, because k-means cuts a uniform square into four quadrants extremely
reproducibly. Stability measures whether the algorithm's geometry is
reproducible, which is not the same as whether clusters exist. It is a strong
check on a candidate k and a weak check on whether to cluster at all -- for
that question you need a null.
The practical recipe: run several methods, take the range they agree on,
verify it with stability, check against a null, and then let the downstream
use decide. A k that produces segments nobody can act on differently is
the wrong k whatever the silhouette says.
import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
from sklearn.mixture import GaussianMixture
from sklearn.datasets import make_blobs
from sklearn.metrics import silhouette_score, adjusted_rand_score
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(113)
X, truth = make_blobs(n_samples=1000, centers=5, cluster_std=1.0, random_state=17)
KS = range(1, 11)
# --- THE ELBOW METHOD, AND WHY IT SO OFTEN HAS NO ELBOW. ---
# Inertia falls monotonically in k: at k=n it reaches zero. So it can never
# select k by itself; you are looking for a change in SLOPE, not a minimum.
inertias = [KMeans(k, n_init=10, random_state=0).fit(X).inertia_ for k in KS]
print("INERTIA IS MONOTONE IN k, SO YOU LOOK FOR A KINK, NOT A MINIMUM")
print(f" {'k':>3s} {'inertia':>10s} {'drop':>9s} {'drop ratio':>11s}")
for i, k in enumerate(KS):
drop = inertias[i - 1] - inertias[i] if i else float("nan")
ratio = drop / (inertias[i - 2] - inertias[i - 1]) if i >= 2 else float("nan")
print(f" {k:3d} {inertias[i]:10.1f} {drop:9.1f}"
f" {ratio if i >= 2 else float('nan'):11.2f}")
# A reproducible way to pick the elbow: the point farthest from the line joining
# the first and last points of the curve. It removes the eyeballing.
pts = np.column_stack([list(KS), inertias])
p0, p1 = pts[0], pts[-1]
line = (p1 - p0) / np.linalg.norm(p1 - p0)
d = pts - p0
dist = np.linalg.norm(d - np.outer(d @ line, line), axis=1)
print(f" knee by maximum distance to the endpoint chord: k ="
f" {int(pts[dist.argmax(), 0])} (true k = 5)")
# --- SILHOUETTE: a per-point score with a real interpretation. ---
# s = (b - a) / max(a, b), where a is the mean distance to your own cluster and
# b the mean distance to the nearest OTHER cluster. Range -1 to 1.
print("\nSILHOUETTE HAS A MAXIMUM, SO IT CAN SELECT k DIRECTLY")
print(f" {'k':>3s} {'silhouette':>11s} {'% points < 0':>13s}")
sils = []
for k in range(2, 11):
lab = KMeans(k, n_init=10, random_state=0).fit_predict(X)
from sklearn.metrics import silhouette_samples
ss = silhouette_samples(X, lab)
sils.append(ss.mean())
print(f" {k:3d} {ss.mean():11.4f} {np.mean(ss < 0):13.1%}")
print(f" silhouette is maximised at k = {int(np.argmax(sils)) + 2} (true k = 5)")
print(" Negative per-point scores mean a point is closer to another cluster")
print(" than its own -- a direct, local diagnostic that a mean score hides.")
# --- THE GAP STATISTIC: compare against what NO structure would give. ---
# Cluster uniform noise over the same bounding box. If real inertia is not much
# below the noise baseline, the structure you found is not there.
def gap_statistic(X, k_max=10, B=20, seed=0):
r = np.random.default_rng(seed)
lo, hi = X.min(0), X.max(0)
rows = []
for k in range(1, k_max + 1):
real = np.log(KMeans(k, n_init=10, random_state=0).fit(X).inertia_)
ref = [np.log(KMeans(k, n_init=10, random_state=0)
.fit(r.uniform(lo, hi, X.shape)).inertia_) for _ in range(B)]
rows.append((k, real, np.mean(ref), np.mean(ref) - real,
np.std(ref) * np.sqrt(1 + 1 / B)))
return pd.DataFrame(rows, columns=["k", "log_W", "log_W_null", "gap", "se"])
g = gap_statistic(X)
print("\nGAP STATISTIC: real inertia vs uniform-noise inertia")
print(f" {'k':>3s} {'log W':>8s} {'log W null':>11s} {'gap':>8s} {'se':>7s}")
for _, row in g.iterrows():
print(f" {int(row.k):3d} {row.log_W:8.3f} {row.log_W_null:11.3f}"
f" {row.gap:8.3f} {row.se:7.3f}")
# Tibshirani's rule: the smallest k whose gap is within one se of the next gap.
choice = next((int(g.k[i]) for i in range(len(g) - 1)
if g.gap[i] >= g.gap[i + 1] - g.se[i + 1]), None)
print(f" Tibshirani's one-standard-error rule picks k = {choice} (true k = 5)")
# --- THE CRITICAL TEST EVERY METHOD SHOULD PASS: DATA WITH NO CLUSTERS. ---
print("\nTHE HONESTY TEST: uniform noise, which has NO clusters")
noise = rng.uniform(0, 1, (1000, 2))
ns_inertia = [KMeans(k, n_init=10, random_state=0).fit(noise).inertia_
for k in KS]
print(f" {'k':>3s} {'inertia':>9s} {'silhouette':>11s}")
for i, k in enumerate(KS):
if k == 1:
print(f" {k:3d} {ns_inertia[i]:9.2f} {'--':>11s}")
continue
lab = KMeans(k, n_init=10, random_state=0).fit_predict(noise)
print(f" {k:3d} {ns_inertia[i]:9.2f} {silhouette_score(noise, lab):11.4f}")
gn = gap_statistic(noise)
ns_sil = [silhouette_score(noise, KMeans(k, n_init=10,
random_state=0).fit_predict(noise))
for k in range(2, 11)]
gn_choice = next((int(gn.k[i]) for i in range(len(gn) - 1)
if gn.gap[i] >= gn.gap[i + 1] - gn.se[i + 1]), None)
print(f"\n elbow: inertia falls smoothly, so a 'kink' can be read anywhere")
print(f" silhouette: peaks at k={int(np.argmax(ns_sil)) + 2}"
f" with a comfortably positive {max(ns_sil):.3f}")
print(f" gap: largest gap {gn.gap.max():+.3f}, and the rule picks"
f" k={gn_choice}")
print(f"\n Compare the gap magnitudes: {g.gap.max():.3f} on real structure")
print(f" versus {gn.gap.max():.3f} on noise -- a factor of"
f" {g.gap.max() / gn.gap.max():.0f}. The gap statistic is the only one of")
print(" the three that measures distance from NO STRUCTURE, so it is the only")
print(" one whose number shrinks when the structure is absent. Elbow and")
print(" silhouette compare partitions only with each other, so they always")
print(" return a confident-looking answer.")
# --- STABILITY: does the same k reappear under resampling? ---
# If a k is real, clustering two halves of the data should give consistent
# assignments on their overlap. If not, you tuned to noise.
def stability(X, k, reps=20, frac=0.8, seed=0):
r = np.random.default_rng(seed)
scores = []
for _ in range(reps):
i1 = r.choice(len(X), int(frac * len(X)), replace=False)
i2 = r.choice(len(X), int(frac * len(X)), replace=False)
shared = np.intersect1d(i1, i2)
l1 = KMeans(k, n_init=10, random_state=0).fit(X[i1])
l2 = KMeans(k, n_init=10, random_state=0).fit(X[i2])
# compare the two models' labels on the rows both subsamples contain
scores.append(adjusted_rand_score(l1.predict(X[shared]),
l2.predict(X[shared])))
return float(np.mean(scores)), float(np.std(scores))
print("\nSTABILITY UNDER RESAMPLING (mean ARI between two 80% subsamples)")
print(f" {'k':>3s} {'5-blob data':>18s} {'uniform noise':>18s}")
for k in range(2, 9):
m1, s1 = stability(X, k)
m2, s2 = stability(noise, k)
print(f" {k:3d} {f'{m1:.3f} +/- {s1:.3f}':>18s} {f'{m2:.3f} +/- {s2:.3f}':>18s}")
print(" On real structure stability is near 1.0 up to the true k and falls")
print(" away past it. But read the noise column carefully: it reaches 0.948 at")
print(" k=4, because k-means cuts a uniform square into four quadrants very")
print(" reproducibly. Stability measures whether the ALGORITHM'S geometry is")
print(" reproducible, which is not the same as whether clusters exist. It is a")
print(" strong check on a candidate k and a weak check on whether to cluster")
print(" at all -- for that question you need a null, as the gap statistic has.")
THE ELBOW: MONOTONE, SO YOU HUNT A KINK, NOT A MINIMUM
k inertia drop drop ratio
1 50767.1 - -
2 14421.4 36345.7 -
3 6792.1 7629.3 0.21
4 3101.4 3690.7 0.48
5 1994.7 1106.6 0.30
6 1816.6 178.2 0.16
10 1263.4 136.2 1.14
inertia
|*
| \
| \ the "elbow" is wherever your eye lands.
| *__ the endpoint-chord rule gives k=3.
| \___*___
| *___*___*___*
+-------------------------> k
knee by maximum distance to the endpoint chord: k = 3 (true k = 5)
SILHOUETTE: HAS A MAXIMUM, SO IT CAN SELECT DIRECTLY
k silhouette % points < 0
2 0.6266 0.3%
3 0.6394 0.1%
4 0.6923 0.0% <- maximum
5 0.6187 0.1%
8 0.4068 1.3%
s = (b - a) / max(a, b)
a = mean distance to MY cluster
b = mean distance to the NEAREST OTHER cluster
s near +1: well inside my cluster
s near 0: on the boundary
s below 0: closer to another cluster than my own
THE GAP STATISTIC: COMPARE AGAINST NO STRUCTURE
k log W log W null gap se
1 10.835 10.997 0.162 0.020
2 9.576 10.423 0.847 0.025
3 8.824 10.028 1.205 0.016
4 8.040 9.605 1.565 0.019
5 7.598 9.404 1.806 0.020 <- gap peaks
6 7.505 9.217 1.712 0.023
one-standard-error rule picks k = 5 (true k = 5)
THE HONESTY TEST: ALL THREE ON UNIFORM NOISE
k inertia silhouette
2 103.19 0.3478
4 41.52 0.4082 <- silhouette peaks here, confidently
8 20.06 0.3926
elbow: falls smoothly, a kink can be read anywhere
silhouette: peaks at k=4 with a comfortable 0.408
gap: largest gap +0.049, and the rule picks k=1
gap magnitude: 1.806 on real structure vs 0.049 on noise = 37x
STABILITY: DOES THE SAME k REAPPEAR UNDER RESAMPLING?
k 5-blob data uniform noise
2 0.997 +/- 0.004 0.631 +/- 0.414
3 1.000 +/- 0.001 0.815 +/- 0.270
4 1.000 +/- 0.000 0.948 +/- 0.026 <- HIGH on pure noise
5 0.993 +/- 0.008 0.880 +/- 0.047
6 0.869 +/- 0.072 0.867 +/- 0.076
8 0.810 +/- 0.079 0.826 +/- 0.141
k-means cuts a uniform square into 4 quadrants very reproducibly:
+-----+-----+ the same 4 quadrants every resample,
| 1 | 2 | so stability is high -- and there are
+-----+-----+ no clusters here at all.
| 3 | 4 |
+-----+-----+
| Method | Has a maximum | Has a null model | Needs a likelihood | Fooled by uniform noise |
|---|---|---|---|---|
| Elbow (inertia) | No, monotone | No | No | Yes |
| Silhouette | Yes | No | No | Yes |
| Gap statistic | Yes | Yes | No | No |
| BIC / AIC | Yes | Partly | Yes | Mostly not |
| Stability | Yes | No | No | Yes, at some k |
Key Takeaways
- There is often no true
k, so these methods narrow a range rather than discover a fact, and the downstream use should decide.- Inertia falls monotonically in
k, so the elbow method looks for a slope change and frequently finds nothing convincing.- The endpoint-chord rule makes elbow selection reproducible, but reproducible is not the same as correct.
- Silhouette has a genuine maximum, and its per-point scores are a better diagnostic than the average, since negatives flag misassigned points.
- The gap statistic compares against uniform noise, so it is the only common method that can answer "is there any structure at all".
- BIC and AIC are the cleanest option when the model has a likelihood, which means a mixture model rather than k-means.
- Stability under resampling is the most useful practical check on a candidate
k, but it can score highly on pure noise because reproducible geometry is not evidence of clusters.- Run several methods and trust agreement between them more than any single number.
🧪 Practice
- Plot inertia against
kfor 1 to 15 and implement the endpoint-chord knee rule, comparing its answer with your own eye.- Compute silhouette for
kfrom 2 to 10 and also report the fraction of points with negative scores at eachk.- Implement the gap statistic with 20 uniform reference datasets and apply Tibshirani's one-standard-error rule.
- Run all three methods on uniform random noise and record what each one claims. Write one sentence on which you would trust.
- Implement subsample stability: cluster two 80% subsamples and compare their labels on the shared rows via ARI, across
kfrom 2 to 10.- Repeat the stability analysis on uniform noise and explain the high score at
k = 4geometrically.- Compare BIC from a GMM against silhouette from k-means on the same dataset and describe where they disagree.
- Interview: Your elbow plot suggests 4 clusters, silhouette suggests 7, and the business can support 3 distinct marketing treatments. What do you deliver? (Hint: which of those three numbers is a constraint rather than an estimate?)
Cluster Validation Metrics
Evaluating a clustering splits cleanly into two situations, and confusing them is the most common mistake in this area.
External metrics compare your clustering against known labels. They exist for benchmarking on datasets where the truth happens to be available, and for comparing against a human-curated grouping. Internal metrics use only the data and the clustering itself, which is what you actually have in production.
Start with why accuracy is unusable. Cluster labels are arbitrary names: nothing obliges cluster "0" to correspond to class "0". A perfect partition with permuted names scores 0.000 accuracy. Every clustering metric must therefore be invariant to relabelling, which they achieve by comparing pairs of points ("do these two land together in both?") or label distributions, never label identities.
The second subtlety is adjustment for chance. The raw Rand index
counts the fraction of point pairs that two labellings agree about. On
random labellings it climbs toward 1.0 as k grows -- 0.499 at k=2 but
0.741 at k=50 -- because with many clusters most pairs get correctly
separated by luck. The adjusted Rand index (ARI) subtracts that
expectation, so random scores 0. Similarly NMI is not chance-adjusted and
drifts upward with k (0.002 at k=2, 0.051 at k=50), while AMI is.
Prefer ARI or AMI, especially when comparing clusterings with different k.
Two external metrics decompose the two distinct ways to be wrong:
- Homogeneity: does each cluster contain only one class? (No mixing.)
- Completeness: does each class land in only one cluster? (No splitting.)
Over-clustering is perfectly homogeneous and incomplete; under-clustering is the reverse. Note that "every point in its own cluster" scores homogeneity 1.000, which is why homogeneity alone is never enough. V-measure is their harmonic mean, and ARI penalises both.
Now the internal metrics -- silhouette, Calinski-Harabasz (higher better), and Davies-Bouldin (lower better). All three measure some ratio of between-cluster separation to within-cluster spread. That sounds neutral, and it is not: they all encode the assumption that clusters are compact and separated.
The demonstration makes the consequence stark. On two interleaved moons, the correct clustering from single linkage (ARI 1.000) scores worse on all three internal metrics than the wrong k-means clustering (ARI 0.249): silhouette 0.331 versus 0.492, Calinski-Harabasz 392 versus 898, Davies-Bouldin 1.159 versus 0.774. Internal metrics measure compactness, not correctness. They are usable when the shape you want happens to be a blob, and actively misleading otherwise -- which also means you must not use them to choose between algorithms with different shape assumptions.
Finally, scoring methods that output noise. DBSCAN's -1 is not a
cluster, and you have two choices: treat it as one, or score only the
clustered points. The second flatters any method that discards hard cases, so
if you do it you must quote coverage alongside. Report both the ARI and
how many points were assigned, plus the noise recall if you know which points
are genuinely outliers.
import numpy as np
import pandas as pd
from sklearn.cluster import KMeans, AgglomerativeClustering, DBSCAN
from sklearn.datasets import make_blobs, make_moons
from sklearn.metrics import (silhouette_score, calinski_harabasz_score,
davies_bouldin_score, adjusted_rand_score, rand_score,
normalized_mutual_info_score, adjusted_mutual_info_score,
homogeneity_score, completeness_score, v_measure_score,
fowlkes_mallows_score, accuracy_score)
rng = np.random.default_rng(127)
# --- WHY ACCURACY CANNOT BE USED: CLUSTER LABELS ARE ARBITRARY NAMES. ---
# A clustering that is PERFECT can score 0 accuracy, because cluster "0" has no
# obligation to be class "0". Every clustering metric must be invariant to
# permuting the labels.
truth = np.array([0, 0, 0, 1, 1, 1, 2, 2, 2])
perfect_but_relabelled = np.array([2, 2, 2, 0, 0, 0, 1, 1, 1])
print("LABEL NAMES ARE ARBITRARY, SO ACCURACY IS MEANINGLESS HERE")
print(f" truth {truth}")
print(f" clustering {perfect_but_relabelled} <- a PERFECT partition, renamed")
print(f" accuracy {accuracy_score(truth, perfect_but_relabelled):.3f}"
" <- nonsense")
print(f" adjusted Rand {adjusted_rand_score(truth, perfect_but_relabelled):.3f}"
" <- correct")
nmi_perm = normalized_mutual_info_score(truth, perfect_but_relabelled)
print(f" NMI {nmi_perm:.3f} <- correct")
print(" Every metric below compares PAIRS of points or label DISTRIBUTIONS,")
print(" never label identities, which is what makes them permutation-invariant.")
# --- ADJUSTMENT FOR CHANCE: the reason to prefer ARI over the raw Rand index. ---
# Random labellings should score 0, not 0.6. Check it empirically.
print("\nADJUSTMENT FOR CHANCE: score RANDOM labellings and see what you get")
print(f" {'k':>3s} {'Rand index':>11s} {'adj. Rand':>10s} {'NMI':>7s} {'AMI':>7s}"
f" {'Fowlkes-M':>10s}")
n = 600
y_true = rng.integers(0, 4, n)
for k in [2, 4, 10, 50]:
rows = np.array([[rand_score(y_true, r.integers(0, k, n)),
adjusted_rand_score(y_true, r.integers(0, k, n)),
normalized_mutual_info_score(y_true, r.integers(0, k, n)),
adjusted_mutual_info_score(y_true, r.integers(0, k, n)),
fowlkes_mallows_score(y_true, r.integers(0, k, n))]
for r in [np.random.default_rng(s) for s in range(20)]])
m = rows.mean(0)
print(f" {k:3d} {m[0]:11.3f} {m[1]:10.3f} {m[2]:7.3f} {m[3]:7.3f} {m[4]:10.3f}")
print(" The raw Rand index climbs toward 1.0 as k grows on PURE NOISE, because")
print(" with many clusters most pairs are correctly separated by luck. ARI and")
print(" AMI subtract that expectation, so they sit at 0. NMI does not adjust")
print(" and drifts upward with k -- prefer AMI when comparing different k.")
# --- INTERNAL METRICS: no labels needed, but they encode a cluster SHAPE. ---
print("\nINTERNAL METRICS ALL ASSUME COMPACT, SEPARATED CLUSTERS")
print(f" {'dataset / method':34s} {'silhou.':>8s} {'Cal-Har':>9s} {'Dav-Bou':>8s}"
f" {'true ARI':>9s}")
Xb, yb = make_blobs(n_samples=600, centers=3, cluster_std=0.8, random_state=2)
Xm, ym = make_moons(n_samples=600, noise=0.06, random_state=2)
for name, Xd, yd, lab in [
("blobs, k-means k=3 (correct)", Xb, yb,
KMeans(3, n_init=10, random_state=0).fit_predict(Xb)),
("blobs, k-means k=6 (too many)", Xb, yb,
KMeans(6, n_init=10, random_state=0).fit_predict(Xb)),
("moons, k-means k=2 (wrong)", Xm, ym,
KMeans(2, n_init=10, random_state=0).fit_predict(Xm)),
("moons, single linkage (correct)", Xm, ym,
AgglomerativeClustering(2, linkage="single").fit_predict(Xm)),
]:
print(f" {name:34s} {silhouette_score(Xd, lab):8.3f}"
f" {calinski_harabasz_score(Xd, lab):9.1f}"
f" {davies_bouldin_score(Xd, lab):8.3f}"
f" {adjusted_rand_score(yd, lab):9.3f}")
print(" Look at the moons rows. The CORRECT clustering scores WORSE on all")
print(" three internal metrics than the wrong one, because crescents are not")
print(" compact. Internal metrics measure compactness, not correctness: they")
print(" are only usable when the shape you want happens to be a blob.")
# --- HOMOGENEITY, COMPLETENESS, V-MEASURE: the two ways to be wrong. ---
# homogeneity : each cluster contains only one class (no MIXING)
# completeness : each class lands in only one cluster (no SPLITTING)
print("\nHOMOGENEITY VS COMPLETENESS: the two distinct kinds of error")
base = np.repeat([0, 1, 2], 100)
cases = {
"perfect": base.copy(),
"each class split in two (over-clustering)": np.repeat(np.arange(6), 50),
"two classes merged (under-clustering)": np.where(base == 2, 1, base),
"all one cluster": np.zeros(300, int),
"every point its own cluster": np.arange(300),
}
print(f" {'clustering':44s} {'homog':>6s} {'compl':>6s} {'V':>6s} {'ARI':>6s}")
for name, lab in cases.items():
print(f" {name:44s} {homogeneity_score(base, lab):6.3f}"
f" {completeness_score(base, lab):6.3f} {v_measure_score(base, lab):6.3f}"
f" {adjusted_rand_score(base, lab):6.3f}")
print(" Over-clustering is perfectly HOMOGENEOUS and incomplete; under-")
print(" clustering is the reverse. 'Every point its own cluster' scores a")
print(" perfect 1.000 homogeneity, which is why homogeneity alone is never")
print(" enough -- V-measure is their harmonic mean, and ARI punishes both.")
# --- NOISE LABELS NEED CARE: DBSCAN's -1 is not a cluster. ---
Xn, yn = make_moons(n_samples=500, noise=0.06, random_state=3)
Xn = np.vstack([Xn, rng.uniform(-1.5, 2.5, (50, 2))])
yn = np.r_[yn, np.full(50, -1)]
db = DBSCAN(eps=0.18, min_samples=8).fit(Xn)
mask = db.labels_ != -1
print("\nSCORING A METHOD THAT OUTPUTS NOISE (DBSCAN on moons + 50 noise points)")
print(f" treating -1 as a cluster: ARI {adjusted_rand_score(yn, db.labels_):.3f}")
print(f" scoring only clustered pts: ARI"
f" {adjusted_rand_score(yn[mask], db.labels_[mask]):.3f}"
f" ({mask.sum()} of {len(Xn)} points)")
print(f" injected noise correctly called noise:"
f" {np.mean(db.labels_[yn == -1] == -1):.1%}")
print(f" genuine points wrongly discarded as noise:"
f" {np.mean(db.labels_[yn >= 0] == -1):.1%}")
print(" Report both numbers. Scoring only the clustered points flatters any")
print(" method that discards the hard cases, so coverage must be quoted too.")
# --- THE SUMMARY TABLE ---
tbl = pd.DataFrame([
("silhouette", "internal", "-1 to 1", "higher", "compact, separated"),
("Calinski-Harabasz", "internal", "0 to inf", "higher", "compact, separated"),
("Davies-Bouldin", "internal", "0 to inf", "LOWER", "compact, separated"),
("adjusted Rand (ARI)", "external", "-0.5 to 1", "higher", "0 for random"),
("NMI", "external", "0 to 1", "higher", "NOT chance-adjusted"),
("AMI", "external", "0 to 1", "higher", "0 for random"),
("homogeneity", "external", "0 to 1", "higher", "no mixing"),
("completeness", "external", "0 to 1", "higher", "no splitting"),
("V-measure", "external", "0 to 1", "higher", "harmonic mean of the two"),
], columns=["metric", "type", "range", "better", "what it assumes / means"])
print("\n", tbl.to_string(index=False))
WHY ACCURACY IS MEANINGLESS
truth [0 0 0 1 1 1 2 2 2]
clustering [2 2 2 0 0 0 1 1 1] <- a PERFECT partition, renamed
accuracy 0.000 <- nonsense
adjusted Rand 1.000 <- correct
NMI 1.000 <- correct
Metrics must compare PAIRS or DISTRIBUTIONS, never label identities.
ADJUSTMENT FOR CHANCE: SCORE RANDOM LABELLINGS
k Rand index adj. Rand NMI AMI Fowlkes-M
2 0.499 -0.001 0.002 0.000 0.353
4 0.625 -0.001 0.006 0.001 0.249
10 0.701 -0.001 0.013 0.001 0.158
50 0.741 -0.000 0.051 -0.000 0.070
^^^^^ ^^^^^^ ^^^^^
climbs with k stays at 0 also stays at 0
The raw Rand index looks respectable on pure noise at large k.
ARI and AMI subtract the expected value. NMI does not.
INTERNAL METRICS ENCODE A SHAPE ASSUMPTION
dataset / method silhou. Cal-Har Dav-Bou true ARI
blobs, k-means k=3 (correct) 0.648 3135.4 0.499 0.965
blobs, k-means k=6 (too many) 0.316 1968.0 1.198 0.558
moons, k-means k=2 (WRONG) 0.492 897.6 0.774 0.249
moons, single linkage (CORRECT) 0.331 392.5 1.159 1.000
^^^^^ ^^^^^ ^^^^^
the correct answer scores WORSE on all three
k-means on moons: single linkage on moons:
ooo|xxx ooooo
oo | xx o xxxxx
compact, so the metrics x x
approve -- and it is wrong curved, so the metrics disapprove
-- and it is right
HOMOGENEITY VS COMPLETENESS: TWO DISTINCT ERRORS
clustering homog compl V ARI
perfect 1.000 1.000 1.000 1.000
each class split in two 1.000 0.613 0.760 0.567
two classes merged 0.579 1.000 0.734 0.570
all one cluster 0.000 1.000 0.000 0.000
every point its own cluster 1.000 0.193 0.323 0.000
^^^^^
a perfect score for a useless clustering
SCORING A METHOD THAT OUTPUTS NOISE
treating -1 as a cluster: ARI 0.957
scoring only clustered pts: ARI 0.961 (509 of 550 points)
injected noise called noise: 80.0%
genuine points wrongly discarded: 0.2%
Quote coverage. Scoring only the assigned points rewards any method
that throws away whatever it found difficult.
THE SUMMARY
metric type range better notes
silhouette internal -1 to 1 higher assumes compact
Calinski-Harabasz internal 0 to inf higher assumes compact
Davies-Bouldin internal 0 to inf LOWER assumes compact
adjusted Rand (ARI) external -0.5 to 1 higher 0 for random
NMI external 0 to 1 higher NOT chance-adjusted
AMI external 0 to 1 higher 0 for random
homogeneity external 0 to 1 higher no mixing
completeness external 0 to 1 higher no splitting
V-measure external 0 to 1 higher harmonic mean
Key Takeaways
- Cluster labels are arbitrary names, so accuracy is meaningless and every valid metric is invariant to relabelling.
- The raw Rand index and NMI drift upward with
kon random labellings; ARI and AMI subtract that expectation and sit at 0.- Homogeneity means no mixing and completeness means no splitting, and either alone can be gamed by a degenerate clustering.
- Internal metrics measure compactness and separation, which is an assumption about cluster shape rather than a neutral quality measure.
- On non-convex clusters the correct clustering can score worse on every internal metric than a wrong one, so internal metrics must not arbitrate between algorithms with different shape assumptions.
- When a method outputs noise, report coverage alongside any score computed on the assigned points only.
- External metrics are for benchmarking; in production you have only internal metrics plus stability and downstream usefulness.
🧪 Practice
- Take a perfect clustering, permute its labels, and compute accuracy, ARI, and NMI to confirm which of the three is broken.
- Score random labellings at
kof 2, 5, 20, and 100 with the Rand index, ARI, NMI, and AMI, and tabulate how each behaves askgrows.- Cluster two moons with k-means and with single linkage, then compare all three internal metrics against ARI, and write one sentence on what that implies.
- Construct four degenerate clusterings and compute homogeneity, completeness, and V-measure for each.
- Score a DBSCAN result both ways -- noise as a cluster and noise excluded -- and report coverage alongside.
- Use silhouette to choose between k-means and DBSCAN on concentric circles, then check the answer against ARI and explain the disagreement.
- Interview: You are asked to pick the better of two production clusterings and you have no labels. What do you measure, and what will you be unable to conclude? (Hint: what do internal metrics assume, and what else could you measure that does not assume a shape?)
<a id="112-dimensionality-reduction"></a>
11.2 Dimensionality Reduction
Dimensionality reduction replaces many columns with few while keeping what matters, which is useful for visualisation, for compression, for removing redundancy, and for escaping the curse of dimensionality; this section covers the linear workhorses, the non-linear methods that produce the plots you have seen in papers, and -- most importantly -- what each of those plots is and is not entitled to tell you.
Principal Component Analysis
You have forty columns and suspect most of them say the same thing. PCA answers a precise version of that suspicion: find the direction along which the data varies most, then the next such direction orthogonal to it, and so on. Those directions are the principal components, and keeping the first few is the compression.
Why variance? Because a direction along which the data barely varies carries almost no information about which point is which -- if every row has nearly the same value, that coordinate cannot distinguish them. Maximising variance is a proxy for maximising retained information, and it is a good proxy right up until it is not, which is the last part of this topic.
Mechanically, PCA is an eigendecomposition of the covariance matrix -- and, equivalently, the SVD of the centred data matrix. The demonstration confirms three routes give identical numbers. Libraries use SVD, because it never forms the covariance matrix and is therefore both faster and better conditioned (chapter 2.1).
Three outputs matter:
- Explained variance ratio: the share of total variance each component carries. In the demonstration, one component captures 96% of a four-column dataset because three of the columns are the same signal plus noise.
- Loadings (
components_): what each component is made of. Component 1 loads+0.755, +0.545, -0.365on the three signal columns -- matching the signs they were built with -- and 0.008 on the noise column. Loadings are what make PCA interpretable, which t-SNE and UMAP are not. - Scores (
transform): each row's coordinates in the new basis.
Centring is required and scaling almost always is. Variance is measured in the squared units of the column, so a column measured in dollars swamps one measured in metres. In the demonstration, PC1 on raw units is 100% "income" because income's variance is ten orders of magnitude larger than height's. Standardising first is PCA on the correlation matrix and is the right default; use the covariance matrix only when every column already shares a unit.
Choosing how many components has three common answers, and they disagree. On data built from 5 latent factors in 40 columns:
- Scree elbow (largest drop between consecutive eigenvalues): 5. Correct.
- Kaiser rule (eigenvalue > 1 on standardised data): 5. Correct.
- Variance threshold: 16 components for 90%, 25 for 95%. Badly wrong.
The variance threshold fails because 35 noise directions each carry a little variance and it adds up. A fixed "95% of variance" rule is the least reliable of the three and unfortunately the most commonly used.
Whitening divides each component by its standard deviation, giving uncorrelated unit-variance features. It helps downstream methods that assume comparable scales and hurts when the variance ordering was information you wanted.
Finally, the limitation that matters most: PCA is unsupervised, and high variance is not high information. The demonstration builds twenty features where exactly one separates the classes and five are inflated pure noise. All twenty raw features give 0.85 test accuracy; the first principal component gives 0.62, and the most class-relevant component is number 3, not number 1. PCA never saw the labels, so it ranked the noise above the signal. When you have labels, LDA (chapter 10.2) targets separation directly; otherwise select components by downstream performance rather than by retained variance.
import numpy as np
import pandas as pd
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(131)
# --- PCA IS A ROTATION: find the axes along which the data actually varies. ---
# Build correlated data so there IS a direction worth finding.
n = 800
z = rng.normal(0, 1, n)
X = np.column_stack([2.5 * z + rng.normal(0, 0.4, n),
1.8 * z + rng.normal(0, 0.4, n),
-1.2 * z + rng.normal(0, 0.4, n),
rng.normal(0, 0.35, n)]) # a genuinely noisy 4th axis
Xc = X - X.mean(0) # PCA requires centring
# The three equivalent routes to the same answer.
cov = np.cov(Xc.T)
evals, evecs = np.linalg.eigh(cov) # eigendecomposition
order = np.argsort(evals)[::-1]
evals, evecs = evals[order], evecs[:, order]
U, S, Vt = np.linalg.svd(Xc, full_matrices=False) # SVD -- preferred route
sk = PCA().fit(Xc)
print("THREE ROUTES TO THE SAME DECOMPOSITION")
print(f" eigenvalues of the covariance {np.round(evals, 4)}")
print(f" s^2/(n-1) from the SVD {np.round(S ** 2 / (n - 1), 4)}")
print(f" sklearn explained_variance_ {np.round(sk.explained_variance_, 4)}")
print(" SVD is the route libraries use: it never forms the covariance matrix,")
print(" which is both faster and numerically better conditioned (chapter 2.1).")
# --- WHAT THE COMPONENTS ARE: directions, ordered by variance captured. ---
print("\nEXPLAINED VARIANCE: how much of the spread each direction carries")
ev = sk.explained_variance_ratio_
print(f" {'component':>10s} {'variance':>9s} {'share':>7s} {'cumulative':>11s}")
for i, (v, r) in enumerate(zip(sk.explained_variance_, ev)):
print(f" {i + 1:10d} {v:9.4f} {r:7.1%} {ev[:i + 1].sum():11.1%}")
print(" One component captures 96% of a 4-column dataset, because three of the")
print(" four columns are the same underlying signal plus noise.")
print("\nLOADINGS: what each component is made of")
print(f" {'component':>10s} " + " ".join(f"{f'col{j}':>8s}" for j in range(4)))
for i in range(4):
print(f" {i + 1:10d} " + " ".join(f"{v:8.3f}" for v in sk.components_[i]))
print(" Component 1 loads on columns 0, 1, 2 with signs matching how they were")
print(" built (+2.5, +1.8, -1.2) and almost nothing on the noise column. The")
print(" loadings are what make PCA interpretable, unlike t-SNE or UMAP.")
# --- SCALING CHANGES THE ANSWER COMPLETELY. ---
# PCA maximises variance, and variance is in squared units of the column.
df = pd.DataFrame({
"height_m": rng.normal(1.7, 0.1, 500), # variance ~0.01
"weight_kg": rng.normal(70, 12, 500), # variance ~144
"income": rng.normal(50_000, 15_000, 500), # variance ~2.25e8
})
df["weight_kg"] += (df["height_m"] - 1.7) * 60 # make height and weight relate
print("\nSCALING: PCA MAXIMISES VARIANCE, WHICH IS UNIT-DEPENDENT")
for label, data in [("raw units", df.values),
("standardised", StandardScaler().fit_transform(df))]:
p = PCA().fit(data)
print(f" {label:14s} PC1 explains {p.explained_variance_ratio_[0]:6.1%}"
f" PC1 loadings {np.round(p.components_[0], 3)}")
print(" In raw units PC1 is entirely 'income', because income's variance is")
print(" ten orders of magnitude larger. Standardising gives every column an")
print(" equal say. Use the covariance matrix only when the columns already")
print(" share a unit; otherwise standardise, which is PCA on the CORRELATION")
print(" matrix.")
# --- CHOOSING THE NUMBER OF COMPONENTS. ---
p_dim = 40
latent = rng.normal(0, 1, (1200, 5))
load = rng.normal(0, 1, (5, p_dim))
Xd = latent @ load + rng.normal(0, 1.0, (1200, p_dim)) # 5 real signals in 40 cols
pca = PCA().fit(StandardScaler().fit_transform(Xd))
evr = pca.explained_variance_ratio_
print(f"\nCHOOSING k: {p_dim} columns built from 5 latent factors")
print(f" {'k':>4s} {'eigenvalue':>11s} {'share':>7s} {'cumulative':>11s}")
for k in [1, 2, 3, 4, 5, 6, 7, 10, 20, 40]:
print(f" {k:4d} {pca.explained_variance_[k - 1]:11.3f} {evr[k - 1]:7.1%}"
f" {evr[:k].sum():11.1%}")
print(f"\n Kaiser rule (eigenvalue > 1 on standardised data):"
f" {int((pca.explained_variance_ > 1).sum())} components")
print(f" 90% of variance needs {int(np.searchsorted(np.cumsum(evr), 0.90)) + 1}"
" components")
print(f" 95% of variance needs {int(np.searchsorted(np.cumsum(evr), 0.95)) + 1}"
" components")
# The scree "elbow": the largest drop between consecutive eigenvalues.
drops = -np.diff(pca.explained_variance_[:12])
print(f" scree elbow (largest eigenvalue drop): after component"
f" {int(np.argmax(drops)) + 1}")
print(" The scree elbow finds the 5 real factors. Variance thresholds do not,")
print(" because 35 noise directions each carry a little variance that adds up.")
print(" A fixed '95% of variance' rule is the least reliable of the three.")
# --- WHITENING: decorrelated AND unit-variance. ---
w = PCA(n_components=3, whiten=True).fit_transform(StandardScaler()
.fit_transform(Xd))
nw = PCA(n_components=3, whiten=False).fit_transform(StandardScaler()
.fit_transform(Xd))
print("\nWHITENING divides each component by its standard deviation")
print(f" not whitened: component variances {np.round(nw.var(0), 3)}")
print(f" whitened: component variances {np.round(w.var(0), 3)}")
print(f" both are uncorrelated: max |off-diagonal correlation|"
f" {np.abs(np.corrcoef(w.T) - np.eye(3)).max():.2e}")
print(" Whitening helps methods that assume comparable scales, and hurts when")
print(" the variance ordering was information you wanted to keep.")
# --- WHERE PCA FAILS: VARIANCE IS NOT THE SAME AS USEFULNESS. ---
m = 2000
cls = rng.integers(0, 2, m)
Xs = rng.normal(0, 1, (m, 20))
Xs[:, 0] += cls * 2.2 # the ONLY discriminative direction
Xs[:, 1:6] *= 9.0 # high-variance, no class information
Atr, Ate, btr, bte = train_test_split(Xs, cls, test_size=0.3, random_state=0,
stratify=cls)
print("\nPCA IS UNSUPERVISED: HIGH VARIANCE IS NOT HIGH INFORMATION")
print(f" {'features used':34s} {'test accuracy':>14s}")
full = make_pipeline(StandardScaler(),
LogisticRegression(max_iter=3000)).fit(Atr, btr)
print(f" {'all 20 raw features':34s} {full.score(Ate, bte):14.4f}")
for k in [1, 2, 3, 5, 10]:
pl = make_pipeline(StandardScaler(), PCA(n_components=k),
LogisticRegression(max_iter=3000)).fit(Atr, btr)
print(f" {f'first {k} principal components':34s} {pl.score(Ate, bte):14.4f}")
sc = StandardScaler().fit(Atr)
pc = PCA().fit(sc.transform(Atr))
corr = [abs(np.corrcoef(pc.transform(sc.transform(Atr))[:, i], btr)[0, 1])
for i in range(8)]
print(f"\n |correlation with the class| by component:"
f" {np.round(corr, 3)}")
print(f" the most class-relevant component is number {int(np.argmax(corr)) + 1},"
" not number 1")
print(" PCA never saw the labels, so it ranked the five inflated noise columns")
print(" above the one column that matters. Keeping 'enough variance' can")
print(" discard exactly the signal you need -- use a supervised method (LDA,")
print(" chapter 10.2) or select components by downstream performance.")
PCA IS A ROTATION ONTO THE AXES OF GREATEST VARIANCE
x2 | . .:. PC1
| .:::::. . \
| .:::::::. \ . .:.
|.:::::::. -> PC2 \.:::::.
+------------- x1 \:::::.
\
eigenvalues of the covariance [10.2132 0.1644 0.1510 0.1273]
s^2/(n-1) from the SVD [10.2132 0.1644 0.1510 0.1273]
sklearn explained_variance_ [10.2132 0.1644 0.1510 0.1273]
EXPLAINED VARIANCE AND LOADINGS
component variance share cumulative
1 10.2132 95.8% 95.8%
2 0.1644 1.5% 97.4%
3 0.1510 1.4% 98.8%
4 0.1273 1.2% 100.0%
component col0 col1 col2 col3
1 0.755 0.545 -0.365 0.008
2 0.643 -0.506 0.574 -0.014
4 0.010 -0.047 -0.029 0.998
cols 0-2 were built as +2.5z, +1.8z, -1.2z and col3 is pure noise.
PC1 recovers that pattern; PC4 isolates the noise column.
SCALING CHANGES THE ANSWER ENTIRELY
raw units PC1 explains 100.0% loadings [-0. -0. 1. ]
standardised PC1 explains 48.7% loadings [0.707 0.706 -0.039]
In raw units PC1 IS income, because income's variance dominates.
CHOOSING k: 40 COLUMNS FROM 5 LATENT FACTORS
k eigenvalue share cumulative
1 10.033 25.1% 25.1%
2 7.115 17.8% 42.8%
3 5.607 14.0% 56.8%
4 4.930 12.3% 69.2%
5 4.578 11.4% 80.6%
6 0.502 1.3% 81.8% <- the cliff
20 0.211 0.5% 92.8%
40 0.082 0.2% 100.0%
eigenvalue
|*
| * * * *
| <- the drop from 4.578 to 0.502 is the scree elbow
| * * * * * * * * * * * * *
+-------------------------------------> k
5
scree elbow: 5 components CORRECT
Kaiser (eigenvalue > 1): 5 components CORRECT
90% of variance: 16 components wrong
95% of variance: 25 components wrong
PCA IS UNSUPERVISED: VARIANCE IS NOT INFORMATION
20 features; feature 0 separates the classes, features 1-5 are
inflated pure noise.
features used test accuracy
all 20 raw features 0.8500
first 1 principal components 0.6217
first 2 principal components 0.6100
first 3 principal components 0.6333
first 5 principal components 0.6767
first 10 principal components 0.7350
|correlation with the class| by component:
[0.174 0.041 0.267 0.190 0.147 0.018 0.069 0.241]
PC1 PC3 PC8
^ the most useful component is the THIRD
| PCA | LDA (chapter 10.2) | |
|---|---|---|
| Uses labels | No | Yes |
| Maximises | Variance | Between-class separation |
| Components available | Up to min(n, p) | At most K - 1 |
| Best for | Compression, decorrelation | Class separation |
| Fails when | Signal has low variance | Classes are not Gaussian |
Key Takeaways
- PCA finds orthogonal directions of maximum variance, computed in practice by SVD of the centred data rather than by forming the covariance matrix.
- Explained variance says how much each component carries, and loadings say what each component is made of, which keeps PCA interpretable.
- Centring is mandatory and standardising almost always is, because variance is measured in squared column units.
- The scree elbow and the Kaiser rule both find the true number of factors where a fixed variance threshold badly over-counts.
- Whitening yields uncorrelated unit-variance components, which helps or hurts depending on whether the variance ordering was meaningful.
- PCA never sees the labels, so a high-variance direction can be pure noise and the discriminative direction can be a late component.
- Retaining "enough variance" can discard exactly the signal you need; select components by downstream performance or use a supervised method.
🧪 Practice
- Compute PCA three ways -- covariance eigendecomposition, SVD of the centred matrix, and
sklearn-- and confirm the eigenvalues agree.- Run PCA on a table with columns in wildly different units, with and without standardisation, and report PC1's loadings in each case.
- Build data from 5 latent factors in 50 columns and compare the scree elbow, the Kaiser rule, and 90% and 95% variance thresholds.
- Reconstruct data from
kcomponents forkfrom 1 topand plot reconstruction error againstk.- Fit PCA with
whiten=Trueandwhiten=Falseand verify the component variances and the off-diagonal correlations of each.- Construct a dataset where the class signal has low variance, then compare classification accuracy on the first 2 PCA components against the first 2 LDA components.
- Interview: A colleague reduces 500 features to 50 components retaining 95% of variance, and the downstream model gets worse. What happened and what would you try? (Hint: what is PCA ranking components by, and is that the same thing the classifier needs?)
Factor Analysis
Factor analysis looks almost identical to PCA -- both produce a small set of components with loadings -- and they answer different questions. The distinction is worth getting right, because the two are routinely confused and one of them is a statistical model while the other is not.
PCA has no noise term. It is a pure rotation of the data: x = (rotation) @ (scores), exactly and reversibly. Every scrap of each variable's variance
must be accounted for by some component, because there is nowhere else for it
to go.
Factor analysis posits a generative model:
x = Lambda @ f + e
where f is a small vector of unobserved common factors, Lambda holds
the loadings, and e is noise with its own variance for each
variable. That last clause is the whole difference. FA says explicitly: part
of each variable is shared structure, and part is that variable's own
measurement error. The split is reported as communality (shared) and
uniqueness (variable-specific).
This matters when variables have genuinely different noise levels. In the demonstration, six variables have noise SD 0.3 and two have 1.7. FA recovers that structure almost exactly -- the fitted uniqueness correlates 0.999 with the true noise variance, and it reports variables 6 and 7 as roughly 84% unique. PCA has no such parameter, so it cannot express "this variable is mostly measurement error" and its components absorb that error as though it were structure.
Note honestly what the demonstration also shows: both methods recover the two-block loading pattern about equally well here (block purity 0.969 versus 0.970). The advantage is not that FA finds better loadings on easy data; it is that FA can say something about the noise and PCA cannot.
Two further consequences of being a real model:
FA has a likelihood, so it can select the number of factors. BIC is
minimised at exactly the true k = 2 in the demonstration. PCA gives you
heuristics -- scree, Kaiser -- rather than a model-selection criterion.
FA loadings can be rotated, and PCA components cannot. Because Lambda @ f is unchanged if you rotate Lambda one way and f the other, any rotation
fits identically. That freedom is a feature: rotate toward
interpretability. Varimax pushes each loading toward 0 or ±1 so each
variable loads on as few factors as possible. In the demonstration, with
factors correlated at 0.5, the unrotated solution is the classic "general
factor plus contrast" -- every variable loads about -0.75 on factor 1 -- and
varimax rotates it into two clean blocks, dropping loading complexity from
1.594 to 1.205 while leaving the fit bit-identical (max |L L' - Lr Lr'| = 2e-16).
Worth noting: a residual cross-loading of about 0.36 survives, and it is the true factor correlation of 0.5. An orthogonal rotation like varimax cannot represent correlated factors, which is exactly what oblique rotations such as promax are for. PCA components cannot be rotated this way without destroying their defining property of maximal variance in order -- which is the cleanest statement of the division of labour: FA for interpretation, PCA for compression.
import numpy as np
import pandas as pd
from sklearn.decomposition import PCA, FactorAnalysis
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(137)
# --- THE MODEL DIFFERENCE, MADE CONCRETE. ---
# Factor analysis assumes x = Lambda @ f + e where e has a PER-VARIABLE
# variance. PCA has no noise term: it is a pure rotation, so it must account for
# every variable's full variance as though all of it were signal.
n, p, k = 3000, 8, 2
F = rng.normal(0, 1, (n, k))
Lam = np.array([[0.9, 0.0], [0.85, 0.0], [0.8, 0.0], [0.75, 0.0],
[0.0, 0.9], [0.0, 0.85], [0.0, 0.8], [0.0, 0.75]])
# The key ingredient: noise variance DIFFERS by variable.
noise_sd = np.array([0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 1.7, 1.7])
X = StandardScaler().fit_transform(F @ Lam.T + rng.normal(0, 1, (n, p)) * noise_sd)
print("8 VARIABLES, 2 LATENT FACTORS, UNEQUAL NOISE PER VARIABLE")
print(f" noise sd by variable: {noise_sd}")
print(" Variables 0-3 load on factor 1, variables 4-7 on factor 2, and")
print(" variables 6-7 are mostly noise.")
pca = PCA(n_components=k).fit(X)
fa = FactorAnalysis(n_components=k, random_state=0).fit(X)
# A clean measure of structure recovery: for each variable, how concentrated is
# its loading on the factor it truly belongs to?
def block_ratio(L):
"""L is (p, k). Mean ratio of |loading on the true factor| to the total."""
A = np.abs(L)
true_f = np.r_[np.zeros(4, int), np.ones(4, int)]
# allow either component ordering
return max(float(np.mean(A[np.arange(p), perm[true_f]] / A.sum(1)))
for perm in [np.array([0, 1]), np.array([1, 0])])
print("\nRECOVERED LOADINGS (truth in the last two columns)")
print(f" {'var':>4s} {'PCA f1':>7s} {'PCA f2':>7s} {'FA f1':>7s} {'FA f2':>7s}"
f" {'true f1':>8s} {'true f2':>8s}")
for j in range(p):
print(f" {j:4d} {pca.components_[0, j]:7.3f} {pca.components_[1, j]:7.3f}"
f" {fa.components_[0, j]:7.3f} {fa.components_[1, j]:7.3f}"
f" {Lam[j, 0]:8.2f} {Lam[j, 1]:8.2f}")
print(f"\n block purity (1.0 = each variable loads on its own factor only)")
print(f" PCA {block_ratio(pca.components_.T):.4f}")
print(f" FA {block_ratio(fa.components_.T):.4f}")
print(" Both recover the two-block structure here. The difference is not in the")
print(" loadings; it is in what each model can SAY about the noise.")
# --- UNIQUENESS: the parameter PCA does not have. ---
comm = (fa.components_ ** 2).sum(0)
print("\nFA SPLITS EACH VARIABLE INTO COMMON AND UNIQUE VARIANCE")
print(f" {'var':>4s} {'communality':>12s} {'uniqueness':>11s}"
f" {'true noise sd':>14s}")
for j in range(p):
print(f" {j:4d} {comm[j]:12.3f} {fa.noise_variance_[j]:11.3f}"
f" {noise_sd[j]:14.2f}")
print(f" correlation(fitted uniqueness, true noise variance):"
f" {np.corrcoef(fa.noise_variance_, noise_sd ** 2)[0, 1]:.4f}")
print(" FA correctly reports variables 6 and 7 as roughly 84% unique -- that is,")
print(" mostly noise. PCA has no noise parameter at all, so it cannot express")
print(" 'this variable is mostly measurement error', and its components absorb")
print(" that error as though it were structure.")
# --- FA IS GENERATIVE, SO IT HAS A LIKELIHOOD AND CAN SELECT k. ---
print("\nFA HAS A LIKELIHOOD, SO IT CAN CHOOSE THE NUMBER OF FACTORS")
print(f" {'k':>3s} {'avg log-lik':>12s} {'BIC':>11s}")
for kk in range(1, 6):
f = FactorAnalysis(n_components=kk, random_state=0).fit(X)
ll = f.score(X) * len(X)
nparam = p * kk + p - kk * (kk - 1) / 2 # loadings + uniquenesses - rot.
print(f" {kk:3d} {f.score(X):12.4f} {-2 * ll + nparam * np.log(len(X)):11.1f}")
print(" BIC is minimised at k=2, the true number of factors. PCA's eigenvalues")
print(" give heuristics (scree, Kaiser) rather than a model-selection criterion.")
# --- ROTATION: the loadings are not unique, and FA turns that into a feature. ---
# Any rotation of the factors fits identically, so you may rotate toward
# interpretability. This matters most when the unrotated solution is a "general
# factor plus contrast", which is what correlated factors produce.
C = np.array([[1.0, 0.5], [0.5, 1.0]]) # the two factors CORRELATE
F2 = rng.normal(0, 1, (n, 2)) @ np.linalg.cholesky(C).T
X2 = StandardScaler().fit_transform(F2 @ Lam.T + rng.normal(0, 0.45, (n, p)))
L = FactorAnalysis(2, random_state=0).fit(X2).components_.T
def varimax(L, iters=200, tol=1e-9):
"""Kaiser varimax: rotate to maximise the variance of squared loadings per
factor, which pushes each loading toward 0 or +/-1."""
L = L.copy()
p_, k_ = L.shape
R, d_old = np.eye(k_), 0.0
for _ in range(iters):
Lr = L @ R
target = L.T @ (Lr ** 3 - Lr @ np.diag((Lr ** 2).sum(0)) / p_)
u, s, vt = np.linalg.svd(target)
R = u @ vt
d = s.sum()
if d_old and d / d_old < 1 + tol:
break
d_old = d
return L @ R
def complexity(L):
"""Hoffman complexity, averaged: 1.0 means every variable loads on ONE factor."""
s = L ** 2
return float(np.mean(s.sum(1) ** 2 / (s ** 2).sum(1)))
Lr = varimax(L)
print("\nROTATION (factors correlated at 0.5, so the raw solution is a blend)")
print(f" {'var':>4s} {'unrotated':>18s} {'varimax':>18s}")
for j in range(p):
print(f" {j:4d} {np.array2string(L[j], precision=3, floatmode='fixed'):>18s}"
f" {np.array2string(Lr[j], precision=3, floatmode='fixed'):>18s}")
print(f"\n loading complexity: {complexity(L):.3f} -> {complexity(Lr):.3f}"
" (1.0 is ideal)")
print(f" the FIT is unchanged: max |L L' - Lr Lr'|"
f" {np.abs(L @ L.T - Lr @ Lr.T).max():.2e}")
print(" Unrotated, EVERY variable loads about -0.75 on the first factor -- a")
print(" 'general factor' -- and the second factor is a contrast between the two")
print(" blocks. Rotated, each block sits on one factor at 0.79 to 0.89. The")
print(" residual cross-loading of about 0.36 is the TRUE factor correlation of")
print(" 0.5, which an ORTHOGONAL rotation cannot remove; that is what oblique")
print(" rotations such as promax are for.")
print(" Rotation is free because a rotation of the factors is absorbed by the")
print(" unobserved factors themselves. PCA components cannot be rotated without")
print(" losing their defining property of maximal variance in order -- which is")
print(" why FA is the tool for interpretation and PCA the tool for compression.")
THE TWO MODELS
PCA: x = W @ scores no noise term at all.
every bit of variance is "signal".
FA: x = Lambda @ f + e e has a PER-VARIABLE variance.
variance is split into shared
(communality) and unique.
variable's total variance
+------------------------------------------+
| communality | uniqueness | <- FA models both
+------------------------------------------+
| all of it | <- PCA has only this
+------------------------------------------+
8 VARIABLES, 2 FACTORS, UNEQUAL NOISE (vars 6-7 have sd 1.7, rest 0.3)
var PCA f1 PCA f2 FA f1 FA f2 true f1 true f2
0 0.502 0.020 0.948 -0.026 0.90 0.00
3 0.497 0.012 0.930 -0.019 0.75 0.00
4 -0.021 0.599 -0.039 -0.945 0.00 0.90
6 -0.004 0.371 -0.005 -0.396 0.00 0.80
7 -0.010 0.380 -0.015 -0.408 0.00 0.75
block purity: PCA 0.9685 FA 0.9697 <- essentially tied
THE DIFFERENCE IS THE NOISE PARAMETER
var communality uniqueness true noise sd
0 0.899 0.101 0.30
3 0.866 0.134 0.30
5 0.893 0.107 0.30
6 0.157 0.843 1.70 <- mostly noise
7 0.167 0.833 1.70 <- mostly noise
correlation(fitted uniqueness, true noise variance): 0.9994
PCA cannot produce this column. It has no parameter for it.
A LIKELIHOOD MEANS MODEL SELECTION
k avg log-lik BIC
1 -8.7908 52873.1
2 -7.8207 47108.3 <- minimum, and the true k
3 -7.8197 47150.7
5 -7.8189 47217.9
ROTATION IS FREE, AND IT BUYS INTERPRETABILITY
(factors correlated at 0.5, so the raw solution is a blend)
var unrotated varimax
0 [-0.772 -0.444] [-0.365 -0.812]
3 [-0.747 -0.430] [-0.353 -0.787]
4 [-0.779 0.442] [-0.891 -0.098]
7 [-0.742 0.429] [-0.852 -0.087]
unrotated: EVERY variable loads ~-0.75 on factor 1
(a "general factor"), factor 2 is a contrast
rotated: each block sits on one factor at 0.79 to 0.89
loading complexity 1.594 -> 1.205 (1.0 = one factor per variable)
the fit is IDENTICAL: max |L L' - Lr Lr'| = 2.22e-16
The residual 0.36 cross-loading is the TRUE 0.5 factor correlation,
which an orthogonal rotation cannot remove -- hence promax.
| PCA | Factor Analysis | |
|---|---|---|
| Is a statistical model | No, a rotation | Yes, generative |
| Noise term | None | Per-variable variance |
| Can say "this variable is noise" | No | Yes, via uniqueness |
| Selecting components | Heuristics (scree, Kaiser) | Likelihood, BIC |
| Rotation allowed | No, destroys the ordering | Yes, and encouraged |
| Solution uniqueness | Unique up to sign | Unique only up to rotation |
| Primary use | Compression, decorrelation | Interpreting latent constructs |
Key Takeaways
- PCA is a rotation with no noise term, so it must attribute every variable's entire variance to some component.
- Factor analysis models
x = Lambda f + ewith a per-variable noise variance, splitting each variable into communality and uniqueness.- That noise parameter is the real difference: FA can identify a variable as mostly measurement error, and PCA structurally cannot.
- On easy data both recover similar loadings, so do not expect FA to find better components -- expect it to tell you more about the noise.
- FA has a likelihood, so BIC selects the number of factors where PCA offers only heuristics.
- FA loadings are only identified up to a rotation, which lets you rotate toward interpretability at zero cost to fit.
- Varimax is orthogonal and cannot represent correlated factors; oblique rotations such as promax exist for that case.
- Use FA when you want to interpret latent constructs and PCA when you want to compress.
🧪 Practice
- Generate data from two factors with unequal per-variable noise, fit both PCA and FA, and compare the recovered loadings side by side.
- Report FA's fitted uniqueness against the true noise variance you used, and compute the correlation between them.
- Compute BIC for 1 to 6 factors and confirm the minimum falls at the true number.
- Implement varimax rotation and verify that
L L'is unchanged to machine precision while loading complexity falls.- Generate data with factors correlated at 0.7, apply varimax, and quantify the cross-loading that survives.
- Refit both methods with equal noise across all variables and describe how the comparison changes.
- Interview: A survey team has 30 questionnaire items and wants to know "what underlying traits are being measured". Would you reach for PCA or FA, and what would you do about the loadings afterwards? (Hint: which method has a parameter for item-specific measurement error, and which output can be rotated for free?)
t-SNE
PCA maximises variance, which is a global criterion: it will happily place two points that are neighbours in the original space far apart in the projection, as long as the overall spread is preserved. For visualisation that is usually the wrong trade. What you want from a two-dimensional plot is that points near each other really were near each other.
t-SNE optimises exactly that, and the way it does so explains every one of its famous caveats. Three steps:
- In high dimensions, convert distances to probabilities. For each point,
place a Gaussian over its neighbours and normalise, so
p_ijis the probability of pickingjasi's neighbour. The Gaussian's width is set per point, so that every point ends up with the same effective number of neighbours. That number is the perplexity. 2. In low dimensions, do the same with a Student-t kernel -- heavy-tailed rather than Gaussian. 3. MinimiseKL(P || Q)by gradient descent on the 2-D coordinates.
Two design choices deserve unpacking. The per-point bandwidth means density is equalised by construction: a dense region and a sparse region are both rescaled to have the same effective neighbour count. The heavy tail in low dimensions is the fix for the "crowding problem": in two dimensions there is simply less room than in fifty, and a heavy-tailed kernel lets moderately distant points sit far apart without a large penalty, which prevents everything collapsing into one blob.
The demonstration quantifies the trade. On 30-dimensional data, t-SNE keeps 0.386 of each point's 15 nearest neighbours against PCA's 0.203 -- nearly double -- while scoring worse on distance-rank correlation (0.863 against 0.961). That is not a defect. It is the objective: t-SNE optimises a distribution over neighbours and never promises anything about distances.
Perplexity is the parameter that matters, and it behaves as "how many neighbours each point considers": 5 to 50 is the useful range, small values emphasising fine local detail and large values blurring toward global structure. Always look at more than one value.
Now the three things you must not read off a t-SNE plot, each demonstrated:
1. Cluster size means nothing. A tight cluster and one 13.4 times more spread out come out with a radius ratio of 1.0 on the plot. The per-point bandwidth equalises density by design, so apparent cluster size is an artefact.
2. Distance between clusters means nothing. Three clusters whose real centre gaps differ by 7.1x come out with a gap ratio of 1.1x. You cannot conclude that two clusters are "more different" because they are further apart in the picture.
3. Pure noise produces convincing clusters. This is the most dangerous
one. On 30-dimensional isotropic Gaussian noise with no structure whatever, a
low-perplexity t-SNE embedding yields visually separated islands that score
silhouette 0.355 at k=4 -- against 0.026 for the same noise in its original
space. The islands are entirely manufactured by the algorithm. Never cluster
a t-SNE embedding, and never treat apparent islands as evidence of groups.
Cluster the original space and use t-SNE only to look at the result.
Two operational notes. Reproducibility depends on init, which is more
nuanced than the usual "t-SNE is non-deterministic". With init="pca" --
scikit-learn's default since version 1.2 -- the starting point is
deterministic and runs agree almost exactly (pairwise-distance correlation
1.000 across seeds). With init="random" the coordinates differ between seeds
(correlations 0.93 to 0.98) while the neighbourhood structure stays
reproducible.
And there is no transform(). t-SNE learns positions, not a mapping, so
adding one new row means refitting the entire embedding. That rules it out of
any pipeline that must embed new data. It is a visualisation tool, not a
preprocessing step.
import time
import numpy as np
import pandas as pd
from sklearn.manifold import TSNE
from sklearn.decomposition import PCA
from sklearn.datasets import make_blobs
from sklearn.neighbors import NearestNeighbors
from sklearn.preprocessing import StandardScaler
from scipy.spatial.distance import pdist
from scipy.stats import spearmanr
rng = np.random.default_rng(139)
# --- THE PROBLEM t-SNE SOLVES: PCA CANNOT SEPARATE NON-LINEAR STRUCTURE. ---
# Three clusters arranged so that no linear projection separates all three.
n_per = 250
t = rng.uniform(0, 2 * np.pi, n_per)
ring = np.column_stack([4 * np.cos(t), 4 * np.sin(t)]) + rng.normal(0, 0.25,
(n_per, 2))
inner = rng.normal(0, 0.5, (n_per, 2))
far = rng.normal([0, 9], 0.5, (n_per, 2))
X2 = np.vstack([ring, inner, far])
lab = np.repeat([0, 1, 2], n_per)
# Embed in 30 dimensions with a random rotation, so the structure is hidden.
Q = np.linalg.qr(rng.normal(0, 1, (30, 30)))[0]
X = np.column_stack([X2, rng.normal(0, 0.4, (len(X2), 28))]) @ Q
# --- HOW WELL DOES EACH METHOD PRESERVE LOCAL NEIGHBOURHOODS? ---
def knn_preservation(Xhigh, Xlow, k=15):
"""Fraction of each point's k nearest neighbours in the original space that
are still among its k nearest in the embedding. The standard local measure."""
a = NearestNeighbors(n_neighbors=k + 1).fit(Xhigh).kneighbors(
Xhigh, return_distance=False)[:, 1:]
b = NearestNeighbors(n_neighbors=k + 1).fit(Xlow).kneighbors(
Xlow, return_distance=False)[:, 1:]
return float(np.mean([len(set(x) & set(y)) / k for x, y in zip(a, b)]))
def global_fidelity(Xhigh, Xlow, sample=600, seed=0):
"""Spearman correlation between pairwise distances before and after. This is
the number t-SNE is NOT trying to optimise."""
r = np.random.default_rng(seed)
idx = r.choice(len(Xhigh), min(sample, len(Xhigh)), replace=False)
return float(spearmanr(pdist(Xhigh[idx]), pdist(Xlow[idx])).statistic)
Xs = StandardScaler().fit_transform(X)
pca2 = PCA(n_components=2).fit_transform(Xs)
tsne = TSNE(n_components=2, perplexity=30, init="pca",
random_state=0).fit_transform(Xs)
print("LOCAL VS GLOBAL STRUCTURE PRESERVATION (30-D data, 2-D embedding)")
print(f" {'method':18s} {'15-NN kept':>11s} {'distance rank r':>16s}")
for name, Z in [("PCA", pca2), ("t-SNE (perp 30)", tsne)]:
print(f" {name:18s} {knn_preservation(Xs, Z):11.3f}"
f" {global_fidelity(Xs, Z):16.3f}")
print(" t-SNE keeps far more of each point's neighbourhood and is WORSE at")
print(" preserving pairwise distance order. That is not a defect: it is the")
print(" objective. t-SNE optimises a distribution over NEIGHBOURS.")
# --- WHAT t-SNE ACTUALLY MINIMISES. ---
print("\nTHE OBJECTIVE, IN THREE LINES")
print(" 1. In high-D, convert distances to probabilities with a Gaussian whose")
print(" width is set per point so each has the same effective neighbour")
print(" count. That count IS the perplexity.")
print(" 2. In low-D, do the same with a Student-t (heavy-tailed) kernel.")
print(" 3. Minimise KL(P || Q) by gradient descent on the 2-D coordinates.")
print(" The heavy tail in step 2 is the 'crowding' fix: it lets moderately")
print(" distant points sit far apart in 2-D without a large penalty, which is")
print(" what stops everything collapsing into one blob.")
# --- PERPLEXITY IS THE ONE PARAMETER THAT MATTERS. ---
print("\nPERPLEXITY = the effective number of neighbours each point considers")
print(f" {'perplexity':>11s} {'15-NN kept':>11s} {'distance rank r':>16s}"
f" {'seconds':>8s}")
for perp in [5, 15, 30, 50, 100]:
t0 = time.perf_counter()
Z = TSNE(n_components=2, perplexity=perp, init="pca",
random_state=0).fit_transform(Xs)
el = time.perf_counter() - t0
print(f" {perp:11d} {knn_preservation(Xs, Z):11.3f}"
f" {global_fidelity(Xs, Z):16.3f} {el:8.2f}")
print(" Small perplexity sees only the nearest few points and can shatter a")
print(" cluster into fragments. Large perplexity blurs toward global structure.")
print(" Typical useful range is 5 to 50; ALWAYS look at more than one value.")
# --- THE THREE THINGS YOU MUST NOT READ OFF A t-SNE PLOT. ---
# 1. CLUSTER SIZE. t-SNE expands sparse clusters and shrinks dense ones.
tight = rng.normal(0.0, 0.15, (300, 10))
loose = rng.normal(0.0, 2.0, (300, 10))
loose[:, 0] += 12.0 # move it away, keep it diffuse
mixed = np.vstack([tight, loose])
Zm = TSNE(n_components=2, perplexity=30, init="pca",
random_state=0).fit_transform(mixed)
def spread(Z):
return float(np.sqrt(((Z - Z.mean(0)) ** 2).sum(1).mean()))
print("\n1. CLUSTER SIZE IS NOT MEANINGFUL")
print(f" original radius: tight {spread(mixed[:300]):.3f}"
f" loose {spread(mixed[300:]):.3f}"
f" ratio {spread(mixed[300:]) / spread(mixed[:300]):.1f}x")
print(f" t-SNE radius: tight {spread(Zm[:300]):.3f}"
f" loose {spread(Zm[300:]):.3f}"
f" ratio {spread(Zm[300:]) / spread(Zm[:300]):.1f}x")
print(" A 13x difference in real spread became roughly 1x on the plot. The")
print(" per-point bandwidth equalises density by construction.")
# 2. DISTANCE BETWEEN CLUSTERS.
threeb, tl = make_blobs(n_samples=600, centers=[[0, 0], [5, 0], [40, 0]],
cluster_std=0.6, n_features=2, random_state=1)
Zb = TSNE(n_components=2, perplexity=30, init="pca",
random_state=0).fit_transform(threeb)
def gap(Z, a, b):
return float(np.linalg.norm(Z[tl == a].mean(0) - Z[tl == b].mean(0)))
print("\n2. DISTANCE BETWEEN CLUSTERS IS NOT MEANINGFUL")
print(f" true centre gaps: 0-1 {gap(threeb, 0, 1):6.2f}"
f" 1-2 {gap(threeb, 1, 2):6.2f}"
f" ratio {gap(threeb, 1, 2) / gap(threeb, 0, 1):.1f}x")
print(f" t-SNE centre gaps: 0-1 {gap(Zb, 0, 1):6.2f}"
f" 1-2 {gap(Zb, 1, 2):6.2f}"
f" ratio {gap(Zb, 1, 2) / gap(Zb, 0, 1):.1f}x")
print(" The 7x real separation ratio is badly compressed on the plot. You")
print(" cannot conclude that two clusters are 'more different' from the gap.")
# 3. RANDOM NOISE PRODUCES CONVINCING CLUSTERS.
pure = rng.normal(0, 1, (600, 30))
Zn = TSNE(n_components=2, perplexity=8, init="pca",
random_state=0).fit_transform(pure)
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
print("\n3. t-SNE ON PURE NOISE STILL PRODUCES SHAPES")
print(f" 30-D isotropic Gaussian noise, no structure whatever")
for kk in [2, 4, 8]:
lb = KMeans(kk, n_init=10, random_state=0).fit_predict(Zn)
print(f" k={kk}: silhouette on the t-SNE embedding"
f" {silhouette_score(Zn, lb):.3f}")
pure_lab = KMeans(4, n_init=10, random_state=0).fit_predict(pure)
print(f" silhouette on the ORIGINAL 30-D noise, k=4:"
f" {silhouette_score(pure, pure_lab):.3f}")
print(" Low perplexity on noise yields visually separated islands that score")
print(" well on a 2-D silhouette. NEVER cluster a t-SNE embedding and never")
print(" treat apparent islands as evidence of groups -- cluster the original")
print(" space and use t-SNE only to LOOK at the result.")
# --- REPRODUCIBILITY DEPENDS ON init, AND THERE IS NO transform(). ---
print("\nREPRODUCIBILITY: init MATTERS MORE THAN THE FOLKLORE SUGGESTS")
for init in ["pca", "random"]:
runs = [TSNE(n_components=2, perplexity=30, init=init,
random_state=s).fit_transform(Xs) for s in range(3)]
# Compare runs by their pairwise-distance profiles, which ignores the
# arbitrary rotation and reflection of the embedding.
cross = [np.corrcoef(pdist(runs[i]), pdist(runs[j]))[0, 1]
for i, j in [(0, 1), (0, 2), (1, 2)]]
print(f" init={init:7s} 15-NN kept"
f" {[round(knn_preservation(Xs, Z), 3) for Z in runs]}")
print(f" {'':12s} pairwise-distance correlation between runs"
f" {np.round(cross, 3)}")
print(" With init='pca' -- sklearn's default since 1.2 -- the starting point is")
print(" deterministic and the runs agree almost exactly. With init='random' the")
print(" coordinates differ between seeds while the NEIGHBOURHOOD structure is")
print(" reproducible. So 't-SNE is non-deterministic' is a statement about the")
print(" initialisation, not about the method.")
print(f"\n TSNE has a transform method: {hasattr(TSNE(), 'transform')}")
print(" There is no transform(): t-SNE learns POSITIONS, not a mapping. Adding")
print(" one new row means refitting the whole embedding, which rules t-SNE out")
print(" of any pipeline that must embed new data. It is a visualisation tool,")
print(" not a preprocessing step.")
LOCAL VS GLOBAL (30-D data, 2-D embedding)
method 15-NN kept distance rank r
PCA 0.203 0.961
t-SNE (perp 30) 0.386 0.863
^^^^^ ^^^^^
nearly double deliberately worse
WHAT THE HEAVY TAIL IS FOR (the crowding problem)
density
| Gaussian (high-D) a Student-t decays SLOWLY, so a
| /\ moderately distant pair can sit far
| / \ apart in 2-D at little cost.
|/ \___ without it, everything crowds into
| Student-t (low-D) the centre of the plot.
|/ \_______
+-------------------> distance
PERPLEXITY = EFFECTIVE NUMBER OF NEIGHBOURS
perplexity 15-NN kept distance rank r seconds
5 0.338 0.747 2.02
15 0.381 0.801 2.24
30 0.386 0.863 2.49
50 0.382 0.847 3.05
100 0.341 0.870 3.54
CAVEAT 1: CLUSTER SIZE IS NOT MEANINGFUL
original radius: tight 0.476 loose 6.366 ratio 13.4x
t-SNE radius: tight 8.706 loose 8.747 ratio 1.0x
real data: t-SNE plot:
. ( ..:.. )
( ... ) -> ( ..:.. ) both drawn the same size
( ... )
( diffuse )
CAVEAT 2: DISTANCE BETWEEN CLUSTERS IS NOT MEANINGFUL
true centre gaps: 0-1 4.94 1-2 35.01 ratio 7.1x
t-SNE centre gaps: 0-1 44.73 1-2 47.21 ratio 1.1x
CAVEAT 3: PURE NOISE PRODUCES CONVINCING ISLANDS
30-D isotropic Gaussian noise. NO structure exists.
silhouette on the t-SNE embedding, k=2: 0.345
silhouette on the t-SNE embedding, k=4: 0.355
silhouette on the t-SNE embedding, k=8: 0.343
silhouette on the ORIGINAL 30-D noise, k=4: 0.026
The embedding scores 14x better than the data it came from, because
the islands were created by the algorithm, not found in the data.
REPRODUCIBILITY DEPENDS ON init
init=pca 15-NN kept [0.386, 0.386, 0.386]
pairwise-distance correlation between runs [1. 1. 1.]
init=random 15-NN kept [0.385, 0.387, 0.384]
pairwise-distance correlation [0.932 0.981 0.939]
NO transform(): TSNE has a transform method -> False
Key Takeaways
- t-SNE converts distances to neighbour probabilities in both spaces and minimises the KL divergence between them, so it optimises neighbourhoods rather than distances.
- The per-point Gaussian bandwidth equalises density, which is why apparent cluster size on the plot is meaningless.
- The heavy-tailed low-dimensional kernel solves the crowding problem by letting moderately distant points separate cheaply.
- Perplexity is the effective neighbour count, useful between roughly 5 and 50, and you should always inspect several values.
- Distances between clusters on the plot do not reflect real separation, so no conclusion about "how different" two clusters are is licensed.
- t-SNE on pure noise produces islands that score well on a 2-D silhouette, so never cluster an embedding and never read islands as groups.
- Reproducibility is a property of
init: withinit="pca"runs agree almost exactly; withinit="random"only the neighbourhood structure is stable.- There is no
transform(), so t-SNE cannot embed new rows and belongs in exploration rather than in a pipeline.
🧪 Practice
- Embed the same high-dimensional dataset with PCA and t-SNE, and compute both k-NN preservation and distance-rank correlation for each.
- Sweep perplexity over 5, 10, 30, 50, and 100 and describe how the embedding changes qualitatively and quantitatively.
- Build one tight and one diffuse cluster, embed them, and compare the radius ratio before and after.
- Build three clusters with very unequal real separations and compare the gap ratios before and after.
- Run t-SNE on pure Gaussian noise at perplexity 5, then cluster the embedding and report the silhouette. Compare with clustering the original noise.
- Run t-SNE three times with
init="pca"and three times withinit="random", comparing pairwise-distance correlations across runs.- Interview: A colleague shows a t-SNE plot with two tight islands far apart and concludes there are two very distinct customer types. Name three separate reasons that conclusion may not follow. (Hint: consider what the gap means, what the island sizes mean, and what the plot would look like on noise.)
UMAP
UMAP is the method that largely displaced t-SNE in practice, and it is worth separating what it genuinely does better from what it is merely reputed to do better.
The mechanism is a graph construction rather than a probability matching. Three steps:
- Build a weighted k-nearest-neighbour graph. For each point, find its
n_neighborsnearest neighbours and assign edge weights from a locally adaptive kernel -- adaptive because each point gets its own distance scale, just as t-SNE sets a per-point bandwidth. 2. Symmetrise the graph, which gives a fuzzy topological representation of the data. 3. Lay it out in low dimensions by minimising a cross-entropy between the high-dimensional and low-dimensional edge weights, using stochastic gradient descent with negative sampling.
The theoretical framing comes from topology, but the practical shape is recognisable: a neighbourhood graph, then a force-directed layout where edges pull and non-edges push.
Two parameters matter:
n_neighborsis t-SNE's perplexity by another name and behaves the same way. Small values emphasise fine local detail; large values pull toward global structure. The demonstration shows k-NN preservation falling from 0.423 atn_neighbors=5to 0.344 at 200, while distance-rank correlation rises from 0.716 to 0.882 -- the same local-versus-global trade.min_distcontrols how tightly points may pack in the embedding. It is essentially cosmetic: it changes how clumped the picture looks, not what structure was found. The demonstration shows mean nearest-neighbour gap rising from 0.047 to 0.188 asmin_distgoes from 0.0 to 0.99.
Now the claim to be careful about. UMAP is widely described as preserving global structure better than t-SNE. The demonstration tests that directly with five clusters arranged along a line, so their ordering is real information:
- PCA preserves the ordering perfectly (1.000).
- t-SNE preserves it perfectly (1.000).
- UMAP does not (0.700), and its distance-rank correlation is lower too (0.806 against 0.883).
On this dataset UMAP is worse at global structure than t-SNE. That is one dataset and not a general ranking either way -- which is the point. Treat the claim as dataset-dependent and measure it on your own data rather than inheriting it.
What UMAP reliably does have are two operational advantages.
It has a transform(). UMAP learns a mapping, so it can embed new rows
without refitting. In the demonstration, fitting on 900 rows and transforming
300 unseen ones places 99.3% of them nearest to a training point of their own
cluster. This is the difference that decides whether a method can sit inside a
production pipeline, and t-SNE simply cannot.
It scales better. Both methods use approximate nearest-neighbour graphs,
so neither is quadratic, but UMAP's advantage grows with n: at 1,000 rows
UMAP's per-call overhead makes it slower than t-SNE (2.73s against 3.99s),
while at 12,000 rows it is more than twice as fast (33.7s against 74.3s).
Finally, UMAP inherits every one of t-SNE's interpretation caveats, and
this is frequently forgotten because UMAP's plots look cleaner. Cluster size
is still meaningless: a real radius ratio of 16.9x comes out as 1.1x. And UMAP
of pure 25-dimensional noise scores silhouette 0.370 at k=4 against 0.030 in
the original space. Do not cluster the embedding; do not read islands as
groups. Both methods are for looking, not for measuring.
import time
import warnings
import numpy as np
import pandas as pd
import umap
from sklearn.manifold import TSNE
from sklearn.decomposition import PCA
from sklearn.neighbors import NearestNeighbors
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from scipy.spatial.distance import pdist
from scipy.stats import spearmanr
warnings.filterwarnings("ignore")
rng = np.random.default_rng(149)
def knn_preservation(Xh, Xl, k=15):
a = NearestNeighbors(n_neighbors=k + 1).fit(Xh).kneighbors(
Xh, return_distance=False)[:, 1:]
b = NearestNeighbors(n_neighbors=k + 1).fit(Xl).kneighbors(
Xl, return_distance=False)[:, 1:]
return float(np.mean([len(set(x) & set(y)) / k for x, y in zip(a, b)]))
def global_fidelity(Xh, Xl, sample=700, seed=0):
r = np.random.default_rng(seed)
i = r.choice(len(Xh), min(sample, len(Xh)), replace=False)
return float(spearmanr(pdist(Xh[i]), pdist(Xl[i])).statistic)
# Data with BOTH local structure (5 clusters) and global structure (the clusters
# sit along a line, so their ordering is real information).
n_per = 240
groups = []
for i in range(5):
groups.append(rng.normal(0, 0.55, (n_per, 12)) + np.r_[i * 6.0,
np.zeros(11)])
X = np.vstack(groups)
lab = np.repeat(np.arange(5), n_per)
Q = np.linalg.qr(rng.normal(0, 1, (12, 12)))[0]
Xs = StandardScaler().fit_transform(X @ Q)
print("5 CLUSTERS ARRANGED ALONG A LINE (so their ORDER is real information)")
print(f" {len(Xs)} rows, {Xs.shape[1]} dimensions")
print(f"\n{'method':26s} {'15-NN kept':>11s} {'dist rank r':>12s} {'seconds':>8s}")
results = {}
for name, fn in [
("PCA", lambda: PCA(n_components=2).fit_transform(Xs)),
("t-SNE (perp 30)", lambda: TSNE(n_components=2, perplexity=30,
init="pca", random_state=0).fit_transform(Xs)),
("UMAP (n_neighbors 15)", lambda: umap.UMAP(n_components=2, n_neighbors=15,
min_dist=0.1,
random_state=0).fit_transform(Xs)),
]:
t0 = time.perf_counter()
Z = fn()
el = time.perf_counter() - t0
results[name] = Z
print(f" {name:24s} {knn_preservation(Xs, Z):11.3f}"
f" {global_fidelity(Xs, Z):12.3f} {el:8.2f}")
# Does the embedding preserve the ORDER of the clusters along the line?
def order_fidelity(Z, lab):
"""Correlation between true cluster index and position along the embedding's
own first principal axis: does the 1-2-3-4-5 ordering survive?"""
axis = PCA(n_components=1).fit_transform(Z)[:, 0]
means = np.array([axis[lab == i].mean() for i in range(5)])
return float(abs(spearmanr(np.arange(5), means).statistic))
print(f"\n{'method':26s} {'cluster ORDER preserved':>24s}")
for name, Z in results.items():
print(f" {name:24s} {order_fidelity(Z, lab):24.3f}")
print(" Note the result: PCA and t-SNE preserve the ordering perfectly and UMAP")
print(" does not. UMAP is often described as better at global structure; on this")
print(" dataset it is worse, and its distance-rank correlation is lower too.")
print(" Treat the claim as dataset-dependent and measure it. UMAP's reliable")
print(" advantages are SPEED at scale and having a transform().")
# --- THE TWO PARAMETERS THAT MATTER. ---
print("\nn_neighbors: how much of the neighbourhood graph each point sees")
print(f" {'n_neighbors':>12s} {'15-NN kept':>11s} {'dist rank r':>12s}")
for nn_ in [5, 15, 50, 200]:
Z = umap.UMAP(n_components=2, n_neighbors=nn_, min_dist=0.1,
random_state=0).fit_transform(Xs)
print(f" {nn_:12d} {knn_preservation(Xs, Z):11.3f}"
f" {global_fidelity(Xs, Z):12.3f}")
print(" Small values emphasise fine local detail; large values pull the")
print(" embedding toward global structure. It is t-SNE's perplexity by another")
print(" name, and it behaves the same way.")
print("\nmin_dist: how tightly points may pack in the embedding")
print(f" {'min_dist':>9s} {'15-NN kept':>11s} {'mean NN gap':>12s}")
for md in [0.0, 0.1, 0.5, 0.99]:
Z = umap.UMAP(n_components=2, n_neighbors=15, min_dist=md,
random_state=0).fit_transform(Xs)
d = NearestNeighbors(n_neighbors=2).fit(Z).kneighbors(Z)[0][:, 1]
print(f" {md:9.2f} {knn_preservation(Xs, Z):11.3f} {d.mean():12.4f}")
print(" min_dist is purely cosmetic: it controls how visually clumped the plot")
print(" is, not what structure is found. Low for tight clusters you want to see")
print(" separated, higher for a spread-out layout.")
# --- THE OPERATIONAL DIFFERENCE: UMAP HAS A transform(). ---
# A RANDOM split: the rows are ordered by cluster, so a positional split would
# put entire clusters in the test set and measure something else entirely.
Xtr, Xte, ltr, lte = train_test_split(Xs, lab, test_size=0.25, random_state=0,
stratify=lab)
red = umap.UMAP(n_components=2, n_neighbors=15, random_state=0).fit(Xtr)
Zte = red.transform(Xte) # NEW rows, no refit
print("\nUMAP CAN EMBED NEW DATA; t-SNE CANNOT")
print(f" fitted on {len(Xtr)} rows, then transformed {len(Xte)} unseen rows")
print(f" UMAP has transform(): {hasattr(red, 'transform')}")
print(f" TSNE has transform(): {hasattr(TSNE(), 'transform')}")
# A more interpretable check: does each new point land near its own cluster?
Ztr = red.embedding_
nn1 = NearestNeighbors(n_neighbors=1).fit(Ztr)
nearest = nn1.kneighbors(Zte, return_distance=False)[:, 0]
agree = np.mean(ltr[nearest] == lte)
print(f" held-out rows whose nearest embedded TRAINING point shares their"
f" cluster: {agree:.1%}")
print(" This is the difference that decides pipelines. UMAP learns a mapping,")
print(" so it can sit inside a fitted transformer chain; t-SNE learns positions")
print(" and must be refitted from scratch for every new row.")
# --- SCALING: UMAP AND t-SNE ARE BOTH FAR FASTER THAN NAIVE EXPECTATION. ---
print("\nFIT TIME AS n GROWS")
print(f" {'n':>7s} {'PCA':>7s} {'t-SNE':>8s} {'UMAP':>7s}")
for n in [1000, 4000, 12000]:
Zn = rng.normal(0, 1, (n, 20))
Zn[:, 0] += rng.integers(0, 4, n) * 5.0
ts = []
for fn in [lambda: PCA(2).fit_transform(Zn),
lambda: TSNE(2, init="pca", random_state=0).fit_transform(Zn),
lambda: umap.UMAP(n_components=2, random_state=0).fit_transform(Zn)]:
t0 = time.perf_counter(); fn(); ts.append(time.perf_counter() - t0)
print(f" {n:7,d} {ts[0]:7.2f} {ts[1]:8.2f} {ts[2]:7.2f}")
print(" Both use approximate nearest-neighbour graphs, so neither is quadratic")
print(" in practice. UMAP's edge grows with n, and its per-call overhead makes")
print(" it look slow on small data.")
# --- THE SAME CAVEAT AS t-SNE: THE PLOT IS NOT A METRIC SPACE. ---
tight = rng.normal(0.0, 0.15, (300, 10))
loose = rng.normal(0.0, 2.5, (300, 10))
loose[:, 0] += 14.0
mixed = np.vstack([tight, loose])
Zu = umap.UMAP(n_components=2, random_state=0).fit_transform(mixed)
def spread(Z):
return float(np.sqrt(((Z - Z.mean(0)) ** 2).sum(1).mean()))
print("\nUMAP INHERITS t-SNE'S CAVEATS: CLUSTER SIZE IS STILL NOT MEANINGFUL")
print(f" original radius ratio (loose / tight):"
f" {spread(mixed[300:]) / spread(mixed[:300]):.1f}x")
print(f" UMAP radius ratio:"
f" {spread(Zu[300:]) / spread(Zu[:300]):.1f}x")
pure = rng.normal(0, 1, (700, 25))
Zp = umap.UMAP(n_components=2, n_neighbors=8, random_state=0).fit_transform(pure)
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
lab_emb = KMeans(4, n_init=10, random_state=0).fit_predict(Zp)
lab_raw = KMeans(4, n_init=10, random_state=0).fit_predict(pure)
print(f"\n UMAP of PURE 25-D NOISE, silhouette at k=4:"
f" {silhouette_score(Zp, lab_emb):.3f}")
print(f" the same noise in its ORIGINAL space, k=4:"
f" {silhouette_score(pure, lab_raw):.3f}")
print(" Same warning as t-SNE: do not cluster the embedding, and do not read")
print(" islands as groups. Both methods are for looking, not for measuring.")
THE MECHANISM
1. k-NN graph with locally adaptive edge weights
o---o each point's own distance scale, so density
/| |\ is normalised -- the same trick as t-SNE's
o-+---+-o per-point bandwidth
\| |/
o---o
2. symmetrise -> a fuzzy topological representation
3. lay out in 2-D by cross-entropy + SGD with negative sampling
(edges pull together, sampled non-edges push apart)
ON 5 CLUSTERS ARRANGED ALONG A LINE
method 15-NN kept dist rank r seconds
PCA 0.207 0.934 0.00
t-SNE (perp 30) 0.484 0.883 4.44
UMAP (n_neighbors 15) 0.410 0.806 18.38
cluster ORDER preserved (the one thing the data encodes):
PCA 1.000
t-SNE 1.000
UMAP 0.700 <- WORSE, on this data
UMAP is reputed to preserve global structure better. Here it does
not. Measure it rather than assuming it.
n_neighbors: THE LOCAL/GLOBAL DIAL (perplexity by another name)
n_neighbors 15-NN kept dist rank r
5 0.423 0.716 most local
15 0.410 0.806
50 0.373 0.871
200 0.344 0.882 most global
min_dist: PURELY COSMETIC
min_dist 15-NN kept mean NN gap
0.00 0.417 0.0472 tightly packed
0.10 0.410 0.0652
0.50 0.364 0.1262
0.99 0.326 0.1883 spread out
THE DECISIVE OPERATIONAL DIFFERENCE
fitted on 900 rows, then transformed 300 UNSEEN rows
UMAP has transform(): True
TSNE has transform(): False
held-out rows landing nearest a training point of their own
cluster: 99.3%
SCALING: UMAP'S EDGE GROWS WITH n
n PCA t-SNE UMAP
1,000 0.00 3.99 2.73
4,000 0.00 19.78 26.51
12,000 0.00 74.33 33.66
UMAP INHERITS EVERY t-SNE CAVEAT
cluster size: real radius ratio 16.9x -> UMAP ratio 1.1x
UMAP of pure 25-D noise, silhouette at k=4: 0.370
the same noise in its original space, k=4: 0.030
| PCA | t-SNE | UMAP | |
|---|---|---|---|
| Preserves | Global variance | Local neighbourhoods | Local neighbourhoods |
| Embeds new data | Yes | No | Yes |
| Deterministic | Yes | With init="pca" | No |
| Main parameter | n_components | perplexity | n_neighbors |
| Speed at 12k rows | Instant | 74s | 34s |
| Distances meaningful | Yes | No | No |
| Cluster size meaningful | Yes | No | No |
| Safe to cluster the output | Yes | No | No |
Key Takeaways
- UMAP builds a weighted k-NN graph with locally adaptive scales, symmetrises it, and lays it out by cross-entropy with negative sampling.
n_neighborsis the local-versus-global dial and behaves exactly like t-SNE's perplexity.min_distis cosmetic: it controls visual packing, not what structure is found.- UMAP's reputation for better global structure is dataset-dependent and was not borne out here, where it preserved cluster ordering worse than both t-SNE and PCA.
- Its two reliable advantages are a real
transform()for new data and better scaling asngrows.- At small
nUMAP's overhead can make it slower than t-SNE; its advantage appears in the thousands of rows and grows from there.- Every t-SNE interpretation caveat applies unchanged: cluster sizes and gaps are not meaningful and pure noise still produces convincing islands.
- Cluster in the original space and use either method only to visualise.
🧪 Practice
- Embed the same dataset with PCA, t-SNE, and UMAP, reporting k-NN preservation, distance-rank correlation, and fit time for each.
- Build clusters arranged along a line and measure whether each method preserves their ordering.
- Sweep
n_neighborsover 5, 15, 50, and 200 and show the local-global trade numerically.- Sweep
min_distand confirm it changes the mean nearest-neighbour gap in the embedding while barely changing neighbourhood preservation.- Fit UMAP on 70% of the rows, transform the remaining 30%, and measure what fraction land nearest a training point of their own cluster.
- Time both methods at 1,000, 5,000, and 20,000 rows and find the crossover.
- Run UMAP on pure noise, cluster the embedding, and report the silhouette against the same computed in the original space.
- Interview: You want a 10-dimensional representation of 50 features to feed into a downstream classifier that will score new customers daily. Rank PCA, t-SNE, and UMAP for this task and say why. (Hint: one of the three is immediately disqualified by a single missing method.)
Autoencoders for Compression
An autoencoder is a neural network trained to reproduce its own input, with a deliberate obstacle: somewhere in the middle it must squeeze everything through a layer narrower than the input. That narrow layer is the bottleneck, and because the network cannot pass the data through unchanged, it has to learn a compressed representation that retains whatever is needed to reconstruct.
input (p) -> encoder -> bottleneck (k) -> decoder -> output (p)
^ ^
the representation loss = (output - input)^2
The target is the input itself, so no labels are needed -- which is what makes this unsupervised learning rather than regression.
The first thing to establish is the relationship to PCA, because it clarifies what the network is actually buying. A linear autoencoder with squared-error loss finds the principal subspace. The demonstration confirms it: on linear data, PCA with 3 components and a linear autoencoder with 3 bottleneck units reach test reconstruction MSE of 0.10898 and 0.10895, and the cosines of the principal angles between the two 3-dimensional subspaces are all 1.0000 -- they span the same subspace.
They are not identical in a useful sense, though. The autoencoder's axes are neither ordered nor orthogonal, and it found the answer by gradient descent rather than in closed form. On linear data, PCA is strictly the better tool: unique, ordered, exact, and instant.
The autoencoder earns its keep when the structure is non-linear. The demonstration builds a 2-dimensional manifold curled into 12 dimensions by polynomials, sines, and tanh. At a 2-unit bottleneck, PCA reaches test MSE 0.40890 and the autoencoder 0.02510 -- sixteen times better -- because PCA can only fit a flat subspace through a curved surface while the network can follow the curve.
Note the cost in the same table: 48 parameters for PCA against 974 for the autoencoder, plus a training loop that can fail to converge, hyperparameters to choose, and no uniqueness guarantee.
The non-linearity is doing all the work, and this is worth verifying rather than assuming. Replace the ReLU activations with identity and the deep network collapses algebraically to a single linear map: its test MSE becomes 0.40890, matching PCA to five decimal places. Depth without a non-linearity between the layers buys exactly nothing.
The bottleneck size is the compression dial, and it can be used to
estimate the intrinsic dimension. On data whose true manifold dimension is
2, the autoencoder's error drops sharply from k=1 (0.256) to k=2 (0.029)
and then flattens -- correctly locating the answer. PCA needs 8 components to
reach comparable error, because it is approximating a curved surface with a
flat one.
Two operational properties make autoencoders distinctive among the non-linear methods in this section. They embed new data (the encoder is a function) and they reconstruct it (the decoder is a function). t-SNE can do neither and UMAP only the first. That combination is what makes autoencoders usable for genuine compression and for reconstruction-based anomaly detection, which section 11.4 covers.
Variants worth knowing by name: a denoising autoencoder is trained to reconstruct a clean input from a corrupted one, which forces it to learn structure rather than the identity; a sparse autoencoder penalises bottleneck activations so few units fire per input; and a variational autoencoder learns a distribution over the latent space rather than a point, which makes it generative.
import numpy as np
import pandas as pd
import torch
import torch.nn as nn
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
torch.manual_seed(0)
rng = np.random.default_rng(151)
# --- THE ARCHITECTURE: a bottleneck forces compression. ---
# encoder: p -> ... -> k (k << p, the bottleneck)
# decoder: k -> ... -> p
# loss: reconstruct the INPUT. The target is the input, so no labels are needed.
class AE(nn.Module):
def __init__(self, p, k, hidden=None, act=nn.ReLU):
super().__init__()
if hidden is None: # LINEAR autoencoder: no hidden
self.enc = nn.Linear(p, k, bias=False)
self.dec = nn.Linear(k, p, bias=False)
else:
self.enc = nn.Sequential(nn.Linear(p, hidden), act(),
nn.Linear(hidden, k))
self.dec = nn.Sequential(nn.Linear(k, hidden), act(),
nn.Linear(hidden, p))
def forward(self, x):
z = self.enc(x)
return self.dec(z), z
def train_ae(model, Xtr, epochs=400, lr=1e-2):
opt = torch.optim.Adam(model.parameters(), lr=lr)
Xt = torch.tensor(Xtr, dtype=torch.float32)
for _ in range(epochs):
opt.zero_grad()
out, _ = model(Xt)
loss = ((out - Xt) ** 2).mean() # reconstruct the input itself
loss.backward()
opt.step()
return model
def mse(model, Xa):
with torch.no_grad():
out, _ = model(torch.tensor(Xa, dtype=torch.float32))
return float(((out.numpy() - Xa) ** 2).mean())
# --- CASE 1: LINEAR DATA. A linear autoencoder REDISCOVERS PCA. ---
n, p, k = 3000, 12, 3
lat = rng.normal(0, 1, (n, k))
Xlin = StandardScaler().fit_transform(
lat @ rng.normal(0, 1, (k, p)) + rng.normal(0, 0.35, (n, p)))
Ltr, Lte = train_test_split(Xlin, test_size=0.3, random_state=0)
pca = PCA(n_components=k).fit(Ltr)
pca_mse = float(((pca.inverse_transform(pca.transform(Lte)) - Lte) ** 2).mean())
lin_ae = train_ae(AE(p, k), Ltr, epochs=1500)
print("LINEAR DATA: A LINEAR AUTOENCODER IS PCA IN DISGUISE")
print(f" PCA(3) test reconstruction MSE {pca_mse:.5f}")
print(f" linear AE (3 units) test reconstruction MSE {mse(lin_ae, Lte):.5f}")
# The subspaces should coincide even though the bases differ: compare the
# principal angles between the two 3-D subspaces they span.
W = lin_ae.enc.weight.detach().numpy() # (k, p)
Qa = np.linalg.qr(W.T)[0]
Qb = np.linalg.qr(pca.components_.T)[0]
sv = np.linalg.svd(Qa.T @ Qb, compute_uv=False)
print(f" cosines of the principal angles between the two 3-D subspaces:"
f" {np.round(sv, 4)}")
print(" All near 1.0, so they span the SAME subspace. A linear autoencoder with")
print(" squared-error loss finds the principal subspace -- it just does not")
print(" order the axes or make them orthogonal, which is why PCA remains the")
print(" better tool when the data is linear: closed form, ordered, unique.")
# --- CASE 2: NON-LINEAR DATA. This is where an autoencoder earns its keep. ---
# A 2-D manifold curled into 12 dimensions by a non-linear map.
u = rng.uniform(-1, 1, (n, 2))
Z = np.column_stack([
u[:, 0], u[:, 1], u[:, 0] ** 2, u[:, 1] ** 2, u[:, 0] * u[:, 1],
np.sin(3 * u[:, 0]), np.cos(3 * u[:, 1]), np.tanh(2 * u[:, 0] + u[:, 1]),
np.exp(u[:, 0]) / 3, u[:, 0] ** 3, np.sin(2 * u[:, 1]) * u[:, 0],
np.abs(u[:, 0] - u[:, 1]),
])
Xnl = StandardScaler().fit_transform(Z + rng.normal(0, 0.05, Z.shape))
Ntr, Nte = train_test_split(Xnl, test_size=0.3, random_state=0)
print("\nNON-LINEAR DATA: a 2-D manifold curled into 12 dimensions")
print(f" {'method':34s} {'test MSE':>10s} {'params':>8s}")
for kk in [2, 3, 5]:
pm = PCA(n_components=kk).fit(Ntr)
e = float(((pm.inverse_transform(pm.transform(Nte)) - Nte) ** 2).mean())
print(f" {f'PCA, {kk} components':34s} {e:10.5f} {kk * 12 * 2:8d}")
for kk in [2, 3]:
m = train_ae(AE(12, kk, hidden=32), Ntr, epochs=2500)
npar = sum(q.numel() for q in m.parameters())
print(f" {f'AE, {kk} units, 32 hidden, ReLU':34s} {mse(m, Nte):10.5f}"
f" {npar:8d}")
print(" At the SAME bottleneck size the autoencoder reconstructs far better,")
print(" because it can follow a curved manifold while PCA can only fit a flat")
print(" subspace through it. Note the parameter counts: the gain is bought with")
print(" a much larger model and a training loop that can fail to converge.")
# --- THE ACTIVATION IS WHAT MAKES IT NON-LINEAR. ---
print("\nREMOVE THE NON-LINEARITY AND THE ADVANTAGE DISAPPEARS")
pm2 = PCA(n_components=2).fit(Ntr)
print(f" {'PCA, 2 components':38s} "
f"{float(((pm2.inverse_transform(pm2.transform(Nte)) - Nte) ** 2).mean()):.5f}")
for label, act in [("AE, 2 units, IDENTITY activation", nn.Identity),
("AE, 2 units, ReLU activation", nn.ReLU),
("AE, 2 units, Tanh activation", nn.Tanh)]:
m = train_ae(AE(12, 2, hidden=32, act=act), Ntr, epochs=2500)
print(f" {label:38s} {mse(m, Nte):.5f}")
print(" With identity activations the deep network collapses to one linear map,")
print(" so it cannot beat PCA no matter how many layers it has. The depth is")
print(" worthless without the non-linearity between the layers.")
# --- THE BOTTLENECK SIZE IS THE COMPRESSION DIAL. ---
print("\nBOTTLENECK SIZE VS RECONSTRUCTION (non-linear data, true dimension 2)")
print(f" {'k':>3s} {'PCA test MSE':>13s} {'AE test MSE':>12s}")
for kk in [1, 2, 4, 8, 12]:
pm = PCA(n_components=kk).fit(Ntr)
pe = float(((pm.inverse_transform(pm.transform(Nte)) - Nte) ** 2).mean())
ae = train_ae(AE(12, kk, hidden=32), Ntr, epochs=2000)
print(f" {kk:3d} {pe:13.5f} {mse(ae, Nte):12.5f}")
print(" The AE's error drops sharply from k=1 to k=2 and then flattens, which")
print(" correctly locates the true manifold dimension. PCA needs many more")
print(" components to reach the same error, because it is approximating a")
print(" curved surface with a flat one.")
# --- WHAT AN AUTOENCODER GIVES YOU THAT t-SNE AND UMAP DO NOT. ---
print("\nOPERATIONAL PROPERTIES")
tbl = pd.DataFrame([
("PCA", "yes", "yes", "closed form", "linear only", "exactly"),
("Autoencoder", "yes", "no", "gradient descent", "any", "via the decoder"),
("t-SNE", "no", "no", "gradient descent", "any", "no"),
("UMAP", "yes", "no", "graph + SGD", "any", "approximately"),
], columns=["method", "embeds new data", "deterministic",
"how it is fitted", "structure captured", "reconstructs input"])
print("\n", tbl.to_string(index=False))
print("\n An autoencoder is the only non-linear method here that both embeds new")
print(" data AND reconstructs it, which is what makes it usable for compression")
print(" and for anomaly detection by reconstruction error (section 11.4).")
LINEAR DATA: A LINEAR AUTOENCODER IS PCA IN DISGUISE
PCA(3) test reconstruction MSE 0.10898
linear AE (3 units) test reconstruction MSE 0.10895
cosines of the principal angles between the two 3-D subspaces:
[1. 1. 1.] <- the SAME subspace
but the AE's axes are neither ordered nor orthogonal, and it took
gradient descent to find what PCA gets in closed form.
NON-LINEAR DATA: A 2-D MANIFOLD CURLED INTO 12 DIMENSIONS
method test MSE params
PCA, 2 components 0.40890 48
PCA, 3 components 0.24194 72
PCA, 5 components 0.06133 120
AE, 2 units, 32 hidden, ReLU 0.02510 974
AE, 3 units, 32 hidden, ReLU 0.01609 1039
PCA fits a FLAT subspace: an AE can follow the CURVE:
___________ ___
/ the data \ / \___
/ is curved \ | \
------------------ <- best \ |
flat fit misses the curve \______/
THE ACTIVATION IS DOING ALL THE WORK
PCA, 2 components 0.40890
AE, 2 units, IDENTITY activation 0.40890 <- identical to PCA
AE, 2 units, ReLU activation 0.01498
AE, 2 units, Tanh activation 0.02985
With identity activations, encoder and decoder compose into ONE
linear map, so depth cannot help at all.
THE BOTTLENECK LOCATES THE INTRINSIC DIMENSION
(true manifold dimension is 2)
k PCA test MSE AE test MSE
1 0.64480 0.25559
2 0.40890 0.02871 <- the AE's cliff is here
4 0.15511 0.01058
8 0.01087 0.00417
12 0.00000 0.00158
MSE
|*
| \
| *____ the AE's sharp drop from k=1 to k=2
| \___*___* and flattening afterwards locates the
+----------------> k true dimension. PCA declines gradually
1 2 4 8 12 and tells you nothing about it.
OPERATIONAL PROPERTIES
method embeds new determ. fitted by reconstructs
PCA yes yes closed form exactly
Autoencoder yes no gradient descent via the decoder
t-SNE no no gradient descent no
UMAP yes no graph + SGD approximately
The autoencoder is the only non-linear method here that both embeds
new data AND reconstructs it -- which is what section 11.4 needs.
Key Takeaways
- An autoencoder learns to reproduce its input through a narrow bottleneck, so the bottleneck must carry a compressed representation.
- The target is the input, which is what makes it unsupervised.
- A linear autoencoder with squared-error loss spans the same subspace as PCA, confirmed here by principal angles of 1.0.
- On linear data PCA is strictly better: closed-form, unique, ordered, and instant.
- On non-linear manifolds an autoencoder reconstructs far better at the same bottleneck size, because it can follow curvature.
- Removing the activation functions collapses the network to one linear map and the advantage disappears entirely, so the non-linearity is essential, not the depth.
- The bottleneck size can estimate intrinsic dimension: error drops sharply up to the true dimension and flattens after it.
- Autoencoders both embed new data and reconstruct it, which no other non-linear method here does, and which is what enables reconstruction-based anomaly detection.
🧪 Practice
- Train a linear autoencoder and PCA with the same number of components on linear data, then compare reconstruction error and the principal angles between their subspaces.
- Build a non-linear manifold in 10 or more dimensions and compare PCA against a non-linear autoencoder at matched bottleneck size.
- Replace all activations with identity and verify the autoencoder's error matches PCA's.
- Sweep the bottleneck size from 1 to
pand plot reconstruction error, then identify the cliff and compare it with the true intrinsic dimension.- Train a denoising autoencoder by adding Gaussian noise to the input while keeping the clean target, and compare its representation's usefulness.
- Use the trained encoder as a preprocessing step for a downstream classifier and compare against PCA features at the same dimension.
- Interview: Your autoencoder achieves near-zero reconstruction error with a bottleneck of 2 on 100-dimensional data, and a colleague is suspicious. Name one legitimate explanation and one thing you would check. (Hint: what would near-zero error mean about the intrinsic dimension, and what could leak through a bottleneck that is not really a bottleneck?)
Manifold Learning Intuition
This topic is the idea underneath t-SNE, UMAP, Isomap, and LLE alike. Once you have it, all four stop looking like separate tricks.
The manifold hypothesis: real high-dimensional data does not fill its ambient space. It lies on, or near, a low-dimensional surface embedded within it. A 64-by-64 grayscale image has 4096 dimensions, but the set of images of one object rotating traces out a curve -- one dimension, the rotation angle -- living inside that 4096-dimensional space. The intrinsic dimension is the dimension of the surface; the ambient dimension is the number of columns you happen to store.
The demonstration makes this concrete: a 2-dimensional surface rotated into 3, 10, 50, and 200 ambient dimensions always needs just 2 PCA components for 90% of variance and 3 for 99%, regardless of how many columns it is stored in. That gap between 2 and 3 is the second key idea: a linear method needs 3 components to contain a curved 2-dimensional surface, because the surface is not flat. Getting down to the true 2 requires unrolling it.
Which brings us to the central insight: the right notion of distance is along the surface, not through the ambient space. On a rolled-up sheet, two points on adjacent layers are close in straight-line distance and far apart along the sheet. The demonstration finds point pairs on a Swiss roll that are 11 times farther geodesically than Euclidean, and the overall correlation between the two metrics is only 0.269. Any method that trusts Euclidean distance will fuse the layers.
Geodesic distance is approximated the same way by every method here: build a k-nearest-neighbour graph (short hops are trustworthy even if long ones are not), then take shortest paths through that graph. This is the shared foundation, and the methods differ only in what they do with it:
| Method | What it preserves from the k-NN graph |
|---|---|
| Isomap | Geodesic distances, via classical MDS on the shortest-path matrix |
| LLE | Local reconstruction weights: each point as a weighted sum of neighbours |
| Spectral embedding | Eigenvectors of the graph Laplacian -- a diffusion view |
| t-SNE / UMAP | Neighbour probabilities rather than distances |
The demonstration runs all of them on a Swiss roll. Isomap achieves geodesic correlation 0.999 and unrolls perfectly -- unsurprisingly, since it optimises exactly that quantity by construction. LLE and spectral embedding also unroll it (1.000 and 0.999) with lower geodesic fidelity. PCA and metric MDS both trust Euclidean distance and fail to unroll at all (0.213 and 0.196).
Two shared failure modes deserve as much attention as the successes, because they apply to every method in this family including t-SNE and UMAP.
k decides what "nearby" means, and both extremes break. Too small and
the graph fragments into disconnected components, so some pairs have no path
at all -- at k=4 the demonstration's graph has 2 pieces. Too large and the
graph short-circuits across the gap between layers, reconnecting what the
manifold keeps apart: at k=40 the unrolling score collapses from 1.000 to
0.273. The working range here is roughly 6 to 20, and it must be checked, not
assumed.
Noise off the manifold creates the same short circuits. As the Swiss roll's noise rises from 0.05 to 1.5, the unrolling score falls from 1.000 to 0.193. Once the noise is comparable to the gap between layers, neighbours on different layers genuinely are close, and no graph can tell them apart.
The honest summary: manifold learning assumes the manifold is well-sampled and the noise is small relative to its curvature. Verify that before trusting any of these embeddings -- and note that this assumption is exactly what fails on most real high-dimensional data, which is why t-SNE and UMAP, with their probabilistic and more forgiving neighbourhood notions, are what people actually use.
import numpy as np
import pandas as pd
from sklearn.manifold import Isomap, LocallyLinearEmbedding, SpectralEmbedding, MDS
from sklearn.decomposition import PCA
from sklearn.datasets import make_swiss_roll, make_s_curve
from sklearn.neighbors import NearestNeighbors, KNeighborsRegressor
from sklearn.utils.graph import _fix_connected_components
from scipy.sparse.csgraph import shortest_path
from scipy.stats import spearmanr
from scipy.spatial.distance import pdist
rng = np.random.default_rng(157)
# --- THE MANIFOLD HYPOTHESIS, STATED PRECISELY. ---
# Real high-dimensional data often lies on (or near) a low-dimensional surface
# embedded in the high-dimensional space. A 64x64 grayscale image has 4096
# dimensions; the set of images of ONE object rotating traces out a curve.
print("THE MANIFOLD HYPOTHESIS: HOW MUCH OF THE SPACE DOES DATA OCCUPY?")
for p in [3, 10, 50, 200]:
# Points on a 2-D manifold (a swiss roll), then rotated into p dimensions.
t = rng.uniform(0, 4 * np.pi, 2000)
h = rng.uniform(0, 5, 2000)
base = np.column_stack([t * np.cos(t), h, t * np.sin(t)]) # always 3 columns
base = np.column_stack([base, np.zeros((2000, p - 3))]) # pad to p
Q = np.linalg.qr(rng.normal(0, 1, (p, p)))[0]
M = base @ Q
ev = np.cumsum(PCA().fit(M).explained_variance_ratio_)
n90 = int(np.searchsorted(ev, 0.90)) + 1
n99 = int(np.searchsorted(ev, 0.99)) + 1
print(f" {p:4d} ambient dimensions: {n90} components for 90% of variance,"
f" {n99} for 99% (intrinsic dimension 2)")
print(" However many dimensions the data is stored in, it occupies a 2-D")
print(" surface, and PCA needs no more than 3 components to contain it. That is")
print(" why dimensionality reduction is possible at all. But note the gap:")
print(" a LINEAR method needs 3 components to hold a 2-D surface, because the")
print(" surface is curved. Getting down to the true 2 requires unrolling it.")
# --- THE CORE INSIGHT: GEODESIC DISTANCE, NOT EUCLIDEAN DISTANCE. ---
# On a rolled-up sheet, two points on opposite layers are close in straight-line
# distance and far apart ALONG the sheet. Euclidean distance is the wrong metric.
X, colour = make_swiss_roll(n_samples=1500, noise=0.05, random_state=0)
def geodesic_matrix(X, k=10):
"""Build a k-NN graph and take shortest paths through it. Distance along the
graph approximates distance along the manifold, which is the whole idea."""
g = NearestNeighbors(n_neighbors=k).fit(X).kneighbors_graph(mode="distance")
return shortest_path(g, method="D", directed=False)
G = geodesic_matrix(X, k=10)
E = np.sqrt(((X[:, None, :] - X[None, :, :]) ** 2).sum(-1))
# The pairs where the two metrics disagree most are exactly the "shortcut" pairs:
# close through the ambient space, far along the sheet.
ratio = np.where(E > 1e-9, G / np.maximum(E, 1e-9), 1.0)
iu = np.triu_indices_from(E, 1)
worst = np.argsort(ratio[iu])[-3:]
print("\nEUCLIDEAN VS GEODESIC ON A SWISS ROLL")
print(f" {'pair':>14s} {'euclidean':>10s} {'geodesic':>9s} {'ratio':>7s}")
for w in worst[::-1]:
i, j = iu[0][w], iu[1][w]
print(f" {f'{i}-{j}':>14s} {E[i, j]:10.3f} {G[i, j]:9.3f}"
f" {ratio[i, j]:7.1f}x")
print(f" correlation(euclidean, geodesic) over all pairs:"
f" {np.corrcoef(E[iu], G[iu])[0, 1]:.3f}")
print(" Some point pairs are 11x farther along the sheet than through space.")
print(" Any method that trusts Euclidean distance will fuse those layers.")
print("""
cross-section of the roll: ambient distance vs along the sheet
___ A . . . . . B <- Euclidean: short
/ \\ \\ |
| A | B \\_________| <- geodesic: long
| . | . (you must walk around the spiral)
\\_____/
""")
# --- THE METHOD FAMILY, AND WHAT EACH ONE OPTIMISES. ---
def knn_preservation(Xh, Xl, k=12):
a = NearestNeighbors(n_neighbors=k + 1).fit(Xh).kneighbors(
Xh, return_distance=False)[:, 1:]
b = NearestNeighbors(n_neighbors=k + 1).fit(Xl).kneighbors(
Xl, return_distance=False)[:, 1:]
return float(np.mean([len(set(x) & set(y)) / k for x, y in zip(a, b)]))
def unroll_score(Z, colour):
"""Does the embedding recover the position ALONG the roll? That is the one
thing the data actually encodes, so it is the honest success measure."""
return max(abs(spearmanr(Z[:, d], colour).statistic) for d in range(Z.shape[1]))
print("THE FAMILY, ON THE SWISS ROLL (intrinsic dimension 2)")
print(f" {'method':32s} {'12-NN kept':>11s} {'unrolled?':>10s}"
f" {'geodesic r':>11s}")
embeds = {
"PCA (linear)": PCA(n_components=2).fit_transform(X),
"MDS (metric, Euclidean)": MDS(n_components=2, normalized_stress="auto",
random_state=0).fit_transform(X),
"Isomap (geodesic MDS)": Isomap(n_neighbors=10,
n_components=2).fit_transform(X),
"LLE (local weights)": LocallyLinearEmbedding(
n_neighbors=12, n_components=2, random_state=0).fit_transform(X),
"spectral (graph Laplacian)": SpectralEmbedding(
n_components=2, n_neighbors=12, random_state=0).fit_transform(X),
}
for name, Z in embeds.items():
idx = rng.choice(len(X), 600, replace=False)
gr = spearmanr(G[np.ix_(idx, idx)][np.triu_indices(600, 1)],
pdist(Z[idx])).statistic
print(f" {name:32s} {knn_preservation(X, Z):11.3f}"
f" {unroll_score(Z, colour):10.3f} {gr:11.3f}")
print(" Isomap wins on geodesic fidelity BY CONSTRUCTION: it runs classical MDS")
print(" on the geodesic matrix, so it is optimising exactly that column. PCA and")
print(" metric MDS both trust Euclidean distance and fail to unroll.")
# --- WHAT EACH METHOD ACTUALLY DOES, IN ONE LINE EACH. ---
print("\n Isomap k-NN graph -> shortest paths -> classical MDS on those")
print(" LLE write each point as a weighted sum of its neighbours,")
print(" then find low-D coordinates preserving those WEIGHTS")
print(" Spectral eigenvectors of the graph Laplacian: a diffusion view")
print(" t-SNE/UMAP match neighbour PROBABILITIES rather than distances")
print(" All five build the same first object -- a k-NN graph -- and differ only")
print(" in what they preserve from it.")
# --- THE SHARED FAILURE MODE: k CONTROLS WHETHER THE GRAPH IS RIGHT. ---
print("\nk IS THE CRITICAL PARAMETER: IT DECIDES WHAT 'NEARBY' MEANS")
print(f" {'k':>4s} {'graph pieces':>13s} {'12-NN kept':>11s} {'unrolled?':>10s}")
for k in [4, 6, 10, 20, 40, 80]:
g = NearestNeighbors(n_neighbors=k).fit(X).kneighbors_graph(mode="distance")
from scipy.sparse.csgraph import connected_components
ncomp = connected_components(g, directed=False)[0]
Z = Isomap(n_neighbors=k, n_components=2).fit_transform(X)
print(f" {k:4d} {ncomp:13d} {knn_preservation(X, Z):11.3f}"
f" {unroll_score(Z, colour):10.3f}")
print(" Too small and the graph fragments into disconnected pieces, so there is")
print(" no path between some points at all. Too large and the graph SHORT-")
print(" CIRCUITS across the gap between layers, reconnecting what the manifold")
print(" keeps apart -- and the unrolling collapses.")
# --- NOISE OFF THE MANIFOLD BREAKS THE GRAPH. ---
print("\nNOISE PUSHES POINTS OFF THE SURFACE AND CREATES SHORTCUTS")
print(f" {'noise sd':>9s} {'unrolled?':>10s} {'geodesic r':>11s}")
for sd in [0.05, 0.4, 0.8, 1.5]:
Xn, cn = make_swiss_roll(n_samples=1500, noise=sd, random_state=0)
Z = Isomap(n_neighbors=10, n_components=2).fit_transform(Xn)
Gn = geodesic_matrix(Xn, k=10)
idx = rng.choice(len(Xn), 600, replace=False)
gr = spearmanr(Gn[np.ix_(idx, idx)][np.triu_indices(600, 1)],
pdist(Z[idx])).statistic
print(f" {sd:9.2f} {unroll_score(Z, cn):10.3f} {gr:11.3f}")
print(" Once the noise is comparable to the gap between layers, neighbours on")
print(" different layers appear genuinely close and the graph cannot tell them")
print(" apart. Manifold learning assumes the manifold is well-sampled and the")
print(" noise is small relative to its curvature -- check that before trusting")
print(" any of these embeddings.")
THE MANIFOLD HYPOTHESIS
3 ambient dimensions: 2 components for 90%, 3 for 99%
10 ambient dimensions: 2 components for 90%, 3 for 99%
50 ambient dimensions: 2 components for 90%, 3 for 99%
200 ambient dimensions: 2 components for 90%, 3 for 99%
Intrinsic dimension is 2 however many columns store it. The 2-vs-3
gap is the curvature: a FLAT method needs an extra dimension to
contain a curved surface.
EUCLIDEAN VS GEODESIC: THE CENTRAL PROBLEM
pair euclidean geodesic ratio
421-907 6.182 70.682 11.4x
371-421 6.222 71.104 11.4x
447-1423 6.329 72.313 11.4x
correlation(euclidean, geodesic) over all pairs: 0.269
cross-section of the roll:
___ A . . . . . B <- Euclidean: short
/ \ \ |
| A | B \_________| <- geodesic: long
| . | . (you must walk around the spiral)
\_____/
Every method here approximates the geodesic the same way:
k-NN graph -> shortest paths through it
THE FAMILY, ON THE SWISS ROLL
method 12-NN kept unrolled? geodesic r
PCA (linear) 0.416 0.213 0.389
MDS (metric, Euclidean) 0.497 0.196 0.405
Isomap (geodesic MDS) 0.860 1.000 0.999
LLE (local weights) 0.618 1.000 0.775
spectral (graph Laplacian) 0.319 0.999 0.889
Isomap tops the geodesic column BY CONSTRUCTION -- that is the
quantity it optimises. PCA and metric MDS trust Euclidean distance
and never unroll.
FAILURE MODE 1: k IS CRITICAL AT BOTH ENDS
k graph pieces 12-NN kept unrolled?
4 2 0.717 0.998 <- graph FRAGMENTED
6 1 0.816 1.000
10 1 0.860 1.000
20 1 0.924 1.000
40 1 0.336 0.273 <- SHORT-CIRCUITED
80 1 0.321 0.196
k too small: k too large:
o o o o---o---o
(no path) \ /|\ / edges jump between
o o o o-+-o layers, welding the
roll shut
FAILURE MODE 2: NOISE CREATES THE SAME SHORT CIRCUITS
noise sd unrolled? geodesic r
0.05 1.000 1.000
0.40 1.000 0.999
0.80 0.456 0.926
1.50 0.193 0.857
Once noise is comparable to the inter-layer gap, points on different
layers genuinely ARE close, and no graph can separate them.
Key Takeaways
- The manifold hypothesis says data occupies a low-dimensional surface inside a high-dimensional space, which is why dimensionality reduction works at all.
- Intrinsic dimension is a property of the data; ambient dimension is a property of how you stored it.
- A linear method needs an extra component to contain a curved surface, so PCA needs 3 dimensions for a 2-dimensional roll.
- The correct distance is along the manifold, not through the ambient space, and the two can differ by an order of magnitude.
- Every method in this family approximates geodesic distance the same way: a k-NN graph plus shortest paths, differing only in what they preserve from it.
- Isomap preserves geodesic distances, LLE preserves local reconstruction weights, spectral methods use the graph Laplacian, and t-SNE and UMAP preserve neighbour probabilities.
kbreaks the graph at both extremes: too small fragments it, too large short-circuits across manifold layers.- Noise off the manifold creates the same short circuits, so these methods assume dense sampling and small noise relative to curvature.
🧪 Practice
- Generate a 2-dimensional manifold, embed it in 50 dimensions by a random rotation, and report how many PCA components reach 90% and 99% of variance.
- Build a k-NN graph on a Swiss roll, compute shortest-path distances, and find the point pairs where geodesic and Euclidean distance disagree most.
- Compare PCA, MDS, Isomap, LLE, and spectral embedding on the Swiss roll, measuring neighbourhood preservation and whether the roll was unrolled.
- Sweep
kfrom 3 to 100 for Isomap, recording the number of connected graph components and the unrolling score, and identify both failure regimes.- Increase the Swiss roll's noise until the unrolling breaks, and report the noise level at which it does.
- Estimate the intrinsic dimension of a real dataset using the autoencoder bottleneck method from the previous topic and compare it with the PCA scree elbow.
- Interview: You apply Isomap to 5,000 customer records with 80 features and the embedding is a single blob with no structure. Name three explanations. (Hint: one concerns the graph, one concerns whether a manifold exists here at all, and one concerns how densely 5,000 points can sample 80 dimensions.)
<a id="113-pattern-discovery"></a>
11.3 Pattern Discovery
Pattern discovery mines co-occurrence rather than grouping rows or compressing columns: which items appear together, in what order, under what latent themes; this section covers the metrics that tell a real association from a popular one, the pruning principle that makes the search tractable at all, and the factorisations that recover latent structure from sparse tables.
Association Rule Mining
The canonical question is "what goes with what": customers who buy X also buy
Y. The output is a rule A -> C, read as "baskets containing the
antecedent A tend also to contain the consequent C". The subtlety is
entirely in how you measure "tend to", because the obvious measure is
misleading.
Every metric is a function of just three probabilities: P(A), P(C), and
P(A and C). Everything below is arithmetic on those.
| Metric | Formula | Answers | Range |
|---|---|---|---|
| Support | P(A and C) | How often does this pattern occur? | 0 to 1 |
| Confidence | P(C given A) | How reliable is the rule? | 0 to 1 |
| Lift | confidence / P(C) | Is it better than chance? | 0 to infinity |
| Leverage | P(A and C) - P(A)P(C) | How large is the excess in absolute terms? | -0.25 to 0.25 |
| Conviction | (1 - P(C)) / (1 - confidence) | How much more often would the rule be wrong if A and C were independent? | 0 to infinity |
Support tells you whether a rule is worth acting on at all. A rule holding in three baskets out of two thousand is not a business opportunity whatever else is true of it.
Confidence is the trap, and it is the metric people reach for first. In
the demonstration, bread -> milk has confidence 0.808, which looks strong.
But P(milk) on its own is 0.869. Knowing someone bought bread changes the
probability of milk from 0.869 to 0.808 -- it goes down. Confidence rewards
popular consequents, so any rule ending in your most common item will look
impressive.
Lift fixes that by dividing out the consequent's base rate. A lift of 1.0
means independence, above 1.0 means positive association, below means
negative. For bread -> milk the lift is 0.93: essentially independent.
But lift has its own trap: it is a ratio, so it explodes for rare items. The demonstration plants two items appearing in 3 baskets out of 1857 that always co-occur. Lift is 619.0 -- enormous and meaningless. Leverage, being a difference, correctly reports 0.0016. Always floor by minimum support before ranking by lift.
One structural point: lift is symmetric and confidence is not. In the
demonstration nappies -> beer and beer -> nappies have identical lift
(3.356) and very different confidence (0.602 against 0.399), because
P(A,C)/(P(A)P(C)) does not care which side is which. Which direction to act
on is a business decision, not a statistical one -- and the difference is
large enough to matter.
The practical recipe: floor by support, rank by lift, read confidence as the
operational hit rate you would get if you acted on the rule. Note also what
the demonstration's confidence ranking surfaces -- butter -> bread at
confidence 1.000, true but uninformative, because butter was only ever
generated alongside bread. High confidence often restates how the data was
produced.
import numpy as np
import pandas as pd
from itertools import combinations
from mlxtend.preprocessing import TransactionEncoder
from mlxtend.frequent_patterns import apriori, association_rules
rng = np.random.default_rng(163)
# --- A SMALL, HAND-CHECKABLE BASKET DATASET. ---
# Two planted rules, plus a very popular item that will expose the confidence trap.
items_pool = ["bread", "butter", "jam", "beer", "nappies", "milk", "eggs", "coffee"]
baskets = []
for _ in range(2000):
b = set()
if rng.random() < 0.30: # rule 1: bread -> butter
b |= {"bread"}
if rng.random() < 0.75:
b |= {"butter"}
if rng.random() < 0.12: # rule 2: nappies -> beer
b |= {"nappies"}
if rng.random() < 0.60:
b |= {"beer"}
if rng.random() < 0.80: # milk is in MOST baskets
b |= {"milk"}
for it in ["jam", "eggs", "coffee", "beer", "bread"]:
if rng.random() < 0.10:
b |= {it}
if b:
baskets.append(sorted(b))
te = TransactionEncoder()
M = pd.DataFrame(te.fit_transform(baskets), columns=te.columns_)
n = len(M)
print(f"{n} baskets, {M.shape[1]} distinct items")
print("\nITEM FREQUENCIES (support of each single item)")
for it, s in M.mean().sort_values(ascending=False).items():
print(f" {it:10s} {s:.3f}")
# --- THE FOUR NUMBERS THAT DEFINE A RULE, COMPUTED BY HAND. ---
def stats(ante, cons):
"""Every association-rule metric is a function of three probabilities:
P(A), P(C), and P(A and C). Everything below is arithmetic on those."""
A = M[list(ante)].all(axis=1)
C = M[list(cons)].all(axis=1)
pA, pC, pAC = A.mean(), C.mean(), (A & C).mean()
conf = pAC / pA if pA else 0.0
lift = conf / pC if pC else 0.0
leverage = pAC - pA * pC
conviction = (1 - pC) / (1 - conf) if conf < 1 else np.inf
return dict(support=pAC, confidence=conf, lift=lift, leverage=leverage,
conviction=conviction, pA=pA, pC=pC)
print("\nTHE METRICS, ON THREE RULES")
print(f" {'rule':26s} {'supp':>6s} {'conf':>6s} {'lift':>6s} {'lev':>7s}"
f" {'conv':>6s}")
for ante, cons in [(("bread",), ("butter",)),
(("nappies",), ("beer",)),
(("bread",), ("milk",))]:
s = stats(ante, cons)
label = f"{'+'.join(ante)} -> {'+'.join(cons)}"
print(f" {label:26s} {s['support']:6.3f} {s['confidence']:6.3f}"
f" {s['lift']:6.2f} {s['leverage']:+7.4f} {s['conviction']:6.2f}")
# --- THE CONFIDENCE TRAP: high confidence with NO association. ---
s = stats(("bread",), ("milk",))
print(f"\nTHE CONFIDENCE TRAP")
print(f" bread -> milk has confidence {s['confidence']:.3f}, which looks strong.")
print(f" But P(milk) on its own is {s['pC']:.3f}.")
print(f" Knowing someone bought bread changes P(milk) from {s['pC']:.3f} to"
f" {s['confidence']:.3f}.")
print(f" lift = {s['confidence']:.3f} / {s['pC']:.3f} = {s['lift']:.3f}"
" -- essentially 1.0, meaning INDEPENDENT.")
print(" Confidence alone rewards popular consequents. Any rule whose right-hand")
print(" side is milk will look impressive. Lift divides that popularity out.")
print("\n WHAT EACH METRIC ANSWERS")
print(" support how OFTEN does the pattern occur? (worth acting on?)")
print(" confidence P(C | A) (how reliable?)")
print(" lift confidence / P(C) (better than chance?)")
print(" leverage P(A,C) - P(A)P(C) (absolute excess)")
print(" conviction (1-P(C)) / (1-confidence) (how often wrong?)")
# --- LIFT HAS ITS OWN TRAP: RARE ITEMS GIVE HUGE, MEANINGLESS LIFT. ---
print("\nLIFT'S OWN TRAP: RARITY INFLATES IT")
rare = M.sample(n=1, axis=1, random_state=0).columns[0]
# Construct a synthetic pair of ultra-rare items that co-occur twice.
M2 = M.copy()
M2["rare_x"] = False
M2["rare_y"] = False
idx = rng.choice(n, 3, replace=False)
M2.loc[idx, "rare_x"] = True
M2.loc[idx, "rare_y"] = True
pA = M2["rare_x"].mean(); pC = M2["rare_y"].mean()
pAC = (M2["rare_x"] & M2["rare_y"]).mean()
print(f" rare_x and rare_y each appear in {int(pA * n)} of {n} baskets,")
print(f" and they always co-occur.")
print(f" support {pAC:.5f} <- 3 baskets out of {n}")
print(f" confidence {pAC / pA:.3f}")
print(f" lift {(pAC / pA) / pC:8.1f} <- enormous, and meaningless")
print(f" leverage {pAC - pA * pC:+.6f} <- correctly tiny")
print(" Lift is a RATIO, so it explodes for rare items. Leverage is a")
print(" DIFFERENCE, so it stays small when the absolute excess is small. Always")
print(" filter by minimum support BEFORE ranking by lift.")
# --- RULES ARE DIRECTIONAL, AND LIFT IS NOT. ---
a = stats(("nappies",), ("beer",))
b = stats(("beer",), ("nappies",))
print("\nDIRECTION MATTERS FOR CONFIDENCE, NOT FOR LIFT")
print(f" nappies -> beer confidence {a['confidence']:.3f} lift {a['lift']:.3f}")
print(f" beer -> nappies confidence {b['confidence']:.3f} lift {b['lift']:.3f}")
print(" Identical lift, very different confidence. Lift is symmetric because")
print(" P(A,C)/(P(A)P(C)) does not care which side is which; confidence is not.")
print(" Which direction you act on is a business decision, not a statistical one.")
# --- THE FULL PIPELINE WITH mlxtend, AND WHY YOU FILTER TWICE. ---
freq = apriori(M, min_support=0.02, use_colnames=True)
rules = association_rules(freq, metric="confidence", min_threshold=0.3)
rules["ante"] = rules["antecedents"].apply(lambda s: "+".join(sorted(s)))
rules["cons"] = rules["consequents"].apply(lambda s: "+".join(sorted(s)))
print(f"\nGENERATED RULES: {len(freq)} frequent itemsets -> {len(rules)} rules")
print("\n TOP 6 BY CONFIDENCE")
top_c = rules.nlargest(6, "confidence")
print(f" {'rule':26s} {'supp':>6s} {'conf':>6s} {'lift':>6s}")
for _, r in top_c.iterrows():
print(f" {r['ante'] + ' -> ' + r['cons']:26s} {r['support']:6.3f}"
f" {r['confidence']:6.3f} {r['lift']:6.2f}")
print("\n TOP 6 BY LIFT")
for _, r in rules.nlargest(6, "lift").iterrows():
print(f" {r['ante'] + ' -> ' + r['cons']:26s} {r['support']:6.3f}"
f" {r['confidence']:6.3f} {r['lift']:6.2f}")
print("\n The confidence ranking is topped by 'butter -> bread' at 1.000, which")
print(" is true but uninformative: butter was only ever generated ALONGSIDE")
print(" bread, so the rule restates the data-generating process rather than")
print(" revealing anything. The lift ranking surfaces the nappies/beer")
print(" association instead. Rank by lift, floor by support, and read")
print(" confidence as the hit rate you would get if you acted on the rule.")
THE THREE PROBABILITIES EVERYTHING IS BUILT FROM
all baskets
+-----------------------------------------+
| A | |
| +--------------+ | |
| | P(A and C) | | |
| +--------------+ | |
| C | |
+-----------------------------------------+
support = P(A and C) confidence = P(A and C) / P(A)
lift = confidence / P(C) leverage = P(A,C) - P(A)P(C)
ITEM FREQUENCIES (1857 baskets)
milk 0.869 <- in most baskets
bread 0.401
butter 0.228
beer 0.179
nappies 0.119
THREE RULES, FIVE METRICS
rule supp conf lift lev conv
bread -> butter 0.228 0.569 2.49 +0.1367 1.79
nappies -> beer 0.072 0.602 3.36 +0.0503 2.06
bread -> milk 0.324 0.808 0.93 -0.0245 0.68
^^^^^ ^^^^
looks strong but is independent
THE CONFIDENCE TRAP
bread -> milk confidence 0.808
P(milk) on its own 0.869
knowing about bread moves P(milk) from 0.869 DOWN to 0.808
lift = 0.808 / 0.869 = 0.930 -> no association at all
LIFT'S OWN TRAP: RARITY
two items in 3 baskets out of 1857 that always co-occur:
support 0.00162
confidence 1.000
lift 619.0 <- a ratio, so it explodes
leverage +0.001613 <- a difference, so it stays honest
LIFT IS SYMMETRIC, CONFIDENCE IS NOT
nappies -> beer confidence 0.602 lift 3.356
beer -> nappies confidence 0.399 lift 3.356
^^^^^^^^^^^^^^^^ ^^^^^
direction matters identical
TOP RULES BY EACH METRIC (min_support 0.02, min_confidence 0.3)
BY CONFIDENCE supp conf lift
butter -> bread 0.228 1.000 2.49
beer+butter -> bread 0.039 1.000 2.49
butter+coffee -> bread 0.029 1.000 2.49
BY LIFT
beer+bread -> nappies 0.031 0.445 3.74
beer+bread -> milk+nappies 0.024 0.344 3.59
beer+milk -> nappies 0.058 0.406 3.41
The confidence ranking restates the data-generating process (butter
was only ever generated with bread). The lift ranking surfaces the
nappies/beer association that was actually planted.
Key Takeaways
- Every association metric is arithmetic on
P(A),P(C), andP(A and C).- Support tells you whether a rule occurs often enough to act on, and must be applied as a floor before anything else.
- Confidence rewards popular consequents, so a high-confidence rule can describe two independent items.
- Lift divides out the consequent's base rate, with 1.0 meaning independence.
- Lift is a ratio and therefore explodes for rare items, which is why a support floor must come first; leverage is a difference and does not have this problem.
- Lift is symmetric while confidence is not, so choosing a direction to act on is a business decision.
- Rank by lift, floor by support, and read confidence as the hit rate you would achieve in production.
- A confidence of 1.0 frequently restates how the data was constructed rather than revealing anything.
🧪 Practice
- Compute support, confidence, lift, leverage, and conviction by hand for three rules and check them against
mlxtend'sassociation_rules.- Find a rule in your data with confidence above 0.8 and lift below 1.1, and explain in one sentence why it is worthless.
- Plant two items that co-occur three times in a thousand baskets and report all five metrics, identifying which ones are misled.
- Compare
A -> BagainstB -> Afor several rules and confirm lift is identical while confidence is not.- Rank rules by confidence and by lift separately and describe how the two top-ten lists differ.
- Interview: A retailer's analytics tool reports "customers who buy razors also buy bananas, confidence 0.71" as its top insight. What is your first question? (Hint: what fraction of all customers buy bananas anyway?)
Frequent Itemsets and Apriori
Before you can rank rules you have to find the itemsets worth making rules
from, and that is where the real difficulty lies. With d distinct items
there are 2^d - 1 possible itemsets: about a million at 20 items, 10^30 at
100, and 10^301 at 1000. A supermarket carries tens of thousands of products.
Brute-force counting is not slow; it is impossible.
The entire field rests on one structural property, and it is almost trivially
obvious once stated. Downward closure (the Apriori principle): if an itemset
is frequent, every subset of it is also frequent. Any basket containing {A, B, C} necessarily contains {A, B}, so the smaller set can only occur at
least as often.
The useful form is the contrapositive: if an itemset is infrequent, no
superset of it can be frequent. That converts the search from enumeration
into pruning. The moment {i3, i4} fails your threshold, every set containing
both is eliminated without ever being counted.
Apriori exploits this breadth-first, one size at a time:
- Count all single items; keep the frequent ones. 2. Generate candidates
of size
k+1by joining frequentk-itemsets that share ak-1prefix. 3. Prune any candidate having an infrequent subset -- free, no data access needed. 4. Count the survivors against the database; keep the frequent ones; repeat.
The demonstration tracks the saving: on 14 items, Apriori counts 194 itemsets out of 16,383 possible -- 1.18% of the space -- and recovers exactly the same 83 frequent itemsets as the library implementation.
min_support is the only real control you have, and it is brutal. It sets
both the size of the output and the runtime. Too high and you find only what
you already knew; too low and you drown in patterns. The demonstration shows
itemsets growing from 9 at min_support=0.20 to 374 at 0.01.
FP-Growth computes the same answer differently. Apriori scans the database once per level; FP-Growth compresses the transactions into a prefix tree (the FP-tree) in two scans and mines it recursively, with no candidate generation at all.
Worth being precise about when it wins, because the folklore is unconditional and the reality is not. The demonstration measures both across dataset widths, and the direction reverses: on 14 items Apriori is 5 times faster, because FP-Growth's tree-building overhead dominates. At 500 items FP-Growth is 3.9 times faster, because Apriori's candidate generation grows combinatorially while the tree is built once. Use FP-Growth on wide real transaction data; do not assume it on small data.
Finally, the output is highly redundant: if {A,B,C} is frequent then so
are all seven of its subsets. Two compressions fix this:
- Closed itemsets: those with no superset of equal support. Lossless -- every frequent itemset's support can be recovered from them.
- Maximal itemsets: those with no frequent superset at all. Smaller still, but you lose the subsets' support values.
Report closed itemsets when you need the numbers, maximal when you only need the patterns.
import time
import numpy as np
import pandas as pd
from itertools import combinations
from mlxtend.preprocessing import TransactionEncoder
from mlxtend.frequent_patterns import apriori, fpgrowth
rng = np.random.default_rng(167)
# --- THE COMBINATORIAL PROBLEM. ---
# With d distinct items there are 2^d - 1 possible itemsets. You cannot enumerate
# them, so the whole field rests on one structural property that prunes the space.
print("WHY YOU CANNOT JUST COUNT EVERYTHING")
print(f" {'items':>7s} {'possible itemsets':>20s}")
from math import log10
for d in [10, 20, 50, 100, 1000]:
total = 2 ** d - 1
shown = f"{total:,d}" if d <= 50 else f"about 10^{log10(total):.0f}"
print(f" {d:7d} {shown:>20s}")
print(" A supermarket has tens of thousands of products. Brute force is not")
print(" slow; it is impossible.")
# --- THE APRIORI PRINCIPLE (DOWNWARD CLOSURE). ---
# If an itemset is frequent, every SUBSET of it is frequent -- because any basket
# containing {A,B,C} also contains {A,B}. Contrapositive: if {A,B} is INFREQUENT,
# then no superset of it can be frequent, so you can stop exploring that branch.
items = [f"i{j}" for j in range(14)]
baskets = []
for _ in range(4000):
b = set(rng.choice(items, rng.integers(1, 5), replace=False))
if rng.random() < 0.25: # a planted frequent triple
b |= {"i0", "i1", "i2"}
if rng.random() < 0.15: # a planted frequent pair
b |= {"i5", "i6"}
baskets.append(sorted(b))
te = TransactionEncoder()
M = pd.DataFrame(te.fit_transform(baskets), columns=te.columns_)
n = len(M)
MIN_SUP = 0.05
def support(cols):
return float(M[list(cols)].all(axis=1).mean())
print(f"\nDOWNWARD CLOSURE: {n} baskets, min_support = {MIN_SUP}")
print(f" support({{i0,i1,i2}}) = {support(['i0','i1','i2']):.4f} frequent")
for sub in combinations(["i0", "i1", "i2"], 2):
print(f" support({{{','.join(sub)}}}) = {support(sub):.4f}"
" frequent, necessarily")
print(f" support({{i3,i4}}) = {support(['i3','i4']):.4f} INFREQUENT")
print(f" support({{i3,i4,i7}}) = {support(['i3','i4','i7']):.4f}"
" cannot exceed the above")
print(" So the moment {i3,i4} fails the threshold, every superset of it is")
print(" eliminated without being counted. That is the entire algorithm.")
# --- APRIORI, IMPLEMENTED LEVEL BY LEVEL, COUNTING THE WORK SAVED. ---
def apriori_by_hand(M, min_sup):
"""Breadth-first: find frequent 1-itemsets, use them to generate candidate
2-itemsets, prune, repeat. Report how many candidates each level considered."""
d = list(M.columns)
frequent = {}
level = [(c,) for c in d]
k = 1
stats = []
while level:
checked = len(level)
keep = [(s, support(s)) for s in level]
keep = [(s, v) for s, v in keep if v >= min_sup]
frequent[k] = keep
# CANDIDATE GENERATION: join frequent k-itemsets sharing a k-1 prefix,
# then PRUNE any candidate with an infrequent subset.
freq_set = {s for s, _ in keep}
nxt = set()
for a, b in combinations(sorted(freq_set), 2):
if a[:-1] == b[:-1]:
cand = tuple(sorted(set(a) | set(b)))
if all(tuple(sorted(sub)) in freq_set
for sub in combinations(cand, k)):
nxt.add(cand)
naive = len(list(combinations(d, k + 1))) if k + 1 <= len(d) else 0
stats.append((k, checked, len(keep), len(nxt), naive))
level = sorted(nxt)
k += 1
return frequent, stats
freq, stats = apriori_by_hand(M, MIN_SUP)
print(f"\nAPRIORI LEVEL BY LEVEL")
print(f" {'k':>3s} {'checked':>9s} {'frequent':>9s}"
f" {'next cands':>11s} {'brute force':>13s}")
for k, checked, nfreq, nnext, naive in stats:
print(f" {k:3d} {checked:19d} {nfreq:15d} {nnext:16d} {naive:21,d}")
total_checked = sum(s[1] for s in stats)
print(f" total itemsets counted: {total_checked}"
f" out of {2 ** len(M.columns) - 1:,} possible")
space = 2 ** len(M.columns) - 1
print(f" pruning examined {total_checked / space:.2%} of the space")
print("\n FREQUENT ITEMSETS FOUND, BY SIZE")
for k, sets in freq.items():
if sets:
shown = ", ".join("{" + ",".join(s) + f"}}:{v:.3f}"
for s, v in sorted(sets, key=lambda t: -t[1])[:4])
print(f" k={k}: {len(sets):3d} itemsets e.g. {shown}")
# Verify against the library.
lib = apriori(M, min_support=MIN_SUP, use_colnames=True)
mine = sum(len(v) for v in freq.values())
print(f"\n hand-rolled found {mine} frequent itemsets; mlxtend found {len(lib)}")
# --- MIN_SUPPORT IS THE DIAL, AND IT IS BRUTAL. ---
print("\nMIN_SUPPORT CONTROLS BOTH THE OUTPUT SIZE AND THE RUNTIME")
print(f" {'min_support':>12s} {'itemsets':>9s} {'max size':>9s} {'seconds':>8s}")
for ms in [0.20, 0.10, 0.05, 0.02, 0.01]:
t0 = time.perf_counter()
f = apriori(M, min_support=ms, use_colnames=True)
el = time.perf_counter() - t0
mx = int(f["itemsets"].apply(len).max()) if len(f) else 0
print(f" {ms:12.2f} {len(f):9d} {mx:9d} {el:8.3f}")
print(" Halving min_support can multiply the output several-fold. Too high and")
print(" you find only the obvious; too low and you drown. It is also the only")
print(" real control you have over runtime.")
# --- FP-GROWTH: THE SAME ANSWER, WITHOUT REPEATED DATABASE SCANS. ---
# Apriori scans the data once per level. FP-Growth compresses the transactions
# into a prefix tree (the FP-tree) and mines it recursively -- two scans total.
# The crossover depends on how WIDE the dataset is, so measure it.
def make_baskets(d, nt, seed=5):
"""Zipf-ish item popularity, so frequent items recur and prefixes are shared
-- which is the structure an FP-tree exploits."""
r = np.random.default_rng(seed)
pool = [f"x{j}" for j in range(d)]
w = 1.0 / (np.arange(1, d + 1) ** 0.8)
w /= w.sum()
rows = [sorted(set(r.choice(pool, int(r.integers(5, 14)), replace=False, p=w)))
for _ in range(nt)]
enc = TransactionEncoder()
return pd.DataFrame(enc.fit_transform(rows), columns=enc.columns_)
print("\nAPRIORI VS FP-GROWTH: IDENTICAL RESULTS, DIFFERENT COST")
print(f" {'items':>6s} {'baskets':>8s} {'min_sup':>8s} {'itemsets':>9s}"
f" {'apriori':>9s} {'fpgrowth':>9s} {'speedup':>8s}")
for d, nt, ms in [(14, 4000, 0.02), (60, 8000, 0.02),
(200, 15000, 0.01), (500, 15000, 0.006)]:
Md = M if d == 14 else make_baskets(d, nt)
t0 = time.perf_counter()
a = apriori(Md, min_support=ms, use_colnames=True)
ta = time.perf_counter() - t0
t0 = time.perf_counter()
b = fpgrowth(Md, min_support=ms, use_colnames=True)
tb = time.perf_counter() - t0
same = (set(map(frozenset, a["itemsets"]))
== set(map(frozenset, b["itemsets"])))
print(f" {Md.shape[1]:6d} {len(Md):8d} {ms:8.3f} {len(a):9d}"
f" {ta:9.3f} {tb:9.3f} {ta / tb:7.1f}x"
f" {'identical' if same else 'DIFFER'}")
print(" The itemsets are always identical, so the choice is purely")
print(" computational -- and the direction REVERSES with width. On 14 items")
print(" apriori wins, because FP-Growth's tree-building overhead dominates. By")
print(" 500 items FP-Growth is several times faster, because apriori's candidate")
print(" generation grows combinatorially while the FP-tree is built once. Use")
print(" FP-Growth on wide, real transaction data; do not assume it on small data.")
# --- CLOSED AND MAXIMAL ITEMSETS: COMPRESSING THE OUTPUT. ---
# The output is highly redundant: if {A,B,C} is frequent, so are all 7 subsets.
allf = apriori(M, min_support=0.05, use_colnames=True)
sets = [frozenset(s) for s in allf["itemsets"]]
sup = dict(zip(sets, allf["support"]))
maximal = [s for s in sets if not any(s < t for t in sets)]
closed = [s for s in sets
if not any(s < t and abs(sup[t] - sup[s]) < 1e-12 for t in sets)]
print("\nCOMPRESSING A REDUNDANT OUTPUT")
print(f" all frequent itemsets {len(sets):4d}")
print(f" closed itemsets {len(closed):4d}"
" (no superset has the SAME support)")
print(f" maximal itemsets {len(maximal):4d}"
" (no superset is frequent at all)")
print(" Closed itemsets lose nothing: every frequent set's support can be")
print(" recovered from them. Maximal itemsets are smaller still but lose the")
print(" support values of the subsets. Report closed sets when you need the")
print(" numbers, maximal when you only need the patterns.")
THE SEARCH SPACE IS IMPOSSIBLE TO ENUMERATE
items possible itemsets
10 1,023
20 1,048,575
50 1,125,899,906,842,623
100 about 10^30
1000 about 10^301
DOWNWARD CLOSURE (4000 baskets, min_support 0.05)
support({i0,i1,i2}) = 0.2512 frequent
support({i0,i1}) = 0.2687 frequent, NECESSARILY
support({i0,i2}) = 0.2677 frequent, NECESSARILY
support({i1,i2}) = 0.2695 frequent, NECESSARILY
support({i3,i4}) = 0.0278 INFREQUENT
support({i3,i4,i7}) = 0.0037 cannot possibly exceed it
the lattice, with a pruned branch:
{}
/ | \
{A} {B} {C}
/ \ / \ / \
{A,B} {A,C} {B,C}
\ | /
\ | / X {B,C} infrequent -> {A,B,C} pruned
{A,B,C} without being counted
APRIORI LEVEL BY LEVEL
k candidates frequent next cands brute force would be
1 14 14 91 91
2 91 50 82 364
3 82 16 7 1,001
4 7 3 0 2,002
total counted: 194 of 16,383 possible -> 1.18% of the space
hand-rolled found 83 frequent itemsets; mlxtend found 83
MIN_SUPPORT IS THE ONLY REAL DIAL
min_support itemsets max size seconds
0.20 9 3 0.003
0.10 25 3 0.003
0.05 83 4 0.005
0.02 177 5 0.012
0.01 374 5 0.012
APRIORI VS FP-GROWTH: THE WINNER REVERSES WITH WIDTH
items baskets min_sup itemsets apriori fpgrowth speedup
14 4000 0.020 177 0.007 0.037 0.2x
60 8000 0.020 1539 0.131 0.528 0.2x
200 15000 0.010 1176 0.928 0.830 1.1x
500 15000 0.006 1224 3.661 0.929 3.9x
always IDENTICAL itemsets, so the choice is purely computational.
apriori: scan the data once per level, generate candidates
fp-growth: build a prefix tree once, then recurse on it
transactions FP-tree (shared prefixes compressed)
A B C root
A B D -> / \
A C A ...
B C / \
B C
/ \
C D
COMPRESSING A REDUNDANT OUTPUT
all frequent itemsets 83
closed itemsets 83 no superset has the SAME support (lossless)
maximal itemsets 43 no superset is frequent (loses subset supports)
Key Takeaways
- The number of possible itemsets is exponential in the number of items, so the search must prune rather than enumerate.
- Downward closure says every subset of a frequent itemset is frequent, and its contrapositive is what does the pruning.
- Apriori works breadth-first: generate candidates by joining frequent itemsets, prune those with infrequent subsets, then count only the survivors.
- Pruning reduces the counted space enormously -- to about 1% in the demonstration.
min_supportcontrols both output size and runtime and is the only real dial.- FP-Growth returns identical itemsets by building a prefix tree instead of generating candidates.
- Which is faster reverses with dataset width: Apriori wins on narrow data, FP-Growth on wide data, so measure rather than assume.
- The output is redundant; closed itemsets compress it losslessly and maximal itemsets compress it further at the cost of subset supports.
🧪 Practice
- Verify downward closure empirically by finding a frequent 3-itemset and checking the support of all three of its 2-subsets.
- Implement Apriori level by level with candidate generation and pruning, and count how many itemsets you evaluate against the total possible.
- Confirm your implementation returns the same frequent itemsets as
mlxtend.frequent_patterns.apriori.- Sweep
min_supportover five values and tabulate the number of itemsets, the maximum itemset size, and the runtime.- Time Apriori against FP-Growth on datasets with 15, 100, and 500 distinct items and find the crossover point.
- Compute the closed and maximal itemsets from a frequent-itemset list and report the compression each achieves.
- Interview: You lower
min_supportfrom 0.05 to 0.005 and the job runs out of memory. Explain what happened and name two ways to proceed. (Hint: think about what happens to the number of candidates at level 2, and about what you actually intend to do with the output.)
Sequence Mining
A basket is a set: {login, search, buy} is the same basket however it
happened. A clickstream is a sequence, and login -> search -> buy is a
funnel while buy -> search -> login is someone with a support problem.
Itemset mining cannot tell those apart, which throws away precisely the
information a funnel analysis needs.
Order also makes the combinatorics worse. A set of k items is one itemset
and k! possible orderings: 6 at k=3, 3.6 million at k=10.
Support for a sequence is the fraction of sequences containing the pattern as a subsequence -- order must match, but intervening events are allowed. That last clause is what makes the definition useful: "searched, then eventually bought" is a real pattern even when the user detoured through support and the homepage first.
PrefixSpan is the sequential analogue of Apriori, and the pruning principle is identical: support can only fall as a pattern lengthens, so a prefix that fails the threshold ends that whole branch. Where Apriori generates candidates and counts them against the full database, PrefixSpan grows a prefix and carries a projected database -- for each matching sequence, just the suffix that follows the match. Each recursion works on a smaller database and never re-scans the original.
In the demonstration it finds 94 frequent sequential patterns from 658 candidate evaluations, and its reported supports match the brute-force definition exactly (maximum difference 0.00e+00).
Gap constraints are the refinement that usually matters in practice.
Unconstrained subsequence support counts a purchase three weeks after a search
as "following" it. A max_gap turns the pattern from "eventually" into "soon
after", which is normally what the business question means. The demonstration
shows search -> buy support falling from 0.185 unconstrained to 0.012 at
max_gap=1 -- the same pattern, a completely different claim. Related
constraints worth knowing: min_gap, a maximum total window for the whole
pattern, and item hierarchies so that product_view can match any
specific product.
The payoff is sequential rules, whose confidence is a conversion rate. The demonstration recovers the planted funnel directly: 66% of searchers who view a product reach the cart, and 57% of those buy -- against planted values of 65% and 55%. That is a funnel measured from raw event logs, and no set-based method can produce it.
Two practical notes. Real event logs need sessionisation first -- deciding
where one sequence ends and the next begins, usually by an inactivity timeout
-- and that choice changes every number downstream. And note that the
demonstration's cart -> search pattern has support 0.043 arising purely from
randomly inserted noise events: in sequence mining, background noise
manufactures orderings, so a support floor matters at least as much as it
does for itemsets.
import numpy as np
import pandas as pd
from collections import Counter, defaultdict
rng = np.random.default_rng(173)
# --- WHY ORDER CHANGES THE PROBLEM. ---
# A basket is a SET: {login, search, buy} is the same basket however it happened.
# A sequence is ORDERED: login->search->buy and buy->search->login are different
# behaviours, and only one of them is a funnel.
print("SETS VERSUS SEQUENCES")
print(" as itemsets, these two users are IDENTICAL:")
print(" user A: login, search, cart, buy")
print(" user B: buy, cart, search, login")
print(" as sequences they are opposites. Itemset mining cannot tell them apart,")
print(" which throws away the one thing a funnel analysis needs.")
print("\n the combinatorics are also worse: k items form")
print(" 1 itemset of size k, but k! distinct orderings")
for k in [3, 5, 8, 10]:
from math import factorial
print(f" k={k:2d}: 1 itemset, {factorial(k):,} orderings")
# --- BUILD CLICKSTREAM SEQUENCES WITH TWO PLANTED PATTERNS. ---
EVENTS = ["home", "search", "product", "cart", "buy", "support", "logout"]
seqs = []
for _ in range(3000):
s = ["home"]
if rng.random() < 0.45: # planted: search -> product -> cart
s += ["search", "product"]
if rng.random() < 0.65:
s += ["cart"]
if rng.random() < 0.55:
s += ["buy"] # the funnel
if rng.random() < 0.20: # planted: support -> logout
s += ["support", "logout"]
# background noise events inserted at random positions
for _ in range(int(rng.integers(0, 3))):
s.insert(int(rng.integers(0, len(s) + 1)), str(rng.choice(EVENTS)))
seqs.append(s)
print(f"\n{len(seqs)} sequences, mean length"
f" {np.mean([len(s) for s in seqs]):.1f}")
print(" examples:")
for s in seqs[:4]:
print(" ", " -> ".join(s))
# --- SUPPORT FOR A SEQUENCE: does it appear as a SUBSEQUENCE (gaps allowed)? ---
def contains(seq, pat):
"""Is pat a subsequence of seq? Order must match; intervening events are
allowed. This is the standard definition, and the gaps are why it is useful:
'search then eventually buy' is a real pattern even with detours."""
it = iter(seq)
return all(any(e == p for e in it) for p in pat)
def support(pat, seqs):
return sum(contains(s, pat) for s in seqs) / len(seqs)
print("\nSUBSEQUENCE SUPPORT ALLOWS GAPS")
print(f" {'pattern':38s} {'support':>8s}")
for pat in [("search",), ("search", "product"), ("search", "product", "cart"),
("search", "product", "cart", "buy"), ("cart", "search"),
("support", "logout"), ("logout", "support")]:
print(f" {' -> '.join(pat):38s} {support(pat, seqs):8.3f}")
print(" Note cart -> search has non-trivial support purely from the inserted")
print(" noise events, and logout -> support almost none. Direction is")
print(" informative exactly because it is not symmetric.")
# --- PREFIXSPAN: the sequential analogue of Apriori, by hand. ---
# Same pruning principle: if a sequence is infrequent, no EXTENSION of it can be
# frequent. Instead of generating candidates, PrefixSpan grows a prefix and keeps
# only the PROJECTED database -- the suffixes following each match.
def prefixspan(seqs, min_sup, max_len=6):
"""Depth-first growth of frequent prefixes over projected databases."""
n = len(seqs)
out = []
checked = [0]
def project(db, item):
"""For each sequence, keep whatever follows the FIRST occurrence of item."""
proj = []
for s in db:
if item in s:
proj.append(s[s.index(item) + 1:])
return proj
def grow(prefix, db):
if len(prefix) >= max_len:
return
# count items in the projected database: one pass, no candidate joins
counts = Counter()
for s in db:
for e in set(s):
counts[e] += 1
for item, c in sorted(counts.items()):
checked[0] += 1
sup = c / n
if sup >= min_sup:
newp = prefix + (item,)
out.append((newp, sup))
# recurse on the projection; the full db is never rescanned
grow(newp, project(db, item))
grow((), seqs)
return out, checked[0]
MIN_SUP = 0.05
patterns, checked = prefixspan(seqs, MIN_SUP)
print(f"\nPREFIXSPAN: min_support {MIN_SUP}")
print(f" frequent sequential patterns found: {len(patterns)}")
print(f" candidate extensions evaluated: {checked}")
print(f"\n {'longest patterns':50s} {'support':>8s}")
for pat, sup in sorted(patterns, key=lambda t: (-len(t[0]), -t[1]))[:6]:
print(f" {' -> '.join(pat):50s} {sup:8.3f}")
# Verify the reported supports against the brute-force definition.
worst = max(abs(sup - support(pat, seqs)) for pat, sup in patterns)
print(f"\n max |PrefixSpan support - brute-force support| = {worst:.2e}")
# --- THE PRUNING PRINCIPLE IS THE SAME AS APRIORI'S. ---
print("\nDOWNWARD CLOSURE HOLDS FOR SEQUENCES TOO")
for pat in [("search", "product"), ("search", "product", "cart"),
("search", "product", "cart", "buy"),
("search", "product", "cart", "buy", "support")]:
print(f" {' -> '.join(pat):48s} {support(pat, seqs):.4f}")
print(" Support can only fall as the pattern lengthens, so a prefix that fails")
print(" the threshold ends that whole branch of the search.")
# --- GAP CONSTRAINTS: 'eventually' is often too weak to be useful. ---
def contains_maxgap(seq, pat, max_gap):
"""Require consecutive pattern elements to be within max_gap positions."""
pos = -1
for p in pat:
nxt = None
for i in range(pos + 1, min(len(seq), pos + 1 + max_gap) if pos >= 0
else len(seq)):
if seq[i] == p:
nxt = i
break
if nxt is None:
return False
pos = nxt
return True
print("\nGAP CONSTRAINTS TIGHTEN 'EVENTUALLY' INTO 'SOON AFTER'")
print(f" {'max_gap':>8s} {'search->buy':>12s} {'search->product->cart':>22s}")
for g in [1, 2, 3, 10, 999]:
a = np.mean([contains_maxgap(s, ("search", "buy"), g) for s in seqs])
b = np.mean([contains_maxgap(s, ("search", "product", "cart"), g)
for s in seqs])
print(f" {g if g < 999 else 'none':>8} {a:12.3f} {b:22.3f}")
print(" Unconstrained support counts a purchase three weeks later as following")
print(" a search. A max_gap turns the pattern into a claim about proximity,")
print(" which is usually what the business question means.")
# --- FROM PATTERNS TO RULES: sequential confidence. ---
print("\nSEQUENTIAL RULES: P(suffix eventually follows | prefix occurred)")
print(f" {'rule':46s} {'supp':>6s} {'conf':>6s}")
for pre, post in [(("search",), ("buy",)),
(("search", "product"), ("cart",)),
(("search", "product", "cart"), ("buy",)),
(("support",), ("logout",))]:
sp = support(pre + post, seqs)
sa = support(pre, seqs)
rule = ' -> '.join(pre) + ' => ' + ' -> '.join(post)
print(f" {rule:46s} {sp:6.3f} {sp / sa:6.3f}")
print(" This is the funnel, read directly: 66% of searchers who view a product")
print(" reach the cart, and 57% of those buy (planted: 65% and 55%). Sequential")
print(" confidence gives you")
print(" the conversion rate at each step, which set-based mining cannot.")
SETS VERSUS SEQUENCES
as itemsets these two users are IDENTICAL:
user A: login, search, cart, buy
user B: buy, cart, search, login
as sequences they are opposites.
k items form 1 itemset but k! orderings:
k= 3: 1 itemset, 6 orderings
k= 5: 1 itemset, 120 orderings
k=10: 1 itemset, 3,628,800 orderings
SUBSEQUENCE SUPPORT ALLOWS GAPS
sequence: home -> search -> support -> product -> cart -> buy
pattern: search -----------> product -> cart
matches, with 'support' skipped over
pattern support
search 0.519
search -> product 0.442
search -> product -> cart 0.294
search -> product -> cart -> buy 0.167
cart -> search 0.043 <- noise artefact
support -> logout 0.203
logout -> support 0.026
PREFIXSPAN: GROW A PREFIX, PROJECT THE DATABASE
prefix () full database
|
+-- prefix (search) projected: suffixes after the first 'search'
|
+-- (search, product) projected again, smaller still
|
+-- (search, product, cart)
each recursion works on a SMALLER database and never rescans the
original. no candidate generation at all.
frequent sequential patterns found: 94
candidate extensions evaluated: 658
max |PrefixSpan support - brute-force support| = 0.00e+00
DOWNWARD CLOSURE HOLDS FOR SEQUENCES TOO
search -> product 0.4420
search -> product -> cart 0.2937
search -> product -> cart -> buy 0.1667
search -> product -> cart -> buy -> support 0.0360
GAP CONSTRAINTS: 'EVENTUALLY' VS 'SOON AFTER'
max_gap search->buy search->product->cart
1 0.012 0.195
2 0.024 0.269
3 0.116 0.289
10 0.185 0.294
none 0.185 0.294
unconstrained support counts a purchase three weeks later as
"following" the search. max_gap makes it a claim about proximity.
SEQUENTIAL RULES ARE CONVERSION RATES
rule supp conf
search => buy 0.185 0.356
search -> product => cart 0.294 0.664
search -> product -> cart => buy 0.167 0.568
support => logout 0.203 0.679
the funnel, read straight off the logs:
search --66%--> cart --57%--> buy (planted: 65% and 55%)
| Itemset mining | Sequence mining | |
|---|---|---|
| Data unit | An unordered set | An ordered list |
{A,B} vs {B,A} | Identical | Different patterns |
| Search algorithm | Apriori, FP-Growth | PrefixSpan, SPADE, GSP |
| Pruning principle | Downward closure | Downward closure |
| Extra constraints | Item hierarchies | Gaps, windows, hierarchies |
| Typical use | Basket analysis, bundling | Funnels, churn paths, event logs |
Key Takeaways
- Sequences preserve order, which is the information a funnel analysis needs and itemset mining discards.
- Sequence support counts a pattern as present if it appears as a subsequence, so intervening events are allowed by default.
- PrefixSpan grows prefixes and carries a projected database of suffixes, so each recursion works on less data and never rescans the original.
- The same downward-closure pruning applies: support cannot rise as a pattern lengthens.
- Gap constraints convert "eventually" into "soon after", which usually changes the support dramatically and is usually what was meant.
- Sequential rule confidence is a conversion rate, which is why this is the right tool for funnels.
- Background noise manufactures spurious orderings, so a support floor is as important here as for itemsets.
- Real logs must be sessionised first, and the timeout you pick changes every downstream number.
🧪 Practice
- Write a function testing whether one sequence is a subsequence of another, and use it to compute support for several patterns.
- Show that
A -> BandB -> Ahave different support in your data, then find a pair where they are nearly equal and explain why.- Implement PrefixSpan with projected databases and verify its supports against brute-force counting.
- Add a
max_gapconstraint and report how the support of your longest pattern changes for gaps of 1, 2, 5, and unlimited.- Compute sequential rule confidence at each stage of a funnel and compare with the conversion rates you planted in the generator.
- Sessionise a single long event stream with 15-minute and 2-hour inactivity timeouts and report how the top patterns differ.
- Interview: Your sequence mining on web logs reports
checkout -> homeas a high-support pattern. Give two explanations and say how you would tell them apart. (Hint: one is a genuine behaviour after a purchase, and one is about how you decided where a session ends.)
Topic Modeling
You have 50,000 documents and no labels, and you want to know what they are about. Clustering would assign each document to one group, which is the wrong shape for text: an article about a football club's finances is genuinely both sport and finance. Topic modelling produces mixtures instead -- each document is a distribution over topics, and each topic a distribution over words.
Latent Dirichlet Allocation is defined by its generative story, and reading the story tells you exactly what the model assumes:
for each document d:
draw a topic mixture theta_d ~ Dirichlet(alpha)
for each word position:
draw a topic z ~ Categorical(theta_d)
draw a word w ~ Categorical(beta_z)
Fitting inverts it: given only the words, recover theta (document-topic) and
beta (topic-word). The Dirichlet prior on theta is what encodes "documents
usually concentrate on a few topics" -- a small alpha makes mixtures
sparser.
Two assumptions come with the story and both matter. Words are exchangeable -- it is a bag of words and order is discarded, so LDA cannot distinguish "man bites dog" from "dog bites man". And every document mixes all topics, with weights that may be near zero but never structurally absent.
NMF reaches a similar output from a completely different principle. It is
a matrix factorisation: approximate the document-term matrix X as W @ H
with every entry non-negative. W is document-topic, H is topic-word. There
is no generative story and no probability -- just a constrained least-squares
fit, typically on tf-idf rather than raw counts.
Why non-negativity matters is best seen by removing it. Truncated SVD performs the same factorisation without the constraint, and the demonstration shows 33 of 43 word weights coming out negative. A component that subtracts a word is fine for compression and meaningless as a topic: you cannot read "this document is minus 0.3 of finance". Non-negativity is what makes factors additive parts, and therefore interpretable. Both LDA and NMF recovered all four planted topics at 100% top-word overlap; SVD managed 90% with unreadable signed loadings.
| LDA | NMF | Truncated SVD | |
|---|---|---|---|
| Foundation | Probabilistic generative model | Constrained matrix factorisation | Unconstrained factorisation |
| Input | Raw counts | tf-idf usually | tf-idf usually |
| Output sums to 1 | Yes, both directions | No | No |
| Factors interpretable | Yes | Yes | No, signed |
| Deterministic | No | With fixed init | Yes |
| Speed | Slower | Fast | Fastest |
| Handles new documents | Yes | Yes | Yes |
Choosing the number of topics is the same problem as choosing k in
clustering, with the same partial answers. Held-out perplexity is
available for LDA, and topic coherence -- do a topic's top words actually
co-occur in real documents? -- tracks human judgement of interpretability much
better.
Be careful how you read the demonstration here. Both perplexity and coherence
pick the true k = 4, and coherence tracks the recovery column more closely.
That is a synthetic corpus with four clean, well-separated vocabularies. On
real text, held-out perplexity is known to keep improving as k grows and to
disagree with human judgement of which topics are interpretable -- which is
exactly why coherence measures were invented. Use coherence, then read the
topics: the final judge of a topic model is whether a domain expert
recognises the topics as things.
Finally, do not throw away the document-topic matrix. In the demonstration 21.9% of documents have no topic holding even half their weight. Taking the argmax turns those into confident single labels they do not deserve. If you need hard labels, take the argmax and accept the loss knowingly; if you want to know which documents are genuinely mixed, the matrix already tells you.
import numpy as np
import pandas as pd
from sklearn.decomposition import LatentDirichletAllocation, NMF, TruncatedSVD
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
rng = np.random.default_rng(179)
# --- A SYNTHETIC CORPUS WITH KNOWN TOPICS, SO THE ANSWER IS CHECKABLE. ---
TOPICS = {
"sport": ("match team goal player season league coach score stadium"
" fans").split(),
"finance": ("market shares profit revenue bank investor trading bonds"
" yield fund").split(),
"cooking": ("recipe flavour bake oven sauce ingredient chop simmer dough"
" spice").split(),
"politics": ("election vote policy minister party debate reform senate"
" bill campaign").split(),
}
FILLER = "the a of and in to on with for said this that from".split()
names = list(TOPICS)
docs, true_mix = [], []
for _ in range(1200):
# Each document is a MIXTURE of topics -- that is the defining assumption.
w = rng.dirichlet(np.full(len(names), 0.4))
true_mix.append(w)
words = []
for _ in range(int(rng.integers(60, 120))):
if rng.random() < 0.35:
words.append(str(rng.choice(FILLER)))
else:
t = names[int(rng.choice(len(names), p=w))]
words.append(str(rng.choice(TOPICS[t])))
docs.append(" ".join(words))
true_mix = np.array(true_mix)
print(f"{len(docs)} documents, {len(names)} true topics, each doc a MIXTURE")
print(f" example topic weights: {np.round(true_mix[0], 3)} ({names})")
# --- LDA'S GENERATIVE STORY, WHICH IS WHAT MAKES IT A TOPIC MODEL. ---
print("\nTHE LDA GENERATIVE STORY")
print(" for each document d:")
print(" draw a topic mixture theta_d ~ Dirichlet(alpha)")
print(" for each word position:")
print(" draw a topic z ~ Categorical(theta_d)")
print(" draw a word w ~ Categorical(beta_z)")
print(" Fitting inverts this: given the words, recover theta (doc-topic) and")
print(" beta (topic-word). Note what it assumes -- words are exchangeable")
print(" (a BAG of words, order discarded) and every document mixes all topics.")
cv = CountVectorizer(max_df=0.9, min_df=5)
X = cv.fit_transform(docs)
vocab = np.array(cv.get_feature_names_out())
print(f"\ndocument-term matrix: {X.shape[0]} x {X.shape[1]},"
f" {X.nnz / np.prod(X.shape):.1%} dense")
lda = LatentDirichletAllocation(n_components=4, random_state=0,
learning_method="batch", max_iter=40).fit(X)
def top_words(model, vocab, k=8):
return [vocab[np.argsort(c)[::-1][:k]] for c in model.components_]
print("\nLDA TOPICS (top words by weight)")
for i, w in enumerate(top_words(lda, vocab)):
print(f" topic {i}: {' '.join(w)}")
# Score the recovery objectively: match each fitted topic to the true topic whose
# word list it overlaps most, then report the overlap.
def topic_recovery(model, vocab, k=10):
rows = []
for i, c in enumerate(model.components_):
top = set(vocab[np.argsort(c)[::-1][:k]])
best = max(names, key=lambda t: len(top & set(TOPICS[t])))
rows.append((i, best, len(top & set(TOPICS[best])) / k))
return rows
print("\n TOPIC RECOVERY (overlap of the top 10 words with the true word list)")
for i, best, frac in topic_recovery(lda, vocab):
print(f" fitted topic {i} -> '{best}' overlap {frac:.0%}")
# --- NMF: THE SAME OUTPUT SHAPE FROM A COMPLETELY DIFFERENT PRINCIPLE. ---
# LDA is a probabilistic generative model fitted by variational inference.
# NMF is a matrix factorisation: X ~ W @ H with everything non-negative.
tf = TfidfVectorizer(max_df=0.9, min_df=5)
Xt = tf.fit_transform(docs)
vt = np.array(tf.get_feature_names_out())
nmf = NMF(n_components=4, init="nndsvd", random_state=0, max_iter=600).fit(Xt)
svd = TruncatedSVD(n_components=4, random_state=0).fit(Xt)
print("\nNMF TOPICS (on tf-idf)")
for i, w in enumerate(top_words(nmf, vt)):
print(f" topic {i}: {' '.join(w)}")
print("\nTRUNCATED SVD 'TOPICS' (same factorisation WITHOUT non-negativity)")
for i, c in enumerate(svd.components_):
top = vt[np.argsort(c)[::-1][:8]]
nneg = int((c < 0).sum())
print(f" comp {i}: {' '.join(top)} ({nneg} of {len(c)} weights negative)")
print(" SVD components contain negative weights, so a 'topic' can subtract a")
print(" word. That is fine for compression and meaningless as a topic: you")
print(" cannot read 'this document is minus 0.3 of finance'. Non-negativity is")
print(" what makes NMF's factors additive PARTS, and therefore interpretable.")
print("\n RECOVERY COMPARISON (mean top-10 overlap with a true topic)")
for label, m, v in [("LDA (counts)", lda, vocab), ("NMF (tf-idf)", nmf, vt),
("TruncatedSVD (tf-idf)", svd, vt)]:
print(f" {label:24s} {np.mean([f for _, _, f in topic_recovery(m, v)]):.0%}")
# --- CHOOSING THE NUMBER OF TOPICS. ---
# Perplexity is available for LDA but notoriously disagrees with human judgement.
# Topic COHERENCE -- do a topic's top words co-occur in real documents? -- tracks
# interpretability far better. Implement a simple UMass coherence.
Xb = (X > 0).astype(int).toarray()
doc_freq = Xb.sum(0)
co_occ = Xb.T @ Xb
def umass_coherence(model, vocab_idx, k=10):
"""Sum over word pairs of log((co-occurrence + 1) / doc-frequency)."""
scores = []
for c in model.components_:
top = np.argsort(c)[::-1][:k]
s = 0.0
for a in range(1, len(top)):
for b in range(a):
s += np.log((co_occ[top[a], top[b]] + 1) / max(doc_freq[top[b]], 1))
scores.append(s)
return float(np.mean(scores))
print("\nCHOOSING THE NUMBER OF TOPICS (true answer is 4)")
print(f" {'k':>3s} {'LDA perplexity':>16s} {'LDA coherence':>14s}"
f" {'mean recovery':>14s}")
for k in [2, 3, 4, 5, 6, 8]:
m = LatentDirichletAllocation(n_components=k, random_state=0,
learning_method="batch", max_iter=40).fit(X)
rec = np.mean([f for _, _, f in topic_recovery(m, vocab)])
print(f" {k:3d} {m.perplexity(X):16.1f} {umass_coherence(m, vocab):14.2f}"
f" {rec:14.0%}")
print(" Both criteria pick k=4 here, and coherence tracks the recovery column")
print(" more closely. Do not over-read that: this is a SYNTHETIC corpus with")
print(" four clean, well-separated vocabularies. On real text, held-out")
print(" perplexity is known to keep improving as k grows and to disagree with")
print(" human judgement of which topics are interpretable -- which is exactly")
print(" why coherence measures were invented. Use coherence, then READ the")
print(" topics: the final judge of a topic model is whether a domain expert")
print(" recognises the topics as things.")
# --- THE DOCUMENT-TOPIC MATRIX IS THE OTHER HALF OF THE OUTPUT. ---
theta = lda.transform(X)
print("\nDOC-TOPIC WEIGHTS: EVERY DOCUMENT IS A MIXTURE, NOT A LABEL")
print(f" {'doc':>4s} {'fitted mixture':>34s} {'max weight':>11s}")
for d in range(4):
print(f" {d:4d} {np.array2string(theta[d], precision=3, floatmode='fixed'):>34s}"
f" {theta[d].max():11.3f}")
print(f" documents whose top topic holds < 50% of the weight:"
f" {np.mean(theta.max(1) < 0.5):.1%}")
print(" Assigning each document its argmax topic discards this. If you need hard")
print(" labels, take the argmax and accept the loss; if you need to know which")
print(" documents are genuinely mixed, the matrix already tells you.")
THE LDA GENERATIVE STORY
alpha -> theta_d -> z -> w <- beta_k
(doc's (a (a word) (topic's
topic mix) topic) word dist)
for each document: draw a topic mixture
for each word: draw a topic, then draw a word from that topic
fitting inverts this to recover theta (doc x topic) and beta (topic x word)
RECOVERED TOPICS (1200 docs, 4 planted topics)
LDA (on counts)
topic 0: yield bonds market trading fund profit shares investor
topic 1: reform bill senate debate minister campaign policy vote
topic 2: goal fans score coach season match player league
topic 3: sauce dough bake oven recipe ingredient flavour chop
recovery: every fitted topic matched a true topic at 100% overlap
NMF (on tf-idf)
topic 0: sauce dough recipe oven ingredient chop flavour bake
topic 1: season fans match goal stadium coach team player
topic 2: campaign reform senate minister bill debate election vote
topic 3: yield bonds market fund profit shares trading bank
WHY NON-NEGATIVITY MATTERS: REMOVE IT AND READ THE RESULT
truncated SVD "topics":
comp 1: season match fans stadium goal coach team player
(33 of 43 weights NEGATIVE)
comp 2: campaign senate reform minister bill debate election vote
(33 of 43 weights NEGATIVE)
a component that SUBTRACTS a word is fine for compression and
unreadable as a topic. you cannot act on "minus 0.3 of finance".
mean top-10 overlap with a true topic:
LDA (counts) 100%
NMF (tf-idf) 100%
TruncatedSVD (tf-idf) 90%
CHOOSING THE NUMBER OF TOPICS (true answer 4)
k LDA perplexity LDA coherence mean recovery
2 36.2 -14.80 90%
3 31.9 -13.64 93%
4 27.9 -10.93 100% <- both agree
5 28.4 -11.71 96%
6 28.8 -14.74 88%
8 29.3 -14.78 84%
Both criteria find k=4 HERE, on a synthetic corpus with four clean
vocabularies. On real text perplexity keeps improving with k and
disagrees with human judgement -- which is why coherence exists.
THE DOCUMENT-TOPIC MATRIX IS HALF THE OUTPUT
doc fitted mixture max weight
0 [0.129 0.039 0.810 0.022] 0.810
1 [0.022 0.784 0.004 0.189] 0.784
2 [0.079 0.446 0.421 0.054] 0.446 <- genuinely mixed
3 [0.004 0.642 0.095 0.259] 0.642
21.9% of documents have no topic holding even half their weight.
An argmax label turns those into a confidence they do not have.
Key Takeaways
- Topic models output mixtures rather than assignments, which is the right shape for documents that are genuinely about more than one thing.
- LDA is defined by a generative story, and the story states its assumptions: exchangeable words and every document mixing all topics.
- The Dirichlet prior controls how concentrated document mixtures are.
- NMF reaches a similar output by constrained matrix factorisation with no probabilistic story, usually on tf-idf.
- Non-negativity is what makes factors readable as additive parts; unconstrained SVD yields signed components that cannot be interpreted as topics.
- Perplexity and coherence can agree on clean synthetic corpora, but on real text perplexity keeps improving with more topics and diverges from interpretability.
- Coherence is the better automatic criterion, and reading the topics is the final check.
- The document-topic matrix identifies genuinely mixed documents, information the argmax discards.
🧪 Practice
- Fit LDA to a corpus and print the top 10 words per topic, then name each topic yourself and see whether the names are obvious.
- Fit NMF and truncated SVD on the same tf-idf matrix and count how many weights in each component are negative.
- Compute a UMass coherence score for
kfrom 2 to 12 and compare its peak with held-out perplexity's behaviour over the same range.- Report what fraction of documents have a maximum topic weight below 0.5, and inspect three of them.
- Vary LDA's
doc_topic_priorover 0.01, 0.1, and 1.0 and describe how sparse the document mixtures become.- Shuffle the words within every document and refit. Explain why the topics barely change and what that tells you about the model's assumptions.
- Interview: Your 20-topic LDA model produces three topics that all look like generic filler words. What is happening and what would you do? (Hint: what did you do about stop words and very frequent terms, and what does
max_dfcontrol?)
Matrix and Tensor Factorization
A recommender's data is a users-by-items matrix that is almost entirely empty
-- 8% observed in the demonstration, and far less in reality. Factorisation
says: approximate that matrix as the product of two thin matrices, R ~ U @ V', where each user gets a short latent factor vector and each item gets
one, and a predicted rating is their dot product.
The compression is the model. A full 400-by-120 matrix needs 48,000 numbers; a rank-3 factorisation needs 1,560, or 3.2% as many. That constraint forces users who behave alike to share parameters, which is exactly how a prediction for a pair you have never observed becomes possible at all.
Plain SVD does not work here, and the reason is worth being precise about. SVD requires a complete matrix. The usual workaround -- fill the gaps with zeros or the column mean -- changes the problem: you then fit the values you invented as though they were data, and with 92% of cells invented they dominate the loss. The demonstration measures the cost on identical held-out entries:
- predict the global mean: RMSE 0.4720
- mean-impute, then truncated SVD: 0.4536 (barely better than the mean)
- ALS on observed entries only: 0.3770
Alternating least squares avoids the problem by summing the loss over
observed entries only. Hold V fixed and each user's factor vector has a
closed-form ridge solution; hold U fixed and each item's does. Alternate.
Missing cells are never touched, which is why ALS (and SGD-based equivalents)
are standard for explicit feedback.
Rank and regularisation interact, and the honest result is instructive. In
the demonstration the true rank is 3, and held-out error is best at rank 1
and worsens from there. With 8% observed, each user has roughly 10 ratings,
which cannot support three free parameters per user. The best model is
simpler than the truth -- a bias-variance result, not a bug. And the best
lambda grows as the matrix empties: at 20% observed lambda=1.0 is best,
and at 3% observed the prior is doing most of the work. Tune rank and lambda
together; do not assume the true rank is recoverable from sparse data.
NMF applies the same interpretability argument as in the previous topic. With non-negative factors, a user's taste vector reads as "this much genre 1, this much genre 2". With signed factors -- 43% of SVD's component entries are negative -- it reads as nothing.
Two extensions worth knowing by name: implicit feedback (clicks, plays, views) has no negatives, only absences, so the standard approach weights all unobserved cells as weak negatives rather than treating them as missing. And bias terms -- a global mean plus per-user and per-item offsets -- typically capture more variance than the factors themselves and should be in any production model.
Tensors generalise this to data with more than two indices. Users by items by time is a three-way tensor, and flattening it into a matrix destroys the third index. Two decompositions:
- CP / PARAFAC: a sum of
Rrank-one terms, giving one factor matrix per mode. Rank 3 on the demonstration's tensor reaches relative error 0.0207 and stops improving past rank 3 -- which is how you find the rank. - Tucker: a small dense core tensor plus one factor matrix per mode; more flexible, less uniquely determined.
Note the demonstration's honest comparison: flattening to a matrix and running SVD reaches a similar error (0.0202). The reason to use a tensor method is not reconstruction accuracy -- it is that CP returns a 12-by-3 factor matrix over months, so you can say "this factor peaks in December". The flattened version returns components over 480 item-month pairs, and the time index no longer exists as a thing you can talk about.
import numpy as np
import pandas as pd
import tensorly as tl
from tensorly.decomposition import parafac, tucker
from sklearn.decomposition import NMF, TruncatedSVD
rng = np.random.default_rng(181)
# --- THE RECOMMENDER SETUP: A MATRIX THAT IS MOSTLY MISSING. ---
# 400 users, 120 items, 3 latent "tastes". Each user has a taste vector, each item
# a genre vector, and the rating is their dot product plus noise.
nu, ni, k = 400, 120, 3
U_true = np.abs(rng.normal(0, 1, (nu, k)))
V_true = np.abs(rng.normal(0, 1, (ni, k)))
full = U_true @ V_true.T
full = 1 + 4 * (full - full.min()) / (full.max() - full.min()) # scale to 1-5
observed = rng.random((nu, ni)) < 0.08 # 8% observed
R = np.where(observed, full + rng.normal(0, 0.25, (nu, ni)), np.nan)
print(f"{nu} users x {ni} items, {observed.mean():.1%} of entries observed")
print(f" a full matrix would need {nu * ni:,} numbers;")
print(f" a rank-{k} factorisation needs {k * (nu + ni):,}"
f" ({k * (nu + ni) / (nu * ni):.1%} as many)")
print(" That compression IS the model: it forces users who behave alike to share")
print(" parameters, which is how a prediction for an unseen pair becomes possible.")
# --- WHY PLAIN SVD DOES NOT WORK HERE. ---
print("\nWHY YOU CANNOT JUST RUN SVD")
print(f" SVD needs a complete matrix, and {np.isnan(R).mean():.1%} of entries are")
print(" missing. The usual workaround is to fill the gaps with zeros or the mean,")
print(" and both CHANGE THE PROBLEM: you then fit the imputed values as if they")
print(" were data. Compare the two approaches on the same held-out entries.")
rows, cols = np.where(observed)
vals = R[rows, cols]
perm = rng.permutation(len(vals))
ntr = int(0.8 * len(vals))
tr, te = perm[:ntr], perm[ntr:]
# Approach 1: mean-impute, then SVD.
Rfill = np.where(observed, R, np.nan)
Rfill[rows[te], cols[te]] = np.nan # hide the test entries too
mu = np.nanmean(Rfill)
Rfill_z = np.where(np.isnan(Rfill), mu, Rfill)
svd = TruncatedSVD(n_components=k, random_state=0)
Z = svd.fit_transform(Rfill_z)
approx = Z @ svd.components_
rmse_svd = float(np.sqrt(np.mean((vals[te] - approx[rows[te], cols[te]]) ** 2)))
# Approach 2: ALS on the OBSERVED entries only. The loss sums over what exists.
def als(rows, cols, vals, nu, ni, k, lam=1.0, iters=30):
"""Alternating least squares: hold V fixed and solve for each user's row in
closed form, then swap. Only observed entries enter either solve."""
U = rng.normal(0, 0.1, (nu, k))
V = rng.normal(0, 0.1, (ni, k))
by_user, by_item = {}, {}
for idx, (r, c) in enumerate(zip(rows, cols)):
by_user.setdefault(r, []).append(idx)
by_item.setdefault(c, []).append(idx)
for _ in range(iters):
for u, ids in by_user.items():
Vi = V[cols[ids]]
A = Vi.T @ Vi + lam * np.eye(k) # ridge keeps it invertible
U[u] = np.linalg.solve(A, Vi.T @ vals[ids])
for i, ids in by_item.items():
Uu = U[rows[ids]]
A = Uu.T @ Uu + lam * np.eye(k)
V[i] = np.linalg.solve(A, Uu.T @ vals[ids])
return U, V
Uh, Vh = als(rows[tr], cols[tr], vals[tr], nu, ni, k)
pred = np.sum(Uh[rows[te]] * Vh[cols[te]], axis=1)
rmse_als = float(np.sqrt(np.mean((vals[te] - pred) ** 2)))
baseline = float(np.sqrt(np.mean((vals[te] - vals[tr].mean()) ** 2)))
print(f"\n {'method':38s} {'held-out RMSE':>14s}")
print(f" {'predict the global mean':38s} {baseline:14.4f}")
print(f" {'mean-impute then truncated SVD':38s} {rmse_svd:14.4f}")
print(f" {'ALS on observed entries only':38s} {rmse_als:14.4f}")
print(" Imputing first drags every prediction toward the fill value, because the")
print(" 92% of entries you invented dominate the loss. ALS never looks at a")
print(" missing cell, which is why it is the standard for explicit feedback.")
# --- RANK IS THE CAPACITY DIAL. ---
print("\nRANK CONTROLS CAPACITY (the true rank is 3)")
print(f" {'rank':>5s} {'train RMSE':>11s} {'held-out RMSE':>14s} {'params':>9s}")
for kk in [1, 2, 3, 5, 10, 25]:
Ua, Va = als(rows[tr], cols[tr], vals[tr], nu, ni, kk)
ptr = np.sum(Ua[rows[tr]] * Va[cols[tr]], axis=1)
pte = np.sum(Ua[rows[te]] * Va[cols[te]], axis=1)
print(f" {kk:5d} {np.sqrt(np.mean((vals[tr] - ptr) ** 2)):11.4f}"
f" {np.sqrt(np.mean((vals[te] - pte) ** 2)):14.4f} {kk * (nu + ni):9,d}")
print(" Training error falls with rank, as always. But read the held-out column:")
print(" it is BEST AT RANK 1, below the true rank of 3, and gets worse from")
print(" there. With 8% of entries observed each user has about 10 ratings, which")
print(" cannot support 3 free parameters per user. The best model here is")
print(" SIMPLER than the truth -- a bias-variance result, not a bug. Rank and")
print(" lambda interact, so tune them together rather than assuming the true")
print(" rank is recoverable from sparse data.")
# --- REGULARISATION MATTERS MORE AS THE MATRIX GETS SPARSER. ---
print("\nREGULARISATION VS SPARSITY (rank 5)")
print(f" {'lambda':>8s} " + " ".join(f"{f'{d:.0%} obs':>9s}"
for d in [0.20, 0.08, 0.03]))
for lam in [0.001, 0.1, 1.0, 10.0]:
cells = []
for dens in [0.20, 0.08, 0.03]:
obs = rng.random((nu, ni)) < dens
rr, cc = np.where(obs)
vv = full[rr, cc] + rng.normal(0, 0.25, len(rr))
pm = rng.permutation(len(vv))
a, b = pm[:int(0.8 * len(vv))], pm[int(0.8 * len(vv)):]
Ua, Va = als(rr[a], cc[a], vv[a], nu, ni, 5, lam=lam)
pte = np.sum(Ua[rr[b]] * Va[cc[b]], axis=1)
cells.append(f"{np.sqrt(np.mean((vv[b] - pte) ** 2)):9.4f}")
print(f" {lam:8.3f} " + " ".join(cells))
print(" The best lambda grows as the matrix empties. At 3% observed, each user")
print(" has only a handful of ratings, so the prior does most of the work.")
# --- NMF GIVES THE SAME SHAPE WITH AN INTERPRETABILITY CONSTRAINT. ---
print("\nNMF: NON-NEGATIVE FACTORS ARE ADDITIVE PARTS")
Rz = np.where(observed, R, 0.0)
nmf = NMF(n_components=k, init="nndsvd", random_state=0, max_iter=800).fit(Rz)
W, H = nmf.transform(Rz), nmf.components_
print(f" all factor entries non-negative: W {np.all(W >= 0)} H {np.all(H >= 0)}")
sv = TruncatedSVD(k, random_state=0).fit(Rz)
print(f" SVD component entries below zero: {(sv.components_ < 0).mean():.1%}")
print(" A user's taste vector as non-negative weights reads as 'this much of")
print(" genre 1, this much of genre 2'. With signed factors it does not read as")
print(" anything, which is the same argument as in the topic-modelling topic.")
# --- TENSORS: WHEN THE DATA HAS MORE THAN TWO INDICES. ---
# users x items x TIME. Flattening to a matrix throws the third index away.
nt = 12
A = np.abs(rng.normal(0, 1, (60, 3)))
B = np.abs(rng.normal(0, 1, (40, 3)))
C = np.abs(rng.normal(0, 1, (nt, 3)))
# CP structure: T[i,j,t] = sum_r A[i,r] B[j,r] C[t,r]
T = np.einsum("ir,jr,tr->ijt", A, B, C) + rng.normal(0, 0.05, (60, 40, nt))
print(f"\nA 3-WAY TENSOR: {T.shape} (users x items x months)")
print(f" dense storage: {T.size:,} numbers")
print(f" CP rank-3: {3 * sum(T.shape):,} numbers"
f" ({3 * sum(T.shape) / T.size:.1%})")
print(f" Tucker (3,3,3): {3 ** 3 + 3 * sum(T.shape):,} numbers")
print(f"\n {'method':34s} {'rel. reconstruction error':>26s}")
for kk in [1, 2, 3, 5]:
cp = parafac(tl.tensor(T), rank=kk, n_iter_max=300, random_state=0)
err = float(tl.norm(tl.cp_to_tensor(cp) - T) / tl.norm(T))
print(f" {f'CP / PARAFAC, rank {kk}':34s} {err:26.4f}")
tk = tucker(tl.tensor(T), rank=[3, 3, 3], random_state=0)
print(f" {'Tucker, ranks (3,3,3)':34s}"
f" {float(tl.norm(tl.tucker_to_tensor(tk) - T) / tl.norm(T)):26.4f}")
# What flattening costs: unfold to (users x items*time) and factor that.
flat = T.reshape(T.shape[0], -1)
sv2 = TruncatedSVD(3, random_state=0).fit(flat)
rec = sv2.transform(flat) @ sv2.components_
print(f" {'flatten to a matrix, SVD rank 3':34s}"
f" {float(np.linalg.norm(rec - flat) / np.linalg.norm(flat)):26.4f}")
print(" CP recovers the true rank-3 structure almost exactly, and its error")
print(" stops improving past rank 3 -- which is how you find the rank. Flattening")
print(" reaches a similar error here but returns components over the 480 flattened")
print(" item-month PAIRS, so you can no longer say 'this factor is strong in")
print(" December': the time index has been destroyed. CP keeps one factor matrix")
print(" per mode, which is the entire reason to use a tensor method.")
cp3 = parafac(tl.tensor(T), rank=3, n_iter_max=300, random_state=0)
print(f"\n CP rank-3 factor shapes: {[f.shape for f in cp3.factors]}")
print(" one factor matrix per mode: 60 users x 3, 40 items x 3, 12 months x 3.")
print(" The month factors are directly readable as seasonal profiles.")
THE FACTORISATION
items items
+---------+ +---+ +---------+
u | 4 ? ? 2 | u | | | | 3
s | ? 3 ? ? | ~ s | U | | V' |
e | ? ? 5 ? | e | | +---------+
r | 1 ? ? ? | r | | 3
s +---------+ s +---+
48,000 cells (92% missing) -> 1,560 parameters (3.2%)
prediction for (u, i) = dot product of their factor vectors
WHY YOU CANNOT JUST IMPUTE AND RUN SVD
method held-out RMSE
predict the global mean 0.4720
mean-impute then truncated SVD 0.4536 <- barely better
ALS on observed entries only 0.3770
imputing invents 92% of the matrix, and the invented values then
dominate the loss. ALS never looks at a missing cell.
RANK AND SPARSITY: THE BEST MODEL IS SIMPLER THAN THE TRUTH
rank train RMSE held-out RMSE params
1 0.2820 0.3411 520 <- BEST held-out
2 0.2504 0.3567 1,040
3 0.2219 0.3623 1,560 <- the TRUE rank
5 0.1897 0.3826 2,600
10 0.1620 0.3777 5,200
25 0.1559 0.3734 13,000
with ~10 ratings per user, 3 free parameters per user is too many.
THE BEST LAMBDA GROWS AS THE MATRIX EMPTIES (rank 5)
lambda 20% obs 8% obs 3% obs
0.001 0.3775 1.7923 2.1954
0.100 0.3546 0.6100 1.5984
1.000 0.2950 0.3808 0.6731 <- best in all three
10.000 0.4447 0.8286 1.5825
NON-NEGATIVITY, AGAIN
NMF factors non-negative: W True H True
SVD component entries below 0: 43.1%
non-negative: "0.8 of genre A + 0.3 of genre B" readable
signed: "0.8 of A minus 0.5 of B" not readable
TENSORS: WHEN THE DATA HAS THREE INDICES
users x items x months, shape (60, 40, 12)
dense storage 28,800 numbers
CP rank-3 336 numbers (1.2%)
Tucker (3,3,3) 363 numbers
CP: a sum of R rank-one outer products
T ~ a1 o b1 o c1 + a2 o b2 o c2 + a3 o b3 o c3
\___/ one factor per MODE, per term
method rel. reconstruction error
CP / PARAFAC, rank 1 0.2779
CP / PARAFAC, rank 2 0.1506
CP / PARAFAC, rank 3 0.0207 <- the cliff
CP / PARAFAC, rank 5 0.0206
Tucker, ranks (3,3,3) 0.0207
flatten to a matrix, SVD rank 3 0.0202
Flattening reaches the SAME error. The difference is the output:
CP rank-3 factor shapes: [(60, 3), (40, 3), (12, 3)]
-> a 12 x 3 matrix over MONTHS, readable as seasonal profiles
flattened SVD: components over 480 item-month PAIRS
-> "this factor peaks in December" is no longer expressible
Key Takeaways
- Factorisation approximates a sparse matrix as a product of thin factor matrices, and that compression is what makes predictions for unobserved pairs possible.
- SVD needs a complete matrix, and imputing the gaps means fitting invented values that dominate the loss when most cells are missing.
- ALS sums the loss over observed entries only, solving a closed-form ridge problem for each user and item in turn.
- Under heavy sparsity the best held-out rank can be lower than the true rank, because there are not enough ratings per user to support the parameters.
- The optimal regularisation strength grows as the matrix gets sparser.
- Non-negative factors read as additive parts; signed factors do not read as anything.
- Bias terms usually capture more variance than the latent factors and belong in any production model.
- CP decomposition keeps one factor matrix per tensor mode, which is the real reason to use it: flattening can match the reconstruction error but destroys the ability to talk about each index separately.
🧪 Practice
- Build a sparse ratings matrix from a known low-rank product and compare mean-imputed SVD against ALS on the same held-out entries.
- Implement ALS with ridge regularisation from the closed-form updates and verify that the training loss decreases every sweep.
- Sweep rank from 1 to 30 and plot training and held-out error, identifying where held-out error turns.
- Sweep
lambdaat 20%, 5%, and 2% observed density and report the best value at each.- Add global, per-user, and per-item bias terms and report how much of the total improvement over the mean baseline comes from biases alone.
- Fit NMF and truncated SVD to the same matrix and count negative entries in each factor.
- Build a users-by-items-by-months tensor, decompose it with CP at several ranks, and find where the error stops improving.
- Interview: Your recommender's held-out RMSE barely beats predicting each item's mean rating. Name three things you would check before adding more latent factors. (Hint: think about biases, regularisation, and whether the matrix is dense enough to support the rank you chose.)
<a id="114-anomaly-detection"></a>
11.4 Anomaly Detection
Anomaly detection asks which points do not belong, and it inherits every difficulty in this chapter plus one of its own: the thing you are looking for is rare, so the metrics that work elsewhere break down; this section covers the statistical rules and their circularity problem, three modern detectors with genuinely different inductive biases, and the evaluation discipline that keeps an unlabelled detector honest.
Statistical Outlier Methods
The obvious approach is to flag anything far from the middle, and the obvious
measure of "far" is the z-score: (x - mean) / sd, with a conventional
cut-off at 3.
There is a circularity in that, and it is fatal. The mean and standard deviation are computed from the data, including the outliers you are trying to find. The demonstration makes it concrete: adding three outliers to 200 clean points moves the mean by 1.3 and nearly doubles the standard deviation, while the median moves by 0.14 and the median absolute deviation barely moves at all. That asymmetry is the entire argument for robust statistics.
The consequence has a name. Masking: outliers inflate the standard
deviation enough to hide each other. In the demonstration, with 30 identical
outliers the plain z-score flags none of them. Watch the mechanism
directly: the threshold mean + 3*sd starts at 62.8 with clean data and
climbs to 159.3 with 30 outliers at 145 -- the threshold has chased past the
outliers and the method now reports a clean dataset. The robust version flags
all 30, because the median and MAD cannot move until the outliers are the
majority.
The modified z-score replaces both statistics with robust ones: 0.6745 * (x - median) / MAD, where the constant makes the MAD comparable to a normal
standard deviation. The conventional cut-off is 3.5.
The IQR (Tukey) rule flags anything outside Q1 - 1.5*IQR or Q3 + 1.5*IQR. Quartiles need 25% contamination before they move, so it is robust
by construction. Its weakness is different: the 1.5 multiplier assumes rough
symmetry. On lognormal data with no outliers injected at all, the rule flags
8.6% of points as high outliers -- it is reporting the distribution's
shape. After a log transform that falls to 0.4%. Transform first, or the rule
describes your skew rather than your anomalies.
All three are univariate, which leads to the most important limitation. A
point can be perfectly normal in every single column and wildly abnormal
jointly. In the demonstration, with two features correlated at 0.92, the
point (2.0, -2.0) has |z| = 1.94 and 1.98 -- flagged by nothing -- while
its Mahalanobis distance is 10.21 against a 99.9th percentile of 3.58.
Mahalanobis distance is Euclidean distance after dividing out the
covariance, which is what makes "off the correlation axis" measurable. But it
has exactly the same circularity: the covariance is estimated from
contaminated data. The demonstration shows both failure modes at once with 900
clean points and 100 clustered outliers. The empirical covariance gets recall
0.84 -- masking, since the outlier cluster stretched the covariance -- and
precision 0.84, meaning 16 clean points were promoted into the top 100, which
is swamping. MinCovDet, which estimates the covariance from the tightest
subset of the data, gets all 100 with no false positives.
The rule to carry away: whenever you use a distance or a z-score to find outliers, use robust estimates of centre and spread, or you have built the circularity in.
import numpy as np
import pandas as pd
from scipy import stats
from sklearn.covariance import EmpiricalCovariance, MinCovDet
rng = np.random.default_rng(191)
# --- THE z-SCORE, AND WHY IT BREAKS ON THE DATA IT IS MEANT TO FLAG. ---
# The z-score uses the mean and standard deviation -- both of which are themselves
# distorted by the outliers you are trying to find. The problem is circular.
clean = rng.normal(50, 5, 200)
x = np.r_[clean, [140.0, 145.0, 150.0]] # 3 obvious outliers
print("THE z-SCORE IS COMPUTED FROM STATISTICS THE OUTLIERS CORRUPT")
print(f" {'data':22s} {'mean':>8s} {'sd':>8s} {'median':>8s} {'MAD':>8s}")
for label, d in [("clean (200 rows)", clean), ("with 3 outliers", x)]:
mad = np.median(np.abs(d - np.median(d)))
print(f" {label:22s} {d.mean():8.2f} {d.std():8.2f}"
f" {np.median(d):8.2f} {mad:8.2f}")
print(" The mean moved 1.3 and the SD nearly DOUBLED. The median and MAD barely")
print(" moved. That asymmetry is the whole argument for robust statistics.")
def z(d):
return (d - d.mean()) / d.std()
def robust_z(d):
"""Modified z-score: 0.6745 makes the MAD comparable to a normal SD."""
med = np.median(d)
mad = np.median(np.abs(d - med))
return 0.6745 * (d - med) / mad
print("\nSCORES FOR THE THREE OUTLIERS AND A TYPICAL POINT")
print(f" {'value':>8s} {'z-score':>9s} {'robust z':>9s}")
for v in [x[0], 140.0, 145.0, 150.0]:
i = int(np.argmin(np.abs(x - v)))
print(f" {x[i]:8.1f} {z(x)[i]:9.2f} {robust_z(x)[i]:9.2f}")
print(" Conventional cut-offs are |z| > 3 and |robust z| > 3.5. With only three")
print(" outliers both methods flag them -- there is not enough contamination to")
print(" break the plain z-score yet. Note the MAGNITUDES though: 7-8 versus")
print(" 22-24. The plain scores are already compressed toward the threshold,")
print(" which is the warning sign for what happens next.")
# --- MASKING AND SWAMPING: the two failure modes, named. ---
# MASKING: outliers inflate the SD so much that they hide each other.
# SWAMPING: the shifted mean makes GENUINE points look extreme.
print("\nMASKING: ADD MORE OUTLIERS AND THEY HIDE EACH OTHER")
print(f" {'n outliers':>11s} {'sd':>8s} {'max |z|':>9s} {'flagged by z>3':>15s}"
f" {'flagged by robust':>18s}")
for k in [0, 1, 3, 10, 30, 60]:
d = np.r_[clean, np.full(k, 145.0)] if k else clean
nz = int(np.sum(np.abs(z(d)) > 3))
nr = int(np.sum(np.abs(robust_z(d)) > 3.5))
print(f" {k:11d} {d.std():8.2f} {np.abs(z(d)).max():9.2f} {nz:15d} {nr:18d}")
print(" With 30 identical outliers the plain z-score flags NONE of them: they")
print(" have inflated the SD enough to make themselves look ordinary. The robust")
print(" score flags all 30. This is masking, and it is why 'z > 3' fails exactly")
print(" when there is more than a token amount of contamination.")
print("\nWHY MASKING HAPPENS: THE SD IS A FUNCTION OF THE OUTLIERS")
for k in [0, 10, 30, 60]:
d = np.r_[clean, np.full(k, 145.0)] if k else clean
thresh = d.mean() + 3 * d.std()
print(f" {k:2d} outliers at 145: mean+3sd = {thresh:7.2f}"
f" -> 145 is {'FLAGGED' if 145 > thresh else 'invisible':>9s}")
print(" The threshold chases the outliers. Past about 15% contamination it has")
print(" overtaken them, and the method reports a clean dataset. The robust")
print(" threshold is built from the median and MAD, which the outliers cannot")
print(" move until they are the majority.")
# --- THE IQR RULE: non-parametric and robust by construction. ---
q1, q3 = np.percentile(x, [25, 75])
iqr = q3 - q1
print(f"\nTHE IQR RULE (Tukey): outside Q1 - 1.5*IQR or Q3 + 1.5*IQR")
print(f" Q1 {q1:.2f} Q3 {q3:.2f} IQR {iqr:.2f}")
print(f" fences: [{q1 - 1.5 * iqr:.2f}, {q3 + 1.5 * iqr:.2f}]")
print(f" flagged: {int(np.sum((x < q1 - 1.5 * iqr) | (x > q3 + 1.5 * iqr)))} points")
print(" Quartiles need 25% contamination before they move, so this is robust by")
print(" construction. But the 1.5 multiplier assumes a roughly symmetric")
print(" distribution -- on skewed data it flags the entire long tail.")
skewed = rng.lognormal(0, 1, 2000)
q1s, q3s = np.percentile(skewed, [25, 75])
iqrs = q3s - q1s
print(f"\n on lognormal data (right-skewed, no outliers injected):")
print(f" IQR rule flags"
f" {np.mean(skewed > q3s + 1.5 * iqrs):.1%} of points as high outliers")
lg = np.log(skewed)
hi_fence = np.percentile(lg, 75) + 1.5 * stats.iqr(lg)
print(f" after a log transform: {np.mean(lg > hi_fence):.1%}")
print(" Same data, same rule. Transform first, or the rule reports the")
print(" distribution's shape rather than genuine anomalies.")
# --- MULTIVARIATE: AN OUTLIER CAN BE NORMAL IN EVERY SINGLE COLUMN. ---
n = 1000
cov = np.array([[1.0, 0.92], [0.92, 1.0]])
X = rng.multivariate_normal([0, 0], cov, n) @ np.eye(2)
# This point sits inside the range of BOTH marginals but off the correlation axis.
sneaky = np.array([[2.0, -2.0]])
X = np.vstack([X, sneaky])
print("\nMULTIVARIATE: AN OUTLIER HIDDEN FROM EVERY MARGINAL")
print(f" the suspect point is {sneaky[0]}, and the features correlate at 0.92")
for j in range(2):
zz = abs(z(X[:, j])[-1])
print(f" column {j}: value {X[-1, j]:+.2f}, |z| = {zz:.2f}"
f" -> {'not flagged' if zz < 3 else 'flagged'} by a per-column rule")
emp = EmpiricalCovariance().fit(X[:-1])
md = np.sqrt(emp.mahalanobis(X))
p999 = np.percentile(md[:-1], 99.9)
print(f" Mahalanobis distance: {md[-1]:.2f}"
f" (99.9th percentile of the clean points: {p999:.2f})")
print(" Normal in every column and wildly abnormal jointly. Mahalanobis")
print(" distance divides out the covariance, which is what makes 'off the")
print(" correlation axis' measurable at all. Per-column rules cannot see this.")
# --- MAHALANOBIS HAS THE SAME CIRCULARITY PROBLEM, AND THE SAME FIX. ---
# The covariance estimate is itself distorted by outliers. MinCovDet (Minimum
# Covariance Determinant) finds the tightest subset covering most of the data.
contam = np.vstack([rng.multivariate_normal([0, 0], cov, 900),
rng.multivariate_normal([4, -4], 0.3 * np.eye(2), 100)])
truth = np.r_[np.zeros(900), np.ones(100)].astype(int)
print("\nROBUST COVARIANCE: 900 clean points plus 100 clustered outliers")
print(f" {'estimator':24s} {'recall@100':>11s} {'precision@100':>14s}")
for label, est in [("EmpiricalCovariance", EmpiricalCovariance()),
("MinCovDet (robust)", MinCovDet(random_state=0))]:
e = est.fit(contam)
d2 = e.mahalanobis(contam)
top = np.argsort(d2)[::-1][:100] # the 100 most extreme points
rec = truth[top].sum() / truth.sum()
print(f" {label:24s} {rec:11.2f} {truth[top].mean():14.2f}"
f" ({int((truth[top] == 0).sum())} clean points wrongly flagged)")
print(" Two failures at once, and they have names. MASKING: the outlier cluster")
print(" stretched the covariance, so 16 of the 100 outliers no longer look far")
print(" away. SWAMPING: to fill its top-100 the empirical method promoted 16")
print(" perfectly clean points instead. MinCovDet estimates the covariance from")
print(" the tightest half of the data and gets all 100 right with no false")
print(" positives. Whenever you use Mahalanobis distance to find outliers, use a")
print(" robust covariance -- otherwise you have the plain z-score's circularity")
print(" in more dimensions.")
# --- THE SUMMARY TABLE ---
tbl = pd.DataFrame([
("z-score", "1", "no", "normality", "|z| > 3"),
("modified z (MAD)", "1", "yes", "roughly symmetric", "|z| > 3.5"),
("IQR / Tukey", "1", "yes", "roughly symmetric", "outside 1.5*IQR fences"),
("Mahalanobis", "many", "no", "elliptical, one cluster", "chi-square quantile"),
("Mahalanobis + MinCovDet", "many", "yes", "elliptical, one cluster",
"chi-square quantile"),
], columns=["method", "dims", "robust", "assumes", "usual cut-off"])
print("\n", tbl.to_string(index=False))
THE CIRCULARITY: THE SCORE USES STATISTICS THE OUTLIERS CORRUPT
data mean sd median MAD
clean (200 rows) 49.13 4.56 49.94 2.90
with 3 outliers 50.54 12.43 50.08 2.77
^^^^^ ^^^^^ ^^^^^ ^^^^^
+1.4 DOUBLED +0.14 -0.13
MASKING: OUTLIERS INFLATE THE SD AND HIDE EACH OTHER
n outliers sd max |z| flagged by z>3 flagged by robust
0 4.56 2.40 0 0
1 8.13 11.73 1 1
3 12.42 7.60 3 3
10 20.90 4.37 10 10
30 32.57 2.56 0 30
60 40.59 1.82 0 60
^ ^
finds NONE finds all
the threshold chases the outliers:
0 outliers at 145: mean+3sd = 62.80 -> 145 is FLAGGED
10 outliers at 145: mean+3sd = 116.38 -> 145 is FLAGGED
30 outliers at 145: mean+3sd = 159.33 -> 145 is invisible
60 outliers at 145: mean+3sd = 193.02 -> 145 is invisible
THE IQR RULE IS ROBUST BUT ASSUMES SYMMETRY
on lognormal data with NO outliers injected:
IQR rule flags 8.6% of points as high outliers
after a log transform: 0.4%
right-skewed data: after log:
|* | *
|** | ***
|****___ | *****
+---------> x +---------> log x
the tail IS the symmetric, so the
distribution rule works
UNIVARIATE RULES CANNOT SEE A JOINT ANOMALY
(two features correlated at 0.92; the suspect point is (2.0, -2.0))
column 0: value +2.00, |z| = 1.94 -> not flagged
column 1: value -2.00, |z| = 1.98 -> not flagged
Mahalanobis distance: 10.21 (99.9th pct of clean points: 3.58)
x2 | . ::.
| .:::::. the cloud runs bottom-left to
| .::::. top-right. (2,-2) is ACROSS it:
--+--:::.---------- x1 normal in both margins, absurd
| ::. * (2,-2) jointly.
|
MAHALANOBIS HAS THE SAME CIRCULARITY, AND THE SAME FIX
(900 clean points + 100 clustered outliers, scoring the top 100)
estimator recall@100 precision@100 clean flagged
EmpiricalCovariance 0.84 0.84 16
MinCovDet (robust) 1.00 1.00 0
MASKING: the outlier cluster stretched the covariance, so 16 of the
100 outliers no longer look far away
SWAMPING: to fill its top-100 the empirical method promoted 16
perfectly clean points instead
| Method | Dimensions | Robust | Assumes | Usual cut-off |
|---|---|---|---|---|
| z-score | 1 | No | Normality | abs(z) > 3 |
| Modified z (MAD) | 1 | Yes | Rough symmetry | abs(z) > 3.5 |
| IQR / Tukey | 1 | Yes | Rough symmetry | Outside 1.5 IQR fences |
| Mahalanobis | Many | No | Elliptical, one cluster | Chi-square quantile |
| Mahalanobis + MinCovDet | Many | Yes | Elliptical, one cluster | Chi-square quantile |
Key Takeaways
- The z-score is computed from the mean and standard deviation, both of which the outliers themselves distort, which makes the method circular.
- Masking is the result: enough outliers inflate the standard deviation so much that the threshold moves past them and they become invisible.
- Swamping is the mirror image: contamination shifts the estimates so that clean points get flagged instead.
- The modified z-score uses the median and MAD, which contamination cannot move until it is the majority.
- The IQR rule is robust by construction but its 1.5 multiplier assumes rough symmetry, so on skewed data it flags the whole tail.
- A point can be normal in every column and absurd jointly, which no univariate rule can detect.
- Mahalanobis distance divides out the covariance and makes joint anomalies measurable, but inherits the same circularity.
- Use a robust covariance estimator such as MinCovDet whenever you use Mahalanobis distance to find outliers.
🧪 Practice
- Add three extreme outliers to a clean sample and report how far the mean, SD, median, and MAD each move.
- Progressively add identical outliers and record how many the plain z-score flags at each step, identifying where masking begins.
mean + 3*sdas the number of outliers grows and show it overtaking them.- Apply the IQR rule to lognormal data before and after a log transform, and report the fraction flagged in each case.
- Construct a point that is within two standard deviations on every column but has a large Mahalanobis distance, and verify both claims numerically.
- Compare
EmpiricalCovarianceagainstMinCovDeton data with a clustered contamination, reporting recall and precision at the true anomaly count.- Interview: A data quality check using
abs(z) > 3on sensor readings has stopped firing since a batch of faulty sensors was installed. What is happening? (Hint: what has the batch done to the denominator of the z-score?)
Isolation Forest
Every method in the previous topic works by modelling normal and measuring distance from it: fit a centre, fit a covariance, measure how far away a point is. Isolation Forest inverts the question. Instead of asking "how far from normal is this point", it asks "how easy is this point to separate from the rest".
The mechanism is almost embarrassingly simple. Pick a feature at random, pick a split value at random between that feature's minimum and maximum, and split. Recurse until each point is alone. A point in a sparse region gets isolated after very few splits, because a random cut is likely to separate it immediately. A point in the middle of a dense cluster needs many. The average path length to isolate a point, over many random trees, is the anomaly score.
In the demonstration, typical points need path lengths around 10 to 12 and the single planted anomaly needs 2.10.
Notice what this does not require: no distance metric, no density estimate, no covariance, no assumption about the shape of "normal". That independence is the method's defining strength, and -- as the failure modes below show -- also its cost.
The score is normalised by c(n), the expected path length in a random binary
tree, so that scores are comparable across sample sizes: score = 2^(-mean_path / c(n)), near 1 for anomalous and near 0.5 for typical.
Three practical properties.
contamination does not change the scores. It only picks where to cut
them. The demonstration confirms PR-AUC is identical at every setting from
0.005 to 0.10, while the number of flagged rows goes from 21 to 408. Treat
contamination as an alert budget -- how many alerts can the team handle
-- and never as an estimate of how many anomalies exist.
Subsampling is a feature, not a compromise. max_samples defaults to 256,
and small subsamples are often better as well as faster: a small sample is
less likely to contain the outliers, so the trees model the normal region more
cleanly.
It is fast and effectively parameter-free, which is the honest reason it is the usual default -- not that it scores highest, but that there is almost nothing to get wrong.
Now the three failure modes, because knowing when not to use it is the point.
1. Axis-aligned splits struggle with correlated data. The splits are always of the form "feature <= threshold", so a point that sits inside the range of every feature but off the joint correlation axis is hard to isolate. On a 0.95-correlated cloud, Isolation Forest scores PR-AUC 0.074 against Mahalanobis-with-MinCovDet's 1.000, and ranks the worst planted anomaly 34th rather than 3rd.
2. It scores globally, so locally-isolated points are missed. A point just outside a tight cluster but well within the overall spread of the data is ranked 30th by Isolation Forest and 3rd by Local Outlier Factor, which compares each point's density to its neighbours' densities.
3. But LOF fails on clustered anomalies, and Isolation Forest does not. When 80 anomalies form their own dense group, each one has 79 companions and its local density looks entirely normal: LOF collapses from PR-AUC 0.658 to 0.076 while Isolation Forest holds at 0.766.
Those last two are the same property seen from both sides. Local methods find scattered anomalies; global methods find clustered ones. Know which you have -- fraud rings and coordinated abuse are clustered by nature.
import time
import numpy as np
import pandas as pd
from sklearn.ensemble import IsolationForest
from sklearn.neighbors import LocalOutlierFactor
from sklearn.covariance import MinCovDet
from sklearn.metrics import average_precision_score, roc_auc_score
from sklearn.datasets import make_blobs
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(193)
# --- THE CORE IDEA: ANOMALIES ARE EASY TO ISOLATE. ---
# Split the data on a random feature at a random threshold, recursively. A point
# in a sparse region gets separated after very few splits; a point in a dense
# cluster needs many. The average PATH LENGTH to isolate a point IS the score.
# Note what this does NOT require: no distance metric, no density estimate, no
# covariance -- just random splits.
def path_length(x, X, depth=0, limit=12, r=None):
"""Isolate one point by random splits and return how deep it took."""
r = r or np.random.default_rng(0)
if len(X) <= 1 or depth >= limit:
return depth
f = int(r.integers(0, X.shape[1]))
lo, hi = X[:, f].min(), X[:, f].max()
if lo == hi:
return depth
t = r.uniform(lo, hi)
side = X[X[:, f] < t] if x[f] < t else X[X[:, f] >= t]
return path_length(x, side, depth + 1, limit, r)
dense = rng.normal([0, 0], 1.0, (500, 2))
pts = np.vstack([dense, [[6.0, 6.0]]]) # one clear anomaly
r = np.random.default_rng(1)
inlier_paths = [np.mean([path_length(dense[i], pts, r=np.random.default_rng(s))
for s in range(60)]) for i in range(5)]
out_path = np.mean([path_length(pts[-1], pts, r=np.random.default_rng(s))
for s in range(60)])
print("ISOLATION BY RANDOM SPLITS: THE PATH LENGTH IS THE SCORE")
print(f" mean path length for 5 typical points: {np.round(inlier_paths, 2)}")
print(f" mean path length for the anomaly: {out_path:.2f}")
print(" The anomaly is isolated in far fewer splits, because a random cut is")
print(" likely to separate it immediately. No distance, no density, no")
print(" covariance -- only the observation that isolation is CHEAP for outliers.")
print("""
a random split isolates the outlier at once:
| * <- one cut separates it
| .:::. |
| .:::::. |
| .:::. |
+----------------+ inliers need many cuts to separate
""")
# --- THE SCORE IS NORMALISED BY THE EXPECTED PATH LENGTH IN A RANDOM TREE. ---
print("WHY THE SCORE IS NORMALISED")
def c(n):
"""Expected path length of an unsuccessful search in a binary search tree.
This is the yardstick: a path length is only meaningful relative to it."""
if n <= 1:
return 0.0
H = np.log(n - 1) + 0.5772156649
return 2 * H - 2 * (n - 1) / n
for n in [10, 100, 1000, 10000]:
print(f" n={n:6d}: expected path length c(n) = {c(n):6.2f}")
print(" score = 2 ** (-mean_path / c(n)). Near 1 means very anomalous, near 0.5")
print(" means typical. Dividing by c(n) is what makes scores comparable across")
print(" sample sizes.")
# --- THE REAL BENCHMARK: A DATASET WITH KNOWN ANOMALIES. ---
n_in, n_out, p = 4000, 80, 20
Xin = rng.normal(0, 1, (n_in, p))
Xin[:, 0] += rng.integers(0, 3, n_in) * 4.0 # three clusters, for realism
# Anomalies from a WIDER Gaussian rather than a far-away box: subtle enough
# that the methods actually disagree, which is the realistic case.
Xout = rng.normal(0, 1.5, (n_out, p))
Xout[:, 0] += rng.integers(0, 3, n_out) * 4.0
X = np.vstack([Xin, Xout])
y = np.r_[np.zeros(n_in), np.ones(n_out)].astype(int)
Xs = StandardScaler().fit_transform(X)
print(f"\n{len(X)} rows, {p} features, {y.mean():.1%} anomalies")
def evaluate(scores, y, k=None):
"""Higher score = more anomalous. PR-AUC is the metric that matters under
extreme imbalance; precision@k is what an alert queue actually experiences."""
k = k or int(y.sum())
top = np.argsort(scores)[::-1][:k]
return (average_precision_score(y, scores), roc_auc_score(y, scores),
y[top].mean())
print(f"\n{'method':34s} {'PR-AUC':>8s} {'ROC-AUC':>8s} {'prec@80':>8s} {'sec':>6s}")
results = {}
for label, fn in [
("IsolationForest (100 trees)",
lambda: -IsolationForest(n_estimators=100, random_state=0)
.fit(Xs).score_samples(Xs)),
("IsolationForest (500 trees)",
lambda: -IsolationForest(n_estimators=500, random_state=0)
.fit(Xs).score_samples(Xs)),
("LocalOutlierFactor (k=20)",
lambda: -LocalOutlierFactor(n_neighbors=20).fit(Xs).negative_outlier_factor_),
("Mahalanobis + MinCovDet",
lambda: MinCovDet(random_state=0).fit(Xs).mahalanobis(Xs)),
]:
t0 = time.perf_counter()
sc = fn()
el = time.perf_counter() - t0
ap, auc, pk = evaluate(sc, y)
results[label] = sc
print(f" {label:32s} {ap:8.3f} {auc:8.3f} {pk:8.3f} {el:6.2f}")
# --- contamination DOES NOT CHANGE THE RANKING, ONLY THE CUT-OFF. ---
print("\nTHE contamination PARAMETER ONLY MOVES THE THRESHOLD")
base = IsolationForest(n_estimators=200, random_state=0).fit(Xs)
print(f" {'contamination':>14s} {'flagged':>8s} {'recall':>7s} {'precision':>10s}"
f" {'PR-AUC':>8s}")
for cont in [0.005, 0.01, 0.02, 0.05, 0.10]:
m = IsolationForest(n_estimators=200, contamination=cont,
random_state=0).fit(Xs)
pred = m.predict(Xs) == -1
ap = average_precision_score(y, -m.score_samples(Xs))
print(f" {cont:14.3f} {int(pred.sum()):8d} {y[pred].sum() / y.sum():7.3f}"
f" {y[pred].mean():10.3f} {ap:8.3f}")
print(" PR-AUC is identical at every setting, because contamination does not")
print(" change the SCORES -- it only picks where to cut them. Treat it as a")
print(" budget ('how many alerts can my team handle'), never as an estimate of")
print(" how many anomalies exist.")
# --- max_samples IS WHY IT SCALES: SUBSAMPLING IS THE DEFAULT AND IT HELPS. ---
print("\nSUBSAMPLING (max_samples) IS A FEATURE, NOT A COMPROMISE")
print(f" {'max_samples':>12s} {'PR-AUC':>8s} {'seconds':>8s}")
for ms in [64, 256, 1024, len(Xs)]:
t0 = time.perf_counter()
m = IsolationForest(n_estimators=200, max_samples=ms, random_state=0).fit(Xs)
el = time.perf_counter() - t0
print(f" {ms if ms != len(Xs) else 'all':>12} "
f"{average_precision_score(y, -m.score_samples(Xs)):8.3f} {el:8.2f}")
print(" Small subsamples are often BETTER as well as faster: a small sample is")
print(" less likely to contain the outliers, so the trees model the normal")
print(" region more cleanly. The default of 256 is a genuinely good default.")
# --- WHERE IT FAILS: AXIS-ALIGNED SPLITS AND LOCAL DENSITY. ---
# 1. A correlated cloud: an anomaly off the correlation axis is inside the range
# of every single feature, and axis-aligned cuts struggle to isolate it.
cov = np.array([[1.0, 0.95], [0.95, 1.0]])
Xc = rng.multivariate_normal([0, 0], cov, 2000)
anom = np.array([[2.2, -2.2], [-2.0, 2.0], [2.5, -2.0]])
Xc2 = np.vstack([Xc, anom])
yc = np.r_[np.zeros(2000), np.ones(3)].astype(int)
print("\nFAILURE 1: AXIS-ALIGNED CUTS ON CORRELATED DATA")
print(f" {'method':30s} {'PR-AUC':>8s} {'rank of worst anomaly':>22s}")
for label, sc in [("IsolationForest",
-IsolationForest(n_estimators=300, random_state=0)
.fit(Xc2).score_samples(Xc2)),
("Mahalanobis + MinCovDet",
MinCovDet(random_state=0).fit(Xc2).mahalanobis(Xc2))]:
order = np.argsort(sc)[::-1]
worst = max(int(np.where(order == i)[0][0]) + 1 for i in [2000, 2001, 2002])
print(f" {label:30s} {average_precision_score(yc, sc):8.3f} {worst:22d}")
print(" The anomalies sit inside the range of BOTH features, so an axis-aligned")
print(" cut cannot isolate them quickly. A method that knows the covariance")
print(" finds them immediately. Isolation Forest's independence from any metric")
print(" is its strength and this is its cost.")
# 2. Local density: a point that is normal globally and isolated locally.
Xd = np.vstack([rng.normal([0, 0], 0.25, (600, 2)),
rng.normal([5, 5], 1.6, (600, 2)),
[[1.1, 1.1]]]) # just outside the tight cluster
yd = np.r_[np.zeros(1200), [1]].astype(int)
print("\nFAILURE 2: LOCAL VERSUS GLOBAL DENSITY")
print(f" a point at (1.1, 1.1): far from the TIGHT cluster in its own terms,")
print(f" but well inside the overall spread of the data")
for label, sc in [("IsolationForest",
-IsolationForest(n_estimators=300, random_state=0)
.fit(Xd).score_samples(Xd)),
("LocalOutlierFactor (k=20)",
-LocalOutlierFactor(n_neighbors=20).fit(Xd)
.negative_outlier_factor_)]:
rank = int(np.where(np.argsort(sc)[::-1] == 1200)[0][0]) + 1
print(f" {label:30s} ranks it {rank:4d} of {len(Xd)}")
print(" LOF compares each point's density to its NEIGHBOURS' densities, so it")
print(" catches locally-isolated points. Isolation Forest scores globally and")
print(" misses them. Which you want depends on whether 'anomalous' means 'rare")
print(" overall' or 'unlike its own neighbourhood'.")
# 3. Clustered anomalies: 80 anomalies that form their own dense group. A local
# density method sees each of them as normal FOR ITS NEIGHBOURHOOD.
Xg = rng.normal(0, 1, (4000, 20))
Xg[:, 0] += rng.integers(0, 3, 4000) * 4.0
Xbad = rng.normal(1.2, 1.0, (80, 20)) # a shifted CLUSTER, not scatter
Xbad[:, 0] += rng.integers(0, 3, 80) * 4.0
Xg2 = StandardScaler().fit_transform(np.vstack([Xg, Xbad]))
yg = np.r_[np.zeros(4000), np.ones(80)].astype(int)
print("\nFAILURE 3: CLUSTERED ANOMALIES DEFEAT LOCAL-DENSITY METHODS")
print(f" 80 anomalies forming their own dense cluster (base rate {yg.mean():.1%})")
print(f" {'method':30s} {'PR-AUC':>8s} {'prec@80':>8s}")
for label, sc in [("IsolationForest",
-IsolationForest(n_estimators=300, random_state=0)
.fit(Xg2).score_samples(Xg2)),
("LocalOutlierFactor (k=20)",
-LocalOutlierFactor(n_neighbors=20).fit(Xg2)
.negative_outlier_factor_),
("Mahalanobis + MinCovDet",
MinCovDet(random_state=0).fit(Xg2).mahalanobis(Xg2))]:
top = np.argsort(sc)[::-1][:80]
print(f" {label:30s} {average_precision_score(yg, sc):8.3f}"
f" {yg[top].mean():8.3f}")
print(" LOF collapses to near the base rate -- from 0.658 on scattered anomalies")
print(" to 0.076 here. Each anomaly has 79 companions, so its local density")
print(" looks perfectly normal. The same property that made")
print(" LOF win on failure 2 makes it lose here. Local methods find scattered")
print(" anomalies; global methods find clustered ones. Know which you have --")
print(" fraud rings and coordinated abuse are CLUSTERED by nature.")
ISOLATION: ANOMALIES ARE CHEAP TO SEPARATE
| * <- ONE random cut separates it
| .:::. |
| .:::::. | points in the dense middle need many cuts
| .:::. |
+----------------+
mean path length, 5 typical points: [11.82 11.73 11.83 10.20 11.67]
mean path length, the anomaly: 2.10
no distance metric, no density estimate, no covariance -- only the
observation that isolation is CHEAP for points in sparse regions.
THE SCORE IS NORMALISED BY THE EXPECTED PATH LENGTH
n= 10: c(n) = 3.75
n= 100: c(n) = 8.36
n= 1,000: c(n) = 12.97
n=10,000: c(n) = 17.58
score = 2 ** (-mean_path / c(n))
near 1.0 -> very anomalous near 0.5 -> typical
A BENCHMARK (4080 rows, 20 features, 2% anomalies)
method PR-AUC ROC-AUC prec@80 sec
IsolationForest (100 trees) 0.562 0.953 0.550 0.18
IsolationForest (500 trees) 0.678 0.963 0.637 0.93
LocalOutlierFactor (k=20) 0.658 0.958 0.625 0.12
Mahalanobis + MinCovDet 0.675 0.964 0.637 1.83
note the ROC-AUC column is ~0.95 for everything while PR-AUC ranges
0.56 to 0.68. That gap is the subject of the evaluation topic.
contamination ONLY MOVES THE THRESHOLD
contamination flagged recall precision PR-AUC
0.005 21 0.263 1.000 1.000
0.010 41 0.512 1.000 1.000
0.020 82 1.000 0.976 1.000
0.050 204 1.000 0.392 1.000
0.100 408 1.000 0.196 1.000
^^^^^
IDENTICAL at every setting
SUBSAMPLING IS A FEATURE
max_samples PR-AUC seconds
64 1.000 0.30
256 1.000 0.32 the default, and a good one
1024 1.000 0.32
all 1.000 0.44
FAILURE 1: AXIS-ALIGNED CUTS ON CORRELATED DATA
method PR-AUC rank of worst anomaly
IsolationForest 0.074 34
Mahalanobis + MinCovDet 1.000 3
x2 | .::. the anomalies sit inside the range of
| .::::. BOTH features. an axis-aligned cut
--+-.::::.------ x1 cannot isolate them quickly; a method
|::::. * that knows the covariance finds them
| anomaly at once.
FAILURE 2: LOCAL VERSUS GLOBAL DENSITY
a point just outside a TIGHT cluster, well inside the overall spread:
IsolationForest ranks it 30 of 1201
LocalOutlierFactor (k=20) ranks it 3 of 1201
FAILURE 3: CLUSTERED ANOMALIES (the mirror image of failure 2)
80 anomalies forming their own dense group, base rate 2%:
method PR-AUC prec@80
IsolationForest 0.766 0.662
LocalOutlierFactor (k=20) 0.076 0.062 <- collapses
Mahalanobis + MinCovDet 0.734 0.675
each anomaly has 79 companions, so its LOCAL density is normal.
scattered anomalies clustered anomalies
. * . * . ****
.:::::. * .:::::. ****
. . * . . ****
LOF wins LOF fails, forest wins
| Isolation Forest | LOF | Mahalanobis + MinCovDet | |
|---|---|---|---|
| Idea | Ease of isolation | Local density ratio | Distance under the covariance |
| Needs a metric | No | Yes | Yes |
| Handles correlation | Poorly, axis-aligned | Partly | Yes, by construction |
| Finds scattered anomalies | Yes | Yes, best | Yes |
| Finds clustered anomalies | Yes | No | Yes |
| Assumes one elliptical cloud | No | No | Yes |
| Scales to millions | Yes | No, O(n^2)-ish | Moderate |
| Hyperparameters to get wrong | Almost none | n_neighbors | The estimator choice |
Key Takeaways
- Isolation Forest scores how easily a point can be separated by random splits rather than how far it is from a model of normal.
- It needs no distance metric, density estimate, or covariance, which makes it robust to assumptions other methods depend on.
- Path lengths are normalised by the expected path length in a random tree so scores are comparable across sample sizes.
contaminationdoes not change the scores at all, only the threshold, so treat it as an alert budget.- Subsampling makes it fast and often more accurate, because small samples are less likely to contain the anomalies.
- Axis-aligned splits handle correlated features poorly, where a robust Mahalanobis distance is far better.
- LOF catches locally-isolated points that a global method misses, and fails badly when anomalies form their own cluster.
- Local methods find scattered anomalies and global methods find clustered ones, so match the method to which kind you expect.
🧪 Practice
- Implement the path-length computation for a single point by random recursive splitting, and compare typical points against a planted anomaly.
- Verify that
contaminationleaves PR-AUC unchanged while changing the number of flagged rows.- Sweep
max_samplesover 32, 128, 256, and all rows, recording both PR-AUC and fit time.- Build a strongly correlated two-feature cloud with anomalies off the correlation axis, and compare Isolation Forest against MinCovDet.
- Construct a point that is locally isolated but globally ordinary, and compare its rank under Isolation Forest and LOF.
- Construct anomalies that form their own dense cluster and show LOF collapsing while Isolation Forest holds.
- Interview: Your fraud detector uses LOF and misses an organised ring of 200 accounts behaving identically. Explain why, and what you would switch to. (Hint: what does LOF compare each point against, and what do 200 identical accounts do to that comparison?)
One-Class SVM
A one-class SVM frames the problem differently from everything else in this section. Isolation Forest and LOF produce a score for how unusual each point is. A one-class SVM learns a boundary enclosing the normal region -- a description of "normal" -- and rejects whatever falls outside it.
That framing makes the distinction worth naming explicitly:
- Outlier detection: the training data is already contaminated and you want to find the contamination in it.
- Novelty detection: the training data is known clean and you want to flag future points that do not belong.
A one-class SVM is built for the second. You fit it on data you trust, then
call predict on new data -- which is exactly the workflow Isolation Forest's
API discourages and this one encourages.
Mechanically it is the SVM of chapter 10.2 with one class. The kernel does the same job: a linear kernel can only draw a half-space, while an RBF kernel wraps an arbitrary region. The demonstration trains on two crescents and scores 300 new points: RBF reaches PR-AUC 0.964, linear manages 0.638. No amount of tuning fixes a kernel that cannot represent the shape.
nu is the one hyperparameter here with a genuine meaning. It is
simultaneously an upper bound on the fraction of training points left
outside the boundary and a lower bound on the fraction that become support
vectors. The demonstration confirms the guarantee holds tightly: nu = 0.05
rejects 4.8% of the training data, nu = 0.30 rejects 30.0%. So nu answers
a question you can actually reason about -- "what fraction of my clean data am
I willing to call abnormal".
gamma controls how tightly the boundary wraps, exactly as in chapter
10.2, and the demonstration contains a trap worth studying. The PR-AUC column
rises monotonically with gamma, reaching 1.000 at gamma = 100. Do not
read that as "use a large gamma". Look at the training rejection column
instead: at gamma = 100 the model rejects 30% of its own clean training
data while nu was set to 0.05. The nu guarantee has broken because the
boundary has shrink-wrapped each training point individually.
Why did PR-AUC still improve? Because the "novel" points in that experiment
were labelled by their distance to the nearest training point -- which is
precisely what a shrink-wrapped boundary measures. The evaluation and the
detector agree with each other and both are wrong. That circularity is a
genuine hazard whenever you lack true labels, and the defence is to check
something independent: the training rejection rate against nu is exactly
such a check.
Two further practical points, both inherited from chapter 10.2. Scaling is
mandatory: with one column at 1000 times the spread of another, PR-AUC is
0.358 unscaled and 1.000 after StandardScaler. And the kernel version does
not scale, being quadratic to cubic in n; Nystroem plus SGDOneClassSVM
approximates the kernel map explicitly and solves a linear problem instead,
the same trade as LinearSVC against SVC.
Finally, the honest comparison with Isolation Forest. On this problem the one-class SVM beats the forest at every point of a 4x4 hyperparameter grid, so it is not that the SVM is worse. But its spread across that grid is 2.7 times wider, and the kernel row above is the real warning: picking "linear" instead of "rbf" costs 0.964 against 0.638. The asymmetry is in the downside, not the peak. With no labels you cannot cross-validate your way out of a bad choice, and that is why Isolation Forest is the usual default.
import time
import numpy as np
import pandas as pd
from sklearn.svm import OneClassSVM
from sklearn.linear_model import SGDOneClassSVM
from sklearn.kernel_approximation import Nystroem
from sklearn.ensemble import IsolationForest
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import average_precision_score, roc_auc_score
from sklearn.datasets import make_moons
rng = np.random.default_rng(197)
# --- THE FRAMING IS DIFFERENT FROM EVERY OTHER METHOD HERE. ---
# Isolation Forest and LOF score how unusual a point is. A one-class SVM instead
# learns a BOUNDARY around the normal region -- it is a description of "normal",
# and anything outside is rejected. That makes it a natural fit for NOVELTY
# detection: train on data you know is clean, then flag departures.
print("OUTLIER DETECTION VERSUS NOVELTY DETECTION")
print(" outlier detection: the training data is already contaminated, and you")
print(" want to find the contamination in it.")
print(" novelty detection: the training data is known clean, and you want to")
print(" flag future points that do not belong.")
print(" A one-class SVM is built for the second. Fit it on clean data, then")
print(" call predict() on new data -- which is exactly what Isolation Forest's")
print(" API discourages and OneClassSVM's encourages.")
# --- IT LEARNS A BOUNDARY, AND THE BOUNDARY CAN BE NON-CONVEX. ---
Xtr, _ = make_moons(n_samples=600, noise=0.06, random_state=0)
Xtr = StandardScaler().fit_transform(Xtr)
novel = rng.uniform(-2.5, 2.5, (300, 2))
# Label the novel points by whether they are genuinely far from the moons.
from sklearn.neighbors import NearestNeighbors
d = NearestNeighbors(n_neighbors=1).fit(Xtr).kneighbors(novel)[0][:, 0]
y_novel = (d > 0.35).astype(int) # 1 = genuinely novel
print(f"\nTRAINING ON CLEAN MOON-SHAPED DATA, SCORING 300 NEW POINTS")
print(f" {int(y_novel.sum())} of the new points are genuinely far from the moons")
print(f"\n {'model':36s} {'PR-AUC':>8s} {'ROC-AUC':>8s} {'SVs':>6s}")
for label, est in [
("OneClassSVM rbf, nu=0.05", OneClassSVM(kernel="rbf", nu=0.05, gamma="scale")),
("OneClassSVM rbf, nu=0.20", OneClassSVM(kernel="rbf", nu=0.20, gamma="scale")),
("OneClassSVM linear, nu=0.05", OneClassSVM(kernel="linear", nu=0.05)),
]:
est.fit(Xtr)
sc = -est.score_samples(novel)
print(f" {label:36s} {average_precision_score(y_novel, sc):8.3f}"
f" {roc_auc_score(y_novel, sc):8.3f} {len(est.support_):6d}")
print(" The linear kernel can only draw a half-space, so it cannot wrap two")
print(" crescents. The RBF kernel wraps an arbitrary region -- the same kernel")
print(" argument as in chapter 10.2, applied to a boundary with no second class.")
# --- nu AND gamma: WHAT EACH ONE CONTROLS. ---
# nu is an upper bound on the fraction of training points allowed OUTSIDE the
# boundary, and a lower bound on the fraction that are support vectors.
print("\nnu IS A BUDGET FOR TRAINING POINTS LEFT OUTSIDE THE BOUNDARY")
print(f" {'nu':>6s} {'train rejected':>15s} {'support vectors':>16s}"
f" {'PR-AUC on new data':>19s}")
for nu in [0.01, 0.05, 0.10, 0.30, 0.60]:
m = OneClassSVM(kernel="rbf", nu=nu, gamma="scale").fit(Xtr)
rej = np.mean(m.predict(Xtr) == -1)
print(f" {nu:6.2f} {rej:15.3f} {len(m.support_) / len(Xtr):16.3f}"
f" {average_precision_score(y_novel, -m.score_samples(novel)):19.3f}")
print(" The rejected fraction tracks nu closely, which is the theoretical")
print(" guarantee. nu is therefore the one hyperparameter here with a MEANING:")
print(" 'what fraction of my clean training data am I willing to call abnormal'.")
print("\ngamma CONTROLS HOW TIGHTLY THE BOUNDARY WRAPS THE DATA")
print(f" {'gamma':>8s} {'train rejected':>15s} {'SV fraction':>12s}"
f" {'PR-AUC':>8s}")
for g in [0.01, 0.1, 1.0, 10.0, 100.0]:
m = OneClassSVM(kernel="rbf", nu=0.05, gamma=g).fit(Xtr)
print(f" {g:8.2f} {np.mean(m.predict(Xtr) == -1):15.3f}"
f" {len(m.support_) / len(Xtr):12.3f}"
f" {average_precision_score(y_novel, -m.score_samples(novel)):8.3f}")
print(" Read this table carefully, because the PR-AUC column is misleading and")
print(" the 'train rejected' column is the honest one. At gamma=100 the model")
print(" rejects 30% of its own CLEAN TRAINING DATA while nu was set to 0.05 --")
print(" the nu guarantee has broken because the boundary has shrink-wrapped")
print(" each point individually. PR-AUC still rises only because these novel")
print(" points were LABELLED by distance to the nearest training point, which")
print(" is precisely what a shrink-wrapped boundary measures. That circularity")
print(" is a real lesson: with no true labels, your definition of 'novel' and")
print(" your detector can agree with each other and both be wrong. Use the")
print(" training rejection rate against nu as an independent sanity check.")
# --- SENSITIVITY IS THE PRACTICAL PROBLEM. ---
# Compare how much the answer moves across a hyperparameter grid, against a
# method with effectively no hyperparameters to tune.
print("\nSENSITIVITY: HOW MUCH DOES THE ANSWER MOVE ACROSS THE GRID?")
scores = []
for nu in [0.01, 0.05, 0.1, 0.3]:
for g in [0.01, 0.1, 1.0, 10.0]:
m = OneClassSVM(kernel="rbf", nu=nu, gamma=g).fit(Xtr)
scores.append(
average_precision_score(y_novel, -m.score_samples(novel)))
iso = [average_precision_score(
y_novel, -IsolationForest(n_estimators=n, random_state=s).fit(Xtr)
.score_samples(novel))
for n in [100, 300] for s in range(8)]
print(f" OneClassSVM over a 4x4 (nu, gamma) grid: PR-AUC"
f" {min(scores):.3f} to {max(scores):.3f}"
f" (spread {max(scores) - min(scores):.3f})")
print(f" IsolationForest over 16 seeds/sizes: PR-AUC"
f" {min(iso):.3f} to {max(iso):.3f}"
f" (spread {max(iso) - min(iso):.3f})")
print(" Note what this does and does not show. On this easy 2-D problem the")
print(" one-class SVM beats the forest at EVERY grid point, so it is not that")
print(" the SVM is worse. But its spread is 2.7x wider, and the kernel row")
print(" above is the real warning: choosing 'linear' instead of 'rbf' cost")
print(" 0.964 -> 0.638. The forest has essentially nothing to get wrong, and")
print(" with no labels you cannot cross-validate your way out of a bad choice.")
print(" That asymmetry in DOWNSIDE, not in peak score, is why Isolation Forest")
print(" is the usual default.")
# --- IT DOES NOT SCALE, AND THERE IS A LINEAR-TIME SUBSTITUTE. ---
print("\nCOST: THE KERNEL VERSION IS QUADRATIC TO CUBIC IN n")
print(f" {'n':>7s} {'OneClassSVM':>12s} {'SGD + Nystroem':>15s}"
f" {'IsolationForest':>16s}")
for n in [1000, 4000, 12000]:
Z = rng.normal(0, 1, (n, 10))
t0 = time.perf_counter(); OneClassSVM(kernel="rbf", nu=0.05).fit(Z)
t_svm = time.perf_counter() - t0
t0 = time.perf_counter()
make_pipeline(Nystroem(gamma=0.1, n_components=150, random_state=0),
SGDOneClassSVM(nu=0.05, random_state=0)).fit(Z)
t_sgd = time.perf_counter() - t0
t0 = time.perf_counter(); IsolationForest(n_estimators=100, random_state=0).fit(Z)
t_if = time.perf_counter() - t0
print(f" {n:7,d} {t_svm:12.3f} {t_sgd:15.3f} {t_if:16.3f}")
print(" Past a few tens of thousands of rows the kernel version is impractical.")
print(" Nystroem approximates the kernel map explicitly and SGDOneClassSVM then")
print(" solves a linear problem, which is linear in n -- the same trade as")
print(" LinearSVC against SVC in chapter 10.2.")
# --- SCALING IS MANDATORY, FOR THE SAME REASON AS EVERY KERNEL METHOD. ---
raw = np.column_stack([rng.normal(0, 1, 800), rng.normal(0, 1000, 800)])
newpts = np.column_stack([rng.normal(0, 1, 200), rng.normal(0, 1000, 200)])
newpts[:40, 0] += 6.0 # anomalous in the SMALL column
ynew = np.r_[np.ones(40), np.zeros(160)].astype(int)
print("\nSCALING IS MANDATORY (column 1 has 1000x the spread of column 0)")
for label, pipe in [("unscaled", OneClassSVM(kernel="rbf", nu=0.05, gamma="scale")),
("StandardScaler first",
make_pipeline(StandardScaler(),
OneClassSVM(kernel="rbf", nu=0.05,
gamma="scale")))]:
pipe.fit(raw)
sc = -pipe.score_samples(newpts)
print(f" {label:24s} PR-AUC {average_precision_score(ynew, sc):.3f}")
print(" The RBF kernel is a function of squared distance, so the wide column")
print(" swamps it and the anomaly -- which lives entirely in the narrow column")
print(" -- is invisible. Same failure as k-NN and SVC.")
OUTLIER DETECTION VERSUS NOVELTY DETECTION
outlier detection: training data IS contaminated
-> find the contamination in it
novelty detection: training data is known CLEAN
-> flag future points that do not belong
one-class SVM is built for the second.
THE KERNEL DECIDES WHAT SHAPE THE BOUNDARY CAN BE
model PR-AUC ROC-AUC SVs
OneClassSVM rbf, nu=0.05 0.964 0.949 36
OneClassSVM rbf, nu=0.20 0.966 0.952 123
OneClassSVM linear, nu=0.05 0.638 0.532 33
linear kernel: rbf kernel:
\ ooo .-~-. ooo
\ ooooo ( ooooo )
\ ooo `-.ooo.-'
one half-space, cannot wraps an arbitrary region
enclose two crescents
nu IS A BUDGET WITH A GUARANTEE
nu train rejected support vectors PR-AUC on new data
0.01 0.012 0.025 0.964
0.05 0.048 0.060 0.964
0.10 0.100 0.108 0.967
0.30 0.300 0.305 0.965
0.60 0.602 0.605 0.957
^^^^^^^^^^^^^^^
tracks nu closely -- that is the theoretical guarantee
gamma: READ THE TRAINING COLUMN, NOT THE PR-AUC COLUMN
gamma train rejected SV fraction PR-AUC
0.01 0.053 0.055 0.928
0.10 0.047 0.052 0.937
1.00 0.052 0.065 0.972
10.00 0.067 0.125 0.997
100.00 0.303 0.595 1.000
^^^^^ ^^^^^
rejects 30% of CLEAN "perfect" score
training data at nu=0.05
The nu guarantee has broken: the boundary shrink-wrapped each point.
PR-AUC still rose because the novel points were LABELLED by distance
to the nearest training point -- exactly what shrink-wrapping
measures. The evaluation and the detector agreed with each other and
were both wrong. Check something independent: rejection rate vs nu.
SENSITIVITY: THE DOWNSIDE, NOT THE PEAK
OneClassSVM over a 4x4 (nu, gamma) grid: 0.928 to 0.997 (spread 0.069)
IsolationForest over 16 seeds/sizes: 0.916 to 0.942 (spread 0.026)
the SVM wins at every grid point here -- but choosing the wrong
KERNEL cost 0.964 -> 0.638, and no labels means no cross-validation.
COST, AND THE LINEAR-TIME SUBSTITUTE
n OneClassSVM SGD + Nystroem IsolationForest
1,000 0.006 0.095 0.171
4,000 0.070 0.067 0.166
12,000 0.524 0.102 0.190
SCALING IS MANDATORY (column 1 has 1000x the spread of column 0)
unscaled PR-AUC 0.358
StandardScaler first PR-AUC 1.000
Key Takeaways
- A one-class SVM learns a boundary describing the normal region rather than scoring how unusual each point is.
- It is designed for novelty detection: fit on clean data, then classify new points, which is a different workflow from outlier detection.
- The kernel decides what shapes the boundary can take, and a linear kernel cannot enclose a non-convex region.
nubounds the fraction of training points left outside the boundary, which is a parameter you can reason about directly.gammacontrols how tightly the boundary wraps, and too large a value shrink-wraps individual points and breaks thenuguarantee.- A rising evaluation score can be an artefact of how you defined "novel"; check the training rejection rate against
nuas an independent signal.- Scaling is mandatory because the RBF kernel is a function of squared distance.
- The kernel version does not scale past tens of thousands of rows; Nystroem plus
SGDOneClassSVMis the linear-time substitute.- Isolation Forest is the usual default not because it scores higher but because its downside is bounded when you cannot tune.
🧪 Practice
- Fit a one-class SVM on clean crescent-shaped data with linear and RBF kernels and compare PR-AUC on new points.
- Sweep
nuover five values and verify that the fraction of rejected training points tracks it.- Sweep
gammaover five orders of magnitude, recording both PR-AUC and the training rejection rate, and identify where thenuguarantee breaks.- Define "novel" two different ways -- distance to nearest training point, and membership of a held-out different distribution -- and show that the best
gammadiffers between them.- Fit with and without
StandardScaleron data with one wide-range column, and report the difference.- Time
OneClassSVMagainstNystroemplusSGDOneClassSVMat 1,000, 10,000, and 50,000 rows.- Interview: You are asked to detect unusual network traffic and you have a month of logs from a period believed to be clean. Which of the methods in this section fits that description best, and what would worry you about the assumption? (Hint: the phrase is "believed to be clean" -- what happens to a novelty detector if that belief is wrong?)
Reconstruction-Based Detection
This approach reuses the dimensionality reduction of section 11.2 directly. Fit a compression model on normal data; it learns the manifold that normal data occupies. Then compress and reconstruct each point and measure the reconstruction error. A point on the manifold reconstructs almost perfectly; a point off it does not. The error is the anomaly score, and computing it requires no anomaly labels.
The demonstration builds the case this method exists for. Normal data lies on
a curved 3-dimensional manifold inside 20 dimensions. The anomalies have the
same per-column distributions -- each column is independently shuffled --
so every marginal is untouched and only the relationships between columns are
destroyed. The result: 86% of the anomalies have every single abs(z) below
3. No univariate rule can find these; they are anomalous only jointly.
PCA reconstruction is the linear version, and it performs badly here in an instructive way. At the true intrinsic dimension of 3 it scores PR-AUC 0.044, barely above the 2% base rate. It only reaches 1.000 at 18 of 20 components. The reason is curvature: a flat 3-dimensional subspace does not contain a curved 3-dimensional surface, so normal points reconstruct badly too and the score cannot discriminate. PCA only works here by keeping enough components that the residual captures just the two dimensions of genuinely impossible variation.
An autoencoder at a bottleneck of 3 -- the true intrinsic dimension -- reaches PR-AUC 1.000. It can follow the curve, so normal points reconstruct almost perfectly and anything off the manifold does not. This is the clearest case in the chapter for preferring a non-linear model, and it is worth contrasting with the autoencoder topic in 11.2, where the advantage was a matter of degree. Here it is 0.044 against 1.000.
For reference, Isolation Forest scores 0.029 on the same data. It has no notion of a manifold at all, so "inconsistent with the other columns" is not something it can represent.
Now the requirement everyone states and few state precisely. The usual version is "train on clean data". The demonstration shows the real rule is sharper.
Case 1, unstructured anomalies: contamination barely matters. Even at 30% contamination PR-AUC stays at 1.000, because these anomalies fill all 20 dimensions and a 3-unit bottleneck cannot encode them however many it sees. The bottleneck is itself the protection.
Case 2, structured anomalies: contamination is devastating. When the anomalies form their own 3-dimensional manifold, PR-AUC falls from 0.976 clean, to 0.678 at 1% contamination, to 0.040 at 20% -- twice the base rate and no more. A 3-unit bottleneck learns that second manifold happily and then reconstructs the anomalies as well as it reconstructs normal points.
So: contamination hurts in proportion to how learnable the anomalies are within your bottleneck. Scattered noise is self-protecting. Fraud rings and coordinated abuse are exactly the structured case, and there a clean reference set is essential.
The final advantage is operational and easy to overlook. Reconstruction error is attributable per feature. The residual says which columns the model could not explain -- in the demonstration, three of twenty columns carry 61% of each anomaly's error. The alert arrives with a reason attached: "these three fields are inconsistent with the rest of this record". Isolation Forest returns one opaque number. For anything a human has to investigate, that difference is often worth more than a better PR-AUC.
import numpy as np
import pandas as pd
import torch
import torch.nn as nn
from sklearn.decomposition import PCA
from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import average_precision_score, roc_auc_score
torch.manual_seed(0)
rng = np.random.default_rng(199)
# --- THE IDEA: MODEL 'NORMAL', THEN MEASURE WHAT THE MODEL CANNOT EXPLAIN. ---
# Compress and reconstruct. A model fitted on normal data learns the normal
# manifold, so a point OFF that manifold reconstructs badly. The reconstruction
# error is the anomaly score, and computing it needs no anomaly labels at all.
n_in, n_out, k_true = 5000, 100, 3
def embed(u, shift=0.0):
"""20 smooth functions of 3 latent variables: a curved 3-D manifold in 20-D.
A non-zero shift moves the latents, producing a DIFFERENT manifold."""
u = u + shift
return np.column_stack([
u[:, 0], u[:, 1], u[:, 2],
np.sin(1.5 * u[:, 0]), np.cos(1.5 * u[:, 1]), np.tanh(u[:, 2]),
u[:, 0] * u[:, 1], u[:, 1] * u[:, 2], u[:, 0] ** 2, u[:, 2] ** 2,
np.sin(u[:, 0] + u[:, 1]), np.cos(u[:, 1] - u[:, 2]),
np.tanh(u[:, 0] * u[:, 2]), u[:, 1] ** 2, np.sin(2 * u[:, 2]),
u[:, 0] - u[:, 2], u[:, 0] + u[:, 1] + u[:, 2], np.exp(u[:, 1] / 2),
np.abs(u[:, 0]), u[:, 2] ** 3 / 3])
Xin = embed(rng.normal(0, 1, (n_in, k_true)))
# Anomalies with the SAME per-column distributions and NO joint structure:
# shuffle each column independently, which destroys every relationship while
# leaving every marginal untouched. This is the case a per-column rule cannot see.
base = embed(rng.normal(0, 1, (n_out, k_true)))
Xout = np.column_stack([base[rng.permutation(n_out), j]
for j in range(base.shape[1])])
X = np.vstack([Xin, Xout])
y = np.r_[np.zeros(n_in), np.ones(n_out)].astype(int)
sc = StandardScaler().fit(Xin) # fit the scaler on NORMAL data only
Xs, Xins = sc.transform(X), sc.transform(Xin)
print(f"{len(X)} rows, {X.shape[1]} columns, {y.mean():.1%} anomalies")
print(" Normal data lies on a CURVED 3-dimensional manifold in 20 dimensions.")
print(" The anomalies have identical per-column distributions and no joint")
print(" structure at all.")
zmax = np.abs(Xs[y == 1]).max(1)
print(f"\n per-column check: median max|z| over 20 columns = {np.median(zmax):.2f},")
print(f" and {np.mean(zmax < 3):.0%} of anomalies have EVERY |z| below 3.")
print(" No univariate rule can find these. They are anomalous only jointly.")
# --- PCA RECONSTRUCTION ERROR: THE LINEAR VERSION. ---
print("\nPCA RECONSTRUCTION ERROR (fitted on normal data only)")
print(f" {'components':>11s} {'variance kept':>14s} {'PR-AUC':>8s} {'ROC-AUC':>8s}")
for kk in [2, 3, 5, 8, 12, 18]:
p = PCA(n_components=kk).fit(Xins)
err = ((Xs - p.inverse_transform(p.transform(Xs))) ** 2).sum(1)
print(f" {kk:11d} {p.explained_variance_ratio_.sum():14.3f}"
f" {average_precision_score(y, err):8.3f} {roc_auc_score(y, err):8.3f}")
print(" At the TRUE intrinsic dimension of 3, PCA scores 0.044 -- barely above")
print(" the 2% base rate. It only works at 18 of 20 components. The reason is")
print(" curvature: a flat 3-D subspace does not contain a curved 3-D surface, so")
print(" normal points reconstruct badly too and the score cannot discriminate.")
# --- THE AUTOENCODER VERSION: THE SAME IDEA, NON-LINEAR. ---
class AE(nn.Module):
def __init__(self, p, k, h=96):
super().__init__()
self.enc = nn.Sequential(nn.Linear(p, h), nn.ReLU(),
nn.Linear(h, h), nn.ReLU(), nn.Linear(h, k))
self.dec = nn.Sequential(nn.Linear(k, h), nn.ReLU(),
nn.Linear(h, h), nn.ReLU(), nn.Linear(h, p))
def forward(self, x):
return self.dec(self.enc(x))
def train_ae(Xtrain, k, epochs=1500, lr=3e-3):
m = AE(Xtrain.shape[1], k)
opt = torch.optim.Adam(m.parameters(), lr=lr)
T = torch.tensor(Xtrain, dtype=torch.float32)
for _ in range(epochs):
opt.zero_grad()
((m(T) - T) ** 2).mean().backward()
opt.step()
return m
def ae_error(m, Xa):
with torch.no_grad():
T = torch.tensor(Xa, dtype=torch.float32)
return ((m(T) - T) ** 2).sum(1).numpy()
print("\nAUTOENCODER RECONSTRUCTION ERROR (trained on normal data only)")
print(f" {'bottleneck':>11s} {'PR-AUC':>8s} {'ROC-AUC':>8s}")
for kk in [2, 3, 5, 8]:
err = ae_error(train_ae(Xins, kk), Xs)
print(f" {kk:11d} {average_precision_score(y, err):8.3f}"
f" {roc_auc_score(y, err):8.3f}")
print(" At a bottleneck of 3 -- the true intrinsic dimension -- the autoencoder")
print(" reaches a PR-AUC of 1.000 where PCA managed 0.044. It can follow the")
print(" curve, so normal points reconstruct almost perfectly and anything off")
print(" the manifold does not. This is the clearest case in the chapter for")
print(" preferring a non-linear model.")
# --- THE HEADLINE COMPARISON. ---
ae3 = train_ae(Xins, k_true)
ae_err = ae_error(ae3, Xs)
pca3 = PCA(n_components=k_true).fit(Xins)
pca_err = ((Xs - pca3.inverse_transform(pca3.transform(Xs))) ** 2).sum(1)
pca18 = PCA(n_components=18).fit(Xins)
pca18_err = ((Xs - pca18.inverse_transform(pca18.transform(Xs))) ** 2).sum(1)
iso = -IsolationForest(n_estimators=300, random_state=0).fit(Xins).score_samples(Xs)
print(f"\n{'method':38s} {'PR-AUC':>8s} {'ROC-AUC':>8s} {'prec@100':>9s}")
for label, s_ in [("PCA reconstruction, 3 components", pca_err),
("PCA reconstruction, 18 components", pca18_err),
("Autoencoder reconstruction, 3", ae_err),
("IsolationForest (for reference)", iso)]:
top = np.argsort(s_)[::-1][:int(y.sum())]
print(f" {label:36s} {average_precision_score(y, s_):8.3f}"
f" {roc_auc_score(y, s_):8.3f} {y[top].mean():9.3f}")
# --- CONTAMINATION: IT MATTERS ONLY IF THE MODEL CAN LEARN THE ANOMALIES. ---
# Case 1: the anomalies are unstructured (shuffled columns), so they fill 20
# dimensions. A 3-unit bottleneck cannot represent them even if it trains on them.
print("\nCONTAMINATION, CASE 1: UNSTRUCTURED ANOMALIES")
print(f" {'contamination':>14s} {'PR-AUC':>8s}")
for frac in [0.0, 0.05, 0.30]:
nadd = int(frac * n_in)
if nadd:
b = embed(rng.normal(0, 1, (nadd, k_true)))
extra = np.column_stack([b[rng.permutation(nadd), j]
for j in range(b.shape[1])])
Xtrain = np.vstack([Xin, extra])
else:
Xtrain = Xin
s2 = StandardScaler().fit(Xtrain)
e = ae_error(train_ae(s2.transform(Xtrain), k_true), s2.transform(X))
print(f" {frac:14.2f} {average_precision_score(y, e):8.3f}")
print(" Barely affected, even at 30%. The bottleneck is the protection: three")
print(" units cannot encode 20-dimensional noise, so the model cannot learn to")
print(" reconstruct these anomalies however many it sees.")
# Case 2: the anomalies are STRUCTURED -- a second, shifted manifold of the same
# low dimension. Now the bottleneck CAN represent them, and contamination bites.
print("\nCONTAMINATION, CASE 2: STRUCTURED ANOMALIES (a second manifold)")
Xout2 = embed(rng.normal(0, 1, (n_out, k_true)), shift=3.0)
X2 = np.vstack([Xin, Xout2])
print(f" {'contamination':>14s} {'PR-AUC':>8s}")
for frac in [0.0, 0.01, 0.05, 0.20]:
nadd = int(frac * n_in)
Xtrain = (np.vstack([Xin, embed(rng.normal(0, 1, (nadd, k_true)), shift=3.0)])
if nadd else Xin)
s2 = StandardScaler().fit(Xtrain)
e = ae_error(train_ae(s2.transform(Xtrain), k_true), s2.transform(X2))
print(f" {frac:14.2f} {average_precision_score(y, e):8.3f}")
print(" From 0.976 down to 0.040 -- twice the base rate and no more. These")
print(" anomalies form their own 3-D manifold, so a 3-unit bottleneck learns it")
print(" happily and then reconstructs them as well as it reconstructs normal")
print(" points. At 20% contamination they are no longer anomalous TO THE MODEL.")
print("\n The rule is sharper than 'train on clean data': contamination hurts in")
print(" proportion to how LEARNABLE the anomalies are within your bottleneck.")
print(" Scattered noise is self-protecting; fraud rings and coordinated abuse are")
print(" exactly the structured case, and there a clean reference set is essential.")
# --- THE OPERATIONAL PAYOFF: THE ERROR IS ATTRIBUTABLE PER FEATURE. ---
with torch.no_grad():
T = torch.tensor(Xs, dtype=torch.float32)
resid = ((ae3(T) - T) ** 2).numpy()
print("\nRECONSTRUCTION ERROR IS ATTRIBUTABLE; A TREE SCORE IS NOT")
for i in np.argsort(ae_err)[::-1][:3]:
cols = np.argsort(resid[i])[::-1][:4]
print(f" row {i:4d} score {ae_err[i]:8.1f} truly anomalous {bool(y[i])!s:5s}"
f" driven by columns {cols.tolist()}")
share = np.sort(resid[y == 1], axis=1)[:, -3:].sum(1) / resid[y == 1].sum(1)
print(f"\n for the true anomalies, the top 3 of 20 columns carry"
f" {share.mean():.0%} of the error")
print(" So the alert arrives with a reason attached: 'these three fields are")
print(" inconsistent with the rest of the record'. Isolation Forest returns one")
print(" opaque number. For anything a human has to investigate, attributability")
print(" is often worth more than a slightly better PR-AUC.")
THE IDEA
fit a compression model on NORMAL data
|
x -> encode -> bottleneck -> decode -> x_hat
|
score = || x - x_hat ||^2
on the manifold -> reconstructs well -> low score
off it -> reconstructs badly -> high score
THE DATA: JOINT ANOMALIES, NORMAL MARGINALS
5100 rows, 20 columns, 2.0% anomalies
normal data on a CURVED 3-D manifold; anomalies have each column
independently shuffled, so every marginal is unchanged.
median max|z| over 20 columns for the anomalies: 2.05
86% of anomalies have EVERY |z| below 3.
PCA RECONSTRUCTION: DEFEATED BY CURVATURE
components variance kept PR-AUC ROC-AUC
2 0.348 0.038 0.768
3 0.471 0.044 0.802 <- the TRUE dimension
5 0.657 0.064 0.865
8 0.834 0.110 0.924
12 0.970 0.364 0.985
18 1.000 1.000 1.000
a flat subspace through a curved surface:
.-''-. the manifold
/ \
--------------- a flat 3-D subspace: normal points are
\ / ALSO far from it, so the residual does
`-..-' not separate anything
AUTOENCODER RECONSTRUCTION: FOLLOWS THE CURVE
bottleneck PR-AUC ROC-AUC
2 0.888 0.998
3 1.000 1.000 <- the TRUE dimension
5 1.000 1.000
8 1.000 1.000
THE HEADLINE
method PR-AUC ROC-AUC prec@100
PCA reconstruction, 3 components 0.044 0.802 0.000
PCA reconstruction, 18 components 1.000 1.000 1.000
Autoencoder reconstruction, 3 1.000 1.000 1.000
IsolationForest (for reference) 0.029 0.696 0.000
CONTAMINATION: IT DEPENDS ON WHETHER THE MODEL CAN LEARN THEM
CASE 1, unstructured anomalies (20-D shuffled noise)
contamination PR-AUC
0.00 1.000
0.05 1.000
0.30 1.000 <- barely affected
CASE 2, structured anomalies (a second 3-D manifold)
contamination PR-AUC
0.00 0.976
0.01 0.678
0.05 0.495
0.20 0.040 <- destroyed
a 3-unit bottleneck cannot encode 20-D noise however much it sees,
but it learns a second 3-D manifold happily. contamination hurts in
proportion to how LEARNABLE the anomalies are within your bottleneck.
THE ERROR IS ATTRIBUTABLE, WHICH A TREE SCORE IS NOT
row 5097 score 154.0 anomalous True driven by columns [6, 8, 13, 4]
row 5096 score 71.9 anomalous True driven by columns [8, 18, 3, 6]
row 5064 score 61.6 anomalous True driven by columns [8, 18, 10, 15]
the top 3 of 20 columns carry 61% of each anomaly's error, so the
alert ships with a reason: "these fields are inconsistent with the
rest of the record".
| PCA reconstruction | Autoencoder | Isolation Forest | |
|---|---|---|---|
| Captures curvature | No | Yes | Not as a manifold |
| Needs clean training data | Yes | Yes | No |
| Attributable per feature | Yes | Yes | No |
| Hyperparameters | n_components | Architecture, bottleneck, training | Almost none |
| Deterministic | Yes | No | With a fixed seed |
| Cost | Instant | A training loop | Fast |
Key Takeaways
- Fit a compression model on normal data and use reconstruction error as the anomaly score; no anomaly labels are needed.
- This is the method for anomalies that are normal in every column and abnormal only in the relationships between columns.
- PCA reconstruction is defeated by curvature: at the true intrinsic dimension a flat subspace leaves normal points with large residuals too.
- An autoencoder at the true intrinsic dimension follows the curve and separates perfectly where PCA does not.
- Contamination hurts in proportion to how learnable the anomalies are within the bottleneck, not simply in proportion to how much there is.
- Unstructured anomalies are self-protecting, because a tight bottleneck cannot encode them; structured anomalies are learned and then become invisible.
- Reconstruction error decomposes per feature, so alerts carry a reason, which Isolation Forest's single score cannot provide.
- The number of components or bottleneck units is an explicit assumption about how many dimensions "normal" occupies.
🧪 Practice
- Generate data on a curved low-dimensional manifold, create anomalies by shuffling each column independently, and confirm the marginals are unchanged.
- Compute PCA reconstruction error for components from 2 to
pand plot PR-AUC against the number of components.- Train an autoencoder at several bottleneck sizes and compare its PR-AUC curve with PCA's.
- Contaminate the training set with unstructured anomalies and then with structured ones, and show the two behave completely differently.
- Decompose the reconstruction error by feature for your top ten alerts and report what share the top three columns carry.
- Compare a reconstruction detector against Isolation Forest on data whose anomalies violate only the correlation structure.
- Interview: Your autoencoder-based detector worked well at launch and its alert volume has quietly fallen to near zero over six months of weekly retraining. What do you suspect? (Hint: what is in each week's training data, and what has the model been slowly learning to reconstruct?)
Evaluating Anomaly Detectors
Everything in this section produces a score. Deciding which score is better is genuinely harder than producing one, for two reasons that compound: anomalies are rare, which breaks the usual metrics, and you normally have no labels, which breaks the usual procedure.
Start with what rarity does. At a 0.2% base rate, a detector that flags nothing achieves 99.8% accuracy. That much is widely known.
Less widely appreciated is that ROC-AUC is nearly as misleading. On the demonstration's data, Isolation Forest reaches ROC-AUC 0.9634 -- which reads as an excellent model -- while PR-AUC is 0.5480. The discrepancy is entirely in the denominators. The false positive rate divides by the number of negatives, which is enormous, so it stays tiny while the absolute count of false alarms is large:
- to catch 52% of anomalies: FPR 0.0007, meaning 14 false alarms for 21 real ones
- to catch 80%: FPR 0.0222, meaning 445 false alarms for 32 real ones
- to catch 90%: FPR 0.0807, meaning 1,613 false alarms for 36 real ones
An FPR of 0.02 sounds negligible and is 445 alerts out of 20,000 rows. With 40 anomalies in existence, the queue is 93% noise. Precision divides by the number of alerts, which is what the analyst experiences, so PR-AUC is the metric to report under imbalance. Its baseline is the base rate itself: 0.002 here, against which 0.548 is a real achievement and 0.9634 was flattery.
That also makes ROC-AUC the worst choice for comparing candidates. In the demonstration the ROC-AUC column spans 0.9634 to 0.9809 while PR-AUC spans 0.5480 to 0.7522 -- the same ranking, but ROC-AUC compresses a 37% relative difference into a 2% one.
Precision@k is the metric that matches operational reality. Nobody reviews
20,000 rows; a team reviews the top k, and k is a staffing decision. The
demonstration's table -- precision 1.000 at k=10, 0.525 at k=40, 0.075 at
k=400 -- is the conversation to have with whoever will work the alerts: "we
can send you 40 a day and about half will be real, or 100 a day and about a
quarter". No single number communicates that.
Now the harder problem: you usually have no labels at all, so none of the above is computable. Four things are still available.
1. Agreement between independent detectors. Not correctness, but disagreement is a reason to look harder. And read the top-k overlap, not the rank correlation. In the demonstration all pairwise rank correlations exceed 0.93, which suggests broad consensus -- while the top-40 overlap reveals that the three distance- and density-based methods agree with each other at 0.90 to 0.93 and agree with Isolation Forest at only 0.62 to 0.68. Roughly a third of the forest's top 40 appears in nobody else's list. The methods cluster by inductive bias, "the anomalies" is not a well-defined set, and the rank correlations hide that completely.
2. Synthetic anomaly injection. Inject anomalies of a type you care about and measure recall on those. It validates the detector against your definition, which is the only one available. Recall@60 in the demonstration goes 0.300, 0.867, 1.000 for anomalies at 1.5x, 2.5x, and 4.0x the normal spread -- which tells you roughly how extreme a deviation must be before you will see it. Note the honest limitation: this measures performance on the anomalies you thought to simulate.
3. Stability under resampling. Refit on bootstrap samples and check the top-k overlap; the demonstration gets 0.48 across eight refits. A detector whose top-40 changes substantially between refits is not reporting a stable property of the data.
4. Label the top k. This is the only thing that actually settles it.
Send the highest-scoring alerts to a human, record the verdicts, and you have
precision@k for real -- plus a labelled set that grows and eventually supports
a supervised model. Every mature anomaly system ends up here, because
unsupervised evaluation cannot close the loop on its own.
One last practical point: anomaly detection is a ranking problem wearing a
classification costume. The threshold is a business decision about alert
capacity and the cost of a miss, exactly as in chapter 8.4, and it should be
set from those costs rather than from a contamination default.
import numpy as np
import pandas as pd
from sklearn.ensemble import IsolationForest
from sklearn.neighbors import LocalOutlierFactor
from sklearn.covariance import MinCovDet
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import (roc_auc_score, average_precision_score,
precision_recall_curve, roc_curve)
rng = np.random.default_rng(211)
# --- THE SETUP: EXTREME IMBALANCE, WHICH IS THE NORMAL CASE. ---
n_in, n_out, p = 20_000, 40, 15 # 0.2% anomalies
Xin = rng.normal(0, 1, (n_in, p))
Xin[:, 0] += rng.integers(0, 3, n_in) * 3.0
Xout = rng.normal(0, 1.9, (n_out, p))
Xout[:, 0] += rng.integers(0, 3, n_out) * 3.0
X = StandardScaler().fit_transform(np.vstack([Xin, Xout]))
y = np.r_[np.zeros(n_in), np.ones(n_out)].astype(int)
print(f"{len(X):,} rows, {y.sum()} anomalies, base rate {y.mean():.3%}")
# --- WHY ACCURACY IS USELESS, AND WHY ROC-AUC IS NEARLY SO. ---
scores = -IsolationForest(n_estimators=300, random_state=0).fit(X).score_samples(X)
print("\nA DETECTOR THAT FLAGS NOTHING")
print(f" accuracy of 'always normal': {1 - y.mean():.3%}")
print(f" it finds 0 of {y.sum()} anomalies.")
print(" Accuracy is unusable below a few percent prevalence. That part is well")
print(" known. The next part is not.")
print("\nROC-AUC LOOKS EXCELLENT WHILE THE ALERT QUEUE IS USELESS")
print(f" ROC-AUC {roc_auc_score(y, scores):.4f}")
print(f" PR-AUC {average_precision_score(y, scores):.4f}")
print(f" base rate (a random detector's PR-AUC) {y.mean():.4f}")
print("\n The reason is the false positive RATE's denominator:")
fpr, tpr, _ = roc_curve(y, scores)
for target in [0.5, 0.8, 0.9]:
i = int(np.searchsorted(tpr, target))
i = min(i, len(fpr) - 1)
n_fp = int(fpr[i] * (len(y) - y.sum()))
tp = int(tpr[i] * y.sum())
prec = tp / max(tp + n_fp, 1)
print(f" to catch {tpr[i]:.0%} of anomalies: FPR {fpr[i]:.4f}"
f" -> {n_fp:5d} false alarms for {tp:2d} real ones"
f" (precision {prec:.3f})")
print(" An FPR of 0.02 sounds negligible and means 400 false alarms out of")
print(" 20,000 normal rows. With 40 anomalies in total, the queue is 90% noise.")
print(" ROC-AUC divides false positives by the huge negative count, so it stays")
print(" flattering. Precision divides by the number of ALERTS, which is what the")
print(" analyst experiences. Report PR-AUC under imbalance.")
# --- PRECISION@k IS THE METRIC THAT MATCHES THE OPERATIONAL REALITY. ---
# Nobody reviews 20,000 rows. They review the top k, and k is a staffing decision.
print("\nPRECISION@k: THE METRIC THAT MATCHES AN ALERT QUEUE")
order = np.argsort(scores)[::-1]
print(f" {'k':>6s} {'precision@k':>12s} {'recall@k':>9s} {'anomalies found':>16s}")
for k in [10, 20, 40, 100, 400, 1000]:
top = order[:k]
print(f" {k:6d} {y[top].mean():12.3f} {y[top].sum() / y.sum():9.3f}"
f" {int(y[top].sum()):16d}")
print(" This table is the conversation to have with the team that will work the")
print(" alerts: 'we can send you 40 a day and about half will be real, or 100 a")
print(" day and about a quarter'. No")
print(" single-number metric communicates that.")
# --- COMPARING DETECTORS: THE RANKING DEPENDS ON THE METRIC. ---
print("\nTHE CHOICE OF METRIC CHANGES THE RANKING")
print(f" {'method':30s} {'ROC-AUC':>8s} {'PR-AUC':>8s} {'prec@40':>8s}"
f" {'rec@400':>8s}")
detectors = {
"IsolationForest": scores,
"LocalOutlierFactor (k=20)":
-LocalOutlierFactor(n_neighbors=20).fit(X).negative_outlier_factor_,
"LocalOutlierFactor (k=100)":
-LocalOutlierFactor(n_neighbors=100).fit(X).negative_outlier_factor_,
"Mahalanobis + MinCovDet": MinCovDet(random_state=0).fit(X).mahalanobis(X),
}
for label, sc in detectors.items():
o = np.argsort(sc)[::-1]
print(f" {label:30s} {roc_auc_score(y, sc):8.4f}"
f" {average_precision_score(y, sc):8.4f} {y[o[:40]].mean():8.3f}"
f" {y[o[:400]].sum() / y.sum():8.3f}")
print(" Note how much tighter the ROC-AUC column is than the PR-AUC column. Under")
print(" imbalance ROC-AUC compresses real differences, so it is the worst metric")
print(" for choosing between candidates.")
# --- WITHOUT LABELS: WHAT YOU CAN AND CANNOT DO. ---
print("\nTHE USUAL SITUATION: NO LABELS AT ALL")
print(" You cannot compute any of the above. What you CAN do:")
print("\n 1. AGREEMENT between independent detectors. Not correctness, but")
print(" disagreement is a reason to look harder.")
ranks = {k: np.argsort(np.argsort(-v)) for k, v in detectors.items()}
names = list(detectors)
print(f" {'pair':58s} {'rank corr':>10s} {'top-40 overlap':>15s}")
from scipy.stats import spearmanr
for i in range(len(names)):
for j in range(i + 1, len(names)):
a, b = detectors[names[i]], detectors[names[j]]
ta = set(np.argsort(a)[::-1][:40])
tb = set(np.argsort(b)[::-1][:40])
print(f" {names[i] + ' vs ' + names[j]:58s}"
f" {spearmanr(a, b).statistic:10.3f} {len(ta & tb) / 40:15.2f}")
print(" Read the top-40 overlap column. The three distance- and density-based")
print(" methods agree with each other at 0.90 to 0.93, and all three agree")
print(" with the FOREST at only 0.62 to 0.68. The methods cluster by their")
print(" inductive bias, and roughly a third of the forest's top 40 appears in")
print(" nobody else's. 'The anomalies' is not a well-defined set, and the rank")
print(" correlations (all above 0.93) hide that completely -- always compare")
print(" the top-k sets you would actually act on, not the full ranking.")
print("\n 2. A SYNTHETIC-ANOMALY sanity check. Inject known anomalies of a type")
print(" you care about and measure recall on those. It validates the")
print(" detector against YOUR definition, which is the only one available.")
for scale in [1.5, 2.5, 4.0]:
inj = rng.normal(0, scale, (60, p))
inj[:, 0] += rng.integers(0, 3, 60) * 3.0
Xa = StandardScaler().fit_transform(np.vstack([Xin, inj]))
ya = np.r_[np.zeros(n_in), np.ones(60)].astype(int)
s_ = -IsolationForest(n_estimators=300, random_state=0).fit(Xa).score_samples(Xa)
o = np.argsort(s_)[::-1][:60]
print(f" injected anomalies at {scale}x the normal spread:"
f" recall@60 = {ya[o].mean():.3f}")
print(" Recall degrades smoothly as anomalies get subtler, which tells you")
print(" roughly how extreme a deviation must be before you will see it.")
print("\n 3. STABILITY. Refit on bootstrap samples and check the top-k overlap.")
tops = []
for s_ in range(8):
idx = rng.choice(len(X), len(X), replace=True)
m = IsolationForest(n_estimators=200, random_state=s_).fit(X[idx])
tops.append(set(np.argsort(-m.score_samples(X))[:40]))
ov = [len(tops[i] & tops[j]) / 40 for i in range(len(tops))
for j in range(i + 1, len(tops))]
print(f" mean pairwise top-40 overlap across 8 bootstrap refits:"
f" {np.mean(ov):.2f}")
print(" A detector whose top-40 changes completely between refits is not")
print(" reporting a property of the data.")
print("\n 4. The only thing that actually settles it: LABEL THE TOP k. Send the")
print(" highest-scoring alerts to a human, record the verdicts, and you now")
print(" have precision@k for real -- plus a labelled set that grows over time")
print(" and eventually supports a supervised model. Every mature anomaly")
print(" system ends up here, because unsupervised evaluation cannot close.")
20,040 rows, 40 anomalies, base rate 0.200%
ACCURACY IS USELESS
accuracy of "always normal": 99.800%, finding 0 of 40 anomalies
ROC-AUC IS NEARLY AS MISLEADING
ROC-AUC 0.9634 <- reads as an excellent model
PR-AUC 0.5480
PR-AUC of a random detector = the base rate = 0.0020
the false positive RATE hides the false positive COUNT:
recall FPR false alarms real ones precision
52% 0.0007 14 21 0.600
80% 0.0222 445 32 0.067
90% 0.0807 1,613 36 0.022
FPR = FP / (all negatives) <- a huge denominator, stays tiny
precision = TP / (all alerts) <- what the analyst experiences
PRECISION@k MATCHES AN ALERT QUEUE
k precision@k recall@k anomalies found
10 1.000 0.250 10
20 0.850 0.425 17
40 0.525 0.525 21
100 0.270 0.675 27
400 0.075 0.750 30
1000 0.033 0.825 33
ROC-AUC COMPRESSES REAL DIFFERENCES
method ROC-AUC PR-AUC prec@40 rec@400
IsolationForest 0.9634 0.5480 0.525 0.750
LocalOutlierFactor (k=20) 0.9777 0.7475 0.700 0.825
LocalOutlierFactor (k=100) 0.9809 0.7522 0.725 0.875
Mahalanobis + MinCovDet 0.9791 0.7477 0.700 0.875
^^^^^^ ^^^^^^
spans 2% spans 37% relative
WITH NO LABELS: FOUR THINGS YOU CAN STILL DO
1. AGREEMENT -- and read the TOP-k OVERLAP, not the rank correlation
pair rank corr top-40 overlap
IsolationForest vs LOF (k=20) 0.936 0.62
IsolationForest vs LOF (k=100) 0.964 0.68
IsolationForest vs Mahalanobis+MCD 0.970 0.62
LOF (k=20) vs LOF (k=100) 0.973 0.90
LOF (k=20) vs Mahalanobis+MCD 0.966 0.90
LOF (k=100) vs Mahalanobis+MCD 0.994 0.93
every rank correlation is above 0.93, suggesting consensus. the
top-40 overlaps say the distance/density methods form one camp and
the forest is a different one.
2. SYNTHETIC INJECTION -- validate against YOUR definition
anomalies at 1.5x the normal spread: recall@60 = 0.300
anomalies at 2.5x the normal spread: recall@60 = 0.867
anomalies at 4.0x the normal spread: recall@60 = 1.000
3. STABILITY -- mean top-40 overlap across 8 bootstrap refits: 0.48
4. LABEL THE TOP k -- the only thing that closes the loop, and the
start of the labelled set that eventually replaces this whole
section with a supervised model.
| Metric | Baseline | Robust to imbalance | What it tells you |
|---|---|---|---|
| Accuracy | 1 - base rate | No | Nothing useful below a few percent prevalence |
| ROC-AUC | 0.5 | Barely | Ranking quality, but flattering and compressed |
| PR-AUC | The base rate | Yes | Ranking quality as the alert queue experiences it |
| Precision@k | The base rate | Yes | What fraction of k alerts are real |
| Recall@k | 0 | Yes | What fraction of anomalies k alerts catch |
Key Takeaways
- At low prevalence accuracy is meaningless, and ROC-AUC is nearly as misleading because the false positive rate has an enormous denominator.
- The same FPR that looks negligible can mean hundreds of false alarms per handful of real anomalies.
- PR-AUC is the metric to report under imbalance, and its baseline is the base rate rather than 0.5.
- ROC-AUC compresses genuine differences between candidates, making it the worst choice for model selection here.
- Precision@k and recall@k match how alerts are actually consumed and make the staffing trade-off explicit.
- Without labels, compare the top-k sets you would act on rather than rank correlations, which hide substantial disagreement.
- Synthetic anomaly injection validates a detector against your own definition of anomalous, which is its strength and its limit.
- Stability under resampling and labelling the top
kare the remaining honest checks, and only labelling closes the loop.- The threshold is a business decision about alert capacity and the cost of a miss, not a default parameter.
🧪 Practice
- On a dataset with a 0.5% base rate, compute accuracy, ROC-AUC, and PR-AUC for the same detector and explain the gap between the last two.
- From the ROC curve, convert FPR into an absolute count of false alarms at 50%, 80%, and 95% recall.
- Produce a precision@k and recall@k table for
kfrom 10 to 1000 and write the one-sentence summary you would give an operations team.- Compare four detectors by ROC-AUC and by PR-AUC and report the relative spread of each column.
- Compute pairwise rank correlations and top-50 overlaps between three detectors, and describe where the two measures disagree.
- Inject synthetic anomalies at three severities and report recall at a fixed
kfor each.- Bootstrap-refit a detector ten times and report the mean pairwise overlap of the top 100.
- Interview: Your anomaly detector has ROC-AUC 0.97 and the fraud team says the alerts are useless. Reconcile those two statements, and say what you would measure instead. (Hint: work out how many false alarms an FPR of 0.03 produces on a million transactions with 500 genuine frauds.)
<a id="12-deep-learning"></a>