16. Interpretability, Ethics, and Governance
A model that cannot be explained cannot be argued with, a model that cannot be audited cannot be trusted, and a model that handles personal data without a lawful basis cannot be deployed at all. This chapter covers the four disciplines that decide whether a technically correct model is allowed to influence real decisions: making predictions explainable, measuring and constraining unequal treatment, protecting the people in the data, and building the organisational machinery that keeps all of it true over time. The recurring theme is that none of these are add-ons applied after modelling -- each one changes what you are permitted to build, and each one is cheapest to satisfy before the first model is trained.
<a id="161-model-interpretability"></a>
16.1 Model Interpretability
Interpretability is the ability to answer "why did the model produce that number?" in terms a human can check; this section covers the choice between models that are readable by construction and explanations bolted on afterwards, the difference between describing a model overall and describing one prediction, the four workhorse post-hoc methods (permutation importance, partial dependence, SHAP, LIME), explanations phrased as actionable changes, and the failure modes that make every method above misleading if used without care.
Most examples in this section share one setup: a loan-default model on synthetic data, deliberately containing a nonlinear interaction and a pair of nearly duplicate features, because those two properties are what break naive explanations.
import numpy as np
def make_data(n=6000, seed=0):
rng = np.random.default_rng(seed)
income = rng.lognormal(10.6, 0.45, n)
debt_ratio = np.clip(rng.beta(2, 5, n) * 1.4, 0, 1.2)
age = rng.integers(21, 70, n).astype(float)
late = rng.poisson(0.8, n).astype(float)
verified = income * rng.normal(1.0, 0.05, n) # near-duplicate of income
z = (-1.9 - 1.0*np.log(income/40000) + 2.6*debt_ratio + 0.45*late
+ 1.3*(age < 25) # a threshold effect
- 2.8*np.log(income/40000)*debt_ratio) # and an interaction
y = (rng.random(n) < 1/(1+np.exp(-z))).astype(int) # 1 = defaulted
X = np.column_stack([np.log(income), debt_ratio, age, late,
np.log(verified)])
return X, y, ["log_income", "debt_ratio", "age", "late_pmts",
"log_verified"]
def auc(y, s): # rank-based AUC, no sklearn needed
o = np.argsort(s); r = np.empty(len(s)); r[o] = np.arange(1, len(s)+1)
n1 = y.sum(); n0 = len(y) - n1
return (r[y == 1].sum() - n1*(n1+1)/2) / (n0*n1)
def logistic(X, y, iters=3000, lr=0.5):
Xs = (X - X.mean(0)) / X.std(0) # coefficients comparable
w = np.zeros(X.shape[1]); b = 0.0
for _ in range(iters):
p = 1/(1+np.exp(-(Xs @ w + b)))
g = (p - y) / len(y)
w -= lr * (Xs.T @ g); b -= lr * g.sum()
return w, b, X.mean(0), X.std(0)
A minimal gradient-boosted model plays the role of "the black box" -- 60 depth-3 regression trees fitted to the residuals, which is enough structure that no human can read it directly.
class Tree: # CART on a squared-error split
def __init__(self, depth=3, min_leaf=20):
self.depth, self.min_leaf = depth, min_leaf
def fit(self, X, g):
self.node = self._build(X, g, 0); return self
def _build(self, X, g, d):
if d >= self.depth or len(g) < 2*self.min_leaf:
return ("leaf", g.mean())
best = None
for j in range(X.shape[1]):
for t in np.unique(np.quantile(X[:, j], np.linspace(.1, .9, 9))):
m = X[:, j] <= t
if min(m.sum(), (~m).sum()) < self.min_leaf: continue
sse = g[m].var()*m.sum() + g[~m].var()*(~m).sum()
if best is None or sse < best[0]: best = (sse, j, t, m)
if best is None: return ("leaf", g.mean())
_, j, t, m = best
return ("split", j, t, self._build(X[m], g[m], d+1),
self._build(X[~m], g[~m], d+1))
def predict(self, X):
out = np.empty(len(X))
for i, x in enumerate(X):
nd = self.node
while nd[0] == "split":
nd = nd[3] if x[nd[1]] <= nd[2] else nd[4]
out[i] = nd[1]
return out
class GBM:
def __init__(self, n=60, lr=0.2, depth=3):
self.n, self.lr, self.depth = n, lr, depth
def fit(self, X, y):
self.base = np.log(y.mean()/(1-y.mean())); self.trees = []
f = np.full(len(y), self.base)
for _ in range(self.n):
p = 1/(1+np.exp(-f))
t = Tree(self.depth).fit(X, y - p) # fit the gradient
f += self.lr * t.predict(X); self.trees.append(t)
return self
def predict_proba(self, X):
f = np.full(len(X), self.base)
for t in self.trees: f += self.lr * t.predict(X)
return 1/(1+np.exp(-f))
Intrinsic vs Post-Hoc Interpretability
There are exactly two ways to be able to explain a prediction. Either the model is simple enough that its mechanism is the explanation, or the model is not, and you build a second, separate artifact whose job is to describe it. The first is intrinsic interpretability; the second is post-hoc.
The distinction matters because the second option carries a hidden cost that
the first does not. An intrinsic explanation is the model: if a logistic
regression says the coefficient on debt_ratio is +0.53, then the model
genuinely adds 0.53 log-odds per standard deviation of debt ratio, in every
case, forever. A post-hoc explanation is an approximation of a different
object, and approximations have error. When the explanation and the model
disagree, the model is what runs in production and the explanation is what
you showed the regulator.
The analogy is the difference between a recipe and a restaurant review. A recipe is the dish's construction -- follow it and you reproduce the dish exactly. A review describes the dish from outside: often useful, sometimes insightful, and occasionally wrong in ways you cannot detect from the review alone.
Interpretability is not a binary property but a rough ordering, and it is not the same thing as simplicity. A linear model with 4,000 features is technically transparent and practically unreadable; a depth-3 tree with five features can be read aloud in a meeting.
MORE READABLE LESS READABLE
| |
scorecard --- linear/logistic --- shallow --- GAM --- random --- deep
(integer (one coefficient tree (one forest/ network
points) per feature) curve boosting
per
feature)
<---------- intrinsic: the model is the explanation ---------->
<-------- post-hoc required -------->
The decision procedure is not "always use the interpretable model". It is: measure what the opacity buys you, then decide whether that is worth the explanation debt.
X, y, names = make_data(); tr, te = slice(0, 4000), slice(4000, None)
w, b, mu, sd = logistic(X[tr], y[tr])
lin = lambda Z: 1/(1+np.exp(-(((Z-mu)/sd) @ w + b)))
tree = Tree(depth=3).fit(X[tr], y[tr].astype(float))
gbm = GBM(n=60, depth=3).fit(X[tr], y[tr])
print("MODEL FAMILY vs WHAT YOU CAN SAY ABOUT IT")
print(f"{'model':22}{'test AUC':>10}{'params to read':>16}")
for nm, s, k in [("logistic regression", lin(X[te]), "6 numbers"),
("depth-3 tree", tree.predict(X[te]), "8 rules"),
("60-tree boosted", gbm.predict_proba(X[te]), "~480 splits")]:
print(f"{nm:22}{auc(y[te], s):10.3f}{k:>16}")
# THE CLASSIC POST-HOC SHORTCUT: FIT AN INTERPRETABLE MODEL TO THE BLACK
# BOX'S OUTPUT, NOT TO THE LABELS. THE QUESTION IS ALWAYS: HOW FAITHFUL?
sur = Tree(depth=3).fit(X[tr], gbm.predict_proba(X[tr]))
pg, ps = gbm.predict_proba(X[te]), sur.predict(X[te])
r2 = 1 - ((pg-ps)**2).sum()/((pg-pg.mean())**2).sum()
print(f"\nsurrogate tree fidelity: R2={r2:.2f}, same decision on "
f"{((pg>0.5) == (ps>0.5)).mean():.1%} of rows")
big = np.abs(pg-ps) > 0.15
print(f"rows where the surrogate is off by >0.15: {big.mean():.1%}"
f" (mean |error| there {np.abs(pg-ps)[big].mean():.2f})")
MODEL FAMILY vs WHAT YOU CAN SAY ABOUT IT
model test AUC params to read
logistic regression 0.770 6 numbers
depth-3 tree 0.738 8 rules
60-tree boosted 0.778 ~480 splits
surrogate tree fidelity: R2=0.80, same decision on 89.8% of rows
rows where the surrogate is off by >0.15: 6.3% (mean |error| there 0.19)
Read the numbers carefully, because they are typical of real tabular problems and they cut against the usual instinct. The boosted ensemble beats the logistic regression by 0.008 AUC. In exchange for that, nobody can state the model's rule, and every explanation you produce will be an inference about a 480-split object.
The surrogate row is the warning. A global surrogate tree reproduces the boosted model's decision on 90% of rows -- which sounds excellent until you invert it. One applicant in ten receives a decision the explanation cannot account for, and those are disproportionately the borderline cases, which are exactly the ones who complain, appeal, or sue.
| Intrinsic | Post-hoc | |
|---|---|---|
| What you explain | The model itself | A model of the model |
| Fidelity | Exact by definition | Measured, always below 1.0 |
| Cost of adding | Chosen up front | Extra pipeline, extra tests |
| Stability | Same answer every time | Depends on sampling, seeds, background data |
| Failure mode | Underfits the problem | Explains confidently and wrongly |
| Regulatory posture | Easiest to defend | Requires a fidelity argument |
The practical rule used in regulated settings: if a decision materially affects a person's rights, money, health, or liberty, start intrinsic and force the black box to justify itself against a measured accuracy gap. Elsewhere, take the accuracy and invest in post-hoc tooling.
Key Takeaways
- Intrinsic interpretability means the model's mechanism is the explanation; post-hoc means you build a second artifact that describes the model.
- Post-hoc explanations are approximations, so they have error, and the model -- not the explanation -- is what runs in production.
- Interpretability is an ordering, not a binary, and it is not the same as having few parameters.
- Always measure the accuracy the opacity buys; on tabular problems it is frequently a fraction of a point.
- A global surrogate's fidelity must be reported as a number; "90% of decisions match" means one in ten explanations is unaccounted for.
- Surrogate error concentrates on borderline cases, which are the cases most likely to be challenged.
- For high-stakes decisions, default to intrinsic and require the complex model to justify the explanation debt.
🧪 Practice
- Fit a logistic regression and a boosted model on the same dataset and report the AUC gap; state whether you would accept the opacity for that gap and why.
- Fit a depth-3 surrogate tree to your black box's predictions and report both R-squared and the fraction of flipped decisions.
- Take the 5% of rows where your surrogate disagrees most with the model and characterise them: are they near the decision threshold, in a sparse region, or a distinct subgroup?
- Rewrite a logistic regression as an integer scorecard (points per band) and measure how much AUC the rounding costs.
- Interview: A stakeholder asks for "an interpretable version" of a production neural network. What do you deliver, and what do you refuse to promise? (Hint: separate a faithful description of the model from a replacement for the model, and think about what number you must attach to whatever you hand over.)
Global vs Local Explanations
Two very different questions get called "explanation", and confusing them produces answers that are technically correct and practically useless.
Global: how does this model behave, in general, across the population? This is what you need to validate a model, document it, brief a risk committee, or notice that it depends on something it should not.
Local: why did this model produce this number for this row? This is what you need to answer an applicant's appeal, debug a surprising prediction, or comply with a rule that requires giving individuals reasons.
The analogy: global explanation is climate, local explanation is weather. Knowing the average August temperature does not tell you whether to carry an umbrella today, and today's rain does not describe the climate. Both are real; neither substitutes for the other.
The trap is that global explanations feel like they answer local questions. "Income is the most important feature in this model" is a statement about the population. For a specific applicant with an unremarkable income and four late payments, income contributed nothing and the reason is the payment history. Handing that applicant the global feature ranking is giving them the wrong reason.
GLOBAL LOCAL
one statement per feature one statement per feature PER ROW
"debt_ratio matters most" "for applicant #8812, late_pmts
pushed the score up by 1.61"
audience: modellers, risk, audience: the affected person,
regulators reviewing the model support agents, the appeals process
built from: coefficients, built from: SHAP, LIME,
permutation importance, PDP counterfactuals, coef x deviation
For a linear model both are available exactly and cheaply, which makes it the clearest place to see the distinction. The global explanation is the coefficient vector. The local explanation is the coefficient multiplied by this row's deviation from the mean -- the contributions sum to the prediction's log-odds, so nothing is hidden.
X, y, names = make_data(); tr, te = slice(0, 4000), slice(4000, None)
w, b, mu, sd = logistic(X[tr], y[tr])
Z = (X - mu) / sd
print("GLOBAL: one number per feature, valid on average")
for n, c in sorted(zip(names, w), key=lambda t: -abs(t[1])):
print(f" {n:14}{c:+.2f} ({'raises' if c > 0 else 'lowers'} log-odds)")
# LOCAL: THE SAME COEFFICIENTS, WEIGHTED BY HOW UNUSUAL THIS ROW IS.
rows = {"A (low income, high debt)": int(np.argmax(Z[te] @ w)),
"B (rejected, but why?)": int(np.argsort(Z[te] @ w)[-40]),
"C (approved)": int(np.argmin(Z[te] @ w))}
print("\nLOCAL: coefficient x this applicant's deviation, in log-odds")
print(f" {'applicant':28}" + "".join(f"{n[:11]:>13}" for n in names))
for label, i in rows.items():
contrib = w * Z[te][i] # sums to the row's log-odds
print(f" {label:28}" + "".join(f"{c:+13.2f}" for c in contrib))
for label, i in rows.items():
top = names[int(np.argmax(np.abs(w * Z[te][i])))]
print(f" top driver for {label:26} -> {top}")
GLOBAL: one number per feature, valid on average
log_income -0.68 (lowers log-odds)
debt_ratio +0.53 (raises log-odds)
late_pmts +0.35 (raises log-odds)
log_verified -0.29 (lowers log-odds)
age -0.22 (lowers log-odds)
LOCAL: coefficient x this applicant's deviation, in log-odds
applicant log_income debt_ratio age late_pmts log_verifie
A (low income, high debt) +1.80 +1.11 +0.26 +0.07 +0.77
B (rejected, but why?) +0.60 -0.12 +0.07 +1.61 +0.30
C (approved) -2.11 -0.21 -0.14 -0.32 -0.85
top driver for A (low income, high debt) -> log_income
top driver for B (rejected, but why?) -> late_pmts
top driver for C (approved) -> log_income
Applicant B is the whole point. Globally, late_pmts is only the third most
important feature. For B it is the dominant reason by a factor of two and a
half, and debt_ratio -- the second most important feature globally --
actually helped B. A reason code generated from the global ranking would
tell B to reduce their debt ratio, which for B is already good.
Two consequences follow for practice. First, adverse action notices, appeal responses, and support scripts must be generated per row, never from a global table. Second, a global explanation can be aggregated up from local ones (average the absolute contributions), but never disaggregated down; the arrow only points one way.
| Question you are asked | Explanation type | Typical tool |
|---|---|---|
| Is this model using something it shouldn't? | Global | Permutation importance, coefficient review |
| What relationship did it learn for income? | Global | PDP, GAM shape function |
| Why was this person declined? | Local | SHAP, coefficient x deviation |
| What would change this person's outcome? | Local | Counterfactual search |
| Does it behave differently for subgroups? | Cohort (global within a slice) | Grouped SHAP, per-slice metrics |
That third row -- cohort explanation -- is the useful middle ground: global explanation computed within a slice. It is how you discover that a model relies on one feature for urban applicants and another for rural ones, which no single global table will reveal.
Key Takeaways
- Global explanations describe the model over the population; local explanations describe one prediction.
- Global importance rankings are the wrong source for individual reason codes, and can name a feature that contributed nothing to that row.
- In a linear model the local explanation is the coefficient times the row's deviation from the mean, and the contributions sum exactly to the prediction.
- Local explanations aggregate into global ones; global explanations cannot be disaggregated into local ones.
- Individual-facing outputs -- adverse action notices, appeals, support scripts -- must be computed per row.
- Cohort explanations (global within a slice) reveal behaviour that whole-population summaries hide.
- Match the explanation type to the question: model validation is global, answering a person is local.
🧪 Practice
- For a linear model of yours, print the global coefficients and the local contributions for three rows, and find a row whose top driver differs from the global top feature.
- Aggregate your local contributions into a global ranking by mean absolute value and compare it with the coefficient ranking; explain any difference.
- Compute global importance separately within two subgroups and report any feature whose rank changes by more than two positions.
- Write the adverse-action template your system would send, and identify exactly which numbers must come from the row rather than the model.
- Interview: A product manager wants a single "top 3 reasons" panel to display for every user, and proposes computing it once from global importance to save latency. What do you tell them? (Hint: think about which users the shortcut is wrong for, and whether that set is random or systematic.)
Permutation Feature Importance
Suppose you have a model you cannot read and you want one number per feature answering "how much does the model rely on this?" The cheapest honest answer is destructive: break the feature and see how much worse the model gets.
Concretely, take the evaluation set, shuffle one column so its values no longer line up with the rows they came from, re-score, and record the drop in your metric. The feature's values are still present and their distribution is unchanged -- only the relationship to the outcome has been destroyed. If the metric collapses, the model depended on that relationship. If nothing happens, the model was ignoring it.
The analogy is testing which cables matter by unplugging one at a time and watching what stops working. It is crude, it is model-agnostic, and it is usually the first diagnostic worth running because it requires nothing but a predict function and a metric.
original debt_ratio shuffled
income debt y income debt y
38k 0.16 1 38k 0.61 1 <- values kept, pairing destroyed
92k 0.61 0 92k 0.08 0
51k 0.08 0 51k 0.16 0
importance(j) = metric(original) - metric(column j shuffled)
Two decisions change the meaning of the result and are routinely made by accident.
Which dataset? Permuting on the training set measures how much the model used the feature to fit, including to fit noise. Permuting on a held-out set measures how much the feature contributes to performance that generalises. A feature with a large training drop and a near-zero test drop is a memorisation channel, not a signal.
How many repeats? A single shuffle is one draw from a random process. Report the mean over several repeats with its spread, or you will rank features by noise.
X, y, names = make_data(); tr, te = slice(0, 4000), slice(4000, None)
gbm = GBM(n=60, depth=3).fit(X[tr], y[tr])
rng = np.random.default_rng(1)
def perm_imp(Xe, ye, reps=5):
base = auc(ye, gbm.predict_proba(Xe)); out = []
for j in range(Xe.shape[1]):
drops = []
for _ in range(reps):
Xp = Xe.copy()
Xp[:, j] = rng.permutation(Xp[:, j]) # break only column j
drops.append(base - auc(ye, gbm.predict_proba(Xp)))
out.append((np.mean(drops), np.std(drops)))
return base, out
base_te, imp_te = perm_imp(X[te], y[te])
base_tr, imp_tr = perm_imp(X[tr], y[tr])
print(f"baseline AUC train {base_tr:.3f} test {base_te:.3f}")
print(f"{'feature':14}{'test drop':>12}{'+/-':>7}{'train drop':>12}")
for n, (m, s), (mt, _) in sorted(zip(names, imp_te, imp_tr),
key=lambda t: -t[1][0]):
print(f"{n:14}{m:12.4f}{s:7.4f}{mt:12.4f}")
# THE CORRELATED-PAIR TRAP: PERMUTE THE TWO INCOME COLUMNS TOGETHER.
Xp = X[te].copy(); p = rng.permutation(len(Xp))
Xp[:, 0], Xp[:, 4] = Xp[p, 0], Xp[p, 4] # same shuffle, both cols
print(f"\nlog_income + log_verified permuted TOGETHER: drop "
f"{base_te - auc(y[te], gbm.predict_proba(Xp)):.4f}")
print(f"correlation of the pair: {np.corrcoef(X[:, 0], X[:, 4])[0, 1]:.3f}")
baseline AUC train 0.790 test 0.778
feature test drop +/- train drop
debt_ratio 0.0625 0.0046 0.0655
log_income 0.0613 0.0093 0.0565
log_verified 0.0511 0.0051 0.0625
late_pmts 0.0224 0.0013 0.0247
age 0.0142 0.0038 0.0289
log_income + log_verified permuted TOGETHER: drop 0.1902
correlation of the pair: 0.994
The last two lines are the single most important thing to understand about this method. Individually, the two income columns look worth 0.061 and 0.051 AUC. Permuted together they are worth 0.190 -- more than three times their sum. The reason is that when you break one, the model reads the other, so each shuffle understates both. With correlated features, permutation importance systematically attributes too little to every member of the correlated group.
There is a second, subtler problem. Shuffling a column independently of the others creates rows that cannot exist: a 23-year-old with 40 years of credit history, a studio apartment with six bedrooms. The model is then evaluated on inputs from outside the data distribution, where its behaviour is undefined and arbitrary. The measured drop mixes "the model needed this feature" with "the model was asked nonsense".
Both problems have standard mitigations:
| Problem | Mitigation |
|---|---|
| Correlated features split credit | Permute correlated groups jointly; report group importance |
| Impossible rows created | Permute within strata of the correlated features (conditional permutation) |
| Ranking driven by noise | Repeat 5-20 times, report mean and standard deviation |
| Train/test confusion | Report both; treat a large gap as an overfitting signal |
| Metric insensitivity | Use a metric that reacts to your actual decision (recall at threshold, not accuracy) |
One more property worth stating explicitly: permutation importance measures importance to this fitted model, not importance in the world. If two features carry the same information and the model happened to key on one, permutation importance will say the other is worthless. That is a true statement about the model and a false statement about the problem, and the difference has misled a great many feature-selection exercises.
Key Takeaways
- Permutation importance shuffles one column and measures the metric drop; it needs only a predict function and a metric.
- Permuting on training data measures fitting; permuting on held-out data measures generalisation, and a large gap between them indicates overfitting to that feature.
- Always repeat the shuffle and report the spread, or the ranking is noise.
- Correlated features split the credit between them, understating every member of the group; permute the group jointly to see its true weight.
- Independent shuffling creates impossible rows, so the drop partly measures the model's behaviour off the data distribution.
- The result describes the fitted model, not the problem: a redundant feature scores near zero even if it is perfectly predictive on its own.
- Choose a metric that matches the decision, since importance is defined relative to whatever you measure.
🧪 Practice
- Compute permutation importance on train and test for one of your models with 10 repeats, and report mean and standard deviation for each feature.
- Find the two most correlated features in your data, permute them jointly, and compare the group drop with the sum of the individual drops.
- Recompute importance using two different metrics (for example AUC and recall at your production threshold) and explain any reordering.
- Construct a dataset with a duplicated column and demonstrate that both copies score near zero individually.
- Interview: A colleague drops every feature whose permutation importance is below 0.001 and the model's AUC falls noticeably. What happened? (Hint: consider what happens to a group of features that were each individually redundant but jointly necessary.)
Partial Dependence and ICE Plots
Feature importance tells you how much a feature matters. It says nothing about which direction or what shape. A feature can be enormously important and its effect can be U-shaped, stepped, or reversed in the tails -- and those are the facts you need to sanity-check a model or explain it to a domain expert.
A partial dependence plot (PDP) answers: if I could set this feature to value v for everybody, leaving all their other characteristics alone, what would the model predict on average? Mechanically, you take a sample of real rows, overwrite the feature of interest with v in every one, score them all, and average. Repeat across a grid of v.
The analogy is a clinical trial run inside the model: everyone gets the same dose, the population is otherwise unchanged, and you record the average response. And, as with a trial, the average response can hide that half the patients improved and half got worse.
That is what an ICE plot (individual conditional expectation) fixes. Instead of averaging, keep one curve per row. The PDP is exactly the mean of the ICE curves, so nothing is added -- what is removed is the averaging that conceals disagreement.
ICE: one line per individual PDP: the average of those lines
p | ___/ p |
| ___/ | ____/
| ___/ low income | ___/
|-------------- |___/
|\___ |
| \____ high income |
+---------------- debt_ratio +---------------- debt_ratio
two populations, opposite one curve, both stories
slopes averaged away
X, y, names = make_data(); tr, te = slice(0, 4000), slice(4000, None)
gbm = GBM(n=60, depth=3).fit(X[tr], y[tr])
grid = np.linspace(0.05, 0.9, 12)
bg = X[te][:400] # background sample of real rows
def ice(rows):
out = np.empty((len(rows), len(grid)))
for k, v in enumerate(grid):
Z = rows.copy(); Z[:, 1] = v # force debt_ratio, keep the rest
out[:, k] = gbm.predict_proba(Z)
return out
curves = ice(bg)
pdp = curves.mean(0) # PDP == mean of the ICE curves
print("PARTIAL DEPENDENCE OF P(default) ON debt_ratio")
print(" debt_ratio " + "".join(f"{v:6.2f}" for v in grid))
print(" PDP " + "".join(f"{p:6.2f}" for p in pdp))
lo = X[te][:, 0] < np.quantile(X[:, 0], 0.25) # bottom income quartile
hi = X[te][:, 0] > np.quantile(X[:, 0], 0.75) # top income quartile
print(" low income " + "".join(f"{p:6.2f}" for p in ice(X[te][lo][:200]).mean(0)))
print(" high income" + "".join(f"{p:6.2f}" for p in ice(X[te][hi][:200]).mean(0)))
sl = curves[:, -1] - curves[:, 0] # per-individual slope
print(f"\nindividual slopes across the grid: mean {sl.mean():+.2f}, "
f"min {sl.min():+.2f}, max {sl.max():+.2f}")
print(f"share of individuals whose curve goes DOWN: {(sl < 0).mean():.1%}")
cnt, _ = np.histogram(X[:, 1], bins=np.r_[grid - 0.0386, 0.9])
print(" rows near grid point " + "".join(f"{c:6d}" for c in cnt))
PARTIAL DEPENDENCE OF P(default) ON debt_ratio
debt_ratio 0.05 0.13 0.20 0.28 0.36 0.44 0.51 0.59 0.67 0.75 0.82 0.90
PDP 0.32 0.32 0.32 0.36 0.37 0.44 0.46 0.46 0.50 0.54 0.54 0.54
low income 0.44 0.45 0.45 0.54 0.55 0.70 0.72 0.73 0.75 0.79 0.79 0.79
high income 0.20 0.20 0.20 0.20 0.20 0.21 0.22 0.22 0.23 0.25 0.25 0.25
individual slopes across the grid: mean +0.22, min -0.02, max +0.47
share of individuals whose curve goes DOWN: 1.0%
rows near grid point 287 546 791 808 791 708 595 515 323 255 156 68
The PDP says debt ratio raises default probability by 22 points across its range. True on average, and badly incomplete. For low-income applicants the effect is 35 points; for high-income applicants it is 5. Anyone who reads only the PDP will build a policy -- "cap debt ratio at 0.5" -- that is far too weak for the group it is meant to protect and pointless for the other.
Three practical cautions follow directly from the mechanics.
The step shape is the model, not the world. The PDP is flat from 0.05 to 0.20 and jumps at 0.28 because tree ensembles are piecewise constant. Those plateaus are artifacts of the splits, and reading a real threshold into them is a mistake.
The last grid points are extrapolation. Only 68 of 6,000 rows lie near debt_ratio = 0.9. The curve is still drawn there, at full visual weight, on almost no evidence. Always plot the feature's distribution beneath the PDP, or restrict the grid to the 5th-95th percentile.
Correlated features make the query points impossible. Setting
debt_ratio = 0.9 for a high-income applicant with no late payments creates
a combination the model never saw. This is the same off-distribution problem
as permutation importance, and it has the same fix: accumulated local
effects (ALE), which measure the model's local differences within narrow
bins of the feature and integrate them, instead of extrapolating globally.
| Method | Answers | Handles interactions | Robust to correlation |
|---|---|---|---|
| PDP | Average effect of setting a feature | No -- averages them away | No |
| ICE | Per-individual effect | Yes -- shows heterogeneity | No |
| Centred ICE | Effect relative to each row's start | Yes | No |
| 2-D PDP | Joint effect of a feature pair | Only that pair | No |
| ALE | Local effect within observed bins | Partially | Yes |
A workable default: plot ICE with the PDP overlaid, colour a few curves by a suspected interacting feature, add a rug or histogram of the feature's distribution, and switch to ALE the moment the feature is strongly correlated with another.
Key Takeaways
- A PDP shows the model's average prediction as one feature is forced across a grid, with all other features left as they are.
- ICE keeps one curve per row; the PDP is exactly the average of the ICE curves, so ICE strictly adds information.
- A flat or modest PDP can hide opposite effects in different subgroups -- here 35 points for low income against 5 for high income.
- Plateaus and steps in a tree ensemble's PDP are artifacts of split locations, not thresholds in the world.
- The tails of a PDP are drawn on very few observations; always show the feature's distribution alongside the curve.
- Forcing a feature value creates impossible rows when features are correlated; ALE avoids this by working within observed bins.
- Colour ICE curves by a suspected interacting feature to find heterogeneity you would otherwise average away.
🧪 Practice
- Plot the PDP and ICE curves for your model's most important feature and report the fraction of individuals whose curve slopes opposite to the PDP.
- Overlay a histogram of that feature and mark which part of the PDP rests on fewer than 1% of rows.
- Split the ICE curves by a second feature's quartiles and report the difference in average slope between the top and bottom quartile.
- Implement a one-dimensional ALE plot (differences within bins, cumulatively summed) and compare it with the PDP for a correlated feature.
- Interview: A PDP for "number of previous claims" is almost flat, but the feature has the highest permutation importance in the model. Give two explanations that could both be true. (Hint: one involves the shape of the effect across subgroups, the other involves interactions with other features.)
SHAP Values
Every local explanation method is trying to do the same thing: split a prediction into per-feature parts. The obvious approaches all fail in ways that are easy to demonstrate. "Remove the feature and see how the prediction changes" gives a different answer depending on which other features you removed first. "Coefficient times value" only works for linear models.
SHAP starts from a different question: is there a unique fair way to divide a prediction among its features? That question was answered in 1953, in cooperative game theory, by Lloyd Shapley -- and it was not about models at all. It was about dividing a payout among players who achieved it together.
Map the game onto the model:
- the players are the features,
- the payout is the prediction minus the average prediction,
- a coalition S is a subset of features whose values you reveal,
- the coalition's value v(S) is the model's expected output when only the features in S are set to this row's values and the rest are drawn from the data.
A feature's Shapley value is its average marginal contribution across every possible order in which features could join the coalition. Averaging over all orders is what removes the arbitrariness of "which feature did we remove first".
ONE ROW, THREE FEATURES: ALL 3! = 6 JOINING ORDERS
{} --income--> {inc} --debt--> {inc,debt} --late--> all
{} --income--> {inc} --late--> {inc,late} --debt--> all
{} --debt--> {debt} --income-->{inc,debt} --late--> all
...
phi(income) = average of income's step size over all six orders
and: phi(income) + phi(debt) + phi(late) = f(x) - E[f(X)]
That last line is the efficiency (or local accuracy) property, and it is what makes SHAP usable in a regulated setting: the contributions add up to the prediction exactly, so no part of the decision is unexplained. Shapley values are the unique attribution satisfying efficiency plus symmetry (identical contributions get identical credit), dummy (a feature the model ignores gets zero), and additivity across ensembled models.
The cost is combinatorial: 2^d coalitions. With five features that is 32 and we can compute it exactly; in practice you use KernelSHAP (sampled coalitions, weighted regression) or TreeSHAP (an exact polynomial-time algorithm for tree ensembles).
from itertools import combinations
from math import factorial
X, y, names = make_data(); tr, te = slice(0, 4000), slice(4000, None)
gbm = GBM(n=60, depth=3).fit(X[tr], y[tr])
bg = X[tr][np.random.default_rng(3).choice(4000, 80, replace=False)]
d = X.shape[1]
def value(x, S):
"""v(S): expected model output with features in S fixed to x's values
and every other feature replaced from the background sample."""
Z = bg.copy()
for j in S: Z[:, j] = x[j]
return gbm.predict_proba(Z).mean()
def shap(x):
v = {frozenset(S): value(x, S) # all 2^d coalitions
for k in range(d+1) for S in combinations(range(d), k)}
phi = np.zeros(d)
for j in range(d):
for k in range(d): # |S| = k, j not in S
for S in combinations([i for i in range(d) if i != j], k):
wt = factorial(k)*factorial(d-k-1)/factorial(d)
phi[j] += wt*(v[frozenset(S+(j,))] - v[frozenset(S)])
return phi, v[frozenset()], v[frozenset(range(d))]
x = X[te][np.argsort(gbm.predict_proba(X[te]))[-25]]
phi, base, full = shap(x)
print("SHAP DECOMPOSITION OF ONE PREDICTION")
print(f" base value E[f(X)] {base:+.3f}")
for n, p in sorted(zip(names, phi), key=lambda t: -abs(t[1])):
print(f" {n:24}{p:+.3f}")
print(f" sum of contributions {phi.sum():+.3f}")
print(f" base + sum {base+phi.sum():.3f}")
print(f" model prediction f(x) {full:.3f} <- identical, by design")
G = np.array([shap(r)[0] for r in X[te][:30]]) # local -> global
print("\nGLOBAL IMPORTANCE = MEAN |SHAP| OVER 30 ROWS")
for n, m in sorted(zip(names, np.abs(G).mean(0)), key=lambda t: -t[1]):
print(f" {n:14}{m:.4f}")
print("\nSIGN DISAGREEMENT ACROSS ROWS (same feature, opposite direction)")
for n, col in zip(names, G.T):
print(f" {n:14} positive for {(col > 0).mean():5.0%} of rows, "
f"negative for {(col < 0).mean():5.0%}")
SHAP DECOMPOSITION OF ONE PREDICTION
base value E[f(X)] +0.420
debt_ratio +0.200
log_verified +0.119
log_income +0.101
late_pmts -0.019
age -0.009
sum of contributions +0.392
base + sum 0.812
model prediction f(x) 0.812 <- identical, by design
GLOBAL IMPORTANCE = MEAN |SHAP| OVER 30 ROWS
debt_ratio 0.0670
log_income 0.0653
log_verified 0.0651
age 0.0354
late_pmts 0.0289
SIGN DISAGREEMENT ACROSS ROWS (same feature, opposite direction)
log_income positive for 40% of rows, negative for 60%
debt_ratio positive for 20% of rows, negative for 80%
age positive for 17% of rows, negative for 83%
late_pmts positive for 50% of rows, negative for 50%
log_verified positive for 30% of rows, negative for 70%
Three things to read out of that output.
The additivity is exact. 0.420 + 0.392 = 0.812, which is the model's prediction to three decimals. That is not an empirical finding; it is a mathematical guarantee, and it is why SHAP is the default for individual-level reason codes.
Averaging the absolute values turns local explanations into a global
one. The ranking it produces is generally more trustworthy than
permutation importance because it is not confounded by metric choice -- but
note that the correlated income pair still splits roughly evenly (0.065 and
0.065), so SHAP does not escape the correlated-feature problem either. The
shap interventional formulation spreads credit across the group; the
tree_path_dependent formulation spreads it differently. Neither is wrong,
and they disagree.
Signs vary by row. debt_ratio pushes 80% of rows down and 20% up. This
is the local/global distinction reappearing: a global bar chart of mean
|SHAP| tells you how much a feature moves predictions, and deliberately
discards which way. Use a beeswarm plot (one dot per row, coloured by
feature value) when direction matters.
Four caveats that matter in production:
| Caveat | Why it matters |
|---|---|
| The base value depends on the background set | Explanations against "all customers" and "declined customers" differ; state which |
| Cost grows with features and background size | KernelSHAP is sampled and therefore noisy; TreeSHAP is exact but tree-only |
| Attribution is in the model's output units | SHAP on log-odds is additive; SHAP on probability is not; pick one and label it |
| Correlated features share credit | A feature can look moderately important while being individually irreplaceable |
And the one misreading to guard against above all: a SHAP value is not a causal effect. It says how much this feature's value moved this model's output relative to a baseline. It does not say what would happen in the world if the feature changed. That question needs the counterfactual and causal machinery, not an attribution.
Key Takeaways
- SHAP divides a prediction into per-feature contributions using Shapley values from cooperative game theory, averaging over all orders in which features could be revealed.
- Contributions plus the base value equal the prediction exactly, which is what makes SHAP defensible for individual reason codes.
- Shapley values are the unique attribution satisfying efficiency, symmetry, dummy, and additivity.
- Exact computation is exponential in the number of features; use TreeSHAP for tree ensembles and sampled KernelSHAP otherwise.
- Mean absolute SHAP aggregates local explanations into a global ranking while discarding direction; beeswarm plots retain it.
- The explanation is relative to a chosen background distribution, so the baseline must be stated alongside the numbers.
- Correlated features still share credit, and different SHAP variants split it differently.
- A SHAP value is an attribution of the model's output, not a causal effect in the world.
🧪 Practice
- Compute exact Shapley values for one row of a 4-feature model by brute force and verify that the contributions plus the base value equal the prediction.
- Recompute SHAP for the same row against two different background sets (all rows, then only approved rows) and explain why the numbers move.
- Compare mean absolute SHAP with permutation importance on the same model and account for any disagreement in the ranking.
- Compute SHAP on log-odds and on probability for the same row and show that only one of them sums to the prediction gap.
- Interview: A stakeholder reads that "income contributed +0.10 to this applicant's default probability" and concludes that raising the applicant's income by any amount will lower their risk by 0.10. Correct them. (Hint: distinguish attribution relative to a baseline population from the effect of an intervention on one person.)
LIME
SHAP's guarantees come from evaluating the model over every coalition of features, which is expensive and requires deciding how to "hide" a feature. LIME (Local Interpretable Model-agnostic Explanations) takes a completely different route, with a simpler intuition: any smooth function looks linear if you zoom in far enough.
So: pick the row you want to explain, generate a cloud of synthetic points around it, ask the black box for its prediction on each, weight those points by how close they are to your row, and fit a sparse linear model to the weighted cloud. The linear model's coefficients are the explanation. You are not describing the model; you are describing a small neighbourhood of it.
The analogy is surveying a hillside. You cannot describe an entire mountain range with a plane, but you can describe the patch of ground you are standing on with a slope and a direction -- and that is precisely what you need to answer "which way is downhill from here?"
the black box's decision surface LIME's local linear fit
___ /
/ \___ /
/ \__ x o o x <- weighted samples
/ \___/ \ o / o near x only
/ \ /
+--------------------------- +------------------------
global behaviour: nonlinear local behaviour: a slope
Four choices define a LIME explanation, and all four are yours to make: how to sample perturbations, how to measure distance, the kernel width that decides what "local" means, and how many features the surrogate may use. Different choices give different explanations of the same prediction.
X, y, names = make_data(); tr, te = slice(0, 4000), slice(4000, None)
gbm = GBM(n=60, depth=3).fit(X[tr], y[tr])
mu, sd = X[tr].mean(0), X[tr].std(0)
x = X[te][np.argsort(gbm.predict_proba(X[te]))[-25]]
def lime(x, n=800, width=0.75, seed=0):
rng = np.random.default_rng(seed)
Xp = mu + rng.normal(0, 1, (n, X.shape[1])) * sd # sample the space
dist = np.linalg.norm((Xp - x) / sd, axis=1) # distance to x
wts = np.exp(-(dist**2) / (2*width*len(x))) # locality kernel
yq = gbm.predict_proba(Xp) # black box answers
A = np.c_[np.ones(n), (Xp - x) / sd]
W = np.diag(wts) # weighted ridge fit
beta = np.linalg.solve(A.T @ W @ A + 1e-3*np.eye(A.shape[1]), A.T @ W @ yq)
ybar = (wts*yq).sum()/wts.sum()
r2 = 1 - (wts*(yq - A @ beta)**2).sum()/(wts*(yq - ybar)**2).sum()
return beta[0], beta[1:], r2, wts
b0, coef, r2, wts = lime(x)
print(f"LIME AROUND ONE APPLICANT f(x)={gbm.predict_proba(x[None])[0]:.3f}"
f" local R2={r2:.2f}")
print(f"effective sample size = {wts.sum()**2/(wts**2).sum():.0f} of 800")
for n, c in sorted(zip(names, coef), key=lambda t: -abs(t[1])):
print(f" {n:14}{c:+.3f} per standard deviation")
print("\nSTABILITY AT A SMALLER PERTURBATION BUDGET (n=100)")
print(f" {'seed':>6}" + "".join(f"{n[:11]:>13}" for n in names) + f"{'R2':>7}")
for s in range(4):
_, c, r, _ = lime(x, n=100, seed=s)
print(f" {s:6}" + "".join(f"{v:+13.3f}" for v in c) + f"{r:7.2f}")
print("\nSENSITIVITY TO THE KERNEL WIDTH (a free parameter you must pick)")
print(f" {'width':>6}" + "".join(f"{n[:11]:>13}" for n in names) + f"{'R2':>7}")
for wd in (0.25, 0.5, 1.0, 4.0):
_, c, r, _ = lime(x, width=wd, seed=0)
print(f" {wd:6.2f}" + "".join(f"{v:+13.3f}" for v in c) + f"{r:7.2f}")
LIME AROUND ONE APPLICANT f(x)=0.812 local R2=0.79
effective sample size = 363 of 800
debt_ratio +0.085 per standard deviation
log_income -0.083 per standard deviation
log_verified -0.083 per standard deviation
late_pmts +0.032 per standard deviation
age -0.022 per standard deviation
STABILITY AT A SMALLER PERTURBATION BUDGET (n=100)
seed log_income debt_ratio age late_pmts log_verifie R2
0 -0.086 +0.087 -0.021 +0.042 -0.086 0.86
1 -0.089 +0.109 -0.011 +0.037 -0.083 0.80
2 -0.070 +0.083 -0.020 +0.024 -0.080 0.82
3 -0.110 +0.080 -0.025 +0.024 -0.078 0.81
SENSITIVITY TO THE KERNEL WIDTH (a free parameter you must pick)
width log_income debt_ratio age late_pmts log_verifie R2
0.25 -0.073 +0.051 -0.001 +0.010 -0.066 0.65
0.50 -0.084 +0.081 -0.018 +0.026 -0.082 0.77
1.00 -0.081 +0.084 -0.023 +0.034 -0.082 0.79
4.00 -0.072 +0.078 -0.025 +0.038 -0.077 0.77
The first block is a perfectly reasonable explanation: debt ratio pushes this applicant's risk up, income pushes it down, and the local linear approximation captures 79% of the model's variation in the neighbourhood. Report that R-squared always -- it is LIME's fidelity measure, and a low value means the explanation does not describe the model even locally.
The next two blocks are the reason LIME is used with caution in high-stakes
settings. At 100 perturbations, the same applicant, the same model, and only
a different random seed produce income coefficients from -0.070 to -0.110, a
57% swing, and the top-driver ranking between log_income and debt_ratio
flips between seeds. At kernel width 0.25 the effect of late_pmts is
+0.010; at width 4.0 it is +0.038, nearly four times larger. Nothing about
the model changed. Two applicants with identical files can receive
different reason codes, and an applicant who appeals can receive a different
answer than the first time.
| SHAP | LIME | |
|---|---|---|
| Foundation | Game theory, axioms | Local linear approximation |
| Adds up to the prediction | Yes, exactly | No, only approximately |
| Free parameters | Background set | Kernel width, distance, sample size, sparsity |
| Stability | High (exact variants) | Low to moderate; must be measured |
| Cost per explanation | High (or tree-specialised) | Moderate, one model fit |
| Works on text/images | Yes, with segmentations | Yes, natively via superpixels/tokens |
| Fidelity report | Guaranteed | Local R-squared, must be reported |
LIME still earns its place. It is the easiest method to apply to text and images, where "features" are tokens or superpixels and turning them off is natural. It produces a genuinely sparse explanation ("these three words drove the classification"), which is often more useful to a human than attributions over 30,000 dimensions. And it is fast enough to run interactively during debugging.
The rule for using it responsibly: fix the seed, fix the kernel width, and report the local fidelity with every explanation. If the explanation is destined for a person affected by the decision, run it several times and verify the top reasons do not move.
Key Takeaways
- LIME explains one prediction by fitting a sparse linear model to perturbed samples around that row, weighted by proximity.
- It describes a neighbourhood of the model, not the model, and its contributions do not sum to the prediction.
- Local R-squared is LIME's fidelity measure and must be reported with every explanation.
- The explanation depends on four free choices: sampling scheme, distance metric, kernel width, and surrogate sparsity.
- With small perturbation budgets, different seeds change coefficients by tens of percent and can reorder the top drivers.
- Kernel width materially changes the answer, and there is no principled way to choose it from the model alone.
- LIME's practical advantage is text and image data, where perturbation by removing tokens or superpixels is natural and the sparse output is readable.
- For individual-facing use, pin the seed and parameters and verify that repeated runs give the same top reasons.
🧪 Practice
- Run LIME 20 times on one row with different seeds and report the standard deviation of each coefficient and how often the top feature changes.
- Sweep the kernel width over two orders of magnitude and plot how the explanation and the local R-squared change.
- Compare LIME and SHAP explanations for the same three rows and characterise where they disagree.
- Apply LIME to a text classifier by deleting random words, and inspect whether the highlighted words are plausible causes.
- Interview: Your compliance team asks whether LIME explanations can be used as adverse action reasons. What conditions would you require first? (Hint: think about reproducibility of the same answer for the same person, and about what fidelity number you would have to commit to.)
Counterfactual Explanations
Attribution methods answer "what drove this outcome?" People affected by a decision almost never ask that. They ask "what do I have to do differently?" Those are different questions, and a SHAP value does not answer the second one.
A counterfactual explanation is the answer to the second question: the nearest set of feature values that would have produced a different outcome. "Your application was declined. It would have been approved with an annual income of 43,800 instead of 38,100, all else equal." That statement is checkable by the applicant, actionable, and does not require them to understand the model at all.
The analogy is the difference between a post-mortem and a set of directions. A post-mortem tells you why you are where you are; directions tell you how to get somewhere else. Both are useful; only one is what you want when you are lost.
Formally, you search for x' minimising a cost that combines three things:
minimise cost(x, x')
subject to model(x') crosses the decision threshold
x' is plausible (a real combination of values)
x' is actionable (only features the person can change)
cost = sum over changed features of (effort_j x size of change_j)
Each constraint corresponds to a real failure. Drop plausibility and you produce "approved if your debt ratio were negative". Drop actionability and you produce "approved if you were 26 instead of 23", which is not recourse, it is an insult -- and in several jurisdictions, evidence of a problem.
X, y, names = make_data(); tr, te = slice(0, 4000), slice(4000, None)
gbm = GBM(n=60, depth=3).fit(X[tr], y[tr])
p_te = gbm.predict_proba(X[te])
cand = np.where((p_te > 0.58) & (p_te < 0.66) & (X[te][:, 3] >= 1))[0]
x = X[te][cand[0]] # a borderline decline
THRESH = 0.5
print(f"applicant: income {np.exp(x[0]):,.0f}, debt_ratio {x[1]:.2f}, "
f"age {x[2]:.0f}, late {x[3]:.0f}")
print(f"p(default) = {gbm.predict_proba(x[None])[0]:.3f} -> DECLINED "
f"(threshold {THRESH})\n")
# EFFORT WEIGHTS AND ALLOWED DIRECTIONS ENCODE ACTIONABILITY. AGE IS ABSENT
# FROM THIS TABLE ON PURPOSE: IT IS NOT A THING THE APPLICANT CAN CHANGE.
COST = {1: ("debt_ratio", 4.0, -1), 0: ("income", 3.0, +1),
3: ("late_pmts", 9.0, -1)}
best = None
for dd in np.arange(0, 0.81, 0.02): # cut debt ratio
for di in np.arange(0, 0.81, 0.02): # raise log income
for dl in (0, 1, 2, 3): # wait out late payments
z = x.copy()
z[1] = max(0.0, x[1]-dd)
z[0] = x[0]+di; z[4] = z[0] # verified income moves too
z[3] = max(0.0, x[3]-dl)
if gbm.predict_proba(z[None])[0] >= THRESH: continue
cost = 4.0*dd/0.1 + 3.0*di/0.1 + 9.0*dl
if best is None or cost < best[0]: best = (cost, dd, di, dl, z)
cost, dd, di, dl, z = best
print("NEAREST ACTIONABLE COUNTERFACTUAL (age and history treated as fixed)")
print(f" reduce debt_ratio by {dd:.2f} ({x[1]:.2f} -> {z[1]:.2f})")
print(f" raise income by {np.exp(di)-1:.1%} "
f"({np.exp(x[0]):,.0f} -> {np.exp(z[0]):,.0f})")
print(f" clear {dl} late payment(s)")
print(f" new p(default) = {gbm.predict_proba(z[None])[0]:.3f} -> APPROVED"
f" (effort score {cost:.0f})")
print("\nSINGLE-FEATURE COUNTERFACTUALS: IS ANY ONE CHANGE ENOUGH?")
for j, (nm, _, sign) in COST.items():
found = None
for step in np.arange(0.02, 3.0, 0.02):
z2 = x.copy(); z2[j] = max(0.0, x[j] + sign*step)
if j == 0: z2[4] = z2[0]
if gbm.predict_proba(z2[None])[0] < THRESH: found = z2[j]; break
print(f" {nm:12} alone: " + (f"reach {found:.2f}" if found is not None
else "no reachable value flips the decision"))
applicant: income 38,079, debt_ratio 0.16, age 23, late 1
p(default) = 0.618 -> DECLINED (threshold 0.5)
NEAREST ACTIONABLE COUNTERFACTUAL (age and history treated as fixed)
reduce debt_ratio by 0.00 (0.16 -> 0.16)
raise income by 15.0% (38,079 -> 43,801)
clear 0 late payment(s)
new p(default) = 0.492 -> APPROVED (effort score 4)
SINGLE-FEATURE COUNTERFACTUALS: IS ANY ONE CHANGE ENOUGH?
debt_ratio alone: no reachable value flips the decision
income alone: reach 10.69
late_pmts alone: no reachable value flips the decision
This example contains the lesson that makes counterfactuals worth the effort.
A SHAP explanation of this applicant would put age near the top: they are
23, and the model learned a sharp penalty below 25. That is a true statement
about the model and a useless statement to the applicant. The counterfactual
search, told that age is not actionable, finds the only route that exists:
15% more income. Note also that reducing debt ratio cannot flip the decision
at all -- it is already 0.16, so there is nothing left to give.
Several properties are worth knowing before you ship this.
Counterfactuals are not unique. There are usually many minimal-ish changes, and which one you show is a design decision. Presenting a diverse set ("any one of these three would have worked") is more useful and more honest than presenting one.
The distance metric is a policy statement. Declaring that a 10% income rise is "cheaper" than clearing a late payment encodes a claim about human effort. Write those weights down, have them reviewed, and do not let them be an accident of feature scaling.
A counterfactual is not a causal promise. The model says it would have scored the applicant differently with that income. Whether earning more income actually reduces their default risk is a causal question the model cannot answer, and if the feature is a proxy rather than a cause, the advice may be gameable without being beneficial -- the "recourse versus gaming" problem.
They must remain valid. If the model is retrained next week, the advice you gave may no longer hold. Serious deployments version the model behind every counterfactual and honour the advice for a stated period.
| Property | Attribution (SHAP/LIME) | Counterfactual |
|---|---|---|
| Question answered | Why this score? | What would change it? |
| Output | A number per feature | An alternative input row |
| Needs actionability model | No | Yes -- which features can change, at what cost |
| Uniqueness | One answer | Many; diversity is a feature |
| Directly usable by a layperson | Rarely | Usually |
| Legal framing | Reason codes | Recourse |
Key Takeaways
- A counterfactual explanation gives the nearest input that would have produced a different decision, answering "what must I change?" rather than "why?".
- The search needs three constraints: it must flip the decision, be plausible, and only touch features the person can act on.
- Immutable attributes must be excluded from the search; recourse that requires being older or a different sex is not recourse.
- The cost function encodes a human-effort policy and should be written down and reviewed, not left to feature scaling.
- Counterfactuals are not unique; presenting a diverse set is more useful and more honest than presenting one.
- A counterfactual describes the model's behaviour, not a causal guarantee about the world, and proxy features make the advice gameable.
- Advice tied to a model version stops being true when the model is retrained, so version and time-bound it.
- Some features offer no recourse at all; discovering that is itself a finding worth escalating.
🧪 Practice
- Implement a grid or random search for the nearest counterfactual on a model of yours and report the change it finds for three declined rows.
- Add a constraint that one feature is immutable and report how much the required effort increases.
- Generate three diverse counterfactuals for the same row and write the customer-facing message you would send.
- Retrain the model on a different data sample and check whether the counterfactuals you produced earlier still flip the decision.
- Interview: An applicant follows your counterfactual advice exactly, reapplies, and is declined again. List the possible causes. (Hint: consider what changed between the two scoring events, and whether the features the applicant changed were the only ones that moved.)
Limits of Explanation Methods
Every method in this section produces confident, plausible, well-formatted output whether or not it is telling you something true. The final skill is knowing what the output cannot establish -- because explanation tooling fails silently, and a wrong explanation is worse than none: it manufactures confidence.
Limit 1: many different models fit the data equally well, and they disagree about why. This is the Rashomon effect, named for the film in which several witnesses describe the same event incompatibly. When two features carry overlapping information, which one the model keys on is decided by accidents of regularisation, seed, and sample -- not by the world.
Limit 2: importance rankings are estimates, and estimates have variance. The ranking that a report presents as a fact is a statistic with a confidence interval that is almost never shown.
X, y, names = make_data(); tr, te = slice(0, 4000), slice(4000, None)
print("TWO MODELS, INDISTINGUISHABLE ACCURACY, DIFFERENT STORY")
for label, cols in [("uses reported income ", [0, 1, 2, 3]),
("uses verified income ", [4, 1, 2, 3]),
("uses both ", [0, 1, 2, 3, 4])]:
w, b, mu, sd = logistic(X[tr][:, cols], y[tr])
a = auc(y[te], ((X[te][:, cols]-mu)/sd) @ w + b)
top = [names[cols[k]] for k in np.argsort(-np.abs(w))[:2]]
print(f" {label} AUC {a:.3f} top drivers: {', '.join(top)}")
print(" The two income columns correlate at "
f"{np.corrcoef(X[:, 0], X[:, 4])[0, 1]:.3f}; a reviewer reading either")
print(" report would name a different variable as the deciding factor.\n")
print("SAME ALGORITHM, DIFFERENT BOOTSTRAP: HOW STABLE IS THE RANKING?")
rng = np.random.default_rng(7)
for rep in range(5):
idx = rng.choice(4000, 400, replace=True) # a small-data regime
g = GBM(n=40, depth=3).fit(X[tr][idx], y[tr][idx])
base = auc(y[te], g.predict_proba(X[te])); drops = []
for j in range(X.shape[1]):
Xp = X[te].copy(); Xp[:, j] = rng.permutation(Xp[:, j])
drops.append(base - auc(y[te], g.predict_proba(Xp)))
order = np.argsort(-np.array(drops))
print(f" replicate {rep}: " + " > ".join(names[k] for k in order))
print("\nHOW MANY PDP QUERY POINTS RESEMBLE A REAL APPLICANT?")
Z = (X - X.mean(0)) / X.std(0); bg = X[te][:300]
print(f" {'debt_ratio':>12}{'median dist to 10th NN':>26}{'% with no close row':>22}")
for v in (0.05, 0.3, 0.6, 0.9):
Q = bg.copy(); Q[:, 1] = v # the PDP's query points
Qz = (Q - X.mean(0)) / X.std(0)
d = np.sqrt(((Qz[:, None, :] - Z[tr][:600][None, :, :])**2).sum(-1))
nn10 = np.sort(d, axis=1)[:, 9] # distance to 10th neighbour
print(f" {v:12.2f}{np.median(nn10):26.2f}{(nn10 > 1.5).mean():22.1%}")
TWO MODELS, INDISTINGUISHABLE ACCURACY, DIFFERENT STORY
uses reported income AUC 0.771 top drivers: log_income, debt_ratio
uses verified income AUC 0.768 top drivers: log_verified, debt_ratio
uses both AUC 0.770 top drivers: log_income, debt_ratio
The two income columns correlate at 0.994; a reviewer reading either
report would name a different variable as the deciding factor.
SAME ALGORITHM, DIFFERENT BOOTSTRAP: HOW STABLE IS THE RANKING?
replicate 0: log_income > debt_ratio > log_verified > age > late_pmts
replicate 1: log_verified > debt_ratio > log_income > late_pmts > age
replicate 2: log_income > debt_ratio > log_verified > age > late_pmts
replicate 3: log_income > debt_ratio > log_verified > late_pmts > age
replicate 4: log_verified > debt_ratio > late_pmts > log_income > age
HOW MANY PDP QUERY POINTS RESEMBLE A REAL APPLICANT?
debt_ratio median dist to 10th NN % with no close row
0.05 1.19 12.7%
0.30 0.89 7.0%
0.60 1.06 8.7%
0.90 1.40 34.0%
Three models with AUC 0.768, 0.770, and 0.771 -- differences well inside sampling noise -- name different top drivers. Resample the training data and the most important feature changes between replicates. And a third of the PDP's query points at debt_ratio = 0.9 have no close analogue in the training data at all, so the model's behaviour there is extrapolation being reported as description.
The remaining limits do not need code to state, only discipline to remember.
| Limit | What it means in practice |
|---|---|
| Not causal | Every method describes the model's input-output mapping, not the world's mechanism |
| Off-manifold evaluation | Permutation, PDP, and SHAP all query impossible rows; the model's answers there are arbitrary |
| Correlated features | Credit is split or shifted arbitrarily among near-duplicates |
| Explanation does not validate | A model with a leaked feature produces a beautiful, coherent explanation of its leakage |
| Interactions are collapsed | A single number per feature cannot express "matters only when X is high" |
| Fairwashing | Explanations can be optimised to look acceptable while the model is unchanged |
| Human over-trust | People accept plausible explanations without checking fidelity, especially when presented graphically |
The fairwashing row deserves emphasis because it is an adversarial risk, not just an error. Given freedom over the background set, kernel width, surrogate family, and which features to display, a motivated analyst can produce an explanation that omits the uncomfortable variable without changing a single model weight. This is why explanation configuration must be version-controlled and reviewed like model code.
A defensible practice, then, is not "run SHAP" but:
- Report fidelity with every explanation -- surrogate R-squared, local R-squared, or the exactness guarantee you are relying on.
- Report stability -- rerun with different seeds, background sets, and bootstrap samples, and publish the variation rather than one draw.
- Report support -- state how many real observations back the region being explained; refuse to plot where there is no data.
- Cross-check with a second method -- agreement between SHAP, permutation importance, and a fitted GAM is weak evidence of truth; disagreement is strong evidence of a problem.
- Never present an explanation as a causal claim without a causal design behind it.
- Version the explanation configuration with the model, so the reasons given to a person can be reproduced years later.
Key Takeaways
- Many models fit the data equally well and attribute the outcome to different features; near-identical accuracy does not imply a shared story.
- Importance rankings are statistics with variance; in small-data regimes the top feature can change between bootstrap samples.
- Permutation, PDP, and SHAP all evaluate the model on inputs that do not occur in reality, especially in the tails.
- No explanation method is causal; all describe the model's mapping, not the world.
- A coherent explanation of a leaking or biased model is still coherent -- explanations do not validate models.
- Single numbers per feature cannot express interactions, so conditional effects are silently collapsed.
- Explanation configuration is adversarially exploitable (fairwashing) and must be version-controlled and reviewed.
- Defensible practice means reporting fidelity, stability, and data support with every explanation, and cross-checking with a second method.
🧪 Practice
- Train five models on bootstrap resamples of your data and report how often the top-ranked feature changes.
- Fit two models on overlapping feature subsets with near-identical accuracy and write the two contradictory explanation summaries they produce.
- For your PDP grid, measure the distance from each query point to its tenth nearest real row and mark the region you should refuse to plot.
- Take a model with a deliberately leaked feature, generate a SHAP summary, and show that the explanation looks entirely reasonable.
- Interview: You are asked to sign off that a model's explanations are "accurate". What would you insist the sign-off statement actually say? (Hint: accuracy of an explanation is fidelity to the model, not truth about the world, and it is a measured quantity with a range.)
<a id="162-fairness-and-bias"></a>
16.2 Fairness and Bias
Fairness in machine learning is not one property but a family of incompatible ones, and choosing between them is a normative decision that cannot be delegated to a metric; this section covers where bias enters before any model is fitted, the standard statistical definitions of fairness and what each one actually asserts, how regulators test for disparate impact, the three families of mitigation and what each costs, the mathematical reason you cannot have everything at once, and how to keep watching after deployment.
The examples share a hiring model on synthetic data constructed so that the two groups have almost identical true success rates, while every observed feature is measured less favourably for one of them. That is the hard case, and it is the common one.
import numpy as np
def make_hiring(n=20000, seed=4):
rng = np.random.default_rng(seed)
g = (rng.random(n) < 0.35).astype(int) # 1 = group B (minority)
skill = rng.normal(0, 1, n) # unobserved true ability
# measurement bias: the test under-measures group B and is noisier
test = skill - 0.35*g + rng.normal(0, 1, n)*(0.9 + 0.2*g)
# access bias: prestige tracks skill for A, barely for B
prestige = skill*(1.3 - 1.2*g) - 0.5*g + rng.normal(0, 0.5, n)
referral = (rng.random(n) < np.where(g == 1, 0.10, 0.30)).astype(float)
perform = (skill + rng.normal(0, 0.7, n) > 0.4).astype(int) # true label
# historical labels: past humans hired with a penalty against group B
hist = ((0.9*test + 0.6*prestige + 0.8*referral - 0.7*g
+ rng.normal(0, 0.7, n)) > 0.6).astype(int)
X = np.column_stack([test, prestige, referral])
return X, perform, hist, g, ["test_score", "prestige", "referral"]
def auc(y, s):
o = np.argsort(s); r = np.empty(len(s)); r[o] = np.arange(1, len(s)+1)
n1 = y.sum(); n0 = len(y) - n1
return (r[y == 1].sum() - n1*(n1+1)/2)/(n0*n1)
def logistic(X, y, sw=None, iters=4000, lr=0.6):
Xs = (X - X.mean(0))/X.std(0)
sw = np.ones(len(y)) if sw is None else sw*len(y)/sw.sum()
w = np.zeros(X.shape[1]); b = 0.0
for _ in range(iters):
p = 1/(1+np.exp(-(Xs @ w + b)))
gr = sw*(p - y)/len(y)
w -= lr*(Xs.T @ gr); b -= lr*gr.sum()
return lambda Z: 1/(1+np.exp(-(((Z-X.mean(0))/X.std(0)) @ w + b))), w
def rates(y, yhat, g):
"""The confusion matrix by group, as the rates fairness criteria use."""
out = {}
for k in (0, 1):
m = g == k
tp = ((yhat==1) & (y==1) & m).sum(); fn = ((yhat==0) & (y==1) & m).sum()
fp = ((yhat==1) & (y==0) & m).sum(); tn = ((yhat==0) & (y==0) & m).sum()
out[k] = dict(sel=(yhat[m]==1).mean(), tpr=tp/max(tp+fn, 1),
fpr=fp/max(fp+tn, 1), ppv=tp/max(tp+fp, 1),
base=y[m].mean(), n=int(m.sum()))
return out
Sources of Bias in Data
The phrase "biased algorithm" is almost always wrong about the location of the problem. Learning algorithms are indifferent: they minimise a loss over whatever they are given. If the output is unequal, the inequality was in the inputs, the labels, the sampling, the framing, or the deployment -- and knowing which one is the difference between a fix that works and a post-processing patch that hides the symptom.
The useful mental model is a chain of measurements between the world and the model, where each link can distort:
the world -> who gets recorded -> what is measured
(true ability) (sampling bias) (measurement bias)
|
what decision <- what label is kept <------+
gets deployed (historical/label bias)
(deployment bias)
|
+-> feeds back into next year's data (feedback loop)
Six named sources, each with a distinct fix:
| Source | What it is | Example | Fix direction |
|---|---|---|---|
| Historical bias | The world itself is unequal, and the data records it faithfully | Past managers hired fewer B candidates | Change the target, not the model |
| Representation bias | A group is too small in the sample to be learned well | 2% of training rows are group B | Collect or oversample; report per-group sample size |
| Measurement bias | The proxy measures the construct differently by group | A test that under-measures B's skill | Fix or drop the instrument; validate per group |
| Aggregation bias | One model imposed on populations with different relationships | Prestige predicts success for A but not B | Per-group models, interactions, or hierarchical models |
| Label bias | The label is a record of a biased decision, not the outcome | "Was hired" instead of "performed well" | Relabel with the outcome you actually care about |
| Deployment bias | The model is used differently from how it was validated | A risk score built for triage used for sentencing | Governance, not modelling |
The single most consequential of these is label bias, because it is invisible to every diagnostic that treats the label as ground truth. If the label is "was hired" and past hiring was discriminatory, a model with perfect accuracy reproduces discrimination perfectly, and every validation metric says it is excellent.
X, y, hist, g, names = make_hiring()
tr, te = slice(0, 14000), slice(14000, None)
print("THE SAME FEATURES, TWO DIFFERENT LABELS")
print(f" true on-the-job success rate A {y[g==0].mean():.3f}"
f" B {y[g==1].mean():.3f}")
print(f" historical hiring decision A {hist[g==0].mean():.3f}"
f" B {hist[g==1].mean():.3f}")
for lab, target in [("trained on historical decisions", hist),
("trained on measured performance", y)]:
f, _ = logistic(X[tr], target[tr])
yhat = (f(X[te]) > 0.5).astype(int)
a, b = yhat[g[te]==0].mean(), yhat[g[te]==1].mean()
print(f" {lab:34} select A {a:.3f} B {b:.3f} ratio {b/a:.2f}")
print("\nMEASUREMENT BIAS: THE FEATURE, NOT THE PERSON")
for j, n in enumerate(names):
print(f" mean {n:12} A {X[g==0, j].mean():+.3f} B {X[g==1, j].mean():+.3f}")
print(" ...while true success rates differ by only "
f"{abs(y[g==0].mean()-y[g==1].mean()):.3f}")
print("\nAGGREGATION BIAS: ONE MODEL FOR TWO DIFFERENT POPULATIONS")
mA, mB = g[te]==0, g[te]==1
f, _ = logistic(X[tr], y[tr]); sp = f(X[te])
print(f" pooled model AUC on A {auc(y[te][mA], sp[mA]):.3f}"
f" AUC on B {auc(y[te][mB], sp[mB]):.3f}")
fa, wa = logistic(X[tr][g[tr]==0], y[tr][g[tr]==0])
fb, wb = logistic(X[tr][g[tr]==1], y[tr][g[tr]==1])
print(f" per-group models AUC on A {auc(y[te][mA], fa(X[te][mA])):.3f}"
f" AUC on B {auc(y[te][mB], fb(X[te][mB])):.3f}")
print(" weights A: " + " ".join(f"{n}={v:+.2f}" for n, v in zip(names, wa)))
print(" weights B: " + " ".join(f"{n}={v:+.2f}" for n, v in zip(names, wb)))
print("\nREPRESENTATION BIAS: HOW MUCH GROUP-B DATA BUYS A GROUP-B MODEL")
print(f" {'B rows in training':>20}{'mean AUC on B':>16}{'sd over 8 seeds':>18}")
rng = np.random.default_rng(0); iB = np.where(g[tr]==1)[0]
for nB in (40, 150, 600, 2400):
aucs = []
for _ in range(8):
idx = rng.choice(iB, nB, replace=False)
fbb, _ = logistic(X[tr][idx], y[tr][idx])
aucs.append(auc(y[te][mB], fbb(X[te][mB])))
print(f" {nB:20}{np.mean(aucs):16.3f}{np.std(aucs):18.3f}")
THE SAME FEATURES, TWO DIFFERENT LABELS
true on-the-job success rate A 0.374 B 0.363
historical hiring decision A 0.433 B 0.125
trained on historical decisions select A 0.397 B 0.166 ratio 0.42
trained on measured performance select A 0.383 B 0.164 ratio 0.43
MEASUREMENT BIAS: THE FEATURE, NOT THE PERSON
mean test_score A +0.001 B -0.362
mean prestige A +0.003 B -0.502
mean referral A +0.300 B +0.101
...while true success rates differ by only 0.010
AGGREGATION BIAS: ONE MODEL FOR TWO DIFFERENT POPULATIONS
pooled model AUC on A 0.861 AUC on B 0.755
per-group models AUC on A 0.871 AUC on B 0.767
weights A: test_score=+0.50 prestige=+1.75 referral=-0.00
weights B: test_score=+1.09 prestige=+0.18 referral=-0.06
REPRESENTATION BIAS: HOW MUCH GROUP-B DATA BUYS A GROUP-B MODEL
B rows in training mean AUC on B sd over 8 seeds
40 0.755 0.015
150 0.757 0.013
600 0.765 0.003
2400 0.767 0.001
The first block sets up the expectation and the second demolishes it. Historical hiring favoured group A more than three to one (0.433 against 0.125) while true success rates differ by one percentage point. Fixing the label -- training on measured performance rather than on past hiring decisions -- barely changes the outcome: the selection ratio moves from 0.42 to 0.43. The bias has already been baked into the features. Group B's test scores average 0.36 standard deviations lower and their prestige scores 0.50 lower, for reasons that have nothing to do with their success rate.
This is the most important practical lesson in the whole topic: relabelling is necessary and not sufficient. When the measuring instrument is biased, no choice of target repairs it. You have to fix, replace, or drop the instrument, or accept that your model encodes its bias.
The aggregation block shows a second, independent problem. The pooled model
assigns prestige a weight of +1.75; the group-B-only model assigns +0.18.
The pooled model is applying, to group B, a relationship that does not hold
for group B. And the representation block quantifies why you cannot always
just fit per-group models: with 40 group-B rows the per-group AUC is both
lower and three times more variable across samples.
A structured way to look for all six before modelling:
- Compare the label to the construct. Write down what you wish you could measure and what you actually recorded. Every gap is a bias candidate.
- Tabulate sample size and outcome rate per group, including intersections. Small cells are a finding, not a footnote.
- Test the features for group differences that the outcome does not justify. A feature gap larger than the outcome gap is a measurement bias signal.
- Fit per-group models and compare coefficients. Large differences mean aggregation bias.
- Ask who is missing entirely. People who never applied, claims never filed, patients who never reached a clinic.
- Trace the feedback loop. If the model's decisions generate next year's training data, the bias compounds.
Key Takeaways
- Bias almost always enters through data, labels, sampling, or use -- rarely through the learning algorithm itself.
- Label bias is the most dangerous because it is invisible to accuracy metrics: a perfect model reproduces a biased decision perfectly.
- Relabelling to the true outcome is necessary but often insufficient, because biased features carry the disparity on their own.
- A feature gap between groups that is much larger than the outcome gap is direct evidence of measurement bias.
- Aggregation bias means one model is imposed on populations with different feature-outcome relationships; compare per-group coefficients to detect it.
- Per-group models are not a free fix: small groups give noisier, weaker models, which is representation bias reappearing.
- Always report sample size and outcome rate per group and per intersection before reporting any fairness metric.
- Deployment bias and feedback loops occur outside the model and need governance, not tuning.
🧪 Practice
- For a dataset of yours, write down the construct you care about and the label you actually have, then list every way they could diverge by group.
- Tabulate row count, outcome rate, and missingness rate per group, and flag every cell with fewer than 100 rows.
- Fit separate models per group and report the two coefficient vectors; identify the feature whose weight differs most.
- Simulate a feedback loop: train, select the top 30%, observe outcomes only for the selected, retrain, and track the selection ratio over five rounds.
- Interview: A team removes the protected attribute from the feature set and reports that the model is now fair. What is wrong with that claim? (Hint: consider what remains in the data after the column is dropped, and how you would test whether it is still there.)
Fairness Definitions and Metrics
"Is this model fair?" is not a question a metric can answer, because there are at least a dozen formal definitions of fairness and they assert different, sometimes contradictory, things. What a data scientist can do is state each definition precisely, measure it, and force the choice between them to be made explicitly by the people accountable for the decision.
Every group-fairness definition has the same shape: pick a quantity from the confusion matrix, and require it to be equal across groups. The definitions differ only in which quantity.
actually succeeded?
yes no
model says yes TP FP -> precision = TP/(TP+FP)
model says no FN TN
| |
recall = TP/(TP+FN) |
FPR = FP/(FP+TN)
selection rate = (TP+FP)/all base rate = (TP+FN)/all
equalise selection rate -> demographic parity
equalise recall -> equal opportunity
equalise recall AND FPR -> equalized odds
equalise precision -> predictive parity
equalise P(y=1 | score) -> calibration by group
| Definition | Requires equal | Assumes the label is | Typical use |
|---|---|---|---|
| Demographic parity (statistical parity) | Selection rate | Possibly biased | Outreach, quotas, advertising |
| Equal opportunity | TPR among the qualified | Trustworthy | Opportunity allocation: hiring, lending, admissions |
| Equalized odds | TPR and FPR | Trustworthy | Where false positives also harm |
| Predictive parity | Precision | Trustworthy | Risk scores consumed as probabilities |
| Calibration by group | P(y=1 given score) | Trustworthy | Any score published as a probability |
| Treatment equality | FN/FP ratio | Trustworthy | Where the error mix is the harm |
| Individual fairness | Similar individuals treated similarly | -- | Requires a defensible similarity metric |
| Counterfactual fairness | Outcome unchanged if group flipped | -- | Requires a causal model |
The column that decides most real arguments is the third one. Demographic parity is the only definition that does not take the label at face value, which is exactly why it is the right choice when the label is a record of a biased decision -- and the wrong choice when the label is a genuine outcome you have no reason to distrust.
X, y, hist, g, names = make_hiring()
tr, te = slice(0, 14000), slice(14000, None)
f, _ = logistic(X[tr], y[tr]) # trained on the TRUE outcome
s = f(X[te]); yt, gt = y[te], g[te]
r = rates(yt, (s > 0.5).astype(int), gt)
print("FAIRNESS MEASURES ON ONE MODEL AT THRESHOLD 0.5")
print(f" {'quantity':34}{'group A':>10}{'group B':>10}{'ratio':>9}{'diff':>9}")
for lab, k in [("selection rate P(yhat=1)", "sel"),
("true positive rate (recall)", "tpr"),
("false positive rate", "fpr"),
("precision P(y=1|yhat=1)", "ppv"),
("base rate P(y=1)", "base")]:
a, b = r[0][k], r[1][k]
print(f" {lab:34}{a:10.3f}{b:10.3f}{b/a:9.2f}{b-a:+9.3f}")
print(f" {'group size':34}{r[0]['n']:10d}{r[1]['n']:10d}")
print("\nCALIBRATION: AMONG THOSE SCORED IN A BAND, WHO ACTUALLY SUCCEEDS?")
print(f" {'score band':16}{'A: n':>8}{'A: rate':>9}{'B: n':>8}{'B: rate':>9}")
edges = [0, .2, .4, .6, .8, 1.0]
for lo, hi in zip(edges[:-1], edges[1:]):
m = (s >= lo) & (s < hi)
a, b = m & (gt == 0), m & (gt == 1)
print(f" [{lo:.1f}, {hi:.1f}) {a.sum():8d}{yt[a].mean():9.3f}"
f"{b.sum():8d}{yt[b].mean():9.3f}")
print("\nWHICH CRITERIA DOES THIS MODEL SATISFY? (tolerance 0.05 absolute)")
for lab, a, b in [("demographic parity (equal selection rate)",
r[0]['sel'], r[1]['sel']),
("equal opportunity (equal TPR)", r[0]['tpr'], r[1]['tpr']),
("predictive parity (equal precision)",
r[0]['ppv'], r[1]['ppv']),
("FPR parity (equal false alarm)",
r[0]['fpr'], r[1]['fpr'])]:
verdict = 'PASS' if abs(a-b) <= 0.05 else 'FAIL'
print(f" {lab:44}{verdict} |diff| {abs(a-b):.3f}")
FAIRNESS MEASURES ON ONE MODEL AT THRESHOLD 0.5
quantity group A group B ratio diff
selection rate P(yhat=1) 0.383 0.164 0.43 -0.219
true positive rate (recall) 0.723 0.323 0.45 -0.400
false positive rate 0.184 0.075 0.41 -0.109
precision P(y=1|yhat=1) 0.697 0.705 1.01 +0.008
base rate P(y=1) 0.369 0.357 0.97 -0.012
group size 3853 2147
CALIBRATION: AMONG THOSE SCORED IN A BAND, WHO ACTUALLY SUCCEEDS?
score band A: n A: rate B: n B: rate
[0.0, 0.2) 1195 0.044 823 0.146
[0.2, 0.4) 818 0.248 769 0.378
[0.4, 0.6) 670 0.434 379 0.580
[0.6, 0.8) 602 0.626 157 0.745
[0.8, 1.0) 568 0.879 19 1.000
WHICH CRITERIA DOES THIS MODEL SATISFY? (tolerance 0.05 absolute)
demographic parity (equal selection rate) FAIL |diff| 0.219
equal opportunity (equal TPR) FAIL |diff| 0.400
predictive parity (equal precision) PASS |diff| 0.008
FPR parity (equal false alarm) FAIL |diff| 0.109
This single output contains the reason fairness debates are so confused. The same model, at the same threshold, on the same data, passes predictive parity almost perfectly and fails equal opportunity by 40 percentage points. A vendor reporting "our model has equal precision across groups" is telling the truth. So is a plaintiff reporting "qualified group-B candidates are selected less than half as often". Both describe this model.
The calibration table explains the mechanism. In every score band, group B's actual success rate is higher than group A's -- at scores of 0.2 to 0.4, B succeeds 37.8% of the time against A's 24.8%. The model systematically under-scores group B, because its features under-measure them. The base rates are nearly identical (0.369 and 0.357), so this is not a difference in the population; it is an error in the instrument.
Three practical rules follow.
Report the whole table, never a single number. Any one row can be cherry-picked. The set of rows is the honest description.
Report ratios and differences. A ratio of 0.43 sounds different from a difference of 0.219, and regulators use ratios (the four-fifths rule) while harm scales with differences. Give both.
State the tolerance and the group sizes. With 19 group-B applicants in the top score band, the "1.000" success rate is one number from nineteen people and carries no information. Every fairness metric needs a confidence interval.
Finally, note what none of these definitions do: they are all group properties, satisfied on average, and a model can satisfy every one of them while treating specific individuals arbitrarily. Individual and counterfactual fairness address that, but require, respectively, a defensible similarity metric and a causal model -- both of which are substantive assumptions you must be prepared to defend.
Key Takeaways
- Every group-fairness definition equalises one quantity from the confusion matrix across groups; they differ only in which quantity.
- Demographic parity is the only common definition that does not treat the label as trustworthy, which makes it right for biased labels and wrong for genuine outcomes.
- The same model can pass predictive parity and fail equal opportunity by 40 points; a single fairness number is always cherry-picked.
- Group-wise calibration reveals whether the score means the same thing for each group, and is the diagnostic closest to "is the instrument fair".
- Report differences and ratios together: regulators reason in ratios, harm scales with differences.
- Every fairness metric needs a group size and a confidence interval; small cells produce meaningless extremes.
- Group fairness says nothing about individuals; individual and counterfactual fairness require a similarity metric or a causal model.
- Choosing the criterion is a normative decision for accountable owners, not a modelling choice.
🧪 Practice
- Compute selection rate, TPR, FPR, precision, and base rate per group for a model of yours and present them as one table with ratios and differences.
- Build a group-wise calibration table in five score bands and state whether the score means the same thing for both groups.
- Find a threshold at which your model passes the four-fifths rule and one at which it fails, and report the accuracy at each.
- Add bootstrap confidence intervals to every fairness number and identify which comparisons are actually resolvable at your sample size.
- Interview: A vendor's model card states "demographic parity difference: 0.01". What three follow-up questions do you ask before accepting it? (Hint: think about the threshold, the population it was measured on, and which other criteria were not reported.)
Disparate Impact Assessment
Disparate impact is a legal concept before it is a statistical one. The distinction it rests on: disparate treatment is using a protected attribute in a decision, and disparate impact is a neutral-looking rule that nevertheless falls much more heavily on a protected group. A model that never sees race can still produce disparate impact, and in United States employment and lending law that alone can establish a prima facie case.
The standard screen is the four-fifths rule from the US Uniform Guidelines on Employee Selection Procedures: if the selection rate for a group is less than 80% of the rate for the most-selected group, that is treated as evidence of adverse impact requiring justification.
impact ratio = selection rate of group B / selection rate of group A
ratio >= 0.80 -> no adverse impact flagged
ratio < 0.80 -> flagged; employer must justify business necessity
AND show no less discriminatory alternative exists
The burden-shifting structure matters more than the arithmetic. Flagging does not end the enquiry: the defendant may show the procedure is job-related and consistent with business necessity, and the plaintiff may then show a less discriminatory alternative exists with similar validity. That last step is a modelling question, and it is where data scientists do the most useful work.
Three things must accompany any impact ratio.
Statistical significance. A ratio computed on 40 applicants is not evidence of anything.
Practical significance. The shortfall -- how many additional selections would reach parity -- is what quantifies the harm and is what courts and regulators actually respond to.
The threshold. The impact ratio is a function of where you cut the score, and it can move from passing to failing across ordinary operating choices.
from math import erf, sqrt
X, y, hist, g, names = make_hiring()
tr, te = slice(0, 14000), slice(14000, None)
f, _ = logistic(X[tr], y[tr])
s = f(X[te]); yt, gt = y[te], g[te]
def phi(z): return 0.5*(1 + erf(z/sqrt(2))) # standard normal CDF
def impact(sel_a, n_a, sel_b, n_b):
p = (sel_a*n_a + sel_b*n_b)/(n_a + n_b) # pooled selection rate
se = sqrt(p*(1-p)*(1/n_a + 1/n_b)) # two-proportion z test
z = (sel_b - sel_a)/se
return sel_b/sel_a, z, 2*(1 - phi(abs(z)))
print("FOUR-FIFTHS RULE AT SEVERAL THRESHOLDS")
print(f" {'thr':>5}{'sel A':>8}{'sel B':>8}{'ratio':>8}{'4/5?':>6}"
f"{'z':>8}{'p':>10}{'shortfall':>11}")
for thr in (0.20, 0.35, 0.50, 0.65):
yhat = (s > thr).astype(int)
a, b = yhat[gt==0].mean(), yhat[gt==1].mean()
na, nb = (gt==0).sum(), (gt==1).sum()
ratio, z, pv = impact(a, na, b, nb)
short = int(round((a-b)*nb)) # extra B selections to parity
print(f" {thr:5.2f}{a:8.3f}{b:8.3f}{ratio:8.2f}"
f"{'ok' if ratio >= 0.8 else 'FAIL':>6}{z:8.1f}{pv:10.1e}{short:11d}")
print("\nSMALL SAMPLES: THE SAME RATIO, DIFFERENT EVIDENTIAL WEIGHT")
print(f" {'n per group':>12}{'sel A':>8}{'sel B':>8}{'ratio':>8}{'z':>8}{'p':>10}")
for n in (40, 200, 1000, 5000):
ratio, z, pv = impact(0.40, n, 0.28, n)
print(f" {n:12}{0.40:8.2f}{0.28:8.2f}{ratio:8.2f}{z:8.1f}{pv:10.1e}")
print("\nLESS DISCRIMINATORY ALTERNATIVE SEARCH")
print(f" {'feature set':32}{'AUC':>7}{'ratio':>8}{'TPR gap':>10}")
for lab, cols in [("all three features", [0,1,2]), ("drop referral", [0,1]),
("drop prestige", [0,2]), ("test_score only", [0])]:
fs, _ = logistic(X[tr][:, cols], y[tr])
ss = fs(X[te][:, cols])
thr = np.quantile(ss, 1 - 0.30) # hold the hire rate at 30%
rr = rates(yt, (ss > thr).astype(int), gt)
print(f" {lab:32}{auc(yt, ss):7.3f}"
f"{rr[1]['sel']/rr[0]['sel']:8.2f}{rr[1]['tpr']-rr[0]['tpr']:+10.3f}")
FOUR-FIFTHS RULE AT SEVERAL THRESHOLDS
thr sel A sel B ratio 4/5? z p shortfall
0.20 0.690 0.617 0.89 ok -5.8 8.9e-09 157
0.35 0.523 0.326 0.62 FAIL -14.7 0.0e+00 423
0.50 0.383 0.164 0.43 FAIL -17.7 0.0e+00 471
0.65 0.266 0.055 0.21 FAIL -19.9 0.0e+00 452
SMALL SAMPLES: THE SAME RATIO, DIFFERENT EVIDENTIAL WEIGHT
n per group sel A sel B ratio z p
40 0.40 0.28 0.70 -1.1 2.6e-01
200 0.40 0.28 0.70 -2.5 1.1e-02
1000 0.40 0.28 0.70 -5.7 1.5e-08
5000 0.40 0.28 0.70 -12.7 0.0e+00
LESS DISCRIMINATORY ALTERNATIVE SEARCH
feature set AUC ratio TPR gap
all three features 0.820 0.42 -0.401
drop referral 0.820 0.40 -0.426
drop prestige 0.774 0.84 -0.066
test_score only 0.774 0.83 -0.068
Read the first table as a warning about how fragile a pass is. At a threshold of 0.20 this model passes the four-fifths rule at 0.89. At 0.65 it is at 0.21. Nothing about the model changed -- only the operating point, which is usually set by capacity, not by fairness. Any fairness claim must name the threshold it was measured at, and any monitoring must re-measure when the threshold moves. Note too that the flagged ratio and the shortfall do not peak together: the worst ratio is at 0.65, the largest shortfall at 0.50, because a stricter rule selects fewer people overall.
The second table is the small-sample caveat in numbers: the identical ratio of 0.70 is statistically unremarkable at 40 per group (p = 0.26) and overwhelming at 5,000 (p < 1e-30). Ratios from small slices should be reported with intervals and never acted on alone.
The third table is the most useful result in the topic. Dropping the
prestige feature moves the impact ratio from 0.42 to 0.84 -- through the
four-fifths threshold -- and shrinks the TPR gap from 40 points to 7, at a
cost of 0.046 AUC. That is a textbook less discriminatory alternative: a
model of materially similar validity with materially less adverse impact. If
this alternative exists and you deployed the other one, the business
necessity defence is in serious trouble.
The corresponding workflow:
- Fix the operating point that will actually be used, and measure the ratio there -- plus at the range of thresholds operations might drift to.
- Report ratio, z-statistic, p-value, shortfall count, and group sizes together.
- Enumerate alternatives: feature subsets, different targets, different model families, different thresholds.
- Plot validity against impact ratio across those alternatives and identify the frontier.
- Document why the chosen point on that frontier was chosen, and who chose it. That document is the business necessity argument.
Key Takeaways
- Disparate treatment is using a protected attribute; disparate impact is a neutral rule with heavily unequal effects, and a model that never sees the attribute can still produce it.
- The four-fifths rule flags a selection-rate ratio below 0.80 as evidence requiring justification, not as a verdict.
- The impact ratio depends strongly on the threshold; a model can pass at one operating point and fail badly at another.
- Always report statistical significance, shortfall count, and group sizes alongside the ratio.
- The worst ratio and the largest shortfall generally occur at different thresholds, so both must be reported.
- Identical ratios carry completely different evidential weight at different sample sizes.
- Searching for a less discriminatory alternative is the highest-value technical work here; a small validity loss for a large impact improvement usually defeats a business-necessity defence.
- Document the validity-impact frontier and who chose the operating point.
🧪 Practice
- Compute the impact ratio, z-statistic, and shortfall for your model at five thresholds spanning its operating range.
- For each single feature, drop it and record the resulting AUC and impact ratio; plot the frontier.
- Determine the minimum sample size per group at which a ratio of 0.75 would be statistically significant at p < 0.05.
- Compute the shortfall in people, not rates, for your worst-performing slice, and write one sentence a non-technical reader would understand.
- Interview: Your model passes the four-fifths rule overall but fails for one geography. How do you decide whether that matters? (Hint: think about the unit at which decisions are actually made and the sample size of the failing slice.)
Bias Mitigation Techniques
Mitigation techniques are organised by where in the pipeline they intervene, and that placement determines their cost, their legal posture, and what they can possibly fix. There are three families.
DATA ----[pre-processing]----> TRAINING ----[in-processing]----> MODEL
|
[post-processing]
v
DECISION
pre: change the data reweight, relabel, resample, repair features
in: change the objective fairness penalty, constrained optimisation,
adversarial debiasing
post: change the decision per-group thresholds, score calibration,
reject-option classification
| Family | Needs training access | Needs group at inference | Accuracy cost | Fixes root cause |
|---|---|---|---|---|
| Pre-processing | Yes | No | Low to moderate | Sometimes |
| In-processing | Yes | Only during training | Tunable | Partly |
| Post-processing | No | Usually yes | Low on ranking, high on accuracy | No |
That "needs group at inference" cell in the post-processing row is the most consequential practical fact in the topic. Applying a different threshold by group requires knowing each individual's group membership at decision time and acting on it -- which in many jurisdictions is disparate treatment, the thing that is most clearly prohibited. Post-processing is technically the easiest family and legally the hardest.
X, y, hist, g, names = make_hiring()
tr, te = slice(0, 14000), slice(14000, None)
Xtr, ytr, gtr = X[tr], y[tr], g[tr]
yt, gt = y[te], g[te]
mu, sd = Xtr.mean(0), Xtr.std(0)
Ztr, Zte = (Xtr-mu)/sd, (X[te]-mu)/sd
def fit(Z, yy, sw=None, gg=None, lam=0.0, iters=4000, lr=0.6):
sw = np.ones(len(yy)) if sw is None else sw*len(yy)/sw.sum()
w = np.zeros(Z.shape[1]); b = 0.0
for _ in range(iters):
p = 1/(1+np.exp(-(Z @ w + b)))
gr = sw*(p - yy)/len(yy)
if lam: # demographic-parity penalty
mA, mB = gg == 0, gg == 1
gap = p[mA].mean() - p[mB].mean() # squared gap in mean score
dp = p*(1-p) # d sigmoid / d logit
gr = gr + lam*2*gap*(dp*(mA/mA.sum() - mB/mB.sum()))
w -= lr*(Z.T @ gr); b -= lr*gr.sum()
return w, b
def report(lab, s, thrs):
yh = np.where(gt == 0, s > thrs[0], s > thrs[1]).astype(int)
r = rates(yt, yh, gt)
print(f" {lab:26}{auc(yt, s):8.3f}{(yh == yt).mean():10.3f}"
f"{r[1]['sel']/r[0]['sel']:11.2f}{r[1]['tpr']-r[0]['tpr']:+10.3f}"
f"{r[1]['ppv']-r[0]['ppv']:+9.3f}")
print("MITIGATION STRATEGIES, ALL AT A 30% OVERALL HIRE RATE")
print(f" {'strategy':26}{'AUC':>8}{'accuracy':>10}{'sel ratio':>11}"
f"{'TPR gap':>10}{'PPV gap':>9}")
w, b = fit(Ztr, ytr)
s0 = 1/(1+np.exp(-(Zte @ w + b))); t0 = np.quantile(s0, 0.70)
report("baseline", s0, (t0, t0))
# PRE: REWEIGHT SO GROUP AND LABEL ARE STATISTICALLY INDEPENDENT IN THE SAMPLE
sw = np.empty(len(ytr))
for gv in (0, 1):
for yv in (0, 1):
m = (gtr == gv) & (ytr == yv)
sw[m] = (gtr == gv).mean()*(ytr == yv).mean()/m.mean()
w, b = fit(Ztr, ytr, sw=sw)
s1 = 1/(1+np.exp(-(Zte @ w + b))); t1 = np.quantile(s1, 0.70)
report("pre: reweighting", s1, (t1, t1))
# IN: PENALISE THE GAP IN MEAN PREDICTED SCORE BETWEEN GROUPS
for lam in (2.0, 10.0):
w, b = fit(Ztr, ytr, gg=gtr, lam=lam)
s2 = 1/(1+np.exp(-(Zte @ w + b))); t2 = np.quantile(s2, 0.70)
report(f"in: parity penalty {lam:g}", s2, (t2, t2))
# POST: PER-GROUP THRESHOLDS
def tpr_at(s, mask, thr):
m = mask & (yt == 1); return ((s > thr) & m).sum()/m.sum()
cand = np.linspace(0.02, 0.98, 400)
target = tpr_at(s0, gt == 0, t0)
tB = min(cand, key=lambda c: abs(tpr_at(s0, gt == 1, c) - target))
report("post: equal-opportunity", s0, (t0, tB))
print(f" thresholds used: A {t0:.3f} B {tB:.3f}")
selA = (s0[gt == 0] > t0).mean()
tB2 = min(cand, key=lambda c: abs((s0[gt == 1] > c).mean() - selA))
report("post: demographic parity", s0, (t0, tB2))
print(f" thresholds used: A {t0:.3f} B {tB2:.3f}")
MITIGATION STRATEGIES, ALL AT A 30% OVERALL HIRE RATE
strategy AUC accuracy sel ratio TPR gap PPV gap
baseline 0.820 0.756 0.42 -0.401 +0.005
pre: reweighting 0.820 0.756 0.43 -0.398 +0.007
in: parity penalty 2 0.813 0.752 0.56 -0.280 +0.005
in: parity penalty 10 0.785 0.732 0.75 -0.147 -0.024
post: equal-opportunity 0.820 0.743 1.29 +0.002 -0.173
thresholds used: A 0.506 B 0.263
post: demographic parity 0.820 0.750 1.00 -0.116 -0.136
thresholds used: A 0.506 B 0.318
Four lessons are visible in that table.
Reweighting did nothing. It moved the selection ratio from 0.42 to 0.43. This is not a bug: the Kamiran-Calders reweighting scheme makes group and label independent in the training sample, and in this dataset they already are (base rates 0.369 and 0.357). The disparity lives in the features, and reweighting does not touch features. Match the mitigation to the diagnosed source of bias -- a technique aimed at label imbalance cannot repair a biased measuring instrument.
In-processing gives a dial with a real price. A penalty weight of 2 halves the TPR gap for 0.007 AUC; a weight of 10 cuts it to 0.147 for 0.035 AUC. This monotone trade is the honest shape of the problem, and the right way to present it to a decision-maker is as a curve, not a single number.
Post-processing is free in ranking quality and not free elsewhere. AUC is unchanged at 0.820 because the score is unchanged; only the cut moved. But equalising TPR overshoots the selection ratio to 1.29 and opens a precision gap of -0.173: group-B hires are now less likely to succeed than group-A hires. Someone will notice that, and you need to have decided in advance whether it is acceptable.
Every row satisfies a different definition, and none satisfies all of them. That is not a failure of the techniques; it is the next topic.
Beyond these three families, two other interventions are often the most effective and are frequently skipped:
- Change the target. Predicting the outcome you care about instead of a biased proxy is usually worth more than any algorithmic fix.
- Change the features. Removing or repairing the specific feature that
carries the disparity -- as the previous topic's
prestigeresult showed -- can move the impact ratio further than any reweighting scheme, and it is far easier to explain and defend.
A note on adversarial debiasing, the most common in-processing method in deep learning: train the predictor jointly with an adversary that tries to recover the protected attribute from the representation, and penalise the predictor when the adversary succeeds. It is powerful and notoriously unstable, needs careful tuning, and its guarantee is only as good as the adversary's capacity -- a weak adversary certifies nothing.
Key Takeaways
- Mitigations are classified by where they intervene: pre-processing on the data, in-processing on the objective, post-processing on the decision.
- Post-processing usually requires the protected attribute at decision time, which can itself constitute disparate treatment.
- Reweighting targets dependence between group and label; it does nothing when the disparity lives in the features.
- Match the technique to the diagnosed source of bias rather than reaching for a standard library call.
- In-processing penalties give a tunable accuracy-fairness trade that should be presented as a curve, not a point.
- Post-processing leaves ranking quality untouched but shifts other fairness quantities, often opening a precision gap in the other direction.
- Changing the target or repairing the offending feature often beats every algorithmic mitigation and is easier to defend.
- Adversarial debiasing is powerful, unstable, and certifies nothing beyond what its adversary could detect.
🧪 Practice
- Implement reweighting on a dataset where group and label ARE dependent and verify it moves the selection ratio there.
- Sweep an in-processing fairness penalty over a range and plot AUC against the selection ratio to produce the trade-off frontier.
- Apply per-group thresholds to equalise TPR and report every other fairness quantity before and after.
- Compare dropping the single most disparity-carrying feature against your best in-processing result, on both accuracy and impact ratio.
- Interview: Legal tells you that you may not use group membership at decision time. Which mitigation families remain available, and what do you lose? (Hint: think about which family needs the attribute at training time only, and whether any post-processing variant can avoid it.)
Tradeoffs Between Fairness Criteria
Once you have several fairness definitions and several mitigations, the natural instinct is to satisfy all of them. That instinct runs into a theorem: when base rates differ between groups, calibration, equal true positive rates, and equal false positive rates cannot all hold simultaneously unless the classifier is perfect. This is the impossibility result of Kleinberg, Mullainathan and Raghavan, and independently Chouldechova, and it is arithmetic, not a limitation of current methods.
The intuition takes one line of algebra. Precision relates to base rate and error rates by
p * TPR
PPV = --------------------------------- where p = base rate
p * TPR + (1 - p) * FPR
Fix TPR and FPR to be equal across groups. Then PPV is a strictly increasing function of p alone -- so if p differs, PPV must differ. Equalising error rates forces unequal precision; equalising precision forces unequal error rates. There is no third option.
X, y, hist, g, names = make_hiring()
tr, te = slice(0, 14000), slice(14000, None)
# USE THE HISTORICAL LABEL: ITS BASE RATE DIFFERS BY GROUP, WHICH IS THE
# CONDITION UNDER WHICH THE CRITERIA BECOME MUTUALLY EXCLUSIVE.
f, _ = logistic(X[tr], hist[tr])
s = f(X[te]); yt, gt = hist[te], g[te]
print(f"base rates differ: A {yt[gt==0].mean():.3f} B {yt[gt==1].mean():.3f}")
def at(ta, tb):
return rates(yt, np.where(gt == 0, s > ta, s > tb).astype(int), gt)
def gapline(lab, ta, tb):
r = at(ta, tb)
print(f" {lab:26}{ta:8.3f}{tb:8.3f}"
f"{r[1]['sel']/max(r[0]['sel'], 1e-9):11.2f}"
f"{r[1]['tpr']-r[0]['tpr']:+9.3f}{r[1]['fpr']-r[0]['fpr']:+9.3f}"
f"{r[1]['ppv']-r[0]['ppv']:+9.3f}")
cand = np.linspace(0.02, 0.98, 300)
base = np.quantile(s, 0.70)
r0 = at(base, base)
print("\nENFORCE ONE CRITERION, MEASURE WHAT BREAKS")
print(f" {'target':26}{'thr A':>8}{'thr B':>8}{'sel ratio':>11}"
f"{'TPR gap':>9}{'FPR gap':>9}{'PPV gap':>9}")
gapline("single threshold", base, base)
for lab, key in [("demographic parity", "sel"), ("equal opportunity (TPR)", "tpr"),
("FPR parity", "fpr"), ("predictive parity (PPV)", "ppv")]:
tb = min(cand, key=lambda c: abs(at(base, c)[1][key] - r0[0][key]))
gapline(lab, base, tb)
tb = min(cand, key=lambda c: (abs(at(base, c)[1]['tpr'] - r0[0]['tpr'])
+ abs(at(base, c)[1]['fpr'] - r0[0]['fpr'])))
gapline("equalized odds (both)", base, tb)
print("\nTHE ALGEBRA: PPV = p*TPR / (p*TPR + (1-p)*FPR)")
print(f" {'':8}{'base rate p':>13}{'TPR':>7}{'FPR':>7}{'implied PPV':>13}")
for lab, p in (("group A", yt[gt==0].mean()), ("group B", yt[gt==1].mean())):
tpr, fpr = 0.70, 0.20 # identical error rates
print(f" {lab:8}{p:13.3f}{tpr:7.2f}{fpr:7.2f}"
f"{p*tpr/(p*tpr + (1-p)*fpr):13.3f}")
print(" Identical TPR and FPR, different base rates -> PPV cannot match.")
base rates differ: A 0.431 B 0.129
ENFORCE ONE CRITERION, MEASURE WHAT BREAKS
target thr A thr B sel ratio TPR gap FPR gap PPV gap
single threshold 0.548 0.548 0.40 -0.042 +0.003 -0.268
demographic parity 0.548 0.081 1.01 +0.179 +0.236 -0.581
equal opportunity (TPR) 0.548 0.498 0.44 +0.001 +0.012 -0.286
FPR parity 0.548 0.559 0.39 -0.057 +0.000 -0.262
predictive parity (PPV) 0.548 0.913 0.14 -0.434 -0.054 +0.000
equalized odds (both) 0.548 0.502 0.43 +0.001 +0.011 -0.285
THE ALGEBRA: PPV = p*TPR / (p*TPR + (1-p)*FPR)
base rate p TPR FPR implied PPV
group A 0.431 0.70 0.20 0.726
group B 0.129 0.70 0.20 0.341
Identical TPR and FPR, different base rates -> PPV cannot match.
Every row is a legitimate policy and every row is unfair by some other row's standard. Look at the two extremes. Enforcing demographic parity requires dropping group B's threshold to 0.081, which achieves an equal selection rate and a false positive rate 23.6 points higher for group B -- a large number of group-B candidates selected who will not succeed, which harms them too. Enforcing predictive parity requires raising group B's threshold to 0.913, which achieves equal precision and a TPR gap of -0.434: qualified group-B candidates are selected at a fraction of the rate of equally qualified group-A candidates.
The algebra block shows why no cleverness escapes this. With base rates of 0.431 and 0.129 and identical error rates, precision must be 0.726 for one group and 0.341 for the other.
So the trade-offs are structural, and the real questions become practical ones:
| Question | Consequence for the choice |
|---|---|
| Do you believe the label? | If not, demographic parity; if yes, an error-rate criterion |
| Who is harmed by a false positive? | If the individual (a loan they cannot repay), do not maximise selection |
| Who is harmed by a false negative? | If the individual (a job they deserved), prioritise equal TPR |
| Is the score consumed as a probability downstream? | Then calibration is close to mandatory |
| Are base rates genuinely different, or an artefact? | If artefact, fix measurement first; criteria stop conflicting as base rates converge |
That last row is the escape hatch, and it is the one worth pursuing. The impossibility bites because base rates differ. If the difference is an artefact of measurement or of a biased label, then repairing the instrument does not just improve one metric -- it dissolves the conflict. In our own dataset with the true label, base rates were 0.369 and 0.357, and the criteria are nearly reconcilable. With the biased historical label they are 0.431 and 0.129, and they are not.
And there is a second escape hatch: improve the model. The impossibility holds for imperfect classifiers, and the tension shrinks as accuracy rises. Better features -- especially features that measure the construct equally well for both groups -- reduce the conflict in a way that no post-processing can.
Key Takeaways
- When base rates differ, calibration, equal TPR, and equal FPR cannot hold together unless the classifier is perfect; this is arithmetic, not a technical limitation.
- Precision is determined by base rate and error rates, so equalising error rates forces unequal precision and vice versa.
- Enforcing demographic parity on unequal base rates raises the false positive rate for the disadvantaged group, which is itself a harm.
- Enforcing predictive parity on unequal base rates collapses the disadvantaged group's true positive rate.
- There is no fairness criterion that is correct in general; the choice follows from who is harmed by which error and whether the label is trustworthy.
- Calibration becomes near-mandatory when a score is consumed downstream as a probability.
- The conflict shrinks as base rates converge, so repairing biased labels and measurement dissolves part of the problem.
- Better accuracy also shrinks the conflict; improving features is the only intervention that helps every criterion at once.
🧪 Practice
- Reproduce the impossibility numerically: fix TPR and FPR equal across two groups with base rates 0.1 and 0.4 and compute both precisions.
- For your own model, find the per-group thresholds that equalise each criterion in turn, and tabulate what each choice does to the others.
- Write a one-page memo recommending a criterion for a concrete decision, naming who bears each error type.
- Simulate reducing the base rate gap (by fixing a biased label) and show the conflict between criteria shrinking.
- Interview: A stakeholder wants both equal selection rates and equal precision across groups. Explain why you cannot deliver both, without using the word "theorem". (Hint: talk about what the selection rate does to who ends up in the selected pool when the groups' qualified fractions differ.)
Auditing Deployed Models
A fairness assessment made at training time expires. The population changes, the sourcing changes, upstream features change meaning, thresholds get adjusted for capacity, and the model's own decisions reshape the data it will be retrained on. A model that passed every check in March can be failing badly in September without a single line of code having changed.
Auditing is therefore a monitoring discipline, and it has the same structure as the model monitoring in the previous chapter -- with three complications that make it harder.
Ground truth is delayed or missing. You learn whether a hire succeeded a year later, and you never learn whether a rejected applicant would have succeeded. Outcome-based metrics (TPR, precision) lag; decision-based metrics (selection rate) are available immediately. Monitor both, at their own cadences.
Slices are small. The whole point is to look at subgroups, and subgroups have few rows, so every number is noisy. Without intervals you will chase phantoms and miss real regressions.
Intersections multiply. Two protected attributes with four levels each make sixteen cells; the group-level number can look fine while a cell is in serious trouble.
X, y, hist, g, names = make_hiring()
tr, te = slice(0, 14000), slice(14000, None)
f, _ = logistic(X[tr], y[tr])
rng = np.random.default_rng(11)
Xte, yt, gt = X[te].copy(), y[te].copy(), g[te]
month = rng.integers(0, 6, len(yt))
# SIMULATE A SOURCING CHANGE: FROM MONTH 3, GROUP B APPLICANTS ARRIVE WITH
# SYSTEMATICALLY LOWER PRESTIGE SCORES. THE MODEL IS UNCHANGED.
drift = (month >= 3) & (gt == 1)
Xte[drift, 1] -= 0.8
s = f(Xte)
thr = np.quantile(f(X[tr]), 0.70) # the threshold set in March
def boot_ratio(mask, reps=300):
out = []; idx = np.where(mask)[0]
for _ in range(reps):
b = rng.choice(idx, len(idx), replace=True) # resample the slice
yh = (s[b] > thr).astype(int); gb = gt[b]
a, bb = yh[gb == 0].mean(), yh[gb == 1].mean()
out.append(bb/a if a > 0 else np.nan)
return np.nanpercentile(out, [2.5, 50, 97.5])
print("MONTHLY DISPARATE IMPACT RATIO WITH A 95% BOOTSTRAP INTERVAL")
print(f" {'month':>6}{'n':>7}{'n_B':>6}{'ratio':>8}{'95% CI':>18}{'flag':>7}")
for m in range(6):
mask = month == m
lo, md, hi = boot_ratio(mask)
flag = "ALERT" if hi < 0.8 else ("watch" if lo < 0.8 else "ok")
print(f" {m:6}{mask.sum():7}{(mask & (gt==1)).sum():6}{md:8.2f}"
f" [{lo:.2f}, {hi:.2f}]{flag:>8}")
print("\nWHY SMALL SLICES CANNOT BE MONITORED THE SAME WAY")
print(f" {'slice size':>12}{'ratio':>8}{'95% CI width':>15}")
for n in (30, 100, 400, 1600):
sub = rng.choice(len(yt), n, replace=False)
mask = np.zeros(len(yt), bool); mask[sub] = True
lo, md, hi = boot_ratio(mask, reps=200)
print(f" {n:12}{md:8.2f}{hi-lo:15.2f}")
print("\nINTERSECTIONAL SLICES: THE GROUP-LEVEL NUMBER HIDES THE WORST CELL")
# A SECOND ATTRIBUTE THE MODEL NEVER SEES: APPLICANTS FROM NON-TARGET
# REGIONS HAVE LOWER PRESTIGE SCORES, AND THE EFFECT COMPOUNDS FOR GROUP B.
X2 = X[te].copy()
region = (rng.random(len(yt)) < 0.4) # 1 = non-target region
X2[region, 1] -= 0.6
X2[region & (gt == 1), 1] -= 0.7 # the compounding part
yh = (f(X2) > thr).astype(int)
base = yh[(gt == 0) & ~region].mean()
print(f" {'slice':24}{'n':>7}{'sel rate':>10}{'ratio vs A/target':>20}")
for lab, m in [("A, target region", (gt==0) & ~region),
("A, other region", (gt==0) & region),
("B, target region", (gt==1) & ~region),
("B, other region", (gt==1) & region)]:
print(f" {lab:24}{m.sum():7}{yh[m].mean():10.3f}{yh[m].mean()/base:20.2f}")
print(f" group-level only: A {yh[gt==0].mean():.3f} B {yh[gt==1].mean():.3f}"
f" ratio {yh[gt==1].mean()/yh[gt==0].mean():.2f}")
MONTHLY DISPARATE IMPACT RATIO WITH A 95% BOOTSTRAP INTERVAL
month n n_B ratio 95% CI flag
0 1016 391 0.42 [0.33, 0.52] ALERT
1 950 324 0.46 [0.36, 0.61] ALERT
2 973 359 0.43 [0.34, 0.54] ALERT
3 1034 352 0.14 [0.08, 0.22] ALERT
4 1045 366 0.11 [0.06, 0.18] ALERT
5 982 355 0.15 [0.09, 0.22] ALERT
WHY SMALL SLICES CANNOT BE MONITORED THE SAME WAY
slice size ratio 95% CI width
30 0.51 1.67
100 0.14 0.42
400 0.34 0.36
1600 0.35 0.14
INTERSECTIONAL SLICES: THE GROUP-LEVEL NUMBER HIDES THE WORST CELL
slice n sel rate ratio vs A/target
A, target region 2303 0.385 1.00
A, other region 1550 0.278 0.72
B, target region 1277 0.156 0.40
B, other region 870 0.016 0.04
group-level only: A 0.342 B 0.099 ratio 0.29
The monthly table shows the regression clearly: a stable ratio around 0.43
for three months, then a drop to 0.11-0.15 once the upstream sourcing change
lands. Nothing about the model changed. What changed was the input
distribution for one group, and the fixed threshold turned that into a step
change in selection. This is why fairness monitoring must be continuous and
must be tied to feature drift monitoring: the root cause is visible in the
prestige distribution weeks before anyone queries the outcome data.
The confidence-interval table is the operational discipline. At 30 rows the 95% interval is 1.67 wide -- the ratio is essentially unmeasured. At 1,600 rows it is 0.14 wide and you can act on it. Set a minimum slice size for alerting and report "insufficient data" rather than a number below it.
The intersectional table is the finding you would otherwise miss. At the group level the ratio is 0.29 -- already bad. But the harm is not spread evenly: group-B applicants in the target region are selected at 0.40 of the reference rate, while group-B applicants from other regions are at 0.04. An audit that reported only the group-level number would have described a uniform 0.29 disadvantage that no individual actually experiences.
A workable audit programme:
- Log what you need. Score, threshold, decision, model version, explanation configuration, and group attributes (stored separately with restricted access, since you need them to audit and must not use them to decide).
- Define the slice set in advance -- protected groups, their pairwise intersections, geography, channel, and time -- so you are not accused of slicing until something looks acceptable.
- Monitor decision metrics daily and outcome metrics at their natural lag, with the lag stated.
- Alert on intervals, not points, with a minimum slice size below which you report insufficient data.
- Re-audit on every change to the model, the threshold, the feature pipeline, or the upstream data source. A threshold change is a fairness change.
- Keep a decision log: every alert, its investigation, and its resolution. This is the artifact a regulator asks for, and reconstructing it later is impossible.
- Watch the feedback loop. If the model's decisions produce the next training set, compare the deployed population with the applicant population every cycle.
One organisational point that matters more than any of the above: the people who build the model should not be the only people who audit it. Independent review, with access to the data and a mandate to publish findings, is the control that makes the rest credible.
Key Takeaways
- Fairness assessments expire; population, sourcing, thresholds, and upstream features all change after deployment.
- Decision metrics are available immediately, outcome metrics lag, and both must be monitored at their own cadence with the lag documented.
- A fairness regression can occur with no model change at all, driven by a shift in one group's input distribution against a fixed threshold.
- Tie fairness monitoring to feature drift monitoring, since drift is visible before outcomes are.
- Every slice metric needs a confidence interval and a minimum sample size; below it, report insufficient data rather than a number.
- Intersectional cells can be far worse than either group-level number, so define the slice set in advance and audit the cells.
- Log score, threshold, model version, and decision for every case, and keep a decision log of every alert and its resolution.
- Independent audit, separate from the build team, is what makes the rest of the programme credible.
🧪 Practice
- Compute your model's impact ratio per month with bootstrap intervals and identify the minimum slice size at which the interval is narrower than 0.15.
- Simulate an upstream feature shift affecting one group and show how long it takes your current alerting to detect it.
- Enumerate every intersectional cell for two protected attributes in your data, report selection rates, and flag the cells below the minimum size.
- Write the alerting policy: which metrics, which slices, which thresholds, which minimum sample sizes, and who is paged.
- Interview: Your fairness dashboard is green but a journalist presents three cases that look clearly unfair. How do you respond? (Hint: consider what a group average can be true of while individual cases are still indefensible, and which slice the three cases share.)
<a id="163-privacy-and-security"></a>
16.3 Privacy and Security
Data about people is the raw material of most data science, and it is also a liability that grows with every copy; this section covers what actually counts as identifying, why removing names is not anonymisation, the one technique that offers a mathematical privacy guarantee, how to train without centralising data, the attacks that target models rather than databases, and the everyday engineering controls that prevent most real incidents. The unifying idea is that privacy is a property of a system's whole behaviour, not of a column, and that it must be designed in rather than certified afterwards.
Personally Identifiable Information
The intuitive model of personal data -- "the name column is sensitive, the rest is fine" -- is wrong in a way that has caused many of the largest privacy failures on record. Identification does not require an identifier. It requires only that the combination of attributes you publish be unique enough to single someone out when joined against anything else.
Three categories matter, and the second is the dangerous one:
| Category | Examples | Risk |
|---|---|---|
| Direct identifiers | Name, national ID, email, phone, account number, device ID | Obvious; usually removed |
| Quasi-identifiers | Postcode, date of birth, sex, job title, employer, timestamps, GPS traces, browser fingerprint | Individually harmless, jointly identifying |
| Sensitive attributes | Diagnosis, salary, religion, sexual orientation, criminal record, biometrics | The thing you are protecting; special legal categories |
The canonical result: Latanya Sweeney showed that the combination of five-digit postcode, date of birth, and sex uniquely identifies roughly 87% of the United States population. The Netflix Prize dataset, released with all identifiers removed, was de-anonymised by joining viewing histories with public IMDb ratings. Anonymised taxi trip records were linked to named celebrities using timestamped paparazzi photographs.
"de-identified" release public voter roll
+----------------------+ +-------------------------+
| zip | dob | sex | dx | <--JOIN--| zip | dob | sex | NAME |
+----------------------+ +-------------------------+
the join key was never removed, because it was never
recognised as a key
import numpy as np, pandas as pd
rng = np.random.default_rng(2)
N = 50000
df = pd.DataFrame({
"zip": rng.choice([f"{z:05d}" for z in range(10001, 10201)], N),
"birth_year": rng.integers(1940, 2006, N),
"birth_month": rng.integers(1, 13, N),
"sex": rng.choice(["F", "M"], N),
"diagnosis": rng.choice(["asthma", "diabetes", "hypertension", "none"],
N, p=[.12, .13, .25, .50]),
})
print(f"a 'de-identified' table of {N:,} people, no name, no id\n")
print(f" {'quasi-identifier set':44}{'unique (k=1)':>14}{'k<5':>8}")
for cols in [("zip",), ("zip", "sex"), ("zip", "birth_year"),
("zip", "birth_year", "sex"),
("zip", "birth_year", "birth_month", "sex")]:
g = df.groupby(list(cols), observed=True).size()
print(f" {' + '.join(cols):44}{(g == 1).sum()/N:13.1%}"
f"{g[g < 5].sum()/N:8.1%}")
print("\nNo single column is an identifier. The COMBINATION is.")
print("\nAND THE DIAGNOSIS LEAKS EVEN WHEN THE PERSON DOES NOT:")
g = df.groupby(["zip", "birth_year", "sex"], observed=True)["diagnosis"]
homog = g.nunique() == 1 # every member shares a value
print(f" groups where every member shares one diagnosis: {homog.mean():.1%}")
print(f" people in such groups: {g.size()[homog].sum()/N:.1%}")
a 'de-identified' table of 50,000 people, no name, no id
quasi-identifier set unique (k=1) k<5
zip 0.0% 0.0%
zip + sex 0.0% 0.0%
zip + birth_year 2.2% 47.8%
zip + birth_year + sex 14.9% 88.0%
zip + birth_year + birth_month + sex 85.5% 100.0%
No single column is an identifier. The COMBINATION is.
AND THE DIAGNOSIS LEAKS EVEN WHEN THE PERSON DOES NOT:
groups where every member shares one diagnosis: 47.8%
people in such groups: 30.1%
Read down that first column. Postcode alone identifies nobody. Postcode plus birth year plus sex uniquely identifies 14.9% of the table, and adding birth month takes it to 85.5% -- reproducing Sweeney's result on synthetic data. Nothing was added to the release except precision.
The second block makes the subtler point: even where individuals are not unique, the sensitive value can still be determined. If everyone in your quasi-identifier group has the same diagnosis, learning your group is enough.
Practical consequences for how you handle data:
- Classify columns, not tables. Every column gets a label -- direct identifier, quasi-identifier, sensitive, or ordinary -- and that label drives access, masking, retention, and logging. Do this in the schema, not in a wiki.
- Count uniqueness before you release anything. The query in the code above takes one line and is the single most informative privacy check available.
- Treat high-dimensional behavioural data as identifying by default. Purchase histories, location traces, app-usage sequences, and clickstreams are effectively unique per person; there is no small set of columns to remove.
- Remember free text and images. Notes fields, uploaded documents, and photographs carry identifiers that no column-level policy catches.
- Assume an auxiliary dataset exists. Your release will be joined against voter rolls, social media, data brokers, and leaked breaches. The question is never "is this anonymous?" but "anonymous against what?"
A final legal note, because the definitions differ in ways that change what you must do. PII (US-centric) generally refers to information that identifies a specific individual. Personal data under the GDPR is much broader: any information relating to an identified or identifiable natural person, which includes IP addresses, cookie IDs, and pseudonymised records where re-identification remains possible. PHI under HIPAA covers health information along with 18 enumerated identifier types. If you build to the narrowest definition, you will be non-compliant under the broadest.
Key Takeaways
- Identification does not require an identifier; a sufficiently precise combination of ordinary attributes singles people out.
- Postcode, date of birth, and sex identify the large majority of a population; here 85.5% of rows were unique on four coarse columns.
- Quasi-identifiers are the dangerous category precisely because each one looks harmless in isolation.
- Even without uniqueness, a homogeneous group leaks the sensitive value to anyone who knows the group.
- High-dimensional behavioural data (locations, purchases, clickstreams) is identifying by default and cannot be fixed by dropping columns.
- Classify columns in the schema, and count uniqueness before every release.
- Privacy is always relative to the auxiliary data an attacker can join against, so "anonymous" is meaningless without naming that threat model.
- PII, personal data (GDPR), and PHI (HIPAA) have different scopes; build to the broadest that applies.
🧪 Practice
- For a dataset you have, compute the fraction of rows that are unique on each candidate quasi-identifier combination of two and three columns.
- Write a column classification for that dataset and note which columns you had to think about longest.
- Identify one auxiliary dataset that is publicly available and could be joined to yours, and name the join keys.
- Take a free-text field and measure how often it contains something that would identify a person.
- Interview: A colleague proposes publishing "anonymous" hourly step counts from a fitness app, with user IDs replaced by random numbers. What do you say? (Hint: think about what a sequence of daily behaviour looks like across a population, and how many days it takes to become unique.)
Anonymization and Pseudonymization
These two words are used interchangeably in conversation and mean completely different things in law and in practice. The difference decides whether a dataset is still regulated personal data.
Pseudonymisation replaces identifiers with surrogate values while keeping the ability to link records -- and, crucially, keeping the possibility of re-identification, whether through a key you hold or through the quasi-identifiers you left behind. Under the GDPR, pseudonymised data is still personal data. It is a risk-reduction control, not an exit from the regulation.
Anonymisation means re-identification is no longer reasonably possible by anyone, including you. Truly anonymised data falls outside the regulation -- and genuine anonymisation is much harder than people expect, because it requires destroying enough information that the release stops being useful for some of the analyses you wanted.
ORIGINAL PSEUDONYMISED ANONYMISED (k=5, generalised)
Alice, 10001, a91f3c, 10001, *, 100**, 1980-1989, F,
1984-03, F, 1984-03, F, asthma
asthma asthma
^ ^
re-linkable with no individual row corresponds
the key or by join to one identifiable person
The tool of anonymisation is generalisation: replace a precise value with a coarser one until every combination of quasi-identifiers is shared by at least k people. That is k-anonymity. The complementary tools are suppression (drop rare rows entirely) and perturbation (add noise).
import numpy as np, pandas as pd, hashlib
rng = np.random.default_rng(2)
N = 50000
zips = np.array([f"{z:05d}" for z in range(10001, 10201)])
# TRUE PREVALENCE VARIES BY AREA AND AGE -- THIS IS THE SIGNAL GENERALISATION
# WILL DESTROY, AND THE REASON ANONYMISATION HAS A COST.
area_rate = rng.uniform(0.10, 0.40, len(zips))
zi = rng.integers(0, len(zips), N)
birth = rng.integers(1940, 2006, N)
p_hyp = np.clip(area_rate[zi] + 0.004*(1980-birth), 0.02, 0.95)
df = pd.DataFrame({"zip": zips[zi], "birth_year": birth,
"sex": rng.choice(["F", "M"], N),
"hypertension": (rng.random(N) < p_hyp).astype(int)})
df["z3"] = df["zip"].str[:3]; df["z2"] = df["zip"].str[:2]
df["band5"] = (df["birth_year"]//5)*5
df["band10"] = (df["birth_year"]//10)*10
print("PSEUDONYMISATION IS NOT ANONYMISATION")
salt = b"per-dataset-salt"
h = lambda v: hashlib.sha256(salt + str(v).encode()).hexdigest()[:12]
df["zip_hash"] = df["zip"].map(h)
rainbow = {h(z): z for z in zips} # attacker hashes the whole domain
print(f" hashed zip looks like: {df['zip_hash'].iloc[0]}")
print(f" domain size to brute force: {len(rainbow)}")
print(f" share of hashes reversed by a table lookup: "
f"{(df['zip_hash'].map(rainbow) == df['zip']).mean():.1%}")
print(" A hash of a low-entropy value is a pseudonym, not a secret.\n")
print("GENERALISATION: TRADING PRECISION FOR GROUP SIZE")
print(f" {'scheme':28}{'min k':>8}{'cells k<5':>11}{'people k<5':>12}"
f"{'area MAE':>10}")
truth = df.groupby("zip", observed=True)["hypertension"].mean()
levels = [("zip5 + year + sex", ["zip", "birth_year", "sex"], "zip"),
("zip3 + year + sex", ["z3", "birth_year", "sex"], "z3"),
("zip3 + 5y band + sex", ["z3", "band5", "sex"], "z3"),
("zip2 + 10y band + sex", ["z2", "band10", "sex"], "z2")]
for lab, cols, geo in levels:
g = df.groupby(cols, observed=True).size()
est = df.groupby(geo, observed=True)["hypertension"].mean()
pred = df[geo].map(est) # best guess per original postcode
mae = np.abs(pred.groupby(df["zip"]).mean() - truth).mean()
print(f" {lab:28}{g.min():8d}{(g < 5).mean():11.1%}"
f"{g[g < 5].sum()/N:12.1%}{mae:10.4f}")
print("\nK-ANONYMITY IS NOT ENOUGH: THE HOMOGENEITY ATTACK")
# ONE CELL IS A RETIREMENT COMMUNITY: LARGE, AND ALMOST ALL POSITIVE.
mask = (df["z3"] == "101") & (df["band10"] == 1940)
df.loc[mask, "hypertension"] = (rng.random(mask.sum()) < 0.97).astype(int)
g = df.groupby(["z3", "band10", "sex"], observed=True)["hypertension"]
wk = g.mean().idxmax()
print(f" smallest cell size k = {g.size().min()} (well past 5-anonymity)")
print(f" worst cell {(wk[0], int(wk[1]), wk[2])}: n={g.size()[wk]}, "
f"positive rate {g.mean()[wk]:.1%}")
PSEUDONYMISATION IS NOT ANONYMISATION
hashed zip looks like: 8b66e715562c
domain size to brute force: 200
share of hashes reversed by a table lookup: 100.0%
A hash of a low-entropy value is a pseudonym, not a secret.
GENERALISATION: TRADING PRECISION FOR GROUP SIZE
scheme min k cells k<5 people k<5 area MAE
zip5 + year + sex 1 95.0% 87.9% 0.0000
zip3 + year + sex 1 27.2% 0.4% 0.0792
zip3 + 5y band + sex 2 3.6% 0.0% 0.0792
zip2 + 10y band + sex 2230 0.0% 0.0% 0.0793
K-ANONYMITY IS NOT ENOUGH: THE HOMOGENEITY ATTACK
smallest cell size k = 7 (well past 5-anonymity)
worst cell ('101', 1940, 'F'): n=1957, positive rate 97.4%
The first block is the mistake that appears in production code constantly: hashing a value with a low-entropy domain protects nothing. There are only 200 postcodes, so an attacker hashes all 200 and reverses every record. The same applies to phone numbers, national ID numbers with checksums, email addresses from a known list, and any small categorical. A hash is only a secret when the input space is too large to enumerate -- which is why real tokenisation uses a keyed function (HMAC) with the key stored separately, as the last topic in this section shows.
The generalisation block quantifies the cost. Going from full postcode to a three-digit prefix eliminates almost all small cells (87.9% of people were in cells of fewer than 5, now 0.4%) -- and it takes the achievable accuracy of a per-postcode prevalence estimate from exact to a mean absolute error of 0.079, which on rates ranging from 0.10 to 0.40 destroys most of the geographic signal. Anonymisation is not free; it trades a specific analysis away. The right way to run this decision is to name the analyses the release must support and check them at each generalisation level.
The last block is the reason k-anonymity alone is not a sufficient standard. Every cell here has at least 7 members, so the release is 5-anonymous, and yet one cell of 1,957 people has a 97.4% positive rate. Knowing only that someone belongs to that cell tells you their diagnosis with near certainty. This is the homogeneity attack, and it is what the successor definitions address:
| Definition | Guarantees | Attack it blocks |
|---|---|---|
| k-anonymity | Each quasi-identifier cell has >= k members | Direct re-identification by join |
| l-diversity | Each cell contains >= l distinct sensitive values | Homogeneity attack |
| t-closeness | Each cell's sensitive distribution is within t of the global one | Skewness / background-knowledge attack |
| Differential privacy | Bounded change in output from any one person's data | All of the above, plus unknown future attacks |
Each row up that table is strictly stronger and strictly more costly in utility. And the crucial limitation of the first three is that they are syntactic: they constrain the shape of a released table, and they do not compose. Two separately k-anonymous releases of the same population can jointly re-identify people. That non-composition is exactly what the next topic fixes.
Key Takeaways
- Pseudonymised data is still personal data under the GDPR; anonymised data is outside it, and genuine anonymisation is hard.
- Hashing a low-entropy value protects nothing: the attacker enumerates the domain, and here recovered 100% of postcodes.
- Tokenisation needs a keyed function with the key stored separately, not a plain hash.
- k-anonymity is achieved by generalisation and suppression, and it always costs a specific analysis; name those analyses before choosing the level.
- k-anonymity does not prevent learning the sensitive value: a homogeneous cell leaks it regardless of size.
- l-diversity and t-closeness patch the homogeneity and skewness attacks at further utility cost.
- All of these are syntactic properties of one table and do not compose; multiple releases can jointly re-identify.
- Decide what the release must support analytically, then find the weakest generalisation that still supports it.
🧪 Practice
- Implement a k-anonymity check and find the coarsest generalisation of your data that reaches k = 10.
- Measure the utility cost of that generalisation on one concrete query you care about, reported as an error, not as a feeling.
- Find a cell in your generalised data with the least diverse sensitive attribute and compute what an attacker learns from group membership alone.
- Demonstrate the rainbow-table attack on a hashed column from your own data, then repeat it with an HMAC and show the difference.
- Interview: Your company publishes two k-anonymous extracts of the same customer base for two different partners. What could go wrong? (Hint: consider what happens when the two partners' quasi-identifier generalisations do not nest inside each other.)
Differential Privacy
Every technique in the previous topic shares a weakness: it protects against the attacks you thought of, using the auxiliary data you imagined. Each new attack requires a new patch, and combining releases breaks guarantees that held individually.
Differential privacy changes the question. Instead of asking "can someone be re-identified from this output?", it asks: would the output look meaningfully different if any one person's data were removed? If the answer is no -- if the distribution of outputs is nearly the same whether you are in the dataset or not -- then nothing anyone learns from the output can be attributed to you, no matter what else they know.
Formally, a randomised mechanism M is epsilon-differentially private if for any two datasets D and D' differing in one person's record, and any set of outputs S:
P(M(D) in S) <= exp(eps) * P(M(D') in S)
eps = 0 : output is independent of any individual -> useless but perfect
eps small : an attacker's belief about you barely moves
eps large : the guarantee is nominal
(eps, delta)-DP adds a delta probability of the guarantee failing outright,
which is what Gaussian noise and DP-SGD accounting use.
The analogy: imagine answering a sensitive yes/no question by first flipping a coin in private. Heads, you answer truthfully; tails, you flip again and answer yes or no according to that. Any individual answer is deniable -- you can always say the coin made you say it -- yet the aggregate proportion is recoverable, because you know the coin's bias. That is randomised response, the oldest differentially private mechanism, invented for survey research in 1965.
The standard mechanism for numeric queries adds noise calibrated to sensitivity: the largest amount the answer could change if one person's record changed. For a count, that is 1.
import numpy as np
rng = np.random.default_rng(5)
N = 10000
disease = (rng.random(N) < 0.18).astype(int)
true_count = disease.sum()
def lap_count(data, eps, sens=1.0):
return data.sum() + rng.laplace(0, sens/eps) # sensitivity 1 for a count
print(f"true count = {true_count} of {N}")
print("\nLAPLACE MECHANISM: NOISE SCALE IS SENSITIVITY / EPSILON")
print(f" {'epsilon':>9}{'noise scale':>13}{'mean |error|':>14}"
f"{'95% of errors within':>22}")
for eps in (0.01, 0.1, 1.0, 10.0):
errs = np.abs([lap_count(disease, eps) - true_count for _ in range(4000)])
print(f" {eps:9.2f}{1/eps:13.1f}{errs.mean():14.1f}"
f"{np.percentile(errs, 95):22.1f}")
print("\nWHAT EPSILON MEANS: THE ATTACKER'S ADVANTAGE IS BOUNDED BY e^eps")
print(f" {'epsilon':>9}{'max likelihood ratio':>22}"
f"{'best posterior from 50% prior':>31}")
for eps in (0.01, 0.1, 1.0, 5.0, 10.0):
r = np.exp(eps)
print(f" {eps:9.2f}{r:22.2f}{r/(1+r):31.1%}")
print("\nCOMPOSITION: A PRIVACY BUDGET IS SPENT, NOT RENEWED")
print(f" {'queries':>9}{'eps each':>10}{'total eps (basic)':>19}"
f"{'total eps (advanced)':>22}")
for q in (1, 10, 100, 1000):
e = 0.1
adv = np.sqrt(2*q*np.log(1/1e-5))*e + q*e*(np.exp(e)-1)
print(f" {q:9d}{e:10.2f}{q*e:19.1f}{adv:22.1f}")
print("\nSAME BUDGET, MORE PEOPLE: PRIVACY GETS CHEAPER AT SCALE")
print(f" {'n':>9}{'true rate':>11}{'noisy rate (eps=1)':>20}{'rel error':>11}")
for n in (100, 1000, 10000, 1000000):
d = (rng.random(n) < 0.18).astype(int)
noisy = (d.sum() + rng.laplace(0, 1.0))/n
print(f" {n:9d}{d.mean():11.3f}{noisy:20.3f}"
f"{abs(noisy-d.mean())/d.mean():11.1%}")
true count = 1827 of 10000
LAPLACE MECHANISM: NOISE SCALE IS SENSITIVITY / EPSILON
epsilon noise scale mean |error| 95% of errors within
0.01 100.0 98.4 287.2
0.10 10.0 10.3 30.6
1.00 1.0 1.0 3.0
10.00 0.1 0.1 0.3
WHAT EPSILON MEANS: THE ATTACKER'S ADVANTAGE IS BOUNDED BY e^eps
epsilon max likelihood ratio best posterior from 50% prior
0.01 1.01 50.2%
0.10 1.11 52.5%
1.00 2.72 73.1%
5.00 148.41 99.3%
10.00 22026.47 100.0%
COMPOSITION: A PRIVACY BUDGET IS SPENT, NOT RENEWED
queries eps each total eps (basic) total eps (advanced)
1 0.10 0.1 0.5
10 0.10 1.0 1.6
100 0.10 10.0 5.9
1000 0.10 100.0 25.7
SAME BUDGET, MORE PEOPLE: PRIVACY GETS CHEAPER AT SCALE
n true rate noisy rate (eps=1) rel error
100 0.170 0.165 2.7%
1000 0.184 0.183 0.5%
10000 0.182 0.182 0.0%
1000000 0.179 0.179 0.0%
Four readings, each with a practical consequence.
Epsilon is an exchange rate, not a switch. At eps = 1 the noise on a count is about 1 person -- negligible against a count of 1,827. At eps = 0.01 it is about 100 people, which destroys the answer. Real deployments sit between: the US Census 2020 used a total budget in the range of tens; Apple and Google use per-report epsilons of a few units per day.
Epsilon has a direct interpretation. The third column of the second table is the most useful thing to tell a stakeholder: starting from a 50% belief that you are in the dataset, an attacker who sees an eps = 1 output can reach at most 73%. At eps = 5 they can reach 99.3%. Epsilon values above about 5 provide a formal guarantee with very little practical content, which is why reported epsilons should always be read sceptically.
The budget composes and does not reset. A hundred queries at eps = 0.1 each cost eps = 10 in total under basic composition, or 5.9 under advanced composition. This is differential privacy's greatest strength -- the guarantee degrades predictably rather than silently, as k-anonymity does -- and its greatest operational burden: you must track spend, and when the budget is exhausted, the honest answer is to stop answering.
Scale is free privacy. The same absolute noise is 2.7% relative error on 100 people and invisible on a million. Differential privacy is nearly costless for population-level statistics on large datasets, and expensive for small subgroups -- which is unfortunately where the interesting questions often are.
For training models the standard method is DP-SGD: bound each example's influence by clipping its gradient, then add noise to the summed gradient before the update.
X = rng.normal(0, 1, (4000, 5)); w_true = np.array([1.2,-0.8,0.5,0.0,0.3])
y = (rng.random(4000) < 1/(1+np.exp(-(X @ w_true)))).astype(int)
Xt = rng.normal(0, 1, (2000, 5))
yt = (rng.random(2000) < 1/(1+np.exp(-(Xt @ w_true)))).astype(int)
def train(clip=None, sigma=0.0, iters=300, lr=0.5, batch=64):
w = np.zeros(5)
for _ in range(iters):
idx = rng.choice(len(X), batch, replace=False)
p = 1/(1+np.exp(-(X[idx] @ w)))
gi = (p - y[idx])[:, None]*X[idx] # PER-EXAMPLE gradients
if clip is not None:
nrm = np.linalg.norm(gi, axis=1, keepdims=True)
gi = gi*np.minimum(1.0, clip/nrm) # bound each influence
g = gi.sum(0)
if sigma:
g = g + rng.normal(0, sigma*(clip or 1.0), 5) # then add noise
w -= lr*g/batch
return w
def acc(w): return ((1/(1+np.exp(-(Xt @ w))) > 0.5) == yt).mean()
print(f" {'setting':32}{'test accuracy':>15}{'|w - w_true|':>14}")
for lab, kw in [("no privacy", dict()),
("clip only (C=1), no noise", dict(clip=1.0)),
("clip + noise sigma=2", dict(clip=1.0, sigma=2.0)),
("clip + noise sigma=10", dict(clip=1.0, sigma=10.0)),
("clip + noise sigma=40", dict(clip=1.0, sigma=40.0))]:
w = train(**kw)
print(f" {lab:32}{acc(w):15.3f}{np.linalg.norm(w-w_true):14.2f}")
setting test accuracy |w - w_true|
no privacy 0.719 0.17
clip only (C=1), no noise 0.721 0.60
clip + noise sigma=2 0.726 0.65
clip + noise sigma=10 0.713 0.79
clip + noise sigma=40 0.626 3.71
Both DP-SGD steps matter and they are not interchangeable. Clipping is what bounds sensitivity -- without it, one outlier can move the gradient arbitrarily far and no amount of noise gives a guarantee. Noise is what buys the epsilon. The table shows the usual shape: modest noise is nearly free, and there is a point (here sigma = 40) where the model degrades sharply. Note also that clipping alone already costs parameter accuracy (0.17 to 0.60), because it biases the gradient.
A last caution. Differential privacy protects one person's record. If a person contributes many rows, the unit of protection must be the person, not the row (user-level rather than example-level DP), or the guarantee you advertise is not the guarantee you provide.
Key Takeaways
- Differential privacy bounds how much any single person's data can change the distribution of outputs, which protects against unknown future attacks.
- The noise scale is sensitivity divided by epsilon; sensitivity is the most an answer could change if one record changed.
- Epsilon is an exchange rate with a direct reading: from a 50% prior, an attacker can reach at most exp(eps)/(1+exp(eps)).
- Epsilons above roughly 5 are formally valid and practically weak; always read a reported epsilon critically.
- The budget composes across queries and does not reset; when it is spent, the correct action is to stop answering.
- Privacy is cheap at population scale and expensive for small subgroups, which is often where the questions are.
- DP-SGD needs both clipping (to bound sensitivity) and noise (to buy epsilon); clipping alone gives no guarantee.
- Protect at the level of the person, not the row, whenever one person can contribute many rows.
🧪 Practice
- Implement the Laplace mechanism for a mean rather than a count, deriving the sensitivity yourself, and check the error at three epsilon values.
- Implement randomised response and recover the true proportion from the noisy answers, reporting the variance of your estimate.
- Track a privacy budget across a sequence of queries and write the code that refuses a query once the budget is exhausted.
- Train a model with DP-SGD at several noise levels and plot accuracy against noise, identifying the knee.
- Interview: A vendor advertises "differentially private analytics with epsilon = 12". What do you conclude? (Hint: compute the bound on an attacker's posterior from a 50% prior, and ask whether it is per query or in total.)
Federated Learning
Every privacy technique so far assumes the data has already been collected into one place. Federated learning asks whether that step is necessary at all. The insight: to fit a model you need gradients, not rows. If each participant computes updates on their own data and only the updates are shared, the raw data never leaves the device or the institution.
This matters in two very different settings. Cross-device federation involves millions of phones with a few hundred examples each -- the original motivating case, for keyboard prediction. Cross-silo federation involves a handful of organisations with large datasets that they cannot legally pool -- hospitals, banks, or national statistical offices.
The standard algorithm is FedAvg:
for each round:
server sends the current model to a sample of clients
each client trains locally for a few steps on its own data
each client uploads its updated weights (not its data)
server averages the uploads, weighted by client dataset size
server --w--> [client 1] --w1--> \
--w--> [client 2] --w2--> > w <- weighted mean
--w--> [client k] --wk--> /
import numpy as np
rng = np.random.default_rng(8)
D, NC, NPER = 6, 20, 300 # dims, clients, rows each
w_true = np.array([1.5, -1.0, 0.8, 0.0, 0.4, -0.6])
def make_clients(non_iid=0.0):
"""non_iid shifts each client's feature distribution and label balance."""
cls = []
for c in range(NC):
shift = non_iid*rng.normal(0, 1, D)
Xc = rng.normal(shift, 1.0, (NPER, D))
z = Xc @ w_true + non_iid*rng.normal(0, 2)
yc = (rng.random(NPER) < 1/(1+np.exp(-z))).astype(int)
cls.append((Xc, yc))
return cls
Xt = rng.normal(0, 1, (4000, D))
yt = (rng.random(4000) < 1/(1+np.exp(-(Xt @ w_true)))).astype(int)
def acc(w): return ((1/(1+np.exp(-(Xt @ w))) > 0.5) == yt).mean()
def local_steps(w, Xc, yc, steps, lr=0.5):
w = w.copy()
for _ in range(steps):
p = 1/(1+np.exp(-(Xc @ w)))
w -= lr*(Xc.T @ (p - yc))/len(yc)
return w
def central(cls, iters=400, lr=0.5): # the baseline: pool the data
X = np.vstack([c[0] for c in cls]); y = np.concatenate([c[1] for c in cls])
w = np.zeros(D)
for _ in range(iters):
p = 1/(1+np.exp(-(X @ w)))
w -= lr*(X.T @ (p - y))/len(y)
return w
def fedavg(cls, rounds, local, frac=1.0):
w = np.zeros(D)
for _ in range(rounds):
sel = rng.choice(NC, max(1, int(frac*NC)), replace=False)
ws = [local_steps(w, cls[i][0], cls[i][1], local) for i in sel]
w = np.mean(ws, axis=0) # equal n, so equal weights
return w
print("FEDAVG vs POOLING THE DATA (20 clients, 300 rows each)")
print(f" {'setting':34}{'rounds':>8}{'local steps':>13}{'accuracy':>10}"
f"{'uploads':>9}")
for label, non_iid in [("IID clients", 0.0), ("non-IID clients", 2.5)]:
cls = make_clients(non_iid)
print(f" {label}")
print(f" {' centralised (data pooled)':34}{'-':>8}{'-':>13}"
f"{acc(central(cls)):10.3f}{0:9d}")
for rounds, local in [(5, 1), (5, 20), (40, 5), (200, 5)]:
print(f" {' fedavg':34}{rounds:8d}{local:13d}"
f"{acc(fedavg(cls, rounds, local)):10.3f}{rounds*NC:9d}")
print("\nCLIENT SAMPLING: NOT EVERY DEVICE IS ONLINE EVERY ROUND")
cls = make_clients(2.5)
print(f" {'fraction per round':>20}{'rounds':>8}{'accuracy':>10}{'uploads':>9}")
for frac in (0.1, 0.25, 0.5, 1.0):
print(f" {frac:20.0%}{60:8d}{acc(fedavg(cls, 60, 5, frac=frac)):10.3f}"
f"{int(60*NC*frac):9d}")
print("\nWHY THE UPDATE IS NOT PRIVATE: A GRADIENT LEAKS ITS BATCH")
Xc, yc = make_clients(0.0)[0]
w0 = np.zeros(D)
for n in (1, 2, 8, 32):
sims = []
for rep in range(40): # average over many batches
idx = rng.choice(len(Xc), n, replace=False)
p = 1/(1+np.exp(-(Xc[idx] @ w0)))
g = (Xc[idx].T @ (p - yc[idx]))/n # the uploaded gradient
v = Xc[idx[0]] # one example in that batch
sims.append(abs(g @ v)/(np.linalg.norm(g)*np.linalg.norm(v)))
print(f" batch of {n:4d}: mean |cosine| between the update and one of "
f"its rows = {np.mean(sims):.3f}")
chance = np.mean([abs(a @ b)/(np.linalg.norm(a)*np.linalg.norm(b))
for a, b in zip(rng.normal(0, 1, (40, D)), Xc[:40])])
print(f" chance level for unrelated vectors in {D} dimensions = {chance:.3f}")
FEDAVG vs POOLING THE DATA (20 clients, 300 rows each)
setting rounds local steps accuracy uploads
IID clients
centralised (data pooled) - - 0.791 0
fedavg 5 1 0.792 100
fedavg 5 20 0.791 100
fedavg 40 5 0.791 800
fedavg 200 5 0.791 4000
non-IID clients
centralised (data pooled) - - 0.757 0
fedavg 5 1 0.753 100
fedavg 5 20 0.776 100
fedavg 40 5 0.763 800
fedavg 200 5 0.763 4000
CLIENT SAMPLING: NOT EVERY DEVICE IS ONLINE EVERY ROUND
fraction per round rounds accuracy uploads
10% 60 0.702 120
25% 60 0.719 300
50% 60 0.737 600
100% 60 0.757 1200
WHY THE UPDATE IS NOT PRIVATE: A GRADIENT LEAKS ITS BATCH
batch of 1: mean |cosine| between the update and one of its rows = 1.000
batch of 2: mean |cosine| between the update and one of its rows = 0.677
batch of 8: mean |cosine| between the update and one of its rows = 0.525
batch of 32: mean |cosine| between the update and one of its rows = 0.378
chance level for unrelated vectors in 6 dimensions = 0.308
On IID clients FedAvg matches centralised training exactly -- 0.791 either way -- which is the result that makes the approach attractive. On non-IID clients it does not: centralised reaches 0.757 and FedAvg lands between 0.753 and 0.776 depending on how the round budget is spent, and client sampling costs more still (0.702 at 10% participation against 0.757 pooled). Heterogeneity, not cryptography, is the main practical difficulty in federated learning.
The last block is the point that must be understood before anyone describes federated learning as private. A single-example update has cosine similarity 1.000 with that example -- the gradient of a logistic model on one row is a scalar multiple of the row, so uploading it is uploading the data. Even at a batch of 32, similarity remains above the chance level of 0.308. Published gradient-inversion attacks reconstruct recognisable training images from updates in realistic deep-learning settings.
Federated learning is therefore a data minimisation technique, not a privacy guarantee. Making it private requires composing it with other mechanisms:
| Mechanism | What it adds | Cost |
|---|---|---|
| Secure aggregation | Server sees only the sum, never an individual update | Crypto protocol, dropout handling |
| DP noise (local or central) | A formal epsilon on what any update reveals | Accuracy, and needs many clients |
| Gradient clipping | Bounded per-client influence; also blocks poisoning | Slower convergence |
| Update compression | Less to intercept, less bandwidth | Some accuracy |
| Client-side validation | Resistance to malicious clients | Cannot fully solve it |
And two costs that are frequently underestimated: you cannot inspect the training data, so debugging is severely limited -- no looking at the distribution, no finding the mislabelled batch -- and a subset of clients can poison the model by sending crafted updates, which requires robust aggregation (trimmed means, median, norm bounds) rather than a plain average.
Key Takeaways
- Federated learning trains on decentralised data by sharing model updates instead of rows, in cross-device (many small clients) or cross-silo (few large ones) settings.
- FedAvg matches centralised training on IID clients and falls short on heterogeneous ones; heterogeneity is the main practical problem.
- Partial client participation costs additional accuracy for the same number of rounds.
- Model updates leak their training data: a single-example gradient is the example up to scale, and inversion attacks reconstruct inputs in practice.
- Federated learning is data minimisation, not a privacy guarantee; it needs secure aggregation and differential privacy to become one.
- Gradient clipping bounds both privacy leakage and the influence of a malicious client.
- Plain averaging is vulnerable to poisoning; robust aggregation is required when clients are not trusted.
- You cannot inspect the data, so ordinary debugging and data-quality work become much harder.
🧪 Practice
- Implement FedAvg and compare it with centralised training under IID and label-skewed client splits.
- Measure the accuracy cost of reducing client participation from 100% to 10% at a fixed number of rounds.
- Reconstruct a training row from a single-example gradient upload for a linear model, and show how batching degrades the reconstruction.
- Implement a trimmed-mean aggregator and show that it survives three clients uploading large adversarial updates while the plain mean does not.
- Interview: A hospital consortium wants federated learning so that "no patient data is shared". What do you have to tell them before they announce it? (Hint: distinguish what leaves the building from what can be inferred from what leaves the building.)
Adversarial Attacks on Models
Once a model is exposed -- through an API, an app, or a decision it makes -- it becomes an attack surface in its own right. The model is not just software that could have bugs; it is a statistical object that leaks information about its training data and can be manipulated through its inputs. Four attack families cover almost everything seen in practice, and they are distinguished by what the attacker wants.
| Attack | Attacker's goal | Access needed | Primary defence |
|---|---|---|---|
| Evasion (adversarial examples) | Make one input be classified wrongly | Query access; gradients help | Adversarial training, input validation, ensembles |
| Poisoning / backdoor | Corrupt the model during training | Influence over training data | Data provenance, anomaly detection in training data |
| Membership inference | Learn whether a record was in training | Query access | Reduce overfitting, DP training |
| Model extraction / inversion | Steal the model, or reconstruct training inputs | Many queries | Rate limits, output coarsening, watermarking |
import numpy as np
rng = np.random.default_rng(13)
D = 40
def feats(n):
X = rng.normal(0, 1, (n, D))
w = np.r_[np.linspace(1.5, 0.2, 8), np.zeros(D-8)]
y = (rng.random(n) < 1/(1+np.exp(-(X @ w)))).astype(int)
return X, y
Xtr, ytr = feats(600); Xte, yte = feats(4000)
def train(X, y, iters=4000, lr=0.4, l2=0.0):
w = np.zeros(X.shape[1]); b = 0.0
for _ in range(iters):
p = 1/(1+np.exp(-(X @ w + b)))
g = (p - y)/len(y)
w -= lr*(X.T @ g + l2*w); b -= lr*g.sum()
return w, b
def prob(X, wb): return 1/(1+np.exp(-(X @ wb[0] + wb[1])))
print("1. EVASION: THE SMALLEST PERTURBATION THAT FLIPS A DECISION")
wb = train(Xtr, ytr, l2=1e-3)
p = prob(Xte, wb)
# FOR A LINEAR MODEL THE MINIMUM L2 PERTURBATION IS THE DISTANCE TO THE
# HYPERPLANE: |margin| / ||w||, APPLIED ALONG -sign(margin)*w.
margin = Xte @ wb[0] + wb[1]
need = np.abs(margin)/np.linalg.norm(wb[0])
print(f" clean test accuracy: {((p > 0.5) == yte).mean():.3f}")
print(f" {'perturbation L2 budget':>24}{'accuracy under attack':>24}")
for eps in (0.0, 0.05, 0.1, 0.25, 0.5, 1.0):
flipped = need <= eps
print(f" {eps:24.2f}{(((p > 0.5) == yte) & ~flipped).mean():24.3f}")
print(f" median row sits {np.median(need):.3f} away from the boundary, while")
print(f" a typical row has norm {np.median(np.linalg.norm(Xte, axis=1)):.1f}"
f" -- the attack is "
f"{np.median(need)/np.median(np.linalg.norm(Xte, axis=1)):.1%} of the input")
print("\n2. MEMBERSHIP INFERENCE: DID THIS ROW APPEAR IN TRAINING?")
def xent(X, y, wb):
p = np.clip(prob(X, wb), 1e-9, 1-1e-9)
return -(y*np.log(p) + (1-y)*np.log(1-p))
def auc(lab, sc):
o = np.argsort(sc); r = np.empty(len(sc)); r[o] = np.arange(1, len(sc)+1)
n1 = lab.sum(); n0 = len(lab)-n1
return (r[lab == 1].sum() - n1*(n1+1)/2)/(n0*n1)
print(f" {'training rows':>14}{'train acc':>11}{'test acc':>10}{'gap':>8}"
f"{'attack AUC':>12}")
for n in (60, 150, 400, 1200, 4000):
TR, TE, AT = [], [], []
for _ in range(8): # average over 8 draws
Xs, ys = feats(n); Xo, yo = feats(n) # members and non-members
wb = train(Xs, ys, iters=8000, lr=0.5, l2=1e-4)
TR.append(((prob(Xs, wb) > 0.5) == ys).mean())
TE.append(((prob(Xte, wb) > 0.5) == yte).mean())
# THE ATTACK: SCORE EACH ROW BY ITS LOSS. LOW LOSS -> LIKELY A MEMBER.
L = np.r_[xent(Xs, ys, wb), xent(Xo, yo, wb)]
AT.append(auc(np.r_[np.ones(n), np.zeros(n)], -L))
print(f" {n:14d}{np.mean(TR):11.3f}{np.mean(TE):10.3f}"
f"{np.mean(TR)-np.mean(TE):+8.3f}{np.mean(AT):12.3f}")
print("\n3. MODEL EXTRACTION: REBUILDING THE MODEL FROM ITS OWN ANSWERS")
wb = train(Xtr, ytr, l2=1e-3)
print(f" {'queries':>9}{'agreement with victim':>24}{'cosine(w_hat, w)':>19}")
for q in (50, 200, 1000, 5000):
Xq = rng.normal(0, 1, (q, D)) # the attacker's own inputs
yq = (prob(Xq, wb) > 0.5).astype(int) # only HARD labels returned
sur = train(Xq, yq, l2=1e-3)
agree = ((prob(Xte, sur) > 0.5) == (prob(Xte, wb) > 0.5)).mean()
cos = (sur[0] @ wb[0])/(np.linalg.norm(sur[0])*np.linalg.norm(wb[0]))
print(f" {q:9d}{agree:24.3f}{cos:19.3f}")
1. EVASION: THE SMALLEST PERTURBATION THAT FLIPS A DECISION
clean test accuracy: 0.801
perturbation L2 budget accuracy under attack
0.00 0.801
0.05 0.783
0.10 0.766
0.25 0.696
0.50 0.569
1.00 0.320
median row sits 0.679 away from the boundary, while
a typical row has norm 6.3 -- the attack is 10.8% of the input
2. MEMBERSHIP INFERENCE: DID THIS ROW APPEAR IN TRAINING?
training rows train acc test acc gap attack AUC
60 1.000 0.694 +0.306 0.649
150 0.936 0.721 +0.215 0.570
400 0.844 0.791 +0.053 0.523
1200 0.826 0.819 +0.007 0.510
4000 0.826 0.828 -0.002 0.507
3. MODEL EXTRACTION: REBUILDING THE MODEL FROM ITS OWN ANSWERS
queries agreement with victim cosine(w_hat, w)
50 0.763 0.747
200 0.936 0.981
1000 0.979 0.998
5000 0.987 1.000
Each block carries a distinct lesson.
Evasion is cheap and it is not a deep-learning phenomenon. A perturbation of L2 norm 1.0 -- about 16% of a typical input's norm -- halves the model's accuracy, and even a budget of 0.1 costs 3.5 points. The reason is geometric, not a quirk of neural networks: in high dimensions most points sit close to the decision boundary relative to their own magnitude, and the shortest path across is along the weight vector. Anything with a decision boundary can be evaded; models trained on attacker-controlled inputs (spam, fraud, malware, content moderation) must assume it.
Membership inference power is the generalisation gap. The attack is embarrassingly simple -- compute each candidate's loss and guess "member" if it is low -- and its AUC tracks the train-test gap almost perfectly: 0.649 at a gap of 0.306, and 0.507 (chance) when the model does not overfit. The practical implication is encouraging: the main defence against membership inference is not exotic, it is the regularisation you should already be doing. For genuinely sensitive training data -- medical records, private messages -- add DP training on top.
Extraction needs only the answers. With 1,000 queries returning nothing but hard labels, the copy agrees with the original on 97.9% of inputs and its weight vector has cosine similarity 0.998. Returning probabilities or explanations makes it dramatically faster. This is why rate limits, query monitoring, coarsened outputs, and abuse detection are part of ML security, not just of API design -- and why "we only return the label" is not a defence.
The fourth family, poisoning, does not appear in the code because it requires control over training data rather than queries, but it is the most serious for any pipeline that ingests public or user-supplied data. A backdoor attack adds a small number of examples containing a trigger pattern paired with the attacker's desired label; the model behaves normally except when the trigger is present. Defences are about provenance and data validation: know where every training row came from, detect distributional anomalies in incoming data, hold out a trusted clean validation set, and treat any externally sourced dataset or pretrained checkpoint as untrusted input.
Key Takeaways
- A deployed model is an attack surface: it leaks information about its training data and can be manipulated through its inputs.
- Evasion is geometric, not specific to deep learning; perturbations of ~10% of an input's norm flip a large share of decisions for a linear model.
- Any system whose inputs are controlled by an adversary must assume evasion attempts.
- Membership inference reduces to thresholding the per-example loss, and its power tracks the train-test gap almost exactly.
- Ordinary regularisation is the first and most effective defence against membership inference; DP training is the second.
- Model extraction works from hard labels alone: 1,000 queries produced a copy agreeing on 97.9% of inputs.
- Returning probabilities or explanations accelerates extraction, so output coarsening and rate limiting are security controls.
- Poisoning and backdoors require provenance controls and validation of incoming training data, including third-party datasets and checkpoints.
🧪 Practice
- Compute the minimum perturbation needed to flip each test prediction of a linear model and plot accuracy against the attack budget.
- Run a loss-threshold membership inference attack against two versions of a model, one overfitted and one regularised, and report both AUCs.
- Extract a copy of a model of yours using only hard labels, and find the number of queries at which agreement exceeds 95%.
- Insert a backdoor by adding 50 poisoned rows with a trigger feature pattern, then verify that overall accuracy is unaffected.
- Interview: Your public API returns a fraud probability to four decimal places. What are the risks, and what would you change? (Hint: think about what extra precision gives an attacker who can query repeatedly, and what the legitimate consumer actually needs.)
Secure Data Handling Practices
Most real privacy incidents are not clever attacks. They are a notebook with customer emails committed to a repository, an S3 bucket left public, a production credential in a config file, a CSV of records emailed to a contractor, or a dataset that should have been deleted two years ago and was still sitting in a backup when the breach happened. The controls that prevent these are unglamorous and they matter more than everything above.
Five principles organise them:
| Principle | What it means in a data science workflow |
|---|---|
| Data minimisation | Pull the columns and rows you need, not the whole table "in case" |
| Least privilege | Access matched to role and to a current, stated purpose |
| Purpose limitation | Data gathered for billing is not automatically available for modelling |
| Defence in depth | Encryption, access control, masking, and logging are layers, not alternatives |
| Auditability | Who accessed which records, when, and why -- retained and reviewable |
import numpy as np, pandas as pd, hashlib, hmac, datetime as dt
rng = np.random.default_rng(21)
N = 200
df = pd.DataFrame({
"user_id": range(1000, 1000+N),
"email": [f"user{i}@example.com" for i in range(N)],
"ssn": [f"{rng.integers(100,999)}-{rng.integers(10,99)}"
f"-{rng.integers(1000,9999)}" for _ in range(N)],
"card": [f"4111 1111 1111 {rng.integers(1000,9999)}" for _ in range(N)],
"zip": rng.choice(["10001", "10002", "94107"], N),
"spend": rng.normal(200, 60, N).round(2),
"created": [dt.date(2021, 1, 1) + dt.timedelta(days=int(d))
for d in rng.integers(0, 1700, N)],
})
print("1. A PII SCANNER YOU CAN PUT IN CI")
PATTERNS = {"email": r"[\w.+-]+@[\w-]+\.[\w.]+",
"us_ssn": r"\b\d{3}-\d{2}-\d{4}\b",
"card_like": r"\b(?:\d[ -]?){13,16}\b"}
for col in df.columns:
vals = df[col].astype(str)
hits = {k: vals.str.contains(p, regex=True).mean()
for k, p in PATTERNS.items()}
flags = [f"{k} {v:.0%}" for k, v in hits.items() if v > 0.5]
print(f" {col:10} {'CLASSIFY: ' + ', '.join(flags) if flags else 'ok'}")
print("\n2. TOKENISATION: STABLE JOIN KEY, NO RECOVERABLE VALUE")
KEY = b"kept-in-a-kms-not-in-git" # rotate it; never commit it
def token(v):
return hmac.new(KEY, str(v).encode(), hashlib.sha256).hexdigest()[:16]
bad = lambda v: hashlib.sha256(str(v).encode()).hexdigest()[:16]
print(f" plain sha256 of an email : {bad(df.email[0])}")
print(f" keyed HMAC of the same : {token(df.email[0])}")
print(" The plain hash is guessable from a wordlist of emails; the HMAC")
print(" is not, without the key. Both join consistently.")
print("\n3. COLUMN-LEVEL ACCESS BY ROLE")
CLASS = {"user_id": "internal", "email": "pii", "ssn": "sensitive",
"card": "sensitive", "zip": "quasi", "spend": "internal",
"created": "internal"}
ROLE_MAX = {"analyst": {"internal"},
"ds_modeller": {"internal", "quasi", "pii"},
"fraud_ops": {"internal", "quasi", "pii", "sensitive"}}
MASK = {"pii": token, "sensitive": lambda v: "***REDACTED***",
"quasi": lambda v: str(v)[:3] + "**"}
def view(role):
allowed = ROLE_MAX[role]; out = df.copy()
for c in out.columns:
cl = CLASS[c]
if cl not in allowed:
out[c] = out[c].map(MASK.get(cl, lambda v: "***"))
elif cl == "pii":
out[c] = out[c].map(token) # pseudonymise even if allowed
return out
for role in ROLE_MAX:
r = view(role).iloc[0]
print(f" {role:12} email={r.email[:12]:14} ssn={r.ssn:14} zip={r.zip}")
print("\n4. RETENTION: DELETION AS A SCHEDULED JOB, NOT A PROMISE")
today = dt.date(2026, 9, 28)
POLICY = {"raw_events": 90, "enriched": 730, "aggregates": None}
print(f" {'dataset':14}{'retention (days)':>18}{'rows kept of 200':>18}")
for ds, days in POLICY.items():
if days is None:
print(f" {ds:14}{'indefinite':>18}{N:18d}")
else:
keep = (pd.to_datetime(df.created) >
pd.Timestamp(today - dt.timedelta(days=days))).sum()
print(f" {ds:14}{days:18d}{keep:18d}")
1. A PII SCANNER YOU CAN PUT IN CI
user_id ok
email CLASSIFY: email 100%
ssn CLASSIFY: us_ssn 100%
card CLASSIFY: card_like 100%
zip ok
spend ok
created ok
2. TOKENISATION: STABLE JOIN KEY, NO RECOVERABLE VALUE
plain sha256 of an email : 9551f6ff6935f400
keyed HMAC of the same : 18691c13cdb0c728
The plain hash is guessable from a wordlist of emails; the HMAC
is not, without the key. Both join consistently.
3. COLUMN-LEVEL ACCESS BY ROLE
analyst email=18691c13cdb0 ssn=***REDACTED*** zip=100**
ds_modeller email=18691c13cdb0 ssn=***REDACTED*** zip=10001
fraud_ops email=18691c13cdb0 ssn=370-79-4474 zip=10001
4. RETENTION: DELETION AS A SCHEDULED JOB, NOT A PROMISE
dataset retention (days) rows kept of 200
raw_events 90 0
enriched 730 33
aggregates indefinite 200
Four controls, each cheap and each preventing a whole class of incident.
Automated classification means the schema knows what each column is, so masking and access rules can be enforced by code rather than remembered by people. Run the scanner in CI over every new table and fail the build on an unclassified match. The same scanner catches the accidental case: a "notes" column that turns out to contain email addresses.
Keyed tokenisation replaces identifiers with values that still join correctly -- essential, because analysis needs to link a customer's rows -- while being useless to anyone without the key. The key lives in a key-management service, is rotated on a schedule, and is never in the repository. A plain hash, as the previous topic showed, is not a substitute.
Column-level access by role is the practical form of least privilege. The analyst gets aggregate-safe columns and a masked postcode; the modeller gets a pseudonymous customer key so features can be joined; only fraud operations, with a documented purpose, sees the raw sensitive fields. Implement this in the warehouse (views, row and column policies, dynamic masking) rather than in each notebook, because a policy that depends on every analyst writing the right query is not a policy.
Retention as a scheduled job is what makes a deletion promise real. The policy is per dataset and per class of data, the job runs unattended, and it must reach every copy: warehouse tables, feature store, training snapshots, model artifacts that embedded the data, logs, exports, and backups. The copies are what make subject-access deletion requests hard, and the only manageable approach is to keep an inventory of where personal data flows.
Beyond the code, the checklist that catches most of the remainder:
- Encrypt in transit and at rest by default; treat an unencrypted store as a finding.
- No secrets in code. Use a secret manager, scan for credentials in pre-commit hooks, and rotate anything that has ever been committed.
- Never copy production data to a laptop. Work in the environment where the data already is; if you must extract, extract a sampled, masked subset with an expiry.
- Use synthetic or masked data for development and tests. Test fixtures containing real records are a recurring source of leaks.
- Strip notebook outputs before committing. A cleared cell is not a cleared file.
- Log access, and review the logs. An audit trail nobody reads deters nothing.
- Have an incident plan and rehearse it. Who is called, what is disclosed, within what deadline (72 hours under the GDPR).
- Review third-party data flows. A dashboard tool, an LLM API, or an analytics SDK receiving personal data is a processor relationship with contractual requirements.
That last item has become the most common new exposure: sending customer records into an external model API is a data transfer, subject to the same rules as any other, and "it is just for a prototype" is not an exemption.
Key Takeaways
- Most privacy incidents come from ordinary handling failures, not from sophisticated attacks.
- Minimise, restrict by role and purpose, layer defences, and make access auditable.
- Classify columns automatically in CI so masking and access can be enforced by code instead of by memory.
- Use keyed tokenisation (HMAC with a managed key) for join keys; a plain hash is reversible for low-entropy values.
- Enforce column-level access in the warehouse, not in each analyst's query.
- Retention must be a scheduled, unattended job that reaches every copy, including snapshots, logs, and backups.
- Keep no secrets in code, no production data on laptops, and no real records in test fixtures; strip notebook outputs before committing.
- Sending personal data to an external model API is a data transfer with the same obligations as any other, prototype or not.
🧪 Practice
- Write a scanner that classifies every column of one of your tables and run it as a pre-commit or CI check.
- Replace a plain-hash join key in an existing pipeline with an HMAC and document where the key lives and how it rotates.
- Define three roles for one of your datasets with the columns each may see, and implement the masked views.
- Trace one personal data field through every system it reaches and write the deletion procedure that covers all of them.
- Interview: You need to share a dataset with an external research partner next week. Walk through your process. (Hint: think about the legal basis first, then minimisation, then the technical release, and finally what happens to their copy afterwards.)
<a id="164-governance-and-regulation"></a>
16.4 Governance and Regulation
Governance is the set of mechanisms that make everything in the previous three sections true in practice rather than in a document; this section covers the frameworks that assign accountability for data, the lineage and catalog infrastructure that makes questions answerable, the regulatory landscape a model must operate inside, what consent actually obliges you to do, how responsible-AI principles become enforceable controls, and the additional duties that apply when your work is research on people. The organising insight: a governance rule that cannot be executed by a machine or attached to a named person is a preference, not a control.
Data Governance Frameworks
Data governance is the answer to a question that has no technical answer: who decides? Who decides what "active customer" means, whether a new field may be collected, how long records are kept, who may see the salary column, and whether a model may be retrained on last week's data. Without an answer, every one of those decisions is made implicitly, by whoever wrote the query.
The recurring failure mode of governance programmes is that they produce documents. A policy in a wiki that nobody can execute changes nothing; it just shifts blame after an incident. Effective governance has three properties: named accountability, machine-checkable rules, and a gate that can actually block something.
The standard role model separates decision rights from execution:
| Role | Accountable for | Typical holder |
|---|---|---|
| Data owner | The decision: who may use this data, for what, how long | A business leader, not a data team |
| Data steward | Day-to-day correctness: definitions, quality, metadata | A domain expert |
| Data custodian | The technical controls: storage, access, backups | Platform engineering |
| Data protection officer | Legal compliance, DPIAs, regulator contact | Legal/compliance (mandatory in some cases under GDPR) |
| Governance council | Cross-domain conflicts, standards, exceptions | A standing forum with authority |
| Consumer | Using data within its declared purpose | Every analyst and data scientist |
Two organisational patterns dominate, and the choice is about where the bottleneck sits rather than about better and worse:
CENTRALISED FEDERATED (data mesh)
one team owns the standards each domain owns its data products
and the catalog a central team owns the platform
and the global rules
+ consistent, one throat to choke + scales; owners have context
- bottleneck; lacks domain context- inconsistency; needs real enforcement
from dataclasses import dataclass, field
from typing import List, Optional
import datetime as dt
@dataclass
class Dataset:
name: str
owner: Optional[str] = None
steward: Optional[str] = None
classification: Optional[str] = None # public|internal|pii|sensitive
retention_days: Optional[int] = None
lawful_basis: Optional[str] = None
upstream: List[str] = field(default_factory=list)
last_reviewed: Optional[dt.date] = None
tests: int = 0
CLASSES = {"public", "internal", "pii", "sensitive"}
BASES = {"consent", "contract", "legal_obligation", "vital_interests",
"public_task", "legitimate_interests"}
TODAY = dt.date(2026, 9, 28)
def check(d: Dataset):
"""Each rule is a policy statement that a machine can enforce."""
out = []
if not d.owner: out.append("no accountable owner")
if not d.steward: out.append("no data steward")
if d.classification not in CLASSES: out.append("unclassified")
if d.classification in {"pii", "sensitive"}:
if d.lawful_basis not in BASES: out.append("no lawful basis recorded")
if d.retention_days is None: out.append("no retention period")
elif d.retention_days > 1095: out.append("retention over 3 years")
if not d.upstream and d.name != "raw_signup":
out.append("no declared lineage")
if d.last_reviewed is None or (TODAY - d.last_reviewed).days > 365:
out.append("review overdue")
if d.tests == 0: out.append("no quality tests")
return out
catalog = [
Dataset("raw_signup", "growth@", "a.rivera", "pii", 365, "consent",
[], dt.date(2026, 4, 2), 6),
Dataset("user_profile", "platform@", "j.chen", "pii", 730, "contract",
["raw_signup"], dt.date(2026, 1, 15), 4),
Dataset("churn_features", "ds@", None, "internal", None, None,
["user_profile"], dt.date(2024, 11, 3), 0),
Dataset("support_transcripts", None, None, None, 4000, None,
["raw_signup"], None, 0),
Dataset("revenue_daily", "finance@", "m.okafor", "internal", None, None,
["user_profile"], dt.date(2026, 6, 30), 9),
Dataset("churn_model_v3", "ds@", "s.haddad", "internal", None, None,
["churn_features"], dt.date(2026, 5, 20), 3),
Dataset("retention_dashboard", "growth@", "a.rivera", "internal", None,
None, ["churn_model_v3", "revenue_daily"], dt.date(2026, 7, 1), 2),
]
print("GOVERNANCE GATE: POLICY AS CODE OVER THE CATALOG")
print(f" {'dataset':22}{'class':>11}{'findings':>10} detail")
fail = 0
for d in catalog:
f = check(d)
fail += bool(f)
print(f" {d.name:22}{str(d.classification):>11}{len(f):10} "
f"{'; '.join(f) if f else 'compliant'}")
print(f"\n {fail} of {len(catalog)} datasets fail the gate")
print("\nBLAST RADIUS: WHAT A POLICY BREACH UPSTREAM AFFECTS")
children = {d.name: [c.name for c in catalog if d.name in c.upstream]
for d in catalog}
def downstream(n, seen=None):
seen = seen or set()
for c in children.get(n, []):
if c not in seen:
seen.add(c); downstream(c, seen)
return seen
for d in catalog:
if check(d):
print(f" {d.name:22} -> affects "
f"{sorted(downstream(d.name)) or ['nothing']}")
GOVERNANCE GATE: POLICY AS CODE OVER THE CATALOG
dataset class findings detail
raw_signup pii 0 compliant
user_profile pii 0 compliant
churn_features internal 3 no data steward; review overdue; no quality tests
support_transcripts None 5 no accountable owner; no data steward; unclassified; review overdue; no quality tests
revenue_daily internal 0 compliant
churn_model_v3 internal 0 compliant
retention_dashboard internal 0 compliant
2 of 7 datasets fail the gate
BLAST RADIUS: WHAT A POLICY BREACH UPSTREAM AFFECTS
churn_features -> affects ['churn_model_v3', 'retention_dashboard']
support_transcripts -> affects ['nothing']
That check is fifteen lines and it is the difference between a governance policy and a governance control. It runs in CI over the catalog, it names specific gaps, and it can fail a build. Every rule inside it corresponds to a sentence someone would otherwise have written in a document and hoped for: "all personal data must have a documented lawful basis" becomes four lines that can be violated visibly.
Notice what the blast radius query adds. churn_features has no steward, no
tests, and has not been reviewed in nearly two years -- and two production
assets depend on it. That combination is what turns a low-priority metadata
gap into an incident waiting to happen, and it is invisible if you look at
datasets one at a time.
A practical adoption sequence, in the order that actually works:
- Inventory first. You cannot govern what you cannot list. An auto-discovered catalog of tables with row counts and last-write times beats a curated one that is three months stale.
- Assign owners before writing policies. An unowned dataset cannot comply with anything.
- Classify. Four levels is enough; more than six will not be used.
- Encode a small number of rules and enforce them. Three enforced rules beat thirty aspirational ones.
- Report, do not police. Publish a per-domain compliance score and let owners fix their own gaps; make exceptions explicit and time-bound.
- Attach governance to the pipeline, not to a review meeting. A rule that only runs when someone remembers is not enforced.
Two framework names worth recognising because they structure many corporate programmes: DAMA-DMBOK (a broad body of knowledge across eleven knowledge areas including quality, metadata, security, and architecture) and DCAM (a capability-maturity assessment model). Both are useful as checklists of what you might be missing and useless as a substitute for the six steps above.
Key Takeaways
- Governance answers "who decides?" about definitions, access, retention, and permitted use; without it those decisions are made implicitly by whoever writes the query.
- Effective governance needs named accountability, machine-checkable rules, and a gate that can block a release.
- Separate the roles: owner decides, steward maintains meaning, custodian operates controls, DPO handles legal compliance.
- Centralised governance is consistent but bottlenecked; federated governance scales but needs real enforcement of global rules.
- Encode policy as code that runs in CI over the catalog, so violations are visible rather than remembered.
- Combine compliance findings with lineage: an unowned dataset with downstream production consumers is the real risk.
- Adopt in order -- inventory, owners, classification, a few enforced rules, published scores -- and keep exceptions explicit and time-bound.
- DAMA-DMBOK and DCAM are useful checklists, not substitutes for enforcement.
🧪 Practice
- Build a minimal catalog of five datasets you work with, listing owner, steward, classification, and retention; note how many fields you cannot fill in.
- Write three executable governance rules for your environment and run them over that catalog.
- Compute the downstream blast radius of your least-governed dataset.
- Draft a one-page exception process: who approves, for how long, and what triggers a review.
- Interview: Your company has a 40-page data governance policy and persistent data quality problems. What do you change first? (Hint: think about which of the three properties of effective governance the policy is missing, and what the smallest enforceable version would be.)
Data Lineage and Cataloging
Every data organisation eventually faces the same four questions, and none of them can be answered from code alone:
- Where did this number come from? (a dashboard figure someone disputes)
- What breaks if I change this column? (before you ship a schema change)
- Where does this person's data live? (a deletion request arrives)
- Who is using this table? (you want to deprecate it)
Lineage is the graph that answers them: nodes are datasets, tables, or columns, and edges record derivation. A catalog is lineage plus metadata -- descriptions, owners, classifications, freshness, quality test results, usage statistics -- made searchable.
The distinction that matters most in practice is granularity. Table-level lineage is easy to capture and answers the coarse questions. Column-level lineage is much harder and is what you need for deletion propagation, impact analysis, and finding out that a "de-identified" feature was derived from an email address.
TABLE-LEVEL COLUMN-LEVEL
raw_signup -> profile -> feat raw_signup.email -> profile.user_key
-> churn_model -> feat.avg_session
raw_signup.country -> profile.region
"the model depends on signup" -> feat.region_onehot
"the model depends on country, not on
email, except through the join key"
import textwrap
EDGES = { # node -> the nodes it is derived FROM
"raw_signup.email": [],
"raw_signup.country": [],
"raw_signup.signup_ts": [],
"events.user_key": ["raw_signup.email"],
"events.session_len": [],
"profile.user_key": ["raw_signup.email"],
"profile.tenure_days": ["raw_signup.signup_ts"],
"profile.region": ["raw_signup.country"],
"feat.tenure_bucket": ["profile.tenure_days"],
"feat.avg_session": ["events.session_len", "events.user_key"],
"feat.region_onehot": ["profile.region"],
"churn_model.score": ["feat.tenure_bucket", "feat.avg_session",
"feat.region_onehot"],
"dash.churn_by_region": ["churn_model.score", "profile.region"],
"campaign.target_list": ["churn_model.score"],
}
CHILD = {}
for n, ups in EDGES.items():
for u in ups: CHILD.setdefault(u, []).append(n)
def walk(start, graph): # transitive closure, either way
seen, stack = set(), list(graph.get(start, []))
while stack:
n = stack.pop()
if n in seen: continue
seen.add(n); stack += graph.get(n, [])
return sorted(seen)
def show(label, items):
for i, line in enumerate(textwrap.wrap(", ".join(items), 48)):
print(f" {label if i == 0 else '':24}{'<- ' if i == 0 else ' '}{line}")
print("1. UPSTREAM: WHERE DID THIS NUMBER COME FROM?")
for n in ("churn_model.score", "campaign.target_list"):
show(n, walk(n, EDGES))
print("\n2. DOWNSTREAM: WHAT BREAKS IF I CHANGE THIS COLUMN?")
print(f" {'column':26}{'blast radius':>14} affected")
for n in ("raw_signup.email", "raw_signup.country", "events.session_len",
"profile.tenure_days"):
d = walk(n, CHILD)
print(f" {n:26}{len(d):14} "
f"{', '.join(dict.fromkeys(x.split('.')[0] for x in d))}")
print("\n3. PROPAGATION: A DELETION REQUEST TOUCHES EVERY DERIVED ASSET")
touched = {"raw_signup.email"} | set(walk("raw_signup.email", CHILD))
print(f" erasing one subject's email must be reflected in "
f"{len(touched)} assets:")
for t in sorted(touched): print(f" {t}")
print("\n4. ORPHANS AND DEAD ENDS: WHAT THE CATALOG SHOULD FLAG")
roots = [n for n, u in EDGES.items() if not u]
leaves = [n for n in EDGES if n not in CHILD]
no_consumer = [n for n in leaves if not n.startswith(("dash.", "campaign."))]
print(f" source columns (no upstream): {len(roots)}")
print(f" terminal columns (no consumer): {len(leaves)}")
print(f" computed but never consumed: {no_consumer}")
1. UPSTREAM: WHERE DID THIS NUMBER COME FROM?
churn_model.score <- events.session_len, events.user_key,
feat.avg_session, feat.region_onehot,
feat.tenure_bucket, profile.region,
profile.tenure_days, raw_signup.country,
raw_signup.email, raw_signup.signup_ts
campaign.target_list <- churn_model.score, events.session_len,
events.user_key, feat.avg_session,
feat.region_onehot, feat.tenure_bucket,
profile.region, profile.tenure_days,
raw_signup.country, raw_signup.email,
raw_signup.signup_ts
2. DOWNSTREAM: WHAT BREAKS IF I CHANGE THIS COLUMN?
column blast radius affected
raw_signup.email 6 campaign, churn_model, dash, events, feat, profile
raw_signup.country 5 campaign, churn_model, dash, feat, profile
events.session_len 4 campaign, churn_model, dash, feat
profile.tenure_days 4 campaign, churn_model, dash, feat
3. PROPAGATION: A DELETION REQUEST TOUCHES EVERY DERIVED ASSET
erasing one subject's email must be reflected in 7 assets:
campaign.target_list
churn_model.score
dash.churn_by_region
events.user_key
feat.avg_session
profile.user_key
raw_signup.email
4. ORPHANS AND DEAD ENDS: WHAT THE CATALOG SHOULD FLAG
source columns (no upstream): 4
terminal columns (no consumer): 3
computed but never consumed: ['profile.user_key']
The third query is the one that turns lineage from a nice-to-have into a
compliance requirement. A single deletion request reaches seven assets, and two
of them -- churn_model.score and dash.churn_by_region -- are derived
artifacts where deletion is not obvious. Does a trained model "contain" the
deleted person's data? That is a real and unsettled question, and the only
defensible position is a documented decision: either the model is retrained on
a schedule that bounds how long any record influences it, or you have a written
argument that the model output is not personal data. Both are acceptable;
having neither is not.
The fourth query illustrates the catalog's quieter value: profile.user_key is
computed and consumed by nothing. Either it is dead code costing money, or it is
an undocumented export somebody is reading outside the graph you modelled. Both
are findings.
How lineage actually gets captured, in rough order of reliability:
| Method | Coverage | Effort |
|---|---|---|
| Query log parsing (warehouse) | Automatic, table and often column level | Low; the best default |
| Transformation-tool metadata (dbt, Spark plans, orchestrator) | Very good within the tool | Low if you already use it |
| Runtime instrumentation of jobs | Broad, including non-SQL | Moderate |
| Manual declaration | Whatever people remember | High, decays immediately |
Prefer automatic capture and treat manual declaration as a gap-filler. A hand-maintained lineage diagram is out of date the week it is drawn, and -- worse -- it is believed.
Two final practices that make a catalog used rather than merely built. Make it searchable from where people work (an IDE or notebook extension, a slash command, a warehouse UI) because a catalog nobody opens documents nothing. And surface freshness and quality state on the asset itself, so the person about to use a table sees that its tests last failed yesterday.
Key Takeaways
- Lineage is the derivation graph that answers where a number came from, what a change breaks, where a person's data lives, and who uses a table.
- A catalog is lineage plus searchable metadata: owners, classifications, freshness, quality results, and usage.
- Column-level lineage is what deletion propagation and real impact analysis require; table-level is only a start.
- A deletion request reaches every derived asset, including models and aggregates, and each needs a documented decision about what deletion means.
- Assets computed but never consumed are either dead cost or undocumented exports; both are findings.
- Capture lineage automatically from query logs and transformation tooling; manual diagrams decay immediately and are trusted anyway.
- Combine lineage with governance metadata so risk can be ranked by blast radius rather than by dataset.
- Put the catalog where people already work and show freshness and quality on the asset itself.
🧪 Practice
- Build a column-level lineage graph for one pipeline you own and compute the blast radius of each source column.
- Trace one personal data field to every downstream asset and write what deletion means for each.
- Find an asset in your environment with no downstream consumer and determine whether it can be removed.
- Extract lineage automatically from your warehouse's query log for one day and compare it with what you believed the dependencies were.
- Interview: A dashboard number disagrees with a finance report. How does lineage help, and what does it not tell you? (Hint: it shows the two derivation paths but not which definition is correct -- think about what other metadata you need.)
Regulatory Landscape: GDPR and Beyond
Regulation is a design constraint, not an afterthought, and the reason is structural: several of the rules restrict things that are easy to change before you collect data and nearly impossible afterwards. If you gather data without a lawful basis, no amount of later engineering makes the resulting model lawful.
The GDPR is the reference framework because most later laws borrow from it. Its core mechanics, in the order they bind you:
You need a lawful basis for every processing purpose, chosen before processing and recorded. There are six, and they are not ranked -- but they have very different consequences:
| Basis | When it applies | Consequence for data science |
|---|---|---|
| Consent | Freely given, specific, informed, withdrawable | Must be withdrawable, which means data must be removable |
| Contract | Necessary to deliver what the person signed up for | Narrow: "necessary", not "useful" |
| Legal obligation | A law requires it | Often overrides erasure requests |
| Vital interests | Life-or-death | Rare |
| Public task | Public authority functions | Public sector |
| Legitimate interests | Your interest, balanced against their rights | Needs a documented balancing test; the usual basis for analytics |
Special category data -- health, race, ethnicity, political opinions, religion, trade union membership, genetics, biometrics, sex life, sexual orientation -- is prohibited by default and requires a specific additional condition, usually explicit consent. Note the trap: a model that infers a special category creates special category data, whether or not you collected it.
Data subjects have rights with a one-month statutory deadline: access, rectification, erasure, restriction, portability, objection, and rights concerning automated decision-making.
Article 22 restricts decisions based solely on automated processing that have legal or similarly significant effects. Where it applies, you need an explicit basis, must provide meaningful information about the logic involved, and must offer human review.
Beyond the GDPR, the landscape a global model operates in:
| Regime | Scope | The thing that most often surprises teams |
|---|---|---|
| GDPR (EU/EEA) | Personal data of people in the EU | Applies extraterritorially; fines up to 4% of global turnover |
| UK GDPR / DPA 2018 | Same structure, separate regulator | Divergence is growing |
| CCPA/CPRA (California) | Consumers of qualifying businesses | Opt-out of "sale/sharing"; broad definition of sharing |
| EU AI Act | AI systems by risk tier | Obligations attach to the use case, with bans and high-risk duties |
| HIPAA (US health) | Protected health information by covered entities | De-identification has a specified standard (Safe Harbor or expert determination) |
| ECOA / Reg B, FCRA (US credit) | Credit decisions | Adverse action notices must give specific principal reasons |
| Illinois BIPA | Biometric identifiers | Private right of action; written consent before collection |
| Brazil LGPD, India DPDP, China PIPL | National regimes | PIPL adds data localisation and export approval |
import datetime as dt
TODAY = dt.date(2026, 9, 28)
DEADLINE_DAYS = 30 # GDPR Art. 12(3): one month
REQUESTS = [
dict(id="DSR-101", kind="access", received=dt.date(2026, 9, 5),
verified=True, note=""),
dict(id="DSR-102", kind="erasure", received=dt.date(2026, 8, 20),
verified=True, note="tax records retained 7y (legal obligation)"),
dict(id="DSR-103", kind="portability", received=dt.date(2026, 9, 20),
verified=False, note="identity not yet verified; clock paused"),
dict(id="DSR-104", kind="objection", received=dt.date(2026, 6, 30),
verified=True, note="no response sent"),
]
print("1. DATA SUBJECT REQUESTS AGAINST THE STATUTORY CLOCK")
print(f" {'id':9}{'type':13}{'day':>5}{'days left':>11}{'state':>16} note")
for r in REQUESTS:
age = (TODAY - r["received"]).days
left = DEADLINE_DAYS - age
if not r["verified"]: state = "on hold"
elif r["note"] and "retained" in r["note"]: state = "partial refuse"
elif left < 0: state = "BREACH"
elif left <= 7: state = "urgent"
else: state = "in progress"
print(f" {r['id']:9}{r['kind']:13}{age:5}{left:11}{state:>16} {r['note']}")
print("\n2. IS EACH PROCESSING ACTIVITY LAWFUL? (a minimal ROPA check)")
BASES = {"consent", "contract", "legal_obligation", "vital_interests",
"public_task", "legitimate_interests"}
ROPA = [
dict(purpose="billing", basis="contract",
special=False, dpia=False, transfer=None, retention=2555),
dict(purpose="churn_model", basis="legitimate_interests",
special=False, dpia=True, transfer=None, retention=730),
dict(purpose="ad_targeting", basis="legitimate_interests",
special=False, dpia=False, transfer="US", retention=1095),
dict(purpose="health_risk_score", basis="consent",
special=True, dpia=False, transfer=None, retention=365),
dict(purpose="employee_monitoring", basis=None,
special=False, dpia=False, transfer="IN", retention=None),
]
ADEQUATE = {"CH", "JP", "KR", "UK", "CA"} # illustrative adequacy list
print(f" {'purpose':22}{'findings':>10} detail")
for p in ROPA:
f = []
if p["basis"] not in BASES: f.append("no lawful basis")
if p["basis"] == "legitimate_interests" and not p["dpia"]:
f.append("LIA/DPIA missing")
if p["special"] and p["basis"] != "consent": f.append("Art.9 basis wrong")
if p["special"] and not p["dpia"]: f.append("DPIA required")
if p["transfer"] and p["transfer"] not in ADEQUATE:
f.append(f"transfer to {p['transfer']} needs SCCs + TIA")
if p["retention"] is None: f.append("no retention limit")
elif p["retention"] > 2000 and p["purpose"] != "billing":
f.append("retention looks excessive")
print(f" {p['purpose']:22}{len(f):10} {'; '.join(f) if f else 'ok'}")
print("\n3. AUTOMATED DECISION-MAKING: DOES ARTICLE 22 APPLY?")
SYSTEMS = [
dict(name="churn score -> email", solely_automated=True, legal_effect=False),
dict(name="credit decline", solely_automated=True, legal_effect=True),
dict(name="fraud hold + review", solely_automated=False, legal_effect=True),
dict(name="CV shortlisting", solely_automated=True, legal_effect=True),
]
print(f" {'system':26}{'Art.22?':>9} obligation")
for s in SYSTEMS:
applies = s["solely_automated"] and s["legal_effect"]
duty = ("explicit basis + logic disclosed + human review"
if applies else
"transparency only" if not s["legal_effect"]
else "human step must be genuine, not a rubber stamp")
print(f" {s['name']:26}{'YES' if applies else 'no':>9} {duty}")
1. DATA SUBJECT REQUESTS AGAINST THE STATUTORY CLOCK
id type day days left state note
DSR-101 access 23 7 urgent
DSR-102 erasure 39 -9 partial refuse tax records retained 7y (legal obligation)
DSR-103 portability 8 22 on hold identity not yet verified; clock paused
DSR-104 objection 90 -60 BREACH no response sent
2. IS EACH PROCESSING ACTIVITY LAWFUL? (a minimal ROPA check)
purpose findings detail
billing 0 ok
churn_model 0 ok
ad_targeting 2 LIA/DPIA missing; transfer to US needs SCCs + TIA
health_risk_score 1 DPIA required
employee_monitoring 3 no lawful basis; transfer to IN needs SCCs + TIA; no retention limit
3. AUTOMATED DECISION-MAKING: DOES ARTICLE 22 APPLY?
system Art.22? obligation
churn score -> email no transparency only
credit decline YES explicit basis + logic disclosed + human review
fraud hold + review no human step must be genuine, not a rubber stamp
CV shortlisting YES explicit basis + logic disclosed + human review
Three observations that translate the law into engineering work.
The clock is real and it starts on receipt. DSR-104 is 90 days old with no response: that is a regulatory breach independent of whether the underlying request was reasonable. Request handling must be a tracked workflow with a timer, not an inbox. Note also that DSR-102 is a legitimate partial refusal -- erasure yields to a legal retention obligation for the tax records, but the other 88 records must still be deleted, and the refusal must be explained.
The records of processing double as a compliance test. The ROPA check flags what an auditor would: analytics relying on legitimate interests with no balancing assessment, a special-category model with no DPIA, and a transfer to a country without an adequacy decision needing standard contractual clauses plus a transfer impact assessment. All of these are answerable before a model exists.
Article 22 hinges on two words: "solely" and "significant". The fraud case escapes it because a human reviews the hold -- but only if that review is genuine. A human who approves 3,000 automated decisions an hour is not meaningful involvement, and regulators have said so. If you build the human step, you must give the human the information and the time to actually use it.
Two further instruments to know by name. A DPIA (data protection impact assessment) is mandatory for high-risk processing -- large-scale profiling, special category data, systematic monitoring -- and is the natural place to put your fairness and privacy analysis. Privacy by design and by default is a legal requirement, not a slogan: the default configuration must be the private one.
A closing caution, which is also a professional boundary: this is a map, not legal advice. Your job is to recognise when a modelling choice has legal consequences and to bring it to someone qualified early, when the choice is still cheap.
Key Takeaways
- Every processing purpose needs a lawful basis chosen and recorded before processing begins; retrofitting one is not possible.
- Consent must be withdrawable, which means the architecture must support removing a person's data and its downstream effects.
- Legitimate interests is the usual basis for analytics and requires a documented balancing assessment.
- Special category data is prohibited by default, and a model that infers a special category creates it.
- Data subject rights carry a one-month deadline from receipt; handle them as a tracked workflow with a timer, and explain partial refusals.
- Article 22 applies to solely automated decisions with significant effects; a rubber-stamp human review does not remove it.
- DPIAs are mandatory for high-risk processing and are the right place for fairness and privacy analysis.
- Cross-border transfers need an adequacy decision or safeguards such as SCCs plus a transfer impact assessment.
- Recognise legal consequences early and escalate; this is not legal advice.
🧪 Practice
- Write the records of processing for one system you work on: purposes, lawful bases, categories of data, retention, recipients, and transfers.
- Determine whether Article 22 applies to a decision system you know, and justify your answer on both "solely" and "significant".
- Implement the data subject request workflow with a deadline timer and a state for identity verification.
- Identify one feature in a model of yours that could function as a proxy for a special category, and state how you would test that.
- Interview: Your team wants to fine-tune a model on support chat transcripts. What do you check before starting? (Hint: work through purpose, lawful basis, what might be inside free text, where the compute runs, and what happens when someone asks to be deleted.)
Consent and Data Ownership
Consent looks like the simplest lawful basis and is in practice the most demanding, because it creates an ongoing obligation that has to be respected at every moment of use, not just at collection.
Valid consent under the GDPR has four properties, each with a concrete engineering consequence:
| Property | What it means | What you must build |
|---|---|---|
| Freely given | No penalty for refusing; not bundled with service access | Separate toggles; the product must work without the optional ones |
| Specific | Per purpose, not a blanket "we use your data" | A purpose taxonomy, and consent stored per purpose |
| Informed | Plain language, before collection | Versioned notices, with the version recorded alongside the consent |
| Withdrawable | As easy to withdraw as to give, at any time | Withdrawal propagating to every downstream use, including models |
The consequence people underestimate: consent is a time-varying state, not a flag. The question is never "did this user consent?" but "was this user consented for this purpose at the moment I used their data?" That requires an append-only ledger of grant and withdrawal events, not a boolean column.
import numpy as np, datetime as dt
rng = np.random.default_rng(31)
TODAY = dt.date(2026, 9, 28)
N = 5000
PURPOSES = ["service", "analytics", "personalisation", "marketing",
"third_party_sharing"]
GRANT_P = [1.00, 0.72, 0.55, 0.38, 0.11]
ledger = []
for uid in range(N):
signup = TODAY - dt.timedelta(days=int(rng.integers(30, 900)))
for p, pr in zip(PURPOSES, GRANT_P):
if rng.random() < pr:
ledger.append((uid, p, "grant", signup))
if rng.random() < 0.12: # some later withdraw
d = signup + dt.timedelta(days=int(rng.integers(1, 800)))
if d <= TODAY: ledger.append((uid, p, "withdraw", d))
def consent_at(events, when):
"""Latest event at or before `when` decides. Absence means NO consent."""
state = False
for kind, d in sorted(events, key=lambda t: t[1]):
if d <= when: state = (kind == "grant")
return state
index = {}
for uid, p, kind, d in ledger:
index.setdefault((uid, p), []).append((kind, d))
print("1. HOW MUCH OF THE POPULATION MAY YOU ACTUALLY USE, PER PURPOSE?")
print(f" {'purpose':22}{'granted ever':>14}{'valid today':>13}{'withdrawn':>11}")
for p in PURPOSES:
ever = sum(1 for (u, pp) in index if pp == p)
now = sum(1 for (u, pp), ev in index.items()
if pp == p and consent_at(ev, TODAY))
print(f" {p:22}{ever/N:13.1%}{now/N:13.1%}{(ever-now)/N:11.1%}")
print("\n2. CONSENT IS EVALUATED AT THE TIME OF USE, NOT AT TRAINING TIME")
train_day = TODAY - dt.timedelta(days=180)
p = "personalisation"
cohort = {u for (u, pp), ev in index.items()
if pp == p and consent_at(ev, train_day)} # the training cohort
still = {u for u in cohort if consent_at(index[(u, p)], TODAY)}
print(f" rows in the training cohort for '{p}': {len(cohort)}")
print(f" of those, still consented today: {len(still)}")
print(f" became unusable but remain inside the model: {len(cohort-still)}")
print("\n3. THE COST OF PURPOSE LIMITATION ON A MODELLING DATASET")
print(f" {'required purposes':56}{'usable rows':>13}{'share':>8}")
combos = [("service", ["service"]),
("service + analytics", ["service", "analytics"]),
("service + analytics + personalisation",
["service", "analytics", "personalisation"]),
("service + analytics + marketing",
["service", "analytics", "marketing"]),
("all five purposes", PURPOSES)]
for lab, c in combos:
ok = sum(1 for uid in range(N)
if all(consent_at(index.get((uid, p), []), TODAY) for p in c))
print(f" {lab:56}{ok:13}{ok/N:8.1%}")
1. HOW MUCH OF THE POPULATION MAY YOU ACTUALLY USE, PER PURPOSE?
purpose granted ever valid today withdrawn
service 100.0% 93.8% 6.2%
analytics 72.8% 67.8% 5.0%
personalisation 55.1% 51.1% 4.1%
marketing 37.9% 34.9% 3.0%
third_party_sharing 10.9% 10.3% 0.7%
2. CONSENT IS EVALUATED AT THE TIME OF USE, NOT AT TRAINING TIME
rows in the training cohort for 'personalisation': 2194
of those, still consented today: 2124
became unusable but remain inside the model: 70
3. THE COST OF PURPOSE LIMITATION ON A MODELLING DATASET
required purposes usable rows share
service 4690 93.8%
service + analytics 3187 63.7%
service + analytics + personalisation 1628 32.6%
service + analytics + marketing 1074 21.5%
all five purposes 58 1.2%
Three consequences for how you plan a project.
Consent rates are not 100% and they compound. Requiring analytics consent alone leaves 63.7% of the population; requiring four purposes leaves 21.5%; all five leaves 1.2%. A model specified over a feature set that spans many purposes quietly becomes a model of a tiny, self-selected subset -- which is also a sampling bias problem, connecting directly back to section 16.2. Design the feature set around the consent you have, or go and ask for more, but do not discover the shrinkage after building.
Withdrawal reaches into trained models. Seventy people in the training cohort have since withdrawn, and their influence remains in the model's parameters until it is retrained. The workable answers are: retrain on a schedule short enough to bound the exposure and say so in the notice; exclude withdrawn users at scoring time as an immediate measure; and use machine unlearning techniques where the stakes justify them. Doing nothing is a position you would have to defend.
The ledger must be queryable at the moment of use. A pipeline should join against the consent state as of the processing timestamp, and that join should be in shared, reviewed code -- not reimplemented per project, where the first mistake is silently using today's consent to justify a computation over last year's data.
On ownership, a clarification that prevents a great deal of confused discussion: in most legal systems, data is not property, and nobody "owns" it in the way one owns a chair. What exists instead is a bundle of specific rights and obligations:
| Actor | Role | Holds |
|---|---|---|
| Data subject | The person the data is about | Rights: access, erasure, portability, objection |
| Controller | Decides why and how data is processed | The obligations; liability |
| Processor | Processes on the controller's instructions | Contractual duties, breach notification |
| Joint controllers | Decide together | Must allocate responsibilities in an arrangement |
Treating this as ownership leads to bad conclusions in both directions -- "the user owns their data so they can demand anything" and "we collected it so we can do anything with it". The accurate framing is stewardship: you hold data under obligations, for declared purposes, for a limited time.
This also determines what happens with third parties. If you send data to a vendor, you are the controller and they are usually the processor, which needs a data processing agreement, a documented purpose, and a deletion commitment. Sending customer records to an external model provider is exactly this relationship.
Key Takeaways
- Valid consent is freely given, specific, informed, and withdrawable, and each property maps to something you must build.
- Consent is a time-varying state; store an append-only ledger of grants and withdrawals per purpose, not a boolean.
- Evaluate consent as of the moment of use, and keep that join in shared, reviewed code.
- Consent rates compound across purposes: requiring several purposes can reduce a usable population to a small, self-selected fraction.
- Population shrinkage from consent is also a sampling bias problem, so it affects model validity, not just legality.
- Withdrawal reaches trained models; bound the exposure with scheduled retraining, exclude withdrawn users at scoring time, and document the approach.
- Nobody "owns" personal data; there are data subjects with rights and controllers and processors with obligations.
- Sending data to a vendor or an external model API creates a controller-processor relationship needing an agreement and a deletion commitment.
🧪 Practice
- Design a consent ledger schema and write the query that returns whether a user was consented for a purpose as of an arbitrary timestamp.
- For a model you work on, list the purposes its features require and compute the usable population under each combination.
- Write the procedure for handling a withdrawal: what changes immediately, what changes at the next retrain, and what you would tell the user.
- Audit one pipeline for whether it filters on consent at all, and at which timestamp.
- Interview: Marketing wants to reuse churn model features for an advertising campaign. What do you check? (Hint: think about the purpose the data was collected under, and what fraction of the population would remain if you filtered correctly.)
Responsible AI Principles
Nearly every organisation now publishes a set of AI principles: fairness, transparency, accountability, privacy, safety, human oversight. They are almost identical everywhere, they are almost always correct, and on their own they change nothing. The reason is that a principle is a statement of intent and an engineering organisation runs on controls -- things that happen automatically, block something, or page someone.
The useful skill is therefore translation, in both directions: turning a principle into a control, and recognising when a proposed control does not actually serve the principle it is filed under.
Two things make that translation tractable. First, obligations should scale with risk, which is the structure of the EU AI Act: a tier determined by the use case rather than the algorithm. Second, documentation should be a structured artifact the pipeline can check, which is what model cards are for.
RISK TIERS AND WHAT THEY ATTACH TO
prohibited social scoring, manipulative techniques, some biometric
categorisation, emotion inference at work or school
high employment, credit, education, essential services, law
enforcement, medical devices, critical infrastructure
limited systems interacting with people, deepfakes, chatbots
-> transparency obligations
minimal spam filters, recommenders, forecasting -> voluntary codes
The SAME model architecture sits in three different tiers depending on
what decision it feeds.
SYSTEMS = [
dict(name="product recommender", domain="retail", affects_rights=False,
safety_critical=False, human_in_loop=True, vulnerable=False),
dict(name="CV screening", domain="employment", affects_rights=True,
safety_critical=False, human_in_loop=False, vulnerable=False),
dict(name="credit scoring", domain="finance", affects_rights=True,
safety_critical=False, human_in_loop=True, vulnerable=False),
dict(name="sepsis early warning", domain="health", affects_rights=True,
safety_critical=True, human_in_loop=True, vulnerable=True),
dict(name="emotion inference at work", domain="employment",
affects_rights=True, safety_critical=False, human_in_loop=False,
vulnerable=True),
dict(name="warehouse demand forecast", domain="logistics",
affects_rights=False, safety_critical=False, human_in_loop=True,
vulnerable=False),
]
BANNED = {("employment", "emotion inference at work")}
HIGH_RISK_DOMAINS = {"employment", "finance", "health", "education",
"law_enforcement", "essential_services"}
def tier(s):
if (s["domain"], s["name"]) in BANNED: return "prohibited"
if s["safety_critical"] or (s["affects_rights"]
and s["domain"] in HIGH_RISK_DOMAINS):
return "high"
if s["affects_rights"]: return "limited"
return "minimal"
DUTIES = {
"prohibited": "do not build; escalate",
"high": "risk mgmt, logging, oversight, conformity assmt",
"limited": "transparency to users, documentation, monitoring",
"minimal": "voluntary codes; normal engineering practice",
}
print("RISK TIER AND WHAT IT OBLIGES")
print(f" {'system':28}{'tier':>12} obligations")
for s in SYSTEMS:
print(f" {s['name']:28}{tier(s):>12} {DUTIES[tier(s)]}")
print("\n2. A MODEL CARD IS A STRUCTURED ARTIFACT, NOT A README")
REQUIRED = ["intended_use", "out_of_scope_use", "training_data",
"evaluation_data", "metrics_overall", "metrics_by_subgroup",
"known_limitations", "ethical_considerations", "owner",
"version", "review_date", "fairness_assessment"]
cards = {
"churn_v3": {"intended_use": 1, "out_of_scope_use": 1, "training_data": 1,
"evaluation_data": 1, "metrics_overall": 1, "owner": 1,
"version": 1, "review_date": 1},
"credit_v7": {k: 1 for k in REQUIRED},
"resume_rank_v1": {"intended_use": 1, "metrics_overall": 1, "owner": 1},
}
print(f" {'model':18}{'completeness':>14} missing")
for name, card in cards.items():
missing = [k for k in REQUIRED if k not in card]
detail = ', '.join(missing[:4]) + (' ...' if len(missing) > 4 else '')
print(f" {name:18}{1-len(missing)/len(REQUIRED):13.0%} "
f"{detail or 'none'}")
print("\n3. FROM PRINCIPLE TO CONTROL: THE ONLY TRANSLATION THAT MATTERS")
MAP = [
("fairness", "subgroup metrics with CIs in CI; block on regression"),
("transparency", "model card required by the deploy pipeline"),
("accountability", "named owner in the registry; on-call rotation"),
("privacy", "column classification + retention job + access logs"),
("robustness", "drift monitors, adversarial test suite, rollback drill"),
("human oversight","appeal route with an SLA and a reviewer who can override"),
("contestability", "reason codes generated per decision and stored"),
]
for pr, control in MAP:
print(f" {pr:17} -> {control}")
RISK TIER AND WHAT IT OBLIGES
system tier obligations
product recommender minimal voluntary codes; normal engineering practice
CV screening high risk mgmt, logging, oversight, conformity assmt
credit scoring high risk mgmt, logging, oversight, conformity assmt
sepsis early warning high risk mgmt, logging, oversight, conformity assmt
emotion inference at work prohibited do not build; escalate
warehouse demand forecast minimal voluntary codes; normal engineering practice
2. A MODEL CARD IS A STRUCTURED ARTIFACT, NOT A README
model completeness missing
churn_v3 67% metrics_by_subgroup, known_limitations, ethical_considerations, fairness_assessment
credit_v7 100% none
resume_rank_v1 25% out_of_scope_use, training_data, evaluation_data, metrics_by_subgroup ...
3. FROM PRINCIPLE TO CONTROL: THE ONLY TRANSLATION THAT MATTERS
fairness -> subgroup metrics with CIs in CI; block on regression
transparency -> model card required by the deploy pipeline
accountability -> named owner in the registry; on-call rotation
privacy -> column classification + retention job + access logs
robustness -> drift monitors, adversarial test suite, rollback drill
human oversight -> appeal route with an SLA and a reviewer who can override
contestability -> reason codes generated per decision and stored
The tiering table makes the key conceptual point: the same model is in different tiers depending on what it decides. A gradient boosting model forecasting warehouse demand is minimal risk; the identical algorithm ranking job applicants is high risk. Risk is a property of the deployment, which is why model governance must be attached to use cases, and why "we already approved this model" is not an answer when someone repurposes it.
The model card table is what converts documentation from good intentions into a gate. Three fields are the ones that are almost always missing and almost always matter most: out-of-scope use (the explicit list of things this model must not be used for, which is your only defence against repurposing), metrics by subgroup (without which no fairness claim is possible), and known limitations (which is where the honest caveats live). Note the pattern in the output: the credit model, which is regulated, is complete; the résumé ranker, which is the highest-risk system in the list, is 25% complete. That inversion is extremely common, because regulation forces documentation and internal tools do not.
The third block is the discipline to internalise. For each principle, write the control, then ask three questions: what does it block?, who is paged when it fires?, and how would I know if it silently stopped working? A principle with no answer to those is decoration.
Finally, three governance structures that make the controls stick:
- A model registry as the single source of truth -- every production model with its owner, version, card, risk tier, and approval state. If a model can reach production without an entry, the registry is documentation, not a control.
- A review gate proportional to tier. Minimal-risk models pass on automated checks; high-risk models require documented sign-off from someone who is not the author, plus fairness and privacy assessments.
- An incident and appeal path with a service level. People affected by decisions need a route to contest them, and the organisation needs a record of the outcomes -- which is also the dataset that tells you where the model is wrong.
Key Takeaways
- Responsible AI principles are nearly identical everywhere and change nothing until they become controls that block, alert, or require sign-off.
- Obligations should scale with risk, and risk is a property of the use case, not the algorithm: the same model can be minimal or high risk.
- Approval must be per use case, so repurposing an approved model requires a new assessment.
- Model cards are structured artifacts whose completeness can be checked by the deploy pipeline.
- The most commonly missing and most valuable card fields are out-of-scope use, subgroup metrics, and known limitations.
- Regulated systems tend to be well documented and internal tools poorly documented, which inverts the actual risk ordering.
- For every control, know what it blocks, who is paged, and how you would detect that it stopped working.
- A model registry, tier-proportional review gates, and an appeal path with an SLA are what make the rest durable.
🧪 Practice
- Assign a risk tier to three systems you work on, justifying each from the decision it feeds rather than the model type.
- Write a model card for one of your models and mark every field you cannot complete honestly.
- Add a CI check that fails a deployment when a required model card field is missing.
- Take two of your organisation's published AI principles and write the control, the owner, and the alert for each.
- Interview: An internal tool built for triage is being used to make final decisions. How did that happen and what do you do? (Hint: think about what the original approval covered, and which single model card field would have made the change visible.)
Research Ethics and IRB Considerations
Most data science is not research on human subjects, and some of it is -- and the boundary is not where practitioners usually assume. If you are running an experiment on people to produce generalisable knowledge, you are doing human subjects research, whether it happens in a university or in a product A/B testing framework.
The framework in use worldwide descends from the Belmont Report, written after a series of abuses in medical research. Its three principles map cleanly onto data work:
| Principle | Means | In a data science project |
|---|---|---|
| Respect for persons | Autonomy; informed consent; extra protection for those with diminished autonomy | Meaningful consent for the actual use; extra care with children, patients, employees, prisoners |
| Beneficence | Maximise benefit, minimise harm; the risk must be justified | A written risk analysis, a stopping rule, and an honest account of who benefits |
| Justice | Fair distribution of burdens and benefits | Do not experiment on the vulnerable while benefiting others; connect directly to section 16.2 |
An IRB (institutional review board, or research ethics committee) reviews protocols against these before the work begins. Three routes exist, and determining which applies is a screening question, not a judgement call:
is it human subjects research? (living individuals AND
| intervention/interaction OR
| identifiable private information)
no -> out of scope / exempt
yes
|
minimal risk, no vulnerable group, no deception?
yes -> expedited review (single reviewer)
no -> full board review (convened committee)
STUDIES = [
dict(name="public corpus, aggregate stats only",
identifiable=False, interaction=False, intervention=False,
sensitive=False, vulnerable=False, deception=False, minimal_risk=True),
dict(name="survey of our own employees",
identifiable=True, interaction=True, intervention=False,
sensitive=False, vulnerable=True, deception=False, minimal_risk=True),
dict(name="newsfeed mood manipulation A/B test",
identifiable=True, interaction=False, intervention=True,
sensitive=True, vulnerable=False, deception=True, minimal_risk=False),
dict(name="retrospective de-identified chart review",
identifiable=False, interaction=False, intervention=False,
sensitive=True, vulnerable=False, deception=False, minimal_risk=True),
dict(name="wearable trial, paediatric clinic",
identifiable=True, interaction=True, intervention=True,
sensitive=True, vulnerable=True, deception=False, minimal_risk=False),
]
def route(s):
"""Human subjects research = living individuals, plus interaction or
identifiable private information. Everything else is out of scope."""
if not s["identifiable"] and not s["interaction"] and not s["intervention"]:
return "exempt / not human subjects"
if not s["minimal_risk"] or s["vulnerable"] or s["deception"]:
return "full board review"
return "expedited review"
def artifacts(s):
a = ["protocol", "data mgmt plan"]
if s["identifiable"]: a.append("informed consent")
if s["deception"]: a.append("debriefing")
if s["vulnerable"]: a.append("vulnerable-group safeguards")
if s["sensitive"]: a.append("restricted storage + breach plan")
if s["intervention"]: a.append("stopping rule")
return a
print(f" {'study':40}{'route':>28}")
for s in STUDIES:
print(f" {s['name']:40}{route(s):>28}")
print(f" required: {', '.join(artifacts(s))}")
study route
public corpus, aggregate stats only exempt / not human subjects
required: protocol, data mgmt plan
survey of our own employees full board review
required: protocol, data mgmt plan, informed consent, vulnerable-group safeguards
newsfeed mood manipulation A/B test full board review
required: protocol, data mgmt plan, informed consent, debriefing, restricted storage + breach plan, stopping rule
retrospective de-identified chart review exempt / not human subjects
required: protocol, data mgmt plan, restricted storage + breach plan
wearable trial, paediatric clinic full board review
required: protocol, data mgmt plan, informed consent, vulnerable-group safeguards, restricted storage + breach plan, stopping rule
Two of these deserve comment because they are the cases practitioners get wrong.
The employee survey goes to full review because employees are a vulnerable population in the technical sense: their consent is not freely given when their employer is asking. The same logic applies to students, patients of the treating clinician, and anyone in a dependent relationship with the researcher.
The newsfeed experiment is the well-known hard case. Product A/B tests are ordinarily not research -- they are operational decisions about your own service, and nobody expects an IRB to review a button colour. The line is crossed when the intervention is designed to affect people's psychological state, when the purpose is generalisable knowledge rather than a product decision, or when harm is plausible. The 2014 Facebook emotional contagion study is the canonical example of that line being crossed without review, and the resulting damage was reputational as much as regulatory.
For everyday data science work that is not formal research, the useful discipline is to borrow the structure without the paperwork:
- Write down who could be harmed and how, before starting. One page, kept with the project.
- State the stopping rule. Under what observed result do you halt the experiment? A test with no stopping rule has no harm bound.
- Ask whether the affected people would be surprised. Surprise is a good proxy for a consent problem, and it is a question anyone can answer.
- Check the burden distribution. Who bears the risk of the experiment and who gets the benefit? If they are different groups, that is the justice principle flagging.
- Do not experiment on vulnerable populations by default, including employees, and be explicit when you must.
- Get an outside opinion for anything you would not describe publicly. The reluctance to describe it is the finding.
Two further obligations that arise in research contexts and increasingly in industry:
- Secondary use. Data collected for one study is not automatically available for the next. Reusing a research dataset for model training usually requires the original consent to have covered it, or a new approval.
- Publication ethics. Pre-registration, reporting negative results, declaring conflicts of interest, and releasing enough detail for replication -- which connects to the reproducibility practices in the next chapter.
Key Takeaways
- Human subjects research is defined by living individuals plus intervention, interaction, or identifiable private information -- not by whether you work at a university.
- The Belmont principles are respect for persons, beneficence, and justice, and each maps to concrete project artifacts.
- Screening determines the route: out of scope, expedited review, or full board review; vulnerable groups or deception force the last one.
- Employees, students, and patients of the treating clinician are vulnerable populations because their consent is not freely given.
- Ordinary product A/B tests are not research, but manipulating psychological state or seeking generalisable knowledge crosses the line.
- Every experiment needs a stopping rule, or it has no bound on harm.
- "Would the affected people be surprised?" is a practical, fast proxy for a consent problem.
- Secondary use of research data needs the original consent to cover it or a fresh approval, and publication carries its own ethical duties.
🧪 Practice
- Apply the screening logic above to three projects you have worked on and state the route each would take.
- Write a one-page harms analysis with a stopping rule for an experiment you are planning.
- Identify a dataset in your organisation that was collected for one purpose and is now used for another, and determine what would make that lawful.
- Draft the informed consent text for a study using your product's data, in language a non-specialist would understand.
- Interview: Your company wants to A/B test messages that increase urgency in a financial product's checkout flow. Is this research, and what would you require? (Hint: consider whether the intervention targets a psychological state, who is most affected by urgency pressure, and what the stopping rule would be.)
<a id="17-communication-and-professional-practice"></a>