17. Communication and Professional Practice

This chapter covers the work that surrounds the analysis: explaining a result to someone who will not read the code, negotiating what the project is actually for, and making the work solid enough that a colleague — or you in six months — can rerun it and reach the same number. These skills decide whether technically correct work changes anything, and they are the most common reason strong analyses are ignored while weak ones get shipped.

<a id="171-communicating-results"></a>

17.1 Communicating Results

An analysis exists to change someone's mind or someone's decision. This section covers how to carry a finding across the gap between the person who computed it and the person who must act on it, without losing the caveats that make it honest.

Knowing Your Audience

The instinct when presenting analysis is to describe what you did, in the order you did it. This is the wrong instinct, and it comes from a real place: you spent three weeks on the work, the hard parts were technical, and the technical parts feel like the content. But the audience did not spend three weeks on it. They have five minutes, a decision to make, and no reason to care how the missing values were imputed.

The fix is to ask what the listener is for. Every audience has a decision they own and a vocabulary they think in, and your job is to enter through that door. An executive owns resource allocation and thinks in money and risk. A product manager owns roadmap and thinks in user behavior and tradeoffs. An engineer owns the system and thinks in latency, failure modes, and interfaces. A fellow data scientist owns whether your work is correct and thinks in assumptions and identification. The same finding — "the churn model scores 0.81 AUC" — is useless to three of those four people until it is translated.

An analogy: a doctor explaining the same MRI to a patient, a surgeon, and an insurance adjuster says three different true things. The patient hears what happens next. The surgeon hears where to cut. The adjuster hears a billing code. None of them is being lied to, and none of them wants the other two explanations.

  SAME RESULT, FOUR TRANSLATIONS
  finding: churn model, AUC 0.81, top decile captures 44% of churners

  EXECUTIVE      -> "We can find nearly half of the customers who will
                     leave by contacting 10% of the base. At current
                     retention cost that is a 3.2x return; I need two
                     engineers for a quarter."

  PRODUCT        -> "The model flags at-risk users a month early. The
                     strongest signal is support tickets without
                     resolution, so the intervention is probably a
                     service fix, not a discount."

  ENGINEER       -> "Batch scoring nightly, 40ms p99, reads from the
                     events table, writes a score column. No online
                     features, so no feature store needed yet."

  DATA SCIENTIST -> "Time-based split, calibrated on the last fold,
                     AUC 0.81 +/- 0.02. Label is 30-day non-renewal.
                     Tenure leaks slightly; I capped it."

The practical technique is a translation table written before the meeting, not improvised during it.

# A finding is a fact plus an audience-specific frame. Keeping the fact in
# one place prevents the frames from drifting away from the numbers.

finding = {
    "metric": "top_decile_recall",
    "value": 0.44,
    "baseline": 0.10,          # random targeting captures 10% of churners
    "cost_per_contact": 12.0,
    "value_per_save": 340.0,
    "monthly_churners": 2_100,
}

def frame_for(audience, f):
    """Return the one sentence that audience can act on."""
    captured = int(f["monthly_churners"] * f["value"])
    contacts = int(f["monthly_churners"] / 0.10)   # decile of the base
    net = captured * 0.25 * f["value_per_save"] - contacts * f["cost_per_contact"]
    return {
        # Executives need the decision and its magnitude, not the metric.
        "executive": (
            f"Targeting 10% of customers reaches {captured} of the {f['monthly_churners']} "
            f"likely leavers each month; at a 25% save rate that is "
            f"${net:,.0f} net per month."
        ),
        # Product needs what to build, which means the driver, not the score.
        "product": (
            f"We can identify {f['value']:.0%} of churners a month ahead, "
            "which is enough lead time for an in-product intervention."
        ),
        # Engineering needs the contract: inputs, cadence, outputs.
        "engineering": (
            "Nightly batch job, one score per account, written to "
            "`accounts.churn_score`; no real-time path required."
        ),
        # Peers need what would falsify it.
        "peer": (
            f"Top-decile recall {f['value']:.2f} vs {f['baseline']:.2f} random, "
            "time-based split, label = 30-day non-renewal."
        ),
    }[audience]

print(frame_for("executive", finding))

Two failure modes bracket this skill. Under-translating buries the decision under method and the audience disengages. Over-translating drops the uncertainty and the audience acts on a number more confidently than the evidence supports. The target is a sentence that is both actionable and still true.

Key Takeaways

• Lead with the decision the audience owns, not the order in which you did the work.
• Executives think in money and risk, product in behavior and tradeoffs, engineers in interfaces and failure modes, peers in assumptions.
• Keep one canonical set of numbers and derive each audience's framing from it, so the framings cannot drift apart.
• Translating is not simplifying away the caveats; a frame that is actionable but false is worse than silence.

🧪 Practice

1. Take a result you produced recently and write four one-sentence versions of it, one each for an executive, a product manager, an engineer, and a peer reviewer.
2. Find a technical blog post and rewrite its conclusion for a reader who has budget authority but no statistics background.
3. List the three questions each of your actual stakeholders asked in the last month. Identify what decision each question was really about.
4. Interview: You present a model to an executive who immediately asks "what is AUC?" What went wrong, and what do you say next? (Hint: the recovery and the root cause are different things; do not fix only one.)

Data Storytelling Structure

People remember narratives and forget lists. This is not a weakness to work around; it is a constraint on how information reaches a decision. An analysis presented as a sequence of findings — here is a chart, here is another chart, here is a table — forces the audience to do the assembly themselves, and they will assemble it wrong or not at all.

A data story imposes structure: a situation everyone accepts, a complication that makes the situation unstable, a question that the complication raises, and an answer supported by evidence. This is the SCQA pattern from consulting, and it works because it earns attention before spending it. The audience agrees with the situation, feels the tension of the complication, and therefore wants the answer.

The contrast with the way analysis is done is important. Analysis is exploratory and works forward: load data, look around, test ideas, arrive somewhere. Communication works backward: state where you arrived, then show only the evidence that supports it. The dead ends were necessary work and they do not belong in the story. An appendix is the correct home for them.

  ANALYSIS ORDER (how you worked)      STORY ORDER (how you present)

  1. pulled 14 months of orders        1. SITUATION
  2. found duplicate rows                 "Repeat purchase drives 70%
  3. rebuilt the cohort join               of revenue."
  4. tried 6 feature sets              2. COMPLICATION
  5. three models, two failed             "Second-order rate fell from
  6. the good one needs recency             31% to 24% since March."
  7. checked it holds by segment       3. QUESTION
  8. conclusion: delivery time            "What changed, and is it
     is the driver                          something we control?"
                                       4. ANSWER
  Presenting in this order forces         "Delivery time. Orders over
  the audience to hold 7 facts             4 days repeat at half the
  before hearing why they matter.          rate; volume shifted to a
                                           slower carrier in March."
                                       5. EVIDENCE + what would change it

The discipline shows up in artifacts. Chart titles carry the finding rather than describing the axes, and the ordering of slides follows the argument rather than the workflow.

import matplotlib.pyplot as plt

# WEAK: the title names the variables and leaves interpretation to the reader.
# ax.set_title("Repeat rate by delivery days")

# STRONG: the title states the claim; the chart becomes its evidence.
fig, ax = plt.subplots(figsize=(7, 4))
ax.bar(buckets["delivery_days"], buckets["repeat_rate"], color="#4C6EF5")
ax.set_title("Orders delivered in over 4 days repeat at half the rate")
ax.set_xlabel("Days to delivery")
ax.set_ylabel("Repeat purchase rate")

# Annotate the specific comparison the sentence in the title refers to,
# so the eye lands where the argument is rather than wandering the bars.
ax.annotate(
    "31% -> 16%",
    xy=(4, buckets.loc[buckets["delivery_days"] == 4, "repeat_rate"].iat[0]),
    xytext=(5.2, 0.28),
    arrowprops={"arrowstyle": "->"},
)
plt.tight_layout()

One caution: narrative pressure and honesty pull in opposite directions. A clean story wants a single cause, and real data usually has several partial ones. The fix is to let the complication carry the mess — "three things changed in March; two are noise and one is not" — rather than pretending the mess was never there.

Element	Purpose	Common failure
Situation	Establish shared ground	Starting with something disputed
Complication	Create the need to know	Omitted, so the answer feels moot
Question	Name what is being decided	Vague ("how do we improve things")
Answer	The finding, stated up front	Withheld for suspense
Evidence	Support, in descending strength	Every chart you made, in order

Key Takeaways

• Present backward from the conclusion; analysis order is not story order.
• SCQA — situation, complication, question, answer — earns attention before it spends attention.
• Chart titles should state the finding, not name the axes.
• Dead ends and robustness checks belong in an appendix, not the narrative.
• Do not buy narrative cleanliness with a false single cause.

🧪 Practice

1. Rewrite the titles of five charts from a past report so each states a claim rather than describing the axes.
2. Take an analysis you did and write its SCQA in four sentences before looking at any of your slides.
3. Reorder an existing deck so that a reader who stops after slide two still knows the conclusion and the recommended action.
4. Interview: A stakeholder says your presentation "did not go anywhere." Which element of SCQA was most likely missing? (Hint: think about which element creates the need for an answer.)

Presenting Uncertainty

Every estimate you produce is wrong by some amount. Communicating the estimate without the amount transfers a false certainty to the decision maker, and they have no way to detect the omission. This is the most common way honest analysts mislead people.

The difficulty is that the standard vocabulary of uncertainty does not survive contact with a general audience. "The 95% confidence interval is [1.2%, 4.8%]" is precise and widely misunderstood — most listeners hear "95% probability the answer is in there," which is a Bayesian credible interval, not a frequentist confidence interval. Arguing about this in a business meeting is a losing move. The productive response is to express uncertainty in terms of the decision rather than in terms of the statistic.

Three techniques do most of the work. First, translate the interval into outcomes: "somewhere between 400 and 1,600 extra signups per month" beats any interval notation. Second, state what the uncertainty means for the decision: "even the pessimistic end pays for the build" ends the conversation, while "the interval spans both profit and loss" is itself the finding. Third, show the distribution rather than a point when the shape matters — a fan chart or a dot-and-whisker plot makes variance visible without requiring vocabulary.

  POINT ESTIMATE              INTERVAL                DECISION FRAME

  "+3.0% conversion"          "+3.0%,                 "Worst case we
                               CI [1.2, 4.8]"          gain 1.2%, which
   |                               |----*----|         still clears the
   *                          1.2  3.0    4.8          2% bar we set.
                                                       Ship it."
  Implies certainty          Correct, often         Correct AND
  the data does not          misread as a           actionable, because
  support.                   probability.           it names the bar.

import numpy as np

# A bootstrap turns "how uncertain is this?" into a distribution you can
# quote in whatever units the audience thinks in.
rng = np.random.default_rng(0)

def bootstrap_lift(control, treatment, n=10_000):
    """Distribution of the treatment-control difference, resampled."""
    diffs = np.empty(n)
    for i in range(n):
        c = rng.choice(control, size=len(control), replace=True)
        t = rng.choice(treatment, size=len(treatment), replace=True)
        diffs[i] = t.mean() - c.mean()
    return diffs

diffs = bootstrap_lift(control_conv, treat_conv)
lo, hi = np.percentile(diffs, [2.5, 97.5])

monthly_visitors = 120_000
# Report in the unit the decision is made in, not in percentage points.
print(f"Expected extra conversions per month: "
      f"{diffs.mean() * monthly_visitors:,.0f} "
      f"(plausible range {lo * monthly_visitors:,.0f} to {hi * monthly_visitors:,.0f})")

# The decision-relevant quantity is often a probability of clearing a bar,
# not the interval itself.
bar = 0.02
print(f"P(lift exceeds the {bar:.0%} bar) = {(diffs > bar).mean():.0%}")

Distinguish two kinds of uncertainty when you present, because they have different remedies. Statistical uncertainty shrinks with more data and is what intervals measure. Structural uncertainty comes from assumptions — the sample represents the population, the relationship is stable, the label means what you think — and more data does not shrink it at all. A narrow interval on a biased estimate is confidently wrong, and only the structural caveat warns anyone.

Key Takeaways

• Express uncertainty in the units of the decision, not in interval notation, and say what it implies for the choice at hand.
• "P(effect clears our bar)" is usually more useful than the interval.
• A confidence interval is not a probability statement about the parameter; avoid the fight by not using the phrase with general audiences.
• Statistical uncertainty shrinks with data; structural uncertainty from assumptions does not, and must be stated separately.

🧪 Practice

1. Take a confidence interval from a past analysis and write three versions of it: raw notation, outcome units, and decision framing.
2. Bootstrap a metric from a dataset you have and report the probability that it exceeds a threshold you choose in advance.
3. Write the structural caveats for an analysis you trust. For each, state whether more data would help.
4. Interview: A stakeholder asks you to "just give one number." How do you respond without either refusing or misleading? (Hint: one number plus one condition is still one sentence.)

Executive Summaries and Memos

An executive summary is not a shortened report. It is a different document with a different job: to let a busy reader make or defer a decision without reading the rest. If the reader must open the appendix to know what you recommend, the summary failed regardless of how accurate it is.

The economics are brutal and worth internalizing. A director may receive forty pages of analysis a week and have ninety minutes to process all of it. Your document competes for that time. Anything that delays the recommendation — background, methodology, a tour of the data — spends attention you have not yet earned.

The structure that survives this is roughly fixed: recommendation first, then the two or three reasons, then what it costs and what could go wrong, then the ask. Length is one page. The rest of the analysis still exists; it lives behind the summary and is referenced, not summarized.

  +--------------------------------------------------------------+
  | Recommendation: switch the EU fulfilment contract to Carrier  |
  | B for orders over 2kg, starting Q1.            [1 sentence]   |
  +--------------------------------------------------------------+
  | Why (3 bullets, each a number):                               |
  |  - Delivery over 4 days halves repeat purchase (31% -> 16%)   |
  |  - Carrier A misses the 4-day mark on 38% of >2kg orders      |
  |  - Carrier B's quoted rate is 6% higher, offset at a 1.5pp    |
  |    repeat-rate recovery; we estimate 4-9pp                    |
  +--------------------------------------------------------------+
  | Cost and risk:  EUR 180k/yr extra freight; risk is that the   |
  | repeat-rate link is correlational — see the holdout test in   |
  | section 4 before committing beyond the pilot.                 |
  +--------------------------------------------------------------+
  | Ask: approve a 3-month pilot in DE/NL and one analyst-week    |
  | to instrument it.                                             |
  +--------------------------------------------------------------+
    Everything above fits on one screen. The 20-page analysis is
    attached and is read only by people who disagree.

Writing this well is mostly ruthless deletion, and it helps to check mechanically that the first paragraph actually contains a recommendation.

import re

def summary_smells(text):
    """Cheap lint for the failure modes that make summaries unreadable."""
    first_para = text.strip().split("\n\n")[0]
    problems = []

    # The recommendation must be in the first paragraph, not deferred.
    if not re.search(r"\b(recommend|should|propose)\b", first_para, re.I):
        problems.append("no recommendation in the opening paragraph")

    # Hedges multiply when the author is uncomfortable; they read as noise.
    hedges = len(re.findall(r"\b(may|might|could|possibly|perhaps)\b", text, re.I))
    if hedges > 3:
        problems.append(f"{hedges} hedges — replace with one stated caveat")

    # Methodology in a summary is almost always misplaced.
    if re.search(r"\b(we used|methodology|imputed|hyperparameter)\b", text, re.I):
        problems.append("method detail belongs in the appendix")

    if len(text.split()) > 500:
        problems.append(f"{len(text.split())} words — target under 400")
    return problems

for p in summary_smells(draft_memo):
    print("-", p)

A note on hedging. Analysts hedge because they are honest and the result is genuinely uncertain, but scattered hedges read as a lack of conviction rather than as calibration. The honest and readable form is one firm recommendation plus one explicit condition: "Pilot Carrier B. If the repeat-rate lift in the pilot is under 1.5 points, stop." That sentence is falsifiable, which is what hedging was trying to achieve.

Key Takeaways

• The summary is a standalone decision document, not a compressed report.
• Recommendation in the first sentence; reasons, cost, risk, ask after it.
• One page, and method detail moves to the appendix without exception.
• Replace scattered hedges with one firm recommendation and one explicit stopping condition.

🧪 Practice

1. Compress a report you wrote into a 250-word memo that opens with the recommendation.
2. Take three memos you received and mark where the recommendation appears. Correlate that position with whether you acted on them.
3. Rewrite a hedged conclusion as a recommendation plus a falsifiable condition.
4. Interview: Your recommendation depends on an assumption you are only 60% confident in. How does the memo read? (Hint: the assumption becomes part of the ask, not a footnote.)

Technical Writing for Data Work

Technical writing is the durable half of communication. Slides are consumed once by people in the room; written analysis is read later, by people who cannot ask you questions, often to decide whether an old result still holds. The reader six months from now is your real audience, and that reader has forgotten everything — including, if the reader is you, that you wrote it.

The central property to optimize is reconstructability: could a competent colleague, reading only this document, understand what question was asked, what data answered it, what was assumed, and what would have changed the conclusion? Prose that reads beautifully but omits the label definition fails this test. Prose that is dry but complete passes.

Several conventions pay for themselves repeatedly:

Convention	Why it matters later
Define every metric in code	Prose definitions drift; code cannot
State the data snapshot and date	Numbers change; the reader needs the as-of
Separate finding from inference	Lets a reader disagree with one, not both
Name what would falsify it	Turns a claim into something testable
Version and date the document	Two versions in circulation is the default
Link to the code, not paste it	Pasted code rots silently

The distinction between finding and inference deserves attention. "Users on the new onboarding completed setup 18% more often" is a finding — it is either in the data or it is not. "The new onboarding is clearer" is an inference, and a reader might accept the first and reject the second. Fusing them into one sentence robs the reader of that option and makes the whole document arguable.

<!-- A documented analysis header that makes the work reconstructable. -->
# Delivery time and repeat purchase

**Author:** A. Rossi  **Date:** 2026-03-14  **Version:** 2
**Data:** `orders_snapshot_2026_03_01`, EU only, 2025-01-01 to 2026-02-28
**Code:** `analyses/delivery_repeat/` at commit `a4f19c2`

## Question
Does time-to-delivery affect the probability of a second order within 90 days?

## Definitions
- `delivery_days` = `delivered_at` - `dispatched_at`, in whole days,
  excluding orders never delivered (n = 2,418 dropped, 1.1%).
- `repeat_90d` = any subsequent order by the same `customer_id`
  within 90 days of `delivered_at`.

## Finding
Repeat rate is 31% for deliveries of 1-4 days and 16% above 4 days.

## Inference
Delivery speed plausibly causes part of this gap, but carrier assignment
correlates with order weight and region, both of which independently
predict repeat rate. The controlled estimate is 4-9 points, not 15.

## What would change this conclusion
A randomized carrier assignment showing under 1.5 points of lift.

Write for the skimmer and the excavator at once. The skimmer reads headings and the first sentence of each section, so those must carry the argument. The excavator reads the definitions and the caveats, so those must be complete. Anything that serves neither reader can be cut.

Key Takeaways

• Optimize for reconstructability: question, data, definitions, assumptions, and what would falsify the result.
• Define metrics in code; prose definitions drift from the implementation.
• Always record the data snapshot, the as-of date, and the code commit.
• Keep findings and inferences in separate sentences so a reader can accept one and reject the other.
• Headings and first sentences serve the skimmer; definitions and caveats serve the excavator.

🧪 Practice

1. Take an analysis notebook and write the header block above for it. Note every field you cannot fill in from memory.
2. Go through a past report and split each sentence that fuses a finding with an inference.
3. Hand a written analysis to a colleague and ask them to state its question and its main caveat without asking you anything.
4. Interview: What single piece of metadata, if missing from an old analysis, most often makes it impossible to trust? (Hint: think about why rerunning the same query today gives a different number.)

Handling Pushback on Findings

Sooner or later you will present a result that someone does not want to be true, and they will push back. How you handle this determines whether your work influences anything, and it is a skill that is almost never taught.

The first move is diagnostic: figure out what kind of objection you are facing, because the three kinds need opposite responses.

A technical objection says the analysis is wrong — the sample was biased, the control group was contaminated, a confounder was omitted. This is a gift. The correct response is to engage seriously, and if they are right, say so immediately and publicly. Analysts who correct themselves quickly build more credibility than analysts who are never wrong, because the second group does not exist.

An interpretive objection accepts your numbers and disputes their meaning: yes, the metric fell, but it fell because of a seasonal effect you did not model. This is usually productive and often correct, since the objector frequently has domain context you lack. The response is to name the disagreement precisely and, if possible, propose the check that would resolve it.

A motivated objection is resistance to the implication, expressed as doubt about the method. Its signature is that it moves: you answer the sampling concern and it becomes a concern about the time window, and then about the definition. Nothing you produce will satisfy it, because the objection is not really about the evidence.

  OBJECTION TRIAGE

  "Your sample excludes enterprise accounts."
     -> specific, checkable, would change the number
     -> TECHNICAL: check it. If right, fix it and say so.

  "That drop is just the holiday period."
     -> accepts the number, disputes the cause
     -> INTERPRETIVE: name it, then propose year-over-year as the test.

  "I do not trust this data."   (then: "the window is wrong",
                                 then: "churn is defined oddly")
     -> unfalsifiable and shifting
     -> MOTIVATED: stop defending. Ask what evidence would convince them,
        and write the answer down.

  The question that separates the third from the first two:
       "What result would change your mind?"
  Technical and interpretive objectors can answer it. Motivated ones cannot.

Preparation blunts most of this. Before presenting, run the objections yourself and have the answers ready as artifacts rather than improvisation.

# Pre-compute the sensitivity table your critics will ask for. Arriving with
# it converts an ambush into a discussion.
variants = {
    "headline":            df,
    "excl. enterprise":    df[df["segment"] != "enterprise"],
    "excl. holiday weeks": df[~df["week"].isin(HOLIDAY_WEEKS)],
    "90d window":          rebuild_with_window(df, days=90),
    "strict churn defn":   rebuild_with_label(df, definition="strict"),
}

for name, frame in variants.items():
    est, lo, hi = effect_with_ci(frame)
    # A finding that survives every reasonable variant is far harder to
    # dismiss than one presented only at its most favourable specification.
    print(f"{name:<22} {est:+.2%}  [{lo:+.2%}, {hi:+.2%}]  n={len(frame):,}")

Two rules hold across all three cases. Never defend a number you have not verified since the objection was raised — "let me check and come back" costs nothing and protects everything. And separate the finding from the recommendation: a stakeholder may accept that churn rose while rejecting your proposed remedy, and that is a legitimate position you should not treat as an attack on the analysis.

Key Takeaways

• Triage objections into technical, interpretive, and motivated; each needs a different response.
• Conceding a real error quickly increases credibility more than never erring appears to.
• "What result would change your mind?" distinguishes a genuine objection from a motivated one.
• Pre-compute the sensitivity table for the obvious criticisms and bring it to the meeting.
• Keep the finding and the recommendation separable; rejecting one is not rejecting the other.

🧪 Practice

1. For an analysis you are about to present, list the five objections you expect and compute the variant for each.
2. Recall a time your work was challenged. Classify the objection and evaluate whether your response matched its type.
3. Write the sentence you would use to concede a real methodological error in front of the people who acted on the earlier version.
4. Interview: A senior stakeholder rejects your result without a specific criticism. What do you do? (Hint: your goal is to convert an unfalsifiable objection into a falsifiable one, in writing.)

<a id="172-working-with-stakeholders"></a>

17.2 Working with Stakeholders

Most failed data science projects are technically sound and organizationally homeless. This section covers the work of finding out what is actually wanted, deciding what to build, setting expectations that survive contact with reality, and proving afterwards that the work mattered.

Requirements Gathering

A stakeholder arrives with a request: "build us a churn model." The request is not the requirement. It is a proposed solution to a problem they have already diagnosed, usually correctly in outline and wrongly in detail. Building exactly what was asked for is the most reliable way to deliver something nobody uses.

The reason is that requests arrive pre-compressed. Behind "build us a churn model" sits a decision — who gets a retention call next Tuesday — and a set of constraints nobody mentioned, such as that the call center handles 200 customers a week, that enterprise accounts are managed separately, and that the model must explain itself because the caller needs a reason to open with. None of that is in the request, and all of it determines whether the model is usable.

The technique is to work backward from the action. Ask what decision changes, who makes it, how often, what they do today, and what happens if the answer is wrong in each direction. An analogy: a customer asking a hardware store for a quarter-inch drill bit wants a quarter-inch hole, and possibly wants a shelf to stay on a wall — which might be better served by a different fastener entirely.

  THE LADDER FROM REQUEST TO REQUIREMENT

  request     "Build a churn model."
     |
     v  Who acts on the output, and when?
  actor       "Retention team, Monday morning list."
     |
     v  What do they do today instead?
  baseline    "Sort by account value and call the top 200."
     |
     v  How many can they act on?
  capacity    "200 calls per week."        -> precision@200, not AUC
     |
     v  What is the cost of each error direction?
  costs       "Missed churner: ~EUR 340. Wasted call: ~EUR 12."
     |
     v  What constrains the solution?
  constraints "Needs a reason to give the customer; enterprise excluded;
               data refreshed nightly is fine."
     |
     v
  requirement "Rank the ~40k SMB accounts weekly; surface the top 200 with
               the two strongest contributing factors each; beat the
               value-sorted baseline on saves per call."

Write the requirement down as an executable contract. Ambiguity that survives into week three is expensive, and a specification you can run is a specification nobody can reinterpret.

from dataclasses import dataclass, field

@dataclass
class ProjectSpec:
    decision: str            # the action that changes
    actor: str               # who takes it
    cadence: str             # how often
    baseline: str            # what happens today, quantified
    capacity: int            # how many actions are possible per cycle
    success_metric: str      # measured against the baseline
    constraints: list = field(default_factory=list)
    out_of_scope: list = field(default_factory=list)

spec = ProjectSpec(
    decision="which accounts receive a retention call",
    actor="retention team",
    cadence="weekly, Monday 08:00",
    baseline="top 200 by account value; 11% of called accounts renew early",
    capacity=200,
    # Note the metric matches the capacity. AUC over the whole base would
    # reward ranking quality in a region nobody will ever act on.
    success_metric="saves per 200 calls, vs the value-sorted baseline",
    constraints=[
        "two contributing factors per account, in plain language",
        "SMB only; enterprise is account-managed",
        "nightly refresh is sufficient; no real-time scoring",
    ],
    out_of_scope=["pricing recommendations", "email campaign targeting"],
)

The out_of_scope field does more work than it looks. Most scope creep is not a decision anyone made; it is an assumption nobody contradicted. Writing down what you are not doing, and getting a nod on it, converts a future argument into a present one, which is far cheaper.

Key Takeaways

• A request is a proposed solution; the requirement is the decision behind it, and they are rarely the same.
• Work backward from the action: actor, cadence, current baseline, capacity, cost of each error direction, constraints.
• Let the action's capacity choose the metric — precision at the number of actions you can take, not a metric over rows nobody will touch.
• Record what is out of scope; unwritten exclusions become scope creep.

🧪 Practice

1. Take a request you received as-is and reconstruct the ladder from request to requirement. Note which rungs you had to guess.
2. Write the ProjectSpec for a project you are running and circulate it. Record every correction you get back.
3. For a model you have built, identify the action capacity and check whether your evaluation metric matched it.
4. Interview: A stakeholder asks for "a dashboard with all our KPIs." What do you ask first? (Hint: a dashboard that informs no decision still costs maintenance forever.)

Scoping and Prioritization

You will always have more requests than capacity. Prioritizing by who asked most insistently is the default in the absence of a system, and it reliably produces a quarter of urgent work with nothing to show. A scoping method is mostly a device for having the hard conversation once, in public, with numbers.

Two quantities drive the decision: what the work is worth if it succeeds, and how likely it is to succeed at all. The second is where data projects differ from engineering projects. A feature request is nearly certain to be buildable; a model is not, because the signal may not exist in the data. This is why feasibility deserves its own explicit estimate rather than being folded into effort.

A third factor is often decisive and usually forgotten: whether the organization can act on the result. A perfectly accurate forecast of regional demand is worth nothing if procurement contracts are locked annually. Ask whether the decision is actually free to move before you optimize it.

Factor	Question	Scale
Value	What changes in money or risk if this works?	Estimated
Confidence	Does the signal plausibly exist in our data?	0.1 - 0.9
Effort	Person-weeks to a usable first version	Weeks
Actionability	Can the decision it informs actually change?	Yes / No
Decay	Does the value drop if delivered a quarter late?	Yes / No

def score(project):
    """Expected value per person-week, with a hard actionability gate."""
    # No amount of value or confidence rescues a decision nobody can change.
    if not project["actionable"]:
        return 0.0
    value = project["annual_value"] * project["confidence"]
    # Time-sensitive work is penalized for the delay it will actually face,
    # not for its duration — a 2-week project queued behind 8 weeks of work
    # is a 10-week project from the stakeholder's point of view.
    if project["decays"]:
        value *= 0.5 ** (project["queue_weeks"] / 13)
    return value / project["effort_weeks"]

backlog = [
    {"name": "churn ranking",     "annual_value": 900_000, "confidence": 0.7,
     "effort_weeks": 6,  "actionable": True,  "decays": False, "queue_weeks": 0},
    {"name": "demand forecast",   "annual_value": 2_000_000, "confidence": 0.6,
     "effort_weeks": 12, "actionable": False, "decays": False, "queue_weeks": 0},
    {"name": "pricing experiment","annual_value": 400_000, "confidence": 0.9,
     "effort_weeks": 3,  "actionable": True,  "decays": True,  "queue_weeks": 8},
]

for p in sorted(backlog, key=score, reverse=True):
    print(f"{p['name']:<20} {score(p):>12,.0f} per person-week")
# demand forecast scores zero despite the largest value: procurement is
# contractually locked, so the forecast cannot change any decision this year.

Scoping the chosen project is a separate discipline. The reliable pattern is to define the smallest version that produces a real decision — often a heuristic or a single query, not a model — ship it, and let its performance justify the next increment. This is not timidity. A rules-based baseline delivered in week one establishes the bar every later version must clear, and surprisingly often it is good enough that the model is never needed.

Key Takeaways

• Prioritize on value times feasibility per unit of effort, and gate on whether the decision can actually change.
• Feasibility is a real unknown for data projects in a way it is not for feature work; estimate it explicitly.
• Penalize time-decaying work by its expected queue time, not its duration.
• Scope to the smallest version that produces a decision; a heuristic shipped early sets the bar the model must beat.

🧪 Practice

1. Score your current backlog with the table above and compare the ordering to what you are actually working on.
2. Identify one project in flight whose decision is not free to change. Write the sentence that would pause it.
3. For your largest current project, define the one-week version that would still produce a decision.
4. Interview: How do you decline a request from a senior stakeholder? (Hint: you are not declining, you are showing what it displaces.)

Managing Expectations About Models

Non-specialists have absorbed a picture of machine learning from journalism and product marketing in which models are approximately correct, improve continuously, and explain themselves. Every part of this is wrong, and the gap between the picture and reality is where projects get labeled failures despite working as designed.

Three expectations need resetting, and the time to do it is before the work starts, when it sounds like professionalism, rather than after, when it sounds like an excuse.

The first is accuracy. Stakeholders hear "85% accurate" as "wrong roughly one time in seven, spread evenly." In reality errors cluster, and on imbalanced problems accuracy is close to meaningless — a fraud model that predicts "no fraud" for everyone is 99.2% accurate and worthless. Replacing accuracy talk with error-cost talk fixes this permanently: "of the 200 accounts we flag, about 70 would have churned; the other 130 calls are the cost of finding them."

The second is stability. A model is a fit to the world as it was during training. When the world moves, performance degrades silently, and nothing in the system announces it. Stakeholders expect software behavior — works today, works next year — and need to be told that a model is closer to a perishable good with a retraining cadence and a monitoring cost.

The third is explanation. A model gives reasons in its own terms — contributing features, not motives. "Support tickets in the last 30 days contributed most to this score" is not the same claim as "this customer is leaving because of support," and conflating them leads to interventions aimed at a correlate.

  WHAT THEY HEAR              WHAT IS TRUE              HOW TO SAY IT

  "85% accurate"        ->  errors cluster in the   ->  "Of 200 flagged,
                            hard cases; accuracy         ~70 are real. The
                            hides the imbalance          rest is the cost
                                                         of finding them."

  "It learns over       ->  it is static until      ->  "It degrades from
   time"                     retrained; drift is         day one. Budget a
                             silent                      retrain each
                                                         quarter."

  "It tells you why"    ->  it reports feature      ->  "It says which
                            contributions, not           signals moved the
                            causes                       score. Testing the
                                                         cause is separate
                                                         work."

Make the tradeoff visible rather than asserted. A threshold table converts an abstract argument about accuracy into a concrete choice the stakeholder is qualified to make.

import numpy as np

def threshold_table(y_true, scores, cost_fp, cost_fn):
    """Show the operating points so the stakeholder picks one, not you."""
    rows = []
    for t in np.arange(0.05, 0.96, 0.05):
        pred = scores >= t
        fp = int(((pred == 1) & (y_true == 0)).sum())
        fn = int(((pred == 0) & (y_true == 1)).sum())
        tp = int(((pred == 1) & (y_true == 1)).sum())
        rows.append({
            "threshold": round(t, 2),
            "flagged": int(pred.sum()),
            "caught": tp,
            "false_alarms": fp,
            "missed": fn,
            # The only column most stakeholders should read.
            "net_cost": fp * cost_fp + fn * cost_fn,
        })
    return rows

# Presenting this beats presenting a single model "accuracy": the choice of
# operating point is a business decision, and this table hands it over.
for row in threshold_table(y_val, val_scores, cost_fp=12, cost_fn=340):
    print(row)

One more expectation is worth setting explicitly: that the first version will be worse than the eventual one, and that shipping it anyway is how you learn what the eventual one needs. Stakeholders who expect a finished artifact treat an honest v1 as a failure. Stakeholders who were told to expect a v1 treat it as progress.

Key Takeaways

• Reset expectations before the work starts; afterwards the same words sound like excuses.
• Replace accuracy claims with error counts and their costs at a named operating point.
• A model is perishable: state the degradation and the retraining cadence as part of the deliverable.
• Feature attribution is not causal explanation; say so before someone acts on a correlate.
• Hand the threshold decision to the stakeholder with a cost table.

🧪 Practice

1. Write the threshold table for a model you have trained and present it without mentioning AUC or accuracy.
2. Draft the three sentences you would use at project kickoff to set accuracy, stability, and explainability expectations.
3. Compute the accuracy of a trivial majority-class predictor on an imbalanced dataset you have, and put it next to your model's.
4. Interview: A stakeholder asks for "at least 95% accuracy" before approving the project. How do you respond? (Hint: find out what decision the 95% was standing in for.)

Cross-Functional Collaboration

Data science sits between functions that each own something it needs. Data engineering owns the pipelines your features come from. Software engineering owns the system your model must run inside. Product owns the surface where the output reaches a user. Legal and compliance own whether you are allowed to use the data at all. You own none of these, and every one of them can block delivery.

This is a structural fact, not a complaint, and it shapes how the work should be sequenced. The instinct is to build the model first and negotiate the integrations afterward, because the model is the interesting part. The result is a finished model that cannot be deployed because the feature it depends on is computed in a nightly batch and the product surface needs it in 50 milliseconds.

The remedy is to agree the interfaces before building anything behind them. For each neighboring function, establish what you receive and what you deliver, in writing, early enough that a "no" is cheap.

Partner	What you need from them	What they need from you
Data engineering	Stable schemas, freshness SLAs,	Which tables matter, tolerable
	notice before breaking changes	lag, expected volume
Backend eng	Where the model runs, latency and	Input/output contract, memory and
	memory budget, rollback path	latency profile, failure behaviour
Product	The decision and the surface,	Honest capability and limits,
	success criteria	the operating point tradeoff
Legal/compliance	Permitted uses, retention limits	Data inventory, purpose, subjects
Domain experts	What the fields actually mean	Findings that contradict intuition

The single most common integration failure is training/serving skew, and it arises directly from the organizational split: features are written twice, once in the analyst's SQL and once in the engineer's service code, and the two drift. Agreeing on a shared feature contract early is worth more than any modeling improvement.

# A feature contract both sides import beats two implementations that agree
# on the day they are written and diverge quietly afterwards.

FEATURE_CONTRACT = {
    "tickets_30d": {
        "dtype": "int64",
        "source": "support.tickets",
        "window_days": 30,
        "as_of": "score_time",       # NOT "today" — this is where leakage enters
        "null_policy": "0",
        "max_staleness_hours": 24,
    },
    "days_since_login": {
        "dtype": "int64",
        "source": "events.sessions",
        "as_of": "score_time",
        "null_policy": "999",        # sentinel, agreed with engineering
        "max_staleness_hours": 6,
    },
}

def validate_batch(df, contract=FEATURE_CONTRACT):
    """Run in training AND in the serving path, so skew surfaces as an error."""
    for name, rules in contract.items():
        if name not in df.columns:
            raise KeyError(f"missing contracted feature: {name}")
        if str(df[name].dtype) != rules["dtype"]:
            raise TypeError(
                f"{name}: expected {rules['dtype']}, got {df[name].dtype}"
            )
        if df[name].isna().any():
            raise ValueError(f"{name}: nulls must be filled per null_policy")
    return True

A note on working with domain experts specifically. They are the cheapest source of the constraint that invalidates your model, and they usually volunteer it in the first ten minutes if asked directly: "what would make you distrust this number?" A warehouse manager who tells you that the scanner logs get re-entered manually on Sundays has just saved you a month.

Key Takeaways

• Agree the interfaces with each neighboring function before building the thing behind them; a late "no" is the expensive one.
• Latency, freshness, and where the model runs are design constraints, not deployment details.
• Training/serving skew is an organizational problem: one shared feature contract, validated on both paths.
• Ask domain experts what would make them distrust the number, early.

🧪 Practice

1. For a model you have deployed or plan to, write the contract table above with a named person for each partner row.
2. Take one feature and compare its training implementation with its serving implementation line by line.
3. Interview a domain expert about a dataset you use and record every data quirk they mention that is not documented anywhere.
4. Interview: A model performs well offline and poorly in production, with no data drift. What do you check first? (Hint: compare the feature values themselves, computed by each path, for the same entity and timestamp.)

Measuring Project Impact

A project that cannot demonstrate its effect will be treated as having had none, and eventually defunded. This is not unfair. The burden of proof is on the analyst, because the counterfactual — what would have happened anyway — is invisible by default.

The trap is measuring model quality instead of business impact. AUC improved, so the project succeeded: this reasoning fails at every step between the score and the outcome. The model might rank well and still change nothing if nobody acts on the output, if the intervention does not work, or if the customers it saves would have stayed anyway. Each of those is a real failure mode and none of them shows up in an offline metric.

The only clean measurement is a holdout. Withhold the intervention from a random subset, keep it withheld, and compare. This is expensive in the sense that you knowingly forgo value on the holdout group, and it is almost always worth it, because without it the effect size is a matter of opinion.

  THE CHAIN FROM MODEL TO MONEY — every link can break

  model ranks well      AUC 0.81                [offline metric]
       |
       v  is it used?
  adoption              retention team uses     [instrumentation:
                        the list 4 weeks in 6    who opened the list?]
       |
       v  does the action work?
  intervention effect   called accounts renew   [holdout: called vs
                        at +6pp vs holdout       not-called, randomized]
       |
       v  does it net out?
  financial impact      6pp * 200 calls/wk *    [finance agrees the
                        EUR 340 - call cost      value per save]
       |
       v  would it have happened anyway?
  incrementality        compare to the old      [the baseline is the
                        value-sorted baseline    old process, not zero]

Note the last link. The comparison that matters is against what the organization did before, not against doing nothing. A model that catches 44% of churners replaces a heuristic that caught 20%, so the project's impact is the difference, not the total. Claiming the total is the most common way data teams lose credibility with finance.

def project_impact(holdout, treated, value_per_save, cost_per_contact,
                   baseline_rate):
    """Incremental value over the previous process, with an interval."""
    treat_rate = treated["retained"].mean()
    ctrl_rate = holdout["retained"].mean()
    lift = treat_rate - ctrl_rate

    # Standard error of a difference in proportions; the interval is the
    # point of the exercise, because a positive point estimate with an
    # interval spanning zero has not demonstrated anything.
    se = ((treat_rate * (1 - treat_rate) / len(treated))
          + (ctrl_rate * (1 - ctrl_rate) / len(holdout))) ** 0.5

    n = len(treated)
    gross = lift * n * value_per_save
    net = gross - n * cost_per_contact
    # Credit only the improvement over the heuristic that was already running.
    incremental = (lift - baseline_rate) * n * value_per_save - n * cost_per_contact
    return {
        "lift_pp": round(lift * 100, 2),
        "lift_ci_pp": (round((lift - 1.96 * se) * 100, 2),
                       round((lift + 1.96 * se) * 100, 2)),
        "net_value": round(net),
        "incremental_vs_baseline": round(incremental),
    }

Agree the measurement plan with the stakeholder and with finance before launching, including who owns the value-per-save figure. A number you compute alone will be disputed; a number finance supplied will be defended by finance.

Key Takeaways

• Model quality is not impact; instrument adoption, intervention effect, and net value separately, because each link can break independently.
• A randomized holdout is the only clean estimate of incremental effect, and forgoing some value to keep it is usually correct.
• Measure against the previous process, not against doing nothing.
• Agree the value assumptions with finance before launch, not after.

🧪 Practice

1. Draw the model-to-money chain for a project you shipped and mark which links you actually measured.
2. Design a holdout for a currently running intervention, including its size and how long it must stay withheld.
3. Recompute a past impact claim as an increment over the pre-existing process rather than over zero.
4. Interview: Your stakeholder refuses a holdout because "we know it works." What do you propose instead? (Hint: staged rollouts and switchbacks both create comparison groups without permanently withholding from anyone.)

<a id="173-reproducibility-and-rigor"></a>

17.3 Reproducibility and Rigor

An analysis nobody can rerun is an opinion with charts. This section covers the practices that make a result verifiable — by a colleague, by a regulator, or by you after you have forgotten everything — and the reasoning errors that most often survive review.

Reproducible Research Practices

Reproducibility means that someone with your code and your data gets your number. It sounds like a low bar. It is routinely missed, and the usual discovery moment is six months later when a stakeholder asks you to refresh a chart and the new figure is different by four points, with no way to tell which version is wrong.

The failure has four common sources, and they compound. The code changed and was not versioned. The data changed underneath a query with no as-of filter. The environment changed — a pandas upgrade altered a default, a library changed a rounding rule. And the execution order differed, because the original was a notebook whose cells were run out of order and whose in-memory state no longer exists anywhere.

The notebook problem deserves emphasis because it is the dominant one in practice. A notebook records the cells, not the order they were executed, so a notebook that produces a correct figure on your screen may produce an error or a different figure when run top to bottom. Hidden state is invisible until it is fatal.

  FOUR LAYERS, EACH INDEPENDENTLY ABLE TO BREAK A RESULT

  +-----------------------------------------------------+
  | EXECUTION  notebook out-of-order? hidden state?      |
  |            fix: restart-and-run-all in CI            |
  +-----------------------------------------------------+
  | ENVIRONMENT  library versions, BLAS, locale, TZ      |
  |              fix: lockfile + container               |
  +-----------------------------------------------------+
  | DATA  which snapshot? late-arriving rows? mutable    |
  |       tables? fix: as-of filter + row count + hash   |
  +-----------------------------------------------------+
  | CODE  which commit? uncommitted edits?               |
  |       fix: version control, fail on a dirty tree     |
  +-----------------------------------------------------+

  A result is reproducible only if ALL FOUR are pinned.
  Pinning three of them produces a reproducible wrong answer.

The practical minimum is a run manifest: a small record emitted by every analysis that captures what would be needed to recreate it.

import hashlib, json, platform, subprocess
from datetime import datetime, timezone

def run_manifest(df, params, path="manifest.json"):
    """Emit everything needed to tell two runs apart."""
    git = subprocess.run(["git", "rev-parse", "HEAD"],
                         capture_output=True, text=True).stdout.strip()
    dirty = subprocess.run(["git", "status", "--porcelain"],
                           capture_output=True, text=True).stdout.strip()

    # A content hash detects the case that ruins refreshes: the same query,
    # the same date range, different rows, because the table was mutable.
    digest = hashlib.sha256(
        df.to_csv(index=False).encode()
    ).hexdigest()[:16]

    manifest = {
        "run_at": datetime.now(timezone.utc).isoformat(),
        "git_commit": git,
        "git_dirty": bool(dirty),        # a dirty tree means the commit lies
        "python": platform.python_version(),
        "rows": len(df),
        "data_sha256_16": digest,
        "params": params,
    }
    with open(path, "w") as f:
        json.dump(manifest, f, indent=2)
    return manifest

Beyond the manifest, three habits carry most of the weight. Pin dependencies in a lockfile and, for anything that matters, run inside a container. Keep raw data immutable and treat every derived table as a rebuildable artifact, so the pipeline can always be replayed from the source. And enforce restart-and-run-all in continuous integration, which converts the notebook hidden-state problem from a latent bug into a failing build.

There is a spectrum here, and it is fine to sit at different points for different work. An exploratory notebook answering a one-hour question does not need a container. Anything that informs a decision someone will be asked to defend does.

Key Takeaways

• Four layers must be pinned — code, data, environment, execution order — and pinning only some yields a reproducibly wrong answer.
• Notebooks record cells, not execution order; enforce restart-and-run-all in CI.
• Emit a run manifest with the commit, dirty flag, row count, data hash, and parameters.
• Keep raw data immutable and treat derived tables as rebuildable.
• Match rigor to stakes; exploration and decision support need different levels.

🧪 Practice

1. Take your most recent analysis, restart the kernel, and run it top to bottom. Record what breaks.
2. Add a run manifest to one pipeline and diff two runs a week apart.
3. Rerun a three-month-old analysis and reconcile any difference to one of the four layers.
4. Interview: An analysis from last quarter now produces a different number with unchanged code. Name three causes and how you would distinguish them. (Hint: one of them leaves a trace in row counts.)

Seed Control and Determinism

Randomness appears throughout a data workflow: train/test splits, weight initialization, bootstrap resampling, stochastic gradient descent, subsampling in tree ensembles. Without control, two runs of identical code produce different numbers, and you lose the ability to tell a real improvement from noise — which is the entire basis of iterative modeling.

Setting a seed makes the pseudo-random sequence repeatable. The subtlety is that there is no single seed. Python's random, NumPy's global state, each library's internal generator, and the GPU all have their own, and setting one does not set the others. Worse, the global-state pattern is fragile: any library that draws from the global NumPy generator changes what your next call returns, so a seemingly unrelated import can shift your results.

import os, random
import numpy as np

def set_all_seeds(seed: int = 42):
    """Cover the generators that actually affect a typical run."""
    os.environ["PYTHONHASHSEED"] = str(seed)   # set-iteration order
    random.seed(seed)
    np.random.seed(seed)                       # legacy global state
    try:
        import torch
        torch.manual_seed(seed)
        torch.cuda.manual_seed_all(seed)
    except ImportError:
        pass

# BETTER than the global seed: pass an explicit generator so that a call
# elsewhere in the process cannot shift your sequence.
rng = np.random.default_rng(42)
sample = rng.choice(population, size=1000, replace=False)

# Reproducible splits should not depend on row order at all. Hashing a
# stable key assigns each entity to a fold deterministically, so the split
# survives re-sorting, re-ingestion, and newly arrived rows.
import hashlib

def stable_fold(entity_id: str, n_folds: int = 5) -> int:
    h = hashlib.md5(f"{entity_id}:v1".encode()).hexdigest()
    return int(h, 16) % n_folds       # bump ":v1" to deliberately reshuffle

Determinism has limits worth knowing. GPU kernels and multi-threaded reductions can sum floating-point values in a nondeterministic order, and floating-point addition is not associative, so results differ in the last digits even with every seed set. Frameworks offer deterministic modes that fix this at a real speed cost. Take the cost when you are debugging or certifying a result; skip it when you are exploring.

A further distinction matters for honesty. Seeds make results repeatable; they do not make them robust. A model that scores 0.84 on seed 42 and 0.79 on seed 7 is not a 0.84 model. If a conclusion depends on the seed, the seed is telling you the effect is within noise.

# Report across seeds, not from the luckiest one.
scores = [fit_and_score(seed=s) for s in range(10)]
print(f"AUC {np.mean(scores):.3f} +/- {np.std(scores):.3f} "
      f"(min {min(scores):.3f}, max {max(scores):.3f})")
# A 0.02 improvement over a baseline whose seed spread is 0.05 is not
# an improvement; it is a selection from the spread.

Key Takeaways

• There is no single seed: Python, NumPy, each framework, and the GPU have separate generators.
• Prefer explicit generator objects to global seeding; global state is perturbed by unrelated code.
• Hash a stable key for splits so folds survive reordering and new rows.
• Full determinism on GPU and multi-threaded reductions costs speed; pay it when certifying, not when exploring.
• Seeds give repeatability, not robustness — report the spread across seeds and treat seed-dependent conclusions as noise.

🧪 Practice

1. Run a model ten times with different seeds and report the spread. Compare it to the improvement you last claimed.
2. Replace a train_test_split(random_state=...) with a hash-based split and verify the folds are stable after shuffling the input rows.
3. Enable deterministic mode in a deep learning framework and measure the runtime cost.
4. Interview: A colleague reports a 0.3% accuracy gain from a new feature. What do you ask? (Hint: one question is about seeds and one is about how many variants were tried.)

Documenting Assumptions

Every analysis rests on assumptions, and most of them are never stated because they were never consciously made. You filtered out refunded orders because that is obviously right. You treated a null country as unknown rather than as a specific country. You assumed the tracking pixel fires equally across browsers. Each is a judgment, each could be wrong, and an undocumented one is unfalsifiable — a reader cannot disagree with a decision they cannot see.

The useful frame is that an assumption is a claim about the world that your result inherits. If the claim is false, the result is wrong in a specific, predictable direction. That direction is the most valuable thing to write down, because it tells a reader how much the assumption matters before anyone spends time testing it.

Assumptions come in layers, and analysts tend to document only the last one.

Layer	Example	Usually documented?
Data capture	The pixel fires on all browsers	Rarely
Data meaning	status='active' means paying	Rarely
Filtering	Refunded orders excluded	Sometimes
Population	EU customers represent all customers	Sometimes
Statistical	Errors independent; variance equal	Often
Causal	No unmeasured confounder	Sometimes
Stability	Relationship holds next quarter	Rarely

from dataclasses import dataclass

@dataclass
class Assumption:
    claim: str
    rationale: str
    if_false: str        # the direction of the error, not just "results change"
    testable: bool
    tested: str = "no"

ASSUMPTIONS = [
    Assumption(
        claim="Refunded orders are excluded from repeat-purchase counts.",
        rationale="A refunded order is not a purchase the customer kept.",
        if_false="Repeat rate is overstated for slow-delivery orders, which "
                 "refund more often — this would inflate the effect we report.",
        testable=True,
        tested="yes — effect is 4.1pp including refunds vs 4.6pp excluding",
    ),
    Assumption(
        claim="Carrier assignment is unrelated to customer quality.",
        rationale="Assignment is by weight and postcode, not customer.",
        if_false="The delivery-time effect absorbs a customer-mix difference "
                 "and is biased upward by an unknown amount.",
        testable=True,
        tested="partially — mix is balanced on tenure and AOV, not on channel",
    ),
    Assumption(
        claim="The 2025 relationship holds into 2026.",
        rationale="No known change to fulfilment or product mix.",
        if_false="The forecast degrades from launch with no alarm.",
        testable=False,           # only time tests this one
    ),
]

# Assumptions that are testable and untested are the project's real risk
# register; sorting by that surfaces the next thing worth an hour.
for a in ASSUMPTIONS:
    if a.testable and a.tested == "no":
        print("UNTESTED:", a.claim, "->", a.if_false)

Two habits make this stick. Write the assumption at the moment you make it, inline in the code, because you will not recall it later — the moment you type a WHERE clause is the moment to say why. And distinguish assumptions you tested from those you merely believe; a list that mixes them gives false comfort, since readers will assume anything written down was checked.

Key Takeaways

• An undocumented assumption is unfalsifiable; readers cannot disagree with a decision they cannot see.
• Record the direction of the error if the assumption is false, not just that "results would change."
• Capture assumptions at data capture and meaning, not only the statistical ones.
• Mark which assumptions were tested; untested-but-testable is your risk register.
• Write it at the moment of the filter, not at write-up time.

🧪 Practice

1. Take one analysis query and document every WHERE clause as an assumption with its failure direction.
2. Build the assumption table for a past project and classify each entry as tested, testable, or untestable.
3. Pick the untested assumption with the largest failure direction and test it.
4. Interview: Which assumption in a typical A/B test is most often violated in practice and least often stated? (Hint: think about what a single user does across devices and sessions.)

Peer Review of Analyses

Code review is standard practice; analysis review is not, and the asymmetry is strange, because an analysis error is usually more expensive. Broken code raises an exception. A broken analysis produces a plausible number that someone acts on.

Reviewing an analysis is a different activity from reviewing code, and treating it as the same is why analysis review often degenerates into style comments. The question is not "is this code correct?" but "does this code answer the question that was asked, and would I believe the answer?" Those require reading the framing and the data, not only the diff.

An effective review proceeds in a fixed order, from cheapest and most likely to catch something to most expensive.

  ANALYSIS REVIEW ORDER — stop early if something fails

  1. QUESTION      Does the analysis answer the question in the brief?
                   (A correct answer to a different question is the most
                    common and most invisible failure.)
       |
  2. DATA          Right source, right window, right population?
                   Row counts plausible? Who was silently dropped?
       |
  3. DEFINITIONS   Does each metric compute what its name claims?
                   Read the SQL, not the variable name.
       |
  4. LEAKAGE       Could any feature have been unknown at decision time?
                   Was the split temporal where it had to be?
       |
  5. INFERENCE     Does the conclusion follow, or does it overreach from
                   correlation to cause?
       |
  6. ROBUSTNESS    Does it survive a reasonable alternative specification?
       |
  7. CODE          Only now: correctness, readability, reuse.

The reviewer's most powerful tool is independent recomputation of one number. Reading code for errors is unreliable; recomputing the headline figure a different way catches whole classes of mistake at once.

# Reviewer's independent check: compute the headline a different way and
# reconcile. Agreement is weak evidence; disagreement is strong evidence.
headline = 0.046          # +4.6pp claimed in the memo

# Deliberately avoid the author's helper functions and joins.
fast = orders.query("delivery_days <= 4")["repeat_90d"].mean()
slow = orders.query("delivery_days > 4")["repeat_90d"].mean()
independent = fast - slow

print(f"claimed {headline:+.3f} | independent {independent:+.3f} "
      f"| delta {independent - headline:+.3f}")
# A delta beyond rounding means the populations differ. Find out how before
# discussing anything else in the review.

Set norms that keep review from being read as an attack. Review the analysis, not the analyst; state whether each comment is blocking or optional; and require the author to state their own biggest doubt before the review starts, which converts the reviewer from adversary to collaborator and usually points straight at the weakest part.

Key Takeaways

• Analysis review asks whether the question was answered, not only whether the code runs.
• Review in order: question, data, definitions, leakage, inference, robustness, then code.
• Independently recompute one headline number a different way; disagreement is far more informative than agreement.
• Have the author state their biggest doubt first; it is usually the right place to start.
• Label comments as blocking or optional so authors can triage.

🧪 Practice

1. Review a colleague's analysis using the seven-step order and note at which step you found the first real issue.
2. Independently recompute the headline number of a report you received.
3. Write the "my biggest doubt" paragraph for your own current analysis before anyone asks for it.
4. Interview: What is the first thing you check when reviewing someone else's model? (Hint: it is not in the model code.)

Common Analytical Fallacies

Certain reasoning errors recur so reliably that they have names, and knowing the names is genuinely useful: it converts a vague unease into a specific check. These are not exotic statistical subtleties. They appear in ordinary dashboards and weekly reports.

Survivorship bias draws conclusions from the units that remained, ignoring those that disappeared. Customers who have been with you three years are happier than new ones — but the unhappy ones left, so you are measuring attrition, not satisfaction growth.

Simpson's paradox is an aggregate trend that reverses within every subgroup, caused by an imbalance in group sizes. It is not a curiosity; it silently flips conclusions in any analysis that pools heterogeneous populations.

Base rate neglect ignores how rare an event is when interpreting a test. A 99%-accurate detector for a condition affecting one in ten thousand will be wrong far more often than it is right, and intuition resists this strongly.

Regression to the mean mistakes the natural return of an extreme value toward average for the effect of an intervention. Coach the worst-performing salespeople and they improve, with or without the coaching.

Multiple comparisons treats the best of many tested hypotheses as if it were the only one tested. Twenty tests at the 5% level produce one "significant" result by construction.

import numpy as np
import pandas as pd

# SIMPSON'S PARADOX: the aggregate and every subgroup disagree.
data = pd.DataFrame({
    "channel": ["web"] * 2 + ["mobile"] * 2,
    "variant": ["A", "B", "A", "B"],
    "conversions": [190, 45, 20, 220],
    "visitors":    [1000, 200, 200, 2000],
})
data["rate"] = data["conversions"] / data["visitors"]

# Within each channel, B beats A:
#   web:    A 19.0%  vs  B 22.5%
#   mobile: A 10.0%  vs  B 11.0%
print(data.pivot(index="channel", columns="variant", values="rate"))

agg = data.groupby("variant")[["conversions", "visitors"]].sum()
agg["rate"] = agg["conversions"] / agg["visitors"]
print(agg["rate"])
# Aggregate: A 17.5%, B 12.0% -> A "wins", because B's traffic is mostly
# mobile, which converts worse for BOTH variants. The mix, not the variant,
# drives the aggregate. Always check the subgroup before believing the total.

# BASE RATE NEGLECT, made concrete.
prevalence, sensitivity, specificity = 1 / 10_000, 0.99, 0.99
n = 1_000_000
true_pos = n * prevalence * sensitivity            # 99
false_pos = n * (1 - prevalence) * (1 - specificity)  # 9,999
print(f"P(condition | positive) = {true_pos / (true_pos + false_pos):.1%}")
# ~1%. A "99% accurate" test on a rare event is mostly false alarms.

Fallacy	Signature	The check
Survivorship	Cohort improves with tenure	Analyze by joining cohort
Simpson's paradox	Aggregate contradicts subgroups	Segment before concluding
Base rate neglect	Rare event, "accurate" test	Compute the positive predictive value
Regression to mean	Extremes improve after treatment	Add a control from the same tail
Multiple comparisons	One of many tests is significant	Pre-register; adjust for the count
Confirmation bias	Analysis stopped when it agreed	Predefine the stopping rule

The practical defense against all of them is the same: write down what you expect before you look, and specify what result would falsify your hypothesis. Most fallacies enter through analytical freedom exercised after seeing the data.

Key Takeaways

• Survivorship bias, Simpson's paradox, base rate neglect, regression to the mean, and multiple comparisons account for most real-world analysis errors.
• An aggregate that contradicts every subgroup is a mix effect; segment before concluding.
• For rare events compute the positive predictive value, not accuracy.
• Improvements measured only on the extremes need a control from the same extreme.
• Pre-specifying the hypothesis and the stopping rule defends against most of these at once.

🧪 Practice

1. Construct a dataset where Simpson's paradox appears, then find a real case of it in a metric you report.
2. Compute the positive predictive value of a classifier you have built at its deployed threshold and prevalence.
3. Audit a past analysis for the number of hypotheses actually tested before the reported one.
4. Interview: A training program's worst-performing participants improved the most. What are two explanations, and how would you distinguish them? (Hint: one requires no treatment effect at all.)

Sanity Checks and Red Teaming Your Own Work

The last defense before a number leaves your machine is you, deliberately trying to break it. This is unnatural — you want the result to be right, especially after three weeks — and therefore must be made procedural rather than left to conscientiousness.

Sanity checks are cheap assertions about what must be true regardless of the finding. Totals reconcile to the source system. Rates lie between zero and one. Segment counts sum to the whole. Time series have no missing days. Each takes seconds to write and catches errors that are otherwise invisible, because a wrong number looks exactly like a right one.

def sanity_checks(df, source_totals):
    """Assertions that must hold whatever the finding turns out to be."""
    checks = []

    # 1. Reconcile to the system of record. Silent join fan-out is the most
    #    common cause of a number that is exactly 2.3x too large.
    ours = df["order_id"].nunique()
    checks.append(("order count vs source",
                   ours == source_totals["orders"],
                   f"{ours:,} vs {source_totals['orders']:,}"))

    # 2. Rows must not multiply through a join.
    checks.append(("no duplicate order rows",
                   len(df) == df["order_id"].nunique(),
                   f"{len(df):,} rows, {df['order_id'].nunique():,} ids"))

    # 3. Ranges must be possible.
    checks.append(("repeat rate in [0,1]",
                   df["repeat_90d"].between(0, 1).all(), ""))
    checks.append(("no negative delivery times",
                   (df["delivery_days"] >= 0).all(), ""))

    # 4. Coverage: a silent gap becomes a silent bias.
    days = df["order_date"].dt.date.nunique()
    expected = (df["order_date"].max() - df["order_date"].min()).days + 1
    checks.append(("no missing days", days == expected,
                   f"{days} of {expected}"))

    for name, passed, detail in checks:
        print(f"[{'PASS' if passed else 'FAIL'}] {name} {detail}")
    return all(c[1] for c in checks)

Red teaming goes further: you argue against your own conclusion as if paid to. The productive question is not "is this right?" — you will answer yes — but "suppose this is wrong; what is the most likely reason?" That reframing gets past the motivated reasoning, because it grants the premise and asks only for the mechanism.

  RED TEAM PROMPTS — work through all six before presenting

  1. If this result is wrong, what is the most likely cause?
  2. What would I expect to see if the effect were entirely a data artifact?
     Do I see it?
  3. Who is missing from this dataset, and would including them flip it?
  4. Which single row, segment, or week is doing the most work? Drop it.
  5. Does the effect exist in a period where it should NOT (a placebo test)?
  6. What is the most favourable decision I made, and what happens
     if I reverse it?

  A finding that survives all six is one you can defend in public.
  A finding that fails #5 is usually a pipeline bug.

The placebo test is worth particular attention because it is so diagnostic. If your intervention shows an effect in a period before it launched, or on a population it never touched, the effect is not the intervention. Running this takes minutes and catches serious errors that every other check passes.

# Placebo test: run the identical pipeline on a pre-launch window, where
# the true effect is known to be zero.
placebo = analyze(orders[orders["order_date"] < LAUNCH_DATE])
print(f"placebo effect: {placebo['lift']:+.3f} (should be ~0)")
# A large placebo effect means the pipeline, not the intervention, is
# producing the result. Stop and debug; do not present.

# Leave-one-out: is the finding carried by one segment?
for seg in orders["region"].unique():
    without = analyze(orders[orders["region"] != seg])
    print(f"without {seg:<8} {without['lift']:+.3f}")

Key Takeaways

• Make self-checking procedural; motivation to find errors drops as investment rises.
• Cheap assertions — reconciliation, uniqueness, ranges, coverage — catch errors that are invisible in the output.
• Ask "suppose this is wrong, why?" rather than "is this right?"
• A placebo test on a pre-launch period or untouched population is the single highest-yield check.
• Leave-one-out by segment reveals findings carried by a single group.

🧪 Practice

1. Add a sanity check block to an existing pipeline and run it against historical data to see whether it would have caught past issues.
2. Run a placebo test on your most recent causal claim.
3. Work through the six red-team prompts on a finished analysis and record what each turns up.
4. Interview: Your model's offline performance jumps sharply after adding a new feature. What do you suspect and how do you confirm it? (Hint: the suspicion should be about timing, and the confirmation is a placebo-style check on when the feature became knowable.)

<a id="174-applied-capstone-practice"></a>

17.4 Applied Capstone Practice

Knowing each technique individually is not the same as carrying a problem from a vague question to a defended result. This section covers executing a complete project, showing the work credibly, handling data that does not cooperate, and keeping current once the coursework ends.

End-to-End Project Execution

A capstone differs from an exercise in one decisive way: nobody tells you what the steps are. There is no clean dataset, no stated target metric, and no assurance that the problem is solvable at all. The skill being tested is sequencing — knowing what to do next when several things are unfinished and one of them may be a dead end.

The reliable pattern is to build the thinnest possible end-to-end path first, then deepen it. Get from raw data to a scored output to an evaluated decision using the crudest method that works — a SQL query and a rule is fine — and only then improve the weakest link. This is the opposite of the instinct, which is to perfect the cleaning before modeling and the model before deployment.

The reason the thin path wins is that it surfaces blockers while they are still cheap. If the target label cannot actually be constructed from available data, you want to discover that on day two, not in week five after building beautiful features.

  DEPTH-FIRST (common, fragile)        THIN-PATH-FIRST (robust)

  week 1  perfect the cleaning         day 1-2  crude end-to-end path:
  week 2  engineer 60 features                  load -> filter -> rule ->
  week 3  tune six model families                evaluate -> one number
  week 4  discover the label is                 |
          unavailable at decision      day 3    the crude path already
          time; restart                         shows the label problem
                                       |
  Everything built rests on an         week 1-3 deepen the weakest link,
  assumption tested last.                       re-measuring every time

# The thin path, written as one function. It is deliberately bad; its job
# is to be complete and to produce a defensible baseline number today.
def thin_path(raw_path):
    df = load(raw_path)

    # Minimal cleaning: only what is needed to get a number at all.
    df = df.dropna(subset=["customer_id", "order_date"])

    # Temporal split FIRST, so every later decision is made on train only.
    cutoff = df["order_date"].quantile(0.8)
    train, test = df[df["order_date"] < cutoff], df[df["order_date"] >= cutoff]

    # The crudest possible "model": a single-rule heuristic.
    rule = lambda d: (d["days_since_login"] > 30).astype(int)

    from sklearn.metrics import precision_score, recall_score
    pred = rule(test)
    return {
        "baseline_precision": precision_score(test["churned"], pred),
        "baseline_recall": recall_score(test["churned"], pred),
        "n_train": len(train),
        "n_test": len(test),
    }

# Every later model is reported as a delta against this. A model that cannot
# beat one rule is a finding worth reporting, not a failure to hide.
print(thin_path("data/raw/orders.parquet"))

Two practices keep a long project from drifting. Keep a decision log — a dated file recording each choice and its reason, because by week four you will not remember why enterprise accounts were excluded. And timebox exploration explicitly: "two days on feature engineering, then I evaluate regardless" prevents the common failure of an interesting subproblem consuming the schedule.

Key Takeaways

• Build the thinnest end-to-end path first; it surfaces blockers while they are cheap to fix.
• Split temporally before making any decision informed by the data.
• A crude rule is the baseline every later model must beat, and reporting that a model fails to beat it is a valid result.
• Keep a dated decision log; you will not reconstruct reasons later.
• Timebox exploration so an interesting subproblem cannot eat the schedule.

🧪 Practice

1. Take a dataset you have not used and build a thin end-to-end path in under two hours, ending in one evaluated number.
2. Start a decision log for a project in flight and backfill the last five choices you made.
3. For a model you built, construct the single-rule heuristic baseline and compare honestly.
4. Interview: You are three weeks into a four-week project and the model barely beats the baseline. What do you deliver? (Hint: the baseline comparison is itself information the organization did not have.)

Portfolio Development

A portfolio exists to answer one question for a person who will spend four minutes on it: can this person turn an ambiguous problem into a defensible result? Most portfolios answer a different question — can this person apply scikit-learn to a tidy dataset — which every applicant can do and which therefore carries no information.

The consequence is counterintuitive. A Titanic notebook with excellent code is worth less than a modest project on data you collected yourself about a question someone actually had. Novelty of the problem beats sophistication of the method, because the scarce skill is problem formulation.

The second consequence concerns presentation. Reviewers do not run notebooks. A repository whose README requires cloning to understand will be closed. Every project needs a top-level summary that states the question, the finding, and the limitation in the first paragraph — the same discipline as an executive summary, applied to your own work.

  WEAK PORTFOLIO PROJECT           STRONG PORTFOLIO PROJECT

  "Titanic survival prediction"    "Which of my city's bike-share
   - Kaggle CSV, already clean      stations run empty at commute
   - accuracy 0.79                  time, and would moving 40 bikes
   - 14 models compared             fix it?"
   - no stated question             - data scraped over 3 months
   - no limitation section          - question came from a real
                                      annoyance
  Signals: can run a library.      - baseline: current allocation
  Does not signal: judgment.       - finding stated with a caveat
                                   - one chart that makes the point

                                   Signals: framing, collection,
                                   evaluation, communication.

A README that does the work:

# Bike-share rebalancing: where 40 bikes would help most

**Question.** Which stations are empty during the 08:00-09:00 commute,
and would a fixed overnight reallocation of 40 bikes reduce failed trips?

**Finding.** Six stations account for 61% of empty-station minutes. A
simulated overnight move of 40 bikes cuts failed pickups by an estimated
23% (bootstrap range 14-31%).

**Limitation.** Failed pickups are inferred from station-empty intervals,
not observed directly; the estimate assumes demand is unchanged by
availability, which likely understates the benefit.

**Data.** Scraped from the operator's public API every 5 minutes,
2026-01-04 to 2026-03-28 (n = 1.2M station-observations). Collection
code in `scrape/`.

**Reproduce.** `make all` (runs in ~6 minutes, Docker image pinned).

Three or four projects of this kind, each with a distinct skill on display — one heavy on data collection, one on causal reasoning, one on production deployment — beats a dozen similar notebooks. Depth is legible; breadth of the same thing is not.

Key Takeaways

• The portfolio answers whether you can handle ambiguity, not whether you can call a library.
• Self-collected or messy data on a real question outperforms a clean benchmark dataset.
• Reviewers read the README and do not run the code; lead with question, finding, and limitation.
• Three or four projects showing distinct skills beats a dozen similar ones.
• Including a limitations section signals judgment more strongly than an extra model does.

🧪 Practice

1. Rewrite the README of your best project to lead with question, finding, and limitation in the first paragraph.
2. Identify a question you have personally wondered about and find or collect the data that would answer it.
3. Have someone unfamiliar with your work read one project README for four minutes and then state your finding back to you.
4. Interview: An interviewer asks what you would do differently on a portfolio project. What makes a strong answer? (Hint: name a specific decision and the evidence that would have changed it.)

Working with Messy Real-World Data

Teaching datasets are cleaned so that the lesson is about the method. Real datasets are not, and the gap is larger than newcomers expect: for most projects the majority of effort goes into establishing what the data means before any modeling is possible.

The mess is not random. It has recurring causes, and recognizing the cause tells you the fix. Data is entered by humans under time pressure, which produces typos, free text in categorical fields, and placeholder values. Systems change, which produces schema drift and fields whose meaning silently shifts. Data is joined across systems whose keys disagree. And measurement itself is incomplete: rows are missing precisely where the process failed, which is rarely at random.

Symptom	Likely cause	Response
Same entity, many spellings	Free-text entry	Fuzzy match to a canonical list
Sentinel values (-1, 9999, 0)	"Unknown" without a null	Map to null; count them
Field meaning changes mid-2024	Upstream release	Segment by era; do not pool
Counts drop on weekends	Batch job, not a business effect	Check the pipeline calendar
Duplicate rows after a join	Key is not unique	Verify cardinality before joining
Timestamps cluster at midnight	Date cast from a date-only field	Do not use hour-of-day

import pandas as pd

def profile_mess(df):
    """First pass on unfamiliar data: find the lies before modeling them."""
    report = {}
    for col in df.columns:
        s = df[col]
        info = {"dtype": str(s.dtype), "null_pct": round(s.isna().mean(), 4)}

        if s.dtype == "object":
            info["n_unique"] = s.nunique()
            # A high-cardinality "categorical" is usually free text.
            info["likely_free_text"] = s.nunique() > 0.5 * len(s)
            # Case and whitespace variants of the same value are common.
            cleaned = s.dropna().astype(str).str.strip().str.lower()
            info["collapses_to"] = cleaned.nunique()

        if pd.api.types.is_numeric_dtype(s):
            # Sentinel values hide as legitimate numbers; they show up as
            # implausible modes at the extremes of the range.
            vc = s.value_counts().head(3)
            info["top_values"] = vc.to_dict()
            info["suspect_sentinels"] = [
                v for v in vc.index
                if v in (-1, 0, 999, 9999, -999) and vc[v] > 0.01 * len(s)
            ]

        if pd.api.types.is_datetime64_any_dtype(s):
            # All-midnight timestamps mean the time component is fabricated.
            info["all_midnight"] = bool((s.dt.time == pd.Timestamp("00:00").time()).all())
            info["range"] = (str(s.min()), str(s.max()))

        report[col] = info
    return pd.DataFrame(report).T

# Then check stability over time, which catches the drift a single profile misses.
monthly_nulls = df.groupby(df["created_at"].dt.to_period("M"))["plan_type"].apply(
    lambda s: s.isna().mean()
)
print(monthly_nulls)
# A null rate that jumps from 2% to 40% in one month is a schema change
# upstream, not a data quality problem to impute away.

Two judgments recur. First, decide whether a problem is worth fixing: a field that is 60% missing and weakly related to the outcome should be dropped, not rescued. Second, never silently impute without recording what you did and how much — a null rate is itself a signal, and replacing it with a mean destroys information about the process that generated it.

The most important habit is to find the person who generated the data and ask them. Ten minutes with the operations team explaining why Tuesday's numbers are always low beats a week of statistical detective work.

Key Takeaways

• Expect most project effort to go into understanding what the data means, not into modeling.
• Mess has recognizable causes — manual entry, schema drift, key mismatches, process-correlated missingness — and the cause dictates the fix.
• Profile null rates over time, not just overall; a jump is a schema change, not noise.
• Missingness is usually informative; record it before imputing anything.
• Ask the people who generate the data; it is the highest-yield hour in the project.

🧪 Practice

1. Run a profiling pass on an unfamiliar public dataset and list every anomaly with a hypothesis about its cause.
2. Plot the null rate of three fields by month and explain any step changes.
3. Take a field you previously imputed and test whether its missingness predicts the outcome.
4. Interview: You find that 15% of timestamps are exactly midnight. What do you conclude and what do you stop doing? (Hint: some analyses become invalid, not merely less accurate.)

Competition and Benchmark Practice

Competitions such as Kaggle and public benchmarks are a fast way to build certain skills and a reliable way to distort others. Treating them as representative of professional work produces a predictable set of bad habits, so it is worth being explicit about what transfers and what does not.

What they teach well is genuine. You get fast feedback, a strong reference point in the form of other competitors' scores, exposure to techniques you would not have tried, and practice at the mechanics of validation — which is where most beginners are weakest. Reading top solutions after a competition closes is among the most efficient available forms of study.

What they distort is equally real. The problem is given, the metric is given, the data is assembled, and the evaluation is fixed — which removes precisely the parts of the job that determine whether a project succeeds. The competitive equilibrium also rewards enormous ensembles for hundredths of a point, which is almost always the wrong tradeoff in production.

Aspect	Competition	Professional work
Problem framing	Given	The hardest part
Metric	Given and fixed	Negotiated, often multiple
Data	Assembled for you	You find and join it
Leakage	Something to exploit	Something to eliminate
Model complexity	Rewarded without limit	Constrained by latency and ops
Maintenance	None	The majority of lifetime cost
Result	A score	A changed decision

# The transferable skill: a validation scheme that mirrors the real
# deployment condition. Getting this right matters far more than the model.

# WRONG for a leaderboard split by time or by group — random KFold leaks
# information across the boundary and flatters every model equally.
# from sklearn.model_selection import KFold

from sklearn.model_selection import GroupKFold, TimeSeriesSplit

# If the test set contains customers never seen in training, validate that way.
cv = GroupKFold(n_splits=5)
for tr, va in cv.split(X, y, groups=df["customer_id"]):
    ...

# If the test set is strictly later, validate that way.
cv = TimeSeriesSplit(n_splits=5)

# The diagnostic that matters more than any single score: does local
# validation move together with the public leaderboard?
import numpy as np
cv_scores = [0.812, 0.818, 0.823, 0.827]
lb_scores = [0.809, 0.815, 0.816, 0.810]
print(f"corr(cv, lb) = {np.corrcoef(cv_scores, lb_scores)[0, 1]:.2f}")
# A weak or negative correlation means the validation scheme does not
# reflect the test distribution. Fix the scheme before tuning anything else.

Use competitions deliberately. Pick one for a technique you want to learn rather than for the ranking, stop when the learning stops rather than when the ranking plateaus, and read the winning write-ups regardless of your own result. Then consciously re-acquire what the format removed: for a portfolio, a competition entry is worth far more if you also state what the problem would have required in production.

Key Takeaways

• Competitions build validation discipline and technique exposure with fast feedback; that part transfers.
• They remove framing, metric choice, data assembly, and maintenance — the parts that decide real projects.
• Ensembling for hundredths of a point is a competition-specific incentive, not an engineering practice.
• The core transferable skill is building a validation scheme that mirrors the deployment condition.
• Read winning write-ups; they are dense, free, and reviewed by thousands.

🧪 Practice

1. Enter a competition with the sole goal of building a validation scheme whose scores track the leaderboard within 0.005.
2. Take a competition you worked on and write what the same problem would require in production: metric, latency, retraining, monitoring.
3. Read three winning solutions and list techniques you had not considered.
4. Interview: Your cross-validation score improves but the leaderboard score falls. What is happening? (Hint: two different explanations, one about the split and one about how many times you looked.)

Continuous Learning and Staying Current

The field moves quickly enough that a fixed skill set depreciates, and slowly enough at its foundations that chasing every new development is wasted effort. Distinguishing the two is the actual skill, and it prevents both stagnation and the exhausting sense of permanent catch-up.

The useful model is a layered one. The bottom layer — linear algebra, probability, inference, causal reasoning, the mechanics of generalization — changes on a timescale of decades and repays deep investment. The middle layer, model families and architectures, changes over years. The top layer, specific libraries and APIs, changes over months and should be learned on demand rather than in advance.

  DEPRECIATION RATE DETERMINES STUDY STRATEGY

  tools & APIs          months    learn when needed, do not stockpile
  ----------------------------------------------------------------
  frameworks            1-3 yrs   learn one deeply, others by analogy
  ----------------------------------------------------------------
  architectures         3-7 yrs   understand the problem each solves
  ----------------------------------------------------------------
  methods & statistics  decades   invest here; it makes the rest fast
  ----------------------------------------------------------------
  domain knowledge      career    compounds, and is rarely portable
                                  between industries

  A practitioner strong at the bottom learns a new top layer in a week.
  The reverse is not true.

A structural point about foundations: they are what make the top layer cheap. Someone who understands attention as a learned weighted average over a set can read a new architecture paper in an afternoon. Someone who learned the API without the concept must relearn from scratch each time.

For keeping up without drowning, a few filters work. Prefer sources with signal density — a well-run paper-reading group, a small number of practising authors — over volume feeds. Give a new method a few weeks before evaluating it; the discourse in the first fortnight is uninformative. And evaluate anything new against a specific problem you have, which converts "should I learn this?" into a question with an answer.

# A concrete learning practice: reproduce, do not just read. The gap
# between understanding a method and being able to use it only closes here.

# Step 1: implement the mechanism from the description, minimally.
import numpy as np

def scaled_dot_product_attention(Q, K, V):
    d_k = Q.shape[-1]
    # Scaling by sqrt(d_k) keeps the softmax out of its saturated region,
    # where gradients vanish — the reason the scaling exists at all.
    scores = Q @ K.T / np.sqrt(d_k)
    weights = np.exp(scores - scores.max(axis=-1, keepdims=True))
    weights /= weights.sum(axis=-1, keepdims=True)
    return weights @ V

# Step 2: test it against the library implementation on the same input.
# Step 3: apply it to a problem you own, where you already know the baseline.
# A method you have not applied to your own data is not yet knowledge.

Finally, contribute. Writing up a method for others, answering questions, or maintaining a small package exposes the gaps in your understanding faster than any amount of reading, because a vague explanation survives private study and does not survive publication.

Key Takeaways

• Layer your learning by depreciation rate: foundations repay depth, tools are learned on demand.
• Strong foundations make new tools cheap to learn; the reverse does not hold.
• Filter for signal density and let new methods age a few weeks before evaluating them.
• Evaluate anything new against a specific problem you have.
• Reproduce and apply rather than only read; teach or write to find the gaps in your understanding.

🧪 Practice

1. Classify your last ten hours of learning by layer and check the balance against the depreciation rates above.
2. Implement one mechanism you currently use only through a library, from the paper description, and test it against the library version.
3. Write a short explanation of a technique you believe you understand and give it to someone who does not. Note every question you cannot answer.
4. Interview: How do you decide whether to adopt a new technique? (Hint: a good answer names a problem you had first, and the technique second.)