1. Foundations of Data Science

This chapter establishes what data science actually is, how a project moves from a vague business question to a monitored production system, and what kinds of data you will be handed along the way. Everything in later chapters — the mathematics, the models, the pipelines — is machinery in service of the decisions made here. Getting the framing wrong is the single most expensive mistake in the field, because a technically flawless model that answers the wrong question is worth nothing.

<a id="11-discipline-and-scope"></a>

1.1 Discipline and Scope

Before learning the tools, it helps to know what the discipline claims as its own: which problems belong to data science, how it differs from neighbouring fields that share its techniques, and who does what on a working team.

What Data Science Is

Organizations have always collected records. What changed is that storage became cheap, computation became abundant, and the volume of recorded behaviour outgrew any human's ability to read it. A supermarket chain with 2,000 stores generates more transactions in an hour than an analyst could inspect in a career. The records contain answers — which products sell together, which customers are about to leave, which supplier is quietly degrading — but those answers are not visible by looking.

Data science is the discipline of extracting reliable, actionable knowledge from data using a combination of statistical reasoning, computational method, and domain understanding. Each of the three is load-bearing. Statistics without computation cannot handle the scale. Computation without statistics produces confident nonsense, because it cannot distinguish a real pattern from a coincidence. Both without domain knowledge produce technically valid answers to questions nobody asked.

A useful analogy: data science is to raw data what medicine is to symptoms. A thermometer reading is not a diagnosis. Turning the reading into a diagnosis requires knowing which measurements matter, how they vary in healthy people, what else could explain them, and what treatment the diagnosis implies. The measurement is the easy part.

        DOMAIN EXPERTISE
               / \
              /   \
             /     \
   "asking  /       \  "traditional
    the    /         \   research"
  right   /  DATA     \
 question/  SCIENCE    \
        /               \
       +-----------------+
      /                   \
 STATISTICS -------------- COMPUTATION
        "machine learning without
         a question to answer"

Remove any one circle and the work degrades in a predictable way.

The practical test of whether work is data science is not the technique used but whether it closes a loop: a question was asked, evidence was gathered, uncertainty was quantified, and a decision changed as a result.

# The same computation can be data science or not, depending on framing.

# NOT data science: a number with no question attached.
average_order = orders["amount"].mean()
print(average_order)  # 47.30 ... and now what?

# Data science: a question, a comparison, and a quantified uncertainty.
from scipy import stats

treated = orders.loc[orders["saw_new_checkout"], "amount"]
control = orders.loc[~orders["saw_new_checkout"], "amount"]

# Welch's t-test does not assume the two groups share a variance,
# which matters because a redesign can change spread as well as level.
result = stats.ttest_ind(treated, control, equal_var=False)

lift = treated.mean() - control.mean()
print(f"lift = {lift:+.2f} per order, p = {result.pvalue:.4f}")
# Now there is a decision to make: ship the checkout redesign or not.

Key Takeaways

• Data science extracts reliable, actionable knowledge from data; it is not defined by any single tool or model family.
• It rests on three legs — statistics, computation, domain expertise — and removing any one produces a recognizable failure mode.
• Computation without statistics cannot tell a real pattern from a coincidence.
• Work qualifies as data science when it closes the loop from question to evidence to changed decision.

🧪 Practice

1. Find three numbers published in a news article this week. For each, write the decision that number should inform. Note any that inform none.
2. Take a dataset you know and write down one question that statistics alone could answer, one that needs domain knowledge to even pose, and one that is infeasible without significant computation.
3. Interview: An engineer says "we do not need data science, we have dashboards." What can data science provide that a dashboard cannot? (Hint: consider what a dashboard does when two metrics move at once.)

Data Science vs Statistics vs Machine Learning

These three fields share equations, so people use the names interchangeably and then talk past one another. The distinction is not the mathematics. It is what each field treats as the goal, and therefore what it considers a good result.

Statistics is primarily concerned with inference: given a sample, what can be concluded about the population it came from, and with what confidence? Its central anxiety is being fooled by randomness, so it invests heavily in uncertainty quantification, assumptions, and the validity of conclusions.

Machine learning is primarily concerned with prediction: given past examples, produce accurate outputs on examples never seen before. Its central anxiety is generalization failure, so it invests in held-out evaluation, regularization, and capacity control. It will happily use a model nobody can interpret if that model predicts better.

Data science is the applied discipline that uses both in service of a decision. It owns the parts neither of the other two claims: deciding which question to ask, obtaining and cleaning the data, and delivering something a non-specialist can act on.

An analogy: statistics is the science of evidence, machine learning is the engineering of prediction, and data science is the practice that decides which of the two a given problem needs and then carries the result to the person who must act.

Dimension	Statistics	Machine Learning	Data Science
Primary goal	Inference about a population	Prediction on new data	Decision support
Success measure	Valid conclusions, calibration	Held-out accuracy or loss	Business or scientific impact
Central worry	Being fooled by randomness	Overfitting	Solving the wrong problem
Model preference	Interpretable, assumption-led	Whatever predicts best	Whatever the decision can use
Typical data size	Small to moderate, designed	Large, often observational	Whatever exists
Typical output	Estimate with interval	Trained model or scores	Recommendation, product, report

The same dataset asks different questions of each field:

import statsmodels.api as sm
from sklearn.ensemble import GradientBoostingRegressor

# STATISTICAL FRAMING: does square footage affect price, and by how much?
X = sm.add_constant(homes[["sqft", "bedrooms", "age"]])
model = sm.OLS(homes["price"], X).fit()
print(model.summary())
# Reads out: coefficient, standard error, confidence interval, p-value.
# The coefficient on sqft IS the answer. Predictive accuracy is secondary.

# MACHINE LEARNING FRAMING: what will this unlisted house sell for?
gbm = GradientBoostingRegressor().fit(X_train, y_train)
print(gbm.score(X_test, y_test))   # held-out R^2 IS the answer
# No interpretable coefficient exists, and for this question none is needed.

# DATA SCIENCE FRAMING: which question does the business actually have?
#   "Should we fund the extension?"  -> the statistical framing
#   "What price do we list at?"      -> the machine learning framing
# Choosing wrongly produces a correct answer to the wrong question.

Key Takeaways

• The fields differ by goal, not by mathematics: inference, prediction, and decision support respectively.
• Statistics optimizes for valid conclusions; machine learning optimizes for held-out performance.
• A model that predicts well can still have meaningless coefficients, and an interpretable model can predict poorly.
• Data science owns problem selection, data acquisition, and delivery, which neither of the other two fields claims.

🧪 Practice

1. Classify each as inference or prediction: estimating a vaccine's effect, flagging fraudulent transactions, measuring a wage gap, routing support tickets.
2. Write a question about customer churn in an inferential framing, then rewrite the same business concern as a prediction problem. Note how the required output changes.
3. Interview: Your model has 94% accuracy and a coefficient saying discounts reduce purchases. Can you tell the business to stop discounting? (Hint: ask what else differs between discounted and undiscounted customers.)

Data Science vs Data Engineering vs Analytics

Inside an organization, three job families work on the same data and are routinely confused. The confusion is costly: hiring a data scientist to fix a broken pipeline wastes both the pipeline and the scientist.

The separation is easiest to see as a supply chain. Data engineering builds and operates the infrastructure that makes data available, reliable, and timely. Data analytics uses that data to describe what happened and why, and puts the description in front of decision-makers. Data science builds models and experiments that estimate what will happen or what to do about it.

An analogy: data engineering lays the water mains and keeps pressure in the pipes. Analytics reads the meters and reports consumption. Data science forecasts next summer's demand and decides whether to build a reservoir. None of the three is senior to the others, and the reservoir is useless if the mains leak.

  RAW SOURCES          DATA ENGINEERING           ANALYTICS
  +-----------+        +------------------+       +---------------+
  | app DB    |        | ingestion        |       | dashboards    |
  | events    | -----> | storage, schemas | ----> | ad-hoc SQL    |
  | 3rd party |        | pipelines, tests |   |   | KPI reporting |
  | files     |        | orchestration    |   |   +---------------+
  +-----------+        +------------------+   |
                                              |   DATA SCIENCE
                                              |   +---------------+
                                              +-> | experiments   |
                                                  | models        |
                                                  | forecasts     |
                                                  +---------------+
        "is the data there?"  "what happened?"   "what will happen,
                                                  and what should we do?"

Aspect	Data Engineering	Data Analytics	Data Science
Core question	Is the data available?	What happened and why?	What will happen or help?
Primary skills	Distributed systems, SQL	SQL, visualization, domain	Statistics, ML, experiments
Main output	Pipelines, tables, contracts	Dashboards, reports	Models, experiments, studies
Time horizon	Continuous operation	Past and present	Future and counterfactual
Failure looks like	Stale or missing data	Misread trend	Wrong prediction or decision
Typical tooling	Orchestrators, warehouses	BI tools, notebooks	ML libraries, notebooks

Boundaries blur by company size. At a ten-person startup one person does all three. At a large firm the roles specialize further into analytics engineering, machine learning engineering, and research science. What does not change is the dependency order: unreliable engineering caps the quality of everything downstream, no matter how sophisticated the modelling.

Key Takeaways

• The three roles form a supply chain: availability, then description, then prediction and decision.
• Data engineering failures propagate downstream and cap the ceiling of all later work.
• Analytics looks backward at what happened; data science looks forward or counterfactually.
• Role boundaries are a function of company size, but the dependency order is constant.

🧪 Practice

1. For each task, name the role that should own it: adding a column to a nightly table, explaining a revenue dip, estimating a price change's effect, fixing a dashboard that times out.
2. Sketch the supply chain for a food delivery app, naming one concrete artifact each role would produce.
3. Interview: A data scientist spends 70% of their time writing ingestion jobs. Describe two organizational causes and what you would change. (Hint: consider both missing infrastructure and missing ownership.)

Roles on a Data Team

A data team is not a pile of interchangeable analysts. Real teams divide labour because the required skills genuinely diverge: the person who can tune a distributed Spark job and the person who can design an unbiased experiment are rarely the same person, and forcing one to do the other's job produces mediocre versions of both.

Understanding roles matters even if you work alone, because it tells you which hat you are wearing at a given moment, and which hat you are neglecting.

Role	Owns	Optimizes for
Data engineer	Ingestion, storage, pipeline reliability	Freshness, correctness
Analytics engineer	Modelled tables, metric definitions	Consistency, reusability
Data analyst	Reporting, ad-hoc investigation	Clarity, speed to answer
Data scientist	Experiments, models, causal studies	Decision quality
Machine learning engineer	Productionized models, serving	Latency, reliability
Research scientist	Novel methods, hard modelling problems	Capability frontier
Data product manager	Prioritization, stakeholder alignment	Impact per effort

A CHURN PROJECT, BY ROLE

 Data engineer      Analytics eng.     Analyst          Data scientist
 +------------+     +------------+     +-----------+    +--------------+
 | land       |     | define     |     | profile   |    | build churn  |
 | subscription --> | "active    | --> | churn by  | -->| model, design|
 | events     |     |  user"     |     | segment   |    | retention    |
 +------------+     +------------+     +-----------+    | experiment   |
                                                        +--------------+
                                                               |
                                    ML engineer  <-------------+
                                    +------------------------+
                                    | serve daily risk scores|
                                    | monitor drift          |
                                    +------------------------+

Note the handoff risk at every arrow: a mismatch in the definition of
"active user" silently invalidates everything to its right.

The most common structural failure is not a missing role but an undefined metric ownership. When three teams each compute "monthly active users" slightly differently, meetings become arguments about whose number is right instead of what to do. The analytics engineering function exists largely to prevent this.

Key Takeaways

• Roles exist because the underlying skills genuinely diverge, not because of hierarchy.
• Every handoff between roles is a place where definitions can silently diverge.
• Ambiguous metric ownership causes more damage than a missing specialist.
• Solo practitioners still switch between these roles and should know which one they are currently neglecting.

🧪 Practice

1. List the roles present on a team you know. Identify which responsibility in the table above currently has no owner.
2. Take a metric your organization uses and trace who defines it, who computes it, and who reports it. Note whether these are the same person.
3. Interview: Two teams report different weekly revenue. How do you diagnose the discrepancy, and what would you put in place so it cannot recur? (Hint: the durable fix is organizational, not a one-off query.)

Common Application Domains

Data science techniques are general, but their value is domain-specific. The same gradient boosting model is a fraud detector in banking, a lead-scoring tool in sales, and a diagnostic aid in radiology — and in each case the constraints, the cost of error, and the acceptable methods differ sharply.

Knowing the domain landscape matters for two reasons. It tells you which problems are already well-solved and worth borrowing from, and it warns you which constraints come attached.

Domain	Representative problems	Dominant constraint
Finance	Credit scoring, fraud, algorithmic trading	Regulation, explainability
Healthcare	Diagnosis support, readmission risk	Safety, privacy, small samples
E-commerce	Recommendation, pricing, demand	Latency, cold start, seasonality
Marketing	Attribution, churn, segmentation	Causality, noisy outcomes
Manufacturing	Predictive maintenance, quality control	Rare failures, sensor drift
Transport	Routing, ETA prediction, demand	Real-time, spatial structure
Public sector	Resource allocation, fraud, forecasting	Fairness, transparency, audit
Science	Genomics, climate, particle physics	Measurement error, reproducibility

COST ASYMMETRY DIFFERS BY DOMAIN

 Spam filter                      Cancer screening
 ------------------------------   ------------------------------
 False positive: user misses      False positive: unnecessary
   one email. Annoying.             biopsy. Costly, invasive.
 False negative: user sees        False negative: missed tumour.
   one spam. Trivial.               Potentially fatal.

 => tune toward precision         => tune heavily toward recall

Identical model, identical metrics, opposite operating point.
The domain, not the algorithm, decides where the threshold goes.

The transferable lesson is that technique portability is high and judgement portability is low. You can move a model architecture from advertising to oncology in an afternoon; moving the threshold, the evaluation protocol, and the tolerance for error takes domain expertise and, often, regulatory review.

Key Takeaways

• Techniques generalize across domains, but constraints and error costs do not.
• The cost asymmetry between false positives and false negatives is set by the domain, not the algorithm.
• Regulated domains restrict which models are permissible regardless of accuracy.
• Surveying how a neighbouring domain solved a structurally similar problem is usually faster than inventing a method.

🧪 Practice

1. Pick two domains from the table and describe a problem that is structurally identical in both but has opposite error-cost asymmetry.
2. For a domain you know, list three constraints a newcomer would not guess from the data alone.
3. Interview: You are asked to port a working credit-risk model to a healthcare client. What do you check before writing any code? (Hint: think about what the regulator and the clinician each require of it.)

<a id="12-the-data-science-lifecycle"></a>

1.2 The Data Science Lifecycle

A data science project moves through recognizable stages, and most project failures can be traced to a stage that was rushed or skipped; this section walks the loop in order and names what each stage owes the next.

Problem Framing

Every project begins with someone describing a problem in the language of their job, not in the language of data. "Customers are unhappy." "Churn is too high." "We need to use AI." None of these can be computed. Framing is the translation step, and it is where the most value is won or lost, because every later stage inherits its assumptions.

Framing answers four questions: what decision will change based on this work, what quantity would inform that decision, what data could estimate that quantity, and what would make the result useless. The last question is the one people skip, and it is the one that saves months.

An analogy: framing is the survey before construction. Pouring a foundation on unsurveyed ground is faster for exactly one day.

FRAMING FUNNEL

 "Churn is too high"                        <- stakeholder statement
        |
        v  Whose decision changes?
 "Retention team picks 5,000 users/month
  for a win-back offer"                     <- the actual decision
        |
        v  What quantity informs it?
 "P(cancels within 30 days) per active
  subscriber, ranked"                       <- the estimand
        |
        v  What data could estimate it?
 "Subscription events, usage logs, support
  tickets, 24 months history"               <- the evidence
        |
        v  What makes it useless?
 "Scores arriving after the monthly
  campaign is locked; or a model that
  only flags users already cancelling"      <- the failure conditions

# A framing document is worth more than the first model. Write it down.
framing = {
    "decision":   "Which 5,000 subscribers receive the win-back offer",
    "owner":      "Retention team, monthly campaign",
    "estimand":   "P(cancel within 30 days | features as of scoring date)",
    "unit":       "active subscriber",         # NOT account, NOT household
    "horizon":    "30 days",                   # changes the label definition
    "baseline":   "tenure-based heuristic, currently 11% hit rate",
    "useless_if": [
        "scores land after the 3rd of the month",   # timing constraint
        "requires features unavailable at scoring time",  # leakage risk
        "cannot be explained to the compliance team",
    ],
}
# Every entry above rules out some modelling choice before it is made.

Note how much the framing constrains: the 30-day horizon defines the label, the scoring date defines which features are legal to use, and the campaign lock date defines the latency budget. A model built without these produces a number that cannot be used.

Key Takeaways

• Framing translates a business statement into a decision, an estimand, and a data requirement.
• Always ask what would make the result useless; it surfaces hard constraints early.
• The unit of analysis and the prediction horizon must be pinned down before any data is pulled.
• Later stages inherit framing assumptions silently, so errors here are the most expensive kind.

🧪 Practice

1. Take the request "we should use AI to improve customer support" and run it through the framing funnel to a concrete estimand.
2. Write the "useless if" list for a demand forecasting project at a grocery chain. Aim for at least four entries.
3. Interview: A stakeholder asks for "a model that predicts which employees will quit." What do you clarify before agreeing? (Hint: consider both the horizon and what anyone is permitted to do with the output.)

Data Collection

With a question defined, the next task is assembling evidence. This stage is consistently underestimated, because it looks like plumbing and behaves like archaeology: the data exists, but it was recorded for someone else's purpose, under conventions nobody documented, with gaps where systems were migrated.

Collection has three distinct modes. Existing data is retrieved from internal systems and is cheap but shaped by past decisions. Purchased or public data extends coverage but arrives with unknown provenance. Instrumented data is newly collected by adding logging or running a study, which is expensive and slow but is the only way to obtain measurements nobody recorded before.

The critical discipline is documenting provenance while collecting, not afterwards. Six months later, nobody remembers why the 2023 rows have a different currency convention.

import pandas as pd

def load_with_provenance(query, source, engine):
    """Return data plus the metadata needed to reproduce and trust it."""
    frame = pd.read_sql(query, engine)
    meta = {
        "source": source,
        "query": query,
        "pulled_at": pd.Timestamp.utcnow(),   # data changes; record when
        "row_count": len(frame),
        "date_range": (frame["event_date"].min(), frame["event_date"].max()),
        "null_rate": frame.isna().mean().round(4).to_dict(),
    }
    return frame, meta

events, events_meta = load_with_provenance(
    "SELECT * FROM subscription_events WHERE event_date >= '2024-01-01'",
    source="warehouse.prod.subscription_events",
    engine=engine,
)

# Check coverage BEFORE modelling: a silent gap here becomes a silent
# bias later, and by then it is attributed to the model.
daily = events.groupby(events["event_date"].dt.date).size()
missing_days = pd.date_range(daily.index.min(), daily.index.max()).difference(
    pd.to_datetime(daily.index)
)
print(f"days with zero events: {len(missing_days)}")  # expect 0, investigate otherwise

WHAT THE DATA ACTUALLY REPRESENTS

  Reality            Recorded            Retrieved          Analysed
  +---------+        +---------+         +---------+        +---------+
  | all     |  --->  | logged  |  --->   | queried |  --->  | after   |
  | customer|        | events  |         | subset  |        | cleaning|
  | actions |        |         |         |         |        |         |
  +---------+        +---------+         +---------+        +---------+
       |                  |                   |                  |
   everything      instrumentation      filters, joins,     dropped rows
   that happened   gaps, outages        date windows        and columns

Each arrow narrows the population. Your conclusions apply to the
rightmost box, but stakeholders will hear them as claims about the
leftmost one. Saying so explicitly is part of the job.

Key Takeaways

• Existing data was recorded for another purpose and carries that purpose's biases.
• Record provenance at collection time: source, query, timestamp, row counts, and coverage.
• Check for gaps and outages before modelling, because they later look like model failures.
• Every stage from reality to analysis narrows the population your conclusions actually describe.

🧪 Practice

1. Pick a table in a system you use and list three recording conventions a newcomer would misinterpret.
2. Write a coverage check that verifies a daily events table has no missing days and no day with implausibly few rows.
3. Interview: Your training data comes from users who opted into a beta. What can and cannot be concluded from a model trained on it? (Hint: think about who is absent from the leftmost box.)

Data Preparation

Raw data is rarely analysable. Fields arrive as text that should be numbers, categories spelled four ways, timestamps in mixed time zones, duplicated rows from retried writes, and missing values encoded as blank, NULL, -1, and 9999 in the same column. Preparation is the work of turning this into a consistent table with known semantics.

The stage has a reputation for consuming most of a project's time, and that reputation is accurate. It is not wasted time: every decision made here is a modelling decision. Choosing to drop rows with missing income, rather than impute it, changes which population the model describes.

The essential discipline is that preparation must be reproducible code, not manual edits. A spreadsheet fixed by hand cannot be rerun next month, and nobody can audit what was changed.

import pandas as pd
import numpy as np

def prepare(raw: pd.DataFrame) -> pd.DataFrame:
    """Every transformation is explicit, ordered, and rerunnable."""
    df = raw.copy()

    # 1. Normalize sentinel values BEFORE any numeric operation, otherwise
    #    -1 and 9999 silently pollute means and quantiles.
    df["income"] = df["income"].replace({-1: np.nan, 9999: np.nan})

    # 2. Standardize categories: strip, lowercase, then map known variants.
    df["plan"] = (df["plan"].str.strip().str.lower()
                  .replace({"prem": "premium", "premium ": "premium"}))

    # 3. Parse timestamps to a single timezone; naive datetimes cause
    #    off-by-one-day errors at month boundaries.
    df["signup_at"] = pd.to_datetime(df["signup_at"], utc=True, errors="coerce")

    # 4. De-duplicate on the true business key, keeping the latest record.
    df = (df.sort_values("updated_at")
            .drop_duplicates(subset=["subscriber_id"], keep="last"))

    # 5. Assert invariants. Failing loudly here beats debugging a model later.
    assert df["subscriber_id"].is_unique, "duplicate subscribers survived"
    assert df["signup_at"].notna().all(), "unparseable signup timestamps"
    return df

clean = prepare(raw_subscribers)

Problem	Naive fix	Consequence of the naive fix
Missing values	Drop the rows	Silently restricts the population
Missing values	Fill with the mean	Shrinks variance, distorts intervals
Outliers	Delete anything extreme	Removes the fraud you meant to detect
Mixed categories	One-hot everything	Explodes dimensionality with typos
Duplicate rows	drop_duplicates()	Wrong if the key is not the full row
Mixed time zones	Ignore	Day-boundary errors in every aggregate

Key Takeaways

• Preparation decisions are modelling decisions: they change which population the results describe.
• Sentinel values must be normalized before any numeric computation.
• Preparation belongs in versioned, rerunnable code, never in manual edits.
• Assert invariants at the end of preparation so violations fail loudly and early.

🧪 Practice

1. Write a preparation function for a messy CSV that has mixed date formats, a numeric column stored as text with thousands separators, and duplicate IDs.
2. For a dataset you know, list every sentinel value used for "missing" and how you would detect each programmatically.
3. Interview: Your cleaning script drops 12% of rows. What do you check before proceeding? (Hint: compare the dropped rows to the kept ones on the outcome variable.)

Modeling

Modeling is the stage most people picture when they hear "data science", and it is usually the shortest. A model is a compressed description of a pattern in data, fitted so it can be applied to cases it has not seen.

The essential discipline is to start with the simplest thing that could work. A baseline serves three purposes: it establishes whether the problem is learnable at all, it gives every later model something to beat, and it is often good enough to ship while the sophisticated version is being built. Teams that begin with a neural network have no way to know whether their 82% accuracy is impressive or embarrassing.

An analogy: a baseline is the tare weight on a scale. Without it every subsequent measurement is uninterpretable.

from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_score

models = {
    # Predicts the majority class. If a real model cannot beat this,
    # the features carry no signal for the label as defined.
    "constant": DummyClassifier(strategy="most_frequent"),
    # Domain heuristic encoded as a model is often the true baseline.
    "logistic": LogisticRegression(max_iter=1000),
    "boosted":  HistGradientBoostingClassifier(),
}

for name, model in models.items():
    # roc_auc is threshold-free, so ranking quality is compared fairly
    # across models before any operating point is chosen.
    scores = cross_val_score(model, X, y, cv=5, scoring="roc_auc")
    print(f"{name:10s} AUC = {scores.mean():.3f} +/- {scores.std():.3f}")

# Typical output:
#   constant   AUC = 0.500 +/- 0.000   <- by construction, the floor
#   logistic   AUC = 0.781 +/- 0.014   <- the real baseline to beat
#   boosted    AUC = 0.824 +/- 0.011   <- is +0.043 worth the complexity?

The last line of that comment is the real modelling question. Extra accuracy has costs: harder debugging, slower inference, more fragile retraining, and reduced explainability. Whether the gain is worth it depends on the decision identified during framing, which is why the two stages are linked.

Key Takeaways

• Always establish a trivial baseline and a simple-model baseline before anything sophisticated.
• A score is meaningless without a reference point to compare it against.
• Added accuracy carries costs in debuggability, latency, and explainability.
• Whether a gain justifies its complexity is decided by the framing, not by the metric.

🧪 Practice

1. For a classification dataset, compute the constant-predictor baseline and explain why accuracy alone is misleading when classes are imbalanced.
2. Fit a linear and a tree-based model on the same data and compare training time, inference time, and score. Decide which you would ship.
3. Interview: Your boosted model beats logistic regression by 0.3% AUC. Do you deploy it? (Hint: ask what the 0.3% is worth in the decision the model feeds.)

Evaluation

Evaluation asks whether the model is good enough for its purpose. The trap is that "good" is not a property of a model; it is a relationship between a model, a decision, and a cost structure. A 95% accurate model is excellent for recommending films and unacceptable for controlling a pacemaker.

Two separate questions must be answered. Statistical evaluation asks whether the model performs on data it has not seen, which requires a held-out set that was never used for any decision during development. Decision evaluation asks whether using the model beats the current practice once its errors are priced.

The most common evaluation failure is the metric that looks impressive and means nothing. Accuracy on imbalanced data is the classic: predicting "no fraud" for every transaction on a dataset with 0.2% fraud scores 99.8%.

from sklearn.metrics import (confusion_matrix, precision_recall_fscore_support,
                             roc_auc_score)
import numpy as np

proba = model.predict_proba(X_test)[:, 1]

# Threshold selection is a BUSINESS decision, not a default of 0.5.
# Suppose a missed fraud costs 300 and a false alarm costs 8 in review time.
COST_FN, COST_FP = 300, 8

thresholds = np.linspace(0.01, 0.99, 99)
costs = []
for t in thresholds:
    tn, fp, fn, tp = confusion_matrix(y_test, proba >= t).ravel()
    costs.append(fn * COST_FN + fp * COST_FP)

best = thresholds[int(np.argmin(costs))]
print(f"cost-minimizing threshold = {best:.2f}, expected cost = {min(costs):,.0f}")
print(f"cost at default 0.50      = {costs[49]:,.0f}")
# The default threshold is almost never the cost-minimizing one.

p, r, f1, _ = precision_recall_fscore_support(y_test, proba >= best,
                                              average="binary")
print(f"AUC={roc_auc_score(y_test, proba):.3f} precision={p:.3f} recall={r:.3f}")

THE EVALUATION HIERARCHY

  4. Does it improve the decision in production?   <- the only one that pays
     (online experiment, monitored over time)
        ^
  3. Does it beat the incumbent on business cost?
     (offline, with errors priced)
        ^
  2. Does it generalize to unseen data?
     (held-out set, cross-validation)
        ^
  1. Does it fit the training data?
     (training score -- necessary, worthless alone)

Most projects stop at level 2 and report it as if it were level 4.

Key Takeaways

• "Good" is a relationship between a model, a decision, and the cost of errors, not a property of the model.
• Accuracy on imbalanced data is actively misleading; use metrics matched to the error costs.
• The 0.5 classification threshold is a default, not a choice; derive it from costs.
• Held-out data stops being held out the moment it influences a development decision.

🧪 Practice

1. Build a confusion matrix for a model at three thresholds and describe how precision and recall trade off across them.
2. Given costs of 500 for a false negative and 20 for a false positive, find the cost-minimizing threshold on a model you have trained.
3. Interview: A colleague tuned hyperparameters using the test set and reports 91% test accuracy. What is that number worth? (Hint: ask what remains that the model has genuinely never influenced.)

Deployment and Monitoring

A model that lives in a notebook changes nothing. Deployment is the stage where the model becomes part of a process that runs without its author present, and monitoring is the recognition that this process will degrade.

Degradation is not a possibility but a certainty, because models encode assumptions about a world that keeps moving. Prices change, competitors launch, a marketing campaign changes the mix of arriving users, an upstream team renames a column. The model does not announce any of this. It keeps producing confident numbers that are quietly wrong, which is far more dangerous than a crash.

An analogy: deploying a model without monitoring is like installing a smoke detector and removing its battery. The absence of an alarm no longer carries information.

DEPLOYMENT PATTERNS

 BATCH SCORING                        ONLINE SERVING
 +----------------------+             +-----------------------+
 | scheduled job nightly|             | request arrives       |
 |   read features      |             |   fetch features      |
 |   score all users    |             |   score one entity    |
 |   write to table     |             |   return in < 50ms    |
 +----------------------+             +-----------------------+
 simple, cheap, tolerant              complex, needs a feature
 of slow models; scores               store and latency budget;
 are stale by up to a day             always fresh

 Choose by the decision's timing, which framing already specified.

# Monitoring is three distinct checks. Skipping any one hides a failure mode.

def monitor(reference: pd.DataFrame, live: pd.DataFrame, preds,
            reference_pred_mean: float, labels=None):
    alerts = []

    # 1. INPUT DRIFT: are incoming features distributed as in training?
    for col in ["tenure_days", "sessions_7d", "support_tickets"]:
        shift = abs(live[col].mean() - reference[col].mean()) / reference[col].std()
        if shift > 0.5:                      # half a training standard deviation
            alerts.append(f"input drift on {col}: {shift:.2f} sd")

    # 2. PREDICTION DRIFT: is the score distribution moving? This catches
    #    upstream breakage even when labels have not arrived yet.
    if preds.mean() > 2 * reference_pred_mean:
        alerts.append("prediction rate doubled vs reference")

    # 3. PERFORMANCE: only computable once labels arrive, which for a
    #    30-day churn horizon means a 30-day blind spot by construction.
    if labels is not None:
        auc = roc_auc_score(labels, preds)
        if auc < 0.75:
            alerts.append(f"AUC degraded to {auc:.3f}")
    return alerts

Note the blind spot in check three. When outcomes take 30 days to observe, performance monitoring is 30 days late, so input and prediction drift are the only early warnings available. This is why all three checks exist.

Key Takeaways

• Model degradation is certain, and a degrading model fails silently rather than loudly.
• Choose batch or online serving based on the decision's timing, which framing already established.
• Monitor inputs, predictions, and performance separately; each catches failures the others miss.
• When labels arrive late, drift monitoring is the only timely signal available.

🧪 Practice

1. Write a drift check comparing a live feature's distribution to a stored training reference, and decide what threshold should page someone.
2. Design a monitoring plan for a model whose outcomes are only observable after 90 days. State what you watch in the meantime.
3. Interview: Your model's accuracy has been stable for six months but business impact has fallen. What could explain it? (Hint: the population reaching the model may no longer be the population that matters.)

CRISP-DM and Alternative Process Models

The stages described above are widely agreed on, but teams need a shared, named process to coordinate around. Several formalizations exist, and knowing them matters mostly because colleagues will reference them by name.

CRISP-DM, the Cross-Industry Standard Process for Data Mining, is the oldest and most cited. It defines six phases and, crucially, draws them as a cycle with backward arrows: understanding the data routinely sends you back to rethinking the business question.

CRISP-DM

        +--------------------+
        | Business           |
   +--->| Understanding      |<---+
   |    +--------------------+    |
   |              |               |
   |              v               |
   |    +--------------------+    |
   +----| Data Understanding |    |
        +--------------------+    |
                  |               |
                  v               |
        +--------------------+    |
        | Data Preparation   |<-+ |
        +--------------------+  | |
                  |             | |
                  v             | |
        +--------------------+  | |
        | Modeling           |--+ |
        +--------------------+    |
                  |               |
                  v               |
        +--------------------+    |
        | Evaluation         |----+
        +--------------------+
                  |
                  v
        +--------------------+
        | Deployment         |
        +--------------------+

The backward arrows are the point. Linear execution is the anti-pattern.

Process	Origin	Emphasis	Weakness
CRISP-DM	Industry, 1999	Business framing, iteration	Silent on deployment and MLOps
KDD	Academia, 1996	Knowledge discovery steps	Research-oriented, dated
SEMMA	SAS	Sample, Explore, Modify, Model, Assess	Tool-centric, no business phase
TDSP	Microsoft	Roles, artifacts, repo structure	Heavier process overhead
ML Ops loops	Modern practice	Continuous retraining, monitoring	Assumes mature infrastructure

The honest summary is that no process model survives contact with a real project unchanged. Their value is not prescriptive but communicative: they give a team vocabulary for saying "we are still in data understanding, so it is too early to promise a launch date." Use one as a checklist, not a schedule.

Key Takeaways

• CRISP-DM's six phases are standard vocabulary; its backward arrows are its most important feature.
• Process models mainly provide shared language and a checklist, not a reliable timeline.
• Older models predate production machine learning and say little about deployment or monitoring.
• Iteration between phases is normal and expected, not a sign of poor planning.

🧪 Practice

1. Map a project you have worked on onto CRISP-DM's six phases and mark every point where you moved backwards.
2. Compare CRISP-DM with a modern MLOps loop and list what each covers that the other omits.
3. Interview: A manager asks for a fixed timeline with one week per CRISP-DM phase. How do you respond? (Hint: identify which phase's duration is genuinely unknowable at the start.)

<a id="13-types-of-data"></a>

1.3 Types of Data

The type of a variable determines which operations are meaningful on it and which methods are valid, so misclassifying data is a silent source of wrong answers; this section covers the classifications that matter in practice.

Structured, Semi-Structured, Unstructured Data

The first question about any dataset is how much of its organization is declared in advance. This determines how it can be stored, queried, and how much work stands between you and analysis.

Structured data has a fixed schema known before the data arrives: rows and columns, each column with a declared type. A relational table is the archetype. Because the shape is guaranteed, querying is fast and analysis can start immediately.

Semi-structured data carries its own structure inside each record rather than in a global schema. JSON documents are the archetype: every record is self-describing, records may have different fields, and nesting is arbitrary. This is flexible for producers and awkward for consumers.

Unstructured data has no machine-readable organization at all. Text, images, audio, video. It is not literally without structure, but the structure is semantic rather than declared, and extracting it is itself a modelling problem.

  STRUCTURED            SEMI-STRUCTURED          UNSTRUCTURED

  id | name  | age      {                        "The delivery arrived
  ---+-------+----      "id": 1,                  two days late and the
   1 | Ada   | 36        "name": "Ada",           box was crushed. Driver
   2 | Grace | 45        "tags": ["a","b"],       was polite though."
   3 | Alan  | 41        "meta": {"src": "web"}
                        }                        [image bytes]
  schema declared       schema per record        no schema
  once, enforced        varies, self-describing  semantics only

  query: SQL            query: JSONPath, unnest  query: embed, then search
  effort to analyse:    effort: moderate         effort: high
  low                   (flatten, type-coerce)   (extract features first)

Roughly 80% of organizational data is unstructured, which is why techniques that convert it into structure — embeddings, entity extraction, transcription — unlock disproportionate value.

import pandas as pd
import json

# Semi-structured to structured: the flattening step every project needs.
records = [
    {"id": 1, "user": {"name": "Ada", "plan": "pro"}, "tags": ["beta"]},
    {"id": 2, "user": {"name": "Grace"}, "tags": []},   # note: no "plan" key
]

flat = pd.json_normalize(records)          # nested keys become dotted columns
print(flat.columns.tolist())
# ['id', 'tags', 'user.name', 'user.plan']
# Grace's missing "plan" becomes NaN rather than an error -- convenient,
# but it means a schema change upstream appears as a null-rate change here.

# Lists still need a decision: explode to rows, or aggregate to a count?
exploded = flat.explode("tags")            # one row per tag, id repeated
tag_counts = flat.assign(n_tags=flat["tags"].str.len())   # one row per record

Key Takeaways

• The classification is about where the schema lives: declared globally, per record, or not at all.
• Semi-structured data trades producer flexibility for consumer effort, and schema changes surface as silent nulls.
• Unstructured data requires a feature-extraction step that is itself a modelling problem.
• Most organizational data is unstructured, so structuring techniques carry outsized value.

🧪 Practice

1. Take a nested JSON API response and flatten it into a table, deciding explicitly how to handle list-valued fields.
2. Classify five data sources in your organization and note the extraction effort each requires before analysis.
3. Interview: An upstream team adds an optional field to a JSON event. What breaks downstream, and when do you notice? (Hint: nothing errors, so ask what quietly changes in your tables.)

Numerical vs Categorical Variables

Once data is in a table, each column must be classified by what it means, not by how it is stored. This distinction drives every later choice: which summary statistic is valid, which plot to draw, which encoding a model needs.

Numerical (quantitative) variables represent magnitudes where arithmetic is meaningful. They split into discrete variables, which are counts and take separable values, and continuous variables, which can take any value in a range.

Categorical (qualitative) variables represent group membership. Arithmetic on them is meaningless even when they are stored as integers, which is the source of a large fraction of beginner errors. They split into nominal, where categories have no order, and ordinal, where they do.

The trap is that storage type lies. Postal codes are integers and are categorical. A five-point satisfaction rating is stored as an integer, is ordinal, and averaging it is defensible only under assumptions people rarely state. A boolean is a two-level categorical that happens to permit a meaningful mean, because the mean of a 0/1 column is a proportion.

import pandas as pd

df = pd.DataFrame({
    "zip_code":   [10001, 94105, 60601],   # integer storage, CATEGORICAL
    "n_orders":   [3, 11, 0],              # discrete numerical
    "revenue":    [42.10, 380.55, 0.0],    # continuous numerical
    "plan":       ["free", "pro", "free"], # nominal categorical
    "csat":       [4, 2, 5],               # ordinal categorical
    "is_active":  [True, False, True],     # binary categorical
})

# The error: pandas will happily compute a meaningless number.
print(df["zip_code"].mean())      # 54835.67 -- a postcode in no city

# The fix: declare intent in the dtype, and the mistake becomes an error.
df["zip_code"] = df["zip_code"].astype("category")
# df["zip_code"].mean()  -> TypeError, which is exactly what you want.

# Ordinal encoding preserves order for models that can use it.
csat_type = pd.CategoricalDtype(categories=[1, 2, 3, 4, 5], ordered=True)
df["csat"] = df["csat"].astype(csat_type)
print(df["csat"].max())           # 5 -- comparison is valid
print((df["csat"] >= 4).mean())   # 0.667 -- proportion satisfied, defensible

Variable type	Example	Valid summary	Typical encoding for models
Continuous	Revenue, latency	Mean, sd, quantiles	Scale or leave as is
Discrete count	Orders, clicks	Mean, median, total	As is, or log for skew
Nominal	Country, plan	Mode, frequencies	One-hot, target encoding
Ordinal	Rating, tier	Median, mode	Ordinal integer codes
Binary	Churned, clicked	Proportion	0/1

Key Takeaways

• Classify columns by meaning, not by storage type; integers are frequently categorical.
• Arithmetic on nominal categories is meaningless even when it computes without error.
• Declaring a categorical dtype converts silent nonsense into a loud TypeError.
• Ordinal variables support comparison and median but not, strictly, averaging.

🧪 Practice

1. Take any dataset and classify every column into the five types in the table. Flag the ones whose storage type misleads.
2. Demonstrate an ordinal variable where the mean and the median lead to different conclusions about a group.
3. Interview: Is it acceptable to report the average of a 1-to-5 star rating? Argue both sides. (Hint: ask whether the gap from 1 to 2 is the same size as the gap from 4 to 5.)

Measurement Scales: Nominal, Ordinal, Interval, Ratio

The numerical-versus-categorical split is the practical version of a finer framework from measurement theory. The four scales form a hierarchy, each adding one permitted operation, and knowing where a variable sits tells you exactly which statements about it are meaningful.

Nominal supports only equality. Categories are labels: red != blue, and nothing more.

Ordinal adds order. You can rank, so small < medium < large, but the gaps between ranks are unknown and not assumed equal.

Interval adds meaningful differences. The gap between 20 and 30 degrees Celsius equals the gap between 30 and 40. But zero is an arbitrary convention, not an absence of the quantity, so ratios are meaningless: 40 degrees is not "twice as hot" as 20.

Ratio adds a true zero, so ratios become meaningful. Zero kilograms is an absence of mass, and 40 kg is genuinely twice 20 kg.

An analogy for the interval/ratio boundary: a calendar year is interval (year 2000 is not "twice" year 1000) while a duration in years is ratio (20 years is twice 10 years). The same unit, different scale, because one has a true origin and the other has a convention.

SCALE HIERARCHY -- each level adds one operation

  NOMINAL    | = !=                    | blood type, country, plan
     |       |                         |
  ORDINAL    | = != < >                | satisfaction, education level
     |       |                         |
  INTERVAL   | = != < > + -            | Celsius, calendar year, IQ
     |       |                         |
  RATIO      | = != < > + - * /        | mass, revenue, duration, count
             |                         |
             +-- permitted operations  +-- examples

  Anything valid at one level is valid at every level below it.

# Why the distinction is not academic: the same operation, two verdicts.

celsius_today, celsius_yesterday = 30, 15
print(celsius_today - celsius_yesterday)   # 15 degrees warmer -- MEANINGFUL
print(celsius_today / celsius_yesterday)   # 2.0 -- MEANINGLESS on interval

# Convert to Kelvin, which is a ratio scale with a true zero, and the
# ratio becomes defensible but also reveals how wrong the naive one was.
kelvin_today, kelvin_yesterday = 303.15, 288.15
print(kelvin_today / kelvin_yesterday)     # 1.052 -- 5% warmer, not 100%

# Practical consequence: percentage-change metrics on interval scales are
# nonsense. "Temperature rose 100%" is not a statement about the world.

Scale	Equality	Order	Differences	Ratios	Valid centre
Nominal	Yes	No	No	No	Mode
Ordinal	Yes	Yes	No	No	Mode, median
Interval	Yes	Yes	Yes	No	Mode, median, mean
Ratio	Yes	Yes	Yes	Yes	All, plus geometric

Key Takeaways

• The four scales form a hierarchy where each level adds exactly one permitted operation.
• Interval scales have an arbitrary zero, so ratios and percentage changes on them are meaningless.
• Ratio scales have a true zero, which is what makes "twice as much" well-defined.
• The scale, not the storage type, determines which summary statistic is valid.

🧪 Practice

1. Assign a scale to each: shoe size, pH, revenue, star rating, date of birth, age, jersey number.
2. Find a published statistic that computes a percentage change on an interval scale and explain the error.
3. Interview: Someone reports that a satisfaction score "improved 20%" from 3.0 to 3.6. What is wrong with that claim? (Hint: consider whether the scale has a true zero and whether its intervals are equal.)

Cross-Sectional, Time Series, Panel Data

Beyond individual columns, the dataset as a whole has a structure defined by how observations relate across entities and time. This structure decides which models are valid, because most standard methods assume observations are independent, and two of these three structures violate that assumption.

Cross-sectional data observes many entities at one point in time. A survey of 5,000 customers taken this Tuesday. Rows are generally independent, which is what most textbook methods assume.

Time series data observes one entity repeatedly over time. Daily revenue for one store across three years. Consecutive observations are correlated, so random train/test splitting leaks the future into the past and produces spectacular, fictional accuracy.

Panel (longitudinal) data observes many entities repeatedly over time. Daily revenue for 200 stores across three years. It combines both complications and is also the most informative, because each entity acts as its own control.

  CROSS-SECTIONAL          TIME SERIES              PANEL

  entity | rev             date  | rev              entity| date | rev
  -------+-----            ------+-----             ------+------+-----
   A     | 120             Jan 1 | 120               A    | Jan 1| 120
   B     |  95             Jan 2 | 131               A    | Jan 2| 131
   C     | 210             Jan 3 | 118               B    | Jan 1|  95
   D     |  88             Jan 4 | 145               B    | Jan 2| 103

  many entities            one entity               many entities
  one time point           many time points         many time points

  rows ~ independent       autocorrelated           both, plus entity
                                                    effects

  random split OK          MUST split by time       split by time, and
                                                    consider entity too

from sklearn.model_selection import train_test_split, TimeSeriesSplit

# WRONG for temporal data: random splitting puts March rows in training
# and February rows in test, so the model "predicts" a past it has seen.
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=0)
# Reported accuracy will be high and will not survive production.

# RIGHT: every test fold lies strictly after its training fold.
splitter = TimeSeriesSplit(n_splits=5)
for fold, (train_idx, test_idx) in enumerate(splitter.split(X_sorted)):
    print(f"fold {fold}: train rows 0-{train_idx[-1]}, "
          f"test rows {test_idx[0]}-{test_idx[-1]}")
# fold 0: train rows 0-199,  test rows 200-399
# fold 4: train rows 0-999,  test rows 1000-1199
# Training windows grow; test windows are always in the future.

# For PANEL data, a time split alone is insufficient if you also need the
# model to generalize to unseen entities. Split by entity as well, or the
# evaluation only answers "new dates for known stores".

Key Takeaways

• The dataset's structure decides whether the independence assumption behind standard methods holds.
• Random train/test splits on temporal data leak the future and produce fictional accuracy.
• Panel data combines temporal and entity-level dependence but lets each entity serve as its own control.
• With panel data, decide whether you must generalize to new times, new entities, or both, and split accordingly.

🧪 Practice

1. Classify these as cross-sectional, time series, or panel: a census, an hourly server metric, quarterly financials for 50 firms.
2. Train a model on temporal data using both a random split and a time-based split, and compare the reported scores.
3. Interview: Your sales forecast scored 0.95 R-squared offline and fails in production. What do you check first? (Hint: look at how the evaluation folds were constructed.)

Data Quality Dimensions

"Is this data any good?" is too vague to act on. Data quality decomposes into distinct dimensions that fail independently and are detected by different checks. A dataset can be perfectly accurate and useless because it arrives three days late.

Naming the dimensions turns a vague worry into a test suite.

Dimension	Question it answers	Typical check
Completeness	Are values present where they should be?	Null rate per column
Accuracy	Do values match reality?	Reconcile against a source
Consistency	Do related values agree with each other?	Cross-field rules
Validity	Do values conform to their declared format?	Type, range, regex, enum
Uniqueness	Is each real entity represented once?	Duplicate key count
Timeliness	Is the data recent enough to act on?	Lag from event to availability
Relevance	Does it measure what the question needs?	Human review against framing

The dimensions are independent. A table can be 100% complete and entirely inaccurate if a sensor is miscalibrated. It can be perfectly valid and inconsistent if start_date is a legal date that falls after end_date.

An analogy: these dimensions are the difference between "the report is finished" and "the report is finished, correct, internally coherent, and on my desk before the meeting." Only the last describes something usable.

def quality_report(df: pd.DataFrame) -> dict:
    """One check per dimension. Each failure implies a different fix."""
    now = pd.Timestamp.utcnow()
    return {
        # COMPLETENESS: which columns are missing values, and how badly?
        "null_rate": df.isna().mean().round(4).to_dict(),

        # UNIQUENESS: duplicates on the business key, not the whole row.
        "duplicate_keys": int(df.duplicated(subset=["subscriber_id"]).sum()),

        # VALIDITY: values inside their declared domain.
        "invalid_plan": int((~df["plan"].isin(["free", "pro", "premium"])).sum()),
        "negative_revenue": int((df["revenue"] < 0).sum()),

        # CONSISTENCY: cross-field rules that must hold jointly.
        "end_before_start": int((df["end_date"] < df["start_date"]).sum()),

        # TIMELINESS: how stale is the freshest record?
        "lag_hours": float((now - df["event_at"].max()).total_seconds() / 3600),
    }

report = quality_report(clean)
# Wire this into the pipeline so it runs on every load, not once by hand.
# A quality check that only runs during development tests nothing.

Key Takeaways

• Data quality is not one property but several that fail independently and need separate checks.
• Valid data can still be inconsistent, and complete data can still be inaccurate.
• Timeliness makes otherwise-perfect data worthless when it misses the decision window.
• Quality checks belong in the pipeline, running on every load, not in a one-off notebook.

🧪 Practice

1. Write a quality report function for a dataset you use, covering at least five of the seven dimensions.
2. Construct an example table that scores perfectly on completeness and validity while being badly inconsistent.
3. Interview: A stakeholder says the data is "clean" because there are no nulls. What do you ask next? (Hint: completeness is one dimension of seven, and nulls can be hidden behind sentinel values.)

<a id="14-framing-analytical-questions"></a>

1.4 Framing Analytical Questions

This section returns to framing with more precision, covering how to classify the type of question being asked, turn business goals into measurable quantities, define what success means before starting, and judge whether the project is possible at all.

Descriptive, Diagnostic, Predictive, Prescriptive Analytics

Analytical questions come in four kinds, and they form a ladder of increasing difficulty and increasing value. Misidentifying which rung a question sits on is a reliable way to under-deliver, because each rung demands strictly more than the one below.

Descriptive analytics answers "what happened?" It summarizes recorded history. Revenue last quarter, churn by segment, page views by hour.

Diagnostic analytics answers "why did it happen?" It requires isolating causes among correlated candidates, which is substantially harder than summarizing, because many explanations fit the same summary.

Predictive analytics answers "what will happen?" It requires that past patterns continue to hold, which is an assumption, not a guarantee.

Prescriptive analytics answers "what should we do?" It requires predicting outcomes under actions not yet taken, which means it needs causal knowledge, not just correlational prediction. This is why it is the hardest rung and why most organizations that claim to be there are not.

 THE ANALYTICS LADDER

  value    ^
           |                                    +----------------+
           |                                    | PRESCRIPTIVE   |
           |                                    | "what should   |
           |                                    |  we do?"       |
           |                   +----------------+  needs CAUSAL  |
           |                   | PREDICTIVE     |  models        |
           |                   | "what will     |                |
           |                   |  happen?"      |                |
           |   +---------------+ needs stable   |                |
           |   | DIAGNOSTIC    |  patterns      |                |
           |   | "why did it   |                |                |
           |   |  happen?"     |                |                |
           |   | needs causal  |                |                |
   +-------+---+  reasoning    |                |                |
   |DESCRIPTIVE|               |                |                |
   |"what      |               |                |                |
   | happened?"|               |                |                |
   +-----------+---------------+----------------+----------------+
                                                       difficulty ->

 A correlational model can climb to PREDICTIVE and no further.

# The same churn dataset, one question per rung. Note how the required
# machinery escalates at each step.

# DESCRIPTIVE: summarize what occurred.
churn_by_plan = df.groupby("plan")["churned"].mean()
# free: 0.22, pro: 0.09, premium: 0.04

# DIAGNOSTIC: why is free-tier churn high? Candidate explanations are
# confounded -- free users differ in tenure, usage, and acquisition channel.
by_plan_tenure = df.groupby(["plan", pd.qcut(df["tenure_days"], 4)])["churned"].mean()
# If free-tier churn is high in EVERY tenure bucket, tenure does not explain it.

# PREDICTIVE: who will churn next month?
risk = model.predict_proba(X_current)[:, 1]

# PRESCRIPTIVE: who should receive a discount? This is NOT "the highest
# risk users". The right target is users whose churn probability CHANGES
# when they receive the offer -- an uplift, which needs an experiment.
#   uplift(user) = P(stay | offer) - P(stay | no offer)
# A high-risk user who leaves regardless is wasted budget, and a user who
# would have stayed anyway is wasted budget too.

That final comment is the central lesson of the ladder. Ranking by predicted risk and ranking by predicted benefit of an action are different rankings, and only the second answers a prescriptive question.

Key Takeaways

• The four rungs demand strictly increasing evidence: summary, cause, extrapolation, then causal effect of an action.
• Correlational prediction cannot answer prescriptive questions, however accurate it is.
• Targeting by predicted risk wastes effort on cases where the action changes nothing.
• Identify the rung before promising a deliverable, because the data requirements differ sharply.

🧪 Practice

1. Write one question at each rung for a hospital's readmission data, and state what evidence each requires.
2. Explain why a perfectly accurate churn model still cannot tell you who to give a discount to.
3. Interview: A client asks for a model to "optimize pricing." Which rung is that, and what does it require beyond historical sales? (Hint: you need to know demand at prices you have never charged.)

Translating Business Goals into Metrics

Business goals are stated in language that cannot be computed: "improve customer experience", "grow the business", "reduce risk". Translation into a metric is an act of interpretation, and the interpretation chosen determines what the organization will actually optimize.

This is consequential because metrics are incentives. Once a number is tracked, people and models steer toward it, including in ways nobody intended. Goodhart's law states the result: when a measure becomes a target, it ceases to be a good measure.

A sound translation follows a chain: goal, then the behaviour that would constitute progress, then an observable quantity, then a specific metric definition, then the guardrails that catch the perverse ways of moving it.

 GOAL -> METRIC TRANSLATION, WITH THE FAILURE AT EACH SHORTCUT

 Goal:        "Improve customer support"
                     |
 Behaviour:   "Customers get their problem solved quickly"
                     |
                     +-- shortcut -> metric: "average handle time"
                     |               gamed by: rushing, closing early,
                     |               transferring. Handle time falls,
                     |               problems go unsolved.
                     |
 Quantity:    "Share of contacts resolved without a repeat contact
               within 7 days, and time to that resolution"
                     |
 Metric:      first_contact_resolution_rate = 
                 contacts with no repeat within 7d / all contacts
                     |
 Guardrails:  csat, repeat-contact rate, escalation rate,
              agent attrition

Stated goal	Naive metric	How it gets gamed	Better metric plus guardrail
Improve support	Average handle time	Close tickets unresolved	First-contact resolution + CSAT
Grow engagement	Sessions per user	Notification spam	Weekly retained users + unsubscribes
Increase revenue	Revenue this month	Discount the future away	Contribution margin + repeat rate
Reduce fraud loss	Fraud caught	Block many legitimate users	Fraud loss + false-decline rate
Improve model	Accuracy	Predict the majority class	Cost-weighted error + worst subgroup

# A metric definition is code, not a sentence. Ambiguity here becomes
# two teams reporting different numbers for the "same" metric.

def first_contact_resolution(contacts: pd.DataFrame, window_days: int = 7):
    """Share of contacts with no follow-up from the same customer in `window`.

    Decisions encoded here that a one-line definition would hide:
      - the window is 7 days, not 24 hours or 30 days
      - a repeat on a DIFFERENT topic still counts as unresolved
      - contacts in the last `window` days are excluded, because their
        outcome is not yet observable (this is the censoring rule)
    """
    cutoff = contacts["created_at"].max() - pd.Timedelta(days=window_days)
    eligible = contacts[contacts["created_at"] <= cutoff].copy()

    nxt = (contacts.sort_values("created_at")
                   .groupby("customer_id")["created_at"].shift(-1))
    eligible["next_contact"] = nxt.loc[eligible.index]

    repeated = (eligible["next_contact"] - eligible["created_at"]
                ) <= pd.Timedelta(days=window_days)
    return 1 - repeated.fillna(False).mean()

Key Takeaways

• Translating a goal into a metric is interpretation, and the chosen interpretation becomes what the organization optimizes.
• Every metric will be gamed, so pair each primary metric with guardrails that detect the perverse route.
• Metric definitions belong in code with their edge cases documented, not in a sentence.
• Censoring rules matter: recent records whose outcome is unobservable must be excluded, not counted as successes.

🧪 Practice

1. Translate "make the product easier to learn" into a primary metric and two guardrails.
2. Take a metric your team reports and write down three ways it could be improved without improving the underlying goal.
3. Interview: A team hits its target of 30% more monthly active users, but revenue is flat. What likely happened? (Hint: examine how "active" was defined and who the new actives are.)

Defining Success Criteria

Before building anything, the team should be able to state the number that would make the project a success and the number that would make it a failure. Without this, evaluation becomes negotiation: whatever the model achieves gets described as a win, and projects that should be stopped continue for quarters.

Success criteria have three parts that are routinely conflated. Model criteria concern statistical performance. Business criteria concern the decision outcome. Operational criteria concern whether the thing can run at all. A project can pass one and fail another, and usually does.

The criteria must be set against a baseline, be measurable with data that will actually exist, and include a stopping rule. The stopping rule is the uncomfortable one, and it is the one that prevents sunk-cost projects.

 THREE LAYERS OF SUCCESS CRITERIA

 +----------------------------------------------------------+
 | OPERATIONAL   scores available by the 3rd of each month   |
 |               inference under 200ms at p99                |
 |               retrainable by one engineer in a day        |
 +----------------------------------------------------------+
 | BUSINESS      win-back campaign hit rate 11% -> 18%       |
 |               incremental margin > cost of the offers     |
 |               no increase in unsubscribe rate             |
 +----------------------------------------------------------+
 | MODEL         AUC >= 0.78 on a time-held-out quarter      |
 |               calibrated: predicted 10% means ~10% actual |
 |               no subgroup with AUC below 0.70             |
 +----------------------------------------------------------+
 | STOP IF       AUC < 0.70 after two iterations, OR         |
 |               required features cannot be served in time  |
 +----------------------------------------------------------+

 Agreed in writing, before modelling, by the team that will act on it.

# Encode the criteria as an executable check. A criterion that cannot be
# evaluated by a function is not specific enough to have been agreed on.

SUCCESS = {
    "auc_min": 0.78,
    "calibration_max_error": 0.05,   # max |predicted - observed| by decile
    "min_subgroup_auc": 0.70,
    "max_scoring_latency_s": 3600 * 12,
    "min_campaign_hit_rate": 0.18,   # measured in the live experiment
}

import operator

# (label, key in `metrics`, comparison, threshold)
CHECKS = [
    ("AUC",         "auc",         operator.ge, SUCCESS["auc_min"]),
    ("calibration", "cal_error",   operator.le, SUCCESS["calibration_max_error"]),
    ("subgroup AUC","min_sub_auc", operator.ge, SUCCESS["min_subgroup_auc"]),
]

def evaluate_criteria(metrics: dict) -> tuple[bool, list[str]]:
    failures = []
    for label, key, compare, threshold in CHECKS:
        value = metrics[key]
        if not compare(value, threshold):        # direction differs per check
            sign = ">=" if compare is operator.ge else "<="
            failures.append(f"{label}: {value:.3f}, required {sign} {threshold}")
    return (not failures), failures

passed, failures = evaluate_criteria(measured)
print("SHIP" if passed else "DO NOT SHIP:\n  " + "\n  ".join(failures))

Key Takeaways

• Write success criteria before building, or evaluation degenerates into negotiation after the fact.
• Separate model, business, and operational criteria; passing one says nothing about the others.
• Every criterion needs a baseline, otherwise the threshold is arbitrary.
• Include an explicit stopping rule, which is the only defence against sunk-cost continuation.

🧪 Practice

1. Write the three layers of success criteria for a delivery-time prediction feature, including a stopping rule.
2. Take a past project and write the criteria you wish had been agreed at the start. Note whether it would have passed.
3. Interview: A stakeholder refuses to name a threshold, saying "just make it as accurate as possible." How do you proceed? (Hint: ask what they will do differently at 70% versus 90%.)

Feasibility and Data Availability Assessment

The last step before committing is checking whether the project can be done at all. This assessment is cheap, takes days rather than months, and is the single highest-return activity in the lifecycle, because the alternative is discovering the blocker after a quarter of work.

Four things get checked. Data existence: does a record of the outcome and the predictors exist, at the right grain, over enough history? Data access: can you legally and practically obtain it, and will it be there at prediction time? Signal plausibility: is there reason to believe the predictors relate to the outcome? Organizational readiness: is there a decision waiting, an owner, and a path to acting on the output?

The most frequently missed check is availability at prediction time. A feature that exists in the warehouse but is only populated after the outcome occurs is a leak, not a feature. Training accuracy will be superb and production accuracy will collapse.

 THE TIMELINE TEST FOR EVERY CANDIDATE FEATURE

                 scoring        outcome
                 moment         observed
   ----+------------|--------------|--------->  time
       |            |              |
  feature known     |              |
  BEFORE scoring    |              |
  => LEGAL          |              |
                    |              |
              feature populated    |
              AFTER scoring        |
              => LEAK. Remove it.  |

 Example leak: "number of support tickets about cancellation" is
 recorded before the churn label but after the scoring date, so it is
 unavailable when the model must actually run.

Check	Question	Cheap way to test it
Outcome exists	Is the label recorded, unambiguously?	Count labelled rows per month
Sufficient history	Enough events, covering enough seasons?	Events per period, minority count
Grain matches	Does the data's unit match the decision's?	Compare keys to the framing doc
Available at inference	Is each feature known before scoring?	Timeline test per feature
Legal and ethical	May it be used for this purpose?	Review with legal and privacy
Signal plausible	Any relationship at all in a quick check?	Simple model on a sample
Decision ready	Is someone waiting to act on the output?	Name the owner and the campaign date

# A one-day feasibility spike. If this fails, do not start the project.

# 1. Label volume: rare outcomes need far more rows than people expect.
labels = df.groupby(df["event_month"])["churned"].agg(["size", "sum"])
print(labels.tail(12))
# Rule of thumb: fewer than a few hundred minority events total means the
# project is a data-collection problem before it is a modelling problem.

# 2. Timeline test: every feature must predate the scoring moment.
for col, populated_at in FEATURE_TIMESTAMPS.items():
    if (populated_at > SCORING_MOMENT).any():
        print(f"LEAK RISK: {col} is sometimes populated after scoring")

# 3. Signal check: a quick model on a sample. Not to be accurate, but to
# distinguish "no signal" from "needs work".
quick = HistGradientBoostingClassifier(max_iter=60).fit(X_s, y_s)
print(f"sample AUC = {roc_auc_score(y_hold, quick.predict_proba(X_hold)[:,1]):.3f}")
# ~0.50 after honest feature work suggests the predictors are unrelated
# to the label as defined -- revisit the framing, not the algorithm.

Key Takeaways

• A feasibility spike costs days and routinely prevents quarters of wasted work.
• Check availability at prediction time, not just presence in the warehouse; the difference is a leak.
• Rare outcomes turn modelling problems into data-collection problems.
• Organizational readiness is a feasibility criterion: without a waiting decision and a named owner, the output will not be used.

🧪 Practice

1. Run the timeline test on five features from a project you know and flag any that could not be computed at scoring time.
2. Write a feasibility checklist for predicting equipment failure in a factory with two years of sensor data and nine recorded failures.
3. Interview: A stakeholder wants to predict which employees will be high performers before hiring. Assess feasibility. (Hint: consider what the label really measures and who is missing from the training data.)

---

<a id="2-mathematical-foundations"></a>