4. Programming and Tooling

The previous chapters established what to compute and why; this one covers the machinery that actually does it, from Python's array and dataframe libraries through R, SQL, and the environment practices that make results reproducible. The emphasis throughout is on the mental models that prevent the specific bugs these tools invite, because a wrong number produced quickly is worse than no number at all.

<a id="41-python-for-data-science"></a>

4.1 Python for Data Science

Python is the default language of data work not because it is fast but because its numerical stack lets you push the slow parts down into compiled code, and this section builds from the language core up through NumPy, pandas, and SciPy with that division of labour always in view.

Core Syntax and Data Structures

Before reaching for a dataframe, it is worth knowing what the language underneath offers, because half of the confusing behaviour in pandas is inherited Python semantics rather than anything pandas invented.

Python's central design choice is that variables are names bound to objects, not boxes containing values. Assignment never copies. When you write b = a, you create a second label pointing at the same object. If that object can be changed in place, both labels see the change. This single fact explains a large share of beginner bugs.

Objects divide into mutable and immutable. An immutable object cannot be altered after creation: integers, floats, strings, and tuples. Any operation that appears to modify one actually builds a new object. Mutable objects, principally lists, dictionaries, and sets, can be altered in place, so aliasing becomes observable.

Think of it as the difference between a printed page and a whiteboard. Two people holding references to the same printed page can never surprise each other. Two people with a marker at the same whiteboard can.

The four workhorse collections each answer a different question:

Type	Ordered	Mutable	Lookup cost	Use when
list	yes	yes	O(n) by value	a sequence you will append to or index
tuple	yes	no	O(n) by value	a fixed record, or a dict key
dict	insertion	yes	O(1) by key	a mapping from key to value
set	no	yes	O(1) membership	deduplication, membership tests

The cost column is the one that matters at scale. Checking x in some_list scans every element; the same check against a set is a single hash lookup. Rewriting one membership test can turn a quadratic loop into a linear one.

# NAMES AND OBJECTS: assignment binds, it does not copy.
a = [1, 2, 3]
b = a                      # b and a label the SAME list object
b.append(4)
print(a)                   # [1, 2, 3, 4] -- a changed too
print(a is b)              # True: identical object, not merely equal

c = a.copy()               # a shallow copy: a NEW list, same elements
c.append(5)
print(a, c)                # [1,2,3,4] [1,2,3,4,5] -- now independent

# IMMUTABILITY: the same pattern with tuples cannot bite you.
t = (1, 2, 3)
u = t
# u.append(4)              # AttributeError: tuples have no append
u = u + (4,)               # builds a NEW tuple; t is untouched
print(t, u)                # (1, 2, 3) (1, 2, 3, 4)

# THE CLASSIC TRAP: a mutable default argument is created ONCE,
# at function definition time, and then shared by every call.
def bad_append(item, target=[]):        # do NOT do this
    target.append(item)
    return target

print(bad_append("a"))     # ['a']
print(bad_append("b"))     # ['a', 'b'] -- the list persisted between calls

def good_append(item, target=None):     # the correct idiom
    if target is None:
        target = []                     # a fresh list per call
    target.append(item)
    return target

print(good_append("a"), good_append("b"))   # ['a'] ['b']

# LOOKUP COST: the difference is asymptotic, not cosmetic.
import time
haystack_list = list(range(200_000))
haystack_set = set(haystack_list)
needles = range(0, 200_000, 1000)

t0 = time.perf_counter()
sum(1 for n in needles if n in haystack_list)    # O(n) per lookup
list_time = time.perf_counter() - t0

t0 = time.perf_counter()
sum(1 for n in needles if n in haystack_set)     # O(1) per lookup
set_time = time.perf_counter() - t0

print(f"list membership: {list_time*1000:7.2f} ms")
print(f"set membership:  {set_time*1000:7.2f} ms")
# Typically a factor of several hundred. Same logic, different container.

  NAMES, OBJECTS, AND WHAT COPYING MEANS

  a = [1, [2, 3]]

     a ----> [ * , * ]
               |   |
               v   v
               1  [2, 3]

  b = a                 b ----> the SAME outer list
  c = a.copy()          c ----> a NEW outer list, but the inner
                                list is still SHARED
  d = copy.deepcopy(a)  d ----> new outer AND new inner

     shallow copy:            deep copy:
     c --> [ * , * ]          d --> [ * , * ]
                 |                        |
                 v                        v
     a --> [ * , * ]          a --> [ * , * ]
                              (inner objects duplicated)

  Mutating a[1] is visible through c but not through d.

Key Takeaways

• Assignment binds a name to an object and never copies; is tests identity while == tests equality.
• Mutable objects (list, dict, set) make aliasing observable; immutable ones (int, str, tuple) cannot surprise you this way.
• A mutable default argument is evaluated once at definition time and shared across all calls; use None as the sentinel instead.
• Choose the container by its lookup cost: sets and dicts give O(1) membership, lists give O(n).
• A shallow copy duplicates the outer container only; nested mutable objects remain shared.

🧪 Practice

1. Write a function that removes duplicates from a list while preserving order, using a set for the membership test, and confirm it is faster than the list-based version on 100,000 elements.
2. Demonstrate the shallow-copy trap: build a list of lists, copy it, mutate an inner list, and show the change is visible in both.
3. Interview: A colleague's caching function returns stale results on the second call but is correct on the first. What do you look at first? (Hint: when is a default argument evaluated?)

Functions and Comprehensions

Functions in Python carry more machinery than their syntax suggests, and comprehensions are the idiom that replaces most of the loops an analyst would otherwise write. Both reward a precise understanding.

The argument system has four layers, and they are always evaluated in the same order: positional parameters, *args for extra positionals, keyword parameters with defaults, and **kwargs for extra keywords. A bare * in the signature forces everything after it to be keyword-only, which is the single best readability investment you can make in a function that takes several flags.

Scoping follows the LEGB rule: a name is resolved in the Local scope, then any Enclosing function scopes, then Global (module) scope, then Builtins. Assignment inside a function creates a local name unless declared otherwise, which is why reading then writing a module-level counter raises an error rather than doing what you meant.

A comprehension is a declarative rewrite of the accumulate-in-a-loop pattern. Instead of describing the steps, you describe the result. This is not only shorter: the interpreter does not have to resolve an append attribute on every iteration, so it is genuinely faster.

The readability rule is worth stating as a rule. A comprehension should fit the shape [transform(x) for x in source if condition(x)]. Once you need two nested loops plus a conditional plus a function call, a real loop is clearer and no slower.

Closures arise when an inner function refers to a variable from the enclosing scope. The inner function keeps that binding alive after the outer function returns. This is the mechanism behind decorators, and also behind the late-binding surprise that catches everyone once.

from functools import reduce

# ARGUMENT FORMS, in signature order.
def summarize(data, *extra, method="mean", digits=2, **options):
    """positional, var-positional, keyword-with-default, var-keyword."""
    return f"{method} of {len(data) + len(extra)} values to {digits} dp"

print(summarize([1, 2, 3], 4, 5, method="median", trim=0.1))

# KEYWORD-ONLY arguments: everything after the bare * must be named.
def scale(values, *, factor=1.0, clip=None):
    out = [v * factor for v in values]
    return [min(v, clip) for v in out] if clip is not None else out

print(scale([1, 2, 3], factor=10, clip=25))     # [10, 20, 25]
# scale([1,2,3], 10)  -> TypeError. The call site is forced to be explicit.

# SCOPING: assignment makes a name local for the WHOLE function body.
counter = 0
def broken():
    # print(counter)      # UnboundLocalError: counter is local here
    counter = 1           # ...because of THIS line, below the read
    return counter

def fixed():
    global counter        # explicit, and usually a smell worth avoiding
    counter += 1
    return counter

# COMPREHENSIONS: the four flavours.
nums = range(10)
squares      = [n ** 2 for n in nums]                  # list
even_squares = [n ** 2 for n in nums if n % 2 == 0]    # with a filter
lookup       = {n: n ** 2 for n in nums}               # dict
distinct     = {n % 3 for n in nums}                   # set

# A conditional EXPRESSION goes before the `for`; a FILTER goes after.
labelled = ["even" if n % 2 == 0 else "odd" for n in nums]

# Nested loops read left to right, exactly like the equivalent for-loops.
pairs = [(i, j) for i in range(3) for j in range(3) if i != j]

# LATE BINDING: the classic closure surprise.
makers = [lambda: i for i in range(3)]
print([m() for m in makers])          # [2, 2, 2] -- all see the FINAL i

makers = [lambda i=i: i for i in range(3)]   # bind at definition time
print([m() for m in makers])          # [0, 1, 2]

# A CLOSURE that does something useful: a parameterized transformer.
def make_clipper(low, high):
    def clip(x):
        return max(low, min(high, x))     # low/high captured from enclosing
    return clip

clip_pct = make_clipper(0, 100)
print([clip_pct(x) for x in (-10, 50, 300)])    # [0, 50, 100]

# reduce is rarely the clearest option; prefer sum/max/any or a loop.
print(reduce(lambda a, b: a * b, [1, 2, 3, 4]))  # 24

Key Takeaways

• Signature order is positional, *args, keyword defaults, **kwargs; a bare * makes the remaining parameters keyword-only.
• Name resolution follows Local, Enclosing, Global, Builtins, and any assignment in a function body makes that name local throughout it.
• Comprehensions state the result rather than the steps and avoid per-iteration attribute lookup, so they are both clearer and faster.
• A conditional expression goes before the for; a filter goes after it.
• Closures capture the variable, not its value; use a default argument to bind at definition time.

🧪 Practice

1. Rewrite a nested loop that builds a list of filtered, transformed values as a single comprehension, then time both on 100,000 items.
2. Write a decorator that times any function it wraps and prints the elapsed milliseconds, using a closure over the wrapped function.
3. Interview: Why does a list of lambdas built in a loop often return the same value from every element? (Hint: what exactly does the lambda hold a reference to?)

Object-Oriented Basics

Most analysis code is functions over dataframes, and that is usually correct. Classes earn their place when you have state and behaviour that travel together, which in data work means transformers, model wrappers, and configuration objects.

A class is a template; an instance is a concrete object built from it. The __init__ method runs at construction and sets up instance state. Attributes assigned on self belong to the instance; attributes assigned in the class body are shared by all instances, which is the class-level analogue of the mutable-default trap.

Three method kinds exist and the distinction is practical. An instance method takes self and acts on one object. A classmethod takes cls and is the idiomatic way to write alternative constructors. A staticmethod takes neither and is really a plain function that lives in the class namespace for organisational reasons.

Dunder methods let your objects participate in Python's syntax. Defining __len__ makes len(obj) work, __getitem__ makes obj[k] work, and __repr__ controls what you see in a notebook. This is how pandas makes a DataFrame feel like a built-in type.

For the common case of a class that only holds data, @dataclass generates the constructor, the repr, and the equality method for you. Use it: it removes the most error-prone boilerplate in the language.

The scikit-learn estimator interface is the canonical example of why this matters in data science. Any object with fit and transform drops into a pipeline, so conforming to the protocol buys you composability for free.

from dataclasses import dataclass, field
import numpy as np

# A DATACLASS for pure configuration: no boilerplate, correct equality.
@dataclass(frozen=True)          # frozen -> immutable and hashable
class SplitConfig:
    test_size: float = 0.2
    seed: int = 0
    stratify: bool = True

cfg = SplitConfig(test_size=0.3)
print(cfg)                        # SplitConfig(test_size=0.3, seed=0, ...)
print(cfg == SplitConfig(test_size=0.3))    # True: value equality, free

# THE CLASS-ATTRIBUTE TRAP, the exact analogue of mutable defaults.
class BadLogger:
    entries = []                  # SHARED by every instance
class GoodLogger:
    def __init__(self):
        self.entries = []         # one per instance

a, b = BadLogger(), BadLogger()
a.entries.append("x")
print(b.entries)                  # ['x'] -- b sees a's entry

# A TRANSFORMER following the scikit-learn protocol.
class StandardScaler:
    """Centre and scale features. fit learns state, transform applies it."""

    def __init__(self, with_mean: bool = True):
        self.with_mean = with_mean          # a hyperparameter
        self.mean_ = None                   # trailing _ = learned at fit
        self.scale_ = None

    def fit(self, X):
        X = np.asarray(X, dtype=float)
        self.mean_ = X.mean(axis=0) if self.with_mean else np.zeros(X.shape[1])
        self.scale_ = X.std(axis=0)
        self.scale_[self.scale_ == 0] = 1.0     # never divide by zero
        return self                              # enables fit().transform()

    def transform(self, X):
        if self.mean_ is None:
            raise RuntimeError("call fit before transform")
        return (np.asarray(X, dtype=float) - self.mean_) / self.scale_

    def fit_transform(self, X):
        return self.fit(X).transform(X)

    # DUNDERS: make the object behave like a native one.
    def __repr__(self):
        state = "fitted" if self.mean_ is not None else "unfitted"
        return f"StandardScaler(with_mean={self.with_mean}, {state})"

X = np.array([[1.0, 100.0], [2.0, 200.0], [3.0, 300.0]])
sc = StandardScaler()
print(sc.fit_transform(X).round(3))
print(sc)                          # StandardScaler(with_mean=True, fitted)

# CLASSMETHOD as an alternative constructor.
class Dataset:
    def __init__(self, rows):
        self.rows = rows

    @classmethod
    def from_csv_text(cls, text):        # cls, not the hard-coded name
        rows = [line.split(",") for line in text.strip().splitlines()]
        return cls(rows)

    @staticmethod
    def is_numeric(token):               # no self, no cls: a helper
        return token.replace(".", "", 1).isdigit()

    def __len__(self):
        return len(self.rows)            # now len(ds) works

ds = Dataset.from_csv_text("1,2\n3,4")
print(len(ds), Dataset.is_numeric("3.5"))    # 2 True

Key Takeaways

• Attributes set on self are per instance; attributes in the class body are shared, which mirrors the mutable-default trap.
• Use instance methods for object behaviour, classmethods for alternative constructors, and staticmethods for namespaced helpers.
• Dunder methods integrate your objects with Python syntax; __repr__ is the highest-value one for interactive work.
• @dataclass removes constructor, repr, and equality boilerplate; frozen=True gives immutability and hashability.
• Conforming to the fit/transform protocol makes a class composable with the wider scikit-learn ecosystem.

🧪 Practice

1. Write a MinMaxScaler class with fit, transform, and fit_transform, storing learned bounds in attributes ending in _.
2. Convert a configuration dictionary used in a script into a frozen dataclass and show that two equal configs compare as equal.
3. Interview: Why does scikit-learn insist that hyperparameters be set in __init__ and learned values only in fit? (Hint: think about what clone and cross-validation need to do with an estimator.)

Iterators, Generators, and Lazy Evaluation

The defining constraint of real data work is that the file is bigger than memory. Lazy evaluation is the answer, and in Python it is built directly into the language through the iterator protocol.

An iterable is anything that can produce an iterator, via __iter__. An iterator is an object with __next__ that yields the next item and raises StopIteration when exhausted. A for loop is syntax sugar over exactly this protocol. Crucially, an iterator is consumed once: after you walk it, it is empty.

A generator function is any function containing yield. Calling it does not execute the body; it returns a generator object. Each next() runs the body until the next yield, then freezes the entire local state and returns. The function resumes exactly where it stopped.

The analogy is a book versus a storyteller. A list is a printed book: every page exists, and the whole thing occupies shelf space. A generator is a storyteller producing one sentence when asked. You cannot flip back, and you cannot know the length in advance, but you can hear a story far longer than any book you could carry.

The practical payoff is constant memory. A generator pipeline that reads, filters, parses, and aggregates ten million rows holds one row at a time. The same pipeline built from lists holds ten million rows four times over.

The cost is real and worth naming. Generators are single-pass, have no length, cannot be indexed, and produce confusing bugs when accidentally consumed twice. If you need the data more than once, materialize it.

import sys
from itertools import islice, chain, groupby, count

# A GENERATOR FUNCTION: the body runs lazily, one yield at a time.
def read_records(lines):
    for line in lines:
        line = line.strip()
        if not line or line.startswith("#"):      # skip blanks and comments
            continue
        yield line.split(",")                     # freeze here, resume later

# MEMORY: the difference is not marginal.
list_version = [n ** 2 for n in range(1_000_000)]
gen_version = (n ** 2 for n in range(1_000_000))   # parentheses, not brackets
print(f"list: {sys.getsizeof(list_version):>10,} bytes")
print(f"gen:  {sys.getsizeof(gen_version):>10,} bytes")
# ~8,000,000 vs ~200. The generator stores a recipe, not results.

# SINGLE-PASS: the trap that catches everyone once.
g = (n for n in range(3))
print(sum(g))        # 3
print(sum(g))        # 0 -- already exhausted, and NO error is raised

# A PIPELINE: each stage is lazy, so nothing is materialized.
raw = (f"2024-01-{d:02d},{d * 3},{'eu' if d % 2 else 'us'}" for d in range(1, 32))
parsed = (r.split(",") for r in raw)                  # stage 1
eu_only = (r for r in parsed if r[2] == "eu")         # stage 2
amounts = (float(r[1]) for r in eu_only)              # stage 3
print(sum(amounts))     # one row in memory at any moment

# yield from DELEGATES to a sub-generator, flattening nesting.
def walk(nested):
    for item in nested:
        if isinstance(item, list):
            yield from walk(item)      # recurse without manual re-yielding
        else:
            yield item

print(list(walk([1, [2, [3, 4]], 5])))     # [1, 2, 3, 4, 5]

# ITERTOOLS: the standard toolkit for lazy sequences.
print(list(islice(count(10), 5)))              # [10,11,12,13,14] from infinite
print(list(chain([1, 2], [3, 4])))             # [1, 2, 3, 4]

rows = [("eu", 1), ("eu", 2), ("us", 3)]       # groupby needs SORTED input
for key, group in groupby(rows, key=lambda r: r[0]):
    print(key, [r[1] for r in group])          # eu [1, 2] / us [3]

# CHUNKING a stream into batches, the most reused generator in data work.
def chunked(iterable, size):
    it = iter(iterable)
    while batch := list(islice(it, size)):     # walrus: assign and test
        yield batch

print([len(b) for b in chunked(range(10), 3)])    # [3, 3, 3, 1]

  EAGER VS LAZY, SAME PIPELINE

  EAGER (lists)                    LAZY (generators)

  read all 10M rows                pull row 1 ------+
        |  10M in RAM                    filter     |
        v                                 parse     | repeat
  filter -> 2M rows                      accumulate |  10M
        |  2M more in RAM                           |  times
        v                                pull row 2 +
  parse -> 2M objects
        |                          peak memory: ONE row
        v                          total allocations: 10M small
  sum                              (garbage collected immediately)

  peak memory: ~14M rows held simultaneously

Key Takeaways

• An iterator yields items one at a time and is exhausted after a single pass, silently returning nothing thereafter.
• A generator function freezes its entire local state at each yield and resumes there on the next request.
• Generator pipelines hold one item at a time regardless of input size, which is what makes larger-than-memory processing possible.
• Generators have no length, no indexing, and no second pass; materialize with list() when you need any of those.
• itertools plus yield from covers most lazy-sequence needs without custom code.

🧪 Practice

1. Write a generator that reads a large CSV line by line and yields only rows whose third field exceeds a threshold, then confirm memory usage stays flat as the file grows.
2. Implement a sliding_window(iterable, n) generator that yields overlapping tuples of length n.
3. Interview: A colleague's function returns a generator, and the caller computes both its sum and its maximum, getting a maximum of negative infinity. What happened? (Hint: how many passes does the data support?)

Virtual Environments and Package Management

An analysis that runs today and fails next month usually has nothing wrong with its code. The environment changed underneath it. Dependency management is therefore not administrative overhead; it is part of the result.

The problem is structural. Python installs packages into a single site-packages directory per interpreter, and only one version of a library can live there. Project A needing pandas 1.5 and project B needing pandas 2.2 are in direct conflict. A virtual environment resolves this by giving each project its own interpreter and its own site-packages.

Two kinds of specification exist and conflating them causes most of the confusion. An abstract dependency is what your code needs, expressed as a range such as pandas>=2.0. A concrete dependency is what you actually installed, pinned exactly, including transitive dependencies you never named. Applications should commit a lockfile of the concrete set; libraries should publish only the abstract ranges.

Think of it as a recipe versus a shopping receipt. The recipe says "flour", which lets a reader substitute. The receipt records the exact brand and batch that produced the cake you actually ate. Reproducing a specific result needs the receipt.

The tooling landscape has consolidated. venv plus pip is built in and sufficient. conda handles non-Python binaries such as CUDA or GDAL, which matters in geospatial and deep learning work. uv is a fast modern resolver that reads the standard pyproject.toml. Choose one per project and record the choice.

# VENV: built in, no installation, one directory per project.
python -m venv .venv                  # create; .venv belongs in .gitignore
source .venv/bin/activate             # Windows: .venv\Scripts\activate
python -m pip install --upgrade pip

# Install ABSTRACT requirements, then freeze the CONCRETE result.
python -m pip install "pandas>=2.0" "scikit-learn>=1.3"
python -m pip freeze > requirements.lock.txt    # every transitive pin

# Reproduce that exact environment elsewhere.
python -m pip install -r requirements.lock.txt

# Verify which interpreter you are actually using. Do this when confused;
# "it works in the terminal but not the notebook" is nearly always this.
python -c "import sys; print(sys.executable)"
python -c "import pandas; print(pandas.__version__, pandas.__file__)"

# UV: the same workflow, resolved in seconds rather than minutes.
uv venv
uv pip install pandas scikit-learn
uv pip compile pyproject.toml -o requirements.lock.txt
uv sync                               # install exactly the lockfile

# pyproject.toml -- the standard, tool-agnostic project declaration.
[project]
name = "churn-analysis"
version = "0.1.0"
requires-python = ">=3.11"            # pin the INTERPRETER too
dependencies = [                       # ABSTRACT: ranges, not pins
    "pandas>=2.0,<3.0",
    "scikit-learn>=1.3",
    "pyarrow>=14.0",
]

[project.optional-dependencies]
dev = ["pytest>=8.0", "ruff>=0.5", "jupyterlab>=4.0"]

  WHY ISOLATION IS NOT OPTIONAL

  WITHOUT VENV                      WITH VENV

  system python 3.11                project-a/.venv -> pandas 1.5
     site-packages/                 project-b/.venv -> pandas 2.2
       pandas 2.2   <-- one slot    system python   -> untouched
       numpy  1.26

  project A needs pandas 1.5        each project resolves
  project B needs pandas 2.2        independently; upgrading one
  -> one of them is broken          cannot break the other

  ABSTRACT vs CONCRETE

  pyproject.toml            requirements.lock.txt
  pandas>=2.0,<3.0     ->   pandas==2.2.1
  (what the code needs)     numpy==1.26.4      <- never named by you
                            python-dateutil==2.9.0
                            pytz==2024.1
                            (what actually ran)

Key Takeaways

• One interpreter holds one version of each package, so per-project virtual environments are the only way to avoid version conflicts.
• Abstract dependencies are ranges your code needs; concrete lockfiles are the exact set that produced a result, including transitive packages.
• Commit lockfiles for applications and analyses; publish ranges only for libraries.
• Pin the Python version alongside the packages, because interpreter changes break code too.
• When behaviour differs between terminal and notebook, print sys.executable first; it is almost always two environments.

🧪 Practice

1. Create a virtual environment, install two packages, freeze the result, and compare the number of lines in the freeze output to the number you installed explicitly.
2. Write a pyproject.toml with abstract ranges and a dev extra, then generate a lockfile from it.
3. Interview: Your model gives different predictions on a colleague's machine from the same code and data. Name three environment-level causes. (Hint: consider library versions, the interpreter, and threading-dependent numerical libraries.)

Why Vectorization Beats Python Loops

NumPy is the reason Python is viable for numerical work, and understanding why it is fast tells you when it will not be, which is the more useful half of the knowledge.

A Python for loop over a list is slow for reasons that compound. Every element is a full PyObject with a type header and a reference count, so the data is scattered across the heap rather than laid out contiguously. Every arithmetic operation dispatches on runtime types. Every iteration allocates a new object. The interpreter executes bytecode for each step.

A NumPy array removes all four costs at once. The data is a single contiguous block of raw machine values with one dtype for the whole array. The loop over elements happens inside compiled C, with the type resolved once before the loop starts. The result is typically 10 to 100 times faster, and the gap widens with array size because the CPU's cache prefetcher can actually predict the access pattern.

The analogy is a customs queue. Interpreted looping inspects each traveller's passport individually, deciding afresh each time which desk applies. A vectorized operation checks that everyone in the coach has the same nationality once, then waves the whole coach through.

The corollary is the one people miss: np.vectorize and df.apply are not vectorization. They are Python loops wearing an array-shaped coat. The speed comes from the operation executing in C, not from the syntax looking array-like.

Vectorization also changes how you think. Conditional logic becomes np.where, accumulation becomes np.cumsum, and "find the rows where" becomes a boolean mask. Expressing a problem without an explicit loop is a skill that transfers to pandas, SQL, and Spark alike.

import numpy as np
import time

n = 2_000_000
py_list = list(range(n))
np_array = np.arange(n)

def timed(label, fn):
    t0 = time.perf_counter()
    result = fn()
    print(f"{label:<28} {(time.perf_counter() - t0) * 1000:8.1f} ms")
    return result

# 1. PURE PYTHON LOOP: interpreter overhead on every element.
timed("python for-loop", lambda: [x * 2 + 1 for x in py_list])

# 2. np.vectorize: LOOKS vectorized, still loops in Python. No speedup.
vec = np.vectorize(lambda x: x * 2 + 1)
timed("np.vectorize (a loop!)", lambda: vec(np_array[:200_000]))

# 3. TRUE VECTORIZATION: the loop happens inside compiled C.
timed("numpy vectorized", lambda: np_array * 2 + 1)

# WHY: contiguous typed memory versus scattered pointers to objects.
print(f"\nlist of ints  : ~{n * 28 / 1e6:.0f} MB of objects + 8 MB pointers")
print(f"int64 ndarray : {np_array.nbytes / 1e6:.0f} MB, one flat block")

# CONDITIONAL LOGIC without a loop.
temps = np.array([-5.0, 12.0, 23.0, 31.0, 40.0])
label = np.where(temps < 0, "freezing",
         np.where(temps < 25, "mild", "hot"))      # nested, still vectorized
print(label)

# Multi-branch logic reads better with select.
band = np.select(
    [temps < 0, temps < 25, temps < 35],           # conditions, in order
    ["freezing", "mild", "hot"],                   # matching choices
    default="extreme",                             # when nothing matches
)
print(band)

# ACCUMULATION without a loop.
daily = np.array([10, -3, 5, 8, -2])
print(np.cumsum(daily))            # [10  7 12 20 18] running balance
print(np.maximum.accumulate(daily))  # [10 10 10 10 10] running maximum

# WHEN VECTORIZATION DOES NOT APPLY: genuinely sequential recurrences
# where each step depends on the previous result.
# x[t] = 0.9 * x[t-1] + noise[t]  cannot be expressed as one array op.
# Options: scipy.signal.lfilter, numba, or accept the loop.
from scipy.signal import lfilter
noise = np.random.default_rng(0).normal(size=5)
print(lfilter([1.0], [1.0, -0.9], noise).round(3))   # compiled recurrence

  WHY THE SPEED DIFFERENCE IS STRUCTURAL

  PYTHON LIST                       NUMPY ARRAY

  [ptr][ptr][ptr][ptr]              [ 8 bytes ][ 8 bytes ][ 8 bytes ]
    |    |    |    |                 one contiguous block, one dtype
    v    v    v    v
  PyObj PyObj PyObj PyObj
  (28 bytes each, scattered
   anywhere in the heap)

  per element the interpreter:      per ARRAY the C loop:
    1 dereference pointer             1 resolve dtype
    2 check the type                  2 run a tight typed loop
    3 dispatch the operation          3 write into preallocated output
    4 allocate a result object
    5 update reference counts       cache prefetch works because the
                                    next address is predictable
  cache misses dominate

Key Takeaways

• Python loops pay per-element costs for boxing, type dispatch, allocation, and bytecode execution; NumPy pays them once per array.
• Contiguous single-dtype memory lets the CPU prefetch, which is why the speed gap grows with array size.
• np.vectorize and .apply are convenience wrappers around Python loops, not vectorization, and deliver no meaningful speedup.
• Replace conditionals with np.where or np.select and accumulation with cumsum or ufunc.accumulate.
• Genuinely sequential recurrences resist vectorization; reach for a compiled helper such as lfilter or numba instead.

🧪 Practice

1. Time a Python loop, np.vectorize, and a true vectorized expression computing x**2 + 3*x on one million elements, and explain why two of the three are similar.
2. Replace a loop that assigns risk bands by nested if-statements with a single np.select call, and verify the outputs match.
3. Interview: A colleague reports that switching to NumPy made their code slower. What would you check? (Hint: consider array size, and whether the operation is actually running in C.)

The ndarray: Shape, dtype, and Strides

Almost every confusing NumPy error becomes obvious once you can describe an array by its three defining attributes. They are worth learning as a set, because they are what the library actually manipulates.

Shape is a tuple giving the length along each dimension. It carries no information about memory; it is metadata describing how to interpret a flat buffer. (3, 4) means three rows of four. The number of dimensions is the length of that tuple, which NumPy calls ndim.

dtype is the fixed element type shared by every element. This is the trade you make for speed: homogeneity. An int64 array cannot hold a string and cannot hold a missing value, which is why NaN forces a float array.

Strides is the attribute nobody teaches and everybody needs. It gives the number of bytes to step forward in memory to advance by one along each axis. It is the mechanism behind views, transposes, and broadcasting, all of which manipulate strides rather than moving data.

The illuminating image is a long ribbon of numbers with a stencil laid over it. Shape and strides are the stencil. Transposing does not rewrite the ribbon; it swaps the stencil's step sizes. This is why a transpose of a one-gigabyte array is instantaneous.

Two dtype hazards recur. Integer arrays overflow silently rather than promoting to a wider type, which can turn a sum into a negative number. And an array containing Python objects has dtype object, which quietly discards every performance benefit while still looking like an array.

import numpy as np

a = np.arange(12).reshape(3, 4)

print(a.shape)       # (3, 4)     rows, columns
print(a.ndim)        # 2          len(shape)
print(a.size)        # 12         total elements, prod(shape)
print(a.dtype)       # int64      one type for all elements
print(a.itemsize)    # 8          bytes per element
print(a.nbytes)      # 96         itemsize * size
print(a.strides)     # (32, 8)    bytes to step one row / one column

# STRIDES EXPLAINED: one row = 4 elements * 8 bytes = 32 bytes.
# To reach a[1, 2] from the buffer start: 1*32 + 2*8 = 48 bytes in.
flat = a.ravel()
print(flat[48 // a.itemsize])     # 6, and a[1,2] is also 6

# TRANSPOSE IS FREE: it swaps strides, it does not move a single byte.
t = a.T
print(t.shape, t.strides)         # (4, 3) (8, 32) -- reversed
print(t.base is a)                # True: t is a VIEW onto a's buffer

# DTYPE CHOICE is a memory decision at scale.
big = np.ones(10_000_000)
print(f"float64: {big.nbytes / 1e6:6.1f} MB")
print(f"float32: {big.astype(np.float32).nbytes / 1e6:6.1f} MB")
# Half the memory, and often faster, at ~7 significant digits instead of 16.

# SILENT INTEGER OVERFLOW: no error, just a wrong answer.
small = np.array([2_000_000_000, 2_000_000_000], dtype=np.int32)
print(small.sum())                # -294967296: wrapped around
print(small.sum(dtype=np.int64))  # 4000000000: correct
# Rule: for counts and sums over large integer arrays, name the accumulator.

# NaN FORCES FLOAT. There is no missing value in an integer array.
ints = np.array([1, 2, 3])
# ints[0] = np.nan          # ValueError
print(ints.astype(float))         # [1. 2. 3.] -- now NaN is representable

# OBJECT DTYPE: an array in name only.
mixed = np.array([1, "two", 3.0])
print(mixed.dtype)                # <U32: everything coerced to string
truly_mixed = np.array([1, "two", {3}], dtype=object)
print(truly_mixed.dtype)          # object: a list of pointers, slow

  ONE BUFFER, MANY VIEWS OF IT

  memory (the ribbon), 12 int64 values:
  +---+---+---+---+---+---+---+---+---+---+---+---+
  | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |10 |11 |
  +---+---+---+---+---+---+---+---+---+---+---+---+
   0   8  16  24  32  40  48  56  64  72  80  88   <- byte offsets

  shape (3,4), strides (32,8)      shape (4,3), strides (8,32)  [the .T]

     0  1  2  3                       0  4  8
     4  5  6  7                       1  5  9
     8  9 10 11                       2  6 10
                                      3  7 11
  step one ROW    -> +32 bytes     step one ROW    -> +8 bytes
  step one COLUMN -> +8 bytes      step one COLUMN -> +32 bytes

  SAME BYTES. Only the stencil changed, which is why .T costs nothing.

Key Takeaways

• An array is a flat buffer plus metadata: shape says how to index it, dtype says what each element is, strides say how far to step in bytes.
• Transposes, slices, and reshapes usually adjust strides rather than copying data, which makes them essentially free.
• Integer arrays overflow silently; pass an explicit dtype to sum when accumulating large counts.
• Integers cannot hold NaN, so any missing value forces a float array.
• object dtype preserves the array interface while discarding every performance benefit; treat its appearance as a bug signal.

🧪 Practice

1. Create a 4x5 array and predict its strides before printing them, then verify your byte-offset arithmetic locates a specific element.
2. Show that .T returns a view by mutating the transpose and observing the change in the original.
3. Interview: An analyst sums a large int32 array of event counts and gets a negative total. Explain the cause and two fixes. (Hint: what is the accumulator's type?)

Array Creation and Initialization

How you create an array determines its dtype, its memory cost, and occasionally whether your results are correct, so the constructors deserve more attention than they usually get.

The constructors fall into four groups. From existing data: np.array and np.asarray. From a shape: zeros, ones, full, empty. From a range: arange and linspace. Structured: eye, diag, identity.

The distinction between array and asarray matters in library code. np.array copies by default; np.asarray returns the input untouched if it is already a suitable array. Writing x = np.asarray(x) at the top of a function accepts lists and arrays alike without a pointless copy.

arange versus linspace is a correctness issue, not a style preference. arange takes a step, and with floating-point steps the endpoint is unreliable because the step cannot be represented exactly. linspace takes a count and guarantees both endpoints. Use arange for integers and linspace for floats; this rule has no exceptions worth remembering.

np.empty allocates without initializing, so it contains whatever was previously in that memory. It is marginally faster and a genuine source of irreproducible bugs when you forget to fill every element. Prefer zeros unless profiling says otherwise.

Finally, prefer the _like family. np.zeros_like(x) inherits shape and dtype together, which keeps an output array in sync with its input when the input's dtype later changes.

import numpy as np

# FROM DATA: note how the dtype is inferred.
print(np.array([1, 2, 3]).dtype)            # int64
print(np.array([1, 2, 3.0]).dtype)          # float64: one float promotes all
print(np.array([[1, 2], [3, 4]]).shape)     # (2, 2)

# array COPIES, asarray does NOT when it can avoid it.
src = np.arange(3)
print(np.array(src) is src, np.asarray(src) is src)      # False True
# The library idiom: accept lists or arrays with no needless copy.
def normalize(x):
    x = np.asarray(x, dtype=float)
    return x / x.sum()

# FROM A SHAPE.
print(np.zeros((2, 3)))                 # float64 zeros
print(np.ones(3, dtype=np.int8))        # dtype is explicit and cheap
print(np.full((2, 2), np.nan))          # a NaN-filled sentinel array
junk = np.empty(3)                      # UNINITIALIZED: arbitrary contents
print(junk)                             # whatever was in that memory

# FROM A RANGE: the float trap.
print(np.arange(0, 1, 0.1).size)        # 10, as expected
print(np.arange(0, 0.3, 0.1))           # [0.  0.1 0.2] -- fine
print(np.arange(1, 1.3, 0.1))           # sometimes 3, sometimes 4 elements
# The step 0.1 is not exact in binary, so the endpoint test is fragile.

print(np.linspace(0, 1, 11))            # 11 points, 0 and 1 BOTH included
print(np.linspace(0, 1, 10, endpoint=False))   # exclusive, like arange
print(np.logspace(0, 3, 4))             # [1. 10. 100. 1000.] for log grids

# STRUCTURED constructors.
print(np.eye(3))                        # identity
print(np.diag([1, 2, 3]))               # diagonal from a vector
print(np.diag(np.arange(9).reshape(3, 3)))     # ...and extract one back

# THE _like FAMILY: shape and dtype inherited together.
template = np.arange(6, dtype=np.float32).reshape(2, 3)
out = np.zeros_like(template)           # (2,3) float32, stays in sync
print(out.shape, out.dtype)

# REPRODUCIBLE RANDOM DATA: always through a seeded Generator.
rng = np.random.default_rng(42)
print(rng.normal(loc=0, scale=1, size=(2, 3)).round(3))

# MEMORY CHECK BEFORE ALLOCATION: a habit that avoids killed kernels.
shape, itemsize = (50_000, 10_000), 8
print(f"would need {np.prod(shape) * itemsize / 1e9:.1f} GB")   # 4.0 GB

Constructor	Gives	Use for
np.array(x)	a copy	converting data you own
np.asarray(x)	no copy if possible	function entry points
np.zeros(shape)	initialized zeros	preallocated output
np.empty(shape)	uninitialized memory	only when you fill every slot
np.arange(a,b,step)	step-defined range	integers
np.linspace(a,b,n)	count-defined range	floats, guaranteed endpoints
np.zeros_like(x)	zeros matching x	outputs tracking an input

Key Takeaways

• np.array copies and np.asarray does not, which makes the latter the right call at a function's entry point.
• Use arange for integer steps and linspace for float ranges, because float steps make the endpoint unreliable.
• np.empty returns uninitialized memory and is a real source of irreproducible bugs; default to zeros.
• The _like constructors keep an output's shape and dtype tied to its input.
• Compute the byte cost of a shape before allocating it, especially in a notebook where a failed allocation kills the kernel.

🧪 Practice

1. Show a case where np.arange with a float step produces a different element count than you expected, and fix it with linspace.
2. Write a function that accepts either a list or an array and doubles it, using asarray so no copy occurs for array input. Verify with is.
3. Interview: Why might a function that preallocates with np.empty produce different results across runs? (Hint: what is in that memory before you write to it?)

Indexing, Slicing, Views vs Copies

This is the topic that produces the most "my data changed and I do not know why" incidents in NumPy. The rule is short, and knowing it precisely is worth more than any amount of defensive copying.

Basic indexing with integers and slices returns a view: a new array object sharing the original's buffer. No data is copied, and writing through the view writes through to the original. This is deliberate, because it makes slicing a gigabyte array free.

Advanced indexing with an integer array or a boolean mask returns a copy. It must, because the selected elements need not be evenly spaced, and a view can only express a regular stride pattern.

That is the whole rule: slices give views, arrays of indices give copies. The useful test is arr.base, which is None for an owning array and points at the parent for a view.

The analogy is a window versus a photograph. A slice is a window into a room: rearranging the furniture is visible from both sides of the glass. A fancy-indexed selection is a photograph: you can scribble on it freely and the room is unaffected.

Slicing syntax itself follows one convention worth internalising: start is inclusive, stop is exclusive, and a negative step reverses direction. For multidimensional arrays, arr[i, j] is one indexing operation while arr[i][j] is two, and the second builds a throwaway intermediate.

import numpy as np

a = np.arange(12).reshape(3, 4)

# BASIC INDEXING -> VIEW. Writing through it writes through to the parent.
row = a[1]                     # a slice along axis 0
row[0] = 999
print(a[1, 0])                 # 999: the parent changed
print(row.base is a)           # True: row does not own its data

block = a[0:2, 1:3]            # a rectangular slice, still a view
block[:] = 0
print(a)                       # the original's top-right block is zeroed

# ADVANCED INDEXING -> COPY. The parent is safe.
a = np.arange(12).reshape(3, 4)
picked = a[[0, 2]]             # a LIST of row indices: fancy indexing
picked[0, 0] = 999
print(a[0, 0])                 # 0: unchanged
print(picked.base is None)     # True: picked owns its data

masked = a[a > 6]              # boolean mask: also a copy, and always 1-D
masked[0] = -1
print(a.max())                 # 11: original untouched

# THE DIAGNOSTIC: does this array own its memory?
print(a.base is None, a[0:2].base is None)      # True False

# STEPS AND REVERSAL.
v = np.arange(10)
print(v[2:8:2])                # [2 4 6] start inclusive, stop exclusive
print(v[::-1])                 # reversed, and still a view (stride -8)
print(v[::-1].strides)         # (-8,)

# MULTIDIMENSIONAL: one operation beats two.
print(a[1, 2])                 # direct: one indexing operation
print(a[1][2])                 # builds an intermediate row first: slower

# ELLIPSIS for unknown dimensionality.
cube = np.arange(24).reshape(2, 3, 4)
print(cube[..., 0].shape)      # (2, 3): last axis indexed, rest untouched
print(cube[0, ...].shape)      # (3, 4)

# np.newaxis ADDS a dimension of length 1, which drives broadcasting.
v = np.arange(3)
print(v[:, np.newaxis].shape)  # (3, 1) column vector
print(v[np.newaxis, :].shape)  # (1, 3) row vector

# WHEN YOU WANT INDEPENDENCE, say so explicitly.
safe = a[0:2].copy()
safe[0, 0] = -1
print(a[0, 0])                 # unchanged

# THE MEMORY TRAP OF VIEWS: a small view keeps a large buffer alive.
huge = np.zeros(50_000_000)
tiny = huge[:10]               # a view holding a reference to 400 MB
del huge                       # the buffer is NOT freed: tiny still uses it
tiny = tiny.copy()             # now the 400 MB can be reclaimed

  THE ONE RULE

  a[1:3]        slice          -> VIEW   -> shares memory -> writes propagate
  a[:, 0]       slice          -> VIEW
  a[[0, 2]]     index array    -> COPY   -> independent
  a[a > 5]      boolean mask   -> COPY   -> and always flattened to 1-D
  a[1, 2]       scalar         -> a copy of one value

  WHY ADVANCED INDEXING MUST COPY

  buffer:  [0][1][2][3][4][5][6][7]
  a[1:7:2] selects  1, 3, 5   -> regular stride 2 -> expressible as a view
  a[[5,0,3]] selects 5, 0, 3  -> no constant step -> must be materialized

  DIAGNOSTIC:  arr.base is None  ->  owns its data
               arr.base is not None -> it is a view of arr.base

Key Takeaways

• Slices return views that share memory; index arrays and boolean masks return copies.
• Advanced indexing must copy because arbitrary selections cannot be described by a constant stride.
• arr.base tells you which case you have: None means the array owns its buffer.
• Boolean masking always flattens to one dimension, regardless of the input's shape.
• A small view keeps its entire parent buffer alive; copy it if you intend to release the original.

🧪 Practice

1. For each of a[1:3], a[[1,3]], a[a>2], and a[:, 1], predict whether it is a view, then verify with .base.
2. Construct a case where forgetting that a slice is a view silently corrupts a training set, then fix it with an explicit copy.
3. Interview: You slice ten elements out of a two-gigabyte array and delete the original, but memory usage does not drop. Why? (Hint: what does the slice still reference?)

Boolean Masking and Fancy Indexing

Boolean masking is how you express "the rows where" without writing a loop, and it is probably the single most-used NumPy idiom in analysis code.

A comparison on an array is itself vectorized: x > 5 returns a boolean array of the same shape, not a single True. Indexing with that array selects the positions where it is True. The mask and the array must have matching shape, and the result is always flattened to one dimension because the selected positions need not form a rectangle.

Combining conditions requires the bitwise operators &, |, and ~, not the keywords and, or, and not. The keywords demand a single truth value and raise "truth value of an array is ambiguous". Bitwise operators also bind more tightly than comparisons, so each condition needs parentheses. Nearly everyone hits both errors once.

Fancy indexing with integer arrays is the other half. It selects elements by position in the order you specify, which makes it the tool for reordering, repeating, and gathering. Passing two index arrays selects element pairs, not a rectangular submatrix, which surprises people coming from MATLAB.

The most valuable derived tool is np.where with one argument, which returns the indices where a condition holds rather than the values. That is what you need when the position matters, such as finding the first day a threshold was crossed.

import numpy as np

rng = np.random.default_rng(0)
sales = rng.integers(0, 100, size=12)
region = np.array(["eu", "us", "eu", "ap"] * 3)

# A COMPARISON IS VECTORIZED: it returns an array of booleans.
mask = sales > 50
print(mask)                     # [ True False ... ]
print(sales[mask])              # only the passing values, 1-D
print(mask.sum())               # counting Trues: sum of a boolean array
print(mask.mean().round(3))     # the PROPORTION passing: a useful idiom

# COMBINING CONDITIONS: bitwise operators, and parentheses are mandatory.
big_eu = sales[(sales > 50) & (region == "eu")]     # & not `and`
print(big_eu)
# sales[sales > 50 and region == "eu"]  -> ValueError: ambiguous truth value
# sales[sales > 50 & region == "eu"]    -> wrong: & binds before >

print(sales[~(region == "ap")].size)                # ~ is NOT
print(sales[np.isin(region, ["eu", "ap"])].size)    # membership in a set

# ASSIGNMENT THROUGH A MASK modifies in place, which IS allowed.
capped = sales.copy()
capped[capped > 80] = 80                            # winsorize at 80
print(capped.max())

# np.where AS A TERNARY: elementwise choose.
flag = np.where(sales > 50, "high", "low")
print(flag[:4])

# np.where WITH ONE ARGUMENT returns POSITIONS, not values.
idx = np.where(sales > 90)[0]           # a tuple per dimension; take [0]
print("positions above 90:", idx)
first = idx[0] if idx.size else None    # first crossing; guard the empty case
print("first crossing:", first)

# FANCY INDEXING: select by position, in the order given.
order = np.argsort(sales)[::-1]         # indices of a descending sort
print(sales[order][:3])                 # the top three values
print(sales[[0, 0, 1]])                 # repetition is allowed

# TWO INDEX ARRAYS select PAIRS, not a submatrix.
m = np.arange(12).reshape(3, 4)
print(m[[0, 2], [1, 3]])                # [m[0,1], m[2,3]] = [1, 11]
print(m[np.ix_([0, 2], [1, 3])])        # np.ix_ gives the 2x2 SUBMATRIX

# MASKS ACROSS AXES: select whole rows by a per-row condition.
row_ok = m.sum(axis=1) > 10             # one boolean per row
print(m[row_ok])                        # keeps full rows, stays 2-D

# any and all reduce a mask to a decision.
print(mask.any(), mask.all())
print((m > 100).any(axis=1))            # per-row: does any element exceed?

  MASKING, STEP BY STEP

  sales   [ 42 | 77 | 13 | 95 | 58 ]
  > 50    [  F |  T |  F |  T |  T ]   <- same shape, dtype bool
  sales[mask]      [ 77 | 95 | 58 ]    <- compressed, always 1-D

  COMBINING: & | ~ , with parentheses around every comparison

  (sales > 50)  [ F | T | F | T | T ]
  (region=="eu")[ T | F | T | F | T ]
  AND           [ F | F | F | F | T ]   <- elementwise

  WHY `and` FAILS: it asks for ONE truth value from a 5-element array.

  TWO KINDS OF ANSWER
    sales[sales > 90]        -> the VALUES   [95]
    np.where(sales > 90)[0]  -> the POSITIONS [3]

Key Takeaways

• Comparisons produce boolean arrays; indexing with one selects matching elements and flattens the result to one dimension.
• Use &, |, and ~ with parentheses around each condition, because and/or require a single truth value.
• mask.sum() counts matches and mask.mean() gives the proportion, both without writing a loop.
• np.where(cond) returns positions while arr[cond] returns values; pick based on whether the index matters.
• Two integer index arrays select element pairs; use np.ix_ when you want the rectangular submatrix instead.

🧪 Practice

1. From an array of daily temperatures, select all readings that are both above the mean and occurred on an even-numbered day.
2. Find the index of the first element exceeding a threshold, returning None safely when none does.
3. Interview: A colleague writes df[col > 5 and col < 10] and gets an ambiguity error. Explain the cause and give the corrected expression. (Hint: what does and need from its operands?)

Broadcasting Rules

Broadcasting is how NumPy applies an operation between arrays of different shapes without materializing the intermediate. It looks like magic until you learn the rule, and then it is completely mechanical.

The motivation is concrete. To centre a data matrix you subtract a per-column mean, a (1000, 5) array minus a (5,) array. Tiling the mean into a full (1000, 5) array would work and would waste memory. Broadcasting achieves the same result by reading the same five values repeatedly.

The rule has two parts. First, align the shapes from the right, padding the shorter one with leading ones. Second, for each axis the sizes must be equal, or one of them must be 1. An axis of size 1 is stretched by reusing its single value. Any other mismatch is an error.

Nothing is copied. Stretching is implemented as a stride of zero on that axis, so the same memory is read repeatedly. That is why broadcasting is both fast and memory-free.

The trap is that broadcasting is sometimes too accommodating. Subtracting a (3,) array from a (3, 3) array succeeds whether you meant rows or columns, and it silently picks rows. When the intent is columns you must add the axis yourself with [:, None]. Silent success with the wrong orientation is far worse than an error.

import numpy as np

# THE MOTIVATING CASE: centre each column.
X = np.arange(12, dtype=float).reshape(4, 3)
col_mean = X.mean(axis=0)              # shape (3,)
centred = X - col_mean                 # (4,3) - (3,) -> broadcast
print(centred.round(2))
print(centred.mean(axis=0).round(12))  # each column now ~0

# ALIGNMENT IS FROM THE RIGHT.
#   X         (4, 3)
#   col_mean     (3,)  -> padded to (1, 3) -> stretched to (4, 3)

# ROWS VS COLUMNS: the silent-wrong-answer trap.
row_mean = X.mean(axis=1)              # shape (4,)
# X - row_mean  -> ValueError (4,3) vs (4,): the last axes 3 and 4 clash.
# Here the error is a gift. With a square matrix there would be none:
S = np.arange(9, dtype=float).reshape(3, 3)
v = np.array([1.0, 2.0, 3.0])
print(S - v)                    # subtracts v from each ROW. Was that meant?
print(S - v[:, None])           # subtracts v from each COLUMN. Be explicit.

# The correct row-centring of X:
print((X - row_mean[:, None]).round(2))

# THE OUTER-PRODUCT PATTERN: (n,1) against (1,m) gives (n,m).
a = np.arange(1, 4)[:, None]           # (3, 1)
b = np.arange(1, 5)[None, :]           # (1, 4)
print(a * b)                           # (3, 4) multiplication table

# PAIRWISE DISTANCES, the classic broadcasting application.
pts = np.array([[0.0, 0.0], [3.0, 4.0], [6.0, 8.0]])
diff = pts[:, None, :] - pts[None, :, :]        # (3,1,2) - (1,3,2) -> (3,3,2)
dist = np.sqrt((diff ** 2).sum(axis=-1))        # (3, 3)
print(dist.round(2))

# MEMORY: broadcasting itself allocates nothing, but the RESULT is real.
big = np.ones((10_000, 1))
wide = np.ones((1, 10_000))
print(f"inputs: {(big.nbytes + wide.nbytes) / 1e6:.2f} MB")
print(f"result: {(big * wide).nbytes / 1e6:.0f} MB")    # 800 MB
# The intermediate is free; the output is not. Watch the OUTPUT shape.

# INSPECT the result shape before computing a large one.
print(np.broadcast_shapes((4, 3), (3,)))        # (4, 3)
print(np.broadcast_shapes((5, 1), (1, 4)))      # (5, 4)
# np.broadcast_shapes((4,3), (4,))   -> ValueError, as it should

# A ZERO STRIDE is how stretching is actually implemented.
bc = np.broadcast_to(np.arange(3), (4, 3))
print(bc.shape, bc.strides)            # (4, 3) (0, 8): axis 0 costs nothing

  THE ALGORITHM, WORKED

  A.shape = (8, 1, 6, 1)
  B.shape =    (7, 1, 5)

  1. Right-align and left-pad with 1s:
       A  8  1  6  1
       B  1  7  1  5
  2. Per axis: equal, or one is 1?
       8 vs 1 -> stretch B      OK
       1 vs 7 -> stretch A      OK
       6 vs 1 -> stretch B      OK
       1 vs 5 -> stretch A      OK
  3. Result: (8, 7, 6, 5)

  A FAILING CASE
       A  4  3
       B     4
       3 vs 4 -> neither equal nor 1 -> ValueError

  ROWS OR COLUMNS?  (3,3) with (3,)
       M  3  3
       v     3     -> aligns to the LAST axis -> applies per ROW
  To apply per COLUMN, make the intent explicit:  v[:, None] -> (3,1)

Key Takeaways

• Shapes are aligned from the right and padded with leading ones; each axis must then match or be 1.
• Stretching uses a zero stride, so no data is copied and the intermediate costs nothing.
• The output array is fully materialized, so check the broadcast result shape before running it on large inputs.
• A one-dimensional array broadcasts along rows by default; use [:, None] when you mean columns.
• np.broadcast_shapes lets you verify the outcome without allocating.

🧪 Practice

1. Given a (100, 5) matrix, scale each column to unit variance using broadcasting, then do the same per row and explain the shape change.
2. Build a pairwise Euclidean distance matrix for 500 two-dimensional points without any loop, and state the output's memory cost.
3. Interview: Subtracting a length-n vector from an n-by-n matrix runs without error but gives wrong results. What went wrong? (Hint: which axis does right-alignment pick?)

Universal Functions and Element-Wise Operations

A ufunc is a function that operates element by element on arrays, with the loop compiled in C. Almost every arithmetic operator and mathematical function in NumPy is one, and they share a set of features that most users never discover.

The mental model is that a ufunc is a scalar function lifted to arrays. It broadcasts its inputs, applies the scalar operation to each aligned tuple, and writes the results into an output array. np.add(a, b) and a + b are the same call.

Three optional arguments turn a ufunc into a precision instrument. The out parameter writes into an existing array, avoiding an allocation. The where parameter applies the operation only where a mask is True, leaving the other positions untouched. And the dtype parameter controls the accumulation type.

Binary ufuncs also carry four methods. reduce collapses an axis, so np.add.reduce is sum. accumulate gives the running version, so np.add.accumulate is cumsum. outer applies the operation to every pair. reduceat performs segmented reductions, which is how you aggregate ragged groups without a loop.

Chained arithmetic allocates a temporary for each intermediate, and on large arrays that dominates the runtime. Using out or in-place operators such as *= removes those allocations, which is the cheapest optimization available in NumPy.

import numpy as np

a = np.array([1.0, 4.0, 9.0, 16.0])
b = np.array([2.0, 2.0, 3.0, 0.0])

# OPERATORS ARE UFUNCS.
print(np.add(a, b) is not None, np.allclose(a + b, np.add(a, b)))
print(np.sqrt(a), np.exp(b), np.log(a).round(3))

# OUT: reuse a buffer instead of allocating a new one.
result = np.empty_like(a)
np.multiply(a, b, out=result)
print(result)

# IN-PLACE OPERATORS avoid temporaries entirely.
x = np.arange(1e6)
x *= 2                  # no new array
x += 1                  # no new array
# versus  x = x * 2 + 1 , which allocates two 8 MB temporaries

# WHERE: apply only at selected positions, leaving the rest alone.
safe = np.zeros_like(a)
np.divide(a, b, out=safe, where=(b != 0))     # skip the division by zero
print(safe)             # last entry stays 0 instead of becoming inf
# Without `where` this emits a RuntimeWarning and produces inf.

# CONTROLLING ERROR BEHAVIOUR explicitly.
with np.errstate(divide="ignore", invalid="ignore"):
    ratio = a / b
print(ratio)            # [0.5 2. 3. inf]
print(np.nan_to_num(ratio, posinf=0.0))        # sanitize afterwards

# UFUNC METHODS.
print(np.add.reduce(a))              # 30.0, same as a.sum()
print(np.add.accumulate(a))          # running sum
print(np.multiply.outer([1, 2], [10, 20]))     # every pair
print(np.maximum.reduce(a))          # 16.0

# reduceat: SEGMENTED reduction, no loop, no groupby.
values = np.array([1, 2, 3, 4, 5, 6, 7, 8])
starts = np.array([0, 3, 5])         # segments [0:3], [3:5], [5:]
print(np.add.reduceat(values, starts))         # [6, 9, 21]

# TYPE PROMOTION follows fixed rules; know them to avoid surprises.
print((np.int8(100) + np.int8(100)))           # overflow inside int8
print((np.array([1], dtype=np.int8) + 1).dtype)      # int8 preserved
print((np.array([1], dtype=np.int8) + 1.0).dtype)    # float64: promoted

# FLOATING-POINT COMPARISON: never use ==.
p = np.array([0.1, 0.2, 0.3])
print(p[0] + p[1] == p[2])                      # False
print(np.isclose(p[0] + p[1], p[2]))            # True
print(np.allclose(p.sum(), 0.6))                # True, for whole arrays

# CUSTOM ELEMENTWISE LOGIC stays vectorized by composing ufuncs.
def stable_sigmoid(z):
    z = np.asarray(z, dtype=float)
    out = np.empty_like(z)
    pos, neg = z >= 0, z < 0                    # two numerically safe branches
    out[pos] = 1.0 / (1.0 + np.exp(-z[pos]))
    ez = np.exp(z[neg])                         # exp of a negative: no overflow
    out[neg] = ez / (1.0 + ez)
    return out

print(stable_sigmoid([-1000, 0, 1000]))         # [0. 0.5 1.] no warnings

Key Takeaways

• A ufunc lifts a scalar operation to arrays with a compiled loop and full broadcasting; operators such as + are ufuncs.
• The out parameter and in-place operators eliminate temporary allocations, which dominate runtime on large arrays.
• The where parameter applies the operation selectively, which is the clean way to avoid division-by-zero warnings.
• reduce, accumulate, outer, and reduceat extend any binary ufunc, with reduceat giving loop-free segmented aggregation.
• Compare floats with np.isclose or np.allclose, never with ==.

🧪 Practice

1. Compute (a * b + c) / d on ten-million-element arrays twice, once naively and once using out parameters, and compare peak memory.
2. Use np.divide with where to compute a ratio that leaves zero denominators as zero rather than infinity.
3. Interview: Why does 0.1 + 0.2 == 0.3 evaluate to False, and what should a test assert instead? (Hint: think about binary representation of decimal fractions.)

Aggregations and Axis Semantics

Aggregation is where most analytical value is extracted, and the axis argument is where most people guess. It has a precise meaning that removes the guessing entirely.

The rule: axis=k is the axis that disappears. The operation runs along that axis and collapses it, so the output shape is the input shape with position k deleted. For a two-dimensional array, axis=0 collapses rows and returns one value per column; axis=1 collapses columns and returns one value per row.

The common mistake is reading axis=0 as "operate on rows". It is the opposite: it operates down the rows to summarize each column. Saying "the axis that vanishes" out loud fixes this permanently.

keepdims=True retains the collapsed axis with length 1 instead of removing it. This matters because the result then broadcasts cleanly against the original, which is exactly what normalization needs.

NaN propagates through every standard aggregation, so one missing value turns an entire column mean into NaN. The nan-prefixed variants skip missing values instead. Neither behaviour is universally right: silent skipping hides data quality problems, so decide deliberately rather than reaching for nanmean by reflex.

import numpy as np

X = np.arange(12).reshape(3, 4)
print(X)

# THE AXIS THAT DISAPPEARS.
print(X.sum())            # 66            no axis: everything collapses
print(X.sum(axis=0))      # [12 15 18 21] shape (3,4) -> (4,): per COLUMN
print(X.sum(axis=1))      # [ 6 22 38]    shape (3,4) -> (3,): per ROW

print(X.sum(axis=0).shape, X.sum(axis=1).shape)      # (4,) (3,)

# KEEPDIMS preserves the axis at length 1, so it broadcasts back.
print(X.sum(axis=1, keepdims=True).shape)            # (3, 1)
row_normalized = X / X.sum(axis=1, keepdims=True)    # works directly
print(row_normalized.round(3))
# Without keepdims: X / X.sum(axis=1) -> ValueError (3,4) vs (3,)

# HIGHER DIMENSIONS: the rule is unchanged.
cube = np.arange(24).reshape(2, 3, 4)
print(cube.sum(axis=0).shape)        # (3, 4)
print(cube.sum(axis=1).shape)        # (2, 4)
print(cube.sum(axis=(0, 2)).shape)   # (3,)  collapse two axes at once

# NaN PROPAGATES. One missing value poisons the whole statistic.
data = np.array([[1.0, 2.0, np.nan], [4.0, 5.0, 6.0]])
print(data.mean(axis=0))             # [2.5 3.5 nan]
print(np.nanmean(data, axis=0))      # [2.5 3.5 6. ] skips the NaN
print(np.isnan(data).sum(axis=0))    # [0 0 1] ALWAYS report what you skipped

# ARGMIN and ARGMAX give POSITIONS along the collapsed axis.
print(X.argmax(axis=0))              # [2 2 2] which row holds each max
print(np.unravel_index(X.argmax(), X.shape))    # (2, 3) global position

# THE FULL AGGREGATION SET.
print(X.min(), X.max(), X.mean(), X.std(), X.var())
print(np.median(X, axis=1))
print(np.percentile(X, [25, 50, 75]))
print(np.average(X, axis=0, weights=[1, 1, 2]))      # weighted mean

# ddof CHANGES THE ANSWER: NumPy defaults to the POPULATION standard
# deviation, while pandas and R default to the SAMPLE one.
s = np.array([2.0, 4.0, 4.0, 4.0, 5.0, 5.0, 7.0, 9.0])
print(f"ddof=0 (population): {s.std():.4f}")
print(f"ddof=1 (sample):     {s.std(ddof=1):.4f}")
# Comparing a NumPy std to a pandas std without checking ddof is a
# reliable source of "the numbers do not match" incidents.

# BOOLEAN AGGREGATION: counting and testing.
print((X > 5).sum(axis=1))           # per-row count of matches
print((X > 5).any(axis=1), (X > 5).all(axis=1))

# NUMERICAL STABILITY: naive summation loses precision on long arrays.
noisy = np.full(1_000_000, 0.1, dtype=np.float32)
print(f"float32 sum : {noisy.sum():.3f}")                  # drifts
print(f"pairwise/f64: {noisy.sum(dtype=np.float64):.3f}")  # 100000.000

  "THE AXIS THAT DISAPPEARS"

  X, shape (3, 4)          axis 0 runs DOWN, axis 1 runs ACROSS

            axis 1 ---->
         [  0   1   2   3 ]
  axis 0 [  4   5   6   7 ]
    |    [  8   9  10  11 ]
    v

  X.sum(axis=0)  collapses the 3  ->  shape (4,)   [12 15 18 21]
                 one value per COLUMN

  X.sum(axis=1)  collapses the 4  ->  shape (3,)   [ 6 22 38]
                 one value per ROW

  keepdims=True  ->  (1,4) or (3,1): the axis stays, length 1,
                     so the result still broadcasts against X

     X / X.sum(axis=1, keepdims=True)      works    (3,4) / (3,1)
     X / X.sum(axis=1)                     fails    (3,4) / (3,)

Key Takeaways

• axis=k names the axis that is collapsed and removed, so axis=0 summarizes each column and axis=1 each row.
• keepdims=True keeps the axis at length 1 so the result broadcasts back against the original, which normalization requires.
• NaN propagates through standard aggregations; the nan-prefixed variants skip it, and you should report how many were skipped.
• argmax returns a position along the collapsed axis; unravel_index converts a flat position into full coordinates.
• NumPy's std defaults to ddof=0 while pandas and R use ddof=1, a frequent cause of mismatched numbers across tools.

🧪 Practice

1. For a (5, 3, 2) array, predict the output shape of summing over each axis and over the pair (0, 2), then verify.
2. Normalize a matrix so every row sums to one, and explain why omitting keepdims raises an error.
3. Interview: A pandas standard deviation and a NumPy standard deviation on the same column differ slightly. Why? (Hint: check the denominator each library uses by default.)

Reshaping, Transposing, Stacking, Splitting

Real data rarely arrives in the shape a function expects. Reshaping operations rearrange the metadata around a buffer, and knowing which ones copy tells you which are free.

Reshape changes the shape tuple while preserving the element order and count. It returns a view whenever the new shape is compatible with the existing strides, which it usually is. Passing -1 for one dimension asks NumPy to infer it, which is how you flatten an image batch without hard-coding sizes.

The critical detail is order. NumPy reads and writes in C order by default, meaning the last axis varies fastest. Reshaping a (2, 3) array to (3, 2) therefore does not transpose it; it re-cuts the same linear sequence into different row lengths. Confusing reshape with transpose produces data that is subtly scrambled rather than obviously broken.

Transpose permutes axes by reordering strides, always free. For arrays with more than two dimensions, np.transpose takes an explicit axis order, and np.moveaxis expresses the common case more readably.

Stacking combines arrays. concatenate joins along an existing axis; stack creates a new one. vstack and hstack are convenience wrappers whose behaviour on one-dimensional inputs surprises people, so prefer the explicit forms in code others will read. All stacking copies, because the result needs contiguous memory.

import numpy as np

a = np.arange(12)

# RESHAPE: same data, new stencil. Usually a view.
m = a.reshape(3, 4)
print(m.base is a)                  # True: no copy
print(a.reshape(2, 6).shape, a.reshape(-1, 2).shape)   # -1 is inferred

# RESHAPE IS NOT TRANSPOSE. Both give (4,3); only one is the transpose.
b = np.arange(12).reshape(3, 4)
print(b.reshape(4, 3))              # re-cuts the SAME linear sequence
print(b.T)                          # genuinely swaps the axes
# Row 0 of reshape is [0,1,2]; row 0 of .T is [0,4,8]. Very different.

# ORDER: C means the LAST axis varies fastest; F means the first.
print(b.ravel())                    # [0 1 2 3 4 5 ...]  C order, a view
print(b.ravel(order="F"))           # [0 4 8 1 5 9 ...]  F order, a copy
print(b.flatten().base is None)     # True: flatten ALWAYS copies

# ravel vs flatten: ravel is free when it can be, flatten never is.
# Use ravel for reading, flatten when you need an independent buffer.

# TRANSPOSE in higher dimensions.
imgs = np.zeros((32, 224, 224, 3))          # NHWC: batch, H, W, channels
chw = imgs.transpose(0, 3, 1, 2)            # NCHW, which PyTorch expects
print(chw.shape, chw.base is imgs)          # (32, 3, 224, 224) True
print(np.moveaxis(imgs, -1, 1).shape)       # the same, more readably

# STACKING: concatenate joins an existing axis, stack makes a new one.
p = np.array([[1, 2], [3, 4]])
q = np.array([[5, 6], [7, 8]])
print(np.concatenate([p, q], axis=0).shape)   # (4, 2) rows appended
print(np.concatenate([p, q], axis=1).shape)   # (2, 4) columns appended
print(np.stack([p, q], axis=0).shape)         # (2, 2, 2) a NEW axis

# The 1-D surprise that motivates preferring explicit forms.
u, v = np.array([1, 2, 3]), np.array([4, 5, 6])
print(np.vstack([u, v]).shape)     # (2, 3): treats each as a row
print(np.hstack([u, v]).shape)     # (6,):   simple concatenation
print(np.column_stack([u, v]).shape)   # (3, 2): each becomes a column

# SPLITTING is the inverse.
big = np.arange(12).reshape(6, 2)
print([c.shape for c in np.split(big, 3, axis=0)])      # equal parts only
print([c.shape for c in np.array_split(big, 4, axis=0)])   # allows unequal
# np.split(big, 4) -> ValueError: 6 is not divisible by 4.

# A REAL PATTERN: flatten images for a classical model, then restore.
batch = np.random.default_rng(0).random((100, 28, 28))
flat = batch.reshape(len(batch), -1)        # (100, 784) for sklearn
print(flat.shape, flat.base is not None)    # a view, so this costs nothing
restored = flat.reshape(-1, 28, 28)
print(np.array_equal(batch, restored))      # True

# SQUEEZE and EXPAND_DIMS manage length-1 axes.
z = np.zeros((1, 5, 1))
print(z.squeeze().shape)                    # (5,)   drop ALL length-1 axes
print(z.squeeze(axis=0).shape)              # (5, 1) drop a specific one
print(np.expand_dims(np.zeros(5), axis=0).shape)   # (1, 5)

  RESHAPE VS TRANSPOSE ON THE SAME DATA

  buffer: 0 1 2 3 4 5 6 7 8 9 10 11

  b = (3,4)          b.reshape(4,3)        b.T  (4,3)
   0  1  2  3         0  1  2              0  4  8
   4  5  6  7         3  4  5              1  5  9
   8  9 10 11         6  7  8              2  6 10
                      9 10 11              3  7 11

  reshape: re-cut the same sequence into rows of 3   (a VIEW)
  transpose: swap the stride meanings                 (a VIEW)
  -> identical shapes, completely different contents

  STACKING VOCABULARY, for two (2,2) arrays

  concatenate axis=0  -> (4,2)   stack on top
  concatenate axis=1  -> (2,4)   stack side by side
  stack       axis=0  -> (2,2,2) a new leading axis

Key Takeaways

• Reshape re-cuts the same linear sequence and is usually a view; it is not a transpose and will scramble data if used as one.
• NumPy defaults to C order, so the last axis varies fastest; changing to Fortran order forces a copy.
• ravel returns a view when possible while flatten always copies.
• concatenate joins along an existing axis and stack adds a new one; both copy.
• Use -1 to infer a dimension so reshaping code survives changes in batch size.

🧪 Practice

1. Take a (2, 6) array and produce both its reshape to (6, 2) and its transpose, then explain element by element why they differ.
2. Convert a batch of images from NHWC to NCHW layout and confirm with .base that no data was copied.
3. Interview: A model trained on flattened images performs at chance after a refactor that changed the reshape order. What likely happened? (Hint: C order versus Fortran order.)

Sorting, Searching, and Unique Values

Sorting in NumPy is about more than ordering. The index-returning variants are what let you carry a sort across several parallel arrays, which is the operation analysis code actually needs.

np.sort returns sorted values. np.argsort returns the indices that would sort the array. The second is the more useful one: applying those indices to a second array reorders it consistently, which is how you answer "who are the top ten customers" rather than merely "what are the top ten values".

Three algorithms are available and the default, introsort, is right almost always. The exception worth knowing is stability. A stable sort preserves the relative order of equal elements, which matters when sorting by a secondary key. Only mergesort and stable are stable.

When you need the top k rather than a full ordering, np.argpartition is asymptotically better. It performs a partial sort in O(n) rather than O(n log n), placing the k smallest elements before position k without ordering them. On a million elements looking for the top ten, this is a large win.

For searching a sorted array, np.searchsorted does binary search and is the vectorized tool for binning continuous values into buckets. np.unique handles deduplication and, with its optional returns, gives value counts and an inverse mapping in a single pass.

import numpy as np

rng = np.random.default_rng(0)
revenue = np.array([300, 120, 950, 480, 120, 770])
customer = np.array(["a", "b", "c", "d", "e", "f"])

# sort GIVES VALUES; argsort GIVES THE PERMUTATION.
print(np.sort(revenue))              # [120 120 300 480 770 950]
order = np.argsort(revenue)          # positions, ascending
print(order)

# The point of argsort: carry the ordering to a PARALLEL array.
top = np.argsort(revenue)[::-1][:3]  # descending, top 3
print(list(zip(customer[top], revenue[top])))    # who, not just how much

# SORTING BY MULTIPLE KEYS: lexsort takes the LAST key as primary.
region = np.array(["eu", "us", "eu", "us", "eu", "us"])
idx = np.lexsort((-revenue, region))     # region asc, then revenue desc
print(list(zip(region[idx], revenue[idx])))

# STABILITY matters when a previous ordering must be preserved.
keys = np.array([2, 1, 2, 1])
print(np.argsort(keys, kind="stable"))   # [1 3 0 2] ties keep input order

# SORTING ALONG AN AXIS sorts each row INDEPENDENTLY, breaking row identity.
m = rng.integers(0, 10, (3, 4))
print(m)
print(np.sort(m, axis=1))            # each row sorted; columns now unrelated
# To sort ROWS by one column, argsort that column and index the whole array.
print(m[np.argsort(m[:, 0])])

# TOP-K WITHOUT A FULL SORT: O(n) instead of O(n log n).
big = rng.random(1_000_000)
k = 10
part = np.argpartition(big, -k)[-k:]     # the k largest, unordered
part = part[np.argsort(big[part])[::-1]] # order just those k
print(big[part][:3].round(4))

# SEARCHSORTED: vectorized binary search, the fast way to bin values.
edges = np.array([0, 18, 35, 60, 120])            # age bands
ages = np.array([5, 22, 35, 61, 99])
bins = np.searchsorted(edges, ages, side="right") - 1
print(bins)                          # [0 1 2 3 3]
labels = np.array(["child", "young", "middle", "senior"])
print(labels[bins])

# side matters exactly at a boundary: 'left' vs 'right' for ties.
print(np.searchsorted(edges, 35, side="left"),
      np.searchsorted(edges, 35, side="right"))    # 2 3

# UNIQUE, and everything it can return in one pass.
vals = np.array(["eu", "us", "eu", "ap", "us", "eu"])
u, counts = np.unique(vals, return_counts=True)
print(dict(zip(u, counts)))          # {'ap': 1, 'eu': 3, 'us': 2}

u, inverse = np.unique(vals, return_inverse=True)
print(inverse)                       # integer codes: the basis of categoricals
print(np.array_equal(u[inverse], vals))          # True: exactly invertible

u, first = np.unique(vals, return_index=True)
print(first)                         # first occurrence of each unique value

# SET OPERATIONS on arrays.
x, y = np.array([1, 2, 3, 4]), np.array([3, 4, 5])
print(np.intersect1d(x, y), np.union1d(x, y), np.setdiff1d(x, y))
print(np.isin(x, y))                 # elementwise membership, keeps shape

Key Takeaways

• argsort returns the permutation, which is what lets you reorder parallel arrays consistently; sort discards that information.
• Sorting along an axis sorts each row independently and destroys the correspondence between columns.
• Use lexsort for multi-key sorts, remembering the last key is primary.
• argpartition finds the top k in linear time when you do not need the remainder ordered.
• searchsorted is the vectorized binning tool, and unique with its optional returns gives counts and an invertible integer encoding.

🧪 Practice

1. Given parallel arrays of customer names and revenues, return the five highest-revenue customers using argsort, then again using argpartition, and time both on a million rows.
2. Bin a continuous array of ages into labelled bands with searchsorted, handling values exactly on a boundary deliberately.
3. Interview: You sort a two-dimensional array with np.sort(arr, axis=0) and the rows no longer make sense. Explain why. (Hint: what happens to each column independently?)

Linear Algebra with numpy.linalg

Chapter 2 covered the mathematics; this topic covers the API and, more importantly, the numerical judgement about which routine to call. The recurring theme is that the textbook formula is rarely the right code.

The central rule: never invert a matrix to solve a system. The mathematical statement x = A^-1 b translates to np.linalg.solve(A, b), not to inv(A) @ b. Solving uses LU decomposition directly, which is both about twice as fast and substantially more accurate, because forming the inverse amplifies rounding error.

Conditioning is the concept that makes this concrete. The condition number measures how much a small perturbation in the input can change the output. A matrix with a condition number of 10^10 loses roughly ten digits of precision, leaving about six from a float64's sixteen. NumPy raises no warning for this; you have to check.

For least squares, np.linalg.lstsq handles rank deficiency gracefully via SVD, while forming the normal equations squares the condition number and can fail outright on data that lstsq handles fine.

The decompositions each have a natural use. SVD is the most numerically robust and underlies PCA and the pseudo-inverse. QR solves least squares stably. Cholesky is twice as fast as LU but requires a symmetric positive-definite matrix, which covariance matrices usually are. And eigendecomposition should use eigh for symmetric matrices, which is faster and guarantees real eigenvalues.

import numpy as np

rng = np.random.default_rng(0)

A = np.array([[3.0, 1.0], [1.0, 2.0]])
b = np.array([9.0, 8.0])

# SOLVING: use solve, not inv.
x = np.linalg.solve(A, b)
print(x, np.allclose(A @ x, b))              # [2. 3.] True

inv_way = np.linalg.inv(A) @ b               # works here, worse in general
print(np.allclose(x, inv_way))

# WHY IT MATTERS: an ill-conditioned system.
n = 12
H = np.array([[1.0 / (i + j + 1) for j in range(n)] for i in range(n)])
true_x = np.ones(n)
rhs = H @ true_x

print(f"condition number: {np.linalg.cond(H):.2e}")      # ~1e16
x_solve = np.linalg.solve(H, rhs)
x_inv = np.linalg.inv(H) @ rhs
print(f"error via solve: {np.abs(x_solve - true_x).max():.3e}")
print(f"error via inv  : {np.abs(x_inv - true_x).max():.3e}")
# Both are poor because the matrix is intrinsically bad, but inv is worse.
# RULE OF THUMB: digits lost is about log10(cond).

# LEAST SQUARES: lstsq handles rank deficiency; normal equations do not.
X = rng.normal(size=(100, 3))
X = np.column_stack([X, X[:, 0]])            # a perfectly collinear column
y = rng.normal(size=100)

coef, residuals, rank, sv = np.linalg.lstsq(X, y, rcond=None)
print(f"rank {rank} of {X.shape[1]} columns")    # 3 of 4: detected
# np.linalg.solve(X.T @ X, X.T @ y)  -> LinAlgError: singular matrix

# DECOMPOSITIONS.
S = rng.normal(size=(5, 5))
S = S @ S.T + 5 * np.eye(5)                  # symmetric positive definite

evals, evecs = np.linalg.eigh(S)             # eigh for SYMMETRIC input
print("eigenvalues:", evals.round(3))        # real, ascending, guaranteed
# np.linalg.eig on a symmetric matrix can return tiny imaginary parts.

L = np.linalg.cholesky(S)                    # S = L @ L.T
print(np.allclose(L @ L.T, S))               # twice as fast as LU

U, s, Vt = np.linalg.svd(X, full_matrices=False)
print("singular values:", s.round(4))        # the last is ~0: the collinearity
print(f"rank from SVD: {(s > 1e-10).sum()}")

# THE PSEUDO-INVERSE handles singular matrices where inv cannot.
print(np.linalg.pinv(X).shape)               # (4, 100)

# NORMS AND PRODUCTS.
v = np.array([3.0, 4.0])
print(np.linalg.norm(v))                     # 5.0, the L2 norm
print(np.linalg.norm(v, ord=1))              # 7.0, the L1 norm
print(np.linalg.norm(S, ord="fro"))          # Frobenius norm of a matrix
print(np.dot(v, v), v @ v)                   # inner product
print(np.linalg.matrix_rank(X))              # 3
print(np.linalg.det(A), np.trace(A))

# PRACTICAL CHECK before trusting any linear model's coefficients.
def diagnose(M, name="matrix"):
    c = np.linalg.cond(M)
    print(f"{name}: cond={c:.2e}, ~{np.log10(c):.0f} digits lost of 16")
    if c > 1e10:
        print("  -> coefficients are unreliable; regularize or drop columns")

diagnose(X.T @ X, "X'X")

Task	Correct call	Avoid
Solve Ax = b	solve(A, b)	inv(A) @ b
Least squares	lstsq(X, y)	solve(X.T@X, X.T@y)
Symmetric eigenvalues	eigh(S)	eig(S)
SPD system	cholesky(S)	general LU
Singular system	pinv(A)	inv(A)
Rank / collinearity	matrix_rank, svd	det(A) == 0

Key Takeaways

• Use solve rather than forming an inverse; it is faster and loses less precision.
• The condition number predicts how many digits you lose, and NumPy will not warn you when that number is large.
• lstsq uses SVD and survives rank deficiency; normal equations square the condition number and fail.
• Prefer eigh for symmetric matrices and cholesky for symmetric positive-definite systems.
• Detect collinearity through matrix_rank or small singular values, not through a determinant test against zero.

🧪 Practice

1. Build a Hilbert matrix of size 10, solve a system with both solve and an explicit inverse, and compare errors against the known solution.
2. Add a perfectly collinear column to a design matrix, then show that lstsq succeeds where the normal equations raise an error.
3. Interview: A regression returns coefficients in the millions with enormous standard errors. What single diagnostic would you run first? (Hint: it concerns the design matrix, not the target.)

Random Number Generation and Reproducible Seeding

Randomness in a data pipeline must be reproducible, which sounds like a contradiction and is not. Computers generate pseudo-random numbers from a deterministic algorithm, so fixing the starting state fixes the entire sequence.

NumPy has two APIs, and this matters because most code online uses the old one. The legacy interface is the module-level np.random.seed and np.random.rand, backed by a single global state. The modern interface, recommended since NumPy 1.17, is np.random.default_rng(seed), which returns an independent Generator object.

The global state is the problem with the legacy API. Any library you import can reseed it or consume from it, so your "reproducible" results depend on code you never wrote. Passing an explicit generator makes the dependency visible and local. The modern bit generator is also statistically better.

For parallel work, never reuse a seed across workers, because every worker then produces identical streams. SeedSequence.spawn creates provably independent child streams from one parent seed, which keeps the whole parallel run reproducible from a single number.

Reproducibility in an ML pipeline needs more than one seed. Data splitting, weight initialization, shuffling, dropout, and augmentation all draw randomness. Seeding one and forgetting the others gives partial reproducibility, which is the most misleading kind.

import numpy as np

# THE MODERN API: an explicit, local generator object.
rng = np.random.default_rng(42)
print(rng.normal(size=3).round(4))

rng2 = np.random.default_rng(42)         # same seed, same stream
print(rng2.normal(size=3).round(4))      # identical

# WHY THE GLOBAL API IS FRAGILE.
np.random.seed(0)
first = np.random.rand(2)
np.random.seed(0)
_ = some_library_call() if False else np.random.rand(5)   # a hidden draw
second = np.random.rand(2)
print(np.allclose(first, second))        # False: the stream moved
# With a Generator object nothing outside your function can touch it.

# PASSING A GENERATOR is the reproducible-function pattern.
def bootstrap_mean(data, n_boot=1000, rng=None):
    rng = np.random.default_rng(rng)     # accepts None, an int, or a Generator
    data = np.asarray(data)
    idx = rng.integers(0, len(data), size=(n_boot, len(data)))
    return data[idx].mean(axis=1)

sample = np.array([2.0, 4.0, 4.0, 5.0, 7.0, 9.0])
boots = bootstrap_mean(sample, rng=7)
print(f"95% CI: [{np.percentile(boots, 2.5):.2f}, "
      f"{np.percentile(boots, 97.5):.2f}]")

# PARALLEL STREAMS: spawn, never reuse a seed.
parent = np.random.SeedSequence(12345)
children = parent.spawn(4)               # 4 independent, reproducible streams
workers = [np.random.default_rng(c) for c in children]
print([w.integers(0, 100) for w in workers])     # all different

bad = [np.random.default_rng(12345) for _ in range(4)]
print([w.integers(0, 100) for w in bad])         # all IDENTICAL: a real bug

# THE DISTRIBUTION SET.
print(rng.normal(loc=10, scale=2, size=3).round(2))
print(rng.uniform(0, 1, size=3).round(3))
print(rng.integers(0, 10, size=5))               # `integers`, not `randint`
print(rng.binomial(n=10, p=0.3, size=3))
print(rng.poisson(lam=4, size=3))
print(rng.choice(["a", "b", "c"], size=5, p=[0.7, 0.2, 0.1]))
print(rng.choice(10, size=3, replace=False))     # sampling without replacement

# SHUFFLING: in place, versus a shuffled copy.
arr = np.arange(10)
rng.shuffle(arr)                                  # modifies arr
print(arr)
print(rng.permutation(10))                        # returns a new array

# PERMUTING PAIRED ARRAYS: shuffle the INDEX, never each array separately.
X, y = np.arange(10), np.arange(10) * 100
perm = rng.permutation(len(X))
X_s, y_s = X[perm], y[perm]                       # correspondence preserved
print(np.all(y_s == X_s * 100))                   # True

# A REPRODUCIBILITY CHECKLIST for a full pipeline.
SEED = 20240101
seeds = {
    "split": SEED,            # train/test partition
    "shuffle": SEED + 1,      # batch order
    "init": SEED + 2,         # model weights
    "augment": SEED + 3,      # data augmentation
}
print(seeds)
# Also required for full determinism: library versions, thread counts
# (OMP_NUM_THREADS), and GPU determinism flags. A seed alone is not enough.

Key Takeaways

• Prefer np.random.default_rng(seed) over the legacy global functions, because global state can be consumed or reset by other code.
• Accept an rng parameter in functions that draw randomness so callers control reproducibility.
• Use SeedSequence.spawn for parallel workers; reusing one seed makes every worker produce the same stream.
• Shuffle a shared index rather than each array separately when rows must stay aligned.
• A single seed does not guarantee reproducibility; library versions, thread counts, and GPU settings also matter.

🧪 Practice

1. Write a bootstrap confidence interval function that takes an optional rng argument and produces identical output across runs when seeded.
2. Create four parallel workers with spawn, verify their streams differ, then show that four separately seeded generators with the same seed do not.
3. Interview: Two runs of the same training script on the same machine give different accuracy despite a seed being set. Name three possible sources. (Hint: how many separate random draws does training make?)

NaN Handling and Masked Arrays

NaN is the float value that represents "not a number", and its behaviour is deliberately contagious: any arithmetic involving it yields NaN. That propagation is a feature, because it prevents a missing value from silently becoming a number, but it makes NaN handling an explicit task.

The property that trips everyone is that NaN is not equal to itself. np.nan == np.nan is False, by IEEE 754 design, because two unknown quantities cannot be asserted equal. Consequently arr == np.nan never finds anything, and you must use np.isnan.

NaN exists only in floating-point arrays. Integer arrays have no representation for it, which is why loading a column with a missing value silently converts it to float. This is the root of a surprising amount of confusion downstream.

For sorting, NaN is treated as larger than everything, so it lands at the end of an ascending sort and, more dangerously, becomes the answer from max. An unguarded np.max on a column with missing values returns NaN, which at least fails loudly, whereas np.argmax returns the NaN's position, which does not.

Masked arrays are the alternative: a separate boolean mask paired with the data, so any dtype can carry missingness and the reason for it is independent of the value. They are cleaner in principle and slower and less supported in practice, so most work uses NaN with explicit handling and pandas for anything richer.

import numpy as np

x = np.array([1.0, 2.0, np.nan, 4.0, np.nan])

# NaN IS CONTAGIOUS and NOT EQUAL TO ITSELF.
print(x.sum(), x.mean())             # nan nan
print(np.nan == np.nan)              # False
print(x == np.nan)                   # all False: this NEVER works
print(np.isnan(x))                   # the correct test

# COUNTING AND REPORTING: always do this before skipping.
print(f"{np.isnan(x).sum()} of {x.size} missing "
      f"({np.isnan(x).mean():.0%})")

# THE nan-PREFIXED AGGREGATIONS skip rather than propagate.
print(np.nansum(x), np.nanmean(x), np.nanstd(x).round(4))
print(np.nanmin(x), np.nanmax(x), np.nanmedian(x))
print(np.nanpercentile(x, 50))

# THE DANGEROUS ASYMMETRY: max fails loudly, argmax does not.
print(np.max(x))                     # nan: obviously wrong, easy to catch
print(np.argmax(x))                  # 2: the NaN's index, silently wrong
print(np.nanargmax(x))               # 3: correct

# SORTING places NaN LAST regardless of direction.
print(np.sort(x))                    # [ 1.  2.  4. nan nan]
print(np.sort(x)[::-1])              # [nan nan  4.  2.  1.] not "largest"

# REMOVING versus FILLING: different questions.
print(x[~np.isnan(x)])                       # drop
print(np.nan_to_num(x, nan=0.0))             # fill with a constant
filled = x.copy()
filled[np.isnan(filled)] = np.nanmean(x)     # mean imputation
print(filled.round(3))
# Imputation changes the variance. Record what you did and how many.

# INF is a separate concept and needs a separate check.
y = np.array([1.0, np.inf, -np.inf, np.nan])
print(np.isnan(y), np.isinf(y), np.isfinite(y))
print(np.nan_to_num(y, nan=0, posinf=1e10, neginf=-1e10))

# ROWS: drop any row containing a NaN, or only fully-missing rows.
M = np.array([[1.0, 2.0], [np.nan, 4.0], [5.0, np.nan], [7.0, 8.0]])
print(M[~np.isnan(M).any(axis=1)])           # complete cases only
print(M[~np.isnan(M).all(axis=1)])           # drop only all-NaN rows

# NaN FORCES FLOAT, which is why integer columns change dtype on load.
ints = np.array([1, 2, 3])
print(ints.dtype, ints.astype(float).dtype)

# MASKED ARRAYS: missingness as metadata, so any dtype can carry it.
mx = np.ma.masked_array([1, 2, -999, 4], mask=[0, 0, 1, 0])
print(mx.mean())                     # 2.333: the sentinel is excluded
print(mx.sum(), mx.count())          # 7 3
print(mx.filled(0))                  # [1 2 0 4] materialize with a fill
# Works on INTEGER data, which NaN cannot. The cost is speed and support:
# most libraries will not accept a masked array.

# CORRELATION with missing data needs explicit pairwise handling.
a = np.array([1.0, 2.0, np.nan, 4.0])
b = np.array([2.0, np.nan, 6.0, 8.0])
both = ~np.isnan(a) & ~np.isnan(b)
print(f"n={both.sum()} usable pairs of {len(a)}")
print(np.corrcoef(a[both], b[both])[0, 1])

Key Takeaways

• NaN propagates through arithmetic and is never equal to itself, so use np.isnan rather than an equality test.
• The nan-prefixed aggregations skip missing values; always report how many were skipped before relying on them.
• np.argmax silently returns the position of a NaN while np.max returns NaN, making the index version the more dangerous one.
• Sorting places NaN at the end regardless of direction, so the "largest" element after a descending sort may be missing.
• Masked arrays support missingness in any dtype but are slower and poorly supported; use NaN or pandas nullable types instead.

🧪 Practice

1. Write a function that reports, for each column of a matrix, the count and percentage of missing values, then the mean computed both ways.
2. Show that np.argmax returns a NaN position and fix it with nanargmax, using an array where the true maximum is elsewhere.
3. Interview: A column of integer identifiers becomes floats after loading a CSV, and some now display in scientific notation. What happened and how would you prevent it? (Hint: what does a missing value require?)

Memory Layout, Contiguity, and Performance

Two arrays with identical shapes, dtypes, and contents can differ in speed by an order of magnitude. The difference is memory layout, and it is invisible until you look for it.

C-contiguous means the last axis varies fastest, so rows are stored consecutively. Fortran-contiguous means the first axis varies fastest, so columns are consecutive. NumPy creates C-contiguous arrays by default, while transposes and some library outputs are Fortran-ordered.

This matters because of the CPU cache. Memory is fetched in cache lines of 64 bytes, so reading eight consecutive float64 values costs one fetch while reading eight scattered values costs eight. Summing along the contiguous axis is therefore several times faster than summing across it, with the same number of arithmetic operations.

A non-contiguous array arises from slicing with a step or transposing. Operations still work, but some routines must make an internal copy first, which appears as an unexplained allocation and slowdown.

The practical discipline is short. Check arr.flags when performance surprises you. Use np.ascontiguousarray before a hot loop. Prefer operations along the last axis for C-ordered data. And be aware that many BLAS routines want Fortran order internally, so a transpose before a matrix multiply is sometimes genuinely faster despite the copy.

import numpy as np
import time

n = 4000
A = np.random.default_rng(0).random((n, n))      # C-contiguous by default

print(A.flags["C_CONTIGUOUS"], A.flags["F_CONTIGUOUS"])   # True False

def timed(label, fn, repeat=3):
    best = min(
        (lambda t0: (fn(), time.perf_counter() - t0)[1])(time.perf_counter())
        for _ in range(repeat)
    )
    print(f"{label:<34} {best * 1000:7.1f} ms")

# SAME ARITHMETIC, DIFFERENT ACCESS ORDER.
timed("sum along axis 1 (contiguous)", lambda: A.sum(axis=1))
timed("sum along axis 0 (strided)", lambda: A.sum(axis=0))
# Identical operation counts. The gap is cache behaviour alone.

# A TRANSPOSE IS FREE BUT CHANGES THE LAYOUT.
At = A.T
print(At.flags["C_CONTIGUOUS"], At.flags["F_CONTIGUOUS"])    # False True
timed("transposed sum axis 1", lambda: At.sum(axis=1))       # now the slow one

# NON-CONTIGUOUS FROM STEPPED SLICING.
strided = A[::2, ::2]
print(strided.flags["C_CONTIGUOUS"])                          # False
timed("strided slice sum", lambda: strided.sum())
packed = np.ascontiguousarray(strided)                        # pay once
timed("packed copy sum", lambda: packed.sum())
# If you will touch the slice repeatedly, compacting it first wins.

# INSPECTING THE FULL FLAG SET when something is unexpectedly slow.
print(A[::2].flags)

# TEMPORARIES dominate chained arithmetic on large arrays.
B = np.random.default_rng(1).random((n, n))
timed("C = A * 2 + B (temporaries)", lambda: A * 2 + B)

C = np.empty_like(A)
def inplace():
    np.multiply(A, 2, out=C)         # no allocation
    np.add(C, B, out=C)              # no allocation
timed("out= chain (no temporaries)", inplace)

# DTYPE IS A PERFORMANCE LEVER TOO: half the bytes, half the cache misses.
A32 = A.astype(np.float32)
timed("float64 sum", lambda: A.sum())
timed("float32 sum", lambda: A32.sum())
print(f"float64 {A.nbytes/1e6:.0f} MB vs float32 {A32.nbytes/1e6:.0f} MB")

# MEASURE, DO NOT GUESS.
print(f"array {A.nbytes / 1e6:.0f} MB; typical L3 cache is 8-32 MB")
# Once an array exceeds cache, layout effects become dominant.

  WHY LAYOUT CHANGES SPEED

  A 3x4 C-contiguous array in memory:
    [ a00 a01 a02 a03 | a10 a11 a12 a13 | a20 a21 a22 a23 ]
      <----- row 0 ---->  <----- row 1 ---->

  cache line = 64 bytes = 8 float64 values

  sum(axis=1)  walks a00,a01,a02,a03  -> all within one or two lines
               ONE fetch serves many elements

  sum(axis=0)  walks a00,a10,a20      -> each is 32 bytes apart here,
               and thousands of bytes apart in a large array
               -> one cache line fetched PER ELEMENT, then discarded

  DIAGNOSIS
    arr.flags["C_CONTIGUOUS"]   rows are consecutive
    arr.flags["F_CONTIGUOUS"]   columns are consecutive
    neither                     a strided view; consider
                                np.ascontiguousarray before hot loops

Key Takeaways

• C order stores rows consecutively and Fortran order stores columns; NumPy defaults to C order.
• Aggregating along the contiguous axis is several times faster because each cache line serves many elements.
• Transposes and stepped slices produce non-contiguous views, which some routines silently copy before use.
• Chained arithmetic allocates a temporary per operation; out= and in-place operators remove them.
• Check arr.flags first when performance is unexpectedly poor, and compact with ascontiguousarray before repeated access.

🧪 Practice

1. Time summing a 4000-by-4000 array along each axis, then repeat on its transpose and explain the reversal.
2. Take a stepped slice of a large array, compare repeated aggregation on it against a contiguous copy, and find the break-even repetition count.
3. Interview: Two identically shaped float64 arrays give very different timings for the same reduction. What would you inspect? (Hint: shape and dtype are not the whole description of an array.)

Structured and Record Arrays

Not all tabular data is homogeneous. A structured array gives each element a compound dtype with named fields of possibly different types, producing something that looks like a lightweight dataframe built directly on NumPy.

The concept is a C struct. Each element occupies a fixed-size block laid out as named fields, and the array is a contiguous sequence of those blocks. You index by field name to get a column and by position to get a row, with no Python objects involved.

The layout question is row-major versus column-major in a different guise. A structured array is an array of structs: all fields for row 0, then all for row 1. Pandas is the opposite, a struct of arrays, storing each column contiguously. Column-wise storage is much faster for analytics, which reads whole columns, and that is why pandas rather than structured arrays became the tabular standard.

Structured arrays remain genuinely useful in three situations: reading fixed-width binary formats where the layout is dictated, interoperating with C or Fortran code that expects structs, and building small typed records without a pandas dependency.

Record arrays add attribute access, so arr.name works alongside arr["name"]. The convenience costs a performance penalty on every access, so prefer plain structured arrays in code that runs hot.

import numpy as np

# DEFINING a compound dtype: named fields with individual types.
dt = np.dtype([
    ("customer_id", "i4"),        # int32
    ("name", "U20"),              # unicode, max 20 chars, FIXED width
    ("revenue", "f8"),            # float64
    ("active", "?"),              # bool
])

data = np.array([
    (101, "Alice", 3400.50, True),
    (102, "Bob", 1200.00, False),
    (103, "Carla", 9800.25, True),
], dtype=dt)

print(data.dtype.itemsize, "bytes per record")    # 4 + 80 + 8 + 1, padded
print(data.shape)                                  # (3,) one axis of records

# ACCESS: by FIELD for a column, by POSITION for a row.
print(data["revenue"])              # [3400.5 1200. 9800.25], a real float64
print(data[0])                      # (101, 'Alice', 3400.5, True)
print(data[0]["name"])              # Alice

# VECTORIZED OPERATIONS work per field, at full NumPy speed.
print(data["revenue"].mean())
print(data[data["active"]]["name"])            # boolean mask over records
print(np.sort(data, order=["active", "revenue"])["name"])   # multi-key sort

# WRITING through a field view updates the parent.
data["revenue"] *= 1.1
print(data["revenue"].round(2))

# THE FIXED-WIDTH STRING TRAP.
short = np.array([("ab",)], dtype=[("s", "U2")])
short["s"] = "abcdef"
print(short["s"])                    # 'ab': SILENTLY TRUNCATED, no warning

# THE PRIMARY REAL USE: parsing a fixed-layout binary file.
binary_dt = np.dtype([
    ("timestamp", "<u8"),            # little-endian uint64
    ("sensor", "<u2"),
    ("value", "<f4"),
])
raw = np.zeros(3, dtype=binary_dt).tobytes()     # stand-in for file bytes
parsed = np.frombuffer(raw, dtype=binary_dt)     # ZERO-COPY parse
print(parsed.dtype.names, parsed.shape)
# np.fromfile(path, dtype=binary_dt) reads a whole file this way.

# RECORD ARRAYS add attribute access at a per-access cost.
rec = data.view(np.recarray)
print(rec.name, rec.revenue.sum().round(2))      # attribute style

# LAYOUT: array of structs versus struct of arrays.
print(data.dtype.fields)             # each field's dtype and BYTE OFFSET

# CONVERTING TO PANDAS is one call, and usually the right move.
import pandas as pd
df = pd.DataFrame(data)
print(df.dtypes.to_dict())
print(df.to_records(index=False).dtype.names)    # and back again

  ARRAY OF STRUCTS VS STRUCT OF ARRAYS

  STRUCTURED ARRAY (numpy)        DATAFRAME (pandas)
  row-major: interleaved          column-major: one block per column

  [id|name|rev|act]               id   : [101][102][103]
  [id|name|rev|act]               name : [Alice][Bob][Carla]
  [id|name|rev|act]               rev  : [3400][1200][9800]
                                  act  : [T][F][T]

  fast: read a whole ROW          fast: read a whole COLUMN
        (one contiguous block)          (one contiguous block)
  slow: mean of one column        slow: assembling a single row
        (stride over the gaps)

  Analytics reads columns, which is why pandas won for tabular work.
  Structured arrays win for binary file layouts and C interop.

Key Takeaways

• A structured array gives elements a compound dtype with named fields, an array of C-style structs.
• Index by field name for a column and by position for a row; field access returns a genuine typed array.
• Row-major interleaving makes column aggregation slower than in pandas, which stores each column contiguously.
• Fixed-width string fields truncate silently when assigned a longer value.
• The strongest use cases are fixed-layout binary parsing with frombuffer or fromfile and interop with compiled code.

🧪 Practice

1. Define a structured dtype for sensor readings with a timestamp, an integer sensor id, and a float value, then compute the mean value per sensor without converting to pandas.
2. Demonstrate the fixed-width string truncation and fix it by widening the field.
3. Interview: Why is computing a column mean typically slower on a structured array than on a pandas column? (Hint: where does the next value of that column live in memory?)

Zero-Copy Interoperability with Other Libraries

A realistic pipeline crosses several libraries: read with Arrow, transform in pandas, train in PyTorch, serve through a C extension. If each boundary copied the data, the copying would cost more than the computation.

The enabling idea is a shared memory protocol. Rather than each library defining its own container, they agree on how to describe a block of memory: its address, dtype, shape, and strides. Any library that speaks the protocol can wrap another's buffer without moving a byte.

Three protocols matter. The buffer protocol is Python's built-in mechanism, which is how memoryview works. The __array_interface__ and its C counterpart are NumPy's. DLPack is the cross-framework tensor standard used by PyTorch, TensorFlow, JAX, and CuPy.

Apache Arrow deserves separate mention because it defines a columnar memory format rather than just an exchange protocol. When pandas uses the Arrow backend and Polars uses Arrow natively, passing data between them is a pointer handoff.

Zero-copy carries obligations. The two views share memory, so a write through one is visible through the other. The original must stay alive or the borrower holds a dangling reference. Some conversions are only zero-copy under conditions, such as contiguity or absence of nulls, and they silently fall back to copying. Verify rather than assume.

import numpy as np

a = np.arange(12, dtype=np.float32).reshape(3, 4)

# THE PROTOCOL: a description of memory, not the memory itself.
info = a.__array_interface__
print(info["shape"], info["typestr"], info["strides"])
print(f"buffer address: {info['data'][0]:#x}, writable: {not info['data'][1]}")

# memoryview: Python's built-in buffer protocol, zero-copy.
mv = memoryview(a)
print(mv.shape, mv.format, mv.nbytes)
back = np.asarray(mv)
print(np.shares_memory(a, back))          # True: same bytes

# THE VERIFICATION TOOL. Use it rather than trusting documentation.
b = a[::2]
print(np.shares_memory(a, b))             # True: a view
print(np.shares_memory(a, a.copy()))      # False: a copy

# PANDAS: zero-copy in one direction, conditionally in the other.
import pandas as pd
df = pd.DataFrame(a, columns=list("wxyz"))
values = df.to_numpy()
print(np.shares_memory(a, values))
# Single-dtype frames can share; MIXED dtypes must copy, because a
# 2-D array needs one dtype and the frame has several.
mixed = pd.DataFrame({"n": [1, 2], "s": ["a", "b"]})
print(mixed.to_numpy().dtype)             # object: a copy, and slow

# A SINGLE COLUMN is usually zero-copy.
col = df["w"].to_numpy()
print(np.shares_memory(col, a))

# ARROW: the columnar interchange standard.
import pyarrow as pa
arrow_arr = pa.array(np.arange(5, dtype=np.int64))
back_np = arrow_arr.to_numpy(zero_copy_only=True)   # raises if it would copy
print(np.shares_memory(back_np, back_np))
# zero_copy_only=True turns a silent copy into an explicit error, which is
# exactly what you want in a performance-critical path.

nullable = pa.array([1, 2, None])
try:
    nullable.to_numpy(zero_copy_only=True)
except pa.ArrowInvalid as e:
    print("copy required:", str(e)[:60])
# Nulls need a validity bitmap that NumPy has no concept of.

# TORCH: from_numpy shares memory; torch.tensor copies.
try:
    import torch
    t = torch.from_numpy(a)               # SHARES
    t[0, 0] = -1.0
    print(a[0, 0])                        # -1.0: the numpy array changed
    t2 = torch.tensor(a)                  # COPIES
    print(t.numpy().__array_interface__["data"][0] ==
          a.__array_interface__["data"][0])
except ImportError:
    print("torch not installed; from_numpy shares, torch.tensor copies")

# THE LIFETIME HAZARD.
def leaky():
    tmp = np.arange(1000)
    return memoryview(tmp)                # tmp is referenced, so it survives
mv2 = leaky()
print(len(mv2))                           # fine in Python, NOT in C code
# In compiled extensions, releasing the owner while a borrower lives is a
# use-after-free. Keep an explicit reference to the owner.

Boundary	Zero-copy call	Copies instead
NumPy to Torch	torch.from_numpy(a)	torch.tensor(a)
Torch to NumPy	t.numpy()	np.array(t)
Arrow to NumPy	to_numpy(zero_copy_only=True)	any array with nulls
pandas to NumPy	df.to_numpy() single dtype	mixed dtypes
Any to any	DLPack __dlpack__	across device boundaries

Key Takeaways

• Libraries interoperate by agreeing on a description of memory, so a handoff can be a pointer rather than a copy.
• Use np.shares_memory to verify zero-copy rather than trusting an API's reputation.
• Zero-copy means shared mutation: writing through one view is visible through the other.
• Conversions fall back to copying under conditions such as mixed dtypes, non-contiguity, or the presence of nulls.
• Arrow's nulls use a validity bitmap that NumPy cannot represent, which forces a copy; zero_copy_only=True makes that explicit.

🧪 Practice

1. Create a NumPy array, convert it to a PyTorch tensor two different ways, and determine which shares memory by mutating each.
2. Build a pandas DataFrame with a single dtype and one with mixed dtypes, then check with np.shares_memory which to_numpy call copies.
3. Interview: An Arrow-to-NumPy conversion that was instant becomes slow after a schema change. What likely changed? (Hint: what does a nullable column require that a NumPy array cannot express?)

Series and DataFrame Internals

pandas is often described as "NumPy with labels", which is accurate enough to be useful and incomplete enough to cause bugs. Understanding what sits underneath explains both the conveniences and the surprises.

A Series is a one-dimensional labelled array: a values array plus an index, which is itself an array of labels. The index is not decoration. It participates in arithmetic, drives joins, and determines what happens when two objects of different length are combined.

A DataFrame is a dict-like collection of Series that share one index. The critical structural fact is that it is stored column-wise: each column is a contiguous typed array. Column operations are therefore fast and row operations are slow, which is the opposite of the intuition people bring from spreadsheets.

Historically pandas grouped same-dtype columns into shared two-dimensional "blocks", which produced confusing copy behaviour. Modern pandas is moving toward one array per column, which is simpler and interoperates better with Arrow.

The practical consequence deserves emphasis: iterating over rows is pathologically slow. Each row must be assembled from separate column arrays into a new object. A row loop over a million-row frame can take minutes where a vectorized expression takes milliseconds.

import pandas as pd
import numpy as np

# A SERIES is values plus an index.
s = pd.Series([10, 20, 30], index=["a", "b", "c"], name="revenue")
print(s.values, type(s.values))       # the underlying numpy array
print(s.index)                        # Index(['a','b','c'], dtype='object')
print(s.name, s.dtype, s.shape)

# A DATAFRAME is aligned columns sharing one index.
df = pd.DataFrame({
    "region": ["eu", "us", "eu"],
    "revenue": [100.0, 250.0, 175.0],
    "units": [4, 9, 6],
})
print(df.dtypes)                      # one dtype PER COLUMN
print(df.index)                       # RangeIndex(start=0, stop=3, step=1)

# COLUMN-WISE STORAGE: each column is its own contiguous array.
print(df["revenue"].values.flags["C_CONTIGUOUS"])     # True
print(df.memory_usage(deep=True))     # per column, deep counts string objects

# WHY ROW ITERATION IS SLOW: measure it once and never do it again.
import time
big = pd.DataFrame({"a": np.arange(200_000), "b": np.arange(200_000)})

t0 = time.perf_counter()
total = 0
for _, row in big.iterrows():         # builds a Series PER ROW
    total += row["a"] * row["b"]
print(f"iterrows      : {(time.perf_counter()-t0)*1000:8.1f} ms")

t0 = time.perf_counter()
total2 = (big["a"] * big["b"]).sum()  # two array multiplications
print(f"vectorized    : {(time.perf_counter()-t0)*1000:8.1f} ms")
print(total == total2)

# iterrows ALSO LOSES DTYPES: a row is a Series, so one dtype for all.
print(df.iloc[0].dtype)               # object: region forced everything up
print(next(df.itertuples()))          # itertuples preserves types, and is
                                      # roughly 10x faster than iterrows

# INSPECTION, the first four calls on any new frame.
print(df.shape, len(df), df.columns.tolist())
df.info()                             # dtypes, non-null counts, memory
print(df.describe())                  # numeric summary only by default
print(df.describe(include="all"))     # adds categorical summaries

# ADDING A COLUMN is cheap; adding a ROW is not.
df["revenue_per_unit"] = df["revenue"] / df["units"]     # one new array
# df.loc[len(df)] = ...   in a loop REALLOCATES every column each time.
# Correct pattern: collect rows in a list, then concat ONCE.
rows = [{"region": "ap", "revenue": 90.0, "units": 3} for _ in range(3)]
df2 = pd.concat([df, pd.DataFrame(rows)], ignore_index=True)
print(len(df2))

  HOW A DATAFRAME IS ACTUALLY STORED

  what you see:                 what exists in memory:

    region  revenue  units      region : ['eu','us','eu']      object array
  0 eu      100.0    4          revenue: [100.0, 250.0, 175.0] float64 array
  1 us      250.0    9          units  : [4, 9, 6]             int64 array
  2 eu      175.0    6          index  : [0, 1, 2]             RangeIndex

  COLUMN access  df["revenue"]   -> hand back an existing array   FAST
  ROW access     df.iloc[0]      -> build a NEW Series by reading
                                    one element from EACH column   SLOW
                                    and upcast to a common dtype

  This is why a 1M-row iterrows loop takes minutes and the equivalent
  column expression takes milliseconds.

Key Takeaways

• A Series is values plus an index, and the index actively participates in alignment rather than merely labelling rows.
• A DataFrame stores one contiguous typed array per column, so column operations are fast and row assembly is slow.
• iterrows builds a new Series per row and upcasts to a common dtype, losing both speed and types; itertuples is faster and type-preserving.
• Appending rows one at a time reallocates every column; collect rows and concatenate once.
• info and memory_usage(deep=True) are the first diagnostics to run on an unfamiliar frame.

🧪 Practice

1. Build a 100,000-row frame and compute a product of two columns using iterrows, itertuples, and a vectorized expression, timing all three.
2. Show that a row extracted with iloc has a single dtype even when the frame has mixed types, and explain the consequence for downstream code.
3. Interview: A script that appends one row per iteration to a DataFrame takes hours on 50,000 records. What is happening and what is the fix? (Hint: what must happen to each column array on every append?)

Index Objects and MultiIndex

The index is pandas' most distinctive feature and the one most often treated as a nuisance. Used deliberately, it gives fast lookup, automatic alignment, and a natural representation of grouped data.

An Index is an immutable array of labels. Immutability matters because indexes are shared between objects: slicing a frame hands the same index to the result rather than copying it. Indexes need not be unique, though non-unique indexes make lookups return multiple rows and turn some operations ambiguous.

The performance property worth knowing is that a unique index maintains a hash table, giving O(1) label lookup. A monotonic index supports binary search slicing in O(log n). An index that is neither requires a linear scan, so the same loc call can vary by orders of magnitude depending on properties you cannot see in the output.

A MultiIndex carries several levels per row, representing hierarchical or naturally multi-dimensional data such as (country, year) or (ticker, date). It is what groupby on several keys returns, and what makes panel data expressible in a two-dimensional structure.

MultiIndexes reward learning three operations: xs for a cross-section at one level, swaplevel with sort_index for reordering, and the fact that slicing a MultiIndex requires it to be sorted. The unsorted case raises a performance warning or an error, and sorting once fixes it.

import pandas as pd
import numpy as np

# INDEX PROPERTIES drive lookup performance.
idx = pd.Index(["c", "a", "b"], name="code")
print(idx.is_unique, idx.is_monotonic_increasing)     # True False
print(idx.dtype, idx.name)
# idx[0] = "z"    -> TypeError: Index does not support mutable operations

rng = np.random.default_rng(0)
n = 500_000
shuffled = pd.Series(rng.random(n), index=rng.permutation(n))
sorted_idx = shuffled.sort_index()
print(shuffled.index.is_monotonic_increasing,
      sorted_idx.index.is_monotonic_increasing)
# Sorted indexes slice by binary search; unsorted ones scan.

# SETTING AN INDEX turns a column into the lookup key.
df = pd.DataFrame({
    "country": ["FR", "FR", "DE", "DE"],
    "year": [2023, 2024, 2023, 2024],
    "gdp": [2.8, 2.9, 4.1, 4.2],
    "pop": [68, 68, 84, 84],
})
by_country = df.set_index("country")
print(by_country.loc["FR"])          # all FR rows, by label
print(by_country.reset_index().head(2))       # and back to a column

# A MULTIINDEX: several levels per row.
panel = df.set_index(["country", "year"]).sort_index()
print(panel)
print(panel.index.names, panel.index.nlevels)

# SELECTING at each level.
print(panel.loc["FR"])                        # all years for FR
print(panel.loc[("FR", 2024)])                # one exact row
print(panel.loc[("FR", 2024), "gdp"])         # one value

# xs takes a cross-section at ANY level, not just the first.
print(panel.xs(2024, level="year"))           # both countries, one year

# SLICING REQUIRES A SORTED INDEX.
unsorted = df.set_index(["country", "year"])
try:
    unsorted.loc[("DE", 2023):("FR", 2023)]
except Exception as e:
    print(type(e).__name__)                   # UnsortedIndexError
print(panel.loc[("DE", 2023):("FR", 2023)].shape)      # works once sorted

# IndexSlice for readable multi-level slicing.
sl = pd.IndexSlice
print(panel.loc[sl[:, 2024], :])              # all countries, year 2024

# RESHAPING BETWEEN INDEX AND COLUMNS.
print(panel["gdp"].unstack("year"))           # years become columns
print(panel["gdp"].unstack("year").stack())   # and back

# swaplevel changes the hierarchy; re-sort afterwards.
print(panel.swaplevel().sort_index().head(2))

# GROUPBY PRODUCES A MULTIINDEX, which is why this matters in practice.
agg = df.groupby(["country", "year"])["gdp"].sum()
print(type(agg.index).__name__)               # MultiIndex

# Flatten it when writing out, since most consumers expect flat columns.
print(agg.reset_index().head(2))

# ALIGNMENT is the index's real job: operations match on LABELS.
a = pd.Series([1, 2, 3], index=["x", "y", "z"])
b = pd.Series([10, 20], index=["y", "z"])
print(a + b)                                  # x is NaN: no match in b

Key Takeaways

• An Index is immutable and shared, which is why slicing does not copy it.
• Unique indexes get O(1) hash lookup and monotonic ones get O(log n) binary search; unsorted non-unique indexes fall back to a linear scan.
• A MultiIndex represents hierarchical keys and is what grouping on several columns returns.
• Slicing a MultiIndex requires sorting it first, which sort_index does once.
• Use xs for a cross-section at any level and IndexSlice for readable multi-level selection.

🧪 Practice

1. Build a frame indexed by country and year, then extract all rows for one year using both xs and IndexSlice.
2. Demonstrate the unsorted MultiIndex slicing error and fix it, then measure lookup time before and after sorting a 500,000-row index.
3. Interview: The same loc lookup is fast on one frame and slow on another with identical shape. What differs? (Hint: what does the index need in order to hash or binary-search?)

Selection with loc, iloc, and Boolean Masks

Selection is the operation you perform most and the one with the most overlapping syntaxes. Learning the small, unambiguous subset is worth more than knowing all of them.

Two accessors cover everything. loc selects by label, and its slices are inclusive of the endpoint, because labels have no notion of "one past the end". iloc selects by integer position, and its slices are exclusive, matching Python. That asymmetry is deliberate and is the detail most worth memorising.

Bare bracket indexing is convenient and ambiguous. df["x"] selects a column, df[0:2] selects rows, and df[mask] selects rows. The same syntax means three different things depending on what you pass. In code that others will read, prefer the explicit accessors.

The real power of loc is that it takes a row selector and a column selector together, and either can be a label, a list, a slice, a boolean array, or a callable. The callable form is what makes method chains possible, because it defers evaluation to the intermediate frame rather than the original.

Assignment must also go through loc or iloc. Chained assignment such as df[df.x > 0]["y"] = 1 operates on a temporary and either fails silently or raises a warning, which the copy-on-write topic covers in detail.

import pandas as pd
import numpy as np

df = pd.DataFrame(
    {"revenue": [100, 250, 175, 320], "units": [4, 9, 6, 11],
     "region": ["eu", "us", "eu", "ap"]},
    index=["r1", "r2", "r3", "r4"],
)

# loc IS LABEL-BASED and its slices INCLUDE the endpoint.
print(df.loc["r2"])                     # one row, as a Series
print(df.loc["r1":"r3"])                # THREE rows: r3 is included
print(df.loc[["r1", "r4"], ["revenue"]])       # explicit rows and columns
print(df.loc["r2", "revenue"])                  # a single scalar

# iloc IS POSITION-BASED and its slices EXCLUDE the endpoint.
print(df.iloc[0:3])                     # three rows: positions 0, 1, 2
print(df.iloc[-1])                      # last row; loc cannot do this
print(df.iloc[[0, 2], [0, 1]])          # positional row and column lists

# THE ASYMMETRY, stated plainly.
print(len(df.loc["r1":"r3"]), len(df.iloc[0:3]))       # 3 3, coincidentally
print(len(df.loc["r1":"r2"]), len(df.iloc[0:2]))       # 2 2
# loc["r1":"r2"] covers r1,r2. iloc[0:2] covers positions 0,1. Same rows,
# different reasoning. With an integer index these diverge visibly.
num = df.reset_index(drop=True)
print(num.loc[0:2].shape[0], num.iloc[0:2].shape[0])   # 3 and 2

# BOOLEAN MASKS, the workhorse.
mask = df["revenue"] > 150
print(df[mask])                         # bracket form
print(df.loc[mask])                     # explicit form, same result
print(df.loc[mask, "region"])           # mask the rows, name the columns

# COMBINING conditions: & | ~ with parentheses, as in NumPy.
print(df.loc[(df["revenue"] > 150) & (df["region"] == "eu")])
print(df.loc[df["region"].isin(["eu", "ap"])])
print(df.loc[df["revenue"].between(150, 300)])

# CALLABLES defer evaluation, which is what makes chaining work.
result = (
    df.assign(margin=lambda d: d["revenue"] / d["units"])
      .loc[lambda d: d["margin"] > 25]          # refers to the NEW column
      .sort_values("margin", ascending=False)
)
print(result)
# Without the lambda you would need an intermediate variable, because the
# original df has no `margin` column.

# ASSIGNMENT must go through a single loc call.
df.loc[df["region"] == "eu", "revenue"] *= 1.1        # correct
# df[df["region"] == "eu"]["revenue"] *= 1.1          # chained: do NOT

# SCALAR ACCESS: at and iat are faster for a single cell.
print(df.at["r1", "units"], df.iat[0, 1])

# SELECTING COLUMNS BY TYPE or by name pattern.
print(df.select_dtypes(include="number").columns.tolist())
print(df.filter(like="re").columns.tolist())           # revenue, region
print(df.filter(regex="^r").columns.tolist())

# query: a string expression, readable for long conditions.
print(df.query("revenue > 150 and region == 'eu'"))
threshold = 150
print(df.query("revenue > @threshold"))                # @ for local variables

Task	Use	Note
One column	df["col"]	returns a Series
Several columns	df[["a","b"]]	returns a DataFrame
Rows by label	df.loc["a":"c"]	endpoint included
Rows by position	df.iloc[0:3]	endpoint excluded
Rows and columns	df.loc[mask, "col"]	one call, safe for assignment
Single cell	df.at[r, c]	fastest scalar access
Inside a chain	df.loc[lambda d: ...]	sees the intermediate frame

Key Takeaways

• loc is label-based with inclusive slice endpoints; iloc is position-based with exclusive endpoints.
• Bare bracket indexing means different things for strings, slices, and masks, so prefer the explicit accessors in shared code.
• loc takes row and column selectors in one call, which is also the only safe form for assignment.
• A callable selector is evaluated against the intermediate frame, which is what makes method chaining possible.
• at and iat are the fast path for single-cell access.

🧪 Practice

1. Using a frame with an integer index starting at zero, show a case where loc[0:2] and iloc[0:2] return different numbers of rows.
2. Write a single chained expression that adds a computed column, filters on it, and sorts by it, without any intermediate variables.
3. Interview: Why does pandas make loc slices inclusive when Python slices are exclusive? (Hint: given a label, what would "the label one past it" mean?)

dtypes: Numeric, Object, Categorical, Nullable, Datetime

A column's dtype determines what operations are legal, how much memory it uses, and how missing values behave. Getting dtypes right at load time prevents a long tail of downstream problems.

The default numeric types are NumPy's int64 and float64. Their defining limitation is that int64 cannot represent missing values, so any column with a gap is read as float. That is why identifiers sometimes appear as 1.0000000e+08 after a CSV load.

object is the dtype that should make you suspicious. It means the column holds pointers to arbitrary Python objects, typically strings. Every operation loops in Python and every element carries object overhead, so an object column uses roughly ten times the memory of the equivalent categorical and runs far slower.

Nullable dtypes, written with a capital letter such as Int64 and boolean, add a separate mask so integers and booleans can hold missing values. They fix the integer-to-float problem at the cost of slightly less mature library support.

Categorical stores a small set of distinct values as integer codes plus a lookup table. For a column with few distinct values relative to its length, the memory saving is dramatic, and grouping becomes faster.

Datetime types are essential rather than optional. A date stored as a string cannot be compared, differenced, or resampled. Converting once at load time with an explicit format is both faster and safer than letting pandas infer per row.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "user_id": [1, 2, 3, None],           # a gap forces float64
    "country": ["FR", "DE", "FR", "FR"],  # object by default
    "signed_up": ["2024-01-15", "2024-02-20", "2024-01-15", "2024-03-01"],
    "active": [True, False, True, None],  # object, because of the None
    "score": [0.91, 0.55, 0.73, 0.88],
})
print(df.dtypes)

# THE INTEGER PROBLEM: one missing value turns an id into a float.
print(df["user_id"])                      # 1.0, 2.0, 3.0, NaN
print(df["user_id"].astype("Int64"))      # nullable: 1, 2, 3, <NA>

# NULLABLE BOOLEANS behave the same way.
print(df["active"].astype("boolean").dtype)

# DATETIME: convert once, explicitly.
df["signed_up"] = pd.to_datetime(df["signed_up"], format="%Y-%m-%d")
print(df["signed_up"].dtype)
print(df["signed_up"].dt.month.tolist())      # the .dt accessor unlocks this
print((df["signed_up"].max() - df["signed_up"].min()).days)
# format= is both faster and safer: inference can silently read 03/04 as
# either 3 April or 4 March depending on the rows it happens to see.
print(pd.to_datetime(["2024-13-01"], errors="coerce"))     # NaT, not a crash

# CATEGORICAL: integer codes plus a lookup table.
df["country"] = df["country"].astype("category")
print(df["country"].cat.categories.tolist())   # ['DE', 'FR']
print(df["country"].cat.codes.tolist())        # [1, 0, 1, 1]

# ORDERED categoricals support comparison, which plain strings do not.
sizes = pd.Series(["small", "large", "medium"], dtype="category")
sizes = sizes.cat.set_categories(["small", "medium", "large"], ordered=True)
print(sizes > "small")                         # [False, True, True]

# MEMORY: the case for categorical, measured.
n = 500_000
s_obj = pd.Series(np.random.default_rng(0).choice(["eu", "us", "ap"], n))
s_cat = s_obj.astype("category")
print(f"object     : {s_obj.memory_usage(deep=True) / 1e6:7.1f} MB")
print(f"categorical: {s_cat.memory_usage(deep=True) / 1e6:7.1f} MB")

# DOWNCASTING numerics when the range allows it.
big = pd.DataFrame({"count": np.random.default_rng(0).integers(0, 200, n)})
print(f"int64: {big['count'].memory_usage(deep=True)/1e6:.1f} MB")
small = big["count"].astype(np.int16)          # range fits; verify first
print(f"int16: {small.memory_usage(deep=True)/1e6:.1f} MB")
print(big["count"].min(), big["count"].max())  # ALWAYS check before casting

# DECLARE DTYPES AT LOAD TIME rather than repairing afterwards.
schema = {"user_id": "Int64", "country": "category", "score": "float32"}
# pd.read_csv(path, dtype=schema, parse_dates=["signed_up"])

# CONVERSION with error control.
raw = pd.Series(["1", "2", "not a number"])
print(pd.to_numeric(raw, errors="coerce"))     # NaN for the bad one
# errors="raise" is the default and the right choice when input is trusted.

# THE GENERAL REPAIR PASS.
print(df.convert_dtypes().dtypes)              # infer best nullable types

dtype	Holds missing	Memory per element	Use for
int64	no	8 bytes	complete integer data
Int64	yes	8 bytes + mask bit	integers with gaps
float64	yes (NaN)	8 bytes	measurements
object	yes	~50+ bytes	avoid; a signal to fix
category	yes	1-4 bytes + table	low-cardinality labels
datetime64[ns]	yes (NaT)	8 bytes	any date or timestamp
bool / boolean	no / yes	1 byte	flags

Key Takeaways

• A missing value in an integer column silently converts it to float; nullable Int64 avoids this.
• object dtype means Python objects with per-element overhead and is usually a sign the column should be categorical or string-typed.
• Categorical stores integer codes and a lookup table, cutting memory dramatically for low-cardinality columns and speeding up grouping.
• Parse dates once with an explicit format, because inference is slower and can misread ambiguous day-month orders.
• Declare dtypes in read_csv rather than repairing them after the fact.

🧪 Practice

1. Load a CSV with an identifier column containing blanks, observe the float conversion, and fix it with a nullable integer dtype.
2. Measure the memory of a 500,000-row string column before and after converting to categorical, and compute the ratio.
3. Interview: An identifier column displays as 1.00000e+08 in output. What happened during loading and how would you prevent it? (Hint: what forces an integer column to become floating point?)

Missing Data Semantics: NaN, NaT, and pd.NA

pandas has three missing-value markers with different types and different behaviour, which is a historical accident you have to work around rather than a design you can appreciate.

np.nan is a float, inherited from NumPy. It propagates through arithmetic and is not equal to itself. NaT is the datetime equivalent. pd.NA is the newer, dtype-agnostic marker used by nullable extension types, and it propagates through comparisons as well as arithmetic.

That last difference is the one to internalise. With np.nan, a comparison returns False, so a filter silently excludes missing rows. With pd.NA, a comparison returns NA, which is three-valued logic: the honest answer to "is an unknown value greater than five" is "unknown". The nullable behaviour is more correct and will surprise code written against the old one.

Crucially, isna detects all three, so the correct test never depends on which marker you have. Never compare against a missing marker directly.

The deeper point is that missingness has causes, and the right treatment depends on which. Missing completely at random can be dropped with only a loss of power. Missing at random can be imputed from observed variables. Missing not at random, where the missingness depends on the unobserved value itself, biases every method and cannot be fixed by imputation.

import pandas as pd
import numpy as np

# THREE MARKERS, THREE TYPES.
print(type(np.nan), type(pd.NaT), type(pd.NA))
print(pd.isna([np.nan, pd.NaT, pd.NA, None, 1]))     # detects them all

# THE COMPARISON DIFFERENCE, which changes filter results.
old = pd.Series([1, 2, np.nan], dtype="float64")
new = pd.Series([1, 2, pd.NA], dtype="Int64")
print((old > 1).tolist())              # [False, True, False] <- silent drop
print((new > 1).tolist())              # [False, True, <NA>]  <- honest
print(old[old > 1].shape, new[(new > 1).fillna(False)].shape)
# With nullable dtypes a boolean mask containing NA must be resolved
# explicitly, which forces you to decide what an unknown row should do.

# NEVER compare to a marker directly.
print(np.nan == np.nan, pd.NA == pd.NA)     # False, and <NA>
print(old.isna().sum())                     # the correct count

df = pd.DataFrame({
    "age": [25, np.nan, 40, 31],
    "income": [50_000, 62_000, np.nan, np.nan],
    "city": ["Paris", None, "Berlin", "Lyon"],
})

# THE FIRST STEP IS ALWAYS TO QUANTIFY, per column and as a pattern.
print(df.isna().sum())
print((df.isna().mean() * 100).round(1))          # percentage per column
print(df.isna().sum(axis=1).value_counts())       # rows by missing count

# CO-OCCURRENCE: which columns go missing TOGETHER? A strong signal of
# a structural cause rather than random loss.
print(df.isna().corr().round(2))

# DROPPING: be specific about the axis and the rule.
print(df.dropna().shape)                        # any missing anywhere
print(df.dropna(subset=["age"]).shape)          # only when age is missing
print(df.dropna(thresh=2).shape)                # keep rows with >=2 values
print(df.dropna(axis=1, how="all").shape)       # drop all-empty COLUMNS

# FILLING: each strategy encodes a different assumption.
print(df["age"].fillna(df["age"].median()))     # centre; shrinks variance
print(df["city"].fillna("unknown"))             # an explicit category
ts = pd.Series([1.0, np.nan, np.nan, 4.0])
print(ts.ffill().tolist())                      # carry forward: time series
print(ts.interpolate().tolist())                # linear: ordered numerics
# ffill on non-time-ordered data invents structure that is not there.

# GROUP-WISE IMPUTATION is usually better than a global constant.
g = pd.DataFrame({"grp": ["a", "a", "b", "b"], "v": [1.0, np.nan, 10.0, np.nan]})
g["v"] = g.groupby("grp")["v"].transform(lambda s: s.fillna(s.mean()))
print(g)

# ALWAYS FLAG WHAT YOU IMPUTED. The flag is often predictive in itself,
# because missingness frequently carries information.
df["income_was_missing"] = df["income"].isna()
df["income"] = df["income"].fillna(df["income"].median())
print(df[["income", "income_was_missing"]])

# SENTINEL VALUES masquerading as data: the most dangerous case, because
# nothing reports them as missing.
sus = pd.Series([25, 30, -999, 45, 9999])
print(sus.mean())                               # 1820.0: meaningless
print(sus.describe())                           # min/max expose the sentinels
cleaned = sus.replace([-999, 9999], np.nan)
print(cleaned.mean())
# Check min, max, and value_counts on every numeric column before trusting
# any statistic. -1, 0, 999, 9999, and 1900-01-01 are common sentinels.

  THE THREE MECHANISMS, AND WHAT EACH ALLOWS

  MCAR  missing completely at random
        the gap is unrelated to anything
        -> dropping is unbiased, you only lose power

  MAR   missing at random, GIVEN observed variables
        income missing more often for younger users, and you observe age
        -> imputation conditioned on age is valid

  MNAR  missing not at random
        high earners decline to state income
        -> the value itself drives its own absence
        -> NO imputation fixes this; every estimate is biased
        -> you need external data or an explicit selection model

  You cannot distinguish MAR from MNAR using the data alone, because the
  evidence you would need is exactly what is missing. This is a domain
  judgement, and it should be written down, not assumed.

Key Takeaways

• np.nan, NaT, and pd.NA are distinct markers, and isna is the only reliable test for all of them.
• Comparisons with np.nan return False and silently drop rows, while pd.NA propagates, forcing an explicit decision.
• Quantify missingness per column and look at which columns go missing together before choosing any treatment.
• Record an indicator column for every imputation, since missingness is often predictive on its own.
• Sentinel values such as -999 are the most dangerous form of missing data because no function reports them.

🧪 Practice

1. Build a frame with missing values, then compare the row counts returned by filtering a float column against filtering the same data as Int64.
2. Write a function that reports per-column missing counts, percentages, and the most common co-missing pair of columns.
3. Interview: Survey income data is missing for 30 percent of respondents, and you impute the median. What could go wrong? (Hint: who is most likely to decline the question?)

Index Alignment in Arithmetic Operations

Automatic alignment is the pandas feature that most distinguishes it from NumPy, and the one that silently produces wrong answers when misunderstood.

The rule is simple to state. When two pandas objects are combined, they are matched by index label, not by position. The result's index is the union of both, and labels present in only one produce NaN.

This is usually exactly right. Joining a revenue series indexed by customer to a cost series indexed by customer gives the correct pairing regardless of row order, which is precisely the bug that positional matching in NumPy would introduce.

The failure mode appears when indexes are accidentally different. Filtering a frame preserves the original labels, so a filtered series has a gappy index. Combining it with a fresh series indexed from zero matches almost nothing, and the result is mostly NaN with no error raised.

The diagnostic is characteristic: a suspicious pile of NaN after an operation between two series that clearly have the same length. When you mean positional combination, use .values or .to_numpy() to strip the index deliberately, or call reset_index(drop=True) on both.

import pandas as pd
import numpy as np

# ALIGNMENT DOING THE RIGHT THING: order does not matter.
revenue = pd.Series([100, 200, 300], index=["a", "b", "c"])
cost = pd.Series([30, 10, 20], index=["c", "a", "b"])       # different order
print(revenue - cost)                  # a:90 b:180 c:270, correctly matched
# NumPy would have subtracted positionally and produced nonsense.

# THE UNION RULE: non-overlapping labels become NaN.
x = pd.Series([1, 2, 3], index=["a", "b", "c"])
y = pd.Series([10, 20], index=["b", "c"])
print(x + y)                           # a:NaN b:12 c:23
print(x.add(y, fill_value=0))          # a:1 b:12 c:23, the usual intent

# THE CLASSIC BUG: a filtered series keeps its ORIGINAL labels.
df = pd.DataFrame({"v": [10, 20, 30, 40, 50]})
high = df.loc[df["v"] > 25, "v"]
print(high.index.tolist())             # [2, 3, 4], not [0, 1, 2]

adjust = pd.Series([1, 2, 3])          # a fresh index: 0, 1, 2
print(high + adjust)                   # 5 rows, 4 NaN: almost all wrong
# No error, no warning. Just mostly-missing output.

# THREE CORRECT FIXES, depending on intent.
print((high.reset_index(drop=True) + adjust).tolist())   # match positionally
print((high.values + adjust.values))                     # strip both indexes
adjust2 = pd.Series([1, 2, 3], index=high.index)         # give the right labels
print((high + adjust2).tolist())

# DATAFRAMES align on BOTH axes.
A = pd.DataFrame({"x": [1, 2], "y": [3, 4]}, index=["r1", "r2"])
B = pd.DataFrame({"y": [10, 20], "z": [30, 40]}, index=["r2", "r3"])
print(A + B)                           # 3 rows x 3 columns, mostly NaN
print(A.add(B, fill_value=0))

# COLUMN ASSIGNMENT ALIGNS TOO, which is the most common real bug.
df = pd.DataFrame({"v": [10, 20, 30, 40, 50]})
subset_mean = df.loc[df["v"] > 25, "v"] * 2
df["doubled"] = subset_mean            # aligned by label
print(df)                              # rows 0 and 1 are NaN: correct here,
                                       # and a silent surprise if unexpected

# ALIGNING EXPLICITLY when you want to see what will match.
left, right = x.align(y, join="inner")
print(left.index.tolist())             # ['b', 'c']: the overlap
print(x.align(y, join="outer", fill_value=0)[1].tolist())

# DUPLICATE LABELS cause a CARTESIAN blow-up rather than an error.
dup = pd.Series([1, 2], index=["a", "a"])
other = pd.Series([10, 20], index=["a", "a"])
print((dup + other).shape)             # 4 rows from 2 and 2
print(dup.index.is_unique)             # check this before combining

# A DEFENSIVE HABIT worth adopting in pipelines.
def combine_checked(a, b, op="+"):
    if not a.index.equals(b.index):
        only_a = a.index.difference(b.index)
        only_b = b.index.difference(a.index)
        raise ValueError(
            f"index mismatch: {len(only_a)} labels only in left, "
            f"{len(only_b)} only in right"
        )
    return a + b if op == "+" else a - b

try:
    combine_checked(x, y)
except ValueError as e:
    print(e)

Key Takeaways

• pandas matches by index label rather than position, and the result index is the union of both inputs.
• Non-overlapping labels produce NaN silently; the arithmetic methods accept fill_value to supply a default.
• A filtered Series keeps the original row labels, which is the usual cause of unexpected NaN when combining it with a fresh Series.
• Strip the index with reset_index(drop=True) or .values when you genuinely want positional combination.
• Duplicate index labels produce a Cartesian expansion rather than an error; check index.is_unique first.

🧪 Practice

1. Filter a frame, add a fresh Series to the filtered column, and explain the resulting NaN pattern, then fix it three different ways.
2. Combine two frames with partially overlapping rows and columns, and predict the output shape before running it.
3. Interview: A colleague's calculation returns a column that is 80 percent NaN despite both inputs being complete and equal in length. What is your first hypothesis? (Hint: equal length is not equal index.)

Copy-on-Write and Chained Assignment

The SettingWithCopyWarning is the most familiar and least understood message in pandas. Copy-on-write, the default since pandas 3.0, resolves the underlying ambiguity, and understanding both states is necessary because plenty of code still runs under the old rules.

The historical problem was that an indexing operation might return a view or a copy depending on the data's memory layout, which the user cannot see. Assigning to the result of a chained indexing operation therefore sometimes modified the original and sometimes modified a temporary that was immediately discarded. Same code, different outcome, no reliable warning.

The chained pattern is df[cond]["col"] = value. Python evaluates this as two separate operations: first df[cond], producing a new object, then an assignment into that object. Whether the original changes depends on whether the intermediate shared memory.

Copy-on-write makes the rule uniform: any indexing operation behaves as if it returned a copy, so a chained assignment never modifies the original, and the intermediate is never silently shared. Copies are still deferred internally until a write occurs, so this is not slower in practice and is often faster because defensive copying becomes unnecessary.

The fix is always the same and always was: perform the selection and the assignment in a single loc call. Under copy-on-write, chained assignment raises a ChainedAssignmentError instead of failing quietly.

import pandas as pd
import numpy as np

pd.options.mode.copy_on_write = True         # the default in pandas 3.0

df = pd.DataFrame({"region": ["eu", "us", "eu"], "revenue": [100.0, 200.0, 300.0]})

# THE WRONG PATTERN: two operations, assignment lands on a temporary.
try:
    df[df["region"] == "eu"]["revenue"] = 0.0
except Exception as e:
    print(type(e).__name__)          # ChainedAssignmentError under CoW
print(df["revenue"].tolist())        # unchanged, which is now GUARANTEED
# Before CoW this could either work or silently do nothing.

# THE CORRECT PATTERN: one loc call with both selectors.
df.loc[df["region"] == "eu", "revenue"] = 0.0
print(df["revenue"].tolist())        # [0.0, 200.0, 0.0]

# WHY THE OLD BEHAVIOUR WAS UNPREDICTABLE.
# df[mask] may produce a view or a copy depending on dtype blocks and
# memory layout, which is invisible at the call site. CoW removes the
# question by making every indexing result behave as an independent object.

# UNDER CoW, A SLICE NEVER LEAKS WRITES BACK.
base = pd.DataFrame({"a": [1, 2, 3]})
part = base.iloc[0:2]
part.loc[0, "a"] = 999
print(base["a"].tolist())            # [1, 2, 3]: the parent is safe
print(part["a"].tolist())            # [999, 2]

# THE OTHER HALF: a function that mutates its argument no longer
# accidentally mutates a caller's slice.
def add_flag(frame):
    frame["flag"] = frame["a"] > 1   # modifies the object it was handed
    return frame

subset = base.iloc[0:2]              # under CoW, an independent object
add_flag(subset)
print("flag" in base.columns)        # False

# EXPLICIT COPY when you intend independence and want it readable.
work = base.copy()
work["a"] *= 10
print(base["a"].tolist(), work["a"].tolist())

# INPLACE=True is discouraged and usually pointless. It does not save
# memory under CoW, it breaks method chaining, and it returns None,
# so a stray assignment silently produces None.
s = pd.Series([3, 1, 2])
s2 = s.sort_values()                 # prefer this
none = s.sort_values(inplace=True)   # returns None
print(none)

# CONDITIONAL UPDATE PATTERNS, all single-call and CoW-safe.
d = pd.DataFrame({"v": [1, 5, 10, 20], "g": list("aabb")})
d.loc[d["v"] > 5, "band"] = "high"                 # loc with a mask
d["band"] = d["band"].fillna("low")
d["capped"] = d["v"].clip(upper=10)                # vectorized alternative
d["label"] = np.where(d["v"] > 5, "high", "low")   # no assignment ambiguity
d = d.assign(doubled=lambda x: x["v"] * 2)         # chain-friendly
print(d)

# DIAGNOSING legacy code: find chained assignment by shape.
# Search for  ][  patterns followed by  =  , such as:
#   df[df.x > 0]["y"] = 1
#   df["y"][df.x > 0] = 1
#   df.loc[df.x > 0]["y"] = 1
# All three are chained. Only  df.loc[df.x > 0, "y"] = 1  is correct.

  WHY CHAINED ASSIGNMENT IS AMBIGUOUS

  df[df.x > 0]["y"] = 1

  step 1  df[df.x > 0]     -> a NEW object, call it TMP
  step 2  TMP["y"] = 1     -> writes into TMP

  Does df change?  Only if TMP shares df's memory. Whether it does
  depended on dtype layout, which is invisible to you.

      pre-CoW:   sometimes yes, sometimes no, warning unreliable
      with CoW:  never -> and pandas raises ChainedAssignmentError

  THE FIX, unchanged for a decade:

  df.loc[df.x > 0, "y"] = 1      one operation, one target, always correct

Key Takeaways

• Chained assignment splits into two operations, and whether the original changed used to depend on invisible memory layout.
• Copy-on-write makes every indexing result behave as an independent object, so chained assignment raises rather than failing silently.
• The correct pattern is a single loc call carrying both the row mask and the column name.
• Under copy-on-write, a function can no longer mutate a caller's slice by accident, and defensive copying becomes unnecessary.
• inplace=True saves no memory, breaks chaining, and returns None; avoid it.

🧪 Practice

1. Write a chained assignment and the equivalent single loc assignment, and confirm which one modifies the original frame.
2. Take a slice of a frame, pass it to a function that adds a column, and verify the parent is unaffected under copy-on-write.
3. Interview: Why did SettingWithCopyWarning sometimes appear when the code was actually correct, and sometimes not appear when it was wrong? (Hint: what was pandas able to detect about the intermediate object?)

apply, map, transform, and When to Avoid Them

These four methods look interchangeable and are not. Choosing correctly determines both the shape of the result and whether your code runs in milliseconds or minutes.

The distinction that matters most is what gets passed and what shape comes back. map works elementwise on a Series. apply passes a whole Series or, on a frame, a whole row or column. transform must return something the same length as its input, which is what makes it usable for group-wise broadcasting. agg reduces to a scalar per group.

The performance reality is blunt. apply is a Python loop with extra overhead, and it is not vectorized in any meaningful sense. On a million-row frame, a vectorized expression runs in milliseconds and the equivalent apply takes tens of seconds. The gap is not a constant factor you can ignore.

That said, apply is not forbidden. It is the right tool when no vectorized equivalent exists, when the row count is small, or when clarity genuinely matters more than speed for a one-off analysis. The mistake is reaching for it first rather than last.

df.apply(func, axis=1) deserves specific warning. It builds a Series per row, which combines the cost of row iteration with the cost of Python dispatch. Almost every row-wise apply can be rewritten as arithmetic between columns.

import pandas as pd
import numpy as np
import time

rng = np.random.default_rng(0)
n = 300_000
df = pd.DataFrame({
    "price": rng.random(n) * 100,
    "qty": rng.integers(1, 10, n),
    "region": rng.choice(["eu", "us", "ap"], n),
})

def timed(label, fn):
    t0 = time.perf_counter()
    out = fn()
    print(f"{label:<32} {(time.perf_counter()-t0)*1000:8.1f} ms")
    return out

# THE SAME COMPUTATION, FOUR WAYS.
timed("row-wise apply", lambda: df.apply(lambda r: r["price"] * r["qty"], axis=1))
timed("Series apply", lambda: df["price"].apply(lambda p: p * 1.2))
timed("vectorized", lambda: df["price"] * df["qty"])
# The first is typically 100x slower than the third for identical output.

# WHAT EACH ONE RECEIVES.
s = pd.Series([1, 2, 3])
print(s.map(lambda x: x * 2).tolist())          # one ELEMENT at a time
print(s.apply(lambda x: x * 2).tolist())        # on a Series: also elementwise
print(df[["price", "qty"]].apply(lambda col: col.max()))      # a COLUMN
print(df[["price", "qty"]].apply(lambda row: row.sum(), axis=1).head(2))  # ROW

# map ALSO TAKES A DICT, which is its best use: fast lookup encoding.
codes = pd.Series(["eu", "us", "ap", "xx"])
lookup = {"eu": "Europe", "us": "Americas", "ap": "Asia"}
print(codes.map(lookup).tolist())               # unmapped -> NaN
print(codes.map(lookup).fillna("Other").tolist())
# This is implemented as a hash join, not a Python loop. Prefer it to
# apply with an if-chain whenever the logic is a lookup.

# transform RETURNS THE SAME LENGTH, which is what broadcasting needs.
g = df.groupby("region")["price"]
print(g.transform("mean").head(3).round(2))     # one value per ROW
print(g.agg("mean").round(2))                   # one value per GROUP
# transform is how you add a group statistic back as a column:
df["price_vs_region_mean"] = df["price"] - g.transform("mean")

# REPLACING apply WITH VECTORIZED EQUIVALENTS, the common cases.
# 1. Conditional logic -> np.where / np.select
slow = lambda: df["price"].apply(lambda p: "high" if p > 50 else "low")
fast = lambda: np.where(df["price"] > 50, "high", "low")
timed("apply conditional", slow)
timed("np.where", fast)

# 2. Multi-branch -> pd.cut for ranges
df["band"] = pd.cut(df["price"], bins=[0, 25, 50, 75, 101],
                    labels=["q1", "q2", "q3", "q4"])
print(df["band"].value_counts().to_dict())

# 3. String work -> the .str accessor
df["region_upper"] = df["region"].str.upper()   # not apply(str.upper)

# 4. Row-wise arithmetic -> column arithmetic
df["total"] = df["price"] * df["qty"]           # not apply(..., axis=1)

# 5. Lookup -> map with a dict, as above

# WHEN apply IS LEGITIMATE: irreducibly row-dependent logic on small data.
small = df.head(500)
def classify(row):
    if row["region"] == "eu" and row["qty"] > 5:
        return "eu-bulk"
    if row["price"] > 90:
        return "premium"
    return "standard"
print(small.apply(classify, axis=1).value_counts().to_dict())
# 500 rows: fine. 5,000,000 rows: rewrite it with np.select.

# raw=True skips Series construction when you only need the values.
timed("apply axis=1", lambda: df.head(50_000).apply(lambda r: r[0] + r[1], axis=1))
timed("apply axis=1 raw", lambda: df[["price", "qty"]].head(50_000)
      .apply(lambda r: r[0] + r[1], axis=1, raw=True))

Method	Receives	Returns	Typical use
Series.map	one element, or a dict key	same length	lookup encoding
Series.apply	one element	any shape	element logic with no vector form
DataFrame.apply	a column, or a row with axis=1	flexible	last resort
groupby.transform	a group	same length as input	add group stats as a column
groupby.agg	a group	one value per group	summarize

Key Takeaways

• apply is a Python loop with overhead, commonly two orders of magnitude slower than the vectorized equivalent.
• df.apply(..., axis=1) builds a Series per row and is the slowest common pattern in pandas.
• transform returns the input's length and is the tool for broadcasting a group statistic back onto rows; agg reduces per group.
• map with a dictionary is a hash lookup, not a loop, and should replace apply-with-if-chains.
• Most apply calls reduce to np.where, np.select, pd.cut, the .str accessor, or plain column arithmetic.

🧪 Practice

1. Rewrite a row-wise apply that computes a weighted score from three columns as pure column arithmetic, and time both on 500,000 rows.
2. Use transform to add a column holding each row's deviation from its group mean, then verify the group means of that column are zero.
3. Interview: When is apply the right choice despite being slow? (Hint: consider both data size and whether a vectorized form exists.)

groupby and the Split-Apply-Combine Pattern

Grouping is the most valuable operation in analytical work, and it has a single conceptual structure that transfers unchanged to SQL, Spark, and R.

Split-apply-combine names the three steps. Split partitions rows by one or more keys. Apply runs a function on each partition independently. Combine assembles the results into one object. Once you see a problem in those terms, the pandas syntax is mechanical.

The crucial efficiency detail is that pandas does not actually build separate frames per group. groupby computes integer group codes and then runs a compiled routine that walks the data once. That is why groupby("k")["v"].mean() is fast while looping over groupby objects in Python is not.

The apply step comes in three flavours distinguished by output shape. Aggregation returns one row per group. Transformation returns one row per input row. Filtration returns a subset of the original rows. Choosing the right one avoids a great deal of index wrangling.

Two defaults deserve attention. Grouping keys become the index unless you pass as_index=False, and rows with a missing key are dropped unless you pass dropna=False. Silently losing the rows with a null key is a common and hard-to-spot data loss.

import pandas as pd
import numpy as np

rng = np.random.default_rng(0)
df = pd.DataFrame({
    "region": ["eu", "us", "eu", "ap", "us", "eu", None],
    "product": ["a", "a", "b", "b", "b", "a", "a"],
    "revenue": [100.0, 250.0, 175.0, 320.0, 90.0, 210.0, 60.0],
    "units": [4, 9, 6, 11, 3, 8, 2],
})

# THE BASIC FORM.
print(df.groupby("region")["revenue"].sum())
print(df.groupby("region")["revenue"].agg(["sum", "mean", "count"]))

# THE KEY BECOMES THE INDEX unless you say otherwise.
print(df.groupby("region", as_index=False)["revenue"].sum())

# MISSING KEYS ARE DROPPED BY DEFAULT: silent row loss.
print(df.groupby("region")["revenue"].sum().sum())          # 1145: 60 lost
print(df.groupby("region", dropna=False)["revenue"].sum())  # NaN group kept
# Always reconcile the total against the ungrouped total.
assert df.groupby("region", dropna=False)["revenue"].sum().sum() == df["revenue"].sum()

# MULTIPLE KEYS produce a MultiIndex.
multi = df.groupby(["region", "product"])["revenue"].sum()
print(multi)
print(multi.unstack(fill_value=0))          # a readable matrix

# NAMED AGGREGATION: the clearest syntax, and flat column names.
summary = df.groupby("region").agg(
    total_revenue=("revenue", "sum"),
    avg_revenue=("revenue", "mean"),
    order_count=("revenue", "size"),         # size counts ALL rows
    valid_count=("revenue", "count"),        # count EXCLUDES NaN
    top_product=("product", lambda s: s.mode().iloc[0]),
)
print(summary)
# size vs count is a frequent bug: they differ exactly by the NaN count.

# THE THREE APPLY SHAPES.
g = df.groupby("region")["revenue"]
print(g.sum().shape)                         # aggregation: one row per group
print(g.transform("sum").shape)              # transformation: one per ROW
print(df.groupby("region").filter(lambda x: x["revenue"].sum() > 200).shape)

# WHY THIS IS FAST: grouping is integer codes plus a compiled pass.
codes, uniques = pd.factorize(df["region"])
print(codes, uniques.tolist())               # what groupby builds internally

# ITERATING is available but usually the wrong instinct.
for key, group in df.groupby("region"):
    print(key, len(group))
# Use it for inspection or genuinely irreducible per-group logic only.

# GROUPING BY SOMETHING OTHER THAN A COLUMN.
df["date"] = pd.to_datetime("2024-01-01") + pd.to_timedelta(range(7), "D")
print(df.groupby(df["date"].dt.dayofweek)["revenue"].mean().round(1))
print(df.groupby(pd.cut(df["units"], [0, 5, 10, 20]), observed=True)
        ["revenue"].mean().round(1))
print(df.groupby(np.where(df["revenue"] > 150, "big", "small"))["units"].sum())

# GROUP-WISE RANKING and SHARE, the two most reused derived columns.
df["rank_in_region"] = df.groupby("region")["revenue"].rank(ascending=False)
df["share_of_region"] = (df["revenue"]
                         / df.groupby("region")["revenue"].transform("sum"))
print(df[["region", "revenue", "rank_in_region", "share_of_region"]].round(3))

# observed=True matters with CATEGORICAL keys: otherwise every unused
# category combination appears as an empty group, which can explode
# the output size on multi-key grouping.
cat = df.assign(region=df["region"].astype("category"))
print(cat.groupby("region", observed=False, dropna=False).size().shape)

  SPLIT - APPLY - COMBINE

  input                 split by region         apply sum        combine

  eu  100              eu: [100, 175, 210]  ->  485          region  revenue
  us  250      --->    us: [250, 90]        ->  340          ap      320
  eu  175              ap: [320]            ->  320          eu      485
  ap  320              (None: [60] dropped                   us      340
  us   90               unless dropna=False)
  eu  210
  N/A  60

  THREE OUTPUT SHAPES FROM THE SAME SPLIT

  aggregate    3 groups -> 3 rows      "total per region"
  transform    3 groups -> 7 rows      "each row's region total, inline"
  filter       3 groups -> N rows      "only rows from regions above 200"

Key Takeaways

• Split-apply-combine is one pattern with three output shapes: aggregation, transformation, and filtration.
• Grouping is implemented as integer codes plus a compiled pass, so built-in aggregations are fast and Python iteration is not.
• Rows with a missing key are dropped unless dropna=False, which silently loses data; reconcile totals to catch it.
• size counts all rows in a group while count excludes missing values.
• Named aggregation produces flat, self-documenting column names and should be the default syntax.

🧪 Practice

1. Compute total revenue, order count, and average order value per region in one named aggregation, then verify the totals reconcile with the ungrouped sum.
2. Add two columns to a frame: each row's rank within its group and its share of the group total, using transform.
3. Interview: A grouped total is smaller than the ungrouped total of the same column. Give two possible explanations. (Hint: consider both the grouping key and the aggregated column.)

Aggregation, Transformation, and Filtration

The three group operations differ by output shape, and picking the wrong one causes the index-mismatch errors that make grouping feel harder than it is. This topic treats each in turn.

Aggregation collapses each group to a single value per column. Built-in aggregations such as sum and mean dispatch to compiled routines, while passing a lambda forces a Python call per group. On many small groups that difference dominates, so prefer named strings where they exist.

Transformation returns a result the same length as the group, aligned back to the original index. This is what you need for group-relative features: deviation from the group mean, within-group z-scores, share of group total, forward fill within a group. These are among the most predictive features in practice, and they are nearly impossible to express cleanly any other way.

Filtration keeps or drops whole groups based on a group-level predicate. It returns original rows, not aggregates. Use it for rules like "exclude customers with fewer than five orders".

The fourth option, groupby.apply, receives the whole group frame and can return anything. It is the flexible, slow escape hatch. Use it when the operation genuinely needs multiple columns from the group at once, and check whether a combination of the other three would work first.

import pandas as pd
import numpy as np

rng = np.random.default_rng(0)
df = pd.DataFrame({
    "store": rng.choice(["s1", "s2", "s3"], 200),
    "day": rng.integers(1, 30, 200),
    "sales": rng.normal(500, 120, 200).round(2),
    "cost": rng.normal(300, 80, 200).round(2),
})

# AGGREGATION: one row per group.
print(df.groupby("store").agg(
    total=("sales", "sum"),
    mean=("sales", "mean"),
    p90=("sales", lambda s: s.quantile(0.9)),     # a lambda where needed
    n=("sales", "size"),
).round(1))

# Different functions per column, in one call.
print(df.groupby("store").agg({"sales": ["sum", "max"], "cost": "mean"}).round(1))

# BUILT-IN STRINGS BEAT LAMBDAS: compiled versus per-group Python.
import time
many = pd.DataFrame({"k": rng.integers(0, 20_000, 400_000),
                     "v": rng.random(400_000)})
t0 = time.perf_counter(); many.groupby("k")["v"].sum()
print(f"built-in sum : {(time.perf_counter()-t0)*1000:7.1f} ms")
t0 = time.perf_counter(); many.groupby("k")["v"].agg(lambda s: s.sum())
print(f"lambda sum   : {(time.perf_counter()-t0)*1000:7.1f} ms")

# TRANSFORMATION: same length as the input, aligned to its index.
df["store_mean"] = df.groupby("store")["sales"].transform("mean")
df["vs_mean"] = df["sales"] - df["store_mean"]
df["z_in_store"] = df.groupby("store")["sales"].transform(
    lambda s: (s - s.mean()) / s.std()
)
df["share"] = df["sales"] / df.groupby("store")["sales"].transform("sum")
print(df[["store", "sales", "vs_mean", "z_in_store", "share"]].head(3).round(3))

# Verify the property: within-group deviations sum to zero.
print(df.groupby("store")["vs_mean"].sum().round(8).tolist())

# GROUP-WISE FILLING, which a global fill would get wrong.
gaps = pd.DataFrame({"g": ["a", "a", "b", "b"], "v": [1.0, np.nan, 10.0, np.nan]})
print(gaps.groupby("g")["v"].transform(lambda s: s.fillna(s.mean())).tolist())
print(gaps.groupby("g")["v"].ffill().tolist())        # within group only

# FILTRATION: keep whole groups by a group-level test.
busy = df.groupby("store").filter(lambda g: len(g) > 60)
print(busy["store"].unique(), len(busy))

profitable = df.groupby("store").filter(lambda g: (g["sales"] - g["cost"]).sum() > 0)
print(profitable["store"].nunique())

# The transform equivalent, which is faster because it avoids per-group
# Python and expresses the same rule as a mask.
counts = df.groupby("store")["sales"].transform("size")
print(df[counts > 60]["store"].unique())

# groupby.apply: the escape hatch, when the logic needs the whole group.
def top_two_days(g):
    return g.nlargest(2, "sales")[["day", "sales"]]

print(df.groupby("store", group_keys=True).apply(top_two_days,
                                                 include_groups=False).head(4))

# A WEIGHTED MEAN genuinely needs two columns at once.
def weighted_margin(g):
    return np.average(g["sales"] - g["cost"], weights=g["sales"])

print(df.groupby("store").apply(weighted_margin, include_groups=False).round(2))

# The vectorized alternative, faster and worth knowing.
df["wnum"] = (df["sales"] - df["cost"]) * df["sales"]
agg = df.groupby("store").agg(num=("wnum", "sum"), den=("sales", "sum"))
print((agg["num"] / agg["den"]).round(2))

Operation	Output rows	Index	Use for
agg	one per group	group keys	summaries and reports
transform	one per input row	original	group-relative features
filter	subset of input	original	dropping whole groups
apply	anything	varies	multi-column group logic

Key Takeaways

• Aggregation collapses to one row per group, transformation preserves row count and alignment, filtration keeps or drops whole groups.
• Built-in aggregation strings dispatch to compiled code while lambdas run Python per group, which matters most with many small groups.
• transform is the tool for group-relative features such as deviation from the group mean, within-group z-scores, and share of group total.
• Group-wise filling respects group boundaries where a global fill would leak values across them.
• groupby.apply is flexible and slow; reach for it only when the logic needs several columns of the group together.

🧪 Practice

1. Add a within-group z-score column and confirm each group's mean of that column is approximately zero and its standard deviation one.
2. Express "keep only stores with more than 50 rows" using both filter and a transform-based mask, then compare timings.
3. Interview: You need each customer's spend as a percentage of their segment's total, kept on every original row. Which group operation and why? (Hint: how many rows must the output have?)

merge, join, and concat Mechanics

Combining tables is where row counts silently change, and a merge that returns more rows than either input is the single most common undetected bug in analytical pipelines.

merge is the general relational join, matching on column values. join is a convenience method that matches on indexes. concat stacks objects along an axis without matching on keys, aligning on the other axis. The three are not interchangeable and the choice should be explicit.

The join types are the SQL ones. inner keeps matching rows only, left keeps all left rows, right the mirror, and outer keeps everything. The default is inner, which means the default silently drops unmatched rows. In analysis this is often wrong, and left is usually the intent.

The mechanic that causes the row explosion is cardinality. If a key appears twice on the left and three times on the right, those rows produce six output rows. When you expect a one-to-many join and the "one" side has duplicates, the output multiplies without any error.

The defence is the validate parameter. Passing validate="one_to_many" makes pandas verify the cardinality assumption and raise if it is violated. It costs a check and prevents an entire category of silent corruption. Use it on every merge in production code.

import pandas as pd
import numpy as np

customers = pd.DataFrame({
    "customer_id": [1, 2, 3, 4],
    "name": ["Alice", "Bob", "Carla", "Dan"],
    "region": ["eu", "us", "eu", "ap"],
})
orders = pd.DataFrame({
    "order_id": [10, 11, 12, 13, 14],
    "customer_id": [1, 1, 2, 2, 99],          # 99 matches no customer
    "amount": [100.0, 150.0, 200.0, 50.0, 75.0],
})

# JOIN TYPES: note the row counts.
print(len(customers.merge(orders, on="customer_id", how="inner")))   # 4
print(len(customers.merge(orders, on="customer_id", how="left")))    # 6
print(len(customers.merge(orders, on="customer_id", how="right")))   # 5
print(len(customers.merge(orders, on="customer_id", how="outer")))   # 7
# inner is the DEFAULT and it dropped Carla, Dan, and order 14.

# indicator=True reports where each row came from: the diagnostic to
# run whenever a join's row count is not what you expected.
chk = customers.merge(orders, on="customer_id", how="outer", indicator=True)
print(chk["_merge"].value_counts().to_dict())
print(chk.loc[chk["_merge"] != "both", ["customer_id", "name", "order_id"]])

# validate CATCHES THE CARDINALITY BUG BEFORE IT SPREADS.
print(customers.merge(orders, on="customer_id", how="left",
                      validate="one_to_many").shape)

dup_customers = pd.concat([customers, customers.iloc[[0]]])   # a duplicate
try:
    dup_customers.merge(orders, on="customer_id", validate="one_to_many")
except pd.errors.MergeError as e:
    print("MergeError:", str(e)[:55])
print(len(dup_customers.merge(orders, on="customer_id")))     # 6, silently

# THE ROW-COUNT ASSERTION, when validate does not express the rule.
before = len(orders)
joined = orders.merge(customers, on="customer_id", how="left")
assert len(joined) == before, f"row count changed: {before} -> {len(joined)}"

# DIFFERENT KEY NAMES on each side.
o2 = orders.rename(columns={"customer_id": "cust"})
print(customers.merge(o2, left_on="customer_id", right_on="cust").shape)

# MULTI-KEY joins: pass a list, and both sides must match on all of them.
a = pd.DataFrame({"k1": [1, 1, 2], "k2": ["x", "y", "x"], "v": [10, 20, 30]})
b = pd.DataFrame({"k1": [1, 2], "k2": ["x", "x"], "w": [100, 200]})
print(a.merge(b, on=["k1", "k2"], how="left"))

# OVERLAPPING COLUMN NAMES get suffixes; name them explicitly.
l = pd.DataFrame({"id": [1, 2], "value": [10, 20]})
r = pd.DataFrame({"id": [1, 2], "value": [99, 98]})
print(l.merge(r, on="id", suffixes=("_left", "_right")).columns.tolist())

# JOIN ON INDEX.
ci = customers.set_index("customer_id")
oi = orders.set_index("customer_id")
print(ci.join(oi, how="left").shape)              # index-to-index
print(orders.join(ci, on="customer_id").shape)    # column-to-index

# CONCAT stacks; it does not match on keys.
q1 = pd.DataFrame({"m": ["jan", "feb"], "v": [1, 2]})
q2 = pd.DataFrame({"m": ["mar", "apr"], "v": [3, 4]})
print(pd.concat([q1, q2], ignore_index=True))     # rows: reset the index
print(pd.concat([q1, q2], keys=["q1", "q2"]).index.names)   # keep provenance

# concat ALIGNS THE OTHER AXIS, filling gaps with NaN.
extra = pd.DataFrame({"m": ["may"], "v": [5], "note": ["new"]})
print(pd.concat([q1, extra], ignore_index=True))  # note is NaN for q1
print(pd.concat([q1, extra], join="inner").columns.tolist())   # common only

# ASOF MERGE: match on the nearest earlier key. Essential for time series
# where events and measurements do not share exact timestamps.
trades = pd.DataFrame({
    "time": pd.to_datetime(["2024-01-01 10:00:03", "2024-01-01 10:00:07"]),
    "price": [101.0, 102.5],
})
quotes = pd.DataFrame({
    "time": pd.to_datetime(["2024-01-01 10:00:00", "2024-01-01 10:00:05"]),
    "bid": [100.5, 102.0],
})
print(pd.merge_asof(trades, quotes, on="time", direction="backward"))
# Each trade takes the most recent quote at or before it. Using a naive
# join here would either miss every row or leak future information.

  WHY A JOIN CAN RETURN MORE ROWS THAN EITHER INPUT

  left            right           inner result
  id  v           id  w           id  v  w
  1   a           1   x           1   a  x
  1   b           1   y           1   a  y
  2   c           2   z           1   b  x
                                  1   b  y     <- 2 x 2 = 4 rows for id 1
  3 rows          3 rows          2   c  z
                                  ------------
                                  5 rows, from 3 and 3

  CARDINALITY, and what to pass to validate

  one_to_one    keys unique on BOTH sides
  one_to_many   unique on the LEFT                (the usual lookup join)
  many_to_one   unique on the RIGHT
  many_to_many  no check: pandas allows it, and you should be sure

  CHECKLIST FOR EVERY MERGE
   1. state the expected row count BEFORE running it
   2. pass validate=
   3. pass how= explicitly; never rely on the inner default
   4. use indicator=True when the count surprises you

Key Takeaways

• merge joins on values, join on indexes, and concat stacks without key matching.
• The default join type is inner, which silently drops unmatched rows; state how explicitly on every merge.
• Duplicate keys multiply rows, so a many-to-many join can return more rows than either input without raising anything.
• The validate parameter enforces your cardinality assumption and turns a silent corruption into an error.
• Use indicator=True to see which side each row came from when the result size surprises you, and merge_asof for nearest-earlier time matching.

🧪 Practice

1. Construct two frames with duplicate keys on both sides, predict the merged row count, then verify and explain it.
2. Write a merge helper that asserts the output row count equals the left input's count, and test that it catches an accidental fan-out.
3. Interview: After a join, your revenue total doubles. What are the two most likely causes, and which parameter would have caught each? (Hint: one concerns duplicates, the other concerns join type.)

Reshaping with pivot, pivot_table, melt, stack, unstack

Data arrives in whichever shape the source system found convenient, and analysis and plotting each want a specific one. Reshaping is the translation layer, and it rests on a single distinction.

Long format has one row per observation, with variable names in a column and values in another. Wide format spreads variables across columns, one column per variable. Long is better for storage, filtering, and most plotting libraries. Wide is better for reading, correlation matrices, and matrix-shaped modelling input.

melt goes wide to long. pivot goes long to wide. The mnemonic is that melting spreads a solid out and pivoting gathers it back up.

pivot requires the index and column combination to be unique; it reshapes without aggregating and raises if two values compete for one cell. pivot_table aggregates duplicates using a function you supply, defaulting to the mean. When pivot raises a duplicate-entry error, the correct response is to decide what aggregation you want, not to reach blindly for pivot_table.

stack and unstack are the index-aware versions, moving levels between the index and the columns. They are the natural tools once a MultiIndex is present, which is what any multi-key groupby produces.

import pandas as pd
import numpy as np

wide = pd.DataFrame({
    "store": ["s1", "s2", "s3"],
    "jan": [100, 150, 120],
    "feb": [110, 140, 130],
    "mar": [125, 160, 128],
})

# WIDE -> LONG with melt.
long = wide.melt(id_vars="store", var_name="month", value_name="sales")
print(long.head(4))
print(long.shape)                      # 9 rows: 3 stores x 3 months

# LONG -> WIDE with pivot. The (index, columns) pair must be UNIQUE.
back = long.pivot(index="store", columns="month", values="sales")
print(back)
print(back.columns.name, back.index.name)     # 'month' 'store'

# WHEN DUPLICATES EXIST, pivot refuses rather than guessing.
dupes = pd.concat([long, long.iloc[[0]]])
try:
    dupes.pivot(index="store", columns="month", values="sales")
except ValueError as e:
    print("ValueError:", str(e)[:50])

# pivot_table AGGREGATES, so you must choose the function deliberately.
print(dupes.pivot_table(index="store", columns="month",
                        values="sales", aggfunc="sum"))
print(dupes.pivot_table(index="store", columns="month",
                        values="sales", aggfunc="mean"))
# These differ for the duplicated cell. Which is right depends entirely
# on whether the duplicate is a genuine second observation or an error.

# pivot_table's fuller feature set: several values, margins, fill.
sales = pd.DataFrame({
    "region": ["eu", "eu", "us", "us", "ap"],
    "product": ["a", "b", "a", "b", "a"],
    "revenue": [100, 200, 150, 250, 90],
    "units": [4, 8, 6, 10, 3],
})
pt = pd.pivot_table(
    sales, index="region", columns="product",
    values=["revenue", "units"], aggfunc="sum",
    fill_value=0, margins=True, margins_name="Total",
)
print(pt)

# STACK and UNSTACK move levels between axes.
multi = sales.groupby(["region", "product"])["revenue"].sum()
print(multi)                           # a MultiIndex Series
print(multi.unstack("product"))        # product level -> columns
print(multi.unstack("product").stack())         # and back
print(multi.unstack(level=0))                   # unstack the OTHER level

# unstack introduces NaN for absent combinations; fill deliberately.
print(multi.unstack("product", fill_value=0))

# WIDE-TO-LONG for paired column names, which melt handles clumsily.
paired = pd.DataFrame({
    "id": [1, 2],
    "score_2023": [10, 20], "score_2024": [12, 22],
    "cost_2023": [5, 6], "cost_2024": [7, 8],
})
tidy = pd.wide_to_long(paired, stubnames=["score", "cost"],
                       i="id", j="year", sep="_").reset_index()
print(tidy)

# EXPLODE turns a list-valued column into one row per element.
nested = pd.DataFrame({"user": ["a", "b"], "tags": [["x", "y"], ["z"]]})
print(nested.explode("tags"))

# CROSSTAB: a frequency pivot, with normalization built in.
print(pd.crosstab(sales["region"], sales["product"]))
print(pd.crosstab(sales["region"], sales["product"],
                  values=sales["revenue"], aggfunc="sum"))
print(pd.crosstab(sales["region"], sales["product"],
                  normalize="index").round(2))     # row proportions

# FLATTENING a MultiIndex column result before export.
pt2 = sales.pivot_table(index="region", values=["revenue", "units"],
                        aggfunc=["sum", "mean"])
pt2.columns = ["_".join(c) for c in pt2.columns]
print(pt2.columns.tolist())

  LONG AND WIDE, THE SAME DATA

  LONG (tidy)                     WIDE
  store  month  sales             store  jan  feb  mar
  s1     jan    100               s1     100  110  125
  s1     feb    110       <-->    s2     150  140  160
  s1     mar    125               s3     120  130  128
  s2     jan    150
  ...                             one row per ENTITY

  one row per OBSERVATION

  good for: filtering, groupby,   good for: reading, correlation,
            plotting libraries,             matrix input to models,
            adding new months               spreadsheet export

  THE OPERATIONS

       melt / stack  ------------>
  WIDE                             LONG
       <------------  pivot / unstack

  pivot        requires unique (index, column) pairs, never aggregates
  pivot_table  aggregates duplicates with aggfunc, default mean

Key Takeaways

• Long format holds one row per observation and suits filtering, grouping, and plotting; wide format suits reading and matrix-shaped input.
• melt widens to long and pivot narrows to wide; stack and unstack do the same across index levels.
• pivot raises on duplicate index-column pairs rather than guessing, which is a feature, not an obstacle.
• pivot_table aggregates duplicates, so choose aggfunc deliberately instead of accepting the mean default.
• Flatten MultiIndex columns before exporting, since most consumers expect a single header row.

🧪 Practice

1. Convert a wide monthly sales table to long format, compute a monthly growth rate, then pivot the result back to wide.
2. Build a pivot table of revenue by region and product with row and column totals, then flatten the column names for CSV export.
3. Interview: pivot raises a duplicate-entry error on your data. What does that tell you about the data, and what should you decide before switching to pivot_table? (Hint: what does aggregation assume about the duplicates?)

DatetimeIndex, Resampling, and Rolling Windows

Time series work has its own vocabulary in pandas, and using it turns awkward manual code into one-liners. The prerequisite is that your timestamps are actually datetimes, not strings.

A DatetimeIndex is an index of timestamps. Once a frame has one, you gain partial-string slicing, resampling, rolling windows, and the whole .dt accessor. Converting a string column and setting it as the index is therefore the first step of any time series task.

Resampling is a groupby where the groups are time bins. Downsampling aggregates to a coarser frequency, so daily to monthly. Upsampling generates finer bins and must decide how to fill them. The frequency strings are compact: D day, W week, ME month end, QE quarter end, h hour.

Rolling windows compute a statistic over a trailing span. The default window is right-aligned and closed on the right, meaning the value at time t uses data up to and including t. That default is what you want for forecasting features, because it never includes the future.

That last point is the one with consequences. A centred rolling window or a bfill leaks future information into a feature, which inflates backtest performance and fails in production. In any predictive time series pipeline, every window must be trailing and every fill must be forward.

Time zones are the other recurring hazard. A naive timestamp has no zone and cannot be compared safely across regions. Store in UTC, convert to local only for display.

import pandas as pd
import numpy as np

rng = np.random.default_rng(0)
idx = pd.date_range("2024-01-01", periods=180, freq="D")
ts = pd.DataFrame({"sales": 500 + rng.normal(0, 50, 180).cumsum()}, index=idx)

print(type(ts.index).__name__, ts.index.freq)

# PARTIAL-STRING SLICING: only a DatetimeIndex allows this.
print(ts.loc["2024-02"].shape)              # a whole month
print(ts.loc["2024-02-01":"2024-02-07"].shape)
print(ts.loc["2024-Q1"].shape)

# THE .dt ACCESSOR on a column, and index attributes on an index.
ts["dow"] = ts.index.dayofweek
ts["is_month_end"] = ts.index.is_month_end
print(ts.groupby("dow")["sales"].mean().round(1).to_dict())

# DOWNSAMPLING: a groupby whose keys are time bins.
print(ts["sales"].resample("ME").mean().round(1).head(3))
print(ts["sales"].resample("W").agg(["min", "max", "mean"]).round(1).head(2))

# The label and the closed edge are choices, not defaults to ignore.
print(ts["sales"].resample("ME", label="left").mean().head(2).round(1))

# UPSAMPLING creates empty bins that you must fill deliberately.
monthly = ts["sales"].resample("ME").mean()
print(monthly.resample("D").ffill().head(3).round(1))       # step function
print(monthly.resample("D").interpolate().head(3).round(1)) # linear
# ffill assumes the value held; interpolate assumes smooth change. Both
# are assumptions, and neither is free of consequence downstream.

# ROLLING WINDOWS: trailing by default, which is what you want.
ts["ma7"] = ts["sales"].rolling(7).mean()
ts["ma30"] = ts["sales"].rolling(30, min_periods=10).mean()
print(ts[["sales", "ma7", "ma30"]].head(3).round(1))
print(ts["ma7"].isna().sum())               # first 6 rows: not enough history

# THE LEAKAGE TRAP: a centred window uses FUTURE data.
ts["centred"] = ts["sales"].rolling(7, center=True).mean()
print(ts[["sales", "ma7", "centred"]].iloc[5:8].round(1))
# `centred` at day 6 already knows days 7, 8, 9. As a smoother for a chart
# that is fine. As a model feature it is leakage and will not reproduce.

# EXPANDING: all history up to each point.
ts["cumulative_mean"] = ts["sales"].expanding(min_periods=5).mean()

# EWM weights recent observations more heavily.
ts["ewm"] = ts["sales"].ewm(span=7, adjust=False).mean()

# TIME-BASED windows, which handle irregular spacing correctly.
irregular = pd.DataFrame(
    {"v": [1.0, 2.0, 3.0, 4.0]},
    index=pd.to_datetime(["2024-01-01", "2024-01-02", "2024-01-09", "2024-01-10"]),
)
print(irregular["v"].rolling("3D").mean().tolist())    # by TIME, not by rows
print(irregular["v"].rolling(2).mean().tolist())       # by row count
# On irregular data these differ substantially. Time-based is usually right.

# SHIFTING creates lag features, the foundation of forecasting inputs.
ts["lag1"] = ts["sales"].shift(1)
ts["lag7"] = ts["sales"].shift(7)
ts["wow_change"] = ts["sales"].pct_change(7)
ts["diff1"] = ts["sales"].diff()
# shift(-1) looks FORWARD and is only ever correct for building a TARGET.
ts["target_next_day"] = ts["sales"].shift(-1)

# GROUPED time operations: per entity, never across entities.
panel = pd.DataFrame({
    "store": ["s1"] * 5 + ["s2"] * 5,
    "date": list(pd.date_range("2024-01-01", periods=5)) * 2,
    "sales": rng.integers(80, 120, 10),
}).set_index("date")
panel["store_ma3"] = panel.groupby("store")["sales"].transform(
    lambda s: s.rolling(3, min_periods=1).mean()
)
print(panel.head(6))
# Rolling without the groupby would average the end of s1 into the start
# of s2, which is silently wrong and produces plausible-looking numbers.

# TIME ZONES: store UTC, display local.
utc = ts.index.tz_localize("UTC")
print(utc[0], utc.tz_convert("Europe/Paris")[0])

  TRAILING VS CENTRED: WHY IT DECIDES CORRECTNESS

  days      1   2   3   4   5   6   7   8   9
  value    10  12  11  13  15  14  16  18  17

  rolling(3) at day 5          -> mean(11, 13, 15)  days 3,4,5
     uses only the PAST                             SAFE as a feature

  rolling(3, center=True) at 5 -> mean(13, 15, 14)  days 4,5,6
     uses day 6, the FUTURE                         LEAKAGE

  A backtest with centred features looks excellent and fails live,
  because at prediction time day 6 does not exist yet.

  RESAMPLING FREQUENCIES
    D day   W week   ME month end   MS month start
    QE quarter end   YE year end    h hour   min minute

Key Takeaways

• Converting timestamps and setting a DatetimeIndex unlocks partial-string slicing, resampling, and time-based rolling windows.
• Resampling is a groupby over time bins; downsampling aggregates while upsampling requires an explicit fill decision.
• Rolling windows are trailing by default, which is what prevents feature leakage; center=True uses future data.
• Use time-based window strings such as "3D" when observations are irregularly spaced, since row-count windows misrepresent the span.
• Always group by entity before any rolling or shift on panel data, or values will bleed across entities.

🧪 Practice

1. Resample a daily series to monthly means, then upsample back to daily with both forward fill and interpolation, and plot the difference.
2. Build lag-1, lag-7, and 7-day trailing mean features for a panel of several stores, ensuring no value crosses a store boundary.
3. Interview: A demand forecast scores well in backtesting and poorly in production. The features include a 7-day moving average. What would you check? (Hint: which days does that average include?)

String Accessor Operations

Text columns arrive dirty. The .str accessor exposes vectorized string methods on a Series, which is both faster and far more readable than applying Python string functions row by row.

The accessor mirrors Python's string API, so s.str.lower() does what you expect on every element. The important addition is automatic missing-value handling: operations skip NaN and propagate it rather than raising an AttributeError, which is exactly the behaviour that makes a manual apply so tedious to write correctly.

The most useful methods cluster into four groups. Cleaning: strip, lower, replace, normalize. Testing: contains, startswith, match, isnumeric. Extracting: extract, findall, split, slice. Measuring: len, count.

Regular expressions appear throughout, and extract with named groups is the cleanest way to parse structured text into columns. contains with regex=False is meaningfully faster when you only need a literal substring, which is the common case.

The performance caveat is that on object dtype these methods still loop in Python under the hood; they are convenient rather than genuinely vectorized. The Arrow-backed string dtype changes that, running the operations in compiled code with a large memory saving.

import pandas as pd
import numpy as np

s = pd.Series(["  Alice Smith ", "BOB JONES", None, "carla o'brien", "42"])

# CLEANING: chainable, and NaN-safe throughout.
clean = s.str.strip().str.lower()
print(clean.tolist())                  # None stays None, no exception

print(s.str.len().tolist())            # NaN for the missing entry
print(s.str.replace(" ", "_", regex=False).tolist())

# TESTING returns a boolean Series with NaN where the input was missing.
print(s.str.contains("o", case=False, na=False).tolist())
# na=False is almost always what a filter needs; without it the mask
# contains NaN and cannot be used for indexing directly.

print(s.str.startswith("B", na=False).tolist())
print(s.str.isnumeric().tolist())

# SPLITTING, with expand to produce columns.
names = pd.Series(["Alice Smith", "Bob Jones", "Carla O'Brien"])
print(names.str.split(" ", expand=True))
print(names.str.split(" ", n=1, expand=True).rename(
    columns={0: "first", 1: "last"}))

# EXTRACTING with named groups: the clean way to parse structured text.
logs = pd.Series([
    "2024-01-15 ERROR db timeout after 30s",
    "2024-01-16 WARN cache miss rate 0.42",
    "2024-01-17 INFO job finished",
])
parsed = logs.str.extract(
    r"(?P<date>\d{4}-\d{2}-\d{2})\s+(?P<level>\w+)\s+(?P<message>.*)"
)
print(parsed)
print(parsed["date"].pipe(pd.to_datetime).dt.day_name().tolist())

# extractall returns EVERY match, with a match number in the index.
codes = pd.Series(["A1 B2", "C3"])
print(codes.str.extractall(r"([A-Z])(\d)"))

# SLICING and PADDING.
ids = pd.Series(["CUST-000123", "CUST-000456"])
print(ids.str.slice(5).tolist())            # drop the prefix
print(ids.str[-3:].tolist())                # bracket syntax also works
print(pd.Series(["7", "42"]).str.zfill(5).tolist())

# CONCATENATION: across a Series, or between Series.
first = pd.Series(["Alice", "Bob"])
last = pd.Series(["Smith", "Jones"])
print(first.str.cat(last, sep=" ").tolist())
print(first.str.cat(sep=", "))              # collapse to one string

# get_dummies for delimited multi-value fields.
tags = pd.Series(["python,sql", "sql", "python,r,sql"])
print(tags.str.get_dummies(sep=","))

# THE PERFORMANCE STORY: object versus Arrow-backed strings.
import time
n = 400_000
rng = np.random.default_rng(0)
words = pd.Series(rng.choice(["alpha", "beta", "gamma", "delta"], n))
arrow = words.astype("string[pyarrow]")

t0 = time.perf_counter(); words.str.upper()
print(f"object  upper: {(time.perf_counter()-t0)*1000:7.1f} ms")
t0 = time.perf_counter(); arrow.str.upper()
print(f"arrow   upper: {(time.perf_counter()-t0)*1000:7.1f} ms")
print(f"object  memory: {words.memory_usage(deep=True)/1e6:6.1f} MB")
print(f"arrow   memory: {arrow.memory_usage(deep=True)/1e6:6.1f} MB")

# LITERAL SEARCH IS FASTER when you do not need a pattern.
t0 = time.perf_counter(); words.str.contains("alp", regex=True)
print(f"contains regex : {(time.perf_counter()-t0)*1000:7.1f} ms")
t0 = time.perf_counter(); words.str.contains("alp", regex=False)
print(f"contains literal: {(time.perf_counter()-t0)*1000:6.1f} ms")

# A REALISTIC CLEANING PIPELINE.
messy = pd.Series(["  New York  ", "new york", "NEW YORK", "N.Y.", None])
normalized = (messy.str.strip()
                   .str.lower()
                   .str.replace(r"[^\w\s]", "", regex=True)   # drop punctuation
                   .str.replace(r"\s+", " ", regex=True))     # collapse spaces
print(normalized.tolist())
print(normalized.value_counts(dropna=False).to_dict())
# Three of the four variants now unify. "ny" still needs a lookup table:
# normalization handles format, not synonyms.

Key Takeaways

• The .str accessor applies string operations across a Series and propagates missing values instead of raising.
• Pass na=False to boolean string tests when the result will be used as an index mask.
• extract with named regex groups parses structured text directly into labelled columns.
• Set regex=False for literal substring searches, which is noticeably faster than pattern matching.
• Arrow-backed string dtype runs these operations in compiled code and uses far less memory than object.

🧪 Practice

1. Parse a Series of log lines into date, level, and message columns using a single extract call with named groups.
2. Normalize a column of inconsistently formatted city names and report how many distinct values remain before and after.
3. Interview: A filter built from str.contains drops more rows than expected on a column with missing values. Why? (Hint: what is the mask value where the input was NaN?)

Categorical Data and Memory Reduction

Memory is the constraint that decides whether an analysis runs on your laptop or needs a cluster, and categorical dtype is usually the single largest saving available.

A categorical column stores two things: an array of small integer codes and a categories lookup table holding each distinct value once. A column of one million rows drawn from four countries stores one million one-byte codes plus four strings, instead of one million separate Python string objects.

The rule of thumb is that categorical wins when the number of distinct values is small relative to the row count, roughly below fifty percent, and the saving becomes dramatic below one percent. For a high-cardinality column such as a unique identifier, categorical is actively worse, because you pay for the lookup table with no deduplication benefit.

Categoricals also carry semantics. An ordered categorical supports comparison and sorting in a meaningful order, which plain strings cannot do: alphabetically, "large" precedes "small". This is the correct representation for ordinal survey responses and size bands.

The behaviours that surprise people all follow from the fixed category set. Assigning an unlisted value produces NaN. Concatenating two categoricals with different categories falls back to object. And grouping includes unused categories unless observed=True, which on multi-key grouping can produce an enormous mostly-empty result.

import pandas as pd
import numpy as np

rng = np.random.default_rng(0)
n = 1_000_000
df = pd.DataFrame({
    "country": rng.choice(["FR", "DE", "IT", "ES"], n),       # 4 distinct
    "user_id": rng.integers(0, n, n).astype(str),             # ~n distinct
    "score": rng.random(n),
    "count": rng.integers(0, 300, n),
})

def mb(s):
    return s.memory_usage(deep=True) / 1e6

# THE WIN: low cardinality.
print(f"country object     : {mb(df['country']):7.1f} MB")
print(f"country categorical: {mb(df['country'].astype('category')):7.1f} MB")

# THE LOSS: high cardinality. Measure before converting.
print(f"user_id object     : {mb(df['user_id']):7.1f} MB")
print(f"user_id categorical: {mb(df['user_id'].astype('category')):7.1f} MB")
# The lookup table is nearly as large as the data, plus the codes.

# THE DECISION RULE, as code.
def should_categorize(s, threshold=0.5):
    if s.dtype != object:
        return False
    ratio = s.nunique() / len(s)
    return ratio < threshold, round(ratio, 4)

print(should_categorize(df["country"]), should_categorize(df["user_id"]))

# UNDER THE HOOD.
cat = df["country"].astype("category")
print(cat.cat.categories.tolist())     # the lookup table
print(cat.cat.codes.dtype)             # int8: one byte per row
print(cat.cat.codes.head(3).tolist())

# ORDERED CATEGORICALS give meaningful comparison and sorting.
sizes = pd.Series(["medium", "small", "large", "small"], dtype="category")
print(sizes.sort_values().tolist())    # alphabetical: wrong
ordered = sizes.cat.set_categories(["small", "medium", "large"], ordered=True)
print(ordered.sort_values().tolist())  # small, small, medium, large
print((ordered >= "medium").tolist())  # comparison now has meaning
print(ordered.max())

# THE SURPRISES, all consequences of a fixed category set.
c = pd.Series(["a", "b"], dtype="category")
c2 = c.copy()
c2.iloc[0] = "z"                       # not in categories
print(c2.tolist())                     # [nan, 'b']: silently lost
c3 = c.cat.add_categories(["z"])
c3.iloc[0] = "z"
print(c3.tolist())                     # ['z', 'b']: correct

x = pd.Series(["a", "b"], dtype="category")
y = pd.Series(["c", "d"], dtype="category")
print(pd.concat([x, y]).dtype)         # object: categories did not match
xu = x.cat.set_categories(["a", "b", "c", "d"])
yu = y.cat.set_categories(["a", "b", "c", "d"])
print(pd.concat([xu, yu]).dtype)       # category, preserved

# GROUPING INCLUDES UNUSED CATEGORIES by default.
g = pd.DataFrame({"k": pd.Categorical(["a", "b"], categories=list("abcde")),
                  "v": [1, 2]})
print(g.groupby("k", observed=False).size().shape)    # 5 rows: 3 are empty
print(g.groupby("k", observed=True).size().shape)     # 2 rows
# With two categorical keys of 100 categories each, observed=False can
# produce 10,000 rows from 50 of data.

# NUMERIC DOWNCASTING: the other large saving.
print(f"count int64: {mb(df['count']):6.1f} MB")
print(f"count int16: {mb(df['count'].astype('int16')):6.1f} MB")
print(f"score f64  : {mb(df['score']):6.1f} MB")
print(f"score f32  : {mb(df['score'].astype('float32')):6.1f} MB")
print(df["count"].min(), df["count"].max())     # verify the range fits

# A GENERAL OPTIMIZER, and always report what it did.
def optimize(frame, cat_threshold=0.5):
    out = frame.copy()
    for col in out.columns:
        s = out[col]
        if s.dtype == object and s.nunique() / len(s) < cat_threshold:
            out[col] = s.astype("category")
        elif pd.api.types.is_integer_dtype(s):
            out[col] = pd.to_numeric(s, downcast="integer")
        elif pd.api.types.is_float_dtype(s):
            out[col] = pd.to_numeric(s, downcast="float")
    return out

before = df.memory_usage(deep=True).sum() / 1e6
after = optimize(df).memory_usage(deep=True).sum() / 1e6
print(f"{before:.0f} MB -> {after:.0f} MB ({after/before:.0%})")
# float downcasting loses precision. For financial values, do not.

Key Takeaways

• Categorical stores integer codes plus a single lookup table, saving memory in proportion to how few distinct values a column has.
• Categorical is counterproductive for high-cardinality columns, so measure the distinct ratio before converting.
• Ordered categoricals give correct sorting and comparison for ordinal data, which string columns cannot provide.
• Assigning an unlisted value yields NaN, and concatenating mismatched category sets falls back to object.
• Use observed=True when grouping on categoricals to avoid generating empty combinations.

🧪 Practice

1. Measure the memory of a one-million-row string column with four distinct values before and after conversion, then repeat with a unique identifier column and explain the difference.
2. Create an ordered categorical for survey responses and show that sorting and comparison behave correctly where strings do not.
3. Interview: After converting two key columns to categorical, a groupby that used to return 50 rows now returns 10,000. What changed? (Hint: which combinations does pandas consider to exist?)

Method Chaining and pipe

How you structure a transformation affects correctness as much as readability. Method chaining expresses a pipeline as a sequence of operations on immutable intermediates, which eliminates a whole class of bugs involving stale variables and accidental mutation.

The problem with the alternative is familiar. Step-by-step code with reassignment produces a sequence of intermediate names, and after a notebook has been run out of order a few times, nobody can say which version of df2 is in memory. Every intermediate is also a chance to mutate something a previous cell still depends on.

A chain has no intermediates to mismanage. Each method returns a new object, the reading order matches the execution order, and the whole transformation is one expression you can move, copy, or wrap in a function.

Two methods make chaining practical. assign adds columns and accepts callables, so a new column can reference columns created earlier in the same chain. pipe inserts an arbitrary function into the chain, which is how you include your own logic without breaking the flow.

The style has real limits. A chain over about ten steps becomes hard to debug because you cannot inspect the middle. The remedy is a debugging pipe that prints the shape as data passes through, or splitting one long chain into two named ones at a meaningful boundary.

import pandas as pd
import numpy as np

rng = np.random.default_rng(0)
raw = pd.DataFrame({
    "Order ID": range(1, 201),
    " Region ": rng.choice(["eu", "us", "ap", None], 200),
    "Revenue": rng.normal(500, 200, 200).round(2),
    "Units": rng.integers(1, 20, 200),
    "Order Date": pd.date_range("2024-01-01", periods=200, freq="D"),
})

# THE STEP-BY-STEP VERSION: correct, but fragile in a notebook.
df1 = raw.copy()
df1.columns = [c.strip().lower().replace(" ", "_") for c in df1.columns]
df1 = df1.dropna(subset=["region"])
df1["unit_price"] = df1["revenue"] / df1["units"]
df1 = df1[df1["unit_price"] > 20]
df1["month"] = df1["order_date"].dt.to_period("M")
result1 = df1.groupby(["region", "month"], observed=True)["revenue"].sum()

# THE SAME THING AS A CHAIN: no intermediates, order matches reading.
result2 = (
    raw
    .rename(columns=lambda c: c.strip().lower().replace(" ", "_"))
    .dropna(subset=["region"])
    .assign(unit_price=lambda d: d["revenue"] / d["units"])
    .loc[lambda d: d["unit_price"] > 20]           # sees the new column
    .assign(month=lambda d: d["order_date"].dt.to_period("M"))
    .groupby(["region", "month"], observed=True)["revenue"]
    .sum()
)
print(result1.equals(result2))

# assign CREATES SEVERAL COLUMNS, and later ones can use earlier ones.
enriched = raw.rename(columns=lambda c: c.strip().lower().replace(" ", "_")).assign(
    unit_price=lambda d: d["revenue"] / d["units"],
    is_bulk=lambda d: d["units"] > 10,
    price_band=lambda d: pd.cut(d["unit_price"], [0, 30, 60, 1e9],
                                labels=["low", "mid", "high"]),
    revenue_rank=lambda d: d["revenue"].rank(pct=True).round(3),
)
print(enriched[["unit_price", "is_bulk", "price_band"]].head(3))

# pipe INSERTS YOUR OWN FUNCTION into the chain.
def drop_outliers(frame, col, z=3.0):
    s = frame[col]
    keep = (s - s.mean()).abs() <= z * s.std()
    return frame.loc[keep]

def add_cumulative(frame, group, col):
    return frame.assign(**{f"cum_{col}": frame.groupby(group)[col].cumsum()})

final = (
    enriched
    .pipe(drop_outliers, "revenue", z=2.5)
    .sort_values("order_date")
    .pipe(add_cumulative, "region", "revenue")
)
print(final[["region", "revenue", "cum_revenue"]].head(3).round(1))

# A DEBUGGING PIPE: the standard remedy for an opaque chain.
def trace(frame, label=""):
    print(f"  [{label:<14}] {frame.shape[0]:>5} rows x {frame.shape[1]} cols")
    return frame

_ = (
    raw
    .rename(columns=lambda c: c.strip().lower().replace(" ", "_"))
    .pipe(trace, "renamed")
    .dropna(subset=["region"])
    .pipe(trace, "dropna")
    .loc[lambda d: d["revenue"] > 300]
    .pipe(trace, "revenue>300")
    .pipe(drop_outliers, "revenue")
    .pipe(trace, "outliers")
)
# Each row loss is now attributable to a specific step, which is the
# information you need when a pipeline output is unexpectedly small.

# WRAPPING A CHAIN IN A NAMED FUNCTION makes it testable and reusable.
def prepare_orders(frame, min_unit_price=20.0):
    return (
        frame
        .rename(columns=lambda c: c.strip().lower().replace(" ", "_"))
        .dropna(subset=["region"])
        .assign(unit_price=lambda d: d["revenue"] / d["units"])
        .loc[lambda d: d["unit_price"] > min_unit_price]
        .reset_index(drop=True)
    )

print(prepare_orders(raw).shape)
# One function, one input, one output, no hidden state: straightforward
# to unit test, and immune to notebook execution order.

Key Takeaways

• Chaining removes intermediate variables, which removes the stale-state bugs that notebooks encourage.
• assign accepts callables so a new column can be defined in terms of columns created earlier in the same chain.
• loc with a lambda filters on the intermediate frame rather than the original, which is what makes filtering inside a chain possible.
• pipe inserts custom functions without breaking the flow, keeping domain logic in testable named functions.
• Long chains are hard to debug; insert a tracing pipe or split at a meaningful boundary.

🧪 Practice

1. Rewrite a six-step step-by-step cleaning script as a single chain and verify both produce identical output.
2. Write a trace pipe that reports row count and the number of rows lost since the previous trace, then use it to locate an unexpected filter.
3. Interview: What class of bug does method chaining prevent that step-by-step assignment does not? (Hint: think about a notebook cell run twice.)

Performance Tuning: eval, query, and Chunked Processing

When a pandas workflow becomes slow, the cause is usually one of three things: unnecessary temporaries, row-wise Python, or data that does not fit in memory. Each has a distinct remedy, and diagnosing which you have is more than half the work.

Temporaries arise from chained arithmetic. Each binary operation allocates a full intermediate array, so a four-term expression on a two-gigabyte frame allocates several gigabytes. pd.eval and df.eval parse the expression and evaluate it in chunks without materializing the intermediates, which both saves memory and improves cache behaviour.

query applies the same engine to filtering. Its practical advantage on large frames is that it avoids building a full boolean mask array per condition. Below roughly 100,000 rows the parsing overhead dominates and plain indexing is faster, so this is a large-data optimization only.

Chunking addresses data larger than memory. read_csv with a chunksize returns an iterator of frames, letting you accumulate a result across pieces. The constraint is that only certain operations decompose cleanly: sums and counts do, medians and exact distinct counts do not.

Before any of this, two cheaper wins usually apply. Read only the columns you need with usecols, and store intermediate data in Parquet rather than CSV. Parquet is columnar, typed, and compressed, so it loads roughly an order of magnitude faster and preserves dtypes that CSV discards.

import pandas as pd
import numpy as np
import time

rng = np.random.default_rng(0)
n = 2_000_000
df = pd.DataFrame({c: rng.random(n) for c in "abcd"})

def timed(label, fn):
    t0 = time.perf_counter()
    out = fn()
    print(f"{label:<30} {(time.perf_counter()-t0)*1000:8.1f} ms")
    return out

# TEMPORARIES: eval avoids materializing every intermediate.
timed("plain arithmetic", lambda: (df.a + df.b) * (df.c - df.d))
timed("df.eval", lambda: df.eval("(a + b) * (c - d)"))

# QUERY: the same engine applied to filtering.
timed("boolean mask", lambda: df[(df.a > 0.5) & (df.b < 0.5) & (df.c > 0.2)])
timed("query", lambda: df.query("a > 0.5 and b < 0.5 and c > 0.2"))

small = df.head(20_000)
timed("small: mask", lambda: small[small.a > 0.5])
timed("small: query", lambda: small.query("a > 0.5"))
# On small frames query is SLOWER: parsing costs more than it saves.

# IN-PLACE eval assignment avoids a copy of the whole frame.
df.eval("e = a * b + c", inplace=True)
print(df.columns.tolist())

# READ ONLY WHAT YOU NEED: usually the largest single win.
path = "/tmp/claude-0/perf_demo.csv"
df.head(300_000).to_csv(path, index=False)
timed("read all columns", lambda: pd.read_csv(path))
timed("read 2 columns", lambda: pd.read_csv(path, usecols=["a", "b"]))
timed("read 2 cols + dtype", lambda: pd.read_csv(
    path, usecols=["a", "b"], dtype={"a": "float32", "b": "float32"}))

# PARQUET versus CSV for intermediate storage.
pq = "/tmp/claude-0/perf_demo.parquet"
df.head(300_000).to_parquet(pq, index=False)
import os
print(f"csv    : {os.path.getsize(path)/1e6:6.1f} MB")
print(f"parquet: {os.path.getsize(pq)/1e6:6.1f} MB")
timed("read csv", lambda: pd.read_csv(path))
timed("read parquet", lambda: pd.read_parquet(pq))
timed("parquet, 2 columns", lambda: pd.read_parquet(pq, columns=["a", "b"]))
# Columnar storage reads only the requested columns off disk, which CSV
# cannot do because it must parse every line in full.

# CHUNKED PROCESSING for data larger than memory.
def chunked_stats(csv_path, chunksize=50_000):
    total, count, maximum = 0.0, 0, -np.inf
    for chunk in pd.read_csv(csv_path, usecols=["a"], chunksize=chunksize):
        total += chunk["a"].sum()          # sums are DECOMPOSABLE
        count += len(chunk)
        maximum = max(maximum, chunk["a"].max())
    return {"mean": total / count, "n": count, "max": maximum}

print(chunked_stats(path))
# A median is NOT decomposable this way: the median of chunk medians is
# not the median. You need a second pass, a t-digest, or a sort.

# CHUNKED GROUPBY: aggregate per chunk, then aggregate the aggregates.
def chunked_group_sum(csv_path, key, value, chunksize=50_000):
    parts = []
    for chunk in pd.read_csv(csv_path, usecols=[key, value],
                             chunksize=chunksize):
        parts.append(chunk.groupby(key)[value].sum())
    return pd.concat(parts).groupby(level=0).sum()

df.head(300_000).assign(g=rng.integers(0, 5, 300_000)).to_csv(path, index=False)
print(chunked_group_sum(path, "g", "a").round(1).to_dict())

# PROFILE BEFORE OPTIMIZING.
# %timeit  in a notebook for a single expression
# %prun    for a function-level profile
# df.memory_usage(deep=True).sort_values()  to find the expensive columns
print(df.memory_usage(deep=True).sort_values(ascending=False).head(3) / 1e6)

# KNOW WHEN TO STOP. Past roughly a few gigabytes, switch tools rather
# than fighting pandas: Polars for single-machine speed, DuckDB for
# SQL over larger-than-memory files, Dask or Spark for a cluster.

Key Takeaways

• Chained arithmetic allocates a temporary per operation; eval evaluates the whole expression without materializing them.
• query helps only on large frames, because its parsing overhead dominates below roughly 100,000 rows.
• Reading only needed columns and using Parquet instead of CSV are usually the largest and cheapest wins.
• Chunked processing works for decomposable aggregations such as sums and counts, but not for medians or exact distinct counts.
• Profile before optimizing, and switch to a different tool rather than fighting pandas past a few gigabytes.

🧪 Practice

1. Compare timings and peak memory for a four-term arithmetic expression computed normally and through eval on a two-million-row frame.
2. Write a chunked reader that computes a per-group sum over a CSV too large to load, then verify it against the in-memory result.
3. Interview: Why can a chunked pass compute an exact mean but not an exact median? (Hint: which statistic can be built from per-chunk partial results?)

Extension Arrays and the Arrow Backend

pandas was built on NumPy, and NumPy's type system was designed for numerical computing rather than tabular data. Extension arrays are the mechanism that lets pandas grow past those limits, and Arrow is the backend that makes it fast.

The three limitations are concrete. NumPy integers cannot represent missing values. NumPy strings are either fixed width or object pointers. And NumPy has no native concept of a nullable boolean, a decimal, or a nested list.

An extension array is an interface a third party can implement to provide a new column type that behaves like a native one. The nullable types Int64, boolean, and string are all implemented this way, as are categoricals and pandas' own date offsets.

Apache Arrow is a columnar memory standard, and the pyarrow backend implements extension arrays over it. This brings three things: compiled string operations, a validity bitmap giving every type proper null support, and zero-copy interchange with Polars, DuckDB, Spark, and any other Arrow consumer.

The caveats are real. Arrow-backed types are newer, some third-party libraries convert them back to NumPy on entry, and a few operations still have a slower path. For new work with heavy string or nullable content the trade is clearly worth it; for a mature pipeline, migrate deliberately.

import pandas as pd
import numpy as np

# THE LIMITATION being fixed.
s = pd.Series([1, 2, None])
print(s.dtype)                            # float64: the None forced it
print(s.astype("Int64").dtype)            # Int64: nullable, stays integer
print(s.astype("Int64").tolist())         # [1, 2, <NA>]

# THE NULLABLE FAMILY.
print(pd.Series([True, False, None], dtype="boolean").tolist())
print(pd.Series(["a", None], dtype="string").tolist())
print(pd.array([1.5, None], dtype="Float64"))

# THE ARROW BACKEND: same semantics, compiled implementation.
arrow_int = pd.Series([1, 2, None], dtype="int64[pyarrow]")
arrow_str = pd.Series(["alpha", "beta", None], dtype="string[pyarrow]")
print(arrow_int.dtype, arrow_str.dtype)

# STRINGS are where the difference is largest.
import time
rng = np.random.default_rng(0)
n = 600_000
obj = pd.Series(rng.choice(["alpha", "beta", "gamma", "delta"], n))
arw = obj.astype("string[pyarrow]")

for label, series in [("object", obj), ("arrow", arw)]:
    t0 = time.perf_counter()
    series.str.upper()
    up = (time.perf_counter() - t0) * 1000
    t0 = time.perf_counter()
    series.str.contains("al")
    cn = (time.perf_counter() - t0) * 1000
    print(f"{label:<7} upper {up:7.1f} ms  contains {cn:7.1f} ms  "
          f"mem {series.memory_usage(deep=True)/1e6:6.1f} MB")

# READING DIRECTLY INTO ARROW TYPES.
path = "/tmp/claude-0/arrow_demo.csv"
pd.DataFrame({"a": [1, 2, None], "s": ["x", None, "z"]}).to_csv(path, index=False)
numpy_backed = pd.read_csv(path)
arrow_backed = pd.read_csv(path, dtype_backend="pyarrow", engine="pyarrow")
print(numpy_backed.dtypes.to_dict())
print(arrow_backed.dtypes.to_dict())

# TYPES NUMPY CANNOT EXPRESS AT ALL.
import pyarrow as pa
nested = pd.Series([[1, 2], [3], None],
                   dtype=pd.ArrowDtype(pa.list_(pa.int64())))
print(nested.dtype)
print(nested.list.len().tolist())         # a list accessor, NaN-aware

decimals = pd.Series(["1.10", "2.20"],
                     dtype=pd.ArrowDtype(pa.decimal128(10, 2)))
print(decimals.dtype)
# Exact decimal arithmetic, which float64 cannot provide. This matters
# for monetary values where 0.1 + 0.2 must equal 0.3 exactly.

# THREE-VALUED LOGIC, the behavioural change to plan for.
a = pd.Series([True, False, None], dtype="boolean")
print((a & True).tolist())                # [True, False, <NA>]
print((a | True).tolist())                # [True, True, True] NA|True=True
print(a.fillna(False).tolist())           # resolve before indexing

# ZERO-COPY INTERCHANGE, the strategic reason to adopt Arrow.
df = pd.DataFrame({"x": [1, 2, 3]}).convert_dtypes(dtype_backend="pyarrow")
table = pa.Table.from_pandas(df)
print(table.schema)
print(table.to_pandas(types_mapper=pd.ArrowDtype).dtypes.to_dict())
# Polars, DuckDB, and Spark all read this table without reserializing.

# MIGRATION CHECK: confirm third-party libraries accept the new dtypes.
X = pd.DataFrame({"f": pd.array([1.0, 2.0, None], dtype="Float64")})
print(X.dtypes.to_dict())
print(X.to_numpy(dtype="float64", na_value=np.nan))
# Many scikit-learn estimators still want a plain NumPy float array;
# convert at the model boundary rather than throughout the pipeline.

Capability	NumPy backend	Arrow backend
Nullable integers	no	yes
String operations	Python loop	compiled
String memory	~50+ bytes each	dictionary-encoded
Nested and decimal types	no	yes
Cross-library zero copy	limited	native
Third-party support	universal	still maturing

Key Takeaways

• Extension arrays let pandas define column types NumPy cannot express, including nullable integers and proper strings.
• The Arrow backend adds compiled string operations, a validity bitmap for nulls on every type, and native nested and decimal types.
• String-heavy columns see the largest gains in both speed and memory.
• Nullable booleans use three-valued logic, so masks may need an explicit fillna before indexing.
• Some libraries still expect NumPy arrays, so convert at the model boundary rather than avoiding Arrow throughout.

🧪 Practice

1. Compare str.contains timing and memory on a 500,000-row string column stored as object and as string[pyarrow].
2. Build a Series of nullable integers and show that it survives a round-trip through CSV with the dtype preserved via dtype_backend.
3. Interview: Why can an Arrow-backed integer column hold nulls when a NumPy one cannot? (Hint: where is the missingness information stored?)

Common Pitfalls and Warning Messages

pandas warns rather than fails for many questionable operations, on the reasonable grounds that interactive exploration should not stop at every ambiguity. The cost is that warnings accumulate until people stop reading them. This topic catalogues the ones that indicate a real bug.

The pattern across nearly all of them is the same: pandas has made a choice on your behalf where two behaviours were plausible. The warning is telling you to make the choice explicitly. Treating warnings as errors in test suites is the cheapest way to enforce that.

Beyond warnings, four silent failures deserve permanent vigilance. Index alignment producing NaN. Merges changing row counts. groupby dropping null keys. And chained assignment that does nothing. None of these raise anything, and all of them produce plausible output.

The habit that catches most of these is reconciliation: after each significant operation, assert something you know must hold. Row count unchanged, total preserved, no new nulls. An assertion that fires during development costs a minute; the same bug reaching a dashboard costs far more.

import pandas as pd
import numpy as np
import warnings

# MAKE WARNINGS VISIBLE AND FATAL where it matters.
# In a test suite:  pytest -W error::FutureWarning
# In a module:      warnings.simplefilter("error", FutureWarning)

df = pd.DataFrame({"g": ["a", "a", "b", None], "v": [1.0, 2.0, 3.0, 4.0]})

# 1. SettingWithCopyWarning / ChainedAssignmentError.
# Cause: two indexing operations, then an assignment.
# Fix: one loc call.
df.loc[df["g"] == "a", "v"] = 10.0                 # correct
# df[df["g"] == "a"]["v"] = 10.0                   # wrong

# 2. FutureWarning / DeprecationWarning.
# Cause: an API that will change or disappear.
# Fix: read the message and migrate. Never suppress globally, because
# the next release will break the code silently.
with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    _ = pd.Series([1.0, np.nan]).fillna(0)
    print(f"{len(caught)} warning(s) during this block")

# 3. PerformanceWarning: highly fragmented frame.
# Cause: inserting columns one at a time in a loop.
bad = pd.DataFrame(index=range(100))
with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    for i in range(60):
        bad[f"c{i}"] = i                            # a reallocation each time
good = pd.concat(
    [pd.Series(i, index=range(100), name=f"c{i}") for i in range(60)], axis=1
)                                                   # one allocation
print(bad.shape == good.shape)

# 4. UnsortedIndexError on a MultiIndex.
mi = pd.DataFrame({"a": [1, 2]}, index=pd.MultiIndex.from_tuples(
    [("z", 1), ("a", 2)]))
print(mi.sort_index().loc["a":"z"].shape)           # sort first

# ---- SILENT FAILURES: no warning at all ----

# 5. ALIGNMENT producing NaN.
filtered = df.loc[df["v"] > 2, "v"]
fresh = pd.Series([1.0, 2.0])
print((filtered + fresh).isna().sum(), "NaN from index mismatch")

# 6. MERGE changing the row count.
left = pd.DataFrame({"k": [1, 1], "x": ["a", "b"]})
right = pd.DataFrame({"k": [1, 1], "y": ["c", "d"]})
print(len(left.merge(right, on="k")), "rows from 2 and 2")

# 7. GROUPBY dropping null keys.
print(df.groupby("g")["v"].sum().sum(), "vs", df["v"].sum())

# 8. dropna defaults differing between methods.
s = pd.Series(["a", None, "a"])
print(s.value_counts().sum(), s.value_counts(dropna=False).sum())
print(s.nunique(), s.nunique(dropna=False))

# 9. SORTING does not reset the index, so positional access misleads.
d = pd.DataFrame({"v": [3, 1, 2]}).sort_values("v")
print(d.index.tolist())                             # [1, 2, 0]
print(d.iloc[0]["v"], d.loc[0, "v"])                # 1 and 3: both "first"

# 10. INPLACE returning None.
z = pd.Series([3, 1, 2])
maybe = z.sort_values(inplace=True)
print(maybe)                                        # None

# ---- THE RECONCILIATION HABIT ----

def check(frame, expect_rows=None, key=None, total_col=None, total=None):
    if expect_rows is not None and len(frame) != expect_rows:
        raise AssertionError(f"rows {len(frame)} != expected {expect_rows}")
    if key is not None and frame[key].duplicated().any():
        raise AssertionError(f"duplicate keys in {key}")
    if total_col is not None and not np.isclose(frame[total_col].sum(), total):
        raise AssertionError(f"{total_col} total drifted")
    return frame

orders = pd.DataFrame({"id": [1, 2, 3], "amount": [10.0, 20.0, 30.0]})
lookup = pd.DataFrame({"id": [1, 2, 3], "region": ["eu", "us", "eu"]})
out = (orders
       .merge(lookup, on="id", how="left", validate="one_to_one")
       .pipe(check, expect_rows=len(orders), key="id",
             total_col="amount", total=orders["amount"].sum()))
print(out.shape)

# THE DIAGNOSTIC SEQUENCE when output looks wrong.
#   1. len(df) at each step: where did rows appear or vanish?
#   2. df.isna().sum():      where did nulls appear?
#   3. df.index.is_unique:   are labels duplicated?
#   4. df.dtypes:            did a column silently become object or float?
#   5. df.describe():        do min and max look like sentinels?

Symptom	Likely cause	Check
Unexpected NaN after arithmetic	index misalignment	a.index.equals(b.index)
Row count grew after a join	duplicate keys	validate=
Totals shrank after grouping	null keys dropped	dropna=False
Assignment had no effect	chained assignment	single loc call
Column became object	mixed types on load	df.dtypes
Identifiers show as floats	missing values in an int column	Int64

Key Takeaways

• Most pandas warnings mean the library chose between two plausible behaviours for you; make the choice explicit instead of suppressing it.
• The dangerous failures are silent: index misalignment, merge fan-out, dropped null group keys, and chained assignment.
• Building a frame by inserting columns in a loop reallocates repeatedly; build the columns and concatenate once.
• Sorting does not reset the index, so iloc[0] and loc[0] can return different rows.
• Reconcile row counts, key uniqueness, and column totals after each significant step rather than inspecting output at the end.

🧪 Practice

1. Write a reusable validation pipe that asserts row count, key uniqueness, and a preserved column total, then deliberately break each condition to confirm it fires.
2. Reproduce four silent failures from this topic and write the assertion that would have caught each.
3. Interview: A daily report's revenue total dropped ten percent with no code change and no errors. Give three pandas-level explanations before you look at the data. (Hint: consider joins, grouping keys, and dtype changes in the source.)

Optimization and Root Finding with SciPy

Fitting a model means minimizing a loss, and when no closed-form solution exists you need a numerical optimizer. SciPy provides the general-purpose ones, and using them well is mostly about choosing the right algorithm and recognising when it has failed.

Root finding answers "where does this function equal zero". Optimization answers "where is this function smallest". They are related, because at a smooth minimum the derivative is zero, so minimization is root finding on the gradient.

The algorithm choice follows from what you can provide. If you can compute a gradient, use a gradient-based method such as BFGS, which converges in far fewer evaluations. If the function is a black box, use a derivative-free method such as Nelder-Mead, which is robust and slow. If you have bounds or constraints, use L-BFGS-B or SLSQP, since unconstrained methods will happily wander into invalid regions.

The property that determines whether the answer is trustworthy is convexity. A convex function has one minimum, so any local method finds the global one. A non-convex function has many, and the optimizer returns whichever basin it started in. Running from several random starting points is the standard, unglamorous defence.

Always inspect the result object. success, message, and nit tell you whether the optimizer converged or simply ran out of iterations, and a non-converged result looks exactly like a converged one if you only read result.x.

import numpy as np
from scipy import optimize

# ROOT FINDING: where does f(x) = 0?
def f(x):
    return x ** 3 - 2 * x - 5

# brentq is the workhorse: bracketed, guaranteed, no derivative needed.
root = optimize.brentq(f, 2, 3)          # f(2) < 0 < f(3): a valid bracket
print(f"root = {root:.6f}, f(root) = {f(root):.2e}")
# It requires a SIGN CHANGE across the bracket. Without one it raises,
# which is better than newton silently diverging.

# newton converges faster with a derivative but can escape entirely.
print(optimize.newton(f, x0=2, fprime=lambda x: 3 * x ** 2 - 2))

# SYSTEMS of nonlinear equations.
def system(v):
    x, y = v
    return [x ** 2 + y ** 2 - 25, x - y - 1]

sol = optimize.root(system, x0=[1.0, 1.0])
print(sol.x.round(6), sol.success)

# MINIMIZATION: the general interface.
def rosenbrock(v):
    x, y = v
    return (1 - x) ** 2 + 100 * (y - x ** 2) ** 2      # minimum at (1, 1)

def rosen_grad(v):
    x, y = v
    return np.array([-2 * (1 - x) - 400 * x * (y - x ** 2),
                     200 * (y - x ** 2)])

for method, jac in [("Nelder-Mead", None), ("BFGS", rosen_grad),
                    ("L-BFGS-B", rosen_grad)]:
    r = optimize.minimize(rosenbrock, x0=[-1.2, 1.0], method=method, jac=jac)
    print(f"{method:<12} x={r.x.round(4)} nfev={r.nfev:<5} success={r.success}")
# The gradient-based methods use a fraction of the function evaluations.

# ALWAYS CHECK THE RESULT OBJECT, not just r.x.
r = optimize.minimize(rosenbrock, x0=[-1.2, 1.0], method="BFGS",
                      options={"maxiter": 3})
print(r.success, r.message, r.nit)       # False: it stopped early
# r.x still holds a number, and it is not the answer.

# BOUNDS and CONSTRAINTS.
# Minimize -(x*y) subject to x + y <= 10, with both in [0, 8].
res = optimize.minimize(
    lambda v: -(v[0] * v[1]),
    x0=[1.0, 1.0],
    method="SLSQP",
    bounds=[(0, 8), (0, 8)],
    constraints=[{"type": "ineq", "fun": lambda v: 10 - v[0] - v[1]}],
)
print(res.x.round(4), -res.fun.round(4))     # (5, 5), product 25

# NON-CONVEXITY: the local minimum problem, demonstrated.
def bumpy(x):
    return np.sin(3 * x) + 0.3 * (x - 2) ** 2

starts = [-2.0, 0.0, 2.0, 5.0]
found = [optimize.minimize(bumpy, [s]).x[0] for s in starts]
print([round(v, 3) for v in found])      # different starts, different answers

best = optimize.differential_evolution(bumpy, bounds=[(-5, 8)], seed=0)
print(f"global: x={best.x[0]:.3f}, f={best.fun:.3f}")
# Multi-start or a global optimizer is the practical answer. Reporting a
# local minimum as "the" optimum is a common and invisible error.

# CURVE FITTING: least squares with a parametric model.
rng = np.random.default_rng(0)
x = np.linspace(0, 4, 60)
true = (2.5, 1.3, 0.5)
y = true[0] * np.exp(-true[1] * x) + true[2] + rng.normal(0, 0.05, 60)

def model(x, a, b, c):
    return a * np.exp(-b * x) + c

params, cov = optimize.curve_fit(model, x, y, p0=[1.0, 1.0, 0.0])
errors = np.sqrt(np.diag(cov))           # standard errors from the covariance
for name, p, e, t in zip("abc", params, errors, true):
    print(f"{name}: {p:.3f} +/- {e:.3f}   (true {t})")
# p0 matters: a poor initial guess on a nonlinear fit can fail to converge
# or land on a different local optimum entirely.

# SCALING: an optimizer struggles when parameters differ by orders of
# magnitude, because a single step size cannot suit both.
def badly_scaled(v):
    return (v[0] - 1) ** 2 + (v[1] * 1e-6 - 1) ** 2

print(optimize.minimize(badly_scaled, [0, 0]).nit)
print(optimize.minimize(lambda u: badly_scaled([u[0], u[1] * 1e6]),
                        [0, 0]).nit)     # reparameterized: far fewer steps

Key Takeaways

• Root finding locates zeros and minimization locates minima; at a smooth minimum the gradient is zero, linking the two.
• Supply a gradient when you can, because gradient-based methods need far fewer function evaluations than derivative-free ones.
• Use bounded or constrained methods when the valid region is restricted, rather than hoping the optimizer stays inside it.
• Non-convex problems return a local minimum depending on the start, so use multiple starts or a global optimizer.
• Always check success, message, and iteration count; a non-converged result still returns plausible-looking numbers.

🧪 Practice

1. Minimize the Rosenbrock function with and without an analytic gradient, comparing function evaluation counts.
2. Fit an exponential decay with curve_fit, extract parameter standard errors from the covariance matrix, and show how a poor p0 changes the outcome.
3. Interview: An optimizer returns a different answer each run on the same data. What does that suggest and what would you do? (Hint: what shape is the loss surface?)

Interpolation and Numerical Integration

Two numerical tasks recur in applied work: estimating values between known observations, and computing the area under a curve you only have as samples or as a function.

Interpolation fits a function that passes exactly through your data points, then evaluates it elsewhere. This is distinct from regression, which fits a function near the points and does not pass through them. Interpolation assumes the observations are exact; regression assumes they contain noise. Using interpolation on noisy data faithfully reproduces the noise, which is almost never what you want.

The method choice trades smoothness for safety. Linear is robust and gives a jagged result. Cubic spline is smooth and can overshoot between points, producing values outside the observed range. PCHIP is shape-preserving, so a monotone input yields a monotone output, which is why it is the right default for quantities that cannot physically reverse.

Extrapolation beyond the data range is unsafe with every method and catastrophically so with polynomials. SciPy's default is to raise an error, which is the correct behaviour, and overriding it deserves a comment explaining why.

Numerical integration comes in two forms. quad integrates a callable function adaptively to high accuracy. The trapezoid and Simpson rules integrate sampled data, where you cannot evaluate at arbitrary points.

import numpy as np
from scipy import interpolate, integrate

# INTERPOLATION: exact through the points.
x = np.array([0.0, 1.0, 2.0, 3.0, 4.0, 5.0])
y = np.array([0.0, 0.8, 0.9, 0.1, -0.8, -1.0])
xq = np.linspace(0, 5, 11)

linear = interpolate.interp1d(x, y, kind="linear")
cubic = interpolate.CubicSpline(x, y)
pchip = interpolate.PchipInterpolator(x, y)

print("linear:", linear(xq).round(3))
print("cubic :", cubic(xq).round(3))
print("pchip :", pchip(xq).round(3))

# OVERSHOOT: cubic can leave the observed range; pchip cannot.
print(f"data range   : [{y.min():.2f}, {y.max():.2f}]")
fine = np.linspace(0, 5, 500)
print(f"cubic range  : [{cubic(fine).min():.2f}, {cubic(fine).max():.2f}]")
print(f"pchip range  : [{pchip(fine).min():.2f}, {pchip(fine).max():.2f}]")
# For a monotone quantity such as cumulative sales, cubic overshoot can
# produce a decrease that never happened. Use pchip there.

# EXTRAPOLATION is refused by default, and that default is correct.
try:
    linear(6.0)
except ValueError as e:
    print("refused:", str(e)[:44])
print(interpolate.interp1d(x, y, fill_value="extrapolate")(6.0).round(3))
print(cubic(8.0).round(3))            # a cubic extrapolates wildly

# INTERPOLATION VS SMOOTHING on noisy data.
rng = np.random.default_rng(0)
xn = np.linspace(0, 10, 40)
yn = np.sin(xn) + rng.normal(0, 0.2, 40)

exact = interpolate.CubicSpline(xn, yn)                 # follows every wiggle
smooth = interpolate.UnivariateSpline(xn, yn, s=1.0)    # s controls smoothing
grid = np.linspace(0, 10, 200)
print(f"interpolating residual: {np.abs(exact(xn) - yn).max():.2e}")  # ~0
print(f"smoothing residual    : {np.abs(smooth(xn) - yn).max():.3f}")
print(f"smoothing vs truth    : "
      f"{np.abs(smooth(grid) - np.sin(grid)).mean():.3f}")
# The interpolant has zero error on the data and worse error on the truth.
# That is overfitting, stated in numerical-methods vocabulary.

# TWO-DIMENSIONAL interpolation, for scattered spatial data.
pts = rng.random((60, 2)) * 10
vals = np.sin(pts[:, 0]) * np.cos(pts[:, 1])
gx, gy = np.mgrid[0:10:20j, 0:10:20j]
grid_z = interpolate.griddata(pts, vals, (gx, gy), method="cubic")
print(f"grid shape {grid_z.shape}, NaN outside hull: {np.isnan(grid_z).sum()}")
# griddata returns NaN outside the convex hull of the input points, which
# is the honest answer rather than an invented one.

# INTEGRATION OF A FUNCTION: adaptive and accurate.
area, err = integrate.quad(lambda t: np.exp(-t ** 2), 0, np.inf)
print(f"quad = {area:.10f} +/- {err:.1e}, truth = {np.sqrt(np.pi)/2:.10f}")

# INTEGRATION OF SAMPLES: when you only have observations.
t = np.linspace(0, np.pi, 101)
f = np.sin(t)                                 # exact integral is 2
print(f"trapezoid: {np.trapezoid(f, t):.8f}")
print(f"simpson  : {integrate.simpson(f, x=t):.8f}")
# Simpson fits parabolas and is markedly more accurate for the same points.

# CONVERGENCE with sample count.
for n in (11, 51, 201):
    tt = np.linspace(0, np.pi, n)
    print(f"n={n:<4} trapz err={abs(np.trapezoid(np.sin(tt), tt)-2):.2e}  "
          f"simpson err={abs(integrate.simpson(np.sin(tt), x=tt)-2):.2e}")

# A PRACTICAL USE: area under an irregularly sampled curve, such as
# cumulative exposure from timestamped sensor readings.
times = np.array([0.0, 0.5, 2.0, 2.1, 5.0])          # irregular spacing
level = np.array([1.0, 3.0, 2.0, 2.5, 1.0])
print(f"total exposure = {np.trapezoid(level, times):.3f}")
# Simply averaging the readings would weight the closely spaced pair far
# too heavily. Integration weights by the interval each value covers.

# SOLVING AN ODE, the third common numerical task.
def decay(t, y):
    return -0.5 * y

sol = integrate.solve_ivp(decay, [0, 10], [100.0], t_eval=np.linspace(0, 10, 5))
print(sol.y[0].round(3))                      # exponential decay from 100

Key Takeaways

• Interpolation passes exactly through the data and assumes it is exact; regression and smoothing assume noise.
• Cubic splines are smooth but can overshoot outside the observed range, while PCHIP preserves monotonicity.
• Extrapolation is unreliable with every method and extreme with polynomials; SciPy refuses it by default for good reason.
• Use quad when you can evaluate the function and trapezoid or Simpson when you only have samples.
• Integration weights each value by the interval it covers, which is why it beats averaging for irregularly sampled data.

🧪 Practice

1. Interpolate a monotone cumulative series with both a cubic spline and PCHIP, and find a point where the cubic produces a decrease.
2. Integrate sin(x) from 0 to pi with the trapezoid and Simpson rules at 11, 51, and 201 points, and tabulate how the errors shrink.
3. Interview: You have hourly sensor readings with occasional gaps and need total exposure over a day. Why is a mean times 24 wrong? (Hint: what does each reading represent in time?)

Sparse Matrices in SciPy

Many real matrices are mostly zeros. A user-item rating matrix, a document-term matrix, or a graph adjacency matrix typically has well under one percent non-zero entries. Storing all those zeros is both wasteful and, past a certain size, impossible.

A sparse matrix stores only the non-zero values along with their positions. The saving is proportional to the sparsity: a matrix that is 99.9 percent zeros needs roughly a thousandth of the memory, turning an infeasible allocation into a routine one.

The formats differ by what they make fast, and this is the whole of the practical knowledge. COO stores triples of row, column, value and is the natural construction format. CSR compresses rows and is fast for row slicing and matrix-vector products. CSC compresses columns and is fast for column operations. LIL and DOK support efficient incremental assignment, which CSR and CSC do not.

The workflow follows directly: build in COO or LIL, convert once to CSR, then compute. Assigning into a CSR matrix in a loop triggers a warning and is pathologically slow, because each insertion reshuffles the compressed structure.

The trap that erases every benefit is densification. Some operations, and most careless NumPy interactions, silently convert a sparse matrix to a dense one, which can exhaust memory instantly. Adding a scalar to a sparse matrix does exactly this, because the zeros stop being zeros.

import numpy as np
from scipy import sparse

rng = np.random.default_rng(0)

# THE MOTIVATION, measured.
n_users, n_items = 50_000, 20_000
density = 0.0005
nnz = int(n_users * n_items * density)
rows = rng.integers(0, n_users, nnz)
cols = rng.integers(0, n_items, nnz)
vals = rng.integers(1, 6, nnz).astype(float)

M = sparse.csr_matrix((vals, (rows, cols)), shape=(n_users, n_items))
dense_bytes = n_users * n_items * 8
sparse_bytes = M.data.nbytes + M.indices.nbytes + M.indptr.nbytes
print(f"dense  : {dense_bytes / 1e9:8.2f} GB")
print(f"sparse : {sparse_bytes / 1e6:8.2f} MB")
print(f"density: {M.nnz / (n_users * n_items):.5%}")

# THE FORMATS, and what each is for.
small = np.array([[1.0, 0, 0, 2.0], [0, 0, 3.0, 0], [0, 4.0, 0, 0]])
coo = sparse.coo_matrix(small)
print("COO triples:", list(zip(coo.row, coo.col, coo.data)))

csr = coo.tocsr()
print("CSR data   :", csr.data)          # non-zeros, row by row
print("CSR indices:", csr.indices)       # their column positions
print("CSR indptr :", csr.indptr)        # where each row starts in data
# Row i occupies data[indptr[i]:indptr[i+1]], which is why row slicing
# is O(nnz in row) and column slicing is not.

# BUILD IN A MUTABLE FORMAT, CONVERT ONCE, THEN COMPUTE.
lil = sparse.lil_matrix((1000, 1000))
for i in range(0, 1000, 10):
    lil[i, i] = 1.0                      # efficient in LIL
work = lil.tocsr()                       # convert once
print(work.nnz)

# Doing the same into a CSR emits a SparseEfficiencyWarning and is
# quadratic, because each insertion rebuilds the compressed arrays.

# OPERATIONS that stay sparse.
v = rng.random(n_items)
print((M @ v).shape)                     # matrix-vector: the core operation
print(M.T.shape, (M > 3).nnz)            # transpose and comparison
print(M.sum(axis=1).shape)               # row sums, returned as a matrix
print(np.asarray(M.sum(axis=1)).ravel()[:3])   # flatten to a 1-D array

# OPERATIONS THAT DENSIFY: the memory hazard.
# M + 1        -> every zero becomes 1: a 8 GB dense matrix
# np.exp(M)    -> exp(0) = 1: same problem
# M.toarray()  -> explicit, and fine only when you know the size
print(sparse.csr_matrix(small).toarray())

# The safe pattern: operate on .data, which touches only the non-zeros.
scaled = M.copy()
scaled.data = np.log1p(scaled.data)      # log1p(0) = 0, so sparsity holds
print(scaled.nnz == M.nnz)

# SPARSE LINEAR ALGEBRA has its own module.
from scipy.sparse import linalg as splinalg
A = sparse.random(500, 500, density=0.01, random_state=0, format="csr")
A = A + sparse.eye(500) * 10             # make it well conditioned
b = rng.random(500)
x = splinalg.spsolve(A.tocsc(), b)       # sparse direct solve
print(f"residual: {np.abs(A @ x - b).max():.2e}")

# Iterative solvers scale to matrices a direct solve cannot handle.
x2, info = splinalg.cg(A, b, rtol=1e-10)
print(f"cg converged: {info == 0}, residual {np.abs(A @ x2 - b).max():.2e}")

# TRUNCATED SVD, the standard sparse dimensionality reduction.
u, s, vt = splinalg.svds(M.astype(float), k=5)
print("top singular values:", np.sort(s)[::-1].round(2))
# Dense SVD on this matrix is impossible; the truncated version is routine.

# WHEN SPARSE IS NOT WORTH IT.
for d in (0.001, 0.05, 0.3):
    sm = sparse.random(2000, 2000, density=d, format="csr", random_state=0)
    sb = sm.data.nbytes + sm.indices.nbytes + sm.indptr.nbytes
    print(f"density {d:<6} sparse {sb/1e6:6.2f} MB vs dense 32.00 MB")
# Above roughly 10-15% density the index overhead outweighs the saving,
# and dense operations are faster anyway because they use BLAS.

Format	Fast at	Use for
COO	construction from triples	initial build
CSR	row slicing, A @ v	the default compute format
CSC	column slicing, spsolve	column work and direct solves
LIL	incremental assignment	building element by element
DOK	dictionary-style assignment	sparse random access during build

Key Takeaways

• Sparse storage keeps only non-zero values and their positions, saving memory in proportion to sparsity.
• Build in COO or LIL, convert once to CSR, then compute; assigning into CSR in a loop is pathologically slow.
• CSR is fast for row access and matrix-vector products while CSC suits column operations and direct solves.
• Operations that make zeros non-zero, such as adding a scalar, densify the matrix and can exhaust memory instantly.
• Above roughly ten to fifteen percent density, dense arrays are both smaller in practice and faster.

🧪 Practice

1. Build a 100,000 by 50,000 matrix at 0.01 percent density, report its sparse memory, and compute what the dense equivalent would require.
2. Apply a log transform to the non-zero entries only, and verify the non-zero count is unchanged.
3. Interview: A recommender pipeline runs out of memory at a line that only adds a constant to a rating matrix. Explain. (Hint: what happens to the stored zeros?)

SciPy Statistical Functions and Distributions

scipy.stats provides the probability distributions and statistical tests that chapter 3 described mathematically. Its value is a uniform interface: every distribution exposes the same methods, so switching between them requires no new syntax.

Each distribution object offers the same set. pdf is the density for continuous variables and pmf the mass for discrete ones. cdf gives the probability below a value and sf the probability above, which is one minus the cdf but computed more accurately in the far tail. ppf is the inverse cdf, the quantile function. rvs draws random samples, and fit estimates parameters from data.

The parameterization is the part that causes errors. SciPy uses a general loc and scale convention rather than each distribution's textbook parameters. For the exponential distribution, scale is the mean, not the rate. For the gamma, the shape is a positional argument called a. Reading the docstring once per distribution saves a great deal of confusion.

For tail probabilities, prefer sf over 1 - cdf. When the cdf is very close to one, subtracting it from one loses most of the significant digits, and the survival function computes the same quantity directly.

The tests follow a common convention too: each returns a statistic and a p-value. Chapter 3 covers what those mean and how they are misused; the point here is the mechanics of getting them.

import numpy as np
from scipy import stats

# THE UNIFORM INTERFACE: the same methods on every distribution.
norm = stats.norm(loc=100, scale=15)
print(f"pdf(115) = {norm.pdf(115):.5f}")
print(f"cdf(115) = {norm.cdf(115):.4f}")      # P(X <= 115)
print(f"sf(115)  = {norm.sf(115):.4f}")       # P(X > 115)
print(f"ppf(.95) = {norm.ppf(0.95):.2f}")     # the 95th percentile
print(f"interval = {np.round(norm.interval(0.95), 2)}")
print(f"mean {norm.mean():.0f}, std {norm.std():.0f}")

# sf BEATS 1 - cdf IN THE TAIL.
z = 9.0
print(f"1 - cdf: {1 - stats.norm.cdf(z):.20e}")     # 0.0: total precision loss
print(f"sf     : {stats.norm.sf(z):.20e}")          # the correct tiny number
# Any tail probability beyond about 8 standard deviations needs sf.

# PARAMETERIZATION: scale is not always what you expect.
# Exponential with RATE 0.5 means scale = 1 / 0.5 = 2.
exp = stats.expon(scale=2.0)
print(f"exponential mean = {exp.mean():.2f}")       # 2.0, not 0.5

# Gamma takes shape positionally.
gam = stats.gamma(a=2.0, scale=3.0)
print(f"gamma mean = {gam.mean():.2f}")             # a * scale = 6

# Binomial and Poisson are discrete: pmf, not pdf.
binom = stats.binom(n=20, p=0.3)
print(f"P(X = 6) = {binom.pmf(6):.4f}")
print(f"P(X <= 6) = {binom.cdf(6):.4f}")
pois = stats.poisson(mu=4)
print(f"P(X >= 8) = {pois.sf(7):.4f}")              # sf(7) is P(X > 7)
# Off-by-one on discrete survival functions is a classic error: for
# integers, P(X >= k) equals sf(k - 1), not sf(k).

# FITTING parameters from data.
rng = np.random.default_rng(0)
sample = stats.gamma(a=2.5, scale=1.8).rvs(3000, random_state=rng)
a_hat, loc_hat, scale_hat = stats.gamma.fit(sample, floc=0)   # fix loc at 0
print(f"fitted a={a_hat:.3f} scale={scale_hat:.3f} (true 2.5, 1.8)")
# floc=0 matters: letting loc float on a distribution bounded at zero
# often yields a better likelihood and a meaningless negative location.

# GOODNESS OF FIT.
ks = stats.kstest(sample, "gamma", args=(a_hat, loc_hat, scale_hat))
print(f"KS statistic {ks.statistic:.4f}, p = {ks.pvalue:.3f}")
# Note that testing fit on the same data used to estimate the parameters
# makes the p-value optimistic. Hold out data or use a bootstrap.

# COMMON TESTS, all returning (statistic, pvalue).
g1 = stats.norm(10, 2).rvs(60, random_state=1)
g2 = stats.norm(11, 3).rvs(70, random_state=2)
print("Welch t  :", np.round(stats.ttest_ind(g1, g2, equal_var=False), 4))
print("Mann-Whit:", np.round(stats.mannwhitneyu(g1, g2), 4))
print("Shapiro  :", np.round(stats.shapiro(g1), 4))
print("Levene   :", np.round(stats.levene(g1, g2), 4))

table = np.array([[45, 55], [30, 70]])
chi2, p, dof, expected = stats.chi2_contingency(table)
print(f"chi2 = {chi2:.3f}, p = {p:.4f}, dof = {dof}")

# CORRELATION, with the right measure for the data.
x = rng.normal(size=200)
y = 2 * x ** 3 + rng.normal(0, 2, 200)                # monotone, not linear
print(f"pearson  {stats.pearsonr(x, y)[0]:.3f}")      # measures LINEARITY
print(f"spearman {stats.spearmanr(x, y)[0]:.3f}")     # measures MONOTONICITY
print(f"kendall  {stats.kendalltau(x, y)[0]:.3f}")

# DESCRIPTIVE helpers.
d = stats.skewnorm(a=6).rvs(1000, random_state=0)
print(f"skew {stats.skew(d):.3f}, kurtosis {stats.kurtosis(d):.3f}")
print(stats.describe(d)._asdict().keys())
print(f"trimmed mean {stats.trim_mean(d, 0.1):.3f} vs mean {d.mean():.3f}")

# BOOTSTRAP for a confidence interval on any statistic.
res = stats.bootstrap((d,), np.median, n_resamples=2000,
                      random_state=0, method="percentile")
print(f"median 95% CI: [{res.confidence_interval.low:.3f}, "
      f"{res.confidence_interval.high:.3f}]")
# This works for statistics with no closed-form standard error, which is
# most of the interesting ones.

Key Takeaways

• Every distribution exposes the same interface, so pdf, cdf, sf, ppf, rvs, and fit work identically across them.
• SciPy parameterizes by loc and scale, so the exponential's scale is the mean rather than the rate.
• Use sf instead of 1 - cdf for tail probabilities, because the subtraction loses precision when the cdf approaches one.
• For discrete distributions, P(X >= k) is sf(k - 1), an easy off-by-one to get wrong.
• stats.bootstrap gives confidence intervals for statistics with no closed-form standard error.

🧪 Practice

1. Compute the probability of exceeding seven standard deviations using both 1 - cdf and sf, and explain the difference in the results.
2. Fit a gamma distribution to simulated data with and without floc=0, and compare the estimated parameters against the truth.
3. Interview: A colleague reports a Pearson correlation near zero and concludes two variables are unrelated. What would you check? (Hint: what relationship shape does Pearson actually measure?)

<a id="42-r-for-statistical-computing"></a>

4.2 R for Statistical Computing

R was designed by statisticians for statistics, which shows in everything from its vectorized defaults to the fact that fitting a model and getting correct standard errors takes one line; this section covers the language and the tidyverse workflow with an eye on what R does better than Python.

R Data Types and Vectors

R's design decision, visible everywhere, is that the vector is the fundamental unit. There is no scalar type: a single number is a vector of length one. Once you accept that, a great deal of otherwise arbitrary behaviour becomes predictable.

An atomic vector holds elements of one type, with a hierarchy that governs automatic coercion: logical becomes integer becomes double becomes character. Mixing types in a single vector silently promotes everything to the most general one present, which is how a numeric column becomes character because one row contained the text "N/A".

A list is the heterogeneous counterpart, holding elements of any type including other lists. Data frames are built on lists, which explains why a data frame is a list of equal-length columns rather than a matrix.

Two features distinguish R from Python here. Recycling repeats the shorter operand when combining vectors of different lengths, warning only when the lengths are not multiples. This is convenient and a genuine source of silent bugs. And NA is a first-class missing value available in every type, which Python needed extension arrays to match.

Indexing is one-based and supports negative indices for exclusion rather than counting from the end, which is the single most common error for someone arriving from Python.

# EVERYTHING IS A VECTOR. There is no scalar.
x <- 5
length(x)          # 1
is.vector(x)       # TRUE

# ATOMIC VECTORS hold ONE type, and mixing coerces silently.
c(1, 2, 3)              # numeric
c(1, "a", TRUE)         # "1" "a" "TRUE"  -- all character
c(TRUE, 1L, 2.5)        # 1.0 1.0 2.5     -- all double
# The hierarchy: logical -> integer -> double -> character
# This is why one "N/A" string turns an entire numeric column to text.

typeof(1L); typeof(1); typeof("a"); typeof(TRUE)   # integer double character logical

# VECTORIZATION IS THE DEFAULT, not an optimization you opt into.
v <- c(1, 4, 9, 16)
sqrt(v)                 # 1 2 3 4
v * 2                   # 2 8 18 32
v[v > 5]                # 9 16

# RECYCLING: the shorter operand repeats.
c(1, 2, 3, 4) + c(10, 20)          # 11 22 13 24  -- silently
c(1, 2, 3) + c(10, 20)             # warning: not a multiple, but it RUNS
# A real hazard: adding a length-2 vector to a 1000-row column succeeds.

# NA is first class and propagates.
y <- c(1, 2, NA, 4)
mean(y)                            # NA
mean(y, na.rm = TRUE)              # 2.333333
is.na(y)                           # FALSE FALSE TRUE FALSE
y == NA                            # NA NA NA NA -- never use this

# FOUR MISSING-LIKE VALUES, with different meanings.
NA        # missing, and typed: NA_integer_, NA_character_, ...
NULL      # absence of an object; drops out of vectors
NaN       # not a number, from 0/0
Inf       # from 1/0
length(c(1, NULL, 3))              # 2: NULL vanishes
length(c(1, NA, 3))                # 3: NA occupies a slot

# INDEXING IS ONE-BASED, and negatives EXCLUDE.
z <- c(10, 20, 30, 40, 50)
z[1]                               # 10, not 20
z[-1]                              # 20 30 40 50 -- drops the first
z[length(z)]                       # 50; R has no z[-1]-as-last idiom
z[c(TRUE, FALSE)]                  # 10 30 50 -- the mask RECYCLES
z[z > 25]                          # 30 40 50

# NAMED VECTORS work as lightweight lookup tables.
ages <- c(alice = 30, bob = 25, carla = 35)
ages["bob"]                        # 25
names(ages)

# LISTS hold anything, including other lists.
info <- list(name = "Alice", scores = c(90, 85), active = TRUE)
info$scores                        # numeric vector
info[["name"]]                     # "Alice"   -- extracts the ELEMENT
info["name"]                       # a LIST of length 1 -- a common confusion
str(info)

# FACTORS: R's categorical type, present since long before pandas had one.
g <- factor(c("low", "high", "medium", "low"),
            levels = c("low", "medium", "high"), ordered = TRUE)
g[1] < g[2]                        # TRUE: ordered comparison works
table(g)
as.integer(g)                      # 1 3 2 1 -- the underlying codes

# THE CLASSIC FACTOR TRAP.
f <- factor(c("10", "20", "30"))
as.numeric(f)                      # 1 2 3   -- the CODES, not the values
as.numeric(as.character(f))        # 10 20 30 -- correct

# THE APPLY FAMILY, R's idiomatic alternative to loops.
sapply(1:5, function(i) i^2)       # simplifies to a vector
lapply(1:3, function(i) i:1)       # always returns a list
vapply(1:3, function(i) i * 2, numeric(1))   # type-safe: declare the result
m <- matrix(1:6, nrow = 2)
apply(m, 1, sum)                   # 9 12    -- by ROW    (MARGIN = 1)
apply(m, 2, sum)                   # 3 7 11  -- by COLUMN (MARGIN = 2)
mapply(function(a, b) a + b, 1:3, 4:6)       # elementwise over two inputs

  R'S TYPE HIERARCHY AND SILENT COERCION

  logical  ->  integer  ->  double  ->  character
  TRUE           1L          1.0        "1"

  Any vector containing a character is ENTIRELY character.
  Any vector containing a double and integers is entirely double.

  Practical consequence: one stray "N/A", "-", or "1,234" in a CSV
  column makes the whole column character, and every subsequent
  numeric operation either fails or coerces to NA.

  RECYCLING

  c(1,2,3,4) + c(10,20)
     1  2  3  4
  + 10 20 10 20      <- the short vector repeats
  = 11 22 13 24      no warning, because 4 is a multiple of 2

  c(1,2,3) + c(10,20)   warns, but still returns 11 22 13

Key Takeaways

• The vector is R's fundamental unit and there is no scalar type, so vectorization is the default rather than an optimization.
• Atomic vectors coerce silently up the hierarchy from logical to character, which is how one text value ruins a numeric column.
• Recycling repeats the shorter operand and warns only on non-multiple lengths, making it a quiet source of bugs.
• Indexing is one-based and negative indices exclude elements rather than counting from the end.
• Converting a factor with as.numeric returns the level codes; convert through as.character first.

🧪 Practice

1. Create a vector mixing numbers and one string, inspect the resulting type, and write a function that reports which columns of a data frame were coerced to character.
2. Demonstrate recycling producing a wrong result silently, then write a guard that checks lengths before combining.
3. Interview: as.numeric on a factor of years returns small integers. What happened and what is the fix? (Hint: what does a factor actually store?)

Data Frames and tibbles

The data frame is R's table, and it predates pandas by two decades. A data.frame is a list of equal-length vectors with a shared row count, which is why it handles mixed column types naturally while a matrix cannot.

The tibble is the tidyverse's modernised replacement. It behaves identically in most respects but removes three behaviours that caused long-standing bugs: it never converts strings to factors, it never partially matches column names, and subsetting always returns a tibble rather than sometimes dropping to a vector.

That last point is the historically painful one. df[, "x"] on a base data frame returns a vector, while df[, c("x", "y")] returns a data frame. A function written assuming a data frame breaks when the selection happens to have one column. Tibbles return a tibble either way, and df[["x"]] is the explicit way to get a vector.

Tibbles also print sensibly. A base data frame with a million rows prints a million rows into your console; a tibble prints ten rows with types in the header. This sounds cosmetic and materially changes how you work.

For a Python reader, the correspondence is close. A tibble is a DataFrame, a column is a Series, and the row-index concept simply does not exist. R tables have row names that nobody uses, and the tidyverse discourages them outright, which eliminates the entire class of alignment bugs.

library(tibble)
library(dplyr)

# A DATA FRAME is a list of equal-length vectors.
df <- data.frame(
  id = 1:4,
  name = c("Alice", "Bob", "Carla", "Dan"),
  score = c(90.5, 78.0, 88.5, 95.0),
  region = c("eu", "us", "eu", "ap")
)
str(df)
nrow(df); ncol(df); dim(df); names(df)

# THE DROP SURPRISE in base R.
class(df[, "score"])              # "numeric"    -- dropped to a vector
class(df[, c("id", "score")])     # "data.frame"
class(df[, "score", drop = FALSE])# "data.frame" -- the explicit fix

# A TIBBLE is consistent: subsetting always returns a tibble.
tb <- as_tibble(df)
class(tb[, "score"])              # "tbl_df" ...
class(tb[["score"]])              # "numeric" -- explicit extraction
tb$score                          # also a vector

# NO PARTIAL MATCHING, which base R allowed and which hid typos.
df$sco                            # 90.5 78.0 ... -- base R guesses!
tb$sco                            # NULL with a warning -- correct

# PRINTING: a tibble shows types and truncates.
print(tb)
# # A tibble: 4 x 4
#      id name  score region
#   <int> <chr> <dbl> <chr>
# 1     1 Alice  90.5 eu
# ...

# CONSTRUCTION, and referencing columns as you build them.
t2 <- tibble(
  x = 1:5,
  y = x^2,                        # can reference x immediately
  label = paste0("row_", x)
)

# ROW-WISE construction for small literal tables, laid out readably.
lookup <- tribble(
  ~region, ~manager,  ~target,
  "eu",    "Nadia",   1000,
  "us",    "Sam",     1500,
  "ap",    "Rin",      800
)

# INSPECTION.
glimpse(tb)                       # the tidyverse equivalent of df.info()
summary(df)                       # per-column statistics
head(tb, 2); tail(tb, 2)

# ADDING AND MODIFYING, the tidyverse way.
tb2 <- tb %>%
  mutate(
    passed = score >= 80,
    grade = case_when(
      score >= 90 ~ "A",
      score >= 80 ~ "B",
      TRUE        ~ "C"           # TRUE is the default branch
    )
  )

# SELECTING with helpers.
tb2 %>% select(id, score)
tb2 %>% select(-region)                      # exclude
tb2 %>% select(starts_with("s"))
tb2 %>% select(where(is.numeric))
tb2 %>% relocate(region, .before = id)

# FILTERING and ARRANGING.
tb2 %>% filter(score > 80, region == "eu")   # commas mean AND
tb2 %>% filter(region %in% c("eu", "ap"))
tb2 %>% arrange(desc(score))
tb2 %>% slice_max(score, n = 2)

# NO ROW INDEX EXISTS, which removes an entire class of pandas bugs.
# Row names exist in base data frames and the tidyverse ignores them.
rownames(df) <- c("a", "b", "c", "d")
as_tibble(df)                                # row names dropped
as_tibble(df, rownames = "key")              # keep them as a real column

# STRINGS ARE NEVER FACTORS (since R 4.0, and always in tibbles).
class(data.frame(x = "a")$x)                 # "character"
class(tibble(x = "a")$x)                     # "character"

Behaviour	data.frame	tibble
Single-column subset	drops to a vector	stays a tibble
Partial name matching	yes, silently	no, warns
Strings to factors	no since R 4.0	never
Printing	all rows	10 rows with types
Column referencing during creation	no	yes

Key Takeaways

• A data frame is a list of equal-length vectors, which is why mixed column types work naturally.
• Base subsetting drops a single-column selection to a vector while tibbles stay tibbles, removing a common source of broken functions.
• Tibbles disable partial column-name matching, turning a silent wrong answer into a warning.
• R tables have no index concept, so the alignment bugs that pandas indexes cause do not arise.
• glimpse is the practical first look at an unfamiliar table.

🧪 Practice

1. Show a base data frame operation that returns a vector where you expected a table, and fix it two ways.
2. Build a tibble whose later columns are defined from earlier ones, and explain why the same expression fails with data.frame.
3. Interview: Why does the tidyverse discourage row names? (Hint: what can a row name hold that a column cannot, and what breaks when rows are filtered?)

The tidyverse Workflow

The tidyverse is a collection of packages sharing a design philosophy, and the philosophy is what makes it worth learning rather than any individual function.

The organising idea is tidy data: each variable is a column, each observation is a row, each type of observational unit is a table. This is the same long format that pandas reshaping targets, stated as a principle rather than a technique. Once data is tidy, every tidyverse verb applies without special cases.

The second idea is that a small set of verbs covers most manipulation, and they compose. filter selects rows, select selects columns, mutate adds columns, arrange orders, summarise aggregates, and group_by changes the scope of the others. Each takes a data frame first and returns a data frame, which is what makes them chainable.

The pipe is what turns the verbs into a workflow. x |> f() is f(x), so a sequence of transformations reads top to bottom in the order it happens, rather than inside out. R now has a native pipe |> alongside the magrittr %>%, and they are interchangeable for typical use.

The feature with no clean Python equivalent is non-standard evaluation. Verbs accept bare column names without quotes because they capture the expression and evaluate it inside the data frame. This makes interactive code much cleaner and makes writing functions over columns harder, which is what the embrace operator {{ }} exists to solve.

library(dplyr)
library(tidyr)

sales <- tibble(
  region  = c("eu", "us", "eu", "ap", "us", "eu"),
  product = c("a", "a", "b", "b", "b", "a"),
  revenue = c(100, 250, 175, 320, 90, 210),
  units   = c(4, 9, 6, 11, 3, 8)
)

# THE SIX CORE VERBS, composed with a pipe.
result <- sales |>
  filter(revenue > 90) |>                        # rows
  mutate(unit_price = revenue / units) |>        # new columns
  group_by(region) |>                            # change the scope
  summarise(
    total   = sum(revenue),
    avg     = mean(unit_price),
    n       = n(),
    .groups = "drop"                             # always state this
  ) |>
  arrange(desc(total))                           # order
print(result)

# .groups = "drop" matters: without it the result stays grouped and the
# NEXT verb silently operates per group. This is the most common
# tidyverse surprise, and it is why summarise warns about it.

# mutate WITH group_by broadcasts, like pandas transform.
sales |>
  group_by(region) |>
  mutate(
    region_total = sum(revenue),
    share        = revenue / region_total,
    rank         = min_rank(desc(revenue))
  ) |>
  ungroup()

# ACROSS applies one operation to many columns.
sales |>
  group_by(region) |>
  summarise(across(c(revenue, units), list(sum = sum, mean = mean)),
            .groups = "drop")

sales |> mutate(across(where(is.numeric), ~ .x / max(.x)))
# ~ .x is a lambda shorthand: .x is the column being operated on.

# TIDYR: reshaping, the direct analogue of melt and pivot.
wide <- tibble(store = c("s1", "s2"), jan = c(100, 150), feb = c(110, 140))

long <- wide |>
  pivot_longer(cols = -store, names_to = "month", values_to = "sales")
print(long)

long |> pivot_wider(names_from = month, values_from = sales)

# SEPARATE and UNITE for composite fields.
tibble(code = c("eu-a-2024", "us-b-2023")) |>
  separate_wider_delim(code, delim = "-",
                       names = c("region", "product", "year"))

# MISSING VALUES, with explicit verbs.
gaps <- tibble(g = c("a", "a", "b"), v = c(1, NA, 3))
gaps |> drop_na(v)
gaps |> replace_na(list(v = 0))
gaps |> group_by(g) |> fill(v, .direction = "down") |> ungroup()

# complete MAKES IMPLICIT MISSINGNESS EXPLICIT, which matters before
# any time series or panel operation.
partial <- tibble(store = c("s1", "s1", "s2"),
                  month = c("jan", "feb", "jan"),
                  sales = c(10, 20, 30))
partial |> complete(store, month, fill = list(sales = 0))
# s2/feb did not exist as a row; now it does, with an explicit zero.

# JOINS mirror SQL, with the same names.
targets <- tibble(region = c("eu", "us"), target = c(400, 300))
sales |> left_join(targets, by = "region")
sales |> inner_join(targets, by = "region")
sales |> anti_join(targets, by = "region")     # rows with NO match: ap
# anti_join has no direct pandas equivalent and is the cleanest way to
# answer "which keys are missing from the lookup table".

# NON-STANDARD EVALUATION: why bare column names work, and how to write
# functions over them.
summarise_by <- function(data, group_col, value_col) {
  data |>
    group_by({{ group_col }}) |>                  # embrace to pass through
    summarise(total = sum({{ value_col }}), .groups = "drop")
}
summarise_by(sales, region, revenue)

# With a column name held in a STRING variable, use .data instead.
col <- "revenue"
sales |> summarise(total = sum(.data[[col]]))

  THE TIDY DATA RULE

  UNTIDY (wide)                    TIDY (long)
  store  jan  feb                  store  month  sales
  s1     100  110                  s1     jan    100
  s2     150  140                  s1     feb    110
                                   s2     jan    150
  "month" is spread across          s2     feb    140
  column NAMES, so it cannot
  be filtered or grouped on         every variable is a column,
                                    every observation a row

  THE PIPE, and why it reads better

  arrange(summarise(group_by(filter(sales, revenue > 90), region),
                    total = sum(revenue)), desc(total))

  sales |>                      read inside out, right to left
    filter(revenue > 90) |>
    group_by(region) |>         versus top to bottom, in order
    summarise(total = sum(revenue)) |>
    arrange(desc(total))

Key Takeaways

• Tidy data puts one variable per column and one observation per row, which is the precondition for every tidyverse verb applying uniformly.
• Six verbs cover most manipulation, and each takes and returns a data frame, which is what makes them compose.
• Always set .groups in summarise, because a silently retained grouping changes what the next verb does.
• across applies an operation to several columns, and complete turns implicit missing combinations into explicit rows.
• Non-standard evaluation allows bare column names; use {{ }} to pass them into your own functions and .data[[ ]] for strings.

🧪 Practice

1. Write one pipeline that filters, adds a derived column, groups, summarises, and sorts, then compare it line by line with the pandas equivalent.
2. Use complete to fill missing store-month combinations with zero, and explain why doing this before a time series step matters.
3. Interview: What does the embrace operator solve, and why is it needed at all? (Hint: what happens to a bare column name passed as a function argument?)

ggplot2 Grammar of Graphics

ggplot2 is the strongest argument for learning R. It implements a grammar of graphics, meaning plots are composed from independent components rather than selected from a catalogue of chart types.

The grammar has five parts. Data is the tidy table. Aesthetics map columns to visual properties: position, colour, size, shape. Geometries are the marks drawn: points, lines, bars. Scales control how data values become visual values. Facets split one plot into a grid of small multiples.

The consequence is compositional. There is no "grouped bar chart with a trend line" function; you add a bar geometry and a smooth geometry to the same aesthetic mapping. Any combination that makes sense can be expressed, which is not true of libraries built around named chart types.

The distinction people get wrong is inside versus outside aes(). Inside, a property is mapped to a column, so it varies by row and gets a legend. Outside, it is set to a constant. Writing colour = "blue" inside aes() creates a single-category legend labelled "blue" and colours the points red, which is confusing exactly once.

Faceting deserves emphasis because it is underused. Splitting a cluttered plot into small multiples by a categorical variable is usually more informative than adding a third colour dimension, and it costs one line.

library(ggplot2)
library(dplyr)

set.seed(42)
d <- tibble(
  x       = rnorm(300),
  y       = NA_real_,
  region  = sample(c("eu", "us", "ap"), 300, replace = TRUE),
  segment = sample(c("smb", "ent"), 300, replace = TRUE),
  size    = runif(300, 1, 5)
) |> mutate(y = 2 * x + rnorm(300, 0, 1.5) + ifelse(region == "eu", 2, 0))

# THE MINIMUM: data, aesthetic mapping, geometry.
ggplot(d, aes(x = x, y = y)) +
  geom_point()

# MAPPING versus SETTING: the distinction that matters.
ggplot(d, aes(x = x, y = y, colour = region)) +   # MAPPED: varies, legend
  geom_point(alpha = 0.6, size = 2)               # SET: constant, no legend

# The classic error:
# geom_point(aes(colour = "blue"))  -> maps every row to the literal
# string "blue", producing one red series with a legend entry "blue".

# LAYERS ACCUMULATE, each optionally with its own data or mapping.
ggplot(d, aes(x = x, y = y)) +
  geom_point(aes(colour = region), alpha = 0.5) +
  geom_smooth(method = "lm", se = TRUE, colour = "black") +
  geom_hline(yintercept = 0, linetype = "dashed", colour = "grey50") +
  labs(
    title    = "Revenue response is steeper in EU",
    subtitle = "Linear fit with 95% confidence band",
    x        = "Standardised spend",
    y        = "Revenue lift",
    colour   = "Region",
    caption  = "n = 300, simulated"
  ) +
  theme_minimal(base_size = 12)

# A title that states the FINDING rather than naming the variables does
# most of the communication work. See chapter 7 for why.

# FACETING: small multiples instead of a third colour dimension.
ggplot(d, aes(x = x, y = y)) +
  geom_point(alpha = 0.5) +
  geom_smooth(method = "lm", se = FALSE) +
  facet_wrap(~ region)                      # one panel per region

ggplot(d, aes(x = x, y = y)) +
  geom_point(alpha = 0.4) +
  facet_grid(segment ~ region)              # rows x columns

# free scales when panels have very different ranges; fixed scales when
# comparing magnitudes across panels is the point.
ggplot(d, aes(x = y)) + geom_histogram(bins = 20) +
  facet_wrap(~ region, scales = "free_y")

# GEOMETRIES for the common cases.
ggplot(d, aes(x = region, y = y)) + geom_boxplot()
ggplot(d, aes(x = region, y = y)) + geom_violin() + geom_jitter(width = 0.15,
                                                                alpha = 0.3)
ggplot(d, aes(x = y)) + geom_density(aes(fill = region), alpha = 0.4)
ggplot(d, aes(x = region, fill = segment)) + geom_bar(position = "dodge")
ggplot(d, aes(x = region, fill = segment)) + geom_bar(position = "fill") +
  scale_y_continuous(labels = scales::percent)

# SCALES control the data-to-visual mapping.
ggplot(d, aes(x = x, y = y, colour = size)) +
  geom_point() +
  scale_colour_viridis_c() +                       # perceptually uniform
  scale_x_continuous(limits = c(-3, 3), breaks = seq(-3, 3, 1))

# LIMITS VERSUS ZOOM: a real correctness difference.
# scale_*_continuous(limits = ...)  DROPS points outside the range, which
#   changes any statistic computed by a later layer, such as a smooth.
# coord_cartesian(xlim = ...)       ZOOMS without dropping data.
ggplot(d, aes(x, y)) + geom_point() + geom_smooth() +
  coord_cartesian(xlim = c(-1, 1))                 # the safe one

# SAVING: specify dimensions, never screenshot.
# ggsave("plot.png", width = 8, height = 5, dpi = 300)

# The grammar composes: any sensible combination is expressible, which
# is why ggplot2 handles plots that a chart-type library cannot.

  THE GRAMMAR AS LAYERS

  ggplot(data, aes(...))      the data and the variable mapping
    + geom_point()            a layer of marks
    + geom_smooth()           another layer, same mapping
    + scale_colour_viridis_c()how values become colours
    + facet_wrap(~ group)     split into small multiples
    + labs(...)               text
    + theme_minimal()         non-data ink

  MAPPED vs SET

  aes(colour = region)              colour VARIES by the region column,
                                    and a legend appears
  geom_point(colour = "blue")       colour is CONSTANT, no legend
  geom_point(aes(colour = "blue"))  WRONG: maps every row to the literal
                                    string, one series, misleading legend

Key Takeaways

• Plots are composed from data, aesthetic mappings, geometries, scales, and facets rather than chosen from a list of chart types.
• Properties inside aes() are mapped to columns and produce a legend; properties outside are set to constants.
• Layers accumulate and can each carry their own data and mapping, which is what makes arbitrary combinations expressible.
• Faceting into small multiples usually communicates better than encoding a third variable as colour.
• Setting scale limits drops data and changes computed layers; use coord_cartesian to zoom safely.

🧪 Practice

1. Build a scatter plot with points coloured by one variable, a linear fit per group, and facets by a second variable, all in one expression.
2. Demonstrate the difference between scale_x_continuous(limits=) and coord_cartesian(xlim=) on a plot containing a smoother.
3. Interview: Someone writes geom_point(aes(colour = "red")) and gets red-free output with an odd legend. Explain. (Hint: what does aes treat its arguments as?)

Statistical Modeling in R

This is where R's origins pay off most directly. Fitting a model, getting correct standard errors, and running diagnostics take three lines, because the language was built by people who needed exactly that.

The formula interface is the core abstraction. y ~ x1 + x2 declares a model specification declaratively, and every modelling function in R accepts it. The notation is compact and expressive: : is an interaction, * is main effects plus interaction, poly(x, 2) is an orthogonal polynomial, and . means all remaining columns.

Factors are handled automatically. A character or factor predictor is expanded into dummy variables with a reference level, and the coefficients are labelled with the level names. There is no manual encoding step, and no opportunity to leak the encoding across a train-test split.

The output objects are rich. summary() gives coefficients, standard errors, t-statistics, p-values, R-squared, and an F-test. confint() gives intervals, anova() compares nested models, and plot() produces four standard diagnostic plots. Compare this with the several libraries and manual assembly the Python equivalent requires.

The broom package converts model output into tidy data frames, which is what makes comparing many models practical: fit them, tidy them, bind the rows, and plot.

library(broom)
library(dplyr)

data(mtcars)

# THE FORMULA INTERFACE.
m1 <- lm(mpg ~ wt + hp, data = mtcars)
summary(m1)
# Coefficients, standard errors, t values, p values, R-squared, and the
# overall F test, all from one call.

# FORMULA NOTATION.
# y ~ x1 + x2          main effects
# y ~ x1:x2            the interaction only
# y ~ x1 * x2          main effects AND the interaction
# y ~ . - x3           every other column except x3
# y ~ poly(x, 2)       orthogonal polynomial, safer than I(x^2)
# y ~ log(x)           transformations inline
# y ~ x + (1 | group)  a random intercept, with lme4

m2 <- lm(mpg ~ wt * factor(cyl), data = mtcars)
coef(m2)      # levels are labelled: factor(cyl)6, factor(cyl)8, ...

# FACTORS EXPAND AUTOMATICALLY, with a reference level.
mtcars$cyl_f <- factor(mtcars$cyl)
levels(mtcars$cyl_f)                       # "4" "6" "8": 4 is the reference
m3 <- lm(mpg ~ wt + cyl_f, data = mtcars)
# Coefficients read as differences FROM the reference level.
mtcars$cyl_f <- relevel(mtcars$cyl_f, ref = "8")    # change the baseline

# EXTRACTORS, uniform across model types.
coef(m1); confint(m1); fitted(m1)[1:3]; resid(m1)[1:3]
AIC(m1); BIC(m1); logLik(m1); nobs(m1)
predict(m1, newdata = data.frame(wt = 3, hp = 150), interval = "prediction")
# interval = "confidence" for the MEAN response, "prediction" for a NEW
# observation. The second is wider, and confusing them understates risk.

# DIAGNOSTICS, in one call.
# par(mfrow = c(2, 2)); plot(m1)
# Residuals vs fitted (linearity), Q-Q (normality),
# scale-location (homoscedasticity), residuals vs leverage (influence).

# MODEL COMPARISON.
m_small <- lm(mpg ~ wt, data = mtcars)
anova(m_small, m1)                         # nested F test
AIC(m_small, m1)

# GENERALIZED LINEAR MODELS, the same interface.
mtcars$efficient <- as.integer(mtcars$mpg > 20)
g1 <- glm(efficient ~ wt + hp, data = mtcars, family = binomial)
summary(g1)
exp(coef(g1))                              # odds ratios
predict(g1, type = "response")[1:3]        # probabilities, not log-odds

# Poisson for counts, with an offset for exposure.
# glm(events ~ x + offset(log(person_years)), family = poisson)

# BROOM turns model output into tidy tables.
tidy(m1, conf.int = TRUE)                  # one row per coefficient
glance(m1)                                 # one row per MODEL
augment(m1) |> select(mpg, .fitted, .resid, .cooksd) |> head(3)
# .cooksd flags influential points, which is the diagnostic most often
# skipped and most often responsible for a surprising coefficient.

# COMPARING MANY MODELS becomes a data frame operation.
specs <- list(
  simple      = mpg ~ wt,
  with_hp     = mpg ~ wt + hp,
  interaction = mpg ~ wt * hp
)
comparison <- purrr::imap_dfr(specs, function(f, name) {
  glance(lm(f, data = mtcars)) |> mutate(model = name, .before = 1)
})
print(comparison |> select(model, r.squared, adj.r.squared, AIC))

# MIXED EFFECTS, where R is clearly ahead of Python.
# library(lme4)
# lmer(score ~ treatment + (1 | school), data = d)
# A random intercept per school, correctly accounting for the fact that
# pupils within a school are not independent observations.

# SURVIVAL ANALYSIS, likewise.
# library(survival)
# coxph(Surv(time, event) ~ treatment + age, data = d)

Key Takeaways

• The formula interface declares model structure compactly and is accepted by essentially every modelling function in R.
• Factor predictors expand into dummy variables automatically with labelled, reference-relative coefficients.
• One summary call gives coefficients, standard errors, p-values, and fit statistics, and plot gives four standard diagnostics.
• Confidence intervals cover the mean response while prediction intervals cover a new observation and are wider.
• broom converts models into tidy tables, making many-model comparison an ordinary data frame operation.

🧪 Practice

1. Fit three nested models, compare them with anova and AIC, and state which you would report and why.
2. Fit a logistic regression, convert the coefficients to odds ratios, and produce predicted probabilities for three hypothetical rows.
3. Interview: A regression coefficient flips sign when one observation is removed. Which diagnostic would have warned you? (Hint: it appears in augment output.)

R Markdown Reporting

A result that lives in a console is not a deliverable. R Markdown, and its successor Quarto, let you write a document in which the code, its output, and the prose live in one file, so the report is regenerated from source every time.

The mechanism is literate programming. A plain-text file contains Markdown prose and fenced code chunks. Rendering executes every chunk in a fresh session and weaves the results into the output document. Because the session is fresh, a document that renders is reproducible by construction: every object it uses is created by code in the file.

That property is the real argument. The alternative workflow, running analysis and pasting numbers into a document, breaks silently whenever the data updates. With a rendered document, updating means re-rendering, and there is no version where the text and the numbers disagree.

Chunk options control what appears. echo shows or hides the code, eval runs it or not, message and warning control noise, and cache stores results of expensive chunks. A report for stakeholders typically hides all code; a technical appendix shows it.

Inline code embeds a computed value in a sentence, so a number in your prose is generated rather than typed. This is a small feature with a large effect, because typed numbers are exactly what goes stale.

Parameterized reports take variables from the render call, letting one template produce a document per region, per month, or per customer.

---
title: "Regional Revenue Review"
author: "Analytics"
date: "`r Sys.Date()`"
params:
  region: "eu"
  min_date: "2024-01-01"
output:
  html_document:
    toc: true
    code_folding: hide      # readers can reveal code if they want it
---

```{r setup, include=FALSE}
# include=FALSE runs the chunk but shows neither code nor output.
knitr::opts_chunk$set(
  echo    = FALSE,     # hide code by default in a stakeholder report
  message = FALSE,     # suppress library loading chatter
  warning = FALSE,
  fig.width = 8, fig.height = 4.5, dpi = 150
)
library(dplyr); library(ggplot2)
```

```{r load, cache=TRUE}
# cache=TRUE stores the result; the chunk re-runs only when its code
# changes. Useful for slow queries, and a trap when the DATA changes
# without the code changing. Set cache.extra = file.mtime(path) to fix.
sales <- readr::read_csv("data/sales.csv") |>
  filter(region == params$region, date >= as.Date(params$min_date))
```

## Summary

Revenue for **`r params$region`** totalled
`r scales::dollar(sum(sales$revenue))` across
`r nrow(sales)` orders, a
`r scales::percent(sum(sales$revenue) / sum(sales$prior_revenue) - 1)`
change on the prior period.

Every number in that paragraph is computed at render time. None of them
can go stale, because none of them was typed.

```{r trend, fig.cap="Daily revenue with a 7-day trailing mean"}
sales |>
  arrange(date) |>
  mutate(ma7 = zoo::rollmean(revenue, 7, fill = NA, align = "right")) |>
  ggplot(aes(date)) +
  geom_line(aes(y = revenue), colour = "grey70") +
  geom_line(aes(y = ma7), linewidth = 1) +
  labs(x = NULL, y = "Revenue") +
  theme_minimal()
```

```{r table}
sales |>
  group_by(product) |>
  summarise(revenue = sum(revenue), orders = n(), .groups = "drop") |>
  arrange(desc(revenue)) |>
  knitr::kable(digits = 0, format.args = list(big.mark = ","))
```

## Appendix

```{r session, echo=TRUE}
sessionInfo()   # record package versions so the run is reproducible
```

# RENDERING, including one document per region from one template.
# rmarkdown::render("report.Rmd", params = list(region = "us"))

for (r in c("eu", "us", "ap")) {
  # rmarkdown::render(
  #   "report.Rmd",
  #   params      = list(region = r),
  #   output_file = paste0("review_", r, ".html")
  # )
}

# QUARTO is the successor, with the same model and multi-language support.
# quarto render report.qmd -P region:us
# The same document can contain R, Python, and Julia chunks sharing state.

# OUTPUT FORMATS from one source: html_document, pdf_document (LaTeX),
# word_document, powerpoint_presentation, and dashboards via flexdashboard.

# THE REPRODUCIBILITY POINT, stated as a test:
#   delete every object in the environment, restart R, render.
#   If it fails, the document depended on hidden state and was never
#   reproducible. This is worth doing before sending anything.

Key Takeaways

• Literate programming keeps code, output, and prose in one file, so the report regenerates rather than being assembled by hand.
• Rendering runs in a fresh session, which makes a document that renders reproducible by construction.
• Chunk options control visibility and caching; stakeholder reports usually hide code while appendices show it.
• Inline code embeds computed values in sentences, eliminating typed numbers that go stale.
• Parameterized reports produce one document per segment from a single template.

🧪 Practice

1. Write a short report with one plot, one table, and at least two inline computed values, then change the input data and re-render.
2. Parameterize the report by region and generate three outputs from one template in a loop.
3. Interview: Why is "it renders from a clean session" a stronger reproducibility claim than "the script runs"? (Hint: what can a script quietly rely on that a fresh render cannot?)

<a id="43-sql-and-query-languages"></a>

4.3 SQL and Query Languages

SQL is where most analytical data actually lives, and pushing work into the database rather than pulling rows into Python is usually the difference between a query that returns in seconds and one that never finishes; this section covers the language from selection through window functions and the optimization habits that keep queries fast.

SELECT, WHERE, ORDER BY

SQL is declarative: you describe the result you want, and the database decides how to produce it. This is the opposite of pandas, where you specify each transformation step. The consequence is that two queries expressing the same result can run identically even when written very differently, and that you cannot debug by stepping through operations.

The single most useful thing to know is the logical evaluation order, which differs from the written order. The database conceptually evaluates FROM, then WHERE, then GROUP BY, then HAVING, then SELECT, then ORDER BY, then LIMIT.

That order explains two persistent confusions. A column alias defined in SELECT cannot be used in WHERE, because WHERE runs first. But it can be used in ORDER BY, because that runs after. Knowing the order turns both from arbitrary rules into consequences.

The other fundamental is three-valued logic. SQL has TRUE, FALSE, and UNKNOWN, and any comparison with NULL yields UNKNOWN. A WHERE clause keeps only rows where the condition is TRUE, so rows where it is UNKNOWN are dropped. This means WHERE status != 'active' silently excludes rows where status is NULL, which is almost never intended.

NULL is not a value; it is the absence of one. = NULL is never true, not even for a null. You must write IS NULL.

-- LOGICAL EVALUATION ORDER, which is not the written order:
--   FROM -> WHERE -> GROUP BY -> HAVING -> SELECT -> ORDER BY -> LIMIT

SELECT
    order_id,
    customer_id,
    amount,
    amount * 0.2                         AS estimated_tax,   -- an alias
    UPPER(region)                        AS region_code
FROM orders
WHERE order_date >= DATE '2024-01-01'
  AND amount > 100
  -- AND estimated_tax > 20             -- ERROR: WHERE runs before SELECT
ORDER BY estimated_tax DESC              -- fine: ORDER BY runs after SELECT
LIMIT 20;

-- THREE-VALUED LOGIC: the most common silent bug in SQL.
-- Suppose 1000 rows, 100 with status = 'active', 50 with status NULL.
SELECT COUNT(*) FROM orders WHERE status = 'active';      --  100
SELECT COUNT(*) FROM orders WHERE status != 'active';     --  850, not 900
SELECT COUNT(*) FROM orders WHERE status IS NULL;         --   50
-- The 50 NULL rows appear in NEITHER of the first two counts, because
-- NULL != 'active' evaluates to UNKNOWN, and WHERE keeps only TRUE.

-- The correct form when NULLs should be included:
SELECT COUNT(*) FROM orders
WHERE status IS DISTINCT FROM 'active';                   --  900
-- or, portably:
SELECT COUNT(*) FROM orders
WHERE status != 'active' OR status IS NULL;

-- NULL COMPARISON never works with =.
SELECT * FROM orders WHERE region = NULL;      -- always zero rows
SELECT * FROM orders WHERE region IS NULL;     -- correct

-- NULL HANDLING functions.
SELECT
    COALESCE(region, 'unknown')      AS region,   -- first non-null argument
    NULLIF(amount, 0)                AS amount,   -- NULL when it equals 0
    COALESCE(discount, 0) + amount   AS total     -- avoid NULL arithmetic
FROM orders;
-- Any arithmetic involving NULL yields NULL, so one missing discount
-- makes the whole total NULL rather than leaving it unchanged.

-- FILTERING vocabulary.
SELECT * FROM orders
WHERE amount BETWEEN 100 AND 500               -- inclusive on both ends
  AND region IN ('eu', 'us')
  AND customer_name LIKE 'A%'                  -- prefix match
  AND customer_email ILIKE '%@example.com'     -- case-insensitive, Postgres
  AND order_date >= CURRENT_DATE - INTERVAL '30 days';

-- CASE: conditional logic, usable anywhere an expression is allowed.
SELECT
    order_id,
    amount,
    CASE
        WHEN amount >= 1000 THEN 'large'
        WHEN amount >=  200 THEN 'medium'
        ELSE 'small'                            -- without ELSE, NULL
    END AS size_band
FROM orders;
-- Branches are evaluated in order and the first match wins, so put the
-- most specific condition first.

-- DISTINCT removes duplicate ROWS, considering every selected column.
SELECT DISTINCT region FROM orders;
SELECT DISTINCT region, status FROM orders;    -- distinct PAIRS

-- ORDER BY: direction and NULL placement are separate choices.
SELECT * FROM orders
ORDER BY region ASC, amount DESC NULLS LAST;
-- Databases disagree on where NULLs sort by default. State it.

-- LIMIT WITHOUT ORDER BY IS NON-DETERMINISTIC. The database may return
-- any 10 rows, and a different 10 on the next run.
SELECT * FROM orders ORDER BY order_id LIMIT 10;      -- reproducible
SELECT * FROM orders LIMIT 10;                        -- arbitrary

-- Deterministic paging needs a unique tiebreaker in the sort.
SELECT * FROM orders
ORDER BY order_date DESC, order_id DESC                -- unique overall
LIMIT 50 OFFSET 100;

  WRITTEN ORDER vs LOGICAL ORDER

  written:              logical:
    SELECT      5         FROM        1   pick the tables
    FROM        1         WHERE       2   filter ROWS
    WHERE       2         GROUP BY    3   form groups
    GROUP BY    3         HAVING      4   filter GROUPS
    HAVING      4         SELECT      5   compute output columns
    ORDER BY    6         ORDER BY    6   sort
    LIMIT       7         LIMIT       7   truncate

  Consequences you can now derive rather than memorise:
    - a SELECT alias is unusable in WHERE   (2 runs before 5)
    - a SELECT alias IS usable in ORDER BY  (6 runs after 5)
    - WHERE filters rows, HAVING filters groups
    - LIMIT without ORDER BY is arbitrary

  THREE-VALUED LOGIC

     x = 5      when x is 5      -> TRUE      kept
                when x is 7      -> FALSE     dropped
                when x is NULL   -> UNKNOWN   dropped

  So a filter and its negation together return FEWER rows than the table.

Key Takeaways

• SQL is declarative, and its logical evaluation order differs from the written order, which explains where aliases can and cannot be used.
• Comparisons with NULL yield UNKNOWN, and WHERE keeps only TRUE, so a filter and its negation both exclude null rows.
• NULL requires IS NULL rather than = NULL, and any arithmetic involving it propagates.
• COALESCE supplies a default and NULLIF creates a null, which together handle most null-safety needs.
• LIMIT without ORDER BY returns arbitrary rows, and stable paging needs a unique tiebreaker in the sort.

🧪 Practice

1. Write a query that counts rows matching a condition, rows failing it, and rows where the column is null, then show the three sum to the table total only when nulls are handled explicitly.
2. Write a query using a computed alias in ORDER BY and explain why the same alias fails in WHERE.
3. Interview: A report of "non-active customers" undercounts. What is the first thing you check? (Hint: what does a comparison against a null column return?)

Joins and Set Operations

Joins combine rows from several tables, and the mental model that makes them predictable is that a join produces every pair of rows satisfying the condition, then the join type decides what happens to unmatched rows.

INNER keeps only matched pairs. LEFT keeps all left rows, padding with nulls where no match exists. RIGHT is the mirror. FULL OUTER keeps everything. CROSS produces every combination with no condition at all.

The behaviour that surprises people is row multiplication. If a key appears twice on the left and three times on the right, the join emits six rows for that key. A SUM over the result is then wrong by a factor nobody notices, because the total is plausible. Checking key uniqueness before joining is the only reliable defence.

The subtlety with LEFT JOIN is the difference between a condition in the ON clause and one in WHERE. A condition in ON restricts what counts as a match, preserving unmatched left rows. The same condition in WHERE filters after the join, and since unmatched rows have nulls there, it removes them, silently converting the left join into an inner join.

Set operations combine query results vertically instead of horizontally. UNION removes duplicates and UNION ALL does not, which makes UNION ALL substantially faster and the right default when you know the inputs are disjoint.

-- SETUP: customers 1-4, orders referencing 1, 1, 2, and 99.

-- INNER: matched pairs only.
SELECT c.customer_id, c.name, o.order_id, o.amount
FROM customers c
INNER JOIN orders o ON c.customer_id = o.customer_id;
-- Customers with no orders vanish; order 99 vanishes.

-- LEFT: every customer, with nulls where there is no order.
SELECT c.customer_id, c.name, o.order_id
FROM customers c
LEFT JOIN orders o ON c.customer_id = o.customer_id;

-- FINDING NON-MATCHES: the anti-join pattern.
SELECT c.customer_id, c.name
FROM customers c
LEFT JOIN orders o ON c.customer_id = o.customer_id
WHERE o.order_id IS NULL;               -- customers who never ordered

-- The same thing with NOT EXISTS, usually clearer and often faster.
SELECT c.customer_id, c.name
FROM customers c
WHERE NOT EXISTS (
    SELECT 1 FROM orders o WHERE o.customer_id = c.customer_id
);
-- Prefer NOT EXISTS over NOT IN: if the subquery returns any NULL,
-- NOT IN returns no rows at all, which is a notorious silent failure.

-- ON versus WHERE with a LEFT JOIN: not interchangeable.
SELECT c.name, o.order_id
FROM customers c
LEFT JOIN orders o
       ON c.customer_id = o.customer_id
      AND o.amount > 100;               -- restricts what MATCHES
-- Customers with only small orders still appear, with NULL order_id.

SELECT c.name, o.order_id
FROM customers c
LEFT JOIN orders o ON c.customer_id = o.customer_id
WHERE o.amount > 100;                   -- filters AFTER the join
-- Unmatched rows have NULL amount, NULL > 100 is UNKNOWN, so they are
-- dropped. This is now an INNER JOIN written misleadingly.

-- ROW MULTIPLICATION: the bug that inflates every downstream total.
-- orders has 2 rows for customer 1; if a shipments table also has 2
-- rows for each of those orders, the join emits 4 rows and SUM(amount)
-- double counts.
SELECT o.customer_id, SUM(o.amount) AS inflated
FROM orders o
JOIN shipments s ON s.order_id = o.order_id
GROUP BY o.customer_id;

-- The fix: aggregate the many side FIRST, then join one-to-one.
SELECT o.customer_id, SUM(o.amount) AS correct_total
FROM orders o
JOIN (
    SELECT order_id, COUNT(*) AS shipment_count
    FROM shipments
    GROUP BY order_id
) s ON s.order_id = o.order_id
GROUP BY o.customer_id;

-- CHECK CARDINALITY BEFORE JOINING. One query, and it settles it.
SELECT COUNT(*) AS rows, COUNT(DISTINCT customer_id) AS distinct_keys
FROM orders;                            -- equal means unique

-- MULTI-TABLE joins chain left to right.
SELECT c.name, o.order_id, p.product_name, oi.quantity
FROM customers  c
JOIN orders     o  ON o.customer_id = c.customer_id
JOIN order_items oi ON oi.order_id   = o.order_id
JOIN products   p  ON p.product_id  = oi.product_id
WHERE o.order_date >= DATE '2024-01-01';

-- SELF JOIN: a table against itself, for hierarchies and comparisons.
SELECT e.name AS employee, m.name AS manager
FROM employees e
LEFT JOIN employees m ON e.manager_id = m.employee_id;

-- CROSS JOIN: every combination. Useful for building a complete grid
-- of dates and entities before a left join, so gaps become explicit.
SELECT d.date, s.store_id, COALESCE(f.sales, 0) AS sales
FROM dates d
CROSS JOIN stores s
LEFT JOIN facts f ON f.date = d.date AND f.store_id = s.store_id;

-- SET OPERATIONS combine vertically; column count and types must match.
SELECT customer_id FROM orders_2023
UNION ALL                                -- keeps duplicates: fast
SELECT customer_id FROM orders_2024;

SELECT customer_id FROM orders_2023
UNION                                    -- deduplicates: needs a sort
SELECT customer_id FROM orders_2024;

SELECT customer_id FROM orders_2023
INTERSECT
SELECT customer_id FROM orders_2024;     -- retained customers

SELECT customer_id FROM orders_2023
EXCEPT                                   -- MINUS in Oracle
SELECT customer_id FROM orders_2024;     -- churned customers

  JOIN TYPES, WITH THE SAME TWO TABLES

  customers          orders
  1 Alice            10 -> 1
  2 Bob              11 -> 1
  3 Carla            12 -> 2
                     13 -> 99

  INNER      3 rows   (10,11 for Alice; 12 for Bob)
  LEFT       4 rows   (+ Carla with NULL order)
  RIGHT      4 rows   (+ order 13 with NULL customer)
  FULL       5 rows   (both extras)
  CROSS      12 rows  (3 x 4, no condition)

  ON vs WHERE ON A LEFT JOIN

  LEFT JOIN o ON c.id = o.cid AND o.amount > 100
     -> Carla still appears, with NULLs.        LEFT JOIN preserved.

  LEFT JOIN o ON c.id = o.cid WHERE o.amount > 100
     -> Carla's amount is NULL, NULL > 100 is UNKNOWN, row dropped.
     -> silently an INNER JOIN.

Key Takeaways

• A join emits every matching pair, so duplicate keys multiply rows and inflate any downstream aggregate.
• Check that the join key is unique on at least one side before joining, and aggregate the many side first when it is not.
• A condition in ON restricts matching while the same condition in WHERE filters after the join and converts a left join to an inner one.
• Prefer NOT EXISTS over NOT IN, because a single null in the subquery makes NOT IN return nothing.
• UNION ALL skips deduplication and is markedly faster than UNION when inputs are known to be disjoint.

🧪 Practice

1. Write the same filtered left join twice, once with the condition in ON and once in WHERE, and explain the different row counts.
2. Construct a case where joining two tables inflates a revenue sum, then fix it by pre-aggregating.
3. Interview: A NOT IN subquery returns zero rows although you can see non-matching records. What is the cause? (Hint: what does the comparison return when the subquery contains a null?)

Aggregations and GROUP BY

Aggregation in SQL is the same split-apply-combine idea as pandas groupby, expressed declaratively. The rules are few and strict, and the strictness is what makes the results unambiguous.

The core rule: every column in SELECT must either appear in GROUP BY or be wrapped in an aggregate function. If a column is neither, the database cannot know which of the group's values to return, so standards-compliant databases raise an error. MySQL historically returned an arbitrary value, which is why that rule is worth stating explicitly.

WHERE and HAVING both filter, at different times. WHERE filters rows before grouping; HAVING filters groups after aggregating. Putting a condition in the right one matters for both correctness and speed: filtering early means fewer rows to aggregate.

The counting functions differ in ways that matter. COUNT(*) counts rows. COUNT(col) counts non-null values in that column. COUNT(DISTINCT col) counts distinct non-null values. Using the wrong one is a routine source of mismatched metrics between teams.

Aggregates other than COUNT(*) ignore nulls, which is usually helpful and occasionally wrong. AVG(col) divides by the count of non-null values, not by the row count. If nulls should count as zero, say so with COALESCE.

-- THE BASIC FORM.
SELECT
    region,
    COUNT(*)                      AS order_count,
    SUM(amount)                   AS total_revenue,
    AVG(amount)                   AS avg_order,
    MIN(amount)                   AS smallest,
    MAX(amount)                   AS largest,
    COUNT(DISTINCT customer_id)   AS unique_customers
FROM orders
WHERE order_date >= DATE '2024-01-01'     -- filters ROWS, before grouping
GROUP BY region
HAVING SUM(amount) > 10000                -- filters GROUPS, after
ORDER BY total_revenue DESC;

-- THE THREE COUNTS, which are not interchangeable.
SELECT
    COUNT(*)                   AS all_rows,          -- includes NULL rows
    COUNT(discount)            AS rows_with_discount,-- excludes NULLs
    COUNT(DISTINCT customer_id)AS distinct_customers
FROM orders;

-- AGGREGATES IGNORE NULLS, which changes the denominator of an average.
SELECT
    AVG(discount)                  AS avg_of_present,  -- n = non-null count
    SUM(discount) / COUNT(*)       AS avg_over_all,    -- n = row count
    AVG(COALESCE(discount, 0))     AS avg_treating_null_as_zero
FROM orders;
-- These three differ, and each answers a different question. Decide
-- which one the stakeholder means before reporting a number.

-- WHERE vs HAVING: both filter, at different stages.
SELECT region, SUM(amount) AS total
FROM orders
WHERE status = 'completed'         -- discard rows first: cheaper
GROUP BY region
HAVING COUNT(*) >= 10;             -- then discard small groups

-- Putting a row condition in HAVING works but scans more data:
-- HAVING can only reference grouped columns and aggregates anyway.

-- MULTI-COLUMN GROUPING produces one row per combination present.
SELECT region, product, SUM(amount) AS total, COUNT(*) AS n
FROM orders
GROUP BY region, product
ORDER BY region, total DESC;

-- CONDITIONAL AGGREGATION: the pivot idiom, and one of the most useful
-- patterns in analytical SQL.
SELECT
    region,
    COUNT(*)                                              AS total_orders,
    COUNT(*) FILTER (WHERE status = 'completed')          AS completed,
    COUNT(*) FILTER (WHERE status = 'cancelled')          AS cancelled,
    SUM(amount) FILTER (WHERE order_date >= DATE '2024-06-01') AS h2_revenue,
    AVG(CASE WHEN is_new_customer THEN amount END)        AS avg_new_customer
FROM orders
GROUP BY region;
-- FILTER is the standard form; CASE inside the aggregate is the portable
-- one, and works because aggregates skip the NULLs that CASE produces.

-- ROLLUP and CUBE add subtotal rows in one pass.
SELECT
    COALESCE(region, 'ALL')  AS region,
    COALESCE(product, 'ALL') AS product,
    SUM(amount)              AS total
FROM orders
GROUP BY ROLLUP (region, product)
ORDER BY region, product;
-- Produces: each region-product pair, a subtotal per region, and one
-- grand total. GROUPING SETS gives arbitrary combinations; CUBE gives all.

-- GROUPING() distinguishes a real NULL from a subtotal marker.
SELECT
    region,
    GROUPING(region) AS is_subtotal,
    SUM(amount)
FROM orders
GROUP BY ROLLUP (region);

-- STATISTICAL aggregates, available without leaving the database.
SELECT
    region,
    STDDEV_SAMP(amount)                                  AS sd,
    VAR_SAMP(amount)                                     AS variance,
    PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY amount)  AS median,
    PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY amount) AS p95,
    CORR(amount, discount)                               AS correlation
FROM orders
GROUP BY region;
-- The median is an ORDERED-SET aggregate and needs the WITHIN GROUP
-- syntax, because it depends on the order of values rather than a
-- running accumulation.

-- STRING and ARRAY aggregation collapse a group into one value.
SELECT
    customer_id,
    STRING_AGG(product_name, ', ' ORDER BY product_name) AS products,
    ARRAY_AGG(DISTINCT region)                           AS regions
FROM order_items
GROUP BY customer_id;

-- GROUPING BY AN EXPRESSION, such as a time bucket.
SELECT
    DATE_TRUNC('month', order_date) AS month,
    SUM(amount)                     AS revenue
FROM orders
GROUP BY DATE_TRUNC('month', order_date)      -- repeat it, or GROUP BY 1
ORDER BY month;

Key Takeaways

• Every selected column must be grouped or aggregated, because otherwise the database cannot choose which value to return.
• WHERE filters rows before grouping and HAVING filters groups after, so filter as early as the logic allows.
• COUNT(*), COUNT(col), and COUNT(DISTINCT col) answer different questions and are a common source of metric disagreement.
• Aggregates ignore nulls, so an average's denominator is the non-null count unless you wrap the column in COALESCE.
• Conditional aggregation with FILTER or CASE pivots categories into columns in a single pass.

🧪 Practice

1. Write one query returning, per region, the total order count plus separate counts of completed and cancelled orders as columns.
2. Show three different averages of a column containing nulls and explain which stakeholder question each answers.
3. Interview: Two teams report different "active customer" counts from the same table. Name three definitional differences that could cause it. (Hint: consider the counting function and the null handling.)

Subqueries and CTEs

As queries grow, the question becomes how to structure them. SQL offers subqueries and common table expressions, and the choice affects readability far more than performance in modern databases.

A subquery is a query nested inside another. It can appear in SELECT as a scalar, in FROM as a derived table, or in WHERE as a filter. The important distinction is between uncorrelated subqueries, which are evaluated once, and correlated ones, which reference the outer query and are conceptually evaluated per outer row.

Correlated subqueries are the performance hazard. A correlated subquery in SELECT over a million-row outer query may execute a million times. Modern optimizers often rewrite them as joins, but not always, and a query that performs fine on test data can fall over in production.

A CTE, written with WITH, names a query and makes it available to the statement that follows. The benefit is structural: a pipeline of named steps reads top to bottom, each step can be inspected independently, and a CTE can be referenced several times without repeating its text.

The historical warning about CTEs being "optimization fences" applied to older PostgreSQL. Since version 12 they are inlined by default unless materialization is requested. Treat CTEs as the default structuring tool and measure if you suspect a problem.

Recursive CTEs handle hierarchies and sequences, which plain SQL cannot express. They are the tool for organizational trees, bill-of-materials explosions, and generating a complete date series.

-- SCALAR SUBQUERY in SELECT: must return exactly one row and column.
SELECT
    order_id,
    amount,
    amount - (SELECT AVG(amount) FROM orders) AS diff_from_overall_avg
FROM orders;
-- Uncorrelated, so the inner query runs ONCE.

-- CORRELATED subquery: references the outer row, runs per row.
SELECT
    o.order_id,
    o.amount,
    (SELECT AVG(amount) FROM orders o2
      WHERE o2.region = o.region) AS region_avg      -- depends on o.region
FROM orders o;
-- Conceptually one execution per outer row. A window function does the
-- same thing in a single pass and is almost always preferable:
SELECT order_id, amount, AVG(amount) OVER (PARTITION BY region) AS region_avg
FROM orders;

-- DERIVED TABLE in FROM: a subquery used as a table.
SELECT region, avg_amount
FROM (
    SELECT region, AVG(amount) AS avg_amount
    FROM orders
    GROUP BY region
) AS region_stats
WHERE avg_amount > 200;

-- IN, EXISTS, and NOT EXISTS in WHERE.
SELECT * FROM customers
WHERE customer_id IN (SELECT customer_id FROM orders WHERE amount > 1000);

SELECT * FROM customers c
WHERE EXISTS (
    SELECT 1 FROM orders o                     -- SELECT 1: the value is
    WHERE o.customer_id = c.customer_id        -- irrelevant, existence is
      AND o.amount > 1000                      -- all that matters
);
-- EXISTS can stop at the first match, so it is often faster than IN,
-- and it is null-safe where NOT IN is not.

-- CTEs: the same pipeline, readable top to bottom.
WITH monthly_revenue AS (
    SELECT
        customer_id,
        DATE_TRUNC('month', order_date) AS month,
        SUM(amount)                     AS revenue
    FROM orders
    WHERE status = 'completed'
    GROUP BY customer_id, DATE_TRUNC('month', order_date)
),
customer_totals AS (
    SELECT
        customer_id,
        SUM(revenue)   AS lifetime_revenue,
        COUNT(*)       AS active_months,
        AVG(revenue)   AS avg_monthly
    FROM monthly_revenue                       -- references the CTE above
    GROUP BY customer_id
),
segmented AS (
    SELECT
        *,
        CASE
            WHEN lifetime_revenue >= 10000 THEN 'platinum'
            WHEN lifetime_revenue >=  2000 THEN 'gold'
            ELSE 'standard'
        END AS segment
    FROM customer_totals
)
SELECT segment, COUNT(*) AS customers, ROUND(AVG(lifetime_revenue)) AS avg_ltv
FROM segmented
GROUP BY segment
ORDER BY avg_ltv DESC;
-- Each step is independently inspectable: replace the final SELECT with
-- "SELECT * FROM monthly_revenue LIMIT 10" to debug the first stage.

-- A CTE REFERENCED TWICE, which a derived table cannot do without repeating.
WITH base AS (
    SELECT region, SUM(amount) AS total FROM orders GROUP BY region
)
SELECT
    b.region,
    b.total,
    ROUND(100.0 * b.total / (SELECT SUM(total) FROM base), 1) AS pct_of_all
FROM base b
ORDER BY b.total DESC;

-- RECURSIVE CTE: an organizational hierarchy.
WITH RECURSIVE org AS (
    -- anchor: the starting rows
    SELECT employee_id, name, manager_id, 1 AS level, name::text AS path
    FROM employees
    WHERE manager_id IS NULL

    UNION ALL

    -- recursive term: joins back to the CTE itself
    SELECT e.employee_id, e.name, e.manager_id, o.level + 1,
           o.path || ' > ' || e.name
    FROM employees e
    JOIN org o ON e.manager_id = o.employee_id
    WHERE o.level < 10                 -- ALWAYS bound the recursion
)
SELECT level, name, path FROM org ORDER BY path;
-- Without the depth bound, a cycle in the data loops forever.

-- RECURSIVE date generation, for filling gaps in a time series.
WITH RECURSIVE dates AS (
    SELECT DATE '2024-01-01' AS d
    UNION ALL
    SELECT d + 1 FROM dates WHERE d < DATE '2024-01-31'
)
SELECT d.d, COALESCE(SUM(o.amount), 0) AS revenue
FROM dates d
LEFT JOIN orders o ON o.order_date = d.d
GROUP BY d.d
ORDER BY d.d;
-- Days with no orders now appear as zero rather than being absent,
-- which any subsequent moving average requires.

Key Takeaways

• Uncorrelated subqueries run once while correlated ones conceptually run per outer row, which is the main performance risk.
• A correlated subquery computing a group statistic is nearly always better expressed as a window function.
• CTEs name pipeline steps so a long query reads top to bottom and each stage can be inspected in isolation.
• A CTE can be referenced several times, which a derived table cannot without repeating its text.
• Recursive CTEs handle hierarchies and generated sequences, and always need an explicit depth bound.

🧪 Practice

1. Rewrite a correlated subquery that computes a per-region average as a window function, and compare the execution plans.
2. Build a three-stage CTE pipeline that aggregates, segments, and summarizes, then debug it by selecting from an intermediate stage.
3. Interview: When would you choose EXISTS over IN? (Hint: consider both early termination and what happens when the subquery yields a null.)

Window Functions

Window functions are the most important analytical feature in SQL and the one most often missing from a self-taught repertoire. They compute a value for each row using a set of related rows, without collapsing the result.

That is the whole distinction. GROUP BY reduces many rows to one per group. A window function keeps every row and attaches a group-derived value to it. Running totals, rankings, period-over-period changes, and percent-of-total all need this, and none can be expressed with GROUP BY alone.

The syntax has three parts inside OVER. PARTITION BY defines the group, analogous to GROUP BY. ORDER BY defines the ordering within it, which ranking and cumulative functions require. The frame clause defines which rows around the current one are included.

The frame default is the trap. When you supply ORDER BY without a frame, the default is RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, which gives a running total. Without ORDER BY the frame is the whole partition. Also, RANGE includes all peer rows with the same sort value while ROWS counts physical rows, so a moving average over data with tied timestamps differs between them.

The three ranking functions differ only in tie handling, and the difference matters: ROW_NUMBER is arbitrary among ties, RANK leaves gaps, and DENSE_RANK does not.

-- THE CONTRAST: GROUP BY collapses, OVER does not.
SELECT region, SUM(amount) FROM orders GROUP BY region;      -- 3 rows

SELECT order_id, region, amount,
       SUM(amount) OVER (PARTITION BY region) AS region_total
FROM orders;                                                  -- all rows

-- RANKING, and how each handles ties.
SELECT
    product,
    revenue,
    ROW_NUMBER() OVER (ORDER BY revenue DESC) AS row_num,   -- 1,2,3,4
    RANK()       OVER (ORDER BY revenue DESC) AS rnk,       -- 1,2,2,4
    DENSE_RANK() OVER (ORDER BY revenue DESC) AS dense,     -- 1,2,2,3
    PERCENT_RANK() OVER (ORDER BY revenue DESC) AS pct_rank,
    NTILE(4)     OVER (ORDER BY revenue DESC) AS quartile
FROM product_revenue;
-- ROW_NUMBER among ties is nondeterministic unless the ORDER BY is
-- unique. Add a tiebreaker column when the result must be stable.

-- TOP N PER GROUP: the canonical window-function problem.
WITH ranked AS (
    SELECT
        region, product, revenue,
        ROW_NUMBER() OVER (PARTITION BY region ORDER BY revenue DESC) AS rn
    FROM product_revenue
)
SELECT region, product, revenue
FROM ranked
WHERE rn <= 3;
-- Window functions cannot appear in WHERE, because WHERE runs before
-- SELECT. Wrapping in a CTE is the standard workaround.

-- OFFSET FUNCTIONS: comparing a row to its neighbours.
SELECT
    order_date,
    revenue,
    LAG(revenue, 1)  OVER (ORDER BY order_date) AS prev_day,
    LEAD(revenue, 1) OVER (ORDER BY order_date) AS next_day,
    revenue - LAG(revenue) OVER (ORDER BY order_date) AS day_change,
    ROUND(100.0 * (revenue - LAG(revenue, 7) OVER (ORDER BY order_date))
          / NULLIF(LAG(revenue, 7) OVER (ORDER BY order_date), 0), 1)
        AS wow_pct,
    FIRST_VALUE(revenue) OVER (ORDER BY order_date) AS first_day,
    LAST_VALUE(revenue)  OVER (
        ORDER BY order_date
        ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
    ) AS last_day
FROM daily_revenue;
-- LAST_VALUE needs an explicit frame: with the default frame ending at
-- the current row, "last" means "current", which is a classic surprise.
-- NULLIF guards the division when the prior value is zero.

-- FRAMES: running totals and moving averages.
SELECT
    order_date,
    revenue,
    SUM(revenue) OVER (ORDER BY order_date) AS running_total,
    -- the default frame above is UNBOUNDED PRECEDING to CURRENT ROW

    AVG(revenue) OVER (
        ORDER BY order_date
        ROWS BETWEEN 6 PRECEDING AND CURRENT ROW      -- 7 rows, trailing
    ) AS ma7,

    AVG(revenue) OVER (
        ORDER BY order_date
        ROWS BETWEEN 3 PRECEDING AND 3 FOLLOWING      -- centred: LEAKAGE
    ) AS ma7_centred,

    SUM(revenue) OVER () AS grand_total               -- no ORDER BY: all rows
FROM daily_revenue;
-- The centred window uses future rows. Fine for a chart, wrong as a
-- feature for a model that must predict before those rows exist.

-- ROWS vs RANGE, which differ whenever the sort key has ties.
SELECT
    order_date, amount,
    SUM(amount) OVER (ORDER BY order_date
                      ROWS BETWEEN 1 PRECEDING AND CURRENT ROW)  AS by_rows,
    SUM(amount) OVER (ORDER BY order_date
                      RANGE BETWEEN INTERVAL '1 day' PRECEDING
                                AND CURRENT ROW)                 AS by_range
FROM orders;
-- ROWS counts physical rows; RANGE covers a value span and includes
-- every peer sharing the current sort value. With several orders per
-- day these produce very different numbers.

-- PERCENT OF TOTAL, at two levels at once.
SELECT
    region, product, revenue,
    ROUND(100.0 * revenue / SUM(revenue) OVER (PARTITION BY region), 1)
        AS pct_of_region,
    ROUND(100.0 * revenue / SUM(revenue) OVER (), 1) AS pct_of_company
FROM product_revenue;

-- NAMED WINDOWS remove repetition when several columns share a frame.
SELECT
    order_date,
    SUM(revenue) OVER w  AS running_total,
    AVG(revenue) OVER w  AS running_avg,
    COUNT(*)     OVER w  AS running_count
FROM daily_revenue
WINDOW w AS (ORDER BY order_date ROWS UNBOUNDED PRECEDING);

-- DEDUPLICATION, one of the most reused window patterns.
WITH numbered AS (
    SELECT *,
           ROW_NUMBER() OVER (PARTITION BY customer_id, order_date
                              ORDER BY updated_at DESC) AS rn
    FROM orders_raw
)
SELECT * FROM numbered WHERE rn = 1;      -- keep the latest per key

-- GAP AND ISLANDS: finding consecutive runs, such as login streaks.
WITH marked AS (
    SELECT
        user_id, activity_date,
        activity_date - (ROW_NUMBER() OVER (PARTITION BY user_id
                                            ORDER BY activity_date))::int
            AS streak_group      -- constant within a consecutive run
    FROM user_activity
)
SELECT user_id, MIN(activity_date) AS streak_start,
       MAX(activity_date) AS streak_end, COUNT(*) AS streak_length
FROM marked
GROUP BY user_id, streak_group
HAVING COUNT(*) >= 3
ORDER BY streak_length DESC;

  GROUP BY VERSUS OVER

  input                GROUP BY region        OVER (PARTITION BY region)

  eu  100              eu  485                eu  100  485
  us  250      --->    us  340        --->    us  250  340
  eu  175              ap  320                eu  175  485
  ap  320                                     ap  320  320
  eu  210              3 rows                 eu  210  485
  us   90                                     us   90  340

                                              6 rows: nothing collapsed

  FRAME DEFAULTS, the source of most surprises

  OVER (ORDER BY d)
     -> RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
     -> a RUNNING total, not the partition total

  OVER ()
     -> the whole partition
     -> the GRAND total

  OVER (ORDER BY d ROWS BETWEEN 6 PRECEDING AND CURRENT ROW)
     -> a 7-row TRAILING window, safe as a model feature

Key Takeaways

• Window functions attach a group-derived value to every row instead of collapsing rows, which GROUP BY cannot do.
• ROW_NUMBER, RANK, and DENSE_RANK differ only in tie handling, and ROW_NUMBER needs a unique sort to be deterministic.
• Supplying ORDER BY without a frame gives a running total by default, not the partition total.
• ROWS counts physical rows while RANGE includes all peers sharing the sort value, which differ whenever ties exist.
• Window functions cannot appear in WHERE, so top-N-per-group requires wrapping the ranking in a CTE.

🧪 Practice

1. Write a query returning the top three products by revenue within each region, using a ranking function and a CTE.
2. Compute a seven-day trailing average and a seven-day centred average in one query, and explain which is safe as a model feature.
3. Interview: LAST_VALUE returns the current row's value instead of the partition's last. Why, and what is the fix? (Hint: what frame applies when you supply ORDER BY?)

Query Optimization Basics

A query that works on ten thousand rows and fails on ten million usually has one of a small number of problems. Recognising them is a more productive skill than memorising database internals.

The first habit is to read the execution plan. EXPLAIN shows what the optimizer intends to do; EXPLAIN ANALYZE actually runs it and reports real timings and row counts. The single most valuable signal is a large discrepancy between estimated and actual rows, because the optimizer chose its strategy from the estimate, and a bad estimate means a bad plan.

The second is understanding indexes. An index is a sorted structure that turns a full scan into a targeted lookup, the same trade as a book's index. It costs storage and slows writes, so indexes are added deliberately, on columns used in WHERE, JOIN, and ORDER BY.

The rule that catches people is that wrapping an indexed column in a function disables the index. WHERE YEAR(order_date) = 2024 cannot use an index on order_date, because the index stores dates, not years. Rewriting as a range comparison restores it. The same applies to leading wildcards in LIKE and to implicit type casts.

For composite indexes, the leftmost prefix rule governs use. An index on (a, b, c) serves queries filtering on a, on a and b, or on all three, but not on b alone. The order of columns in the index is a design decision, not an arbitrary one.

Beyond indexing, the principle is to reduce data early. Filter before joining, select only needed columns, and aggregate before joining when the join would multiply rows.

-- READ THE PLAN FIRST. Never optimize from intuition.
EXPLAIN ANALYZE
SELECT c.name, SUM(o.amount)
FROM customers c
JOIN orders o ON o.customer_id = c.customer_id
WHERE o.order_date >= DATE '2024-01-01'
GROUP BY c.name;

-- What to look for, in priority order:
--  1. Seq Scan on a large table where a filter should have used an index
--  2. estimated rows vs actual rows differing by 10x or more
--  3. Nested Loop over a large outer input
--  4. Sort or Hash spilling to disk ("external merge Disk: 240MB")
--  5. the node with the largest actual time, which is where to work

-- INDEXES: the basic kinds.
CREATE INDEX idx_orders_date     ON orders (order_date);
CREATE INDEX idx_orders_cust     ON orders (customer_id);
CREATE INDEX idx_orders_cust_date ON orders (customer_id, order_date);
CREATE UNIQUE INDEX idx_customers_email ON customers (email);

-- THE LEFTMOST PREFIX RULE for composite indexes.
-- With an index on (customer_id, order_date):
--   WHERE customer_id = 5                          uses it
--   WHERE customer_id = 5 AND order_date > '...'   uses it fully
--   WHERE order_date > '...'                       CANNOT use it
-- Put the equality-filtered column first, the range-filtered one second.

-- SARGABILITY: a predicate the index can use.
-- NOT sargable: the column is wrapped in a function.
SELECT * FROM orders WHERE EXTRACT(YEAR FROM order_date) = 2024;
-- Sargable: the same condition as a range.
SELECT * FROM orders
WHERE order_date >= DATE '2024-01-01'
  AND order_date <  DATE '2025-01-01';

-- NOT sargable: a leading wildcard forces a full scan.
SELECT * FROM customers WHERE name LIKE '%smith';
-- Sargable: a prefix match can seek into the sorted index.
SELECT * FROM customers WHERE name LIKE 'smith%';

-- NOT sargable: arithmetic on the column.
SELECT * FROM orders WHERE amount * 1.2 > 100;
SELECT * FROM orders WHERE amount > 100 / 1.2;          -- sargable

-- An EXPRESSION INDEX when the function genuinely must stay.
CREATE INDEX idx_orders_year ON orders (EXTRACT(YEAR FROM order_date));

-- IMPLICIT CASTS silently disable indexes too.
-- If customer_id is a bigint:
SELECT * FROM orders WHERE customer_id = '12345';   -- may cast per row
SELECT * FROM orders WHERE customer_id = 12345;     -- correct type

-- REDUCE DATA EARLY.
-- Slower: join everything, then filter.
SELECT c.name, o.amount
FROM customers c
JOIN orders o ON o.customer_id = c.customer_id
WHERE o.order_date >= DATE '2024-06-01' AND c.region = 'eu';

-- Often faster: filter each side before joining.
WITH recent AS (
    SELECT customer_id, amount FROM orders
    WHERE order_date >= DATE '2024-06-01'
),
eu AS (
    SELECT customer_id, name FROM customers WHERE region = 'eu'
)
SELECT e.name, r.amount FROM eu e JOIN recent r USING (customer_id);
-- A good optimizer does this itself. Many do not, especially across
-- views and federated sources.

-- SELECT ONLY WHAT YOU NEED. SELECT * prevents index-only scans and
-- moves columns you never read across the network.
SELECT customer_id, amount FROM orders WHERE order_date > DATE '2024-01-01';

-- A COVERING INDEX lets the database answer without touching the table.
CREATE INDEX idx_cover ON orders (order_date) INCLUDE (customer_id, amount);

-- AGGREGATE BEFORE JOINING when the join would multiply rows.
SELECT c.name, o.total
FROM customers c
JOIN (
    SELECT customer_id, SUM(amount) AS total
    FROM orders GROUP BY customer_id          -- now one row per customer
) o ON o.customer_id = c.customer_id;

-- EXISTS instead of COUNT when you only need "any".
-- Slow: counts every matching row.
SELECT * FROM customers c
WHERE (SELECT COUNT(*) FROM orders o WHERE o.customer_id = c.customer_id) > 0;
-- Fast: stops at the first match.
SELECT * FROM customers c
WHERE EXISTS (SELECT 1 FROM orders o WHERE o.customer_id = c.customer_id);

-- UNION ALL when duplicates are impossible: skips a deduplicating sort.

-- KEEP STATISTICS CURRENT. Stale statistics cause bad estimates, which
-- cause bad plans, which is the most common cause of a query that was
-- fast last month and is slow today with no code change.
ANALYZE orders;

-- PARTITIONING for very large tables: the planner skips whole partitions.
-- CREATE TABLE orders (...) PARTITION BY RANGE (order_date);
-- A query filtered to one month then reads one partition, not the table.

Symptom in the plan	Likely cause	Remedy
Seq Scan on a big table	no usable index	add one, or make the predicate sargable
estimate far from actual	stale statistics	ANALYZE
Nested Loop, large outer	bad join order	fix estimates, or pre-filter
Sort spilling to disk	insufficient work memory	reduce rows earlier, or raise the limit
Index exists but unused	function or cast on the column	rewrite as a range

Key Takeaways

• Start from EXPLAIN ANALYZE, and treat a large gap between estimated and actual rows as the primary signal of a bad plan.
• Indexes speed reads and slow writes, so add them deliberately on columns used in filters, joins, and sorts.
• Wrapping an indexed column in a function, a leading wildcard, or an implicit cast all disable the index.
• Composite indexes serve only leftmost prefixes, so column order is a design decision.
• Reduce rows and columns as early as possible, and aggregate before a join that would otherwise multiply rows.

🧪 Practice

1. Run EXPLAIN ANALYZE on a query filtering by year with a function, rewrite it as a date range, and compare the plans.
2. Create a composite index and demonstrate a query that uses it and one that cannot, explaining why.
3. Interview: A nightly query that ran in 30 seconds now takes 20 minutes with no code change. What are your first three checks? (Hint: consider data volume, statistics, and index health.)

Analytical SQL Patterns

Certain analytical questions recur across every organization, and each has a standard SQL form. Learning the patterns is more valuable than learning more syntax, because the syntax is already covered.

The patterns below share a structure: a CTE pipeline that prepares a clean base, a window function that computes the per-row analytic, and a final aggregation. Recognising which pattern a question maps to turns a vague request into a query you can write directly.

Two cautions apply throughout. First, define the metric before writing the query. "Active user" and "churn" have several defensible definitions, and the SQL is the definition, so it belongs in code review alongside the logic. Second, generate a complete date spine before any time series calculation, because absent rows are silently treated as gaps rather than zeros and a moving average over sparse data is wrong.

Cohort retention, funnel conversion, running totals, period comparison, sessionization, and top-N-per-group cover the large majority of analytical requests. Each appears below in a form you can adapt.

-- 1. COHORT RETENTION: do users acquired in month M still return later?
WITH first_purchase AS (
    SELECT customer_id,
           DATE_TRUNC('month', MIN(order_date)) AS cohort_month
    FROM orders GROUP BY customer_id
),
activity AS (
    SELECT DISTINCT
           o.customer_id,
           f.cohort_month,
           DATE_TRUNC('month', o.order_date) AS active_month
    FROM orders o
    JOIN first_purchase f USING (customer_id)
),
sized AS (
    SELECT cohort_month,
           active_month,
           COUNT(DISTINCT customer_id) AS active_customers,
           (EXTRACT(YEAR  FROM active_month) - EXTRACT(YEAR  FROM cohort_month)) * 12
         + (EXTRACT(MONTH FROM active_month) - EXTRACT(MONTH FROM cohort_month))
             AS months_since
    FROM activity
    GROUP BY cohort_month, active_month
)
SELECT
    cohort_month,
    months_since,
    active_customers,
    ROUND(100.0 * active_customers
          / FIRST_VALUE(active_customers) OVER (
                PARTITION BY cohort_month ORDER BY months_since), 1)
        AS retention_pct
FROM sized
ORDER BY cohort_month, months_since;

-- 2. FUNNEL CONVERSION: how many users reach each step, in order?
WITH steps AS (
    SELECT
        user_id,
        MIN(CASE WHEN event = 'view'     THEN event_time END) AS t_view,
        MIN(CASE WHEN event = 'add_cart' THEN event_time END) AS t_cart,
        MIN(CASE WHEN event = 'checkout' THEN event_time END) AS t_checkout,
        MIN(CASE WHEN event = 'purchase' THEN event_time END) AS t_purchase
    FROM events
    GROUP BY user_id
)
SELECT
    COUNT(*)                                                    AS viewed,
    COUNT(*) FILTER (WHERE t_cart     > t_view)                 AS carted,
    COUNT(*) FILTER (WHERE t_checkout > t_cart)                 AS checked_out,
    COUNT(*) FILTER (WHERE t_purchase > t_checkout)             AS purchased,
    ROUND(100.0 * COUNT(*) FILTER (WHERE t_purchase > t_checkout)
          / NULLIF(COUNT(*), 0), 2)                             AS overall_pct
FROM steps
WHERE t_view IS NOT NULL;
-- Comparing timestamps rather than mere presence enforces ORDER, so a
-- user who purchased before viewing is not counted as converted.

-- 3. THE DATE SPINE: generate every date, then left join the facts.
WITH spine AS (
    SELECT generate_series(DATE '2024-01-01', DATE '2024-03-31',
                           INTERVAL '1 day')::date AS d
),
daily AS (
    SELECT s.d, COALESCE(SUM(o.amount), 0) AS revenue
    FROM spine s
    LEFT JOIN orders o ON o.order_date = s.d
    GROUP BY s.d
)
SELECT
    d,
    revenue,
    SUM(revenue) OVER (ORDER BY d) AS running_total,
    ROUND(AVG(revenue) OVER (ORDER BY d ROWS BETWEEN 6 PRECEDING
                                            AND CURRENT ROW), 2) AS ma7
FROM daily
ORDER BY d;
-- Without the spine, a zero-revenue day is an ABSENT row, so the 7-row
-- window spans more than 7 calendar days and the average is wrong.

-- 4. PERIOD-OVER-PERIOD comparison.
WITH monthly AS (
    SELECT DATE_TRUNC('month', order_date) AS month, SUM(amount) AS revenue
    FROM orders GROUP BY 1
)
SELECT
    month,
    revenue,
    LAG(revenue)     OVER (ORDER BY month) AS prev_month,
    ROUND(100.0 * (revenue - LAG(revenue) OVER (ORDER BY month))
          / NULLIF(LAG(revenue) OVER (ORDER BY month), 0), 1) AS mom_pct,
    LAG(revenue, 12) OVER (ORDER BY month) AS same_month_last_year,
    ROUND(100.0 * (revenue - LAG(revenue, 12) OVER (ORDER BY month))
          / NULLIF(LAG(revenue, 12) OVER (ORDER BY month), 0), 1) AS yoy_pct
FROM monthly
ORDER BY month;
-- Year-over-year comparison controls for seasonality that
-- month-over-month cannot.

-- 5. SESSIONIZATION: group events into sessions with a 30-minute gap.
WITH gaps AS (
    SELECT
        user_id, event_time,
        CASE WHEN event_time - LAG(event_time) OVER (PARTITION BY user_id
                                                     ORDER BY event_time)
                  > INTERVAL '30 minutes'
             OR LAG(event_time) OVER (PARTITION BY user_id
                                      ORDER BY event_time) IS NULL
             THEN 1 ELSE 0 END AS is_new_session
    FROM events
),
sessions AS (
    SELECT user_id, event_time,
           SUM(is_new_session) OVER (PARTITION BY user_id
                                     ORDER BY event_time) AS session_id
    FROM gaps                        -- a running sum of the new-session flag
)                                    -- assigns a stable id per session
SELECT
    user_id, session_id,
    MIN(event_time) AS started,
    MAX(event_time) AS ended,
    COUNT(*)        AS events,
    MAX(event_time) - MIN(event_time) AS duration
FROM sessions
GROUP BY user_id, session_id
ORDER BY user_id, started;

-- 6. RFM SEGMENTATION: recency, frequency, monetary value.
WITH rfm AS (
    SELECT
        customer_id,
        CURRENT_DATE - MAX(order_date) AS recency_days,
        COUNT(*)                       AS frequency,
        SUM(amount)                    AS monetary
    FROM orders
    WHERE order_date >= CURRENT_DATE - INTERVAL '365 days'
    GROUP BY customer_id
),
scored AS (
    SELECT *,
        NTILE(5) OVER (ORDER BY recency_days DESC) AS r_score,  -- recent = 5
        NTILE(5) OVER (ORDER BY frequency)         AS f_score,
        NTILE(5) OVER (ORDER BY monetary)          AS m_score
    FROM rfm
)
SELECT
    customer_id, recency_days, frequency, monetary,
    r_score, f_score, m_score,
    CASE
        WHEN r_score >= 4 AND f_score >= 4 THEN 'champion'
        WHEN r_score >= 4                  THEN 'promising'
        WHEN r_score <= 2 AND f_score >= 4 THEN 'at risk'
        ELSE 'standard'
    END AS segment
FROM scored
ORDER BY monetary DESC;

-- 7. TOP N PER GROUP with a deterministic tiebreaker.
WITH ranked AS (
    SELECT region, product, revenue,
           ROW_NUMBER() OVER (PARTITION BY region
                              ORDER BY revenue DESC, product ASC) AS rn
    FROM product_revenue
)
SELECT * FROM ranked WHERE rn <= 3 ORDER BY region, rn;

Key Takeaways

• Most analytical requests map to a small set of patterns built from a CTE pipeline, a window function, and a final aggregation.
• Define the metric in writing before coding it, since the query becomes the operative definition.
• Generate a complete date spine before any time series calculation, or absent rows will distort moving averages and running totals.
• Funnels should compare timestamps rather than mere presence, so that step order is actually enforced.
• Sessionization uses a running sum over a new-session flag, which is the same gaps-and-islands technique that finds streaks.

🧪 Practice

1. Write a cohort retention query and present the output as a triangle with cohort months as rows and months-since as columns.
2. Build a sessionization query with a configurable inactivity gap, then report median session length per user segment.
3. Interview: A seven-day moving average of daily signups looks wrong on low-traffic days. What is the most likely structural cause? (Hint: what happens to a day with zero signups in the source table?)

<a id="44-development-environment-and-practices"></a>

4.4 Development Environment and Practices

Analytical code is still code, and the practices that keep software maintainable apply with one addition: data changes underneath you, so reproducibility requires pinning the environment and the data as well as the logic; this section covers notebooks, version control, project structure, testing, and containers.

Jupyter Notebooks and Notebook Hygiene

Notebooks are the dominant interface for exploratory data work, for good reasons and at a real cost. Understanding both determines whether they help or quietly undermine your results.

The strength is the tight feedback loop. Load an expensive dataset once, then iterate on transformations against the live object with plots and tables inline. For exploration, nothing else comes close.

The weakness is hidden state. A notebook's output reflects the order cells were executed, not the order they appear. Deleting a cell does not remove the variables it created. Editing a cell and not re-running it leaves output that no longer matches the code. A notebook can display a result that no execution of its current code could reproduce.

The execution counter is the tell. Cells numbered [1] [5] [2] [12] mean the visible order is not the run order, and any conclusion drawn from that state is unverified. The habit that fixes this is restart and run all before trusting or sharing anything.

The second problem is version control. A notebook is JSON containing outputs, execution counts, and base64-encoded images, so a trivial change produces an unreadable diff and merges conflict constantly. Stripping outputs before commit solves most of it.

The mature workflow is a division of labour: explore in the notebook, and once logic stabilises, move it into an importable module that the notebook then calls. The notebook stays a thin narrative; the logic becomes testable.

# THE HIDDEN STATE PROBLEM, made concrete.
# Cell [1]:  df = load_data()
# Cell [2]:  df = df[df.amount > 0]
# Cell [3]:  print(len(df))          -> 950
#
# You now edit cell [2] to df[df.amount > 100] but re-run only cell [3].
# It still prints 950, because df in memory reflects the OLD filter.
# The notebook now shows code that does not produce the output beside it.

# DIAGNOSTIC: list variables and check the execution counter.
# %whos                       shows every variable in memory
# %reset -f                   clears everything, forcing a clean rerun

# THE DISCIPLINE, in order of value:
#  1. Restart and Run All before trusting, sharing, or committing.
#  2. Keep imports and configuration in the FIRST cell.
#  3. Never define a function in one cell and edit it in another.
#  4. Write cells to be idempotent: running twice gives the same result.

# NOT IDEMPOTENT: each run mutates the same object further.
df["amount"] = df["amount"] * 1.1          # run twice, compounded twice

# IDEMPOTENT: derived from a source that does not change.
df["amount_adjusted"] = df["amount_raw"] * 1.1

# AUTORELOAD lets you edit a module and see changes without restarting,
# which is what makes the notebook-plus-module workflow practical.
# %load_ext autoreload
# %autoreload 2
from src.features import build_features      # your own module
# Edit src/features.py, re-run the cell, and the new code is used.

# USEFUL MAGICS.
# %timeit expr           microbenchmark a single expression
# %%time                 wall time for a whole cell
# %prun function()       profile by function
# %debug                 post-mortem debugger after an exception
# %matplotlib inline     render figures in the notebook

# PARAMETERS AT THE TOP, never scattered through the document.
CONFIG = {
    "input_path": "data/raw/orders.parquet",
    "start_date": "2024-01-01",
    "min_amount": 100.0,
    "random_seed": 42,
}
# This also makes the notebook executable by papermill with overrides.

# VERSION CONTROL: strip outputs so diffs are readable.
pip install nbstripout
nbstripout --install            # installs a git filter for this repo
# Committed notebooks now contain code and markdown only.

# Human-readable diffs for review.
pip install nbdime
nbdiff notebook.ipynb           # cell-level diff, not JSON noise
nbdime config-git --enable

# PAIRING with a plain .py file: jupytext keeps both in sync.
pip install jupytext
jupytext --set-formats ipynb,py:percent notebook.ipynb
# Commit only the .py, which diffs and merges like normal source.

# EXECUTE NON-INTERACTIVELY, which is the real reproducibility test.
jupyter nbconvert --to notebook --execute analysis.ipynb \
        --output executed.ipynb
# If this fails, the notebook depended on state that no longer exists.

# PARAMETERIZED EXECUTION for one report per segment.
pip install papermill
papermill analysis.ipynb out_eu.ipynb -p region eu -p min_amount 50

  WHY EXECUTION ORDER MATTERS

  what the notebook SHOWS          what actually RAN

  [1]  import pandas               1. import pandas
  [5]  df = load("v2.csv")         2. df = load("v1.csv")   <- edited since
  [2]  df = clean(df)              3. df = clean(df)
  [12] model.fit(df)               4. plot(df)
  [4]  plot(df)                    5. df = load("v2.csv")   <- rerun alone
                                   12. model.fit(df)        <- UNCLEANED df

  The counters [1][5][2][12][4] are the warning. The model was fitted on
  data that never passed through clean(), and nothing on screen says so.

  THE ONLY RELIABLE CHECK: Restart Kernel and Run All Cells.
  If the conclusions change, they were never real.

Key Takeaways

• Notebook output reflects execution order, not cell order, so a notebook can display results its current code cannot reproduce.
• Out-of-sequence execution counters are the visible warning sign; restart and run all before trusting or sharing anything.
• Write cells to be idempotent so that re-running one does not compound a transformation.
• Strip outputs with nbstripout or pair with jupytext so notebooks diff and merge like source code.
• Move stabilised logic into an importable module and use autoreload, keeping the notebook as narrative rather than implementation.

🧪 Practice

1. Deliberately create a notebook whose displayed output contradicts its code, then show that restart-and-run-all exposes it.
2. Configure nbstripout on a repository and compare the diff of a small change before and after.
3. Interview: A colleague's notebook produces a different number on your machine with the same data. Name three notebook-specific causes. (Hint: consider what lives in memory rather than in the file.)

IDEs for Data Work

An editor is where you spend the day, and the features that matter for data work differ somewhat from general software development. The main additions are interactive execution and data inspection.

VS Code is the common default. It runs notebooks natively, supports an interactive window where a plain .py file can be executed cell by cell using # %% markers, and gives a variable explorer and dataframe viewer. Crucially, it treats the file as real Python, so linting, refactoring, and version control work normally.

PyCharm offers deeper static analysis, the strongest refactoring tools, and an integrated debugger that handles remote interpreters well. It is heavier and worth it on larger codebases.

JupyterLab is the notebook-native environment, with a file browser, terminals, and extensions in one interface. It is the right choice when the work is genuinely notebook-shaped and shared with people who will not install an editor.

RStudio and Positron serve R work, and Positron is notable for treating Python and R as equal citizens with a shared console, variable pane, and plot history.

Whichever you choose, the setup that pays for itself is the same: a formatter on save, a linter, a type checker, and the debugger. The # %% cell marker convention deserves particular mention, because it gives notebook-style interactivity in a file that diffs cleanly.

# THE #%% CELL CONVENTION: notebook interactivity in a plain .py file.
# VS Code, PyCharm, and Spyder all recognise these markers.

# %% [markdown]
# ## Load and inspect
# Prose cells work too, so the file reads like a notebook.

# %%
import pandas as pd
import numpy as np

df = pd.read_parquet("data/orders.parquet")
df.head()                      # rendered in the interactive window

# %%
summary = df.groupby("region")["amount"].agg(["sum", "mean", "count"])
summary                        # run this cell alone with shift+enter

# %%
# The file is ordinary Python: it diffs cleanly, imports normally,
# and can be executed end to end from the command line.

// .vscode/settings.json -- the setup worth copying into every project.
{
  "python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python",
  "editor.formatOnSave": true,
  "editor.codeActionsOnSave": { "source.organizeImports": "explicit" },
  "[python]": { "editor.defaultFormatter": "charliermarsh.ruff" },
  "python.analysis.typeCheckingMode": "basic",
  "jupyter.interactiveWindow.textEditor.executeSelection": true,
  "notebook.output.textLineLimit": 60,
  "files.trimTrailingWhitespace": true
}

# pyproject.toml -- one formatter and linter, configured once.
[tool.ruff]
line-length = 88
target-version = "py311"

[tool.ruff.lint]
select = ["E", "F", "I", "UP", "B", "SIM", "PD"]
# E/F pycodestyle and pyflakes   I  import sorting
# UP  modern syntax              B  likely bugs
# SIM simplifications            PD pandas-specific checks
ignore = ["E501"]                # the formatter handles line length

[tool.mypy]
python_version = "3.11"
warn_return_any = true
ignore_missing_imports = true    # many data libraries lack stubs

# THE DEBUGGER beats print statements, and data work is where it helps
# most, because you can inspect a whole dataframe at the breakpoint.
def compute_margin(df):
    breakpoint()            # execution stops here; inspect df interactively
    return (df["revenue"] - df["cost"]) / df["revenue"]

# At the prompt:
#   df.shape               check the size
#   df.dtypes              check the types
#   df["cost"].isna().sum()check the nulls
#   (df["revenue"] == 0).sum()   find the division-by-zero source
#   c                      continue

# POST-MORTEM debugging after an exception, in a notebook: %debug
# From the command line:  python -m pdb -c continue script.py

# TYPE HINTS pay off in data code, where shape and type errors dominate.
from typing import Literal
import pandas as pd

def summarize(
    df: pd.DataFrame,
    group_col: str,
    value_col: str,
    how: Literal["sum", "mean", "median"] = "sum",
) -> pd.DataFrame:
    """Aggregate value_col by group_col.

    Returns a frame with columns [group_col, value_col].
    """
    return df.groupby(group_col, as_index=False)[value_col].agg(how)
# The signature documents the contract, the editor autocompletes it, and
# mypy catches a wrong `how` before the code runs.

Environment	Strongest at	Choose when
VS Code	notebooks plus real editing	general data work
PyCharm	refactoring, static analysis	large Python codebases
JupyterLab	notebook-native workflow	shared, notebook-shaped work
RStudio	R, R Markdown, Shiny	primarily R
Positron	Python and R side by side	genuinely bilingual teams

Key Takeaways

• The # %% cell marker gives notebook-style interactivity in a plain Python file that diffs and imports normally.
• Configure the interpreter path to the project's virtual environment, or the editor will silently use a different one.
• A formatter on save plus a linter removes an entire category of review comments and keeps diffs focused on logic.
• The debugger is more useful than print statements in data work, because you can interrogate a whole frame at the breakpoint.
• Type hints document shape and type contracts that data code violates most often.

🧪 Practice

1. Convert an exploratory notebook into a # %%-annotated Python file and confirm it both runs end to end and diffs readably.
2. Configure ruff and mypy in pyproject.toml, then fix every issue they report on an existing script.
3. Interview: Your editor reports an import error for a package you know is installed. What is the first thing you check? (Hint: which interpreter is the editor actually using?)

Version Control with Git

Git tracks the history of a project so that any past state can be restored and concurrent work can be merged. For analytical work it also serves a second purpose: it is the record of which code produced which result.

The model is worth understanding rather than memorising commands. Git stores snapshots of the whole tree, not diffs, and each commit points to its parent, forming a graph. A branch is just a movable pointer to a commit, which is why branching is instant and cheap.

Work moves through three places. The working directory holds your files. The staging area holds what will go into the next commit. The repository holds committed history. git add moves changes to staging, git commit moves staging to history, and understanding this removes most confusion about why a change did not appear in a commit.

For data work there are two specific concerns. First, never commit large data or secrets. Git keeps everything forever, so a committed hundred- megabyte CSV bloats every future clone, and a committed credential must be treated as compromised even after removal.

Second, notebooks need the handling described earlier. Stripping outputs turns an unmergeable JSON blob into reviewable source.

# THE CORE LOOP.
git status                        # run this constantly
git add src/features.py           # stage specific files
git add -p                        # stage selected HUNKS: review as you go
git commit -m "Add rolling window features for churn model"
git log --oneline --graph --all   # see the shape of history

# BRANCHING is cheap; use one per piece of work.
git switch -c feature/churn-model
git switch main
git merge feature/churn-model
git branch -d feature/churn-model

# A GOOD COMMIT MESSAGE explains WHY, since the diff shows WHAT.
git commit -m "Fix revenue double-counting in regional report

The orders-to-shipments join fanned out for multi-shipment orders,
inflating regional totals by roughly 12%. Aggregate shipments per
order before joining.

Affects: reports/regional.sql, tests/test_regional.py"

# UNDOING, from safest to most destructive.
git restore file.py                    # discard working-directory changes
git restore --staged file.py           # unstage, keep the edit
git commit --amend                     # rewrite the last commit (unpushed)
git revert <sha>                       # a NEW commit undoing an old one
git reset --soft HEAD~1                # undo the commit, keep the changes
git reset --hard HEAD~1                # undo the commit AND the changes
# --hard discards work permanently. Prefer revert on anything pushed.

git stash                              # shelve work in progress
git stash pop

# FINDING THINGS.
git log -S "groupby" --oneline         # commits that changed this string
git blame src/features.py              # who last touched each line, and when
git bisect start                       # binary search for a breaking commit
git log --follow -p src/features.py    # full history of one file
git reflog                             # recover from a bad reset

# WHAT NOT TO COMMIT.
cat > .gitignore <<'EOF'
# data
data/raw/
data/processed/
*.csv
*.parquet
*.db

# secrets
.env
*.pem
credentials.json

# environments and caches
.venv/
__pycache__/
.ipynb_checkpoints/
.pytest_cache/

# models and outputs
models/*.pkl
outputs/
EOF

# If a secret was committed, ROTATE IT. Removing it from history does not
# undo the exposure, because clones and forks retain it.
pip install detect-secrets
detect-secrets scan > .secrets.baseline

# LARGE FILES: Git LFS stores pointers in git and blobs elsewhere.
git lfs install
git lfs track "*.parquet"
git add .gitattributes

# DATA VERSIONING: DVC tracks data alongside code without storing it in git.
dvc init
dvc add data/raw/orders.parquet    # creates a small .dvc pointer file
git add data/raw/orders.parquet.dvc .gitignore
dvc remote add -d storage s3://my-bucket/dvc
dvc push                           # data to S3, pointer to git
# git checkout <old-commit> && dvc pull  restores the matching data.
# This is what makes "which data produced this result" answerable.

# PRE-COMMIT HOOKS enforce hygiene automatically.
cat > .pre-commit-config.yaml <<'EOF'
repos:
  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.5.0
    hooks:
      - id: ruff
        args: [--fix]
      - id: ruff-format
  - repo: https://github.com/kynan/nbstripout
    rev: 0.7.1
    hooks:
      - id: nbstripout
  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.5.0
    hooks:
      - id: detect-secrets
EOF
pre-commit install

  THE THREE AREAS

  working directory  --git add-->  staging area  --git commit-->  repository
       (edits)                      (what goes                     (history)
                                     in next)

  git restore <f>          discard working changes
  git restore --staged <f> unstage, keeping the edit
  git commit               staging -> history

  WHY A CHANGE "DID NOT APPEAR": it was never staged.

  BRANCHES ARE POINTERS

     A---B---C        main
          \
           D---E      feature

  merge   -> a new commit F with parents C and E, history preserved
  rebase  -> replay D,E on top of C, linear history, rewritten SHAs
             Never rebase a branch others have pulled.

Key Takeaways

• Git stores snapshots linked to parents, and a branch is a movable pointer, which is why branching costs nothing.
• Changes must be staged before they enter a commit, which explains most "my change is missing" confusion.
• Write commit messages that explain why, since the diff already shows what changed.
• Never commit data or secrets; a committed credential is compromised and must be rotated even after removal.
• Pair git with DVC or LFS so the data that produced a result is recoverable alongside the code.

🧪 Practice

1. Make three commits on a branch, then use git log -S to find which one introduced a specific function call.
2. Configure pre-commit with a formatter, notebook stripping, and secret detection, then verify each hook fires.
3. Interview: You committed an API key three weeks ago and have since pushed. Removing it from history is not enough. Why? (Hint: who else has a copy?)

Command Line Essentials

The shell is faster than any GUI for the things data work needs constantly: inspecting a file too large to open, counting matches, chaining transformations, and running work on a remote machine.

The organising principle is composition. Each tool does one thing and reads from standard input, writing to standard output. A pipe connects them, so arbitrary processing pipelines assemble from small pieces. This is the same idea as method chaining, applied to processes.

The property that makes this valuable for large data is streaming. A pipe passes data through without materializing it, so counting matching lines in a fifty-gigabyte file uses almost no memory and starts producing output immediately.

The tools worth knowing are few. head, tail, and less inspect. wc counts. grep filters by pattern. cut, sort, uniq, and awk manipulate delimited text. find and xargs operate over many files. That set covers most of what a data practitioner needs day to day.

Two modern additions earn their place: jq for JSON, and duckdb for running SQL directly against CSV and Parquet files without loading them anywhere.

# INSPECTION without loading the file.
head -5 data.csv                 # first rows, including the header
tail -100 app.log                # last rows
tail -f app.log                  # follow a growing file live
wc -l data.csv                   # line count, minus one for the header
du -h data.csv                   # file size
file data.csv                    # type and encoding

# COUNTING AND FILTERING, streamed.
grep -c "ERROR" app.log                      # count matching lines
grep -i "timeout" app.log | head -20         # case-insensitive
grep -v "^#" config.txt                      # INVERT: drop comments
grep -rn "TODO" src/                         # recursive, with line numbers
grep -E "^(2024-0[1-3])" app.log             # extended regex

# COLUMN OPERATIONS on delimited text.
cut -d',' -f2,4 data.csv | head              # select fields 2 and 4
awk -F',' '{print $2, $4}' data.csv | head   # same, more flexible
awk -F',' 'NR > 1 {sum += $3} END {print sum}' data.csv    # sum column 3
awk -F',' 'NR > 1 && $3 > 100' data.csv      # filter rows by a value
awk -F',' 'NR==1 {print; next} $5 == "eu"' data.csv   # keep the header

# THE MOST USEFUL PIPELINE IN DATA WORK: value counts on a column.
tail -n +2 data.csv | cut -d',' -f5 | sort | uniq -c | sort -rn | head
#   skip header  | take a field  | sort | count | rank | show top
# This answers "what are the distinct values and how common" on a file
# far too large to open, in seconds, with no Python involved.

# FINDING DUPLICATES and uniques.
sort data.csv | uniq -d          # lines appearing more than once
sort data.csv | uniq -u          # lines appearing exactly once
sort -u data.csv | wc -l         # distinct line count

# SAMPLING a large file, which beats loading it to look.
shuf -n 1000 data.csv > sample.csv
head -1 data.csv > sample.csv && tail -n +2 data.csv | shuf -n 1000 >> sample.csv

# SPLITTING for parallel processing.
split -l 1000000 big.csv chunk_      # one million lines per piece

# FIND and XARGS over many files.
find data/ -name "*.csv" -size +100M
find . -name "*.py" -mtime -7                       # changed in a week
find data/ -name "*.csv" | xargs wc -l | tail -1    # total lines
find . -name "*.parquet" -print0 | xargs -0 -P 4 -I{} python process.py {}
# -P 4 runs four in parallel; -print0/-0 handles spaces in filenames.

# JSON with jq.
cat events.json | jq '.[] | select(.amount > 100) | .user_id'
cat events.json | jq -r '.[] | [.id, .amount] | @csv' > out.csv
cat nested.json | jq '.results | length'

# SQL DIRECTLY OVER FILES with duckdb: often the fastest way to answer
# a question about a Parquet or CSV file on disk.
duckdb -c "SELECT region, COUNT(*), SUM(amount)
           FROM 'data/*.parquet' GROUP BY region ORDER BY 2 DESC"
duckdb -c "COPY (SELECT * FROM 'big.csv' WHERE amount > 100)
           TO 'filtered.parquet' (FORMAT parquet)"
# No loading step, no memory limit tied to file size, and it reads the
# whole glob as one table.

# LONG-RUNNING WORK on a remote machine.
nohup python train.py > train.log 2>&1 &     # survives logout
tmux new -s training                          # detachable session
# ctrl-b d to detach, tmux attach -t training to return

# MONITORING.
htop                             # processes, CPU, memory
df -h                            # disk space: check before a big write
nvidia-smi                       # GPU utilisation

# RESOURCE LIMITS and timing.
/usr/bin/time -v python script.py 2>&1 | grep "Maximum resident"
timeout 300 python maybe_hangs.py

# A SCRIPT THAT FAILS LOUDLY. These three options belong in every script.
cat > run.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
#  -e  exit on any command failure
#  -u  error on an undefined variable
#  -o pipefail  a pipeline fails if ANY stage fails, not just the last

python -m src.ingest  --date "${1:?usage: run.sh YYYY-MM-DD}"
python -m src.features --date "$1"
python -m src.train    --date "$1"
EOF
chmod +x run.sh
# Without pipefail, `python broken.py | tee log` reports success because
# tee succeeded. That is how a broken pipeline passes CI.

Key Takeaways

• Shell tools compose through pipes and stream data, so they inspect files far larger than memory with immediate output.
• The sort-uniq-sort pipeline gives value counts on any column of any size file in seconds.
• find with xargs applies an operation across many files and can parallelize it with a single flag.
• duckdb runs SQL directly over CSV and Parquet on disk, often the fastest path to an answer.
• Every shell script should start with set -euo pipefail, or failures inside pipelines are reported as success.

🧪 Practice

1. Without loading it into Python, report the row count, distinct values of one column, and their frequencies for a large CSV.
2. Write a script with set -euo pipefail that runs three stages and stops correctly when the middle one fails.
3. Interview: A nightly shell pipeline reports success while producing an empty output file. What shell setting would have caught it? (Hint: which command's exit status does a pipeline report by default?)

Project Structure and Packaging

A script that only its author can run is a liability. Project structure is what turns exploratory work into something a colleague can clone, install, and execute, and the conventions are stable enough to adopt without deliberation.

The core principle is separation by role. Code that transforms data lives in an importable package. Notebooks tell a story and call that code. Configuration lives in files, not in code. Data lives outside version control. Outputs are regenerable and never edited by hand.

The structure that follows makes the data flow explicit: raw data is immutable and never modified, interim data is a working stage, and processed data is model-ready. Writing over raw data is the mistake this layout exists to prevent, because it makes the pipeline unrepeatable.

Making the project an installable package is the step people skip, and it is what fixes import paths. With pip install -e ., your code is importable from anywhere, so from src.features import build works identically in a notebook, a test, and a scheduled job. The sys.path manipulation at the top of notebooks disappears.

Configuration deserves its own file. Hard-coded paths and thresholds scattered through modules make the pipeline impossible to run against a different date range or environment without editing source.

churn-analysis/
├── README.md               what it does, how to run it, who owns it
├── pyproject.toml          dependencies, build config, tool settings
├── requirements.lock.txt   exact pinned versions for reproduction
├── Makefile                the documented entry points
├── .gitignore
├── .pre-commit-config.yaml
├── .env.example            names of required variables, never values
│
├── config/
│   ├── config.yaml         paths, thresholds, feature lists
│   └── logging.yaml
│
├── data/                   ALL git-ignored
│   ├── raw/                immutable: never written to after arrival
│   ├── interim/            intermediate stages
│   ├── processed/          model-ready
│   └── external/           third-party reference data
│
├── notebooks/
│   ├── 01-eda.ipynb        numbered so the reading order is obvious
│   └── 02-model-eval.ipynb
│
├── src/churn/              the importable package
│   ├── __init__.py
│   ├── config.py           load and validate configuration
│   ├── data/
│   │   ├── load.py         read from source systems
│   │   └── validate.py     schema and quality checks
│   ├── features/
│   │   └── build.py        feature engineering
│   ├── models/
│   │   ├── train.py
│   │   └── predict.py
│   └── viz/plots.py
│
├── tests/
│   ├── conftest.py         shared fixtures
│   ├── test_features.py
│   └── test_validate.py
│
├── scripts/                thin CLI wrappers over src
│   └── run_pipeline.py
│
└── outputs/                git-ignored: figures, reports, models

# pyproject.toml -- dependencies, packaging, and tool config in one file.
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "churn-analysis"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = [
    "pandas>=2.0",
    "scikit-learn>=1.3",
    "pyarrow>=14.0",
    "pydantic>=2.0",
    "pyyaml>=6.0",
]

[project.optional-dependencies]
dev = ["pytest>=8.0", "pytest-cov", "ruff", "mypy", "pre-commit"]

[project.scripts]
churn-train = "churn.models.train:main"    # a real command after install

[tool.hatch.build.targets.wheel]
packages = ["src/churn"]

[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-v --cov=churn --cov-report=term-missing"

# src/churn/config.py -- typed, validated configuration.
from pathlib import Path
import yaml
from pydantic import BaseModel, Field

class Paths(BaseModel):
    raw: Path
    processed: Path
    outputs: Path

class TrainConfig(BaseModel):
    target: str
    test_size: float = Field(0.2, gt=0, lt=1)     # validated at load
    random_seed: int = 42
    features: list[str]

class Config(BaseModel):
    paths: Paths
    train: TrainConfig

def load_config(path: str | Path = "config/config.yaml") -> Config:
    with open(path) as fh:
        return Config(**yaml.safe_load(fh))       # raises on a bad config
# Validating at load means a typo in a YAML file fails immediately with a
# clear message, rather than three stages later as a confusing TypeError.

# INSTALL THE PROJECT so imports work everywhere.
pip install -e ".[dev]"

# Now this works in a notebook, a test, and a cron job alike:
#   from churn.features.build import make_features
# No sys.path.append, no relative-import errors, no cwd dependence.

# SECRETS from the environment, never from source.
# .env (git-ignored):        DB_PASSWORD=...
# .env.example (committed):  DB_PASSWORD=

# Makefile -- the documented, discoverable entry points.
.PHONY: help setup lint test pipeline clean

help:            ## show this help
	@grep -E '^[a-z-]+:.*##' $(MAKEFILE_LIST) | sed 's/:.*##/ -/'

setup:           ## create the environment and install
	python -m venv .venv && . .venv/bin/activate && pip install -e ".[dev]"
	pre-commit install

lint:            ## format and check
	ruff format src tests && ruff check --fix src tests && mypy src

test:            ## run the test suite with coverage
	pytest

pipeline:        ## run the full pipeline end to end
	python -m churn.data.load
	python -m churn.features.build
	python -m churn.models.train

clean:           ## remove generated artifacts
	rm -rf data/interim/* outputs/* .pytest_cache .coverage

Key Takeaways

• Separate roles: importable code, narrative notebooks, file-based configuration, git-ignored data, and regenerable outputs.
• Treat raw data as immutable and write derived stages to separate directories, or the pipeline cannot be repeated.
• Install the project with pip install -e . so imports resolve identically in notebooks, tests, and scheduled jobs.
• Validate configuration at load with a schema, so a typo fails immediately with a clear message.
• A Makefile documents the entry points and makes them discoverable without reading the source.

🧪 Practice

1. Restructure a flat script-and-notebook folder into the layout above, install it editable, and confirm imports work from both a test and a notebook.
2. Move every hard-coded path and threshold into a YAML file loaded through a validated schema, and show that a bad value fails at load.
3. Interview: Why is pip install -e . preferable to appending to sys.path at the top of each notebook? (Hint: think about tests, cron jobs, and the current working directory.)

Testing Data Code

Data code fails differently from application code. The function is often correct while the data violates an assumption the function never stated. Testing therefore has two halves: testing the logic, and validating the data.

Unit tests cover transformation logic with small, hand-built inputs. The discipline that makes them useful is constructing the input by hand so the expected output is obvious. A test against a sample of real data tests nothing, because nobody can say what the right answer is.

Data validation checks assumptions about data at runtime, in the pipeline, every time it runs. Schema, ranges, uniqueness, null rates, referential integrity. These are not tests that run in CI; they are assertions that run in production, because the data changes after your tests pass.

The most valuable cases to test are the ones real data produces: empty inputs, a single row, all values identical, all values null, duplicate keys, and unexpected categories. Each has caused a production incident somewhere, and each takes three lines to test.

Property-based testing generates many inputs and checks invariants rather than specific outputs. For numerical code this finds edge cases no one would write by hand, such as the empty group or the zero denominator.

For models, test the pipeline rather than the accuracy. Accuracy varies with data; what must hold is that training is deterministic given a seed, that the model saves and loads identically, and that no target column leaks into the features.

# tests/test_features.py
import pandas as pd
import numpy as np
import pytest
from churn.features.build import add_rolling_mean, compute_margin

# FIXTURES: small, hand-built, obviously correct.
@pytest.fixture
def simple_frame():
    return pd.DataFrame({
        "store": ["a", "a", "a", "b", "b"],
        "date": pd.to_datetime(["2024-01-01", "2024-01-02", "2024-01-03",
                                "2024-01-01", "2024-01-02"]),
        "sales": [10.0, 20.0, 30.0, 100.0, 200.0],
    })

def test_rolling_mean_values(simple_frame):
    out = add_rolling_mean(simple_frame, window=2)
    # Hand-computed: store a row 2 is mean(10, 20) = 15.
    assert out.loc[1, "sales_ma2"] == pytest.approx(15.0)

def test_rolling_mean_does_not_cross_groups(simple_frame):
    out = add_rolling_mean(simple_frame, window=2)
    # Store b's first row must NOT include store a's last value.
    first_b = out[out["store"] == "b"].iloc[0]
    assert pd.isna(first_b["sales_ma2"]) or first_b["sales_ma2"] == 100.0

def test_rolling_mean_uses_no_future_data(simple_frame):
    out = add_rolling_mean(simple_frame, window=2)
    # The last value must not change when later rows are removed.
    truncated = add_rolling_mean(simple_frame.iloc[:2], window=2)
    assert out.loc[1, "sales_ma2"] == truncated.loc[1, "sales_ma2"]
# This is the leakage test, and it is the most valuable one in the file.

# THE EDGE CASES THAT ACTUALLY OCCUR.
def test_empty_input():
    empty = pd.DataFrame({"store": [], "date": [], "sales": []})
    out = add_rolling_mean(empty, window=2)
    assert len(out) == 0                       # must not raise

def test_single_row():
    one = pd.DataFrame({"store": ["a"], "date": [pd.Timestamp("2024-01-01")],
                        "sales": [5.0]})
    assert len(add_rolling_mean(one, window=7)) == 1

def test_all_null_column():
    d = pd.DataFrame({"revenue": [np.nan] * 3, "cost": [1.0, 2.0, 3.0]})
    result = compute_margin(d)
    assert result.isna().all()                 # defined behaviour, not a crash

def test_zero_denominator():
    d = pd.DataFrame({"revenue": [0.0, 100.0], "cost": [10.0, 40.0]})
    result = compute_margin(d)
    assert np.isnan(result.iloc[0]) or np.isinf(result.iloc[0])

@pytest.mark.parametrize("window,expected_nulls", [(2, 1), (3, 2), (7, 3)])
def test_warmup_nulls(simple_frame, window, expected_nulls):
    a_only = simple_frame[simple_frame["store"] == "a"]
    out = add_rolling_mean(a_only, window=window)
    assert out["sales_ma2" if window == 2 else f"sales_ma{window}"].isna().sum() \
        == min(expected_nulls, len(a_only))

# PROPERTY-BASED TESTING: invariants over generated inputs.
from hypothesis import given, strategies as st
import hypothesis.extra.pandas as hpd

@given(hpd.series(dtype=float, elements=st.floats(-1e6, 1e6,
                                                  allow_nan=False)))
def test_normalization_bounds(s):
    if len(s) == 0 or s.std() == 0:
        return
    z = (s - s.mean()) / s.std()
    assert abs(z.mean()) < 1e-6                # invariant, not a fixed value
    assert abs(z.std(ddof=0) - 1) < 1e-6

# src/churn/data/validate.py -- runtime validation, not a unit test.
import pandas as pd

def validate_orders(df: pd.DataFrame) -> pd.DataFrame:
    """Assert every assumption the pipeline relies on. Fail loudly."""
    required = {"order_id", "customer_id", "amount", "order_date"}
    missing = required - set(df.columns)
    if missing:
        raise ValueError(f"missing columns: {sorted(missing)}")

    if df["order_id"].duplicated().any():
        n = int(df["order_id"].duplicated().sum())
        raise ValueError(f"{n} duplicate order_id values")

    if (df["amount"] < 0).any():
        raise ValueError(f"{int((df['amount'] < 0).sum())} negative amounts")

    null_rate = df["customer_id"].isna().mean()
    if null_rate > 0.01:
        raise ValueError(f"customer_id null rate {null_rate:.1%} exceeds 1%")

    future = df["order_date"] > pd.Timestamp.now()
    if future.any():
        raise ValueError(f"{int(future.sum())} orders dated in the future")

    return df

# Declarative alternatives express the same rules as a schema:
#   pandera:          a DataFrameSchema with typed, checked columns
#   Great Expectations: a suite with data docs and profiling
# Both are worth adopting once the checks outgrow a function.

# tests/test_model_pipeline.py -- test the pipeline, not the accuracy.
def test_training_is_deterministic(training_data):
    m1 = train_model(training_data, seed=42)
    m2 = train_model(training_data, seed=42)
    np.testing.assert_array_equal(m1.predict(training_data),
                                  m2.predict(training_data))

def test_no_target_leakage(training_data, feature_columns):
    assert "churned" not in feature_columns
    # A feature perfectly correlated with the target is the usual tell.
    for col in feature_columns:
        if pd.api.types.is_numeric_dtype(training_data[col]):
            r = training_data[col].corr(training_data["churned"])
            assert abs(r) < 0.99, f"{col} correlates {r:.3f} with the target"

def test_model_roundtrip(tmp_path, training_data):
    model = train_model(training_data, seed=0)
    before = model.predict(training_data)
    path = tmp_path / "model.pkl"
    save_model(model, path)
    after = load_model(path).predict(training_data)
    np.testing.assert_array_equal(before, after)

Layer	Question	Runs
Unit test	is the logic right on known input?	CI
Property test	do invariants hold on any input?	CI
Data validation	does today's data meet assumptions?	every pipeline run
Pipeline test	is training reproducible and leak-free?	CI
Monitoring	has the live distribution drifted?	continuously

Key Takeaways

• Unit tests use small hand-built inputs so the expected output is unarguable; testing against real data samples tests nothing.
• Data validation runs in the pipeline rather than in CI, because the data changes after your tests pass.
• Test the edge cases real data produces: empty, single row, all null, duplicate keys, and unexpected categories.
• Property-based tests check invariants over generated inputs and find cases nobody would write by hand.
• For models, test determinism, serialization round-trip, and absence of target leakage rather than an accuracy threshold.

🧪 Practice

1. Write unit tests for a group-wise rolling feature covering empty input, a single row, and the guarantee that no value crosses a group boundary.
2. Write a validation function asserting schema, key uniqueness, a value range, and a maximum null rate, then run it against deliberately corrupted data.
3. Interview: Why is asserting a minimum model accuracy a poor CI test? (Hint: what makes it fail, and is that always a code problem?)

Reproducible Environments with Containers

A lockfile pins Python packages. It does not pin the operating system, the system libraries those packages link against, the locale, or the presence of a compiler. Containers pin all of it.

A container image is a filesystem snapshot plus metadata describing how to run it. A container is a running instance. Unlike a virtual machine, containers share the host kernel, so they start in under a second and add almost no overhead, which is what makes them practical for development rather than only deployment.

The mental model is a shipping container. The contents are packed once and the same box moves by truck, ship, and train without repacking. Your image runs identically on a laptop, in CI, and in production, because the environment travels with the code.

Images build in layers, each instruction creating one, and layers are cached. Ordering matters enormously: copy the dependency file and install before copying your source, and a code change reuses the cached dependency layer instead of reinstalling everything. This single ordering choice is the difference between a five-second and a five-minute rebuild.

The rules that matter for data work: pin the base image by digest rather than a floating tag, never bake credentials into an image, mount data as a volume rather than copying it in, and run as a non-root user.

# Dockerfile -- a reproducible analysis environment.

# PIN THE BASE IMAGE. "python:3.11" is a moving target that changes
# underneath you; a digest is immutable.
FROM python:3.11-slim@sha256:2d9f3a1b0c8e7f6a5b4c3d2e1f0a9b8c7d6e5f4a3b2c1d0e9f8a7b6c5d4e3f2a

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PIP_NO_CACHE_DIR=1

# System libraries some data packages link against.
RUN apt-get update && apt-get install -y --no-install-recommends \
        build-essential \
        libgomp1 \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

# LAYER ORDER IS THE WHOLE OPTIMIZATION: dependencies before source.
COPY requirements.lock.txt .
RUN pip install --no-cache-dir -r requirements.lock.txt

# Source changes invalidate only the layers below this line.
COPY src/ ./src/
COPY pyproject.toml .
RUN pip install --no-deps -e .

# Run as a non-root user.
RUN useradd --create-home --uid 1000 analyst && chown -R analyst:analyst /app
USER analyst

# Data and outputs are MOUNTED, never copied into the image.
VOLUME ["/data", "/outputs"]

HEALTHCHECK --interval=30s CMD python -c "import churn" || exit 1
CMD ["python", "-m", "churn.models.train"]

# docker-compose.yml -- the whole local stack in one file.
services:
  analysis:
    build: .
    volumes:
      - ./data:/data:ro          # read-only: raw data cannot be overwritten
      - ./outputs:/outputs
      - ./src:/app/src           # live code for development
    environment:
      - DB_HOST=postgres
      - DB_PASSWORD=${DB_PASSWORD}   # from .env, never in the image
    depends_on:
      postgres:
        condition: service_healthy

  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_DB: analytics
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      retries: 5

  jupyter:
    build: .
    command: jupyter lab --ip=0.0.0.0 --no-browser --allow-root
    ports: ["8888:8888"]
    volumes:
      - ./notebooks:/app/notebooks
      - ./data:/data:ro

volumes:
  pgdata:

# BUILDING AND RUNNING.
docker build -t churn-analysis:0.1.0 .
docker run --rm -v "$(pwd)/data:/data:ro" -v "$(pwd)/outputs:/outputs" \
    churn-analysis:0.1.0

docker compose up -d
docker compose logs -f analysis
docker compose exec analysis bash      # a shell inside the container
docker compose down -v                 # -v also removes named volumes

# TAG WITH THE COMMIT so an image is traceable to its source.
docker build -t churn-analysis:$(git rev-parse --short HEAD) .

# INSPECT LAYER SIZES when an image is unexpectedly large.
docker history churn-analysis:0.1.0

# A MULTI-STAGE BUILD keeps compilers out of the final image.
cat > Dockerfile.slim <<'EOF'
FROM python:3.11 AS builder
WORKDIR /app
COPY requirements.lock.txt .
RUN pip install --user --no-cache-dir -r requirements.lock.txt

FROM python:3.11-slim
COPY --from=builder /root/.local /root/.local    # only the installed packages
ENV PATH=/root/.local/bin:$PATH
WORKDIR /app
COPY src/ ./src/
CMD ["python", "-m", "churn.models.train"]
EOF
# Often cuts the image from over a gigabyte to a few hundred megabytes.

# DEV CONTAINERS: VS Code opens the project INSIDE the container, so the
# editor, terminal, and debugger all use the pinned environment.
mkdir -p .devcontainer
cat > .devcontainer/devcontainer.json <<'EOF'
{
  "name": "churn-analysis",
  "dockerComposeFile": "../docker-compose.yml",
  "service": "analysis",
  "workspaceFolder": "/app",
  "customizations": {
    "vscode": {
      "extensions": ["ms-python.python", "charliermarsh.ruff"],
      "settings": { "python.defaultInterpreterPath": "/usr/local/bin/python" }
    }
  }
}
EOF

  WHAT EACH LAYER OF REPRODUCIBILITY PINS

  requirements.lock.txt   python packages                  weakest
  + pyproject.toml        the python version
  + Dockerfile            OS, system libraries, locale
  + image digest          the exact base image bytes
  + DVC / data hash       the exact input data
  + git commit            the exact code                   strongest

  A result is reproducible only to the weakest layer you pinned.
  "It works on my machine" is a statement about unpinned layers.

  LAYER CACHING: why order matters

  BAD                             GOOD
  COPY . .                        COPY requirements.lock.txt .
  RUN pip install -r req.txt      RUN pip install -r req.txt
                                  COPY . .

  any source edit invalidates     a source edit reuses the cached
  the install layer               install layer
  -> full reinstall every build   -> seconds

Key Takeaways

• A lockfile pins Python packages only; containers additionally pin the operating system, system libraries, and locale.
• Images build in cached layers, so copying dependencies and installing before copying source turns minutes of rebuild into seconds.
• Pin the base image by digest, because a floating tag changes underneath you without warning.
• Mount data as a read-only volume rather than copying it into the image, and keep credentials in the environment.
• A result is reproducible only to the weakest layer you pinned, so pin code, data, packages, and the operating system together.

🧪 Practice

1. Write a Dockerfile for an existing project, then reorder the copy and install steps and measure the rebuild time after a one-line source change.
2. Build a compose stack with an analysis service and a database, and confirm the analysis container waits for the database health check.
3. Interview: Your pipeline gives different numerical results in production despite an identical lockfile. Name two environment-level causes a container would eliminate. (Hint: what do numerical libraries link against, and what can vary by locale?)

---

<a id="5-data-acquisition-and-storage"></a>