5. Caching and Content Delivery

Caching is the highest-leverage performance technique in system design: it turns expensive work into a memory lookup and removes most traffic from your origin before it ever arrives. This chapter covers why caching works at all, the strategies for reading and writing through a cache, the distributed systems problems that appear once the cache spans machines, and the content delivery networks that push all of it to the edge of the network.

5.1 Caching Fundamentals

Before choosing a caching strategy, it is worth understanding why caches work, how to measure whether yours is working, and where in the stack a cache can sit.

Why Caching Works: Locality

A cache is a bet: that work you did recently will be asked for again, and that storing the answer is cheaper than recomputing it. If requests were uniformly random across a billion items, that bet would lose — a small cache would almost never hold what you need. Caching works because real access patterns are overwhelmingly non-uniform, and that property has a name: locality.

Two kinds matter. Temporal locality means an item accessed now is likely to be accessed again soon: a trending article, a logged-in user's profile, a configuration value read on every request. Spatial locality means items near one another get accessed together: the next row in a table scan, the next byte in a file, the other fields of the same record.

The analogy is a desk. The papers you are working with sit on the desk (cache); the rest live in the filing cabinet across the room (origin). The desk is tiny and the cabinet is huge, but you spend most of your day reaching only for the desk — because what you needed a minute ago is usually what you need now.

 REAL ACCESS DISTRIBUTIONS ARE ZIPFIAN, NOT UNIFORM

  requests
     |*
     |*
     |**
     |***
     |*****
     |*********
     |*******************
     +-------------------------------------> items ranked by popularity
      ^^^^^^^^                    ^^^^^^^^^^^^^^^^^^^^
      top 20% of items            the long tail: 80% of items,
      = ~80% of requests          ~20% of requests

  Consequence: a cache holding only the top 20% of items serves ~80% of
  traffic. You do not need to cache much to remove most of the load.

# Why the shape of the distribution decides whether caching is worth it.
def simulate(distribution, cache_size, requests):
    cache, hits = set(), 0
    for item in distribution(requests):
        if item in cache:
            hits += 1
        else:
            if len(cache) >= cache_size:
                cache.pop()                    # crude eviction; see 5.2
            cache.add(item)
    return hits / requests

# Uniform over 1M items, cache of 10k -> hit rate ~1%. Caching is pointless.
# Zipfian over 1M items, cache of 10k -> hit rate ~85%. Caching is transformative.
#
# The practical test before building a cache: plot request counts per key from
# your access logs. If the curve is flat, a cache will not help you.

Locality is also why caching appears at every level of every computer system, with the same structure repeated at different scales:

Level	Cache	Backing store	Speed ratio
CPU	L1/L2/L3 cache	Main memory	~100x
Operating system	Page cache	Disk	~1,000x
Database	Buffer pool	Data files	~1,000x
Application	In-process / Redis	Database	~100x
Network	CDN edge	Origin server	~10x
Browser	HTTP cache	Network	~1,000x

The corollary is the part people forget: a cache is only useful if the working set fits. If the actively-requested data is 500 GB and your cache is 10 GB, you will thrash — evicting entries just before they are needed again — and pay cache overhead for almost no benefit.

Key Takeaways

Caches work because access patterns are non-uniform, not because memory is fast.

Temporal locality (recently used again soon) is what most application caches exploit.

Zipfian traffic means caching ~20% of items typically serves ~80% of requests.

Check the distribution first; a flat access curve or a working set larger than the cache makes caching useless.

🧪 Practice

Take an access log you have and count requests per key. Plot or rank them — is the distribution Zipfian or flat?

Explain why a cache in front of a randomly-accessed 1 TB dataset with 8 GB of RAM will perform badly.

Interview: When is adding a cache the wrong solution to a slow endpoint? (Hint: think about write-heavy paths, uniform access, and what a cache does to correctness that an index does not.)

Cache Hit Ratio and Effectiveness

"We added a cache" is not a result. The only number that matters is the hit ratio — the fraction of requests served from the cache — and its relationship to latency and origin load is non-linear in a way that changes how you prioritize work.

The hit ratio is simple to define: hits / (hits + misses). Its consequences are less obvious, because the same one-percentage-point improvement is worth wildly different amounts depending on where you start.

CACHE_MS, ORIGIN_MS = 1, 50
TOTAL_RPS = 10_000

def effective_latency(hit_ratio):
    # A miss usually costs cache lookup PLUS origin, but the cache lookup is
    # negligible here, so this is the standard simplification.
    return hit_ratio * CACHE_MS + (1 - hit_ratio) * ORIGIN_MS

def origin_rps(hit_ratio):
    return TOTAL_RPS * (1 - hit_ratio)

for h in (0.0, 0.50, 0.90, 0.95, 0.99, 0.999):
    print(f"{h:6.1%}  latency {effective_latency(h):6.2f} ms   origin {origin_rps(h):8.0f} RPS")
#   0.0%  latency  50.00 ms   origin    10000 RPS
#  50.0%  latency  25.50 ms   origin     5000 RPS
#  90.0%  latency   5.90 ms   origin     1000 RPS
#  95.0%  latency   3.45 ms   origin      500 RPS
#  99.0%  latency   1.49 ms   origin      100 RPS
#  99.9%  latency   1.05 ms   origin       10 RPS

Read that table twice. Going from 0% to 90% cuts origin load by 10x. Going from 90% to 99% cuts it by another 10x — the same nine percentage points that barely move average latency (5.9 ms to 1.5 ms) remove 900 requests per second from your database. Origin load scales with the miss rate, so improvements at the top of the range are where the capacity savings live.

 THE MISS RATE IS WHAT YOU ARE ACTUALLY OPTIMIZING

  hit ratio   miss rate   origin load (relative)
     90%         10%            1.0x
     95%          5%            0.5x     <- half the origin traffic
     99%          1%            0.1x     <- one tenth
   99.9%        0.1%            0.01x    <- one hundredth

  Talking about "hit ratio going from 99% to 99.5%" sounds trivial.
  Saying "origin load halved" does not.

But hit ratio alone can mislead in two ways, and both show up in real systems:

Trap	What happens
Measuring the wrong keys	A 99% hit ratio dominated by one trivially cacheable key hides a 20% ratio on everything else
Ignoring miss cost	If a miss costs 2 seconds, a 95% hit ratio still means a terrible p99
Counting only reads	Invalidation churn from writes can make a cache do net harm
Cold-start behaviour	A restart drops to 0% instantly — can the origin survive that?

The metrics worth exporting, therefore, are hit ratio segmented by key prefix, miss latency as its own distribution, eviction rate (rising evictions mean the working set no longer fits), and cache memory utilization.

Key Takeaways

Hit ratio is the primary cache metric, but origin load scales with the miss rate.

Improvements near the top (95% to 99%) deliver the largest absolute reduction in backend traffic.

Segment hit ratio by key type; one hot key can mask a badly-performing cache.

Track eviction rate and miss latency too, and know what happens on a cold start.

🧪 Practice

At 20,000 RPS with a 92% hit ratio, compute origin RPS. What hit ratio would halve it?

A cache has a 97% hit ratio, 1 ms hits and 400 ms misses. Compute the mean and estimate the p99. Which is the user's experience?

Interview: Your cache reports a 99% hit ratio but the database is still overloaded. Give three explanations. (Hint: consider which requests are counted, which keys dominate, and what writes are doing.)

Cache Layers Across the Stack

A single request passes through many places where an answer could already be waiting. Each is a cache, each has different scope and control, and designing deliberately across them is what turns a 200 ms request into a 5 ms one — or into a confusing bug where users see stale data and nobody can say which layer served it.

The organizing principle is that the earlier a request is answered, the cheaper it is — and the less control you have over invalidating it. A browser cache costs you nothing and you cannot clear it; a database buffer pool you control completely but it saves the least.

 THE FULL CACHE CHAIN FOR ONE REQUEST

  [browser memory/disk cache]     0 ms    you cannot invalidate it
        | miss
  [ISP / corporate proxy]         ~5 ms   you cannot even see it
        | miss
  [CDN edge PoP]                  ~10 ms  purgeable via API, seconds to minutes
        | miss
  [reverse proxy / gateway]       ~1 ms   you control it fully
        | miss
  [application in-process cache]  ~0.1 ms per-instance, inconsistent across fleet
        | miss
  [distributed cache: Redis]      ~1 ms   shared, invalidatable, one more hop
        | miss
  [database buffer pool]          ~0.1 ms managed by the database
        | miss
  [disk]                          ~1-10 ms the actual work

  Each layer that answers removes ALL load from every layer below it.

# A layered read path. Note that each layer has a different TTL — shorter as
# you go outward, because outer layers are harder to invalidate.
def get_product(product_id, local, redis, db):
    key = f"product:{product_id}"

    # L1: in-process. Nanoseconds, but each instance has its own copy, so a
    # short TTL bounds how long the fleet can disagree with itself.
    if (v := local.get(key)) is not None:
        metrics.increment("cache.hit", tags={"layer": "local"})
        return v

    # L2: shared. One network hop, but consistent across all instances and
    # invalidatable from anywhere.
    if (v := redis.get(key)) is not None:
        local.set(key, v, ttl=5)            # populate L1 on the way back
        metrics.increment("cache.hit", tags={"layer": "redis"})
        return v

    # L3: the origin.
    v = db.query_product(product_id)
    redis.setex(key, 300, v)                # 5 minutes shared
    local.set(key, v, ttl=5)                # 5 seconds per-instance
    metrics.increment("cache.miss")
    return v

Layer	Scope	Latency	Invalidation control	Best for
Browser	One user	0 ms	None (only via URL)	Immutable, versioned assets
CDN edge	One region	~10 ms	Purge API	Static and public content
Reverse proxy	One datacenter	~1 ms	Full	Rendered pages, API responses
In-process (L1)	One instance	~0.1 ms	Full but per-instance	Tiny hot values, config
Distributed (L2)	Whole fleet	~1 ms	Full and consistent	Sessions, query results
Database buffer	One database	~0.1 ms	Automatic	Pages the engine chooses

The multi-layer design brings one specific hazard: the shortest TTL you can invalidate is the longest one in the chain. If a browser holds a response for an hour, purging the CDN changes nothing for that user. This is why the standard pattern for assets is content-hashed URLs with immutable, year-long caching — you never invalidate, you change the URL.

Key Takeaways

Every layer that answers removes all load from every layer beneath it.

Control decreases and cheapness increases as you move outward toward the client.

Use shorter TTLs for layers you cannot invalidate precisely, longer for ones you can.

You cannot recall what a browser has cached — version the URL instead of invalidating.

🧪 Practice

Trace a request for a product page through every cache layer and give each a TTL with justification.

Explain why an in-process cache with a 5-minute TTL causes users to see different data on refresh across a 20-instance fleet.

Interview: You deployed a fix but some users still see the old behaviour after an hour. Walk through the diagnosis. (Hint: enumerate every layer that could hold a copy, starting with the one you control least.)

Client, CDN, Application, and Database Caches

The previous topic laid out the chain; this one is about what each layer is actually for, because putting the right data in the right layer is most of the design work. The decision rests on three properties of the data: how public it is, how often it changes, and how bad staleness would be.

Client caches live in the user's browser or app. They are free, infinitely scalable, and completely outside your control once served. That makes them perfect for immutable content and dangerous for anything else.

CDN caches hold public, shareable content near users. Their unit of caching is the URL, so anything varying by user is either uncacheable there or requires careful Vary handling (Chapter 2.4).

Application caches hold computed results — an assembled dashboard, a rendered template, a query result — where the expense is CPU or database work rather than distance.

Database caches are largely automatic: the buffer pool holds hot pages, and the query plan cache holds parsed plans. You influence them by sizing memory and by keeping the hot dataset small, not by writing code.

 WHAT BELONGS WHERE

  data                             layer            TTL          why
  ------------------------------------------------------------------------
  app.a3f9c2.js (hashed filename)  browser + CDN    1 year       immutable
  product image                    CDN              30 days      public, static
  product catalogue page           CDN              5 min        public, changes
  logged-in user's dashboard       application      30 s         private, computed
  user session                     distributed      30 min       private, shared
                                                                 across instances
  feature flags                    in-process       10 s         tiny, hot, read
                                                                 on every request
  account balance                  NOT CACHED       -            staleness is
                                                                 unacceptable

### Client + CDN: immutable asset. Never invalidated; the hash IS the version.
GET /static/app.a3f9c2.js
Cache-Control: public, max-age=31536000, immutable

### CDN only, not the browser: shared cache holds it, clients revalidate.
GET /api/products
Cache-Control: public, max-age=0, s-maxage=300, stale-while-revalidate=60
#                             ^browser: always revalidate
#                                       ^shared cache: 5 minutes

### Private: may be stored by the browser, never by a shared cache.
GET /api/me/dashboard
Cache-Control: private, max-age=30

### Never cache anywhere.
GET /api/accounts/7/balance
Cache-Control: no-store

# Application-layer caching is where judgement is needed, because you choose
# the key — and the key defines correctness.
def dashboard(user, redis, db):
    # The key MUST include every input that changes the output. Forgetting the
    # tenant, the locale, or the permission set is how one user sees another's
    # data — the most serious caching bug there is.
    key = f"dash:v3:{user.tenant_id}:{user.id}:{user.locale}:{user.role}"
    #           ^ a version prefix lets you invalidate the whole namespace by
    #             bumping it, without deleting keys one by one
    if (cached := redis.get(key)) is not None:
        return cached
    result = build_dashboard(user, db)          # expensive: several queries
    redis.setex(key, 30, result)
    return result

Layer	Caches	Keyed by	Typical TTL	Main risk
Client	Assets, private responses	URL	Long / short	Cannot be recalled
CDN	Public responses	URL (+ `Vary`)	Minutes to days	Serving private data publicly
Application	Computed results	A key you design	Seconds to minutes	A key missing an input
Database	Pages and plans	Managed	Automatic	Working set exceeding memory

Two rules carry most of the weight. Include every input in the key — tenant, user, locale, role, feature flags, API version. And never cache what you cannot afford to serve stale: balances, permissions checks, inventory counts during checkout. When you are unsure, cache with a short TTL and measure whether the hit ratio justifies the risk at all.

Key Takeaways

Match data to layer by how public it is, how fast it changes, and how bad staleness would be.

Immutable, content-hashed assets get year-long client and CDN caching.

private versus public in Cache-Control is what keeps one user's data out of a shared cache.

An application cache key must include every input that affects the output.

🧪 Practice

Assign a layer, TTL, and Cache-Control header to: a company logo, a search results page, a user's notification count, a PDF invoice.

Write the cache key for a paginated, filtered, localized product list and justify each component.

Interview: A bug shows one customer another customer's dashboard. What caching mistake would produce that? (Hint: the bug is in one line, and it is about what is absent from it.)

5.2 Caching Strategies

A caching strategy answers two questions — who populates the cache on a read, and what happens to the cache on a write — and the four canonical combinations have very different consistency and durability properties.

Cache-Aside

Cache-aside (also called lazy loading) is the most widely used caching pattern because it requires nothing of the cache except get and set. The application owns the logic: check the cache, and on a miss, fetch from the database and populate the cache itself.

Its defining property is that the cache is never in the write path of the database. If the cache is unavailable, every read simply becomes a database read — degraded but working. That resilience is why cache-aside is the default choice.

 CACHE-ASIDE READ                      CACHE-ASIDE WRITE

  app --get--> [cache]                  app --write--> [database]
       hit? return                       app --delete-> [cache]
       miss:                                            ^
  app --query--> [database]                             |
  app --set----> [cache]                    invalidate, do NOT update:
  return                                    a concurrent write could otherwise
                                            leave a stale value behind

def get_user(user_id, cache, db):
    key = f"user:{user_id}"

    if (cached := cache.get(key)) is not None:
        return cached                                   # hit

    user = db.query_user(user_id)                       # miss: load from origin
    if user is not None:
        cache.setex(key, 300, user)                     # populate, with a TTL
    return user

def update_user(user_id, data, cache, db):
    db.update_user(user_id, data)                       # 1. write the origin first
    cache.delete(f"user:{user_id}")                     # 2. then INVALIDATE
    # Order matters. Deleting first leaves a window where a concurrent read
    # repopulates the cache with the OLD value just before the write lands.
    #
    # Delete rather than update: two concurrent writers that both "update" the
    # cache can apply their writes to the database in one order and to the
    # cache in the other, leaving the cache permanently wrong. A delete forces
    # the next reader to reload the truth.

Cache-aside has one well-known race that no ordering fully removes:

 THE STALE-REPOPULATION RACE

  reader                          writer
  ------                          ------
  cache.get(k) -> MISS
  db.read(k)   -> v1
                                  db.write(k, v2)
                                  cache.delete(k)      (deletes nothing)
  cache.set(k, v1)                                     <- stale value, for the
                                                          full TTL

  Mitigations, in increasing order of cost:
    - a short TTL, so the damage is bounded
    - set only if absent (SET NX), so a fresher value is not overwritten
    - delayed double-delete: invalidate again a few hundred ms after the write
    - versioned keys, so a write changes the key rather than the value

Property	Cache-aside
Who populates	The application, on a miss
Cache failure	Degrades to direct database reads — the system survives
Data in cache	Only what has been requested (no wasted memory)
First request cost	Always a miss (cold start is slow)
Consistency	Eventual; a narrow stale-repopulation race exists
Code	Duplicated at every call site unless wrapped

That last row is its main drawback: the get/miss/load/set logic appears everywhere, and one call site that forgets to invalidate on write becomes a lasting correctness bug. Wrapping it in a repository or decorator is the usual remedy — which is essentially reinventing read-through, the next topic.

Key Takeaways

The application controls the cache: check, load on miss, populate.

The cache is not in the database write path, so cache failure degrades rather than breaks.

On writes, update the database first, then delete the key — do not update it.

A stale-repopulation race remains; bound it with short TTLs or versioned keys.

🧪 Practice

Implement cache-aside for a product lookup, including the invalidation on update and delete.

Draw the interleaving that leaves a stale value in the cache and show how a delayed double-delete narrows it.

Interview: Why delete the cache entry on write instead of updating it with the new value? (Hint: consider two writers racing, and which order each of the two systems ends up applying.)

Read-Through

Cache-aside scatters cache logic across the codebase. Read-through moves it behind the cache itself: the application asks the cache for a key, and if the cache does not have it, the cache fetches from the origin, stores the result, and returns it. The caller sees one uniform interface and never writes a if miss: load branch.

The gain is not just tidiness. Centralizing the load path means you can implement miss coalescing, consistent TTL policy, and metrics once, correctly, instead of relying on every developer to repeat the pattern.

 CACHE-ASIDE (app orchestrates)     READ-THROUGH (cache orchestrates)

  app -> cache   miss                 app -> cache   miss
  app -> db                                  cache -> db
  app -> cache   set                         cache stores
  app returns                         app <- cache   value

  3 app-side steps, repeated          1 app-side step; the loader is
  at every call site                  configured once

class ReadThroughCache:
    def __init__(self, backend, loader, ttl=300):
        self._backend = backend          # Redis, or an in-process store
        self._loader  = loader           # how to fetch on a miss
        self._ttl     = ttl
        self._locks   = {}               # per-key locks for miss coalescing

    def get(self, key):
        if (v := self._backend.get(key)) is not None:
            return v

        # Miss coalescing: without this, 500 concurrent misses on the same key
        # issue 500 identical database queries (see Cache Stampede in 5.3).
        lock = self._locks.setdefault(key, threading.Lock())
        with lock:
            if (v := self._backend.get(key)) is not None:
                return v                            # another thread filled it
            v = self._loader(key)                   # exactly one origin call
            if v is not None:
                self._backend.setex(key, self._ttl, v)
            return v

# One configuration, used everywhere. No call site can forget the pattern.
users = ReadThroughCache(redis, loader=lambda k: db.query_user(int(k.split(":")[1])),
                         ttl=300)
user = users.get("user:7")

Many caching libraries and products implement read-through natively — Caffeine and Guava in Java, and cache products configured with a "cache loader". The pattern is also what a CDN does: on a miss it fetches from origin, stores, and serves, without the client knowing.

Aspect	Cache-aside	Read-through
Load logic lives in	Every call site	One configured loader
Miss coalescing	Must be added per site	Natural to implement once
Flexibility	Any custom logic per call	Uniform; awkward for special cases
Cache failure	Degrades to direct reads	Cache is on the read path — failure needs an explicit fallback
Consistency on write	Application invalidates	Application still invalidates (read-through says nothing about writes)

Two clarifications that avoid common confusion. Read-through addresses reads only — you still need a write strategy, which is the next three topics. And because the cache is now on the read path, an unavailable cache breaks reads unless the loader explicitly falls back to the origin — so the resilience advantage of cache-aside has to be rebuilt deliberately.

Key Takeaways

Read-through hides the miss path inside the cache, giving one uniform read interface.

Centralizing the loader is what makes miss coalescing and consistent TTLs practical.

It governs reads only; a write strategy is still required.

The cache sits on the read path, so add an explicit origin fallback for cache outages.

🧪 Practice

Convert a cache-aside implementation into a read-through wrapper with a configurable loader and TTL.

Add miss coalescing and demonstrate that 100 concurrent misses cause one origin call.

Interview: What does read-through give you that cache-aside does not, and what does it cost? (Hint: think about what happens when the cache itself is down, and who is responsible for noticing.)

Write-Through

Read strategies decide what happens on a miss; write strategies decide what happens to the cache when data changes. Write-through takes the strictest approach: every write goes to the cache and the origin, synchronously, before the write is acknowledged.

The benefit is that the cache is never stale. A reader that hits the cache immediately after a write sees the new value, so read-your-writes holds through the cache — the property that makes stale-read bugs disappear.

 WRITE-THROUGH

  app --write--> [cache] --write--> [database]
                                        |
                 <---------- ack -------+
       <-- ack --

  Both are updated before the client is told "done".
  Read after write: guaranteed fresh from the cache.
  Cost: every write pays BOTH latencies, and a cache failure fails the write.

def write_through(key, value, cache, db):
    # Order: origin first, so a cache failure cannot leave the cache holding a
    # value the database never accepted.
    db.write(key, value)                      # durable, authoritative
    try:
        cache.setex(key, 300, value)          # keep the cache exactly in step
    except CacheUnavailable:
        cache_delete_best_effort(key)         # never leave a KNOWN-stale entry
        raise                                 # or degrade, but decide explicitly

# The subtlety people miss: strict write-through means writes populate the
# cache even for data nobody reads. On a write-heavy, read-light table that
# fills the cache with cold entries and evicts the hot ones — actively
# LOWERING the hit ratio. Write-through pays off only when written data is
# read soon afterwards.

Because the cache and the origin are updated under two separate operations, write-through is not atomic. A crash between the two leaves them inconsistent, which is why the origin is written first and the cache entry is deleted rather than left wrong on failure.

Property	Write-through
Cache freshness	Always current — no stale reads through the cache
Write latency	Origin latency plus cache latency
Durability	Full — the origin is written before acknowledgement
Cache utilization	Poor if written data is rarely read
Failure of the cache	Write fails, or you accept an invalidation instead
Good for	Read-after-write workloads: profiles, settings, carts

Write-through pairs naturally with read-through: together they give a cache that is always populated and always current, at the cost of write latency and wasted memory on write-only data. When writes vastly outnumber reads, cache-aside with invalidation is the better pairing — it only caches what someone actually asked for.

Key Takeaways

Write-through updates cache and origin synchronously, so the cache is never stale.

It guarantees read-your-writes through the cache, eliminating a whole class of bugs.

Every write pays both latencies, and written-but-unread data pollutes the cache.

The two writes are not atomic — write the origin first and delete the key on cache failure.

🧪 Practice

Implement write-through for a user settings object and handle cache failure explicitly.

Explain when write-through reduces the hit ratio, with a concrete workload.

Interview: Write-through guarantees the cache matches the database. Under what failure does that stop being true? (Hint: there are two operations and no transaction spanning them.)

Write-Behind

Write-through makes every write pay the origin's latency. Write-behind (also called write-back) removes that: the write goes to the cache, is acknowledged immediately, and is flushed to the origin asynchronously — usually batched with other writes.

This is the highest-throughput write strategy by a wide margin, because it turns many small random origin writes into few large batched ones (the sequential-I/O advantage from Chapter 4.1), and because the client never waits for the origin at all.

It is also the only caching strategy that can lose acknowledged data, which makes it a durability decision rather than merely a performance one.

 WRITE-BEHIND

  app --write--> [cache]  --ack immediately-->  app     (~0.5 ms)
                    |
                    | buffered, coalesced, flushed on a timer or batch size
                    v
                [database]                              (later: ~seconds)

  If the cache dies with unflushed writes, those writes are GONE — and the
  client was told they succeeded.

  Coalescing is the hidden win: 100 updates to the same key in one window
  become ONE database write.

class WriteBehindCache:
    def __init__(self, cache, db, flush_interval=1.0, max_batch=500):
        self._cache, self._db = cache, db
        self._dirty = {}                      # key -> latest value (coalescing)
        self._lock = threading.Lock()
        threading.Thread(target=self._flush_loop, daemon=True).start()

    def write(self, key, value):
        self._cache.set(key, value)
        with self._lock:
            self._dirty[key] = value          # later writes overwrite earlier
        return "ok"                           # acknowledged BEFORE durability

    def _flush_loop(self):
        while True:
            time.sleep(self._flush_interval)
            with self._lock:
                batch, self._dirty = self._dirty, {}
            if not batch:
                continue
            try:
                self._db.write_batch(batch)   # one round trip for many keys
            except Exception:
                with self._lock:
                    # Re-queue, but do not clobber newer values written since.
                    for k, v in batch.items():
                        self._dirty.setdefault(k, v)
                metrics.increment("write_behind.flush_failed", len(batch))

The pattern is everywhere in systems programming — a database buffer pool is write-behind onto data files, and an operating system's page cache is write-behind onto disk. Both are safe only because a write-ahead log makes the change durable first (Chapter 4.1), which is exactly the mitigation an application-level write-behind cache usually lacks.

Property	Write-behind
Write latency	Cache latency only — the fastest option
Throughput	Highest; batching and coalescing multiply it
Durability	Data loss window equal to the flush interval
Origin load	Dramatically reduced
Read-after-write	Correct through the cache; wrong if reading the origin
Complexity	High: flush failures, ordering, shutdown draining
Good for	Counters, metrics, view counts, telemetry, session activity timestamps

The suitability test is a single question: if the last N seconds of writes vanished, would anyone be harmed? For a page-view counter, no. For an order or a payment, catastrophically yes. Use write-behind only where the data is tolerant of loss, and always drain the buffer on graceful shutdown.

Key Takeaways

Write-behind acknowledges at the cache and flushes to the origin asynchronously in batches.

It gives the best write latency and throughput, and coalescing collapses repeated writes to one.

It can lose acknowledged writes — the loss window is the flush interval.

Restrict it to loss-tolerant data, and drain the buffer on shutdown.

🧪 Practice

Implement a write-behind counter cache with coalescing and compute the origin write reduction for 10,000 increments across 100 keys per second.

Describe exactly what is lost when the cache process is killed with a 1-second flush interval at 5,000 writes per second.

Interview: Which data in a social media app is safe for write-behind and which is not? (Hint: sort by what a user would notice, and by what an auditor would.)

Refresh-Ahead

All the strategies so far are reactive: the cache learns a value is stale only when someone asks for it and pays the miss. Refresh-ahead is proactive — the cache reloads entries before they expire, so a popular key is never missing when a user arrives.

The motivation is the shape of the latency distribution. With a reactive cache, one unlucky user per TTL period pays the full origin latency for every hot key. At scale that is exactly what shows up as an ugly p99 — and it is entirely predictable, so it is entirely preventable.

 REACTIVE TTL                          REFRESH-AHEAD

  |---- served from cache ----|X|       |---- served from cache --------|
                              ^                          |
                     TTL expires; the next               | at 80% of the TTL, a
                     request pays 200 ms                 | background refresh
                                                         | reloads the value
  Every TTL period, one user is                          v
  punished for the whole cohort.        No request ever sees a miss on a hot key.

def get_with_refresh_ahead(key, cache, loader, ttl=300, refresh_at=0.8):
    entry = cache.get_with_metadata(key)

    if entry is None:                                   # true miss: load inline
        value = loader(key)
        cache.setex(key, ttl, value)
        return value

    age_fraction = entry.age_seconds / ttl
    if age_fraction >= refresh_at and cache.try_lock(f"refresh:{key}", ttl=10):
        #                             ^ only ONE worker refreshes; the rest
        #                               keep serving the still-valid value
        submit_background(lambda: cache.setex(key, ttl, loader(key)))

    return entry.value                                  # always served instantly

A closely related and often better technique is stale-while-revalidate: serve the expired value immediately and refresh in the background. HTTP has this built in (Chapter 5.4), and it converts an expiry from a latency spike into a brief window of slightly stale data.

Cache-Control: max-age=60, stale-while-revalidate=300
#              ^ fresh for 60s
#                            ^ for 300s after that, serve the stale copy
#                              instantly AND refresh in the background

Aspect	Reactive (TTL)	Refresh-ahead / stale-while-revalidate
p99 on hot keys	Spikes each TTL period	Flat
Origin load	Proportional to misses	Proportional to refreshes, whether needed or not
Wasted work	None	Refreshes keys that nobody asks for again
Freshness	Never stale past TTL	May serve slightly stale data
Complexity	Trivial	Needs background workers and a refresh lock

The cost is real: refreshing everything wastes origin capacity on keys that will never be requested again. The practical rule is to refresh only demonstrably hot keys — those with recent access counts above a threshold — and let the long tail expire normally.

Key Takeaways

Refresh-ahead reloads entries before expiry so users never pay the miss on hot keys.

It flattens the p99 spike that a plain TTL creates once per period per key.

stale-while-revalidate achieves the same effect by serving stale and refreshing behind it.

Refresh only hot keys, and take a lock so one worker refreshes rather than all of them.

🧪 Practice

Implement refresh-ahead triggering at 80% of TTL with a single-refresher lock.

Explain how stale-while-revalidate differs from refresh-ahead in what the user receives.

Interview: Your p50 is 3 ms and your p99 is 250 ms, matching the origin latency exactly. What is happening and how do you fix it? (Hint: the p99 requests are not different requests — they are unluckily timed ones.)

Eviction Policies: LRU, LFU, FIFO, TTL

A cache is finite, so eventually it must remove something to make room. The eviction policy decides what, and it directly determines the hit ratio — a bad policy discards exactly the entries you were about to need.

The theoretical optimum (Bélády's algorithm) evicts the item that will be requested furthest in the future, which requires knowing the future. Every real policy is a heuristic that predicts future access from past behaviour.

 THE FOUR CLASSIC POLICIES

  LRU  Least Recently Used     evict what has not been touched longest
       assumes temporal locality; the workhorse default
       weakness: a single large scan evicts the whole working set

  LFU  Least Frequently Used   evict what has been requested fewest times
       resists scans; better for stable popularity
       weakness: an old, once-popular item is hard to dislodge (needs ageing)

  FIFO First In, First Out     evict the oldest inserted, regardless of use
       trivially cheap; ignores access entirely, so hit ratios are poor

  TTL  Time To Live            evict when the entry is older than its TTL
       not really an eviction policy: a FRESHNESS policy, used alongside one

from collections import OrderedDict

class LRUCache:
    """LRU in O(1): a hash map for lookup, ordering for recency."""
    def __init__(self, capacity):
        self._data = OrderedDict()
        self._capacity = capacity

    def get(self, key):
        if key not in self._data:
            return None
        self._data.move_to_end(key)          # mark as most recently used
        return self._data[key]

    def put(self, key, value):
        if key in self._data:
            self._data.move_to_end(key)
        self._data[key] = value
        if len(self._data) > self._capacity:
            self._data.popitem(last=False)   # evict the least recently used

# The scan problem, and why modern caches are not plain LRU.
#
# A nightly report reads 1,000,000 rows once. Plain LRU dutifully caches every
# one, evicting the hot working set that serves live traffic. The hit ratio
# collapses at 02:00 and recovers slowly.
#
# Real caches defend against this:
#   - SEGMENTED LRU: new entries enter a small probation segment and are only
#     promoted to the protected segment on a SECOND access. A one-time scan
#     never reaches the protected area.
#   - TinyLFU (Caffeine): a compact frequency sketch decides whether a new
#     entry deserves to displace the eviction candidate at all.
#   - Redis: approximate LRU/LFU by sampling a few keys rather than maintaining
#     exact global order — O(1) memory, nearly the same hit ratio.

Policy	Cost	Strength	Weakness
LRU	O(1)	Matches temporal locality	Destroyed by scans
LFU	O(1) approx	Resists scans; stable popularity	Stale popularity without ageing
FIFO	O(1)	Cheapest, simplest	Ignores access; low hit ratio
Random	O(1)	No metadata at all	Surprisingly decent; unpredictable
Segmented LRU / TinyLFU	O(1)	Scan-resistant, high hit ratio	More complex
TTL	O(1)	Bounds staleness	Not a capacity policy on its own

Two practical notes. TTL and eviction are orthogonal: TTL bounds how wrong a value may be, while eviction bounds how much memory you use — you almost always want both. And rising eviction rate is the metric to alert on: it means the working set has outgrown the cache, and the hit ratio is about to fall.

Redis makes this explicit through maxmemory-policy, where the choice between allkeys-lru (evict anything, cache behaviour) and volatile-lru (evict only keys with a TTL, protecting persistent data) is one of the more consequential configuration decisions in a shared Redis instance.

Key Takeaways

Eviction policy predicts future access from past behaviour; LRU is the sensible default.

A large scan destroys a plain LRU cache — segmented LRU and TinyLFU exist to prevent it.

TTL bounds staleness, eviction bounds memory; they are complementary, not alternatives.

Alert on eviction rate: it rises before the hit ratio falls.

🧪 Practice

Trace LRU and LFU on the access sequence A B C A B D A B E with a capacity of 3, and compare final contents.

Explain why allkeys-lru is dangerous on a Redis instance that also holds non-cache data, and what to use instead.

Interview: Your hit ratio drops every night at 02:00 and recovers by 08:00. What is happening? (Hint: something reads a great deal of data exactly once, and your cache believes it matters.)

5.3 Distributed Caching

Once a cache is shared by many application instances it becomes a distributed system of its own, with its own partitioning, failure, and consistency problems.

Redis and Memcached Architectures

An in-process cache is fast but private: twenty application instances hold twenty independent copies, disagree with each other, and lose everything on deploy. A distributed cache is a separate service all instances share, which gives one consistent view, survives application restarts, and can hold far more than any single process. The two dominant choices — Redis and Memcached — make opposite bets on what a cache should be.

Memcached is deliberately minimal: an in-memory key-value store of opaque byte strings, multi-threaded, with no persistence and no replication. Its simplicity is the feature — it scales vertically across cores and essentially never surprises you.

Redis is a data-structure server that happens to be used as a cache. Values can be strings, hashes, lists, sets, sorted sets, streams, and bitmaps, with atomic operations on each. It is single-threaded for command execution (so every operation is atomic without locks), and it offers persistence, replication, pub/sub, and server-side scripting.

 MEMCACHED                            REDIS

  [key] -> opaque bytes                [key] -> string | hash | list | set |
                                                sorted set | stream | bitmap
  multi-threaded                       single-threaded command execution
  no persistence                       RDB snapshots + AOF log (optional)
  no replication                       replication + Sentinel + Cluster
  client-side sharding only            native Cluster mode (16384 hash slots)
  LRU eviction                         8 maxmemory policies
  ~1 MB value limit (default)          512 MB value limit

  Memcached: "a cache, and nothing else."
  Redis:     "a fast data store you can use as a cache."

# Redis's value types are the reason to choose it — each avoids a read-modify-
# write round trip that a plain key-value store would force.

r.hset("user:7", mapping={"name": "Ada", "tier": "pro"})
r.hget("user:7", "tier")                    # read ONE field, not the whole object

r.zadd("leaderboard", {"ada": 4820})
r.zrevrange("leaderboard", 0, 9, withscores=True)   # top 10, O(log n + 10)

r.incrby("views:post:42", 1)                # atomic; no read-modify-write race
r.pfadd("uniques:2026-03-11", user_id)      # HyperLogLog: ~12 KB for millions
r.pfcount("uniques:2026-03-11")             #   of unique values, ~0.8% error

# Single-threaded execution means a slow command blocks EVERYTHING.
r.keys("user:*")        # O(n) over the whole keyspace — never in production
r.scan_iter("user:*")   # cursor-based, incremental, safe

Dimension	Memcached	Redis
Data types	Strings only	Rich structures
Threading	Multi-threaded	Single-threaded commands (I/O threads separate)
Memory efficiency	Slightly better for plain strings	More per-key overhead
Persistence	None	RDB and/or AOF
Replication	None	Built in
Clustering	Client-side hashing	Native Cluster mode
Atomic operations	Basic (incr, cas)	Extensive, plus Lua scripting
Best when	Pure cache, huge scale, simplicity	You need structures, atomicity, or persistence

The choice in practice: Redis unless you specifically want the constraint of Memcached. Redis's extra capability costs little, and most teams eventually need at least atomic counters or sorted sets. Memcached remains attractive at very large scale for pure string caching, where its multi-threaded simplicity and lower per-key overhead pay off.

One warning that applies to Redis specifically: because it can persist, teams start storing data there that is not reconstructible, and then a cache eviction becomes data loss. Decide explicitly whether an instance is a cache (evictable, allkeys-lru) or a store (never evict, noeviction), and never mix the two in one instance.

Key Takeaways

A distributed cache gives all instances one shared, consistent view that survives deploys.

Memcached is a minimal multi-threaded string cache; Redis is a single-threaded data-structure server.

Redis's structures (hashes, sorted sets, counters, HyperLogLog) remove round trips and races.

Redis executes commands single-threaded — one O(n) command stalls every client. Never mix cache and store in one instance.

🧪 Practice

Rewrite "read a user object, increment its login count, write it back" using a Redis hash and one atomic command.

Explain why KEYS * on a production Redis with 10 million keys causes an outage, and what to use instead.

Interview: When would you choose Memcached over Redis in 2026? (Hint: think about what you gain by having fewer features, and at what scale that starts to matter.)

Cache Sharding and Replication

One cache node has a memory ceiling and is a single point of failure. Scaling past it means the same two mechanisms as any data system: sharding to distribute keys across nodes, and replication to keep copies for availability. What differs is that a cache is derived data, which changes the calculus considerably — losing a shard is a performance event, not a data-loss event.

Sharding a cache uses consistent hashing (Chapter 4.5) for the reason established there: plain hash % N remaps almost every key when the cluster changes, and for a cache that means a near-total cold start precisely when you are adding capacity because you are under load.

 REDIS CLUSTER: 16384 HASH SLOTS

  CRC16(key) mod 16384 -> slot -> node

  node A: slots     0 - 5460      node B: slots  5461 - 10922
  node C: slots 10923 - 16383

  Adding node D: move whole SLOTS (not a rehash) to D. Only the migrated
  slots' keys are affected; everything else keeps serving.

 HASH TAGS: forcing related keys onto one node

  user:7:profile   and  user:7:settings   -> may land on different nodes
  user:{7}:profile and  user:{7}:settings -> only "7" is hashed, so BOTH land
                                             on the same node
  Required for multi-key operations (MGET, transactions, Lua) in Cluster mode.

# Replication for a cache is about AVAILABILITY, not durability.
# Redis Cluster: each shard has one primary and one or more replicas.
#
#   client writes -> primary (asynchronous replication to replica)
#   primary dies  -> replica is promoted; some very recent writes may be lost
#
# For a cache, losing the last few writes is fine: the next read is a miss and
# repopulates from the origin. That is exactly why cache replication can be
# asynchronous where a database's often cannot.

def get_with_failover(key, cluster, db):
    try:
        if (v := cluster.get(key)) is not None:
            return v
    except (ConnectionError, ClusterDownError):
        # NEVER let a cache outage become an application outage. Degrade to the
        # origin, and shed load if the origin cannot take the full traffic.
        metrics.increment("cache.unavailable")
        return db.query(key)
    v = db.query(key)
    try:
        cluster.setex(key, 300, v)
    except Exception:
        pass                     # a failed cache write is not a failed request
    return v

Concern	Cache	Database
Losing a node	Hit ratio drops; origin load spikes	Data loss or unavailability
Replication mode	Asynchronous is fine	Often must be synchronous
Rebalancing	Can simply cold-start the new node	Must move all data correctly
Consistency across replicas	Tolerable	Critical
The real risk	Origin cannot absorb the miss traffic	Data integrity

That final row is the one that causes outages. A cache node holding 25% of a 10,000 RPS workload at a 95% hit ratio is absorbing roughly 2,375 RPS. If it dies and those keys all miss, the origin's load jumps by that amount instantly. Capacity-plan the origin for partial cache loss, not just for the steady state — and consider whether your system needs to shed load rather than melt.

Two sharding refinements are worth knowing. Replicating hot keys across several shards (with a random read replica chosen per request) fixes the single-hot-key problem that sharding alone cannot. And client-side consistent-hashing libraries (the Memcached approach) avoid running a cluster manager at the cost of every client needing the same node list.

Key Takeaways

Shard a cache with consistent hashing so adding capacity does not cold-start everything.

Redis Cluster moves whole hash slots; hash tags {...} co-locate related keys for multi-key operations.

Cache replication buys availability, not durability — asynchronous is acceptable.

Plan origin capacity for partial cache loss, and never let a cache outage fail requests.

🧪 Practice

Compute the origin RPS spike when one of four cache nodes fails at 40,000 RPS and a 96% hit ratio.

Explain why MGET user:7:profile user:7:settings fails in Redis Cluster and how hash tags fix it.

Interview: Your cache cluster is healthy but one node's CPU is at 100%. What is likely wrong? (Hint: sharding distributes keys evenly, which is not the same as distributing requests evenly.)

Cache Invalidation Techniques

The old joke is that there are only two hard problems in computer science: cache invalidation and naming things. The reason invalidation is genuinely hard is that a cache entry has no way to know the underlying data changed — something must tell it, and that something has to know every key derived from every piece of data.

There are three families of answer, and mature systems use all three at once.

 1. TTL (EXPIRY)                    simplest; staleness is bounded but guaranteed
    set key with ttl=300            "wrong for at most 5 minutes"
    no coordination needed          works even when invalidation messages are lost

 2. EXPLICIT INVALIDATION           precise; requires knowing every derived key
    on write -> delete key(s)       fails silently when someone forgets a path
                                    or when data changes outside your code

 3. VERSIONED / KEY MUTATION        never invalidate; change the key instead
    key = f"product:{id}:v{ver}"    old entries age out naturally
    bump ver on write               no delete storm, no race — but memory churn

# TECHNIQUE: key versioning by namespace. One counter invalidates a whole
# family of derived keys without enumerating or deleting any of them.
def product_list_key(tenant_id, filters, redis):
    version = redis.get(f"ver:products:{tenant_id}") or 0
    return f"products:{tenant_id}:v{version}:{hash_filters(filters)}"

def on_product_changed(tenant_id, redis):
    redis.incr(f"ver:products:{tenant_id}")     # every derived key is now unreachable
    # No scanning, no delete storm, no missed key. The old entries simply
    # become garbage and are evicted by TTL/LRU in their own time.

# TECHNIQUE: tag-based invalidation, when you need precision.
def cache_with_tags(key, value, tags, redis, ttl=300):
    pipe = redis.pipeline()
    pipe.setex(key, ttl, value)
    for tag in tags:                            # e.g. ["product:42", "category:9"]
        pipe.sadd(f"tag:{tag}", key)            # reverse index: tag -> keys
        pipe.expire(f"tag:{tag}", ttl * 2)
    pipe.execute()

def invalidate_tag(tag, redis):
    keys = redis.smembers(f"tag:{tag}")
    if keys:
        redis.delete(*keys, f"tag:{tag}")       # every page mentioning product 42

 FAN-OUT INVALIDATION ACROSS INSTANCES

  One instance writes; every instance's LOCAL (L1) cache must be told.

  [instance 1] --write--> [database]
        |
        +-- publish "invalidate:product:42" --> [Redis pub/sub or a topic]
                                                   |
                    +------------------+-----------+------------+
                    v                  v                        v
             [instance 1]        [instance 2]  ...       [instance 20]
             drop local key      drop local key           drop local key

  Pub/sub is fire-and-forget: an instance that was restarting misses the
  message and keeps a stale entry until its TTL. This is precisely why L1
  TTLs must be short even with explicit invalidation.

Technique	Precision	Cost	Fails when
TTL only	None	Zero	Staleness window is unacceptable
Explicit delete	Exact	Must know every key	A write path forgets to invalidate
Key versioning	Namespace	Memory churn from orphans	You need per-key precision
Tag-based	Exact	A reverse index to maintain	Tag sets grow unbounded
Pub/sub fan-out	Exact	Best-effort delivery	An instance misses the message
CDC-driven (4.6)	Exact	A pipeline to operate	Nothing forgets — it reads the log

The most robust design combines them: CDC or explicit invalidation for precision, key versioning for whole namespaces, and a TTL underneath everything as the backstop. The TTL is what saves you when the precise mechanism fails, and it will fail — someone will write to the database from a migration script, an admin tool, or another service that never publishes an invalidation.

Key Takeaways

A cache cannot know data changed; something must tell it, and that path can always be missed.

TTL bounds staleness with zero coordination — always have one as a backstop.

Key versioning invalidates an entire namespace with one counter increment and no deletes.

Local per-instance caches need short TTLs even with pub/sub fan-out, because delivery is best-effort.

🧪 Practice

Design invalidation for a product page cached under several keys (product, category listing, search results, sitemap).

Implement namespace versioning and explain what happens to the orphaned old entries.

Interview: A nightly batch job updates prices directly in the database and users see old prices all morning. What went wrong and what are the fixes? (Hint: the application's invalidation code is correct — and irrelevant.)

Cache Stampede and Thundering Herd

A cache is most valuable for exactly the keys that are requested most, which means the moment a hot key expires, every concurrent request for it misses at once. All of them go to the origin, all of them compute the same value, and the origin — sized for the cached traffic rate — collapses. This is a cache stampede (or thundering herd), and it is the most common way a caching layer causes an outage rather than preventing one.

The arithmetic is brutal. A key served 5,000 times per second with a 60-second TTL protects the origin from 300,000 requests per minute. When it expires, if a miss takes 200 ms, roughly 1,000 requests arrive before the first one finishes — and every one of them is a database query.

 THE STAMPEDE

  t=0.000  key expires
  t=0.001  req 1 misses -> starts a 200 ms origin query
  t=0.002  req 2 misses -> starts an IDENTICAL query
  ...
  t=0.200  req 1000 misses -> 1000 concurrent identical queries
  t=0.200  req 1 finishes and populates the cache
           ...the other 999 queries are still running, doing nothing useful

  Worse: the origin is now slow, so misses take 2 s instead of 200 ms, so
  10,000 requests pile up instead of 1,000. The system does not recover on
  its own.

# DEFENCE 1: request coalescing (single-flight). Only one caller loads; the
# rest wait for its result. This is the single most effective fix.
def get_coalesced(key, cache, loader, ttl=300):
    if (v := cache.get(key)) is not None:
        return v
    lock_key = f"lock:{key}"
    if cache.set(lock_key, "1", nx=True, ex=10):        # atomic: exactly one winner
        try:
            v = loader(key)                              # the ONLY origin call
            cache.setex(key, ttl, v)
            return v
        finally:
            cache.delete(lock_key)
    # Lost the race: wait briefly for the winner rather than hitting the origin.
    for _ in range(50):
        time.sleep(0.02)
        if (v := cache.get(key)) is not None:
            return v
    return loader(key)          # last resort after ~1 s: better slow than wrong

# DEFENCE 2: probabilistic early expiration. Each request independently decides
# whether to refresh early, so refreshes spread out instead of synchronizing.
def should_refresh_early(entry, ttl, beta=1.0):
    # XFetch: probability rises as expiry approaches, scaled by how long the
    # recompute takes — expensive values start refreshing sooner.
    gap = entry.recompute_seconds * beta * math.log(random.random())
    return time.time() - gap >= entry.expires_at

# DEFENCE 3: jittered TTLs, so keys populated together do not expire together.
cache.setex(key, 300 + random.randint(-30, 30), value)

Three related herd problems have the same shape and different triggers:

Variant	Trigger	Fix
Stampede (one key)	A hot key expires	Coalescing, early refresh, jitter
Synchronized expiry	Many keys populated together expire together	TTL jitter
Cold start	Cache restart or deploy empties it	Warm-up on boot; gradual traffic ramp
Cache failure	The cache cluster goes down	Origin capacity headroom, load shedding, circuit breakers

The last two are why the origin must never be sized assuming the cache is always there. Either provision enough origin capacity for a plausible cache loss, or install a circuit breaker (Chapter 9.2) that sheds load and serves degraded responses rather than letting the database die and take everything with it.

Key Takeaways

When a hot key expires, every concurrent request misses simultaneously and hits the origin.

Request coalescing — one loader, everyone else waits — is the most effective defence.

Jitter TTLs so keys populated together do not expire together.

Never size the origin assuming the cache is up; plan for cold starts and cache outages explicitly.

🧪 Practice

Compute the number of concurrent origin queries for a key at 3,000 RPS with a 150 ms miss, without coalescing.

Implement single-flight coalescing and prove one origin call under 200 concurrent misses.

Interview: Your cache cluster restarts and the database immediately falls over. Describe both the immediate response and the permanent fix. (Hint: the permanent fix is not "make the cache more reliable".)

Negative Caching

Caches usually store answers. But "there is no answer" is also an answer, and failing to cache it creates a specific vulnerability: every request for a key that does not exist bypasses the cache entirely and hits the origin. Under normal traffic that is a minor inefficiency. Under a malicious or buggy client enumerating random ids, it is a cache penetration attack that turns your cache into a pass-through and your database into the bottleneck.

Negative caching stores the absence of a value — a "not found" marker — so repeated lookups for a missing key are answered from the cache like any other.

 WITHOUT NEGATIVE CACHING            WITH NEGATIVE CACHING

  GET /users/999999                   GET /users/999999
   cache miss                          cache miss
   db query -> not found               db query -> not found
   return 404                          cache.setex(key, 60, NOT_FOUND)
   (nothing cached)                    return 404

  Attacker requests 100,000           Second request onward is served from
  random ids -> 100,000 database      the cache. The origin sees each missing
  queries. The cache is useless.      id at most once per 60 seconds.

NOT_FOUND = object()             # a distinct sentinel; NOT None, NOT empty string

def get_user(user_id, cache, db):
    key = f"user:{user_id}"
    cached = cache.get(key)

    if cached is NOT_FOUND_MARKER:            # negative hit
        return None
    if cached is not None:                    # positive hit
        return cached

    user = db.query_user(user_id)
    if user is None:
        # Short TTL: a negative entry must not outlive the creation of the
        # real record by long. 30-60 s is typical; positive entries live longer.
        cache.setex(key, 60, NOT_FOUND_MARKER)
        return None

    cache.setex(key, 300, user)
    return user

def create_user(user_id, data, cache, db):
    db.insert_user(user_id, data)
    cache.delete(f"user:{user_id}")           # CRITICAL: clear any negative entry,
                                              # or the new user "does not exist"
                                              # for up to 60 seconds

For very large keyspaces, a Bloom filter is a cheaper defence: a compact probabilistic structure that answers "definitely not present" or "possibly present" without storing the keys.

# A Bloom filter of all valid ids sits in front of the cache. It can produce
# false positives (says "maybe" for a missing key) but never false negatives,
# so a "no" is always trustworthy.
def get_user_bloom(user_id, bloom, cache, db):
    if not bloom.might_contain(user_id):
        return None                  # definitely does not exist: zero I/O
    return get_user(user_id, cache, db)

# ~1.2 MB holds 1 million ids at a 1% false-positive rate — far less than
# caching a million negative entries.
# The catch: deletions are awkward (standard Bloom filters cannot remove), so
# the filter is usually rebuilt periodically.

Aspect	Guidance
Negative TTL	Much shorter than positive TTL (30-60 s typical)
Marker value	A distinct sentinel — never `None`, which is indistinguishable from a miss
On creation	Delete the negative entry, or the new record appears missing
Memory risk	Enumeration attacks fill the cache with negatives — cap it or use a Bloom filter
Large keyspaces	Bloom filter in front; rebuild it periodically
Rate limiting	Complementary: limit lookups of non-existent ids per client

Key Takeaways

Uncached "not found" results let missing-key traffic bypass the cache entirely.

Store a distinct sentinel for absence, never None — the two must be distinguishable.

Give negative entries a much shorter TTL, and delete them when the record is created.

For huge keyspaces, a Bloom filter rejects missing keys with far less memory.

🧪 Practice

Add negative caching to a lookup, including the sentinel and the invalidation on creation.

Explain the bug that appears if a negative entry has the same TTL as a positive one, from a user's point of view.

Interview: An attacker requests millions of random product ids and your database is overwhelmed. Give three layered defences. (Hint: one caches, one filters, one throttles.)

Cache Consistency Trade-offs

Every cache is a copy, and every copy can disagree with the original. There is no configuration that removes this — you can only choose how the cache is allowed to be wrong, for how long, and what that costs. Being explicit about that choice per data type is what separates a caching design from a caching accident.

The question to ask about any cached value is: what is the worst thing that happens if a user sees a value that is N seconds old? The answer sets the TTL, the invalidation strategy, and sometimes the decision not to cache at all.

 THE STALENESS TOLERANCE SCALE

  0 s        account balance during a transfer      -> do not cache
             inventory count at checkout            -> do not cache, or lock
  1-5 s      permissions and feature flags          -> short TTL + pub/sub
  30 s       a user's own dashboard                 -> TTL, keyed per user
  5 min      product catalogue, prices              -> TTL + explicit invalidation
  1 hour     public article content                 -> CDN, long TTL, purge on edit
  1 year     hashed static assets                   -> immutable, never invalidated

  The scale is a business question, not a technical one. "How stale is too
  stale?" is answered by the domain, not by the cache.

# Make the trade-off a declared property of the data, not an ad-hoc choice at
# each call site. This is the single most useful discipline in caching.
CACHE_POLICY = {
    "account_balance":  {"cacheable": False,
                         "reason": "financial correctness; must be authoritative"},
    "permissions":      {"ttl": 5,    "invalidate": "pubsub",
                         "reason": "stale permissions are a security issue"},
    "user_dashboard":   {"ttl": 30,   "invalidate": None},
    "product":          {"ttl": 300,  "invalidate": "explicit"},
    "article_html":     {"ttl": 3600, "invalidate": "cdn_purge"},
    "static_asset":     {"ttl": 31_536_000, "invalidate": "url_versioning"},
}
# Reviewable, testable, and greppable. When someone asks "why is this stale for
# five minutes?", the answer is written down.

Two specific hazards deserve naming because they are the ones that cause incidents rather than annoyance:

Hazard	Why it is worse than ordinary staleness
Cached authorization	A revoked user keeps access for the TTL — a security hole, not a UX issue
Cached negative permission	A newly-granted user is locked out; support tickets
Cached aggregates over cached data	Staleness compounds: a 5-minute cache built from 5-minute caches can be 10 minutes stale
Per-instance L1 disagreement	Two refreshes return different answers; users think the app is broken

For the compounding case, the rule is to build derived caches from the origin, not from other caches, or to accept and document the summed staleness.

Finally, the honest framing for the whole chapter: caching trades correctness for performance. Everything in Chapter 4 was about making data correct; caching deliberately relaxes that in exchange for speed and capacity. That is a good trade for most data and a terrible one for some, and knowing which is which is the actual skill.

Key Takeaways

You cannot eliminate cache inconsistency — only choose its shape, duration, and cost.

Ask "what breaks if this is N seconds old?" and let the domain set the TTL.

Declare cache policy per data type in one place rather than deciding at each call site.

Cached authorization is a security decision; staleness compounds when caches are layered on caches.

🧪 Practice

Write a cache policy table for a banking app covering balances, transaction history, exchange rates, and branch locations.

Explain the security consequence of a 15-minute permissions cache when an employee is terminated, and design the fix.

Interview: A stakeholder asks for "caching, but always fresh". How do you respond? (Hint: reframe the request as a measurable tolerance, then price each level.)

5.4 Content Delivery Networks

A CDN is a cache you rent, distributed across the planet; this subchapter covers its architecture, how content gets into it, the HTTP headers that control it, and what it can do beyond caching bytes.

CDN Architecture and PoPs

Chapter 2.4 introduced edge networks as a latency tool. This topic looks inside one, because the internal structure explains behaviour that otherwise seems arbitrary — why a "purge" is not instant, why hit ratios differ per region, and why a CDN can serve a terabit while your origin serves a gigabit.

A CDN is a hierarchy. Points of presence (PoPs) — hundreds of them — sit in internet exchange facilities near population centres. Each holds edge servers with a local cache. Behind them, a smaller number of shield or mid-tier caches aggregate misses from many PoPs, so the origin sees one request rather than one per PoP.

 CDN TOPOLOGY

   [users in Sydney]      [users in Tokyo]       [users in Berlin]
          |                      |                       |
      [PoP SYD]              [PoP NRT]               [PoP BER]      ~300 PoPs
       cache                  cache                   cache
          \                     |                      /
           \                    |                     /
            +------------ [ SHIELD / mid-tier cache ] +           ~10-30 shields
                                 |
                                 v
                            [ ORIGIN ]                            1 (or a few)

  WITHOUT a shield: a cold object is fetched from origin once PER PoP.
                    300 PoPs -> 300 origin requests for the same object.
  WITH a shield:    the shield fetches once; every PoP fetches from the shield.
                    Origin requests: 1. This is "origin shielding" and it is
                    the difference between a 95% and a 99.7% origin offload.

Three mechanisms make the whole thing work, and each has an operational consequence:

Mechanism	What it does	Consequence you will notice
Anycast routing	One IP announced from every PoP; BGP routes users to the nearest	You cannot choose a PoP; routing follows network topology, not geography
Tiered caching	Shields absorb PoP misses	Origin offload jumps dramatically
Per-PoP independence	Each PoP caches separately	Hit ratio is per-PoP; a rarely-visited region has a cold cache

That last row explains a common confusion. A global 95% hit ratio can hide a PoP in a low-traffic region running at 40%, because objects there expire before enough requests arrive to justify holding them. It is also why purging is not instantaneous: the purge must reach every PoP, which typically takes seconds and is best-effort.

# What a CDN edge does per request, conceptually.
def edge_handle(request, cache, shield):
    key = cache_key(request)         # usually: host + path + query + Vary headers

    entry = cache.get(key)
    if entry and entry.fresh:
        return entry.response                        # HIT: ~5-15 ms for the user

    if entry and entry.stale_while_revalidate_ok:
        background_revalidate(key, shield)
        return entry.response                        # stale but instant (5.2)

    # MISS: go to the shield, not the origin. Collapse concurrent misses for the
    # same key into ONE upstream request ("request collapsing") — the CDN's
    # built-in defence against the stampede from 5.3.
    with cache.collapse(key):
        response = shield.fetch(request)

    if is_cacheable(response):                       # driven by Cache-Control
        cache.store(key, response, ttl=compute_ttl(response))
    return response

The cache key deserves attention because it decides your hit ratio more than any other setting. By default it includes host, path, and query string — which means a tracking parameter like ?utm_source=twitter creates a separate cache entry for every campaign, fragmenting what should be one object. Normalizing the cache key (stripping irrelevant query parameters, ignoring cookies, limiting Vary) is usually the highest-value CDN tuning available.

Key Takeaways

A CDN is tiered: many PoPs, fewer shields, one origin; shielding is what produces very high origin offload.

Anycast routes users to the topologically nearest PoP — you do not control which one.

Each PoP caches independently, so hit ratios vary by region and purges take seconds to propagate.

Cache-key normalization (query parameters, cookies, Vary) is the biggest lever on CDN hit ratio.

🧪 Practice

Compute origin requests for one cold object across 200 PoPs, with and without a shield tier.

Explain why appending ?utm_source=... to shared links can collapse your CDN hit ratio, and how to fix it.

Interview: Your CDN reports a 94% hit ratio but origin traffic is still high. What do you investigate? (Hint: a global average hides both regional variation and key fragmentation.)

Push vs. Pull CDNs

Content has to get into the CDN somehow, and there are two models. In a pull CDN, you change nothing: the CDN fetches from your origin on the first request for each object and caches the result. In a push CDN, you upload content to the CDN in advance, and the CDN serves it without ever contacting an origin.

Pull is the overwhelming default because it requires no publishing pipeline and automatically adapts to what people actually request. Push exists for cases where the first-request penalty is unacceptable or where there is no origin to pull from.

 PULL (origin-fetch)                  PUSH (pre-populated)

  first request -> MISS               you upload  -> [CDN storage]
      CDN -> origin -> CDN            every request -> HIT, always
      user waits for the full
      origin round trip               No origin needed at request time.
  later requests -> HIT               You must publish, version, and expire
                                      content yourself.
  Origin must stay available.         Storage cost for everything, whether
  Cold objects are always slow once.  requested or not.

# PULL: nothing to do but set headers correctly. The CDN discovers content.
#   origin serves /images/product-42.jpg with Cache-Control: public, max-age=86400
#   first user in Tokyo pays the origin fetch; the next 10,000 do not

# PUSH: an explicit publishing step, usually in CI/CD.
def publish_release(build_dir, cdn):
    for path in walk(build_dir):
        # Content-hashed names mean a push never overwrites — it only adds.
        # Old versions stay available for users still running the old client.
        cdn.upload(
            key=f"static/{content_hash(path)}/{basename(path)}",
            body=read(path),
            cache_control="public, max-age=31536000, immutable",
            content_encoding="br",              # pre-compressed at build time
        )
    cdn.upload(key="manifest.json", body=build_manifest(),
               cache_control="public, max-age=60")   # the only mutable object

Dimension	Pull CDN	Push CDN
Setup	Point a DNS record; done	Build a publishing pipeline
First request	Slow (origin fetch)	Fast (already there)
Origin dependency	Required and must stay up	None at request time
Storage cost	Only what is requested	Everything you upload
Content updates	Automatic on TTL expiry	Explicit re-publish
Best for	Websites, APIs, most assets	Large media, software releases, launch-day spikes
Stale content risk	Bounded by TTL	Until you push a correction

The practical pattern most teams end up with is pull for everything, plus pre-warming for known events. Pre-warming is push-like behaviour on a pull CDN: before a product launch or a major release, you issue requests for the objects you know will be hot so they are already cached when real traffic arrives.

# Pre-warming: make the CDN fetch the objects before your users do.
# Requests must reach many PoPs to be useful — this is why CDNs offer a
# prefetch API rather than expecting you to curl from one machine.
curl -X POST https://api.cdn.example/v1/prefetch \
     -d '{"urls": ["https://cdn.example.com/launch/hero.mp4"], "regions": "all"}'

Key Takeaways

Pull CDNs fetch on first request and need no publishing pipeline — the default choice.

Push CDNs pre-populate content, removing the first-request penalty and the origin dependency.

Push suits large media and software distribution; pull suits websites and APIs.

Pre-warming gives pull CDNs push-like behaviour for predictable traffic spikes.

🧪 Practice

Decide push or pull for: a marketing site's images, a 4 GB game patch, an API's JSON responses, a video-on-demand library.

Design the publishing step for a push CDN with content-hashed filenames and explain what happens to old versions.

Interview: A product launch at 09:00 will drive 50x normal traffic to one page. What do you do beforehand? (Hint: the goal is that the first real user is not the one who populates the cache.)

Cache-Control and ETags

Everything a CDN, proxy, or browser does with your content is driven by response headers. Getting them right is the difference between a 99% offload and an origin that receives every request — and unlike most performance work, it costs nothing but attention.

There are two independent mechanisms. Freshness (Cache-Control) says how long a copy may be used without asking. Validation (ETag, Last-Modified) lets a cache ask cheaply whether its copy is still good, receiving a tiny 304 instead of the whole body.

### FRESHNESS: the directives that matter, and who they speak to
Cache-Control: public, max-age=60, s-maxage=3600, stale-while-revalidate=86400, stale-if-error=604800
#              |       |           |             |                            |
#              |       |           |             |                            +- serve stale for
#              |       |           |             |                               7 days if the origin
#              |       |           |             |                               is DOWN (free uptime)
#              |       |           |             +- serve stale instantly and refresh behind it
#              |       |           +- SHARED caches (CDN): 1 hour
#              |       +- everyone else (browser): 60 seconds
#              +- may be stored by shared caches (never use with private data)

### VALIDATION: a cheap freshness check when max-age has elapsed
HTTP/1.1 200 OK
ETag: "v7-9f2c"
Last-Modified: Wed, 11 Mar 2026 10:04:22 GMT

GET /api/products
If-None-Match: "v7-9f2c"

HTTP/1.1 304 Not Modified          # no body: a few hundred bytes instead of 200 KB
ETag: "v7-9f2c"

# Generating a correct ETag. Two variants with different guarantees.
def strong_etag(body_bytes):
    # Strong: byte-for-byte identity. Required for range requests and for
    # If-Match optimistic concurrency (Chapter 3.2).
    return '"' + hashlib.sha256(body_bytes).hexdigest()[:16] + '"'

def weak_etag(resource):
    # Weak: semantically equivalent, not byte-identical. Cheaper — no need to
    # render the body to compute it.
    return f'W/"{resource.id}-{int(resource.updated_at.timestamp())}"'

def handle(request, resource):
    etag = weak_etag(resource)
    if request.headers.get("If-None-Match") == etag:
        return Response(304, headers={"ETag": etag})     # body never rendered
    body = render(resource)                              # only on a real miss
    return Response(200, body, headers={
        "ETag": etag,
        "Cache-Control": "public, max-age=60, s-maxage=600",
    })

Directive	Effect
`public` / `private`	May / may not be stored by shared caches
`max-age=N`	Fresh for N seconds (all caches)
`s-maxage=N`	Overrides `max-age` for shared caches only
`no-cache`	Store it, but revalidate before every use (not "do not cache")
`no-store`	Never write it to any cache — the actual "do not cache"
`must-revalidate`	Once stale, never serve without revalidating
`immutable`	Never revalidate during `max-age`, even on a reload
`stale-while-revalidate=N`	Serve stale for N seconds while refreshing in background
`stale-if-error=N`	Serve stale for N seconds when the origin errors

Two directives are worth singling out. stale-if-error gives you free availability: when the origin is down, the CDN keeps serving the last good copy instead of an error page — one header that turns an outage into a degraded but working site. And no-cache is the most misread directive in HTTP: it means "revalidate before use", not "do not store". The directive people usually intend is no-store.

Finally, Vary multiplies your cache entries. Vary: Accept-Encoding is necessary and cheap (two or three variants). Vary: User-Agent creates a separate entry per browser string — effectively disabling the cache. Vary: Cookie on a site that sets any cookie does the same.

Key Takeaways

Cache-Control sets freshness; ETag/Last-Modified enable cheap 304 revalidation.

s-maxage targets shared caches separately from browsers — use it to cache longer at the CDN than at the client.

no-cache means revalidate; no-store means never store.

stale-if-error keeps the site up during an origin outage; a careless Vary destroys the hit ratio.

🧪 Practice

Write Cache-Control for: a hashed JS bundle, a news homepage, a logged-in dashboard, a one-time password endpoint.

Implement conditional responses with weak ETags and measure the bytes saved on revalidation.

Interview: Explain the difference between no-cache and no-store, and give a case where using the wrong one is a security problem. (Hint: think about what is written to a shared disk on a shared computer.)

Edge Computing and Edge Functions

A CDN that only caches can serve one thing: what the origin already produced. Anything personalized, authenticated, or dynamic falls through to the origin and pays the full round trip. Edge computing removes that limit by running your code at the PoP — so decisions that need logic but not your database can be made 10 ms from the user instead of 150 ms away.

The constraint that shapes everything about edge functions is that they run in hundreds of locations. They must start in single-digit milliseconds, use little memory, and cannot assume a database connection — so they run in lightweight isolates (typically V8 or WebAssembly) rather than containers, with short CPU budgets and no persistent local state.

 WHAT MOVES TO THE EDGE, AND WHAT CANNOT

  GOOD AT THE EDGE                     STAYS AT THE ORIGIN
  ----------------------------------------------------------------
  redirects and rewrites               anything needing your database
  A/B bucket assignment                complex business logic
  auth token validation (signature)    transactions and writes
  geo/device-based routing             large computations
  header manipulation                  anything needing shared state
  request/response transformation      long-running work
  bot filtering, rate limiting
  serving personalized fragments
  from cached pieces

// An edge function doing four things that would each cost a 150 ms origin trip.
export default async function handler(request) {
  const url = new URL(request.url);
  const country = request.headers.get("cf-ipcountry") ?? "US";   // set by the PoP

  // 1. Geo redirect at the edge: ~10 ms instead of a full origin round trip.
  if (country === "DE" && !url.pathname.startsWith("/de/")) {
    return Response.redirect(new URL("/de" + url.pathname, url), 302);
  }

  // 2. Validate a signed token WITHOUT calling the auth service. Signature
  //    verification needs only the public key — no network, no database.
  const token = request.headers.get("authorization")?.slice(7);
  const claims = token && (await verifyJwt(token, PUBLIC_KEY));
  if (!claims) return new Response("Unauthorized", { status: 401 });

  // 3. A/B assignment at the edge keeps the cached page identical per bucket,
  //    so both variants stay cacheable — personalization without cache loss.
  const bucket = (hashString(claims.sub) % 100) < 50 ? "a" : "b";

  // 4. Fetch the CACHED shell and inject the per-user fragment. The expensive
  //    part is cached globally; only the small personal piece is dynamic.
  const shell = await fetch(new URL(`/shell/${bucket}`, url), {
    cf: { cacheEverything: true, cacheTtl: 3600 },
  });
  return new HTMLRewriter()
    .on("#greeting", { element: (e) => e.setInnerContent(`Hi ${claims.name}`) })
    .transform(shell);
}

Property	Edge function	Origin server
Latency to user	~5-20 ms	~50-250 ms
Cold start	Sub-millisecond (isolates)	Hundreds of ms (containers)
CPU/memory budget	Tight (tens of ms, ~128 MB)	Whatever you provision
Database access	None, or high-latency	Local and fast
State	Stateless, or eventually-consistent edge KV	Full
Runtime	JavaScript/WASM subset	Anything

The pattern that gets the most value is caching the expensive part and computing only the cheap personal part at the edge: a globally cached page shell plus a small edge-rendered fragment. That preserves a high hit ratio while still delivering personalized content, which is otherwise the fundamental trade-off of CDN caching.

Key Takeaways

Edge functions run your code at the PoP, so logic that needs no database avoids the origin round trip entirely.

They run in isolates with tight CPU, memory, and runtime limits and no local state.

Ideal work: redirects, signature validation, routing, A/B assignment, header and HTML transformation.

Cache the expensive shell globally and compute only the small personal piece at the edge.

🧪 Practice

Write an edge function that redirects mobile users to a lightweight page and explain why it beats an origin redirect.

Explain why JWT signature validation works at the edge but a token revocation check generally does not.

Interview: How would you serve a personalized homepage without destroying your CDN hit ratio? (Hint: split the response into a part that is the same for everyone and a part that is not.)

Static Asset Optimization

Static assets — JavaScript, CSS, fonts, images — are usually the majority of the bytes a page transfers and the majority of the time before it becomes usable. They are also the easiest thing in the entire chapter to optimize, because they never change once built, which means they can be cached forever and never invalidated.

The technique that makes that possible is content hashing: name each file after a hash of its contents. app.js becomes app.a3f9c2.js. Because the name changes whenever the content does, the old URL never needs invalidating — you simply stop referencing it.

 THE VERSIONED-URL PATTERN

  BAD                                  GOOD
  /static/app.js                       /static/app.a3f9c2.js
  Cache-Control: max-age=300           Cache-Control: max-age=31536000, immutable

  Must be revalidated every 5 min      Never revalidated. Ever.
  Deploy -> stale for up to 5 min      Deploy -> new filename -> instant pickup
  Cannot cache aggressively            Perfect cache hit ratio
                                       Old versions remain valid for users
                                       still running the old page

  Only the HTML that REFERENCES the assets is short-lived:
  /index.html   Cache-Control: no-cache   (revalidate every time; it is tiny)

// The build emits hashed names and a manifest; the server injects them.
// build output:
//   dist/app.a3f9c2.js      dist/vendor.7b1e04.js      dist/main.9c2d81.css
//   dist/manifest.json  ->  {"app.js": "app.a3f9c2.js", ...}

// Splitting matters as much as hashing: vendor code changes rarely, app code
// changes every deploy. Separate bundles mean a deploy invalidates only the
// small app bundle, not the 400 KB of dependencies.

# Serving the two classes correctly.
location ~* \.[0-9a-f]{6,}\.(js|css|woff2)$ {   # hashed filenames
    add_header Cache-Control "public, max-age=31536000, immutable";
    gzip_static on;                              # serve pre-built .gz
    brotli_static on;                            # and pre-built .br (smaller)
}

location = /index.html {
    add_header Cache-Control "no-cache";         # revalidate; it points at the hashes
}

Technique	Typical saving	Notes
Brotli compression	15-25% smaller than gzip	Pre-compress at build time, not per request
Content hashing	Enables 1-year caching	The single highest-value change
Bundle splitting	Only changed code re-downloads	Split vendor from application
Modern image formats	WebP ~30%, AVIF ~50% vs JPEG	Serve via `<picture>` or CDN negotiation
Responsive images	Large — no 4K image on a phone	`srcset` with width descriptors
`font-display: swap`	Removes invisible-text delay	Text renders before the font loads
Preload / Early Hints	Removes discovery round trips	`103 Early Hints` for critical assets
Tree shaking / minification	30-70% of JavaScript	Build-time, no runtime cost

Two rules of thumb worth internalizing. Compress at build time, not per request — Brotli at maximum quality is far too slow to run per request but essentially free when done once during the build. And the fastest asset is the one you do not send: auditing for unused dependencies and unshipped polyfills usually beats any compression setting.

Key Takeaways

Content-hashed filenames let you cache assets for a year and never invalidate them.

Keep the referencing HTML short-lived; it is small and it points at the hashes.

Split bundles so a deploy invalidates only the code that changed.

Pre-compress with Brotli at build time, and serve modern, correctly-sized image formats.

🧪 Practice

Set up caching headers for a build output containing hashed JS/CSS, an unhashed favicon, and index.html.

Explain why splitting vendor and application bundles improves repeat-visit performance, with a concrete byte estimate.

Interview: A deploy goes out but users see a broken page mixing old and new assets. What happened? (Hint: consider which file they cached and which one it points to.)

Media Streaming Delivery

Video is the hardest content type a CDN handles: files are enormous, viewers have wildly different bandwidth, and a user who waits more than a couple of seconds leaves. Downloading a whole video before playing is impossible, and serving one fixed quality means either buffering on slow connections or wasting bandwidth on fast ones.

Adaptive bitrate streaming (ABR) solves both. The video is encoded at several quality levels and cut into small segments (typically 2-10 seconds). The player downloads a manifest listing the available renditions, then picks a segment quality for each segment based on measured throughput and buffer level — switching mid-playback as conditions change.

 ADAPTIVE BITRATE STRUCTURE

  master.m3u8  (manifest of manifests)
    |-- 1080p/  -> index.m3u8 -> [seg0.ts][seg1.ts][seg2.ts]...   5000 kbps
    |-- 720p/   -> index.m3u8 -> [seg0.ts][seg1.ts][seg2.ts]...   2500 kbps
    |-- 480p/   -> index.m3u8 -> [seg0.ts][seg1.ts][seg2.ts]...   1000 kbps
    `-- 240p/   -> index.m3u8 -> [seg0.ts][seg1.ts][seg2.ts]...    400 kbps

  Playback:  [720p seg0][720p seg1][1080p seg2][480p seg3][720p seg4]
                                                    ^
                            bandwidth dropped mid-stream; the player stepped
                            down rather than buffering. The viewer sees a
                            quality dip instead of a spinner.

  Every segment is a normal, immutable HTTP GET — which is why a CDN can cache
  video at all. This is the whole trick: streaming became a caching problem.

# Segments are perfectly cacheable; manifests are not (for live).
CACHE_RULES = {
    "*/segment-*.ts":  "public, max-age=31536000, immutable",  # never changes
    "*/index.m3u8":    "public, max-age=2",                    # LIVE: rewritten
                                                               # as segments are added
    "*/master.m3u8":   "public, max-age=3600",                 # rendition list is stable
}
# For video on demand, even the media manifest is immutable and cached for a
# year. For live, the media manifest changes every segment duration, so its TTL
# must be shorter than the segment length — this one setting is the difference
# between a working live stream and one stuck several minutes behind.

Protocol	Origin	Segment format	Notes
HLS	Apple	`.ts` or fMP4	Universal support; the safe default
DASH	MPEG	fMP4	Codec-agnostic; common outside Apple
CMAF	Standard	fMP4	One set of segments serves HLS and DASH — halves storage and doubles CDN hit ratio
WebRTC	W3C	RTP	Sub-second latency; not CDN-cacheable

Three design points decide the viewer's experience:

Segment duration trades latency against efficiency. Two-second segments give ~6-10 s of live latency and more requests; ten-second segments are more cache-efficient but push live latency to 30 s or more.
The ladder (the set of renditions) should be chosen from the audience's actual bandwidth distribution, not copied from a blog post.
CMAF with a single segment set is the biggest infrastructure win available: without it you store and cache two copies of every segment for HLS and DASH separately, halving your hit ratio for no benefit.

For live streaming, the additional constraint is that content is created as it is consumed, so low-latency modes (LL-HLS, LL-DASH) use partial segments and HTTP chunked transfer to get latency to 2-5 seconds. True sub-second interaction requires WebRTC, which bypasses CDN caching entirely and therefore does not scale the same way.

Key Takeaways

ABR encodes several qualities in short segments; the player adapts per segment to available bandwidth.

Segments are immutable HTTP objects, which is what makes video CDN-cacheable at all.

For live, the media manifest TTL must be shorter than the segment duration.

Segment length trades live latency against caching efficiency; CMAF lets one segment set serve both HLS and DASH.

🧪 Practice

Design a bitrate ladder for an audience that is 60% mobile, and justify each rendition.

Compute the storage for a 2-hour film at four renditions and explain what CMAF saves.

Interview: Viewers report a live stream running 45 seconds behind. What do you check? (Hint: two settings compound — how long each segment is, and how long each cache layer holds the manifest.)

← System Design