6. Scalability and Load Management

Scalability is the property that lets a system absorb more work by adding resources; load management is the set of techniques that decide where each request goes and what happens when there is more work than capacity. This chapter covers how systems grow — vertically, horizontally, and by removing state — how traffic is spread across the machines that result, how a system protects itself when demand exceeds supply, and how services find one another once there are too many instances to configure by hand.

<a id="61-scaling-approaches"></a>

6.1 Scaling Approaches

Before choosing load balancers or auto-scaling rules, you need a clear model of what "adding capacity" actually means, which parts of a system can absorb it, and why state is the thing that decides whether scaling is easy or painful.

Vertical Scaling

Vertical scaling — "scaling up" — means giving one machine more resources: more CPU cores, more RAM, faster disks, a bigger network card. The program does not change; the box underneath it gets bigger.

The reason to start here is that vertical scaling is free of distributed systems problems. One machine has one clock, one memory space, one copy of the data, and no network between its parts. Every problem that dominates later chapters — replication lag, consensus, partial failure, split brain — simply does not exist inside a single process. An engineer who reaches for a cluster before exhausting a single machine often pays a large complexity bill to solve a problem they did not yet have.

Modern hardware makes this argument stronger than most people assume. A single cloud instance can be had with 128 or more cores and multiple terabytes of RAM. A well-indexed relational database on such a machine comfortably serves tens of thousands of transactions per second — more than the majority of production systems will ever need.

The analogy is a restaurant kitchen. Vertical scaling is buying a bigger stove and hiring a faster chef for the same kitchen. It works beautifully, until the kitchen itself is full — and then no amount of money buys a bigger one.

Vertical scaling has three hard limits:

1. A physical ceiling. There is a largest instance type, and you cannot buy past it.
2. Superlinear cost. Price per unit of capacity rises steeply at the top of the range; the biggest instance often costs far more than twice the one half its size.
3. A single failure domain. One machine means one thing to lose, and restarting it for a kernel patch is a full outage. Vertical scaling improves capacity and does nothing at all for availability.

 COST PER UNIT OF CAPACITY AS THE MACHINE GROWS

  $/vCPU
    |                                        *
    |                                   *
    |                            *
    |                *
    |      *    *
    | *
    +--------------------------------------------> instance size
      2      8     32     64      96       128 vCPU

  The flat middle is the sweet spot. Past it you pay a premium for the
  privilege of not having to distribute -- sometimes worth it, but that is a
  purchase decision, not an engineering one.

# When does scaling up stop being the cheaper option?
# Illustrative on-demand prices for one instance family.
prices = {4: 0.20, 8: 0.40, 16: 0.80, 32: 1.70, 64: 3.60, 128: 9.80}

for vcpu, hourly in prices.items():
    per_vcpu = hourly / vcpu
    # Cost of the same vCPU count assembled from 8-vCPU machines instead.
    horizontal = (vcpu / 8) * prices[8]
    print(f"{vcpu:>4} vCPU  ${hourly:>5.2f}/h  ${per_vcpu:.4f}/vCPU  "
          f"horizontal equivalent ${horizontal:.2f}/h")

#    4 vCPU  $ 0.20/h  $0.0500/vCPU  horizontal equivalent $0.20/h
#    8 vCPU  $ 0.40/h  $0.0500/vCPU  horizontal equivalent $0.40/h
#   16 vCPU  $ 0.80/h  $0.0500/vCPU  horizontal equivalent $0.80/h
#   32 vCPU  $ 1.70/h  $0.0531/vCPU  horizontal equivalent $1.60/h
#   64 vCPU  $ 3.60/h  $0.0563/vCPU  horizontal equivalent $3.20/h
#  128 vCPU  $ 9.80/h  $0.0766/vCPU  horizontal equivalent $6.40/h
#
# Up to ~16 vCPU the premium is zero: scale up, it is strictly simpler.
# At 128 vCPU you pay ~53% extra to avoid distributing. That may still be
# cheaper than the engineering time to shard -- compute it, do not guess.

Vertical scaling is also the correct first move for components that are genuinely hard to distribute: a primary write node, a single-writer ledger, a graph traversal that touches everything. For those, "buy a bigger box" buys years of runway.

Key Takeaways

• Vertical scaling adds resources to one machine and introduces no distributed systems problems — it is the simplest capacity increase available.
• Modern single machines are far larger than most workloads need; exhaust this option before sharding.
• Its limits are a physical ceiling, superlinear cost near the top, and an unchanged single point of failure.
• Scaling up improves capacity, never availability.

🧪 Practice

1. Using the price table above, compute the cost premium of a 128-vCPU instance versus sixteen 8-vCPU instances, and state one reason to pay it.
2. Your database runs at 70% CPU on a 32-vCPU machine and traffic grows 20% per quarter. How many quarters of headroom does doubling the instance buy?
3. Interview: A team says they must shard Postgres because it is "at capacity". What do you ask before agreeing? (Hint: capacity of which resource — and is the bottleneck the hardware or one unindexed query?)

Horizontal Scaling

Horizontal scaling — "scaling out" — means adding more machines and spreading work across them. Instead of one server handling 10,000 requests per second, ten servers handle 1,000 each.

The reason to pay its complexity cost is that it removes both of vertical scaling's fatal limits at once. There is no ceiling: capacity becomes a number you choose. And because the work is spread over independent machines, losing one removes a fraction of capacity rather than the whole service. Horizontal scaling is the only approach that improves availability and capacity together.

Back to the kitchen: horizontal scaling is opening a second and third kitchen. Now you need a way to route orders to a kitchen (load balancing), a way to know which kitchens are open (health checks and discovery), and a shared understanding of the menu and inventory (state). Those three needs are the entire rest of this chapter.

 VERTICAL                          HORIZONTAL

   +----------------+               +------+  +------+  +------+
   |                |               | node |  | node |  | node |
   |   one large    |               +------+  +------+  +------+
   |     server     |                   ^         ^         ^
   |                |                   |         |         |
   +----------------+                   +---------+---------+
           ^                                      |
           |                                +-----------+
        clients                             |    LB     |
                                            +-----------+
                                                  ^
                                                  |
                                               clients

  Failure of the single box  = 100% outage.
  Failure of one node of 3   = 33% capacity loss, zero downtime.

The catch is that horizontal scaling does not scale everything. Amdahl's law applies directly: any serialized component — one primary database, one global lock, one sequence generator — caps the whole system no matter how many application servers you add.

# Adding app servers only helps until a shared dependency saturates.
DB_CAPACITY = 12_000        # queries/sec the single primary can serve
PER_REQUEST_QUERIES = 3     # each request issues 3 queries
APP_CAPACITY = 900          # requests/sec one app server handles

for servers in (1, 5, 10, 40):
    app_limit = servers * APP_CAPACITY
    db_limit = DB_CAPACITY / PER_REQUEST_QUERIES      # 4000 rps, FIXED
    effective = min(app_limit, db_limit)
    bound_by = "app tier" if app_limit < db_limit else "DATABASE"
    print(f"{servers:>3} servers -> {effective:>6.0f} rps  (bound by {bound_by})")

#   1 servers ->    900 rps  (bound by app tier)
#   5 servers ->   4000 rps  (bound by DATABASE)
#  10 servers ->   4000 rps  (bound by DATABASE)
#  40 servers ->   4000 rps  (bound by DATABASE)
#
# Past ~5 servers you buy capacity you cannot use. The next dollar belongs in
# caching, read replicas, or query reduction -- not in more app nodes.

Dimension	Vertical scaling	Horizontal scaling
Ceiling	Largest instance available	Effectively unbounded
Cost curve	Superlinear at the top	Roughly linear
Complexity added	None	Balancing, discovery, coordination
Availability	Unchanged (single domain)	Improves with instance count
Downtime to grow	Usually a restart	None (add an instance)
Data consistency	Trivial (one copy)	Replication and partitioning

The rule most teams converge on: scale up the hard-to-distribute tier, scale out the easy-to-distribute tier. Application servers, usually stateless, scale out. Databases scale up first and out only when forced.

Key Takeaways

• Horizontal scaling adds machines and is the only path with no capacity ceiling.
• It improves availability as a side effect: one instance failing costs a fraction of capacity.
• Its cost is coordination — balancing, discovery, replication, consistency.
• The system scales only as far as its least-scalable shared dependency.

🧪 Practice

1. Rework the code above assuming a cache absorbs 70% of queries. How many app servers become useful?
2. Draw the request path for a horizontally scaled web tier and mark every component that is still a single instance.
3. Interview: You add ten servers and throughput does not move. Name three plausible causes and the metric that distinguishes them. (Hint: look for the resource whose utilization did not change when you added capacity.)

Stateless Service Design

A service is stateless when any instance can serve any request, because everything needed to handle that request either arrives with it or is fetched from shared storage. Nothing important lives only in that instance's memory.

This is the single property that makes horizontal scaling practical. If any instance can serve any request, instances become interchangeable, and a family of hard problems dissolves at once: the load balancer routes freely, a crashed instance loses nothing, deployments replace instances one at a time, and auto-scaling adds or removes capacity without migrating anything.

Stateless does not mean the system has no state — sessions, carts, and uploads all exist. It means the state does not live in the application process. It is pushed outward into a database, a cache, or the request itself.

The analogy is a bank teller. A stateless teller keeps nothing about you at their window: your balance is in the bank's ledger, so any teller can serve you. A stateful teller has your file in their drawer — and when they go to lunch, you wait for them specifically.

 STATEFUL: instance memory holds the truth

  client A --> [ node 1 ]  session{A}, cart{A}, upload-buffer{A}
  client B --> [ node 2 ]  session{B}
                    X   node 2 dies -> B's session is gone

 STATELESS: instance memory holds only what the current request needs

  client A --> [ node 1 ] --\
  client B --> [ node 2 ] ---+--> [ session store / DB / cache ]
  client A --> [ node 3 ] --/
                    X   node 2 dies -> nothing lost, retry anywhere

// Stateful: correctness depends on the client returning to the same instance.
const carts = new Map();                       // lives in THIS process only

app.post('/cart/items', (req, res) => {
  const cart = carts.get(req.userId) ?? [];    // lost on restart, and invisible
  cart.push(req.body.item);                    // to every other instance
  carts.set(req.userId, cart);
  res.json({ items: cart.length });
});

// Stateless: the instance is a pure function of (request, shared state).
app.post('/cart/items', async (req, res) => {
  const userId = verifyJwt(req.headers.authorization).sub;  // identity travels
                                                            // with the request
  await redis.rpush(`cart:${userId}`, JSON.stringify(req.body.item));
  await redis.expire(`cart:${userId}`, 60 * 60 * 24 * 7);   // bounded lifetime
  const items = await redis.llen(`cart:${userId}`);
  res.json({ items });                          // ANY instance produces this
});

Making a service stateless is mostly a hunt for hidden state. The usual offenders:

Hidden state	Symptom when you scale out	Fix
In-memory sessions	Random logouts	External session store or signed token
Local file uploads	"File not found" on a later request	Object storage
In-process schedulers	The job runs N times, once per instance	Distributed lock or scheduler service
Local rate-limit counters	Effective limit is N x the configured one	Shared counter in Redis
In-memory caches	Inconsistent reads between instances	Acceptable if TTL-bounded, else shared
WebSocket connections	Cannot push to a user on another instance	Pub/sub fan-out between instances

The last two are often kept local on purpose. A per-instance cache with a short TTL is fine when staleness is tolerable. The test is not "does memory hold anything?" but "if this instance vanished mid-request, would anything be lost that the system cannot reconstruct?"

Key Takeaways

• Stateless means any instance can serve any request; state lives in shared storage or travels with the request.
• It is the precondition for free load balancing, painless deploys, and auto-scaling.
• Statelessness is about durability of location, not about having zero memory — bounded local caches are fine.
• Audit for hidden state: sessions, local files, in-process timers, local counters, long-lived connections.

🧪 Practice

1. Take the stateful cart handler above and list every failure that appears once a second instance sits behind a round-robin load balancer.
2. A service writes uploaded images to /tmp and serves them on a later request. Describe two ways to make it stateless and the trade-off between them.
3. Interview: Your service runs a nightly report from an in-process cron. What breaks at three instances, and how do you fix it? (Hint: you need exactly one execution — what primitive produces a single winner across machines?)

Session Management and Sticky Sessions

Sessions are the most common form of state in a web system, so how you store them largely determines how freely you can scale. There are three viable designs, one of which is a compromise worth understanding well enough to avoid.

1. Sticky sessions (session affinity). Keep sessions in instance memory and have the load balancer send each client back to the same instance every time, usually via a cookie or a hash of the client IP.

2. External session store. Keep sessions in a shared store (Redis, Memcached, a database). Any instance serves any request after one extra hop.

3. Client-side sessions (tokens). Put the session data in a signed token held by the client. No server-side lookup at all.

 STICKY                       SHARED STORE              CLIENT-SIDE TOKEN

 client --cookie:node2--+     client --> LB --> any     client --JWT--> any
                        |                       |                        |
      LB pins to node2  |                       v                    verify
                        v                 +-----------+              signature
                  [ node 2 mem ]          |   Redis   |              locally
                        X  dies           +-----------+
                  session lost           survives node loss       no lookup

Sticky sessions need no code change, but they undercut most of the benefits of horizontal scaling:

• Uneven load. Long-lived clients pile onto whichever instance they landed on; a newly added instance receives only new sessions and stays idle for hours.
• Failure is user-visible. Losing an instance logs out everyone pinned to it.
• Deploys hurt. Every rolling restart drops sessions unless you drain slowly.
• Auto-scaling is hobbled. You cannot remove an instance without affecting the users pinned to it.

// Client-side session: state travels in a signed token.
const token = jwt.sign(
  { sub: user.id, roles: user.roles, tier: 'premium' },  // small, non-secret
  process.env.JWT_SECRET,
  { expiresIn: '15m' }        // SHORT: a token cannot be un-issued, so the
);                            // expiry IS your revocation window

const claims = jwt.verify(token, process.env.JWT_SECRET);  // no network call

// Server-side session: one extra hop, but revocable instantly.
await redis.setex(`sess:${sid}`, 1800, JSON.stringify(sessionData));
await redis.del(`sess:${sid}`);   // logout/ban takes effect on the next request

Approach	Scale-out	Revocation	Failure impact	Extra latency	Size limit
Sticky	Poor	Immediate	Pinned users lose their sessions	None	Memory-bound
Shared store	Good	Immediate	None (store is HA)	~1 ms lookup	Large
Signed token	Excellent	Only at expiry	None	None	~4-8 KB cookie

The common production answer is a hybrid: a short-lived signed access token for cheap, lookup-free authentication, plus a server-side refresh token so sessions stay revocable. That gives the scaling properties of tokens with the control of a session store.

Sticky sessions remain legitimately useful for stateful protocols such as WebSockets, in-progress multipart uploads, and per-instance caches warmed for one tenant. Use affinity as a performance optimization the system can survive losing — never as a correctness requirement.

Key Takeaways

• Sticky sessions require no code change but reintroduce the problems statelessness solves: uneven load, session loss, painful deploys.
• A shared session store scales well and keeps sessions revocable, at the cost of a lookup per request.
• Signed tokens remove the lookup entirely but cannot be revoked before expiry — so keep expiry short.
• Prefer a short access token plus a server-side refresh token; treat affinity as an optimization, not a requirement.

🧪 Practice

1. A JWT is issued with a 24-hour expiry and the user is banned an hour later. Describe exactly what that user can still do, and for how long.
2. Your load balancer uses IP-hash affinity and one corporate customer behind a single NAT gateway sends 30% of traffic. What happens, and what do you change?
3. Interview: Design session handling for a service that must support instant logout across devices at 100k requests/second. (Hint: the fast path can stay lookup-free if only the rare, slow-changing case needs a lookup.)

Auto-Scaling Policies

Auto-scaling adjusts the number of instances automatically in response to load. The promise is obvious — pay for what you use, absorb spikes without paging anyone — but a naive policy oscillates, reacts too late, or scales the wrong tier entirely.

The core difficulty is that scaling is not instantaneous. Between a metric crossing a threshold and a new instance serving traffic sits a chain of delays: metric aggregation, alarm evaluation, provisioning, boot, application startup, cache warming, and health-check passes. The total is often two to five minutes. Auto-scaling is therefore a control system with significant lag, and control systems with lag oscillate unless they are damped.

 THE SCALING LAG

  load  |          _____________________
        |         /                     \
        |        /                       \
        |_______/                         \______
        +----------------------------------------------> time
               ^      ^       ^          ^
               |      |       |          |
            spike   metric   alarm    instance
            starts  scraped  fires    in service
               |<-60s->|<-30s->|<---- 120s ---->|
               |<--------- ~3.5 min of degraded service ------->|

  You cannot remove this lag; you can only start earlier (predictive or
  scheduled scaling) or absorb it (headroom, queues, load shedding).

There are four policy types, and mature systems combine them:

Policy type	Trigger	Best for
Target tracking	Keep a metric at a set point	The default for steady workloads
Step scaling	Tiered thresholds, bigger steps for bigger breaches	Sharp, unpredictable spikes
Scheduled	Time of day or day of week	Known patterns (business hours)
Predictive	Forecast from history	Strong daily or weekly seasonality

Choosing the metric matters more than choosing the policy. CPU is the default and is often wrong: an I/O-bound service saturates on connections or queue depth while CPU sits at 30%. The right metric is the one that correlates with user-visible latency — usually backlog per consumer for a queue worker, and concurrent requests per instance for a request service.

# Target tracking on a demand-side metric, with damping.
scaling_policy:
  metric: requests_in_flight_per_instance   # not CPU: this tracks latency
  target: 25                                # load tests: p99 degrades past ~35
  min_instances: 4                          # survive an AZ loss at minimum
  max_instances: 60                         # a cost guardrail AND a blast-radius
                                            # limit on runaway scale-out
  scale_out:
    cooldown_seconds: 60                    # short: being late is expensive
    max_step: 100%                          # allow doubling under a real spike
  scale_in:
    cooldown_seconds: 300                   # long: being early causes flapping
    max_step: 10%                           # remove capacity slowly

# Why asymmetric cooldowns matter: the cost of being wrong is asymmetric.
#
#   Scaling out unnecessarily -> you pay for one extra instance for 5 minutes.
#   Scaling in unnecessarily  -> you drop below capacity, latency spikes, and
#                                recovery costs the FULL scaling lag.
#
# Therefore: scale out fast and aggressively, scale in slowly and timidly.
import math

def desired_instances(current_rps, rps_per_instance, headroom=0.30, azs=3):
    """Capacity sizing with explicit headroom to cover the scaling lag."""
    raw = current_rps / rps_per_instance
    with_headroom = raw / (1 - headroom)          # never plan to run at 100%
    # Round up to a multiple of the AZ count so each AZ carries an equal share.
    return max(azs, math.ceil(with_headroom / azs) * azs)

print(desired_instances(9_000, 400))   # 33 -> the headroom IS the lag budget

Two failure modes deserve names. Flapping is repeated scale-out/scale-in caused by thresholds that sit too close together or cooldowns that are too short; fix it with hysteresis (different thresholds for out and in) and a longer scale-in cooldown. Scaling into a bottleneck is worse: the app tier scales out, each new instance opens a database connection pool, and the database falls over. Always ask what the new instances will consume, and cap max_instances at what the dependencies can survive.

Key Takeaways

• Auto-scaling has an unavoidable lag of minutes; headroom, schedules, or prediction are the only ways to cover it.
• Scale on a metric that tracks user-visible latency (in-flight requests, queue backlog), not reflexively on CPU.
• Make cooldowns asymmetric: out fast, in slow — the costs of error are not symmetric.
• max_instances is a blast-radius control for downstream dependencies, not only a budget cap.

🧪 Practice

1. A service scales out at 70% CPU and in at 65%. Describe the oscillation this produces and give two fixes.
2. Write a scaling policy for a video-transcoding worker pool. Which metric do you target, and why is it not CPU?
3. Interview: Traffic quadruples in 30 seconds when a marketing email lands, and auto-scaling takes 4 minutes. How do you serve those 4 minutes? (Hint: auto-scaling is not the only lever — what can absorb or refuse load meanwhile?)

Scaling Reads vs. Scaling Writes

Reads and writes are not symmetric, and treating them as one workload is a common design error. Most systems are read-dominated — often 100:1 or more — and reads have a property writes do not: they can be served from any copy of the data.

That difference is everything. To serve more reads you add copies: caches, read replicas, CDN nodes, materialized views. Each copy is independent, so read capacity scales close to linearly. Writes must be applied to the authoritative copy in a defined order, so adding machines does not help unless you split the data itself into independent partitions.

The analogy is a library. Adding photocopies of a popular book lets a hundred people read it at once. But if the book must be edited, every copy has to agree on the new text — and now the copies are a liability, not an asset.

 SCALING READS: add copies                SCALING WRITES: split the data

              [ primary ]                    [ shard A ]  users a-h
                   |  replication            [ shard B ]  users i-p
        +----------+----------+              [ shard C ]  users q-z
        v          v          v
   [replica]  [replica]  [replica]           Each shard is an independent
        ^          ^          ^              write path, so capacity scales
        |          |          |              with shard count -- but a write
      reads      reads      reads            spanning two shards now needs a
                                             distributed transaction.
   Cheap and near-linear, but STALE
   by the replication lag.

The read-scaling ladder, cheapest and most effective first:

Technique	Read capacity gain	Cost
Client/CDN cache	Enormous	Staleness bounded by TTL
Application cache	10-100x	Invalidation complexity
Read replicas	Linear in replica count	Replication lag; read-your-writes bugs
Materialized views	Large for specific queries	Refresh cost and staleness
Denormalization	Removes joins	Write amplification, update anomalies

The write-scaling ladder is shorter, and each rung costs more:

Technique	Write capacity gain	Cost
Batching / coalescing	5-50x	Added latency, partial-failure handling
Write-behind buffering	Large	Durability risk while buffered
Async queue absorption	Smooths peaks	Eventual consistency, backlog management
Sharding	Linear in shards	Cross-shard queries and transactions, rebalancing

# Read/write asymmetry decides where the money goes.
READ_QPS, WRITE_QPS = 48_000, 600          # a typical ~80:1 social feed
REPLICA_CAPACITY, PRIMARY_CAPACITY = 8_000, 12_000

cache_hit_rate = 0.92                      # a modest, realistic cache
reads_to_db = READ_QPS * (1 - cache_hit_rate)
replicas_needed = -(-reads_to_db // REPLICA_CAPACITY)      # ceil division

print(f"reads reaching the DB: {reads_to_db:.0f}/s -> {replicas_needed:.0f} replica(s)")
print(f"writes: {WRITE_QPS}/s -> {WRITE_QPS / PRIMARY_CAPACITY:.1%} of one primary")

# reads reaching the DB: 3840/s -> 1 replica(s)
# writes: 600/s -> 5.0% of one primary
#
# The cache did more for read capacity than any number of replicas would have,
# and sharding this system would be wasted complexity: the primary is at 5%.
# Shard when WRITES run out of room, not reads.

The sequencing rule: cache reads, batch writes, shard last. Reach for sharding only when a single primary genuinely cannot absorb the write rate or hold the working set, because sharding is the one step that permanently changes which queries and transactions your application is allowed to express.

Key Takeaways

• Reads scale by adding copies (caches, replicas, views); writes scale only by splitting data or reducing write volume.
• Most systems are read-dominated, so caching usually beats every other lever on cost-effectiveness.
• Every read-scaling technique trades freshness for capacity — set the staleness budget explicitly.
• Shard for write volume or dataset size, never for read volume.

🧪 Practice

1. Recompute the example with a 60% cache hit rate. How many replicas are needed, and what does that say about cache tuning versus replica count?
2. Give a concrete example of a read that cannot be served from a replica, and explain what you would do instead.
3. Interview: An e-commerce system handles 50k product views/sec and 200 orders/sec. Which side do you scale first, and how? (Hint: compute what fraction of a single primary the write path actually uses.)

<a id="62-load-balancing"></a>

6.2 Load Balancing

Once there is more than one instance, something must decide which instance each request goes to. That decision — made at a layer, by an algorithm, over a set of backends known to be healthy — is load balancing.

Layer 4 vs. Layer 7 Load Balancing

The first decision about a load balancer is how deeply it inspects traffic, and the answer is expressed in OSI layers. A Layer 4 balancer works with TCP/UDP connections: it sees IP addresses and ports and forwards packets or connections without reading their contents. A Layer 7 balancer terminates the connection, parses the application protocol (usually HTTP), and makes decisions using URLs, headers, cookies, and methods.

The trade-off is understanding versus cost. L4 is a postal sorter reading only the address on the envelope: fast, cheap, protocol-agnostic, and completely unable to help you if routing depends on what is inside. L7 opens the envelope and reads the letter: it can route /api/v2/* to one pool and /images/* to another, retry a failed request on a different backend, and compress responses — but it must do TLS termination and protocol parsing for every request.

 LAYER 4                                LAYER 7

  client                                 client
     |  TCP connection                      |  TCP + TLS terminated HERE
     v                                      v
  [ L4 LB ]  reads: src/dst IP:port      [ L7 LB ]  reads: method, path,
     |       forwards the connection         |      headers, cookies, body
     |       (often NAT or DSR)              |      opens a SEPARATE
     v                                       v      connection to the backend
  [ backend ]  sees the raw stream       [ backend ]  sees a NEW request
                                                      (client IP only via
                                                       X-Forwarded-For)

A crucial operational consequence: because an L7 balancer opens its own connection to the backend, the backend no longer sees the client's IP address. The real client IP must be carried in a header (X-Forwarded-For, or the standardized Forwarded), and any code that logs, rate-limits, or geolocates by IP must read that header — while trusting it only when it comes from your own proxy, since clients can forge it.

Property	Layer 4	Layer 7
Inspects	IP, port	URL, headers, cookies, method
Throughput	Very high (millions of pps)	Lower (parsing + TLS per request)
Added latency	Microseconds	Sub-millisecond to milliseconds
Protocol support	Any TCP/UDP	Specific (HTTP, gRPC, WebSocket)
Content-based routing	No	Yes
Retry a failed request	No (connection-level only)	Yes
TLS termination	Passthrough	Usually terminates
Sees client IP	Yes	Only via forwarded headers
Typical products	AWS NLB, IPVS, MetalLB	NGINX, Envoy, HAProxy, AWS ALB

# Layer 7: routing decisions that require reading the request.
upstream api_v2   { server 10.0.1.10:8080; server 10.0.1.11:8080; }
upstream images   { server 10.0.2.10:8080; }

server {
    listen 443 ssl;

    location /api/v2/ {
        proxy_pass http://api_v2;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;  # backend
        proxy_set_header X-Forwarded-Proto $scheme;   # loses the client IP
        proxy_next_upstream error timeout http_502;   # L7 can RETRY the request
    }

    location /images/ {
        proxy_pass http://images;
        proxy_cache image_cache;                      # only possible at L7
    }
}

The mature pattern is not to choose but to layer them: an L4 balancer at the edge absorbing raw connection volume and spreading it across a fleet of L7 proxies, which then make the smart routing decisions. This is how most large sites are built — L4 for scale, L7 for intelligence.

Key Takeaways

• L4 balances connections using IP and port; L7 balances requests using URL, headers, and cookies.
• L7 costs more per request but enables content routing, request retries, caching, and header manipulation.
• L7 hides the client IP from backends; carry it in X-Forwarded-For and trust the header only from your own proxies.
• Large systems commonly use L4 at the edge in front of an L7 tier.

🧪 Practice

1. List three routing rules that are impossible at L4 and explain why.
2. A backend logs every client as 10.0.0.5. Diagnose the cause and give the fix, including the security caveat.
3. Interview: You must balance a custom binary protocol over TCP at two million connections. Which layer, and what do you give up? (Hint: which layer needs to understand the protocol to work at all?)

Round Robin and Weighted Algorithms

Round robin is the simplest balancing algorithm: keep a pointer, hand each new request to the next backend, wrap around at the end. It requires no state about backends, no measurement, and no coordination — which is exactly why it is the default nearly everywhere.

Round robin is correct when two assumptions hold: backends are identical, and requests cost roughly the same. When either fails, it distributes requests evenly while distributing load unevenly, which is the thing you actually care about.

Weighted round robin repairs the first assumption. Each backend gets a weight proportional to its capacity, and receives that share of requests. This is what you use during a migration to a larger instance type, or when running a mixed fleet.

 ROUND ROBIN (equal)              WEIGHTED ROUND ROBIN (3:2:1)

 req 1 -> A                       req 1 -> A      A: weight 3 (16 vCPU)
 req 2 -> B                       req 2 -> A      B: weight 2 (8 vCPU)
 req 3 -> C                       req 3 -> A      C: weight 1 (4 vCPU)
 req 4 -> A                       req 4 -> B
 req 5 -> B                       req 5 -> B
 req 6 -> C                       req 6 -> C      -> then repeat

 Smooth WRR interleaves instead of bursting: A B A C A B -- the same
 ratio, but no backend receives three requests back to back.

# Smooth weighted round robin (the algorithm NGINX and Envoy use).
# Naive WRR sends w_i requests in a row to each backend, which bursts.
# This version interleaves while preserving the exact ratio.
class SmoothWRR:
    def __init__(self, backends):           # backends: {name: weight}
        self.weights = dict(backends)
        self.current = {name: 0 for name in backends}
        self.total = sum(backends.values())

    def next(self):
        for name, w in self.weights.items():
            self.current[name] += w         # everyone accumulates their weight
        best = max(self.current, key=self.current.get)   # highest credit wins
        self.current[best] -= self.total    # winner pays the full total back
        return best

lb = SmoothWRR({"A": 3, "B": 2, "C": 1})
print([lb.next() for _ in range(6)])
# ['A', 'B', 'A', 'C', 'A', 'B']   -> exact 3:2:1 ratio, no bursts

Round robin's real weakness is that it is open-loop: it never observes the consequences of its own decisions. If backend B is garbage collecting, swapping, or stuck behind a slow dependency, round robin keeps feeding it one request in three anyway. Worse, a backend that fails fast — instantly returning 500 — finishes requests quickly and therefore looks perfectly healthy to any algorithm that counts requests rather than outcomes.

Variant	State needed	Handles uneven backends	Handles uneven requests
Round robin	A counter	No	No
Weighted round robin	Static weights	Yes	No
Random	None	No	No (but no sync bursts)
Random with two choices	Live conn counts	Partly	Yes

That last row deserves a mention because it is remarkably effective for its cost: pick two backends at random, send the request to whichever has fewer in-flight requests. It needs almost no coordination yet removes nearly all of the imbalance of pure random selection — a well-known result usually called "the power of two choices."

Key Takeaways

• Round robin is stateless and fair in request count, which equals fairness in load only when backends and requests are uniform.
• Weighted round robin handles heterogeneous fleets; use smooth WRR to avoid bursting onto one backend.
• Round robin is open-loop: it cannot notice that a backend has become slow.
• Two random choices plus a load comparison gets most of the benefit of least-connections at almost no coordination cost.

🧪 Practice

1. Trace smooth WRR by hand for weights A=5, B=1 over eight requests.
2. Half your fleet is 16-vCPU and half is 4-vCPU under plain round robin. Describe the observable symptom and compute the correct weights.
3. Interview: One backend returns errors in 5 ms while healthy ones take 200 ms. What happens under round robin versus least-connections? (Hint: which backend has the fewest in-flight requests when it fails instantly?)

Least Connections and Least Response Time

Least connections is a closed-loop algorithm: instead of assuming backends are equivalent, it measures how many requests each is currently handling and sends the next request to the least busy one. It is the natural answer when request costs vary widely — one request returns a cached value in 2 ms while another runs a report for 4 seconds.

The insight behind it is Little's Law. For a backend, concurrency = arrival rate x service time. In-flight connection count is therefore a live signal that folds in both how much work a backend has been given and how slowly it is handling it. A backend that is slow for any reason — GC pause, cold cache, noisy neighbour, a slow dependency — accumulates in-flight requests, and least connections automatically routes around it without anyone configuring anything.

 SAME MOMENT, TWO ALGORITHMS

  backend   in-flight   avg latency
  --------  ---------   -----------
     A          2          20 ms
     B         31         340 ms     <- GC pause, slow dependency, whatever
     C          3          25 ms

  Round robin      -> next request goes to B (it is simply B's turn).
  Least conns      -> next request goes to A (2 < 3 < 31).
  Least resp time  -> A as well, and it would keep avoiding B even if B's
                      queue drained while its latency stayed high.

Least response time goes one step further and ranks backends by a blend of in-flight count and observed latency (usually an exponentially weighted moving average). It catches a case least connections misses: a backend that is slow but lightly loaded, for example because it is serving a smaller share of a misconfigured weighted pool.

# Peak EWMA: the scoring most modern proxies use (Envoy, Linkerd, Finagle).
import time

class PeakEwma:
    def __init__(self, decay_seconds=10.0):
        self.rtt = {}          # backend -> smoothed latency estimate (ms)
        self.inflight = {}     # backend -> current in-flight request count
        self.decay = decay_seconds

    def observe(self, backend, latency_ms):
        prev = self.rtt.get(backend, latency_ms)
        # Weight recent samples more heavily; a spike is reflected immediately,
        # while recovery is only believed gradually.
        alpha = 0.2
        self.rtt[backend] = max(latency_ms, prev * (1 - alpha) + latency_ms * alpha)

    def choose(self, backends):
        # Cost estimate = what this request will likely wait for.
        # Adding 1 accounts for the request we are about to send.
        return min(backends,
                   key=lambda b: self.rtt.get(b, 1.0) * (self.inflight.get(b, 0) + 1))

lb = PeakEwma()
lb.inflight = {"A": 2, "B": 31, "C": 3}
lb.rtt      = {"A": 20, "B": 340, "C": 25}
print(lb.choose(["A", "B", "C"]))   # A -> score 60 vs B 10880 vs C 100

Two cautions apply in practice.

First, least connections needs an accurate in-flight count, which is easy for a single load balancer and hard for a fleet of them. Ten balancers each seeing only their own connections have ten partial views, and all ten may simultaneously conclude that the same idle backend is the least busy — a stampede onto the newest instance. Randomized choice among the best few, or "two random choices," fixes this cheaply.

Second, the fail-fast trap from the previous topic is real here too: a backend returning instant errors keeps a near-zero in-flight count and looks maximally attractive. Least response time with error-aware scoring — counting failures as very expensive rather than very fast — is the defense.

Algorithm	Signal used	Good at	Weak at
Round robin	Nothing	Uniform workloads	Variable request cost
Least connections	In-flight count	Variable request cost	Multi-LB view splits; fail-fast nodes
Least response time	Latency + in-flight	Slow-but-idle backends	Needs tuning; reacts to noise
Two random choices	In-flight of two picks	Scale-out balancing	Slightly worse than a perfect view

Key Takeaways

• Least connections measures actual backend load and adapts automatically to uneven request costs and slow backends.
• It works because in-flight count is Little's Law made observable.
• Least response time additionally catches backends that are slow without being busy.
• Both need error-aware scoring, or a fast-failing backend becomes the most attractive one.

🧪 Practice

1. Compute the peak-EWMA scores for the table above after B's queue drains to 4 but its latency stays at 340 ms. Which backend is chosen?
2. Explain why ten independent load balancers using least connections can stampede a newly added instance, and give a fix.
3. Interview: When would you deliberately choose round robin over least connections? (Hint: think about what the extra state costs, and when the assumption it removes was true anyway.)

Hash-Based Routing

Sometimes you do not want an even spread — you want the same key to reach the same backend every time. That is hash-based routing: compute a hash of some request attribute (client IP, session ID, user ID, cache key) and use it to pick the backend deterministically.

The reason to want this is locality. If every request for user:42 lands on the same instance, that instance's local cache stays warm, its open connections stay reusable, and any per-user in-memory state stays valid. Hash routing turns a fleet of interchangeable instances into a de facto partitioned system without a coordinator.

The naive implementation is backend = hash(key) % N — and it has a catastrophic flaw. Change N (add a server, lose a server) and almost every key maps somewhere new. For a cache tier, that is a near-total cache flush at the exact moment the system is already under stress.

 MODULO HASHING: one node change remaps almost everything

  N = 4:  key "user:42" -> hash % 4 = 2 -> node C
  N = 5:  key "user:42" -> hash % 5 = 1 -> node B     REMAPPED
          ~80% of all keys move.

 CONSISTENT HASHING: a ring; only the departed node's keys move

              node A
          .-----*-----.
        /     k1       \        Each key walks clockwise to the first
   node D                node B node it meets.
        \      k2      /
          '-----*-----'         Remove node C: only the keys between B
              node C            and C move -- and they move to D alone.
                                Everything else is untouched.

Consistent hashing places both nodes and keys on a hash ring and assigns each key to the next node clockwise. Adding or removing one node relocates only 1/N of the keys on average. Real implementations add virtual nodes: each physical node is placed at many ring positions, which smooths out the uneven arc lengths a handful of random placements would produce.

import hashlib, bisect

class ConsistentHashRing:
    def __init__(self, nodes, vnodes=150):
        # vnodes: more replicas per node -> smoother distribution.
        # 150 is a common default; ~1% standard deviation across nodes.
        self.vnodes, self.ring, self.keys = vnodes, {}, []
        for node in nodes:
            self.add(node)

    def _hash(self, value):
        return int(hashlib.md5(value.encode()).hexdigest()[:8], 16)

    def add(self, node):
        for i in range(self.vnodes):
            h = self._hash(f"{node}#{i}")     # many ring positions per node
            self.ring[h] = node
            bisect.insort(self.keys, h)       # keep positions sorted for search

    def remove(self, node):
        for i in range(self.vnodes):
            h = self._hash(f"{node}#{i}")
            del self.ring[h]
            self.keys.remove(h)

    def get(self, key):
        h = self._hash(key)
        idx = bisect.bisect(self.keys, h) % len(self.keys)   # walk clockwise,
        return self.ring[self.keys[idx]]                     # wrapping at the end

ring = ConsistentHashRing(["A", "B", "C", "D"])
before = {f"user:{i}": ring.get(f"user:{i}") for i in range(10_000)}
ring.remove("C")
moved = sum(1 for k, v in before.items() if ring.get(k) != v)
print(f"{moved / 100:.1f}% of keys moved")   # ~25% -- exactly C's share,
                                             # versus ~80% under modulo

A refinement worth knowing is bounded-load consistent hashing, which caps how much load any single node may take: if a key's target node is already above its cap, the key overflows to the next node on the ring. This keeps a single viral key from melting one node while the rest idle — the hot-key problem that plain consistent hashing does nothing about.

Hash routing has one structural downside: it prioritizes affinity over balancing. If your key distribution is skewed, your load will be too. Use it where locality pays for itself (cache tiers, sharded stateful services, per-tenant workers) and use a load-aware algorithm everywhere else.

Key Takeaways

• Hash routing sends the same key to the same backend, preserving cache warmth and per-key state.
• hash % N remaps almost all keys when N changes; consistent hashing moves only about 1/N.
• Virtual nodes are what make consistent hashing distribute evenly in practice.
• Hash routing trades load balance for affinity, so hot keys become hot nodes unless you bound per-node load.

🧪 Practice

1. Compute the fraction of keys remapped when going from 4 to 5 nodes under modulo hashing versus consistent hashing.
2. Modify the ring class to return the top two nodes for a key, so a client can fail over without a full remap.
3. Interview: A single celebrity user's key overloads one cache node. The ring is working exactly as designed. What do you change? (Hint: either the key stops being one key, or the node stops being allowed to take it all.)

Health Checks and Draining

A load balancer is only as good as its picture of which backends can serve traffic. Health checking builds that picture; draining is how a backend leaves the pool without dropping the requests it is already handling.

There are two kinds of health check and you want both. Active checks are probes the balancer sends on a schedule — request a /healthz endpoint every few seconds, mark the backend down after N consecutive failures, mark it up again after M consecutive successes. Passive checks (outlier detection) observe real traffic: if a backend returns a burst of 5xx or times out repeatedly, eject it temporarily. Active checks catch a dead process; passive checks catch a backend that answers probes fine but fails real requests.

The design of the health endpoint itself is where teams most often go wrong. The distinction that matters:

Probe type	Question it answers	If it fails
Liveness	Is this process wedged?	Restart the instance
Readiness	Can it serve traffic right now?	Remove from the LB pool only
Startup	Has it finished initializing?	Wait; do not restart yet

 THE CASCADING HEALTH CHECK ANTI-PATTERN

  Database has a brief hiccup
        |
        v
  /healthz on EVERY instance queries the DB -> every check fails
        |
        v
  LB marks the ENTIRE fleet unhealthy -> 100% of traffic rejected
        |
        v
  DB recovers, but nothing is in the pool; instances restart, cold caches,
  thundering herd back onto the DB -> the hiccup became a 20-minute outage.

  Rule: a health check must report on the HEALTH OF THIS INSTANCE, not on
  the health of its dependencies. If everything is equally sick, taking
  everything out of rotation converts degradation into an outage.

// Liveness: cheap, local, no dependencies. Failing this means "restart me".
app.get('/livez', (req, res) => res.status(200).send('ok'));

// Readiness: local capacity only. Note what is deliberately NOT checked.
app.get('/readyz', (req, res) => {
  if (shuttingDown)          return res.status(503).send('draining');
  if (!warmupComplete)       return res.status(503).send('warming');
  if (inFlight > MAX_INFLIGHT) return res.status(503).send('saturated');

  // NOT checked here: database reachability, downstream APIs.
  // If the DB is down, every instance is equally affected; removing them all
  // helps nobody and prevents cached or degraded responses from being served.
  res.status(200).send('ok');
});

// Draining: stop accepting NEW work, finish what is in flight.
process.on('SIGTERM', async () => {
  shuttingDown = true;                       // /readyz now fails
  await sleep(15_000);                       // KEY: wait for the LB to notice
                                             // (check interval x threshold)
  server.close(() => process.exit(0));       // then finish in-flight requests
});

That sleep before server.close() is the detail almost everyone misses. The load balancer learns a backend is unready only on its next few probes. If the process stops listening the instant it receives SIGTERM, every request the balancer sends during that window is refused. The correct order is: fail readiness → wait longer than the balancer's detection window → stop accepting new connections → finish in-flight work → exit.

Tuning the probe parameters is a trade-off between detection speed and stability. Aggressive settings (1 s interval, 1 failure) eject healthy backends during a GC pause; slow settings (30 s interval, 5 failures) leave a dead backend receiving traffic for over two minutes. A common starting point is a 5-second interval with a 2-second timeout, 3 failures to eject and 2 successes to restore, which detects a genuine failure in roughly 15 seconds without reacting to transient blips.

Key Takeaways

• Use active probes to find dead processes and passive outlier detection to find backends that fail real requests.
• Separate liveness (restart me) from readiness (stop sending me traffic).
• Never fail a health check because a shared dependency is down — it converts partial degradation into a total outage.
• Draining must fail readiness first and wait out the balancer's detection window before the process stops listening.

🧪 Practice

1. With a 5-second interval and 3 failures to eject, compute the worst-case time a dead backend keeps receiving traffic.
2. Write a readiness endpoint for a service that needs a warm local cache before it can meet its latency target.
3. Interview: A deploy causes a burst of 502s even though the new instances are healthy. What is the likely cause? (Hint: look at what the old instances did the moment they received SIGTERM.)

Global Server Load Balancing

Everything so far balances traffic inside one region. Global server load balancing (GSLB) decides which region a user reaches in the first place.

There are two reasons to care. The first is latency: the speed of light imposes roughly 100 ms of round trip between continents, and no amount of backend optimization recovers it — sending a European user to a European region is the single largest latency win available. The second is availability: if a whole region fails, something must move its users elsewhere, and that decision cannot be made by a load balancer inside the failed region.

GSLB is implemented in one of three ways, and they differ mostly in how fast they can fail over:

Mechanism	How it routes	Failover speed	Client IP visibility
DNS-based	Returns different IPs per resolver	Slow (TTL + caching)	Only the resolver's
Anycast	Same IP announced from many sites; BGP picks	Seconds (route withdrawal)	Full
HTTP redirect	302 to a regional hostname	Immediate	Full

 DNS-BASED GSLB WITH HEALTH-AWARE ANSWERS

   user in Frankfurt                 user in Oregon
          |                                 |
          v                                 v
   resolver asks: app.example.com?   resolver asks: app.example.com?
          |                                 |
          v                                 v
   +--------------------- GSLB authority ---------------------+
   | geo/latency table + per-region health from real probes   |
   +----------------------------------------------------------+
          |                                 |
          v                                 v
      52.x.x.x (eu-central)             34.x.x.x (us-west)
          |                                 |
      [ regional LB ]                   [ regional LB ]
          |                                 |
      [ instances ]                     [ instances ]

  If eu-central fails health checks, the authority stops handing out its IP.
  Existing clients keep using the cached answer until the TTL expires --
  which is why GSLB TTLs are short (30-60 s) and why DNS alone is never a
  complete failover story.

The routing policies available at the global layer are worth distinguishing, because they answer different questions:

• Geolocation: route by where the user is. Predictable, and required for data-residency rules ("EU users must be served from the EU").
• Latency-based: route by measured network latency, which is not the same as geographic distance — network topology often makes a farther region faster.
• Weighted: send a fixed percentage to each region. The mechanism behind gradual migrations and multi-region canaries.
• Failover: an ordered list; secondary regions receive traffic only when the primary is unhealthy.

# Sizing multi-region capacity: the question GSLB forces you to answer.
regions = {"us-east": 40_000, "eu-central": 25_000, "ap-south": 15_000}  # rps
total = sum(regions.values())

for lost in regions:
    survivors = {r: v for r, v in regions.items() if r != lost}
    # Assume the failed region's traffic is spread across survivors in
    # proportion to their normal share.
    absorbed = {r: v + regions[lost] * v / sum(survivors.values())
                for r, v in survivors.items()}
    peak = max(absorbed.values())
    print(f"lose {lost:<11} -> busiest survivor must serve {peak:,.0f} rps "
          f"({peak / regions[max(survivors, key=survivors.get)]:.2f}x normal)")

# lose us-east     -> busiest survivor must serve 50,000 rps (2.00x normal)
# lose eu-central  -> busiest survivor must serve 58,182 rps (1.45x normal)
# lose ap-south    -> busiest survivor must serve 49,231 rps (1.23x normal)
#
# "Active-active across three regions" means every region must be able to run
# at up to 2x its normal load -- or you must accept shedding during failover.

That calculation is the real content of a multi-region strategy. Regional failover only works if the surviving regions have somewhere to put the traffic; otherwise failover simply moves the outage.

Key Takeaways

• GSLB chooses a region; regional load balancers choose an instance.
• DNS-based GSLB is universal but fails over only as fast as TTLs and resolver caches allow; anycast fails over at BGP speed.
• Routing policies answer different questions: geolocation for residency, latency for performance, weighted for migrations, failover for DR.
• Multi-region failover requires pre-provisioned headroom in the surviving regions — compute it explicitly.

🧪 Practice

1. With a 60-second DNS TTL, what is the realistic worst case for a client to stop using a failed region, and why is it longer than 60 seconds?
2. Design GSLB policy for a service that must keep EU user data in the EU but wants the lowest latency for everyone else.
3. Interview: You run active-active in two regions, each at 65% utilization. What happens when one fails? (Hint: add the two numbers before answering.)

Anycast Routing

Anycast is a routing trick with a strange premise: announce the same IP address from many locations at once, and let the internet's own routing protocol deliver each packet to whichever location is nearest in network terms.

Normally an IP address identifies one destination (unicast). With anycast, dozens or hundreds of sites announce the same prefix via BGP, and every router along the way independently picks the best path it knows. A user in Tokyo and a user in São Paulo type the same IP and land on different continents, with no DNS trickery, no redirect, and no per-user decision anywhere.

The analogy is a chain of stores that all share one phone number, where the phone network connects you to the nearest branch automatically. Nobody looks up a branch number; the network's own routing does the selection.

 ANYCAST: one prefix, many origins

     Tokyo user                   São Paulo user
         |                              |
   [ ISP routers pick the shortest BGP path they see ]
         |                              |
         v                              v
   +-----------+                  +-----------+
   | Tokyo PoP |                  | GRU PoP   |     Both announce 198.51.100.0/24
   +-----------+                  +-----------+

  A PoP fails -> it withdraws the BGP announcement -> routers converge on
  the next-nearest PoP within seconds. No DNS TTL, no client change.

Anycast has three properties that make it the backbone of DNS root servers, public resolvers, and every large CDN:

1. Automatic proximity routing with no lookup step and no client cooperation.
2. Fast failover. Withdrawing a route reconverges in seconds, not TTLs.
3. DDoS absorption. An attack from many sources is delivered to many PoPs by the routing system itself, splitting the flood across the whole footprint instead of concentrating it on one target.

The catch is that BGP chooses paths by policy and AS-path length, not by latency. "Nearest" means nearest in routing terms, which is frequently not the nearest geographically or the fastest in milliseconds. Anycast gets you close; it does not get you optimal.

The subtler catch is statefulness. BGP can reconverge mid-connection and send subsequent packets of the same TCP flow to a different PoP, which has no knowledge of that connection and resets it. In practice this is rare and short-lived, and it is why anycast is a natural fit for:

Workload	Anycast suitability
DNS (single-packet UDP)	Ideal — no connection state to lose
CDN edge / TLS termination	Very good — flows are short, resets are rare
Long-lived WebSockets	Risky — a reroute drops the connection
Multi-hour uploads	Poor — use unicast regional endpoints

 ANYCAST vs DNS-BASED GSLB

                      Anycast                 DNS GSLB
  Selection made by   BGP routers             Authoritative DNS server
  Granularity         Network path            Per resolver (not per user)
  Failover time       Seconds (reconvergence) TTL + resolver cache (minutes)
  Control over choice Coarse (BGP policy)     Fine (geo, latency, weights)
  Requires            Own AS + IP space       Nothing special
  Connection safety   Reroutes can reset      Stable once resolved

  Large providers use BOTH: anycast for the edge, DNS for coarse steering
  and for pulling whole regions out of service deliberately.

The operational barrier is that anycast requires running your own autonomous system with provider-independent address space and BGP peering — which is why most teams consume anycast through a CDN or cloud provider rather than operating it themselves.

Key Takeaways

• Anycast announces one IP from many locations and lets BGP route each user to a nearby site with no lookup step.
• It gives fast failover through route withdrawal and spreads DDoS traffic across the whole footprint.
• BGP optimizes for path policy, not latency, so "nearest" is approximate.
• Route changes can break long-lived connections; anycast suits DNS and short HTTP flows best.

🧪 Practice

1. Explain why anycast is a near-perfect fit for DNS but a poor fit for a multi-hour file upload.
2. Describe how anycast changes the economics of a volumetric DDoS attack.
3. Interview: A user in Toronto reports being served from London instead of Montreal. Is anycast broken? (Hint: what does BGP actually minimize?)

<a id="63-traffic-control"></a>

6.3 Traffic Control

Load balancing decides where work goes. Traffic control decides how much work is allowed in at all — because every system has a capacity, and the behaviour of a system past that capacity is a design choice, not an accident.

Rate Limiting Algorithms

A rate limiter enforces a cap on how much a client may ask for in a period: 100 requests per minute, 10 uploads per hour, 5 login attempts per 15 minutes. It is the most basic form of traffic control and it serves three distinct purposes that are worth keeping separate, because they imply different limits:

1. Protection. Keep one client from consuming capacity that belongs to everyone (accidental retry storms, runaway scripts).
2. Fairness. Give each tenant a predictable share in a multi-tenant system.
3. Business policy. Enforce plan tiers — free users get 1,000 calls a day, enterprise gets 1,000,000.

Before choosing an algorithm, three questions decide the design.

What is the limit keyed on? API key, user ID, IP address, or endpoint. IP is the weakest key: it punishes everyone behind a corporate NAT and is trivially evaded with a botnet. Authenticated identity is far better where you have it.

Where is it enforced? At the edge (gateway), where it protects everything behind it and rejects cheaply; or in the service, where it can see costs the gateway cannot. Serious systems do both.

Is the counter local or shared? A local counter on each of N instances means the effective limit is N times the configured one. A shared counter is accurate but adds a network round trip and a dependency to every request.

 LOCAL vs SHARED COUNTERS

  Limit: 100 rps per API key, fleet of 10 instances

  LOCAL              each instance allows 100 -> real limit is 1000 rps
                     (fast, no dependency, WRONG by a factor of N)

  SHARED             all instances INCR the same Redis key -> real limit 100
                     (correct, but +1 ms and a hard dependency per request)

  HYBRID             each instance takes a LEASE of 10 rps from Redis every
                     few seconds and enforces locally -> approximately
                     correct, one round trip per lease instead of per request

A rate limiter must also tell the client what happened, and there is a standard vocabulary for it. Returning 429 Too Many Requests without a Retry-After is an invitation for the client to retry immediately and make things worse.

HTTP/1.1 429 Too Many Requests
Retry-After: 30                       ; seconds -- tells the client to back off
RateLimit-Limit: 100                  ; the quota for the window
RateLimit-Remaining: 0                ; what is left
RateLimit-Reset: 30                   ; seconds until the quota refills

# The decision surface of a rate limiter, before any algorithm is chosen.
LIMITS = {
    # key pattern         limit  window  scope     rationale
    ("free",  "GET /*"):  (1_000, 3600, "api_key"),   # business policy
    ("free",  "POST /*"): (   60, 3600, "api_key"),   # writes cost more
    ("*",     "POST /login"): (5,  900, "ip+user"),   # credential stuffing
    ("*",     "POST /export"): (2, 3600, "api_key"),  # protects a heavy job
}
# Note that one number per client is almost never right: an endpoint that
# writes to the database is not interchangeable with one that reads a cache.
# Limits should track the COST of the work, not merely the request count.

Finally, distinguish rate limiting from throttling and load shedding, which appear later in this subchapter. Rate limiting is per-client and policy-based — it rejects you because you exceeded your quota, regardless of whether the system is busy. Load shedding is global and condition-based — it rejects requests because the system is overloaded, regardless of whose they are. You need both, and confusing them produces limiters that stay wide open during an overload or reject good traffic on an idle system.

Key Takeaways

• Rate limits serve protection, fairness, and business policy; be explicit about which one a given limit is for.
• Key limits on authenticated identity where possible; IP-based limits punish shared egress and are easy to evade.
• Local counters multiply the real limit by the instance count; shared counters are accurate but add a dependency per request.
• Always return 429 with Retry-After and quota headers, and set limits by the cost of the work, not just request count.

🧪 Practice

1. Your limit is 1,000 rpm per key, enforced locally on 12 instances. What is the actual limit a client experiences, and give two ways to fix it.
2. Design limits for a public API with free and paid tiers, one expensive export endpoint, and a login endpoint. Justify each key and window.
3. Interview: A customer says they are rate-limited at well under their quota. List three plausible causes. (Hint: consider what "one client" means when they run 40 pods behind one NAT gateway, and which key you chose.)

Token Bucket and Leaky Bucket

Token bucket and leaky bucket are the two classic rate-limiting algorithms. They are often confused because both use a bucket metaphor, but they answer different questions and produce visibly different behaviour.

Token bucket models permission. A bucket holds up to capacity tokens and refills at a steady rate. Each request removes one token; if the bucket is empty, the request is rejected or must wait. Because tokens accumulate while the client is idle, a client that has been quiet can spend the saved tokens in a burst — which is usually what you want, since real clients are bursty and a limiter that forbids all bursts feels broken.

Leaky bucket models pacing. Requests enter a queue and drain at a fixed rate, like water leaking from a bucket with a hole. Output is perfectly smooth regardless of how bursty the input was; if the queue is full, new requests are dropped. It protects a downstream system that genuinely cannot absorb bursts.

 TOKEN BUCKET                        LEAKY BUCKET

     tokens drip in at rate R          requests pour in (bursty)
              |                                  |
              v                                  v
        +-----------+                      +-----------+
        | ooooo     |  capacity C          | rrrrr     |  queue size Q
        +-----------+                      +-----------+
              |                                  |
       request takes 1 token             drains at fixed rate R
              |                                  |
              v                                  v
        allowed / 429                     perfectly smooth output

  Idle time ACCUMULATES tokens          Idle time accumulates NOTHING
  -> bursts up to C are allowed         -> output is never bursty
  -> output is bursty                   -> latency is added by queuing

import time

class TokenBucket:
    """Allows bursts up to `capacity`, sustained rate of `rate` per second."""
    def __init__(self, rate, capacity):
        self.rate, self.capacity = rate, capacity
        self.tokens = float(capacity)      # start full: a fresh client may burst
        self.last = time.monotonic()

    def allow(self, cost=1):
        now = time.monotonic()
        # Lazy refill: no background timer needed. Compute how many tokens
        # accrued since the last call, capped at the bucket size.
        self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
        self.last = now
        if self.tokens >= cost:
            self.tokens -= cost            # `cost` lets expensive calls charge more
            return True
        return False

bucket = TokenBucket(rate=10, capacity=50)   # 10 rps sustained, 50-request burst
print(sum(bucket.allow() for _ in range(80)))   # 50 -> the burst allowance
time.sleep(1)
print(sum(bucket.allow() for _ in range(80)))   # ~10 -> the refill in 1 second

class LeakyBucket:
    """Smooths output to exactly `rate`; excess beyond `size` is dropped."""
    def __init__(self, rate, size):
        self.rate, self.size = rate, size
        self.level = 0.0                    # how full the queue currently is
        self.last = time.monotonic()

    def allow(self):
        now = time.monotonic()
        # Drain first: the bucket has been leaking since the last request.
        self.level = max(0.0, self.level - (now - self.last) * self.rate)
        self.last = now
        if self.level + 1 <= self.size:
            self.level += 1
            return True
        return False                        # bucket overflows -> drop

Property	Token bucket	Leaky bucket
Bursts	Allowed, up to capacity	Never
Output shape	Bursty then rate-limited	Perfectly constant
Idle credit	Accumulates tokens	None
Adds latency	No (reject or pass immediately)	Yes, when queuing
State per client	Two floats	Two floats (or a real queue)
Typical use	Public API quotas	Protecting a fragile downstream

Two practical notes. First, token bucket's cost parameter is underused: charge a search request 5 tokens and a cached lookup 1, and the limit finally reflects real capacity instead of request count. Second, in a distributed setting the lazy-refill formulation is what makes token bucket cheap to implement in Redis — you store (tokens, last_refill) and update both atomically in a small Lua script, so no background process is needed.

Key Takeaways

• Token bucket allows saved-up bursts and enforces a sustained average; it is the right default for API quotas.
• Leaky bucket produces perfectly smooth output and adds queuing latency; use it to protect something that cannot tolerate bursts.
• Both are implemented with two numbers and lazy refill — no timers required.
• Weight requests by cost so the limit reflects work, not request count.

🧪 Practice

1. A token bucket has rate 5/s and capacity 100. A client is idle for a minute then sends 200 requests instantly. How many succeed, and why?
2. Implement token bucket as a Redis Lua script so the check-and-decrement is atomic across instances.
3. Interview: Your API allows 100 requests/minute. A client sends all 100 in the first second every minute and overloads a downstream service. Which algorithm fixes this, and what does the client lose? (Hint: which one refuses to let bursts exist at all?)

Fixed and Sliding Window Counters

Window counters are the other family of rate limiters — the ones that count events inside a time window rather than modelling a bucket. They are popular because they map directly onto a Redis INCR and onto how limits are usually expressed ("1,000 requests per hour").

Fixed window divides time into aligned buckets — 10:00:00-10:00:59, 10:01:00-10:01:59 — and counts requests in the current one. It is trivial to implement and has one well-known flaw: a client can send a full window's worth of requests at the end of one window and another full window's worth at the start of the next, achieving twice the limit across a period shorter than one window.

 THE FIXED-WINDOW BOUNDARY BURST (limit: 100/minute)

  window 1: 10:00:00 --------------------------- 10:00:59
                                          [100 requests at 10:00:59]
  window 2: 10:01:00 --------------------------- 10:01:59
            [100 requests at 10:01:00]

  Both windows are within their limit. But in the 2 seconds spanning the
  boundary the client sent 200 requests -- 2x the intended rate.

Sliding window log fixes this exactly: store a timestamp for every request, and on each new request drop the timestamps older than the window and count what remains. It is perfectly accurate and costs memory proportional to the limit — a 10,000/hour limit means storing up to 10,000 timestamps per client, which for a million clients is untenable.

Sliding window counter is the compromise everyone actually ships. Keep the count for the current window and the previous one, and estimate the rate by weighting the previous window by how much of it the sliding window still covers.

import time

def sliding_window_counter(prev_count, curr_count, limit, window, now):
    """Weighted estimate: accurate to within a few percent, O(1) memory."""
    elapsed = now % window                     # seconds into the current window
    prev_weight = (window - elapsed) / window  # how much of prev is still in view
    estimate = prev_count * prev_weight + curr_count
    return estimate < limit, estimate

# The boundary-burst scenario that defeats a fixed window:
LIMIT, WINDOW = 100, 60
now = 1_000_000 + 1        # 1 second into the new window
ok, est = sliding_window_counter(prev_count=100, curr_count=0,
                                 limit=LIMIT, window=WINDOW, now=now)
print(f"allowed={ok} estimate={est:.1f}")
# allowed=False estimate=98.3
#
# The previous window's 100 requests still count for 59/60 of their weight,
# so the client is correctly refused. A fixed window would have allowed 100 more.

# Redis implementation: two keys, one round trip, O(1) memory per client.
LUA = """
local curr_key, prev_key = KEYS[1], KEYS[2]
local limit, window, elapsed = tonumber(ARGV[1]), tonumber(ARGV[2]), tonumber(ARGV[3])
local curr = tonumber(redis.call('GET', curr_key) or '0')
local prev = tonumber(redis.call('GET', prev_key) or '0')
local estimate = prev * ((window - elapsed) / window) + curr
if estimate >= limit then
  return {0, math.floor(estimate)}          -- rejected, report the estimate
end
redis.call('INCR', curr_key)
redis.call('EXPIRE', curr_key, window * 2)  -- keep it one extra window for the
return {1, math.floor(estimate) + 1}        -- next window's weighting
"""

Algorithm	Accuracy	Memory per client	Boundary burst
Fixed window	Poor at boundaries	One counter	Up to 2x limit
Sliding window log	Exact	One entry per request	None
Sliding window counter	Within a few percent	Two counters	Effectively none
Token bucket	Exact for the model	Two floats	Bounded by capacity

The choice between sliding window counter and token bucket is mostly about how you want bursts to behave. Token bucket says "you may save up to C requests and spend them at once." Sliding window says "you may never exceed N in any N-window, period." For public API quotas the token bucket is friendlier; for abuse prevention the sliding window's strictness is the point.

Key Takeaways

• Fixed windows are trivial but allow up to twice the limit across a window boundary.
• Sliding window logs are exact but store one entry per request — unusable at high limits.
• The weighted sliding window counter gives near-exact behaviour with two counters and is the usual production choice.
• Window counters forbid bursts; token buckets permit bounded ones. Pick by whether bursts are abuse or normal client behaviour.

🧪 Practice

1. With a limit of 60/minute, construct the exact request pattern that gets the most requests through a fixed window in a 10-second period.
2. Compute the memory needed for a sliding window log with a 10,000/hour limit across 500,000 active clients, and explain why that rules it out.
3. Interview: How would you rate-limit an API across 30 gateway instances with sub-millisecond overhead and no more than ~5% error? (Hint: perfect accuracy per request costs a round trip — what can each instance be given in advance?)

Throttling and Admission Control

Rate limiting answers "has this client exceeded its quota?" Admission control answers a harder question: "does the system have capacity to serve this request at all?" The distinction matters because a system can be overloaded while every individual client is within quota — for instance when the number of clients doubles.

The reason admission control exists is that queueing systems degrade catastrophically, not gracefully, past saturation. As utilization approaches 100%, queueing delay does not rise linearly; it rises hyperbolically. The result is that accepting a little too much work does not make the system a little slower — it makes it unusable.

 QUEUEING DELAY vs UTILIZATION (M/M/1: wait ~ rho / (1 - rho))

  wait  |                                            *
        |                                        *
        |                                   *
        |                             *
        |                    *
        |          *   *
        | *  *
        +---------------------------------------------> utilization
         10%  30%  50%   70%   80%   90%  95%  99%

  50% -> 1x service time     90% ->  9x
  80% -> 4x                  99% -> 99x

  This is why "we are only 15% over capacity" is never a small problem, and
  why the target utilization for a latency-sensitive service is 60-70%, not 95%.

Admission control means deciding, at the door, whether to let a request in. There are three signals to decide on, in increasing order of quality:

Signal	Measures	Weakness
CPU / memory	Resource use	Misses I/O-bound saturation entirely
Concurrency (in-flight)	Little's Law load	Needs a known good limit
Latency vs target	Actual user experience	Reacts after degradation starts

The most robust designs use adaptive concurrency limiting: rather than configuring a fixed maximum, continuously estimate the concurrency at which latency starts to degrade, and admit up to that. This is TCP congestion control applied to requests — probe upward while things are fast, back off sharply when latency rises.

# Adaptive concurrency limit, in the style of Netflix's concurrency-limits
# and Envoy's adaptive concurrency filter (AIMD: additive increase,
# multiplicative decrease).
class AdaptiveLimiter:
    def __init__(self, initial=20, min_limit=4, max_limit=1000):
        self.limit = initial
        self.min_limit, self.max_limit = min_limit, max_limit
        self.inflight = 0
        self.best_rtt = float('inf')     # the no-queue baseline latency

    def try_acquire(self):
        if self.inflight >= self.limit:
            return False                 # admission DENIED -> 503 immediately
        self.inflight += 1
        return True

    def release(self, rtt_ms, dropped=False):
        self.inflight -= 1
        self.best_rtt = min(self.best_rtt, rtt_ms)   # baseline = fastest seen
        if dropped or rtt_ms > self.best_rtt * 2:
            # Latency has doubled: we are queueing. Back off hard.
            self.limit = max(self.min_limit, self.limit * 0.9)
        elif self.inflight >= self.limit * 0.9:
            # We are near the limit and still fast -> there may be more room.
            self.limit = min(self.max_limit, self.limit + 1)

# The key insight: the limit is DISCOVERED, not configured. It adapts when a
# dependency slows down, when instance size changes, or when the workload mix
# shifts -- all situations that make a hand-tuned constant wrong.

Two design rules make admission control worth having:

Reject fast and cheap. A rejection that costs a database query is not protection. Admission checks belong before authentication lookups, before deserialization of large bodies, and ideally at the edge proxy.

Reject with the right signal. 503 Service Unavailable with Retry-After tells clients the system is overloaded; 429 tells them they specifically are over quota. Clients (and their retry libraries) behave differently in response, and mislabeling causes well-behaved clients to retry when they should back off.

Key Takeaways

• Rate limiting is per-client quota; admission control is system-wide capacity.
• Queueing delay grows hyperbolically near saturation, so exceeding capacity slightly produces disproportionate latency.
• Concurrency and latency are better admission signals than CPU, especially for I/O-bound services.
• Adaptive limiters discover the safe concurrency instead of relying on a hand-tuned constant; reject early, cheaply, and with the correct status code.

🧪 Practice

1. Using wait = rho / (1 - rho), compute the queueing multiplier at 85%, 95%, and 98% utilization, and explain the target you would set.
2. Extend AdaptiveLimiter to keep a small reserved share of the limit for health checks and internal traffic.
3. Interview: Where in the request path do you put an admission check, and what must be true of it? (Hint: whatever the check costs, you pay it on every request you are trying to not do work for.)

Load Shedding

Load shedding is admission control's blunt sibling: when the system is over capacity, deliberately drop some work so the rest can be served correctly. The name comes from power grids, where controlled blackouts in one district prevent an uncontrolled collapse of the entire network.

The reason shedding beats the alternative is arithmetic. A system at 150% of capacity that accepts everything serves 100% of requests slowly and badly — often so slowly that clients time out, retry, and push the load to 200%. The same system shedding one third of requests serves two thirds of them perfectly and fails the rest immediately, which is both a better outcome and a stable one.

 WITHOUT SHEDDING (metastable failure)          WITH SHEDDING

  load 150% of capacity                         load 150% of capacity
        |                                             |
  all requests queue                            33% rejected in <1 ms
        |                                             |
  latency exceeds client timeouts               67% served at normal latency
        |                                             |
  clients retry -> load becomes 200%            clients back off on 503
        |                                             |
  goodput collapses toward ZERO                 goodput stays at 100% of
  and stays there after load drops              capacity, recovers instantly

  The left path is a "metastable failure": the system stays broken even
  after the original trigger goes away, because retries now sustain it.

The critical property of the right-hand path is goodput — successfully completed work — as distinct from throughput. Under overload without shedding, throughput stays high while goodput approaches zero, because everything the system produces arrives after the client gave up.

Shedding well means shedding selectively. Dropping requests at random wastes the capacity you saved on low-value work. The dimensions to prioritize on:

Dimension	Shed first	Keep
Criticality	Analytics, prefetch, recommendations	Checkout, auth, payments
Client class	Free tier, bulk API	Paid tier, interactive users
Request age	Already past its deadline	Fresh requests
Retry status	Retries (they had a chance)	First attempts
Cost	Expensive queries	Cheap cached reads

# Priority-aware shedding driven by observed system health.
CRITICALITY = {
    "CRITICAL_PLUS": 0,    # auth, payments -- shed only at total collapse
    "CRITICAL":      1,    # core user actions
    "SHEDDABLE_PLUS":2,    # recommendations, related items
    "SHEDDABLE":     3,    # analytics beacons, prefetch, background sync
}

def should_admit(request, queue_depth, max_queue, now):
    # 1. Drop anything that can no longer be useful. Work on a request whose
    #    client has already given up is pure waste -- and it is worse than
    #    waste, because it consumes capacity a live request needed.
    if request.deadline_ms and now > request.deadline_ms:
        return False, "deadline_exceeded"

    # 2. Shed by criticality as pressure rises: the fuller the queue, the
    #    higher up the priority ladder the cutoff climbs.
    pressure = queue_depth / max_queue                 # 0.0 .. 1.0+
    cutoff = 3 if pressure < 0.7 else (2 if pressure < 0.85 else
             (1 if pressure < 0.95 else 0))
    if CRITICALITY[request.criticality] > cutoff:
        return False, "shed_by_priority"

    # 3. Retries are shed before first attempts: the retry already consumed
    #    one chance, and admitting it now can sustain a retry storm.
    if request.is_retry and pressure > 0.8:
        return False, "shed_retry"

    return True, "admitted"

Three implementation details separate shedding that helps from shedding that does not:

• Shed at the front, not the back. A request that has already done 90% of its work should be finished, not dropped. Reject at admission.
• Prefer LIFO under overload. Counter-intuitively, serving the newest request first is better when overloaded: the oldest queued requests are the ones most likely to have already timed out, so FIFO spends capacity on work nobody is waiting for any more.
• Make it observable. Shedding is a system deliberately failing requests. If the rate, reason, and criticality of shed requests are not on a dashboard, the next incident will be diagnosed as an unexplained error spike.

Key Takeaways

• Shedding trades a controlled partial failure for an uncontrolled total one; the metric that matters is goodput, not throughput.
• Without it, overload plus client retries produces metastable failure that outlives the original trigger.
• Shed by criticality, client class, deadline, and retry status — never at random.
• Drop at admission (cheap), consider LIFO queuing under overload, and instrument every shed decision.

🧪 Practice

1. A service at 150% load has a client timeout of 2 s and an average queue wait of 5 s. Compute goodput with and without shedding one third of requests.
2. Classify the endpoints of a system you know into the four criticality tiers above and justify the boundaries.
3. Interview: Why does LIFO beat FIFO when a queue is overloaded, and why is it a bad default the rest of the time? (Hint: think about which request in the queue is most likely to still have a client waiting for it.)

Backpressure

Backpressure is the mechanism by which a slow consumer tells a fast producer to slow down. It is not a component you add; it is a property a pipeline either has or lacks, and the systems that lack it fail in a characteristic way — unbounded queue growth ending in memory exhaustion.

The reason it matters is that every pipeline has a mismatch somewhere. A service accepting 10,000 requests per second while its database handles 3,000 has three options for the excess: buffer it (memory grows until the process dies), drop it (shedding), or refuse to accept it in the first place (backpressure). Only the third both preserves the work and keeps the system stable, and it works by letting the slowness travel upstream to whoever can actually decide what to do.

The analogy is a factory conveyor belt. If the packing station is slower than the assembly line, you can pile boxes on the floor (unbounded queue), throw some away (shedding), or stop the line (backpressure). Stopping the line is the only option that neither loses product nor buries the building.

 NO BACKPRESSURE                        WITH BACKPRESSURE

  producer 10k/s                         producer 10k/s
       |                                       |
       v                                  [ bounded queue, 1k ]
  [ unbounded queue ]  <- grows                | full -> producer BLOCKS
       |                    forever            |         or is refused
       v                                       v
  consumer 3k/s                          consumer 3k/s

  Queue depth after 60s: 420,000         Queue depth: 1,000 (constant)
  Memory: OOM kill                       Producer learns the true rate and
  Latency: minutes (useless work)        slows down / sheds at the SOURCE

The single most important design rule follows directly: every queue must be bounded. An unbounded queue does not prevent overload, it merely converts a fast, visible failure into a slow, catastrophic one — and along the way it inflates latency to the point where everything in the queue is stale.

// Node streams implement backpressure explicitly. This is what ignoring it
// looks like -- and it is one of the most common memory bugs in production.
readable.on('data', (chunk) => {
  writable.write(chunk);          // BUG: return value ignored. If the writer
});                               // is slow, chunks buffer in memory forever.

// Correct: honour the signal. write() returns false when the buffer is full.
readable.on('data', (chunk) => {
  const ok = writable.write(chunk);
  if (!ok) {
    readable.pause();                                  // stop reading upstream
    writable.once('drain', () => readable.resume());   // resume when it clears
  }
});

// Or let the runtime do it -- pipeline() propagates backpressure end to end.
const { pipeline } = require('stream/promises');
await pipeline(readable, transform, writable);

# A bounded queue turns "unbounded memory growth" into an explicit decision.
import asyncio

queue = asyncio.Queue(maxsize=1000)      # THE bound: never omit this argument

async def producer(items):
    for item in items:
        try:
            # Do not simply await put(): that gives unbounded LATENCY instead
            # of unbounded memory. Bound the wait, then make a real decision.
            await asyncio.wait_for(queue.put(item), timeout=0.5)
        except asyncio.TimeoutError:
            metrics.increment("producer.shed")   # shed, or propagate 503
                                                 # upstream to the caller

Backpressure must propagate, not stop at one hop. If the database is slow, the service should slow down; if the service slows, the gateway should see it; if the gateway sees it, the client should be told. A pipeline where each stage buffers privately looks healthy at every layer until it collapses everywhere at once. The mechanisms differ by protocol but the intent is identical:

Layer	Backpressure mechanism
TCP	Receive window shrinks; sender stalls
HTTP/2, gRPC	Per-stream and per-connection flow-control windows
Reactive streams	Consumer requests N items; producer may not exceed it
Message queues	Bounded queue depth; producer blocks or gets an error
Thread pools	Bounded work queue with a rejection policy
Connection pools	Bounded size with an acquisition timeout

One anti-pattern deserves a specific warning: an unbounded thread pool or an unbounded connection pool defeats backpressure completely. newCachedThreadPool and a connection pool with no maximum both respond to overload by creating more of the resource that is already scarce. Bound them, and give them a rejection policy that fails fast.

Key Takeaways

• Backpressure is the consumer's slowness travelling back to the producer so the producer can slow or shed at the source.
• Every queue, pool, and buffer must be bounded — unbounded ones convert overload into memory exhaustion and useless stale work.
• Blocking forever on a full queue trades unbounded memory for unbounded latency; bound the wait and then decide.
• Backpressure must propagate across every hop, or the pipeline looks healthy right up until it collapses.

🧪 Practice

1. A producer emits 5,000 msg/s, the consumer handles 2,000 msg/s, and each message is 2 KB. Compute the memory used after 10 minutes with an unbounded queue.
2. Take a service you know and list every queue, pool, and buffer in its request path. Mark which are bounded.
3. Interview: A gRPC streaming service OOMs under load even though CPU is at 30%. Where do you look? (Hint: what does the server do with messages it has received but not yet processed?)

Request Prioritization and Fair Queuing

Once a system can shed load, the next question is whose requests get served with the capacity that remains. A single FIFO queue answers "whoever arrived first," which is exactly wrong under contention: one client running a bulk job can fill the queue and starve every interactive user behind it.

Request prioritization classifies work by importance and serves higher classes first. Fair queuing ensures each client, tenant, or class receives a guaranteed share regardless of how much they ask for. They are complementary: priority decides which kind of work matters; fair queuing prevents any one source of that work from taking everything.

 SINGLE FIFO QUEUE                  FAIR QUEUING (per-tenant)

  [A A A A A A A A B C]              tenant A: [A A A A A A]  --\
        |                            tenant B: [B]            ---+--> round
   one bulk client fills             tenant C: [C]            --/    robin
   the queue; B and C wait
   behind 8 of A's requests          Each dequeue takes from a DIFFERENT
                                     tenant, so B and C are served after
   B's latency = 8 x service          at most 2 other requests, no matter
                                     how much A submits.

The most effective fair-queuing scheme in practice is weighted fair queuing with a virtual clock: each request is stamped with a virtual finish time that accounts for the tenant's weight and how much service they have already received, and the queue always serves the smallest stamp. It gives each tenant their configured share when everyone is busy, and lets any tenant use the whole system when others are idle — work-conserving fairness.

import heapq
from collections import defaultdict

class WeightedFairQueue:
    """Each tenant gets a share proportional to its weight when contended,
    and may use spare capacity when others are idle."""
    def __init__(self, weights):
        self.weights = weights                  # tenant -> weight
        self.virtual_time = 0.0
        self.last_finish = defaultdict(float)   # tenant -> its last stamp
        self.heap = []

    def submit(self, tenant, request, cost=1.0):
        # A tenant that has been idle starts from the current virtual clock,
        # so it is not punished for its past silence -- and one that has been
        # greedy carries its accumulated finish time forward.
        start = max(self.virtual_time, self.last_finish[tenant])
        finish = start + cost / self.weights[tenant]   # heavier weight ->
        self.last_finish[tenant] = finish              # smaller increment ->
        heapq.heappush(self.heap, (finish, tenant, request))  # served sooner

    def dequeue(self):
        if not self.heap:
            return None
        finish, tenant, request = heapq.heappop(self.heap)
        self.virtual_time = finish              # clock advances with service
        return tenant, request

q = WeightedFairQueue({"enterprise": 10, "free": 1})
for i in range(6):
    q.submit("free", f"free-{i}")          # free tier floods the queue
q.submit("enterprise", "ent-0")            # one enterprise request arrives late

order = [q.dequeue()[0] for _ in range(4)]
print(order)   # ['free', 'enterprise', 'free', 'free']
#              -> the enterprise request jumps ahead of the free backlog
#                 without starving free entirely.

Strict priority queues (always serve class 0 before class 1) are simpler but introduce starvation: if high-priority work never stops arriving, low- priority work never runs. The usual remedies are a reserved minimum share for each class, or priority aging — increasing a request's priority the longer it waits.

Scheme	Guarantees	Risk
FIFO	Arrival order	One noisy client starves others
Strict priority	High class always first	Starvation of low classes
Round robin per tenant	Equal share	Ignores request cost differences
Weighted fair queuing	Proportional share, work-conserving	More state and bookkeeping
LIFO (under overload)	Freshest work served	Unfair; only for overload

Two supporting practices make prioritization work end to end. Propagate the priority — a criticality label attached at the edge should travel through every downstream call, or an internal service will happily spend its last capacity on a background job. And isolate the pools: give critical traffic its own connection pool and thread pool, so a flood of low-priority work cannot exhaust the resources critical work needs. That is the bulkhead pattern applied to queuing, and it is the difference between prioritization that works during an incident and prioritization that exists only on paper.

Key Takeaways

• FIFO under contention lets one heavy client starve everyone; fair queuing guarantees each source a share.
• Weighted fair queuing gives proportional shares when busy and full access to spare capacity when idle.
• Strict priority risks starvation; mitigate with reserved shares or priority aging.
• Propagate criticality across service boundaries and isolate pools per class, or the priority scheme stops at the first hop.

🧪 Practice

1. Trace the weighted fair queue above with weights 10:1 and ten queued free requests plus three enterprise requests. What is the service order?
2. Design the criticality labels for a food-delivery system and describe how they propagate from the mobile client to the payment service.
3. Interview: A batch export job makes the interactive API slow even though CPU is at 40%. What is happening and what do you change? (Hint: something bounded and shared is fully occupied — and it is not the CPU.)

<a id="64-service-discovery-and-routing"></a>

6.4 Service Discovery and Routing

Scaling out produces a fleet whose membership changes constantly — instances start, stop, move, and get replaced on every deploy. Service discovery is how callers find the current members; routing is how the choice among them is made and changed deliberately.

Client-Side vs. Server-Side Discovery

In a static world you configure a service's address once. In an elastic world that address is a moving target: auto-scaling adds instances, deploys replace them, and container schedulers relocate them to different hosts and ports every few minutes. Something must answer "where is the payment service right now?"

There are two architectures for answering it, and the difference is who holds the list.

Server-side discovery puts a load balancer between caller and callee. The caller knows one stable address — a DNS name pointing at the balancer — and the balancer knows the live instances. This is the model behind every cloud load balancer and Kubernetes Service.

Client-side discovery gives the caller the list. It queries a registry, caches the result, and picks an instance itself using whatever algorithm it likes. This is the model behind Netflix's Ribbon/Eureka, gRPC's built-in name resolvers, and every service mesh (where the "client" is a local sidecar).

 SERVER-SIDE DISCOVERY                CLIENT-SIDE DISCOVERY

  caller                                caller
    |                                     |  1. "instances of payments?"
    | one stable address                  v
    v                              +------------+
 [ load balancer ]  <-- registers  |  registry  |
    |     ^                        +------------+
    |     |                              |  2. [10.0.1.4, 10.0.1.9, ...]
    v     |                              v
 [ instances ]                     caller picks one itself
                                         |  3. direct connection
  + Callers stay simple                  v
  + One place for policy            [ instance ]
  - An extra hop of latency
  - The LB is a component to run   + No extra hop; caller-side algorithms
                                   - Discovery logic in every client
                                   - Per-language libraries to maintain

The trade-off is concentration versus distribution. Server-side discovery keeps all the complexity in one component that every language and framework can use, at the price of one extra network hop and one more thing to operate. Client-side discovery removes the hop and enables sophisticated per-caller load balancing, at the price of embedding a discovery library into every service in every language you use — and of updating all of them when policy changes.

// Server-side: the caller knows nothing about instances.
// "payments" resolves to a stable virtual IP; the LB does the rest.
HttpResponse<String> response = client.send(
    HttpRequest.newBuilder(URI.create("http://payments/charge")).build(),
    BodyHandlers.ofString());

// Client-side: the caller resolves, caches, and chooses.
List<Instance> instances = registry.getHealthy("payments");  // cached locally,
                                                             // refreshed in bg
Instance target = loadBalancer.choose(instances);  // any algorithm you like,
                                                   // e.g. zone-aware + EWMA
HttpResponse<String> response = client.send(
    HttpRequest.newBuilder(URI.create("http://" + target.hostPort() + "/charge"))
        .build(),
    BodyHandlers.ofString());
// The caller must also handle: stale cache entries, registry unavailability
// (keep serving the last known good list), and ejecting failing instances.

Aspect	Server-side	Client-side
Where the list lives	Load balancer	Every caller
Network hops	Two (caller -> LB -> target)	One (caller -> target)
Language coupling	None	A library per language
Load-balancing choice	Whatever the LB supports	Anything the client implements
Policy change rollout	Update the LB	Redeploy every caller
Failure of the mediator	LB down = service unreachable	Registry down = use cached list

The service mesh, covered later in this subchapter, is the design that refuses the trade-off: it uses client-side discovery for the routing benefits, but puts the logic in a sidecar proxy rather than in the application, so there is exactly one implementation regardless of language.

Key Takeaways

• Discovery exists because instance addresses change constantly in an elastic system.
• Server-side discovery hides everything behind one address: simple callers, one extra hop, a component to operate.
• Client-side discovery removes the hop and enables smarter balancing, at the cost of a library in every language.
• A registry outage should degrade to the last known good list, never to a hard failure.

🧪 Practice

1. List the failure modes introduced by each model and how each degrades when its mediator (LB or registry) is unavailable.
2. Your fleet uses Java, Go, and Python. Argue for one model and state what it costs you.
3. Interview: Why does client-side discovery make zone-aware routing easier? (Hint: who knows which zone the caller is in?)

Service Registries

A service registry is the database of "what is running where." It holds an entry per instance — service name, address, port, health status, and metadata like version or zone — and every discovery mechanism ultimately reads from one.

The interesting design questions are not about storing the data but about keeping it true. A registry that lists dead instances is worse than no registry at all, because callers will confidently route to nothing.

Registration happens one of two ways. Self-registration has the instance announce itself on startup and renew a lease periodically; simple, but couples application code to the registry and leaves stale entries when an instance dies without deregistering. Third-party registration has a separate agent — the container orchestrator, typically — register instances based on observed state; this is what Kubernetes does, and it is why nothing in a Kubernetes pod needs registry code.

Freshness is maintained by one of two mechanisms, and the difference determines how fast failures are noticed:

Mechanism	How it works	Stale window
Heartbeat / TTL lease	Instance renews every N seconds or is evicted	Up to the TTL
Active health check	Registry (or agent) probes instances	Interval x failure threshold

 REGISTRY LIFECYCLE

  1. START     instance boots, passes its own startup checks
  2. REGISTER  PUT /v1/register {name, ip, port, zone, version}
  3. RENEW     lease renewed every 10s (TTL 30s) -- three chances to miss
  4. DISCOVER  callers/LBs watch the registry and update their view
  5. DRAIN     SIGTERM -> deregister FIRST, then finish in-flight work
  6. EXPIRE    if the process dies hard, the lease expires and the entry
               is evicted after the TTL -- this is the safety net that
               makes step 5 an optimization rather than a requirement

# Self-registration with a lease. Note the two independent safety mechanisms.
import threading, atexit, signal

class RegistryClient:
    def __init__(self, registry, service, addr, ttl=30, renew_every=10):
        self.registry, self.service, self.addr = registry, service, addr
        self.ttl, self.renew_every = ttl, renew_every
        self.instance_id = f"{service}-{addr}"

    def start(self):
        self.registry.register(self.instance_id, self.service, self.addr,
                               ttl=self.ttl, meta={"version": VERSION,
                                                   "zone": ZONE})
        # Renew at 1/3 of the TTL: two consecutive renewals may fail without
        # the instance being evicted. Renewing at the TTL itself would make
        # a single dropped packet look like a dead instance.
        self._timer()
        atexit.register(self.stop)                 # graceful path
        signal.signal(signal.SIGTERM, lambda *_: self.stop())

    def _timer(self):
        self.registry.renew(self.instance_id, ttl=self.ttl)
        t = threading.Timer(self.renew_every, self._timer)
        t.daemon = True
        t.start()

    def stop(self):
        # Deregister explicitly so callers stop routing here immediately
        # instead of waiting out the TTL. The TTL remains the backstop for
        # a hard crash, where this code never runs at all.
        self.registry.deregister(self.instance_id)

The hardest property to get right is behaviour during a network partition. A registry that insists on strong consistency (etcd, Consul, ZooKeeper — all Raft-based) will refuse writes when it loses quorum, meaning instances cannot register during exactly the incident when you most need capacity to come online. Eureka takes the opposite position with self-preservation mode: when it sees a mass loss of heartbeats, it assumes the network is broken rather than that every instance died, and it stops evicting entries.

Registry	Consistency	Health model	Notable property
etcd	Strong (Raft)	TTL leases + watches	Backs Kubernetes; refuses writes without quorum
Consul	Strong (Raft)	Agent-run checks	Rich health checks, multi-datacenter
ZooKeeper	Strong (ZAB)	Ephemeral znodes	Entry vanishes when the session drops
Eureka	AP, eventual	Client heartbeats	Self-preservation prefers stale over empty

The underlying principle generalizes beyond registries: for a discovery system, availability usually beats consistency. Routing to an instance that died two seconds ago costs one failed request and a retry. Being unable to resolve any address at all takes the entire system down.

Key Takeaways

• A registry maps service names to live instances with health state and metadata; its hard problem is staleness, not storage.
• Renew leases well inside the TTL so transient packet loss does not evict a healthy instance.
• Deregister on shutdown for speed and rely on TTL expiry as the crash backstop.
• Prefer availability over consistency for discovery: a slightly stale list beats an empty one.

🧪 Practice

1. With a 30-second TTL renewed every 10 seconds, how long can a hard-crashed instance keep receiving traffic? How would you reduce it, and what breaks if you go too far?
2. Explain what self-preservation mode prevents, and construct a scenario where it makes an incident worse.
3. Interview: Your registry is a Raft cluster that loses quorum during an AZ outage. What happens to running traffic, and to a scale-out? (Hint: reads from a cached list and writes to the registry have very different fates.)

DNS-Based Discovery

DNS is the oldest service discovery system, and it is still the most widely deployed one — every language can resolve a hostname with no library, no sidecar, and no code change. Using DNS for discovery means putting instance addresses into DNS records and letting normal resolution find them.

Three record types do the work:

• A / AAAA records map a name to one or more IP addresses. Multiple records give crude round robin: resolvers typically rotate the order.
• SRV records map a name to host and port, plus priority and weight — which matters when instances land on random ports, as they do under most container schedulers.
• CNAME aliases one name to another, used for regional or blue/green indirection.

 DNS RECORDS FOR DISCOVERY

  payments.internal.        30  IN  A      10.0.1.4
  payments.internal.        30  IN  A      10.0.1.9      <- multiple A records,
  payments.internal.        30  IN  A      10.0.2.7         resolver rotates

  _grpc._tcp.payments.      30  IN  SRV    10 60 50051 pay-a.internal.
  _grpc._tcp.payments.      30  IN  SRV    10 40 50052 pay-b.internal.
                                            ^  ^  ^     ^
                                            |  |  |     target host
                                            |  |  port  (dynamic!)
                                            |  weight (60/40 split)
                                            priority (lower wins first)

DNS's appeal is universality, and its weakness is that it was designed for records that change rarely. Four problems follow, and you must plan for all of them.

Caching you do not control. A TTL is a hint. OS resolvers, language runtimes, and connection pools all cache, sometimes ignoring the TTL entirely. The infamous case is the JVM, which historically cached successful lookups forever by default:

// Without this, a long-running JVM may keep using an IP that was retired
// hours ago. networkaddress.cache.ttl is a JVM-wide security property.
java.security.Security.setProperty("networkaddress.cache.ttl", "30");
java.security.Security.setProperty("networkaddress.cache.negative.ttl", "5");

No health awareness by default. A plain A record does not know its target is down. Health-aware DNS exists (Route 53 health checks, Consul DNS), but the removal still only takes effect as records expire from caches.

No load awareness. Resolver rotation is not load balancing; it cannot see that one instance is saturated.

Response size limits. A UDP DNS response is practically limited to about 512 bytes without EDNS0, which caps how many instances fit in one answer — a real constraint for fleets of hundreds.

# What a DNS-discovering client must do that a registry client gets for free.
import socket, time, random

class DnsDiscovery:
    def __init__(self, name, refresh=30):
        self.name, self.refresh = name, refresh
        self.addrs, self.fetched_at = [], 0.0
        self.ejected = {}                      # addr -> retry-after timestamp

    def resolve(self):
        # Re-resolve on OUR schedule rather than trusting the platform to
        # honour the TTL. Long-lived connection pools otherwise pin to IPs
        # that were retired long ago.
        if time.monotonic() - self.fetched_at > self.refresh:
            infos = socket.getaddrinfo(self.name, None, proto=socket.IPPROTO_TCP)
            self.addrs = list({i[4][0] for i in infos})
            self.fetched_at = time.monotonic()
        now = time.monotonic()
        live = [a for a in self.addrs if self.ejected.get(a, 0) < now]
        return live or self.addrs              # never return an empty list:
                                               # better to try a suspect host
                                               # than to fail with no target

    def eject(self, addr, seconds=30):
        # Passive health checking, because DNS will not tell us about failures.
        self.ejected[addr] = time.monotonic() + seconds

The correct way to think about DNS is as a bootstrap and coarse-grained routing mechanism, not a fine-grained load balancer. Use it to find the regional entry point or the stable service address; use a registry, a load balancer, or a mesh for instance-level decisions that must react in seconds. This is exactly the layering Kubernetes uses: DNS resolves payments to a stable virtual IP, and the actual instance selection happens below it in kube-proxy or the mesh.

Key Takeaways

• DNS discovery is universal and needs no client library; SRV records add ports and weights for dynamic environments.
• TTLs are advisory — OS, runtime, and pool caches routinely outlive them, so re-resolve on your own schedule.
• DNS is neither health-aware nor load-aware by default, and UDP response size limits the number of instances per answer.
• Use DNS for coarse routing and bootstrap; use a registry or mesh for instance-level decisions.

🧪 Practice

1. Explain why lowering a TTL to 5 seconds does not guarantee 5-second failover, and name three caches involved.
2. Write the SRV records for a service with two instances split 70/30 across dynamic ports.
3. Interview: A service keeps sending traffic to a terminated instance an hour after it was removed from DNS. What do you check first? (Hint: what does a long-lived connection pool do after its first successful resolution?)

Service Mesh and Sidecar Proxies

A service mesh moves discovery, load balancing, retries, timeouts, mutual TLS, and telemetry out of application code and into a proxy deployed next to every instance. The application talks to localhost; the proxy handles everything that used to require a library.

The motivation is a duplication problem. In a polyglot fleet, every cross- cutting concern — retry policy, circuit breaking, tracing headers, mTLS — must be implemented in every language, kept consistent, and upgraded everywhere at once. That upgrade is the killer: changing a retry policy across forty services in five languages is a quarter-long project. A mesh makes it a configuration push.

 SIDECAR TOPOLOGY

  +---------------------------+        +---------------------------+
  |  pod: orders              |        |  pod: payments            |
  |  +---------+  +--------+  |  mTLS  |  +--------+  +---------+  |
  |  |  app    |->| proxy  |--|--------|->| proxy  |->|  app    |  |
  |  +---------+  +--------+  |        |  +--------+  +---------+  |
  +--------|------------------+        +--------|------------------+
           |    DATA PLANE (Envoy sidecars)     |
           +------------------+-----------------+
                              |  xDS config + telemetry
                     +--------------------+
                     |   CONTROL PLANE    |  Istio / Linkerd
                     |  policy, certs,    |
                     |  service registry  |
                     +--------------------+

  The app makes a plain HTTP call to localhost. The proxy adds: discovery,
  load balancing, retries, timeouts, circuit breaking, mTLS, and metrics.

The split between data plane (the proxies, on the request path) and control plane (configuration, certificates, policy — off the request path) is the architecture's central idea. Traffic keeps flowing when the control plane is down, because proxies run on their last-known configuration.

# Istio: retries, timeouts, and outlier ejection as CONFIGURATION.
# None of this exists in the application's source code.
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata: { name: payments }
spec:
  hosts: [payments]
  http:
    - timeout: 3s                     # end-to-end budget for the whole call
      retries:
        attempts: 3
        perTryTimeout: 800ms          # 3 x 800ms < 3s, so retries fit the budget
        retryOn: 5xx,reset,connect-failure
                                      # NOTE: no retry on 4xx -- retrying a
                                      # client error just wastes capacity
      route:
        - destination: { host: payments, subset: v1 }
---

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata: { name: payments }
spec:
  host: payments
  trafficPolicy:
    connectionPool:
      http: { http2MaxRequests: 100, maxRequestsPerConnection: 10 }
    outlierDetection:                 # passive health checking
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50          # CRITICAL: never eject more than half,
                                      # or a bad deploy empties the whole pool

What you get, and what it costs:

Gain	Cost
Uniform policy across all languages	One extra proxy hop each way (~0.5-2 ms p99)
mTLS everywhere with automatic rotation	50-150 MB memory and some CPU per pod
Consistent golden metrics for free	A complex control plane to operate and upgrade
Traffic splitting without code changes	Debugging now spans app and proxy layers
Retries and circuit breaking as config	A new class of failure: proxy misconfiguration

That maxEjectionPercent line is worth dwelling on, because it encodes a general lesson about automated ejection: if a bad deploy makes every instance return 5xx, unrestricted outlier detection will eject the entire pool and turn a degraded service into a dead one. Every automatic removal mechanism needs a floor.

A mesh is justified when you have many services in multiple languages and need uniform security and policy. It is over-engineering for a handful of services in one language, where a good client library gives you most of the benefit for a fraction of the operational cost. Newer designs reduce the overhead by moving L4 concerns into a per-node component and keeping a sidecar only where L7 features are actually needed.

Key Takeaways

• A mesh moves discovery, retries, timeouts, mTLS, and telemetry into a sidecar proxy so every language gets identical behaviour.
• Data plane (proxies) stays on the request path; control plane (config, certs) does not, so traffic survives a control-plane outage.
• The cost is real: an extra hop, per-pod resources, and a complex control plane to operate.
• Cap automatic ejection — unbounded outlier detection turns a bad deploy into a total outage.

🧪 Practice

1. Compute the added latency for a request that traverses four services in a mesh, at 1 ms per proxy hop.
2. Rewrite the retry policy above for an endpoint that is not idempotent, and explain each change.
3. Interview: A team wants a mesh for eight Go services in one cluster. What do you ask before agreeing? (Hint: which of the mesh's benefits do they not already get from one shared client library?)

Traffic Splitting and Canary Routing

Traffic splitting is the ability to send a controlled percentage of requests to a different version of a service. It is the mechanism behind canary releases, blue/green deployments, and A/B tests — and it is what turns a deployment from an all-or-nothing event into a measured, reversible experiment.

The reason it matters is that testing environments never fully reproduce production: real traffic patterns, real data distributions, real client versions, real concurrency. A canary release accepts this by exposing the new version to a small fraction of real traffic and watching the metrics that matter, so a bad release harms 1% of requests for five minutes rather than 100% of requests until someone notices.

 CANARY PROGRESSION

  t0   v1: 100%  v2:   0%   deploy v2, no traffic, verify it starts
  t1   v1:  99%  v2:   1%   watch: error rate, p99 latency, business metrics
  t2   v1:  95%  v2:   5%   still healthy after a full metric window
  t3   v1:  75%  v2:  25%
  t4   v1:   0%  v2: 100%   promote; keep v1 ready to receive traffic again

  At ANY step, an automated check that fails sends the split back to 0%.
  Rollback is a routing change -- seconds -- not a redeploy.

The comparison with the neighbouring strategies:

Strategy	Traffic to new version	Rollback speed	Extra capacity needed
Rolling	Grows as instances swap	A full rolling back	Small
Blue/green	0% then 100% at once	Instant (flip back)	2x during the switch
Canary	Small, increasing	Instant (set to 0%)	Small
Shadow/mirror	0% (a copy is sent)	Not applicable	1x for the shadow

# Weighted split at the routing layer, independent of instance counts.
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata: { name: checkout }
spec:
  hosts: [checkout]
  http:
    # Rule order matters: the first match wins, so targeted rules go first.
    - match:
        - headers:
            x-canary: { exact: "true" }     # internal testers opt in
      route:
        - destination: { host: checkout, subset: v2 }

    - route:                                 # everyone else: weighted split
        - destination: { host: checkout, subset: v1 }
          weight: 95
        - destination: { host: checkout, subset: v2 }
          weight: 5
      mirror: { host: checkout, subset: v3 }  # v3 receives a COPY of traffic;
      mirrorPercentage: { value: 10 }         # its responses are DISCARDED

Two details decide whether a canary is meaningful.

Sticky assignment. Randomizing per request means one user's session bounces between versions, which produces incoherent behaviour and makes results unreadable. Hash a stable identifier so a given user consistently sees one version.

import hashlib

def variant_for(user_id, canary_percent):
    """Stable assignment: the same user always lands on the same version,
    and raising the percentage only ever ADDS users to the canary."""
    h = int(hashlib.sha256(f"checkout:{user_id}".encode()).hexdigest()[:8], 16)
    return "v2" if (h % 100) < canary_percent else "v1"

# Including the experiment name in the hash prevents correlated assignment:
# without it, the same users would be canaried for every experiment forever.

Automated analysis. A canary that a human watches is a canary that gets promoted on a Friday afternoon because the dashboard "looked fine." Define the comparison up front, and let a controller promote or roll back on it.

# Argo Rollouts: the promotion criteria are code, evaluated automatically.
analysis:
  metrics:
    - name: error-rate
      interval: 1m
      successCondition: result < 0.01        # absolute floor
      failureLimit: 2                        # two bad windows -> roll back
    - name: p99-latency-vs-baseline
      successCondition: result < 1.2         # no worse than 1.2x the stable
    - name: checkout-completion-rate         # the business metric: a release
      successCondition: result >= 0.98       # can be fast, healthy, AND wrong

That last metric is the one teams forget. A new version can have a lower error rate and better latency while quietly failing to complete purchases — a technically healthy release that is a business outage. Canary analysis must include at least one metric that measures whether users are still succeeding at what they came to do.

Finally, note the constraint that traffic splitting cannot solve on its own: both versions run simultaneously against the same data. A schema change, a cache format change, or a message format change must therefore be forward-and-backward compatible for the duration of the rollout. The routing layer makes the code rollback instant; it does nothing for data you have already written in the new format.

Key Takeaways

• Traffic splitting exposes a new version to a controlled fraction of real traffic, making rollback a routing change measured in seconds.
• Assign variants by hashing a stable user identifier so behaviour is coherent and results are readable.
• Automate promotion and rollback against explicit thresholds, including at least one business metric.
• During any split, both versions share the data layer — schema and message changes must be compatible in both directions.

🧪 Practice

1. Design a canary schedule for a service handling 100k rps, with the shortest step long enough to detect a 0.5% error-rate regression.
2. Explain why mirrored traffic must not perform writes, and describe what you would change to make a write path safely mirrorable.
3. Interview: A canary shows lower latency and fewer errors than the stable version, but revenue drops during the window. What happened, and what should the canary analysis have included? (Hint: an endpoint that returns 200 has not necessarily done anything useful.)

---

<a id="7-asynchronous-processing-and-messaging"></a>