13. Storage and Data Persistence

A web server's memory is temporary: it disappears when the process restarts, it is not shared between instances, and it is far too small for a growing user base. Every real application therefore keeps its data in a dedicated store. This chapter first builds the vocabulary needed to choose and use one: relational and NoSQL models, consistency guarantees, connection management and data modelling (13.1). It then studies the two stores most often paired with Node.js. MongoDB is a general-purpose document database (13.2). Redis is an in-memory data structure server used for caching, sessions, rate limiting and messaging (13.3). It builds on the Node.js runtime (Chapter 11) and on the Express services from Chapter 12, whose in-memory repositories, session stores and rate limiters this chapter finally backs with real storage.

<a id="131-persistence-concepts-for-web-applications"></a>

13.1 Persistence Concepts for Web Applications

This subchapter covers the ideas that apply to every database, whatever the product: the relational and NoSQL families, the guarantees they give (ACID, BASE, CAP), the main data models, how applications talk to a database through drivers and connection pools, and how to design data around the way a web application actually reads and writes it.

Relational vs NoSQL Data Stores

Why a database at all? Imagine storing users in a JSON file. It works for one user and one process. Then two requests write at the same time and one overwrites the other. The file grows to 2 GB and every lookup reads all of it. The server crashes halfway through a write and leaves the file corrupted. A database management system (DBMS) solves these problems once: concurrent access, fast lookups (indexes), crash safety, a query language, access control, backups and replication.

The relational model. Proposed by Edgar F. Codd in 1970, the relational model stores data in tables (relations) made of rows (tuples) and columns (attributes). Every column has a type, and the set of tables and columns (the schema) is declared in advance. Rows are linked through keys: a primary key identifies a row, and a foreign key points to the primary key of another table. Data is queried with SQL, a declarative language: you describe what you want, and the database's query planner decides how to fetch it. Popular relational databases (RDBMS) include PostgreSQL, MySQL/MariaDB, SQLite, Microsoft SQL Server and Oracle.

Relational design favours normalisation: each fact is stored exactly once, and related data is split into separate tables that are recombined with joins at query time. Think of a well-organised library card catalogue: authors in one drawer, books in another, each card pointing to the other by number. Nothing is duplicated, so nothing can become inconsistent.

-- Schema: declared up front, enforced by the database
CREATE TABLE authors (
  id    SERIAL PRIMARY KEY,
  name  TEXT NOT NULL
);

CREATE TABLE books (
  id         SERIAL PRIMARY KEY,
  title      TEXT NOT NULL,
  isbn       CHAR(13) UNIQUE NOT NULL,              -- the database rejects duplicates
  author_id  INTEGER NOT NULL REFERENCES authors(id) -- foreign key: must point to an existing author
);

-- Recombine the normalised data with a join
SELECT b.title, a.name AS author
FROM books b
JOIN authors a ON a.id = b.author_id
WHERE a.name = 'Frank Herbert';

NoSQL. In the late 2000s, companies such as Google (Bigtable, 2006) and Amazon (Dynamo, 2007) needed to spread data across thousands of commodity machines with predictable latency. Joins and multi-row transactions across machines are expensive, so they built stores that gave them up in exchange for horizontal scale and flexible schemas. "NoSQL" (usually read as "not only SQL") is an umbrella term for these non-relational stores. It is not one model but several (see "Document, Key-Value, Column and Graph Models"):

• Document stores (MongoDB, Couchbase, Firestore) keep self-contained JSON-like documents.
• Key-value stores (Redis, Valkey, DynamoDB, etcd) map a key to a value.
• Wide-column stores (Cassandra, ScyllaDB, HBase) organise data in rows with flexible column families, partitioned across nodes.
• Graph databases (Neo4j, Amazon Neptune) store nodes and relationships as first-class data.

The same book in a document store keeps related data together instead of normalising it:

// MongoDB document: the author is embedded; one read returns everything a page needs
{
  _id: ObjectId('66fe1c2a9b1e8a3f4c2d7e10'),
  title: 'Dune',
  isbn: '9780441172719',
  author: { id: 17, name: 'Frank Herbert' },   // duplicated from the authors collection on purpose
  tags: ['sci-fi', 'classic'],                 // arrays are native: no extra join table
  ratings: { count: 1287, average: 4.6 }
}

Comparison.

Aspect	Relational (SQL)	NoSQL (typical)
Data shape	Tables of rows with fixed columns	Documents, key-value pairs, wide rows, graphs
Schema	Declared and enforced up front (schema-on-write)	Flexible; often enforced by the application (schema-on-read) or optional validators
Relationships	Foreign keys and joins	Embedding, duplication, application-side joins, $lookup
Query language	Standard SQL	Product-specific APIs (MongoDB query language, Redis commands, CQL, Cypher)
Transactions	Multi-row, multi-table ACID by default	Single-record atomicity always; multi-record transactions in some products
Scaling	Vertical first; read replicas; sharding is possible but harder	Designed for horizontal partitioning (sharding)
Strengths	Integrity, ad hoc queries, reporting, complex relationships	Flexible shapes, very high throughput on known access patterns, easy scale-out
Weaknesses	Schema migrations on huge tables, scaling writes across machines	Ad hoc queries and cross-entity integrity are harder; denormalised data can drift

The lines are blurring. PostgreSQL stores and indexes JSON (JSONB), and MongoDB offers multi-document ACID transactions and schema validation. Distributed SQL databases (CockroachDB, YugabyteDB, Google Spanner) shard relational data horizontally. The choice is rarely "SQL or NoSQL forever"; it is about which model fits each workload.

Polyglot persistence. Many web systems use several stores, each for what it does best:

                       +----------------------------+
  browser --> API ---> | PostgreSQL / MongoDB       |  system of record: users, orders, products
                |      +----------------------------+
                +----> | Redis                      |  cache, sessions, rate limits, queues
                |      +----------------------------+
                +----> | Elasticsearch / OpenSearch |  full-text search over products
                |      +----------------------------+
                +----> | S3-compatible object store |  images, uploads, backups
                       +----------------------------+

The system of record is the authoritative copy. Every other store holds derived data that can be rebuilt from it. Keep that distinction clear: losing a cache is an inconvenience, losing the system of record is a disaster.

Choosing. Ask questions, not slogans:

1. How are entities related, and how often do you query across those relationships in unpredictable ways? Many ad hoc relationships favour relational.
2. Do the records of one kind have very different shapes, or does the shape change often? That favours documents.
3. Which operations must be atomic across several records (money transfers, stock reservations)?
4. What are the read/write volumes and latency targets, and will one machine be enough for years? It usually is.
5. What does the team already know how to operate, back up and monitor?

Key Takeaways

• A DBMS provides concurrency control, indexing, crash safety, querying and replication that files cannot.
• Relational databases store normalised tables with a declared schema, linked by keys and recombined with SQL joins.
• NoSQL is a family of models (document, key-value, wide-column, graph) that trade joins and sometimes cross-record transactions for flexibility and horizontal scale.
• Modern products blur the boundary (JSONB in PostgreSQL, transactions in MongoDB); choose by workload, not by trend.
• Polyglot persistence pairs a system of record with specialised stores for caching, search and files.

Practice

1. (Beginner) Model a blog with users, posts and comments twice: once as relational tables with primary and foreign keys, once as MongoDB documents. Write the query that returns a post with its author name and comments in each model.
2. (Intermediate) For an e-commerce site, list five kinds of data (catalogue, carts, orders, sessions, product search) and choose a store for each. Justify each choice in one sentence and identify the system of record.
3. (Interview) "When would you choose MongoDB over PostgreSQL, and when would you not?" Hint: talk about data shape variability, relationships queried ad hoc, transactional requirements, the team's operational experience, and the fact that PostgreSQL also stores JSON.

ACID vs BASE and the CAP Theorem

A database's guarantees decide which bugs your application can and cannot have. If two customers buy the last item at the same moment, will both orders succeed? If the server loses power right after confirming a payment, is the payment still there after the restart? These questions are answered by the consistency models below.

ACID. A transaction is a group of operations treated as a single unit. Relational databases (and MongoDB for multi-document transactions) guarantee four properties:

Property	Meaning	Bank transfer example
Atomicity	All operations happen, or none do	Debit and credit both apply, or neither does
Consistency	The transaction moves the database from one valid state to another; constraints (unique, foreign keys, checks) always hold	A CHECK (balance >= 0) constraint is never violated
Isolation	Concurrent transactions do not see each other's intermediate states	Another reader never sees money that has left one account but not arrived in the other
Durability	Once committed, data survives crashes	After "transfer complete", a power failure does not undo it

Durability is usually implemented with a write-ahead log (WAL): the change is appended to a log file and flushed to disk before the commit is acknowledged. After a crash, the database replays the log.

BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
UPDATE accounts SET balance = balance + 100 WHERE id = 2;
-- if anything fails before this point, ROLLBACK undoes both updates
COMMIT;

Isolation levels. Perfect isolation (each transaction behaves as if it ran alone) is expensive, so SQL defines weaker levels. Each level permits fewer anomalies:

Level	Dirty read	Non-repeatable read	Phantom read	Typical default in
Read uncommitted	possible	possible	possible	rarely used
Read committed	prevented	possible	possible	PostgreSQL, SQL Server, Oracle
Repeatable read	prevented	prevented	possible (prevented in PostgreSQL)	MySQL InnoDB
Serializable	prevented	prevented	prevented	opt-in

• Dirty read: you see data another transaction has not committed (and may roll back).
• Non-repeatable read: reading the same row twice in one transaction returns different values.
• Phantom read: re-running a range query returns new rows that another transaction inserted.

A classic bug at read committed is the lost update: two requests read stock = 1, both compute stock - 1 = 0 in application code, and both write 0; two items are sold, one is in stock. The fix is to let the database do the arithmetic atomically (UPDATE ... SET stock = stock - 1 WHERE stock > 0), to lock the row (SELECT ... FOR UPDATE), or to use optimistic concurrency with a version number.

Distributed systems and CAP. Once data is replicated across several machines (for availability or scale), a new problem appears: the network between them can fail. The CAP theorem (Eric Brewer, 2000; proved by Gilbert and Lynch, 2002) says that a distributed data store cannot guarantee all three of the following at the same time:

• Consistency (C): every read sees the most recent write (linearizability; not the same "C" as in ACID).
• Availability (A): every request to a non-failing node receives a response.
• Partition tolerance (P): the system keeps working even if messages between nodes are lost.

Partitions will happen in any real network, so P is not optional. The real choice is what to do during a partition:

            Node A  ----X----  Node B          the link between A and B is broken
          (has write v2)     (still has v1)

  A client asks Node B for the value. Node B can:

  CP: refuse to answer (error or timeout)  --> stays consistent, gives up availability
  AP: answer v1 (stale)                    --> stays available, gives up consistency

Choice	Behaviour during a partition	Examples (default configurations)
CP	Minority side rejects writes (and possibly reads) until the partition heals	MongoDB with majority write concern, etcd, ZooKeeper, HBase
AP	Every side keeps accepting reads and writes; conflicts reconciled later	Cassandra, DynamoDB (eventually consistent reads), CouchDB

PACELC (Daniel Abadi) extends CAP with the more common case: if there is a Partition, choose A or C; Else (normal operation), choose between Latency and Consistency. Even without failures, waiting for several replicas to confirm a write costs latency. Many databases let you choose per operation (MongoDB write and read concerns, Cassandra consistency levels).

BASE. Systems that favour availability describe their guarantees as BASE:

• Basically Available: the system answers, even if some data is stale or some parts are degraded.
• Soft state: the state may change over time without new input, as replicas converge.
• Eventual consistency: if writes stop, all replicas eventually return the same value.

An analogy: ACID is a bank counter where the clerk will not let you leave until the ledger is updated and signed. BASE is a social network's "like" counter: different users may briefly see 41 or 42 likes, and nobody is harmed because it converges within a second.

ACID vs BASE.

Aspect	ACID	BASE
Priority	Correctness of every operation	Availability and scale
Consistency	Strong, immediate	Eventual
Typical uses	Payments, inventory, bookings, account balances	Feeds, counters, analytics, product views, caches
Cost	Coordination, locking, latency	Application must tolerate and sometimes reconcile stale or conflicting data

Practical guidance for web applications.

• Identify the few operations that must be strongly consistent (money, stock, unique usernames) and protect them with transactions, atomic updates or unique constraints.
• Accept eventual consistency where users will not notice (counters, recommendation lists, search indexes updated a second later).
• Watch out for read-your-own-writes: a user saves a profile and the next page, read from a lagging replica, shows the old value. Route such reads to the primary or use causal consistency (see "Replica Sets and Sharding").

Key Takeaways

• ACID transactions guarantee atomicity, consistency, isolation and durability; isolation comes in levels that permit different anomalies.
• Lost updates happen when application code reads, modifies and writes; prefer atomic updates, locks or optimistic versioning.
• CAP: during a network partition a replicated store must choose between consistency and availability; PACELC adds the latency/consistency trade-off during normal operation.
• BASE systems are basically available with eventual consistency; they suit data where brief staleness is harmless.
• Most applications mix models: strong consistency for critical operations, eventual consistency elsewhere.

Practice

1. (Beginner) For each feature, state whether it needs strong or eventual consistency and why: checking out a cart, the "views" counter on a video, changing a password, the list of "people you may know".
2. (Intermediate) Write two concurrent scripts that each read a stock value, decrement it in JavaScript and write it back, 1,000 times, against any database. Show the lost updates, then fix it with an atomic update and verify the final count.
3. (Interview) "Is MongoDB CP or AP?" Hint: the answer depends on configuration; discuss primary elections, write concern majority versus w: 1, and reads from secondaries.

Document, Key-Value, Column and Graph Models

A data model is the shape in which a database lets you store and query information. The model decides which questions are cheap to ask. A useful analogy is storage furniture: a filing cabinet (documents), a wall of numbered lockers (key-value), a giant spreadsheet split across rooms (wide-column), and a corkboard with pins and strings (graph). You can store anything in any of them, but each makes certain retrievals effortless and others painful.

Key-value model. The simplest model: a key maps to an opaque value. The store can find a value by its exact key extremely quickly, but it cannot search inside values or ask "which keys have a value greater than 10?" unless it adds extra structures. Redis extends the idea with typed values (lists, hashes, sets, sorted sets) that support server-side operations.

 key                         value
 ------------------------    ----------------------------------------
 session:9f2c...             {"userId":42,"role":"admin"}            (string, JSON-encoded)
 ratelimit:203.0.113.7       17                                      (counter)
 cart:42                     { "sku-1": 2, "sku-9": 1 }              (Redis hash)

Strengths: very low latency, trivial to partition by key. Weaknesses: only lookups by key; relationships and secondary queries are your job. Use for: caches, sessions, feature flags, counters, rate limits.

Document model. Each record is a document: a nested structure of fields, arrays and sub-documents (JSON or a binary form of it, such as BSON). Documents of the same kind are grouped in collections. Unlike key-value stores, the database understands the document's structure: it can query and index any field, including nested ones and array elements.

// One document holds an order with its line items: the unit that is read and written together
{
  _id: 'ord_1029',
  customer: { id: 42, name: 'Ada Lovelace' },
  status: 'paid',
  items: [
    { sku: 'BK-DUNE', qty: 1, price: 9.99 },
    { sku: 'BK-EMMA', qty: 2, price: 7.50 }
  ],
  total: 24.99
}

Strengths: maps naturally to objects in code, no impedance mismatch, flexible shapes, one read for an aggregate. Weaknesses: many-to-many relationships and cross-document consistency need care. Use for: content, catalogues, user profiles, orders, most general application data.

Wide-column (column-family) model. Data is stored in tables with a partition key that decides which node holds the row and clustering columns that sort rows within a partition. Rows in the same table can have different columns. It is optimised for massive write volumes and queries that read one partition in order. Cassandra and ScyllaDB are the main examples. Note that "wide-column" is different from "columnar" analytic databases (ClickHouse, BigQuery), which store each column contiguously to speed up aggregations.

-- Cassandra Query Language (CQL): readings partitioned by sensor, sorted by time
CREATE TABLE readings (
  sensor_id  text,
  day        date,
  ts         timestamp,
  value      double,
  PRIMARY KEY ((sensor_id, day), ts)       -- (partition key), clustering column
) WITH CLUSTERING ORDER BY (ts DESC);

-- Efficient: reads one partition, already sorted
SELECT ts, value FROM readings WHERE sensor_id = 's-17' AND day = '2026-10-03' LIMIT 100;
-- Not allowed without a full scan: WHERE value > 30 (not part of the key)

Strengths: linear write scalability, high availability, time-ordered data. Weaknesses: you must design a table per query; no joins; limited ad hoc queries. Use for: IoT telemetry, event logs, messaging histories at very large scale.

Graph model. Data is stored as nodes (entities) and edges (relationships), both with properties. Traversing a relationship is a pointer hop rather than a join, so queries that follow many hops ("friends of friends who like the same band") stay fast however large the data grows.

 (:User {name:'Ada'}) -[:FOLLOWS]-> (:User {name:'Alan'}) -[:LIKES]-> (:Band {name:'Kraftwerk'})
          |                                                                 ^
          +--------------------------[:LIKES]-------------------------------+

// Cypher (Neo4j): recommend bands liked by people Ada follows, but not yet by Ada
MATCH (me:User {name: 'Ada'})-[:FOLLOWS]->(:User)-[:LIKES]->(b:Band)
WHERE NOT (me)-[:LIKES]->(b)
RETURN b.name, count(*) AS score
ORDER BY score DESC LIMIT 5;

Strengths: deep relationship queries, path finding, recommendations, fraud rings. Weaknesses: less suited to bulk aggregate scans; harder to shard. Use for: social graphs, knowledge graphs, access-control graphs, network topologies.

Other specialised models often found next to the main database:

Model	Examples	Typical web use
Time series	TimescaleDB, InfluxDB, Prometheus	Metrics, monitoring, sensor data
Search engine (inverted index)	Elasticsearch, OpenSearch, Meilisearch	Full-text search, faceted navigation
Vector	pgvector, MongoDB Atlas Vector Search, Qdrant	Semantic search, retrieval for AI features
Object storage	Amazon S3 and compatibles	Files, images, backups

Summary comparison.

Model	Unit of data	Lookup by	Relationships	Scale-out
Key-value	Value behind a key	Key only	None (application)	Easiest
Document	Nested document	Any indexed field	Embedding, references	Good (sharding)
Wide-column	Row in a partition	Partition key + clustering columns	None; one table per query	Excellent
Graph	Node and edge	Pattern traversal	First-class	Harder
Relational	Row in a table	Any indexed column, joins	Foreign keys, joins	Harder for writes

Key Takeaways

• Key-value stores give the fastest lookups by key and little else; Redis adds typed values with server-side operations.
• Document stores keep nested, flexible records and can query and index fields inside them.
• Wide-column stores partition rows by key for enormous write throughput but require one table design per query.
• Graph databases make multi-hop relationship traversal cheap.
• Pick a model by the questions your application asks most often.

Practice

1. (Beginner) Represent a user with two addresses and three roles in each of the four models. Which representation makes "find all admins" easiest, and which makes "load user 42" easiest?
2. (Intermediate) Design a Cassandra-style table for "the last 50 messages in a chat room" and explain your partition and clustering keys. What happens if one room becomes extremely popular?
3. (Interview) "Why are graph databases faster than SQL joins for friend-of-friend queries?" Hint: compare index lookups per join step with index-free adjacency, and consider how cost grows with depth.

Connection Pooling and Drivers

What a driver does. Your application does not speak a database's wire protocol by itself. A driver (client library) does that work: it opens TCP connections, performs TLS and authentication, encodes queries and parameters into the protocol (PostgreSQL's frontend/backend protocol, MongoDB's OP_MSG with BSON, Redis's RESP), decodes results into language values, and handles timeouts, retries and server discovery in clusters. Examples for Node.js: pg (PostgreSQL), mysql2, the official mongodb driver, and redis or ioredis. Higher-level libraries sit on top: query builders (Knex, Kysely), ORMs for relational databases (Prisma, Drizzle, Sequelize, TypeORM) and ODMs for documents (Mongoose, 13.2).

 your code  -->  ODM / ORM (optional)  -->  driver  -->  wire protocol over TCP/TLS  -->  database
 repo.find()     Mongoose, Prisma           mongodb, pg   BSON / PostgreSQL messages

Why connections are expensive. Opening a database connection involves a TCP handshake, a TLS handshake, authentication (often a deliberately slow SCRAM hash exchange), and on PostgreSQL the creation of a whole server process with its own memory. That can take from a few milliseconds on a local network to over 100 ms across regions. A web request that opens and closes its own connection pays that cost every time, and the database pays even more.

Connection pooling. A pool keeps a set of open connections and lends them out. A request borrows a connection, runs its queries, and returns it; the next request reuses it immediately. An analogy is a taxi rank: instead of building a car for every passenger and scrapping it at the destination, a fixed fleet of taxis waits at the rank, and passengers queue when all are busy.

              +----------------- pool (max 10) -----------------+
 request A -->| [conn 1: busy A] [conn 2: busy B] [conn 3: idle] |---> database
 request B -->| [conn 4: idle]   ...                             |
 request C -->| waiting queue: (empty) --> gets conn 3           |
              +--------------------------------------------------+

Node.js is single-threaded, but database work is I/O: while one query waits for the network, the event loop runs other requests. A pool with several connections lets those queries run concurrently on the database.

A pool in practice (PostgreSQL with pg).

// src/db/pg.js: one pool per process, created once at start-up
import pg from 'pg';

export const pool = new pg.Pool({
  connectionString: process.env.DATABASE_URL,
  max: 10,                         // upper bound of open connections from this process
  idleTimeoutMillis: 30_000,       // close connections idle for 30 s
  connectionTimeoutMillis: 2_000,  // fail fast if no connection is free within 2 s
});

pool.on('error', (err) => console.error('idle client error', err)); // e.g. server restarted

// Simple queries: the pool checks out and releases a connection for you
export async function findBook(id) {
  const { rows } = await pool.query('SELECT * FROM books WHERE id = $1', [id]); // parameterised: no SQL injection
  return rows[0] ?? null;
}

// Transactions need the SAME connection for every statement
export async function transfer(from, to, amount) {
  const client = await pool.connect();             // borrow
  try {
    await client.query('BEGIN');
    await client.query('UPDATE accounts SET balance = balance - $1 WHERE id = $2', [amount, from]);
    await client.query('UPDATE accounts SET balance = balance + $1 WHERE id = $2', [amount, to]);
    await client.query('COMMIT');
  } catch (err) {
    await client.query('ROLLBACK');
    throw err;
  } finally {
    client.release();                              // ALWAYS return it, or the pool leaks
  }
}

The MongoDB and Redis drivers manage their pools internally: you create one client per process and reuse it everywhere (see "The Node.js MongoDB Driver" and "Redis Clients in Node.js").

Sizing the pool. Bigger is not better. Each database connection consumes server memory and CPU scheduling; beyond the number of queries the database can execute in parallel (roughly its CPU cores times a small factor), extra connections only add contention. Little's Law gives a first estimate of how many connections are busy on average:

 busy connections = queries per second x average query time (in seconds)

 example: 400 queries/s x 0.005 s = 2 connections busy on average
          size the pool for peaks (say 3-5x) => max 10

Remember that pools multiply: 8 Node.js processes (cluster workers, containers) with max: 20 each can open 160 connections, which exceeds PostgreSQL's default max_connections of 100.

Pool setting	Purpose	Consequence of a bad value
Max size	Cap concurrent connections	Too low: requests queue; too high: database overload
Min size / idle	Keep warm connections ready	Too high: wasted server resources
Acquire timeout	How long a request waits for a free connection	Too long: requests pile up under load instead of failing fast
Idle timeout	Close connections that sit unused	Too long: firewalls silently drop them
Max lifetime	Recycle connections periodically	Helps after failovers and credential rotation

External poolers. When many processes or serverless functions connect to one database, put a pooler in between: PgBouncer or a cloud proxy (for example Amazon RDS Proxy) for PostgreSQL. Thousands of client connections are multiplexed onto a few dozen server connections. Serverless platforms are the extreme case: each function instance creates its own pool, so create the client outside the handler (to reuse it across invocations) and keep max small.

Common pitfalls.

• Creating a new client or pool inside each request handler: connection storms, exhausted server limits.
• Forgetting release() on an error path: the pool slowly leaks until every request times out.
• Long-running work (HTTP calls, file uploads) while holding a connection inside a transaction: a few slow requests block the whole pool.
• Not closing the pool on shutdown: the process hangs or the database sees abrupt disconnects. Call pool.end() / client.close() in the graceful shutdown handler (11.5).

Key Takeaways

• A driver implements the database's wire protocol, authentication, encoding and failover; ORMs and ODMs build on drivers.
• Opening connections is slow and expensive for the database; a pool reuses a bounded set of open connections.
• Create one pool or client per process at start-up, reuse it everywhere, and close it on shutdown.
• Size pools from measured throughput and query time (Little's Law) and remember that every process has its own pool.
• Use an external pooler for many processes or serverless functions.

Practice

1. (Beginner) Write an Express route that creates a new pg.Client per request and another that uses a shared pg.Pool. Load both with autocannon and compare latency and the number of connections shown by SELECT count(*) FROM pg_stat_activity.
2. (Intermediate) Introduce a bug where client.release() is skipped when a query throws. Show how the service degrades under load, then add a metric (pool totalCount, idleCount, waitingCount) that would have revealed the leak.
3. (Interview) "Our API scaled from 2 to 30 containers and the database started refusing connections. What happened and what would you do?" Hint: multiply pool sizes, then consider smaller pools, an external pooler, and whether the database is actually the bottleneck.

Data Modeling for Web Workloads

Start from access patterns, not entities. In a university course you might design a schema by listing entities and normalising them. Web workloads add a sharper question: which reads and writes happen, how often, and how fast must they be? A product page might be read 50,000 times for every edit. A chat message is written once and read a few times. An analytics event is written millions of times and read in aggregate. The schema should make the most frequent operations cheap, even if rarer ones become more expensive.

A practical method:

1. List the screens and API endpoints, and for each the data it needs.
2. Estimate frequency and size: reads/writes per second, records per user, growth per year.
3. Identify consistency requirements (13.1 "ACID vs BASE"): which updates must be atomic together.
4. Shape records so that hot reads need one query (or one index lookup), and hot writes touch as few records as possible.
5. Add indexes for every hot query, and verify them with the query plan.

 Endpoint                         Frequency      Data needed                       Design decision
 -----------------------------    -----------    ------------------------------    ------------------------------------
 GET /products/:slug              very high      product, price, stock, rating     one document/row, rating precomputed
 GET /products?category=&sort=    high           list of summaries, paginated      index (category, price), keyset paging
 POST /reviews                    low            insert review, update rating      transaction or atomic $inc on summary
 GET /users/:id/orders            medium         last 20 orders, summary only      index (userId, createdAt desc)

Denormalisation and computed values. To make hot reads cheap, store data in the shape it is read: copy an author's name into each book, or keep a precomputed ratings.average instead of averaging thousands of reviews on every page view. The price is that writes must update every copy. Denormalise data that is read often and changes rarely; keep data that changes often in one place.

Identifiers. Every record needs a stable ID. The common choices have different trade-offs:

ID type	Example	Pros	Cons
Auto-increment integer	1042	Compact, ordered, readable	Reveals volume, guessable (enumeration attacks), needs one central counter
UUID v4	3f1c0b2e-...	Generated anywhere, not guessable	Random order fragments B-tree indexes, 16 bytes
UUID v7 (RFC 9562)	0192...	Generated anywhere, time-ordered, index-friendly	Embeds creation time
MongoDB ObjectId	66fe1c2a9b1e8a3f4c2d7e10	Generated by driver, roughly time-ordered, 12 bytes	MongoDB-specific, embeds creation time
Natural key	ISBN, email	Meaningful	Can change (emails do), may be personal data

Expose opaque IDs in URLs, and never rely on IDs being unguessable as an access-control measure: always check authorisation (12.3).

Time. Store timestamps in UTC (TIMESTAMPTZ in PostgreSQL, BSON Date in MongoDB, epoch milliseconds in Redis) and convert to local time only for display. Keep createdAt and updatedAt on most records; they cost little and help debugging, syncing and sorting.

Pagination. Lists must be paginated. Offset pagination (LIMIT 20 OFFSET 10000) is simple but the database still walks through the skipped rows, and pages shift when rows are inserted. Keyset (cursor) pagination remembers the last value seen and asks for the next ones, which is fast at any depth:

-- Offset: page 501 reads and discards 10,000 rows
SELECT id, title, created_at FROM posts ORDER BY created_at DESC, id DESC LIMIT 20 OFFSET 10000;

-- Keyset: continue after the last row of the previous page, using an index on (created_at, id)
SELECT id, title, created_at FROM posts
WHERE (created_at, id) < ('2026-09-30T12:00:00Z', 8812)   -- values from the last row the client saw
ORDER BY created_at DESC, id DESC
LIMIT 20;

The API returns an opaque nextCursor (for example the last row's values, base64-encoded) instead of a page number.

Deleting. A hard DELETE is final. A soft delete (deletedAt set to a timestamp) allows undo and auditing, but every query must then filter it out, and unique constraints must account for it. Personal data laws (such as the GDPR's right to erasure) may require real deletion or anonymisation, so decide per data type.

Unbounded growth. Anything that grows without limit inside a single record is a time bomb: an array of all comments inside a post, all events inside a user. Records get slower to read and write, and document stores impose size limits (16 MB in MongoDB). Put unbounded collections in their own table or collection, or split them into buckets (see "Embedding vs Referencing in Schema Design").

Multi-tenancy. SaaS applications serve many customers (tenants) from one system. Options include a tenantId column on every record (simplest, needs strict filtering everywhere), a schema or database per tenant (stronger isolation, more operational work), or a hybrid. With the shared approach, put tenantId first in compound indexes and enforce it in the repository layer, not in each route.

Schema evolution. Requirements change; data outlives code. Relational databases change schemas with migrations: versioned scripts applied in order (tools: Knex, Prisma Migrate, Flyway). Document databases accept new shapes immediately, but old documents keep their old shape, so code must handle both or a migration must rewrite them. A safe pattern for any store is expand and contract:

 1. expand:    add the new field/column (nullable); deploy code that writes both old and new
 2. migrate:   backfill existing records in batches
 3. switch:    deploy code that reads the new field
 4. contract:  stop writing the old field, then remove it

Each step is deployable on its own and can be rolled back, which matters when several versions of the application run at the same time during a rolling deployment.

Key Takeaways

• Design around access patterns: list endpoints, estimate frequency, then shape records and indexes so hot operations are cheap.
• Denormalise data that is read often and changes rarely; precompute expensive aggregates.
• Choose IDs consciously (time-ordered UUIDs or ObjectIds are good defaults); never treat them as secrets.
• Store UTC timestamps, paginate with keysets for deep lists, and avoid unbounded growth inside one record.
• Evolve schemas with migrations and the expand-and-contract pattern so old and new code can coexist.

Practice

1. (Beginner) List the endpoints of a simple recipe-sharing site and, for each, the fields it needs and whether it is read-heavy or write-heavy.
2. (Intermediate) Implement keyset pagination for GET /api/v1/books from 12.3, returning a nextCursor. Explain how you handle ties on the sort field.
3. (Interview) "How would you rename a column used by a live API with zero downtime?" Hint: describe expand and contract, and what happens when old and new application versions run at the same time during a rolling deploy.

<a id="132-mongodb"></a>

13.2 MongoDB

MongoDB is the most widely used document database and a natural partner for JavaScript back ends: its records look like JSON, its queries are JSON objects, and its Node.js driver is maintained by MongoDB itself. This subchapter covers the document model and tooling, the query and update language, schema design and indexing, aggregation, transactions and change streams, replication and sharding, and finally the official Node.js driver and the Mongoose object modelling library.

Document Model, Collections and BSON

The idea. In an application, an order is one object with a customer, a list of items and a total. A relational database splits it into several tables and reassembles it with joins; MongoDB stores it as one document, close to the object your code already uses. The hierarchy is:

 deployment (mongod / replica set / cluster)
   └── database            e.g. "shop"            (like a schema in SQL)
         └── collection    e.g. "orders"          (like a table, but no fixed columns)
               └── document  { _id: ..., ... }    (like a row, but nested and flexible)
                     └── field: value             (value can be a scalar, array or sub-document)

SQL term	MongoDB term
Database	Database
Table	Collection
Row	Document
Column	Field
Primary key	_id field
Index	Index
Join	Embedding, or $lookup in aggregation
ALTER TABLE migration	Not needed to add fields (but existing documents keep their old shape)

Databases and collections are created implicitly the first time you write to them. Documents in the same collection may have different fields, but in practice they share a shape, and you can make the database enforce one (see "JSON Schema validation" below).

The _id field. Every document has an _id that is unique within its collection and cannot be changed after insertion. MongoDB creates a unique index on it automatically. If you do not provide one, the driver generates an ObjectId: 12 bytes made of a 4-byte timestamp (seconds since the Unix epoch), a 5-byte random value unique to the process, and a 3-byte incrementing counter. ObjectIds are therefore unique without coordination and roughly sorted by creation time.

// mongosh
const id = ObjectId();             // ObjectId('66fe1c2a9b1e8a3f4c2d7e10')
id.getTimestamp();                 // ISODate('2024-10-03T...'): creation time is embedded in the first 4 bytes

BSON. MongoDB stores and transmits documents as BSON (Binary JSON). JSON is a text format with only six types; BSON is a binary format with length prefixes (fast to skip over) and a richer type system:

BSON type	JavaScript (Node driver)	Why it matters
Double	number	Default for numbers from JavaScript
32-bit / 64-bit integer	Int32, Long (or number / bigint)	Exact integers, counters
Decimal128	Decimal128	Exact decimal arithmetic for money
String	string	UTF-8
Boolean, Null	boolean, null
Date	Date	Milliseconds since epoch, always UTC; JSON has no date type
ObjectId	ObjectId	Default _id
Binary	Binary / Buffer	Hashes, small files, UUIDs (subtype 4)
Array, Embedded document	Array, Object	Nesting
Regular expression	RegExp

Types matter in queries: { price: 10 } and { price: '10' } do not match the same documents, and a date stored as the string '2026-10-03' cannot be compared with a real Date. When MongoDB data must travel as JSON (for example in exports), it uses Extended JSON, which wraps special types: { "$oid": "66fe..." }, { "$date": "2026-10-03T10:00:00Z" }.

Limits and rules to remember.

• A document can be at most 16 MiB. This limit exists to stop unbounded documents; larger files go to object storage or GridFS.
• Nesting depth is limited to 100 levels.
• Field names should not start with $ or contain ., since those characters have special meaning in queries.
• Field order is preserved, and _id is always first.

JSON Schema validation. A flexible schema does not have to mean an unchecked one. A collection can declare a validator, and the server rejects writes that do not match:

// mongosh: create a collection with server-side validation
db.createCollection('books', {
  validator: {
    $jsonSchema: {
      bsonType: 'object',
      required: ['title', 'isbn', 'price'],
      properties: {
        title: { bsonType: 'string', minLength: 1 },
        isbn:  { bsonType: 'string', pattern: '^[0-9]{13}$' },
        price: { bsonType: ['double', 'int', 'decimal'], minimum: 0 },
        tags:  { bsonType: 'array', items: { bsonType: 'string' } },
      },
    },
  },
  validationLevel: 'strict',     // apply to all inserts and updates ('moderate': only to already-valid documents)
  validationAction: 'error',     // reject invalid writes ('warn': only log them)
});

Special collection types. MongoDB also offers capped collections (fixed size, oldest documents overwritten, useful for logs), time series collections (optimised storage for measurements over time), and clustered collections (documents stored in _id order).

Key Takeaways

• MongoDB stores documents in collections inside databases; collections are created on first write and need no fixed columns.
• Every document has an immutable, unique _id; the default ObjectId embeds a creation timestamp.
• BSON is a binary format with more types than JSON (dates, ObjectIds, 64-bit integers, Decimal128, binary); store values with their real type.
• Documents are limited to 16 MiB; design to avoid unbounded growth.
• $jsonSchema validators let the server enforce a document shape.

Practice

1. (Beginner) Insert three documents into a products collection, one with a price stored as a string. Query { price: { $gt: 5 } } and explain which documents are returned and why.
2. (Intermediate) Add a $jsonSchema validator to products that requires name, a non-negative numeric price and an optional array of string tags. Show the error returned for an invalid insert.
3. (Interview) "Why does MongoDB use BSON instead of storing JSON text?" Hint: consider type richness (dates, integers, binary), traversal speed thanks to length prefixes, and parsing cost.

Installation, Shell and Compass

Ways to run MongoDB.

Option	Best for	Notes
Docker container	Local development, CI	Reproducible, easy to reset
MongoDB Atlas	Managed production, quick start	Hosted by MongoDB on AWS, Azure or Google Cloud; has a free tier
Native install (Community Server)	Servers you manage, learning internals	Packages for Linux, macOS, Windows
Kubernetes operator	Self-managed clusters on Kubernetes	Community or Enterprise operators

Local development with Docker.

# Single node with authentication
docker run -d --name mongo -p 27017:27017 \
  -e MONGO_INITDB_ROOT_USERNAME=root \
  -e MONGO_INITDB_ROOT_PASSWORD=devpassword \
  -v mongo-data:/data/db \
  mongo:8
# -v mongo-data:/data/db: a named volume, so data survives container removal

# Transactions and change streams need a replica set: a single-member one is enough locally
docker run -d --name mongo-rs -p 27017:27017 mongo:8 --replSet rs0 --bind_ip_all
docker exec -it mongo-rs mongosh --eval "rs.initiate()"

A single-member replica set advertises its container hostname, which the host machine cannot resolve. Connect from the host with mongodb://localhost:27017/?directConnection=true, or initiate it with rs.initiate({ _id: 'rs0', members: [{ _id: 0, host: 'localhost:27017' }] }).

A docker-compose.yml keeps the set-up next to the code:

services:
  mongo:
    image: mongo:8
    command: ["--replSet", "rs0", "--bind_ip_all"]
    ports: ["27017:27017"]
    volumes: ["mongo-data:/data/db"]
    healthcheck:                                   # initiates the replica set once, then reports health
      test: ["CMD", "mongosh", "--quiet", "--eval", "try { rs.status().ok } catch (e) { rs.initiate().ok }"]
      interval: 5s
volumes:
  mongo-data:

Connection strings. Applications and tools connect with a URI:

 mongodb://appuser:s3cret@db1.example.com:27017,db2.example.com:27017/shop?replicaSet=rs0&authSource=admin
 └─scheme─┘ └─credentials─┘ └──────────── host list (replica set seeds) ──────┘ └db┘ └──────── options ───────┘

 mongodb+srv://appuser:s3cret@cluster0.ab12c.mongodb.net/shop
 └─ +srv: the host list is looked up in DNS SRV records, and TLS is enabled by default (Atlas uses this form)

Special characters in passwords must be percent-encoded (@ becomes %40). Keep the full URI in an environment variable (11.5), never in source code.

The MongoDB Shell (mongosh). mongosh is a Node.js-based REPL: you can write JavaScript in it, and the API is close to the Node driver's.

// $ mongosh "mongodb://root:devpassword@localhost:27017"
show dbs                                 // list databases
use shop                                 // switch database (created on first write)
show collections
db.books.insertOne({ title: 'Dune', isbn: '9780441172719', price: 9.99 })
db.books.find({ price: { $lt: 20 } })    // results printed in pages; type "it" for more
db.books.countDocuments()
db.books.getIndexes()
db.stats()                               // data size, storage size, number of objects

// Create a least-privilege user for the application
use admin
db.createUser({
  user: 'shopapp',
  pwd: passwordPrompt(),                 // asks interactively instead of leaving it in shell history
  roles: [{ role: 'readWrite', db: 'shop' }],
})

MongoDB Compass. Compass is the official desktop GUI. It is useful for:

• browsing and editing documents, with filters written in the query language;
• schema analysis: sampling a collection to show which fields exist, their types and value distributions;
• the explain plan view, which shows whether a query used an index (see "Indexes and Query Performance");
• the aggregation pipeline builder, which previews the output of each stage;
• creating and dropping indexes, validation rules and import/export of JSON or CSV.

Compass also contains an embedded mongosh. The VS Code extension "MongoDB for VS Code" offers similar features in the editor.

Database Tools. Command-line utilities installed separately:

mongodump   --uri="$MONGO_URL" --out=backup/2026-10-03          # binary BSON backup
mongorestore --uri="$MONGO_URL" backup/2026-10-03
mongoimport --uri="$MONGO_URL" --collection=books --file=books.json --jsonArray
mongoexport --uri="$MONGO_URL" --collection=books --type=csv --fields=title,isbn,price --out=books.csv

Security basics. Always enable authentication, never expose port 27017 to the internet (many data breaches came from open MongoDB instances), use TLS outside localhost, and give the application a user restricted to its own database.

Key Takeaways

• Use Docker for local development and Atlas or a managed operator for production; transactions and change streams need a replica set, even a one-member one.
• Applications connect with a mongodb:// or mongodb+srv:// URI kept in an environment variable.
• mongosh is a JavaScript shell for administration and ad hoc queries; Compass is a GUI with schema analysis, explain plans and a pipeline builder.
• Database Tools (mongodump, mongorestore, mongoimport, mongoexport) handle backups and data transfer.
• Enable authentication, use least-privilege users and never expose the database port publicly.

Practice

1. (Beginner) Start MongoDB with Docker, create a shop database with a books collection from mongosh, and inspect it in Compass.
2. (Intermediate) Convert the container to a single-member replica set using Docker Compose, create an application user with readWrite on shop only, and verify that this user cannot read another database.
3. (Interview) "What is the difference between mongodb:// and mongodb+srv://?" Hint: think about where the host list comes from, default TLS, and what happens when cluster nodes are replaced.

CRUD Operations and Query Operators

Create, Read, Update, Delete. MongoDB operations are methods on a collection, and queries are themselves documents: a filter describes the documents to match. The examples below use mongosh; the Node driver uses the same method names with await.

Create.

db.books.insertOne({ title: 'Dune', author: { name: 'Frank Herbert' }, year: 1965, price: 9.99, tags: ['sci-fi', 'classic'] });
// { acknowledged: true, insertedId: ObjectId('...') }

db.books.insertMany(
  [
    { title: 'Emma', author: { name: 'Jane Austen' }, year: 1815, price: 7.5, tags: ['classic', 'romance'] },
    { title: 'Neuromancer', author: { name: 'William Gibson' }, year: 1984, price: 11, tags: ['sci-fi', 'cyberpunk'] },
    { title: 'Hyperion', author: { name: 'Dan Simmons' }, year: 1989, price: 12.5, tags: ['sci-fi'], stock: 0 },
  ],
  { ordered: false },  // default true: stop at the first error; false: insert all valid documents, then report errors
);

Read. find(filter) returns a cursor over all matches; findOne(filter) returns the first match or null. An empty filter {} matches every document. Several fields in one filter are combined with an implicit AND.

db.books.find({ year: 1965 })                               // equality
db.books.find({ 'author.name': 'Jane Austen' })             // dot notation reaches into sub-documents (quotes required)
db.books.find({ price: { $gte: 8, $lt: 12 } })              // range on one field
db.books.find({ year: { $gt: 1980 }, tags: 'sci-fi' })      // implicit AND across fields
db.books.findOne({ title: 'Emma' })

Query operators.

Category	Operators	Example
Comparison	$eq, $ne, $gt, $gte, $lt, $lte, $in, $nin	{ year: { $in: [1965, 1984] } }
Logical	$and, $or, $nor, $not	{ $or: [{ price: { $lt: 8 } }, { tags: 'classic' }] }
Element	$exists, $type	{ stock: { $exists: true } }, { price: { $type: 'string' } }
Evaluation	$regex, $expr, $mod, $text, $jsonSchema	{ title: { $regex: '^Hyp' } }
Array	$all, $elemMatch, $size	{ tags: { $all: ['sci-fi', 'classic'] } }

// $or with a nested condition
db.books.find({ $or: [{ year: { $lt: 1900 } }, { price: { $gt: 12 } }] })

// $and is needed only when the same field or operator appears twice
db.books.find({ $and: [{ $or: [{ year: 1965 }, { year: 1984 }] }, { $or: [{ price: { $lt: 10 } }, { stock: 0 }] }] })

// $expr compares two fields of the same document, using aggregation expressions
db.products.find({ $expr: { $gt: ['$reserved', '$stock'] } })   // overbooked products

// Case-insensitive prefix search: anchored regexes can use an index; the /i flag usually prevents efficient use
db.books.find({ title: { $regex: '^neu', $options: 'i' } })

Arrays have special matching rules. This is the area that most surprises newcomers:

// Data: { tags: ['sci-fi', 'classic'] }
db.books.find({ tags: 'sci-fi' })                   // MATCHES: the array CONTAINS 'sci-fi'
db.books.find({ tags: ['sci-fi', 'classic'] })      // matches only the EXACT array (same elements, same order)
db.books.find({ tags: { $all: ['classic', 'sci-fi'] } }) // contains both, any order
db.books.find({ tags: { $size: 2 } })               // exactly two elements

// Data: { reviews: [ { user: 'ada', stars: 5 }, { user: 'bob', stars: 2 } ] }
db.books.find({ 'reviews.user': 'bob', 'reviews.stars': 5 })                // MATCHES: conditions may be met by DIFFERENT elements
db.books.find({ reviews: { $elemMatch: { user: 'bob', stars: 5 } } })        // no match: both on the SAME element

Null and missing fields. { stock: null } matches documents where stock is null or does not exist. Use { stock: { $exists: false } } for "missing" and { stock: { $type: 'null' } } for "explicitly null".

Update and delete (overview). Updates take a filter and an update document made of operators (detailed in "Update Operators and Upserts"):

db.books.updateOne({ title: 'Dune' }, { $set: { price: 8.99 } })     // first match only
db.books.updateMany({ tags: 'classic' }, { $inc: { price: 1 } })     // every match
db.books.replaceOne({ title: 'Emma' }, { title: 'Emma', year: 1815, price: 6 }) // whole document replaced (except _id)

db.books.deleteOne({ title: 'Hyperion' })
db.books.deleteMany({ year: { $lt: 1800 } })
db.books.deleteMany({})                                              // deletes EVERYTHING: double-check filters

Counting. countDocuments(filter) counts matches accurately (it runs a query). estimatedDocumentCount() reads collection metadata, which is instant but ignores filters.

Security: query injection. Because filters are objects, passing user input directly can change the query's meaning:

// Express handler, request body: { "email": "a@b.c", "password": { "$ne": null } }
const user = await users.findOne({ email: req.body.email, password: req.body.password }); // matches ANY password!

Validate input types at the boundary (12.3: schemas with zod) so that password must be a string, and never compare plain-text passwords in a query anyway. Libraries such as express-mongo-sanitize strip $-prefixed keys, but validation is the real fix.

Key Takeaways

• Filters are documents; multiple fields mean AND, and dot notation reaches into nested fields.
• Comparison, logical, element, evaluation and array operators cover most queries; $expr compares fields with each other.
• A scalar condition on an array field matches if any element matches; use $elemMatch to apply several conditions to the same element.
• { field: null } matches both null and missing fields.
• Validate input types before building filters to prevent operator injection.

Practice

1. (Beginner) Write queries for: science-fiction books published after 1980, books cheaper than 10 or tagged classic, and books without a stock field.
2. (Intermediate) Create a courses collection where each document has an array of { student, grade }. Write a query that finds courses where the same student has a grade above 28, and explain why the dot-notation version gives different results.
3. (Interview) "How can a NoSQL database be vulnerable to injection if there is no SQL?" Hint: consider what happens when a JSON request body containing an object such as { "$gt": "" } reaches a filter, and how input validation prevents it.

Projection, Sorting, Limiting and Cursors

Why shape and limit results? A list page needs 20 book titles, not 50,000 full documents with reviews and descriptions. Returning less data saves network bandwidth, memory in Node.js, and serialisation time. MongoDB lets you choose which fields (projection), in which order (sort) and how many (skip and limit), and returns results through a cursor so large result sets are never loaded all at once.

Projection. The second argument of find lists fields to include (1) or exclude (0):

db.books.find({ tags: 'sci-fi' }, { title: 1, price: 1 })           // inclusion: title, price and _id
db.books.find({ tags: 'sci-fi' }, { title: 1, price: 1, _id: 0 })   // _id is the only field you may exclude in an inclusion projection
db.books.find({}, { reviews: 0, description: 0 })                   // exclusion: everything except these
// db.books.find({}, { title: 1, reviews: 0 })                      // ERROR: cannot mix inclusion and exclusion

// Array projections
db.books.find({}, { title: 1, reviews: { $slice: -3 } })            // only the last three reviews
db.books.find({ 'reviews.stars': 5 }, { title: 1, 'reviews.$': 1 }) // positional $: the first matching review only
db.books.find({}, { title: 1, 'author.name': 1 })                   // nested field

Projection also supports computed fields using aggregation expressions, for example { title: 1, priceWithVat: { $multiply: ['$price', 1.22] } }.

Sorting. sort takes field names with 1 (ascending) or -1 (descending). The order of keys matters: earlier keys sort first.

db.books.find().sort({ year: -1, title: 1 })      // newest first; same year sorted by title

When no index supports the sort, MongoDB performs an in-memory blocking sort, limited to 100 MB unless it is allowed to spill to disk. Sorting by an indexed field is much faster (see "Indexes and Query Performance"). Comparison is binary by default ('Zebra' sorts before 'apple'); a collation gives language-aware, optionally case-insensitive ordering:

db.books.find().sort({ title: 1 }).collation({ locale: 'en', strength: 2 }) // strength 2: ignore case

Limit and skip.

db.books.find().sort({ year: -1 }).skip(40).limit(20)   // page 3 with 20 items per page

Whatever order you call them in, the server applies sort, then skip, then limit. As explained in 13.1 ("Data Modeling for Web Workloads"), large skip values are slow because skipped documents are still read. Use keyset pagination for deep lists:

// First page
let page = await books.find({}).sort({ _id: -1 }).limit(20).toArray();
// Next page: continue after the last _id the client received
const lastId = page.at(-1)._id;
page = await books.find({ _id: { $lt: lastId } }).sort({ _id: -1 }).limit(20).toArray();

Cursors. find() does not return documents; it returns a cursor, a server-side pointer into the result set. The driver fetches results in batches (the first batch has up to 101 documents; later batches are up to 16 MiB each) using getMore commands as you iterate.

 client                                  server
 find({...})      ---------------------> runs query, returns batch 1 + cursorId
 iterate batch 1
 (batch empty)    --- getMore cursorId -> returns batch 2
 ...
 (no more)        <--------------------- cursorId 0: cursor exhausted and closed

Three ways to consume a cursor in Node.js:

const cursor = books.find({ tags: 'sci-fi' }).project({ title: 1 }).sort({ title: 1 });

// 1. toArray(): loads everything into memory. Fine for bounded results (always use limit())
const firstTwenty = await books.find({}).limit(20).toArray();

// 2. for await: processes one document at a time, constant memory
for await (const doc of cursor) {
  await exportToCsv(doc);
}

// 3. stream(): a Readable stream for piping (11.4), e.g. a large export in an HTTP response
import { pipeline } from 'node:stream/promises';
import { Transform } from 'node:stream';

app.get('/api/export/books.ndjson', async (req, res) => {
  res.type('application/x-ndjson');
  const toLine = new Transform({
    writableObjectMode: true,
    transform(doc, _enc, cb) { cb(null, JSON.stringify(doc) + '\n'); },
  });
  await pipeline(books.find({}).stream(), toLine, res);           // backpressure handled for you
});

Idle cursors are closed by the server after 10 minutes. If you break out of a loop early, the driver closes the cursor for you; with manual iteration, call cursor.close().

Key Takeaways

• Projection selects fields (inclusion or exclusion, not both, except _id); $slice and $ trim arrays.
• sort with several keys sorts by the first, then the next; collations give case-insensitive, locale-aware ordering.
• The server always applies sort, skip, limit in that order; large skips are slow, so prefer keyset pagination.
• find returns a cursor that fetches batches lazily; use limit with toArray, or for await/streams for large results.

Practice

1. (Beginner) Return the titles and prices of the five most expensive books, without _id.
2. (Intermediate) Implement GET /api/v1/books?after=<id>&limit=20 in Express using keyset pagination on _id, and return nextCursor (or null on the last page).
3. (Interview) "A colleague exports a 5-million-document collection with find().toArray() and the process crashes. Why, and how would you fix it?" Hint: consider heap memory, cursor batching, streams and backpressure.

Update Operators and Upserts

Why operators instead of rewriting documents? A naive update reads a document, changes it in JavaScript, and writes it back. Between the read and the write another request may change it too, and one change is lost (the "lost update" from 13.1). Update operators describe the change itself ("increment stock by -1", "add this tag"), and MongoDB applies it atomically on the server: a single-document update is never interleaved with another update to the same document.

Field operators.

Operator	Effect	Example
$set	Set field(s), creating them if missing	{ $set: { price: 8.99, 'author.country': 'US' } }
$unset	Remove field(s)	{ $unset: { discount: '' } }
$inc	Add a number (negative to subtract)	{ $inc: { stock: -1, sold: 1 } }
$mul	Multiply	{ $mul: { price: 1.1 } }
$min / $max	Update only if the new value is lower / higher	{ $max: { highScore: 870 } }
$rename	Rename a field	{ $rename: { 'qty': 'quantity' } }
$currentDate	Set to the server's current date	{ $currentDate: { updatedAt: true } }
$setOnInsert	Set only when an upsert inserts	{ $setOnInsert: { createdAt: new Date() } }

Array operators.

Operator	Effect
$push	Append (with modifiers $each, $slice, $sort, $position)
$addToSet	Append only if not already present
$pull	Remove all elements matching a value or condition
$pop	Remove the first (-1) or last (1) element
$	Positional: update the first element matched by the filter
$[]	Update all elements
$[name]	Update elements matching arrayFilters

// Atomic stock reservation: the filter and the change are applied together, so stock never goes negative
const result = await products.updateOne(
  { _id: productId, stock: { $gte: qty } },             // guard condition
  { $inc: { stock: -qty }, $currentDate: { updatedAt: true } },
);
if (result.modifiedCount === 0) throw new HttpError(409, 'out of stock');

// Tags without duplicates
await books.updateOne({ _id: id }, { $addToSet: { tags: { $each: ['award-winner', 'classic'] } } });

// Keep only the 10 most recent activity entries (a bounded array)
await users.updateOne(
  { _id: userId },
  { $push: { recentActivity: { $each: [{ type: 'login', at: new Date() }], $sort: { at: -1 }, $slice: 10 } } },
);

// Remove all reviews by a user
await books.updateMany({}, { $pull: { reviews: { user: 'spammer42' } } });

// Positional $: change the quantity of the item matched in the filter
await carts.updateOne({ _id: cartId, 'items.sku': 'BK-DUNE' }, { $inc: { 'items.$.qty': 1 } });

// Filtered positional: mark every unread notification older than a week as read
await users.updateOne(
  { _id: userId },
  { $set: { 'notifications.$[n].read': true } },
  { arrayFilters: [{ 'n.read': false, 'n.at': { $lt: new Date(Date.now() - 7 * 864e5) } }] },
);

Update results. updateOne and updateMany return { matchedCount, modifiedCount, upsertedId, upsertedCount }. matchedCount may be 1 while modifiedCount is 0 if the new values equal the old ones.

Returning the updated document. findOneAndUpdate updates atomically and returns the document, before or after the change. Use it when you need the new state, for example to generate sequential order numbers:

// Node driver (v6+): returns the document directly (or null)
const counter = await db.collection('counters').findOneAndUpdate(
  { _id: 'orderNumber' },
  { $inc: { value: 1 } },
  { upsert: true, returnDocument: 'after' },    // 'before' is the default
);
const orderNumber = counter.value;               // 1, 2, 3, ... never duplicated

Upserts. With { upsert: true }, an update that matches nothing inserts a new document built from the equality conditions of the filter plus the update. It means "update if exists, otherwise insert" in a single atomic operation, ideal for counters, "last seen" records and idempotent imports:

// Daily page-view counter: one document per (page, day), created on the first view
await db.collection('pageViews').updateOne(
  { page: '/books/dune', day: '2026-10-03' },          // equality fields become part of the new document
  {
    $inc: { views: 1 },
    $setOnInsert: { firstViewAt: new Date() },          // only written when the document is created
  },
  { upsert: true },
);

Upsert races. Two concurrent upserts with the same filter may both find nothing and both insert, creating duplicates. Prevent this with a unique index on the filter fields ({ page: 1, day: 1 }, unique). With that index in place, MongoDB retries a conflicting upsert internally as an update, so both increments are counted.

Replacement vs update. replaceOne(filter, doc) and an update document without operators replace the whole document; fields not listed disappear. Most of the time you want $set.

Update pipelines. Updates can also be an array of aggregation stages, which lets the new value depend on other fields of the same document:

await orders.updateMany(
  { status: 'paid', total: { $exists: false } },
  [{ $set: { total: { $sum: { $map: { input: '$items', in: { $multiply: ['$$this.qty', '$$this.price'] } } } } } }],
);

Key Takeaways

• Update operators apply changes atomically on the server; prefer them over read-modify-write in application code.
• Put guard conditions in the filter (stock: { $gte: qty }) to make conditional updates safe under concurrency.
• Array operators ($push with $slice, $addToSet, $pull, positional $, $[], arrayFilters) modify arrays in place.
• findOneAndUpdate returns the document before or after the update; upsert: true inserts when nothing matches.
• Back upserts with a unique index to avoid duplicates under concurrent requests.

Practice

1. (Beginner) Write updates that raise every classic book's price by 10%, add the tag sale without duplicates, and remove a discount field.
2. (Intermediate) Implement POST /api/cart/items that adds a SKU to a user's cart or increments its quantity if it is already there, in at most two atomic operations. Test it with concurrent requests.
3. (Interview) "How would you implement a 'like' button so a user can like a post only once and the counter stays accurate?" Hint: consider a separate likes collection with a unique index on (postId, userId) plus $inc on the post, versus $addToSet on an array, and what happens at a million likes.

Embedding vs Referencing in Schema Design

The central design decision. In MongoDB, related data can be stored embedded (inside the same document) or referenced (in another document, linked by ID). Relational design asks "how do I avoid duplication?"; document design asks "what data is read and written together?". Data accessed together should be stored together.

An analogy: embedding is a printed recipe card listing every ingredient; referencing is a recipe that says "see the sauce on page 42". The card is faster to use, but if the sauce recipe changes, every card that copied it must be reprinted.

// Embedded: the post and its few tags and the author summary are one unit
{ _id: 1, title: 'Hello', author: { _id: 42, name: 'Ada' }, tags: ['intro'], body: '...' }

// Referenced: comments live in their own collection and point back to the post
// posts:    { _id: 1, title: 'Hello', ... }
// comments: { _id: 9001, postId: 1, user: 'bob', text: 'Nice!', at: ISODate(...) }

Decision guide.

Embed when...	Reference when...
The child data is always (or usually) read with the parent	The child data is often accessed on its own
The relationship is one-to-one or one-to-few	The relationship is one-to-many with many, or unbounded
The child belongs to one parent only	The child is shared by many parents (many-to-many)
The child changes together with the parent, or rarely	The child is updated often and independently
You need atomic updates across parent and child	The combined document would approach 16 MiB

Relationship cardinality gives a useful rule of thumb:

 one-to-few       (user -> 3 addresses)            embed an array in the parent
 one-to-many      (product -> 500 reviews)         reference: child stores parentId (or a bounded array of IDs in the parent)
 one-to-squillions (server -> billions of log lines) reference: child stores parentId; NEVER an array in the parent
 many-to-many     (students <-> courses)           arrays of IDs on one or both sides, or a linking collection

Reading referenced data. Either run two queries in the application, or join on the server with $lookup in an aggregation (see "The Aggregation Pipeline"):

// Two queries (simple, uses indexes, often fastest)
const post = await posts.findOne({ _id: postId });
const latest = await comments.find({ postId }).sort({ at: -1 }).limit(20).toArray();   // index { postId: 1, at: -1 }

// One round trip with $lookup
const [postWithComments] = await posts.aggregate([
  { $match: { _id: postId } },
  { $lookup: {
      from: 'comments',
      localField: '_id',
      foreignField: 'postId',
      pipeline: [{ $sort: { at: -1 } }, { $limit: 20 }],     // only the latest comments
      as: 'comments',
  } },
]).toArray();

Schema design patterns. MongoDB's documentation names recurring solutions:

Pattern	Problem	Solution
Extended reference	You always need the author's name with a book	Store { authorId, name } in the book; update copies when the name changes (rare)
Subset	A product has 5,000 reviews but the page shows 10	Embed the 10 most recent reviews; keep all reviews in a reviews collection
Computed	Averaging reviews on every view is expensive	Store ratings: { count, sum, average } and update it with $inc when a review is added
Bucket	One document per sensor reading creates billions of tiny documents	Group readings per sensor per hour in one document with an array (bounded by time)
Outlier	A few celebrities have millions of followers; most users have hundreds	Embed for normal users; add hasOverflow: true and overflow documents for outliers
Schema versioning	Old and new document shapes coexist	Add schemaVersion and let code handle each version, migrating lazily
Polymorphic	Different product types have different attributes	One collection, common fields plus type-specific fields, with a type discriminator

A worked example: a blog.

// users
{ _id: ObjectId('...42'), username: 'ada', displayName: 'Ada L.', bio: '...' }

// posts: author summary embedded (extended reference), tags embedded (few), counters computed
{
  _id: ObjectId('...p1'),
  slug: 'hello-mongodb',
  title: 'Hello MongoDB',
  author: { _id: ObjectId('...42'), displayName: 'Ada L.' },   // copied; refreshed if the name changes
  tags: ['mongodb', 'intro'],
  stats: { comments: 132, likes: 950 },                        // computed: updated with $inc
  recentComments: [ /* the 3 latest: subset pattern */ ],
  publishedAt: ISODate('2026-10-01T09:00:00Z'),
}

// comments: unbounded, so referenced by postId
{ _id: ObjectId('...c1'), postId: ObjectId('...p1'), author: { _id: ..., displayName: 'Bob' }, text: '...', at: ISODate(...) }

The article page needs one query for the post and one for the next comments page. Adding a comment is an insert plus a $inc/$push on the post, which can be wrapped in a transaction if the counters must be exact.

Duplication is a decision, not an accident. For every copied field, write down who updates the copies and how quickly they must converge. An author's display name in old posts can be updated by a background job; a price in a past order should never change, because the order must record the price actually paid.

Key Takeaways

• Store together what is read together; embed for one-to-few, owned, co-changing data; reference for large, shared or independently accessed data.
• Never embed unbounded arrays; the one-to-squillions side stores the parent's ID.
• Join referenced data with two indexed queries or $lookup.
• Patterns such as extended reference, subset, computed, bucket and schema versioning solve recurring performance problems.
• Every duplicated field needs an explicit update strategy; some copies (prices in orders) must deliberately never change.

Practice

1. (Beginner) For an online course platform (courses, lessons, students, enrolments, quiz answers), decide for each relationship whether to embed or reference, and justify it with cardinality and access patterns.
2. (Intermediate) Apply the bucket pattern to store temperature readings every 10 seconds from 1,000 sensors. Show the document shape and the update that appends a reading, and estimate the number of documents per day with and without the pattern.
3. (Interview) "Your product documents embed all reviews and some now exceed 16 MiB. How do you migrate without downtime?" Hint: combine the subset and computed patterns with a reviews collection, and apply expand and contract (13.1).

Indexes and Query Performance

Why indexes. Without an index, MongoDB answers a query by reading every document in the collection: a collection scan (COLLSCAN). That is fine for 1,000 documents and unacceptable for 10 million. An index is a separate, sorted data structure (a B-tree) that maps field values to document locations, like the index at the back of a book: to find "replication" you jump to the letter R instead of reading every page.

 index on { year: 1 }                        collection (unordered on disk)
 +-------+-----------+                       +--------------------------------+
 | 1815  | -> rec 7  |                       | rec 3: Neuromancer, 1984 ...   |
 | 1965  | -> rec 12 |   query year: 1984    | rec 7: Emma, 1815 ...          |
 | 1984  | -> rec 3  |  ---------------->    | rec 12: Dune, 1965 ...         |
 | 1989  | -> rec 21 |   one B-tree lookup   | rec 21: Hyperion, 1989 ...     |
 +-------+-----------+                       +--------------------------------+

Indexes speed up filters, sorts and some aggregations, but each one costs disk space and RAM, and every insert, update and delete must update every relevant index. Index what your queries need, not every field.

Creating indexes.

db.books.createIndex({ isbn: 1 }, { unique: true })                  // single field, uniqueness enforced
db.books.createIndex({ 'author.name': 1, year: -1 })                // compound, on a nested field
db.books.createIndex({ tags: 1 })                                   // multikey: one entry per array element
db.sessions.createIndex({ expiresAt: 1 }, { expireAfterSeconds: 0 }) // TTL: delete when expiresAt is reached
db.users.createIndex(
  { email: 1 },
  { unique: true, partialFilterExpression: { deletedAt: { $exists: false } } }, // unique among non-deleted users only
)
db.books.createIndex({ title: 'text', description: 'text' })        // full-text (one text index per collection)
db.places.createIndex({ location: '2dsphere' })                     // geospatial queries ($near, $geoWithin)
db.events.createIndex({ userId: 'hashed' })                         // hashed: used for hashed sharding
db.products.createIndex({ 'attributes.$**': 1 })                    // wildcard: arbitrary attribute names

Index type	Use
Single field	Equality and range on one field
Compound	Queries on several fields, and sorts
Multikey	Fields containing arrays (created automatically when an indexed field holds an array)
Unique	Enforce uniqueness (duplicate writes fail with error code 11000)
Partial	Index only documents matching a filter: smaller, and allows conditional uniqueness
Sparse	Index only documents that have the field (partial indexes are more flexible)
TTL	Automatically delete documents after a time; a background task runs about every 60 seconds
Text	Simple full-text search with stemming (Atlas Search is far more capable)
2dsphere	Geospatial queries on GeoJSON
Hashed	Even distribution for sharding; equality only
Wildcard	Unknown or highly variable field names

Compound indexes and the ESR rule. A compound index is sorted by its first field, then by the second within equal first values, and so on, like a phone book sorted by surname then first name. It can serve queries on any prefix of its fields: an index on { a: 1, b: 1, c: 1 } supports queries on a, on a, b, and on a, b, c, but not on b alone. When designing compound indexes, order fields by the ESR rule:

1. Equality fields first (exact matches).
2. Sort fields next.
3. Range fields last ($gt, $lt, $in on many values, regex).

// Query: books by author, published after 1980, newest first
db.books.find({ 'author.name': 'Dan Simmons', year: { $gt: 1980 } }).sort({ year: -1 })

// Good: equality (author.name), then sort/range on year
db.books.createIndex({ 'author.name': 1, year: -1 })

// Query: in-stock products of a category sorted by price, price filter as range
db.products.find({ category: 'books', inStock: true, price: { $lt: 20 } }).sort({ rating: -1 })
db.products.createIndex({ category: 1, inStock: 1, rating: -1, price: 1 })   // E, E, S, R

Reading a query plan. explain('executionStats') shows what the server actually did:

db.books.find({ 'author.name': 'Dan Simmons' }).sort({ year: -1 }).explain('executionStats')

 winningPlan:
   stage: FETCH                       <- loads full documents
     inputStage: IXSCAN               <- used an index (COLLSCAN would mean "no index")
       indexName: author.name_1_year_-1
 executionStats:
   nReturned: 12                      <- documents returned
   totalKeysExamined: 12              <- index entries read
   totalDocsExamined: 12              <- documents read
   executionTimeMillis: 1

The health check: totalDocsExamined should be close to nReturned. A ratio of 100,000 to 12 means the index is missing or poorly ordered. A SORT stage in the plan means an in-memory sort that an index could avoid.

Covered queries. If every filtered and projected field is in the index (and _id is excluded), MongoDB answers from the index alone without fetching documents (totalDocsExamined: 0):

db.books.createIndex({ isbn: 1, title: 1 })
db.books.find({ isbn: '9780441172719' }, { _id: 0, isbn: 1, title: 1 })   // covered

Finding slow queries.

• The server logs operations slower than 100 ms by default; the database profiler (db.setProfilingLevel(1, { slowms: 50 })) records them in system.profile.
• $indexStats shows how often each index is used, which helps find indexes worth dropping.
• Atlas offers a Performance Advisor that suggests indexes.
• Hide an index (db.books.hideIndex('year_1')) to test the effect of removing it without rebuilding it later.

Memory: the working set. MongoDB's storage engine (WiredTiger) caches data and indexes in RAM. Performance stays good while the working set (the frequently used indexes and documents) fits in memory; when it does not, queries wait for disk.

Key Takeaways

• Without an index, queries scan the whole collection; B-tree indexes make lookups and sorts fast at the cost of write speed and memory.
• MongoDB offers unique, compound, multikey, partial, TTL, text, geospatial, hashed and wildcard indexes.
• Compound indexes serve their prefixes; order fields by Equality, Sort, Range.
• Use explain('executionStats'): look for IXSCAN, no blocking SORT, and totalDocsExamined close to nReturned.
• Keep the working set in RAM and remove unused indexes.

Practice

1. (Beginner) Insert 500,000 generated books, run a query on year with explain('executionStats'), add an index and compare totalDocsExamined and execution time.
2. (Intermediate) For the query find({ status: 'active', country: 'IT', age: { $gte: 18 } }).sort({ signupAt: -1 }), design the best compound index with the ESR rule and verify with explain that no in-memory sort occurs.
3. (Interview) "Why not just index every field?" Hint: consider write amplification, RAM usage for the working set, index build time, and the query planner's choices.

The Aggregation Pipeline

Why aggregation. find retrieves documents; it cannot answer "revenue per month per category" or "top 10 authors by average rating". The aggregation pipeline processes documents through a sequence of stages, each transforming the stream of documents and passing the result to the next, like an assembly line or a Unix pipe (cat | grep | sort | uniq -c).

 orders --> [$match status: paid] --> [$unwind items] --> [$group by category] --> [$sort by revenue] --> results
             filter (uses index)       one doc per item     sum per group            order

Common stages.

Stage	Purpose	SQL analogy
$match	Filter documents (same syntax as find)	WHERE / HAVING
$project, $set ($addFields), $unset	Reshape, compute or remove fields	SELECT expressions
$group	Group by a key and compute accumulators	GROUP BY
$sort, $limit, $skip	Order and paginate	ORDER BY, LIMIT, OFFSET
$unwind	One output document per array element	Joining with a child table
$lookup	Join with another collection	LEFT OUTER JOIN
$facet	Several sub-pipelines on the same input	Several queries at once
$bucket, $bucketAuto	Group into ranges	CASE ranges + GROUP BY
$count	Count documents	COUNT(*)
$out, $merge	Write results to a collection	INSERT INTO ... SELECT

Accumulators used inside $group: $sum, $avg, $min, $max, $first, $last, $push (collect into an array), $addToSet, $count, $top, $topN.

Expressions. Inside stages, '$field' refers to a field's value, and operators such as $add, $multiply, $cond, $dateToString, $concat, $size, $map and $filter compute new values.

A complete example: monthly revenue per category.

// orders: { _id, status, createdAt, items: [ { sku, category, qty, price } ] }
const report = await db.collection('orders').aggregate([
  { $match: {                                          // 1. filter first: can use an index on { status, createdAt }
      status: 'paid',
      createdAt: { $gte: new Date('2026-01-01'), $lt: new Date('2027-01-01') },
  } },
  { $unwind: '$items' },                               // 2. one document per line item
  { $group: {                                          // 3. group by (month, category)
      _id: {
        month: { $dateToString: { format: '%Y-%m', date: '$createdAt', timezone: 'Europe/Rome' } },
        category: '$items.category',
      },
      revenue: { $sum: { $multiply: ['$items.qty', '$items.price'] } },
      units:   { $sum: '$items.qty' },
      orders:  { $addToSet: '$_id' },                  // distinct orders containing this category
  } },
  { $set: { orderCount: { $size: '$orders' } } },     // 4. computed field
  { $unset: 'orders' },
  { $sort: { '_id.month': 1, revenue: -1 } },          // 5. order the output
]).toArray();

// [ { _id: { month: '2026-01', category: 'books' }, revenue: 18234.5, units: 2104, orderCount: 1630 }, ... ]

Joins with $lookup.

// Top 5 authors by average rating, with author details from another collection
await db.collection('books').aggregate([
  { $match: { 'ratings.count': { $gte: 50 } } },
  { $group: { _id: '$authorId', avgRating: { $avg: '$ratings.average' }, books: { $sum: 1 } } },
  { $sort: { avgRating: -1 } },
  { $limit: 5 },                                         // limit BEFORE the join: only 5 lookups
  { $lookup: { from: 'authors', localField: '_id', foreignField: '_id', as: 'author' } },
  { $unwind: '$author' },                                // $lookup always returns an array
  { $project: { _id: 0, name: '$author.name', avgRating: { $round: ['$avgRating', 2] }, books: 1 } },
]).toArray();

Faceted search in one query. $facet computes the result page and filter counts for a catalogue page at once:

await db.collection('products').aggregate([
  { $match: { category: 'books', price: { $lte: 30 } } },
  { $facet: {
      results:    [{ $sort: { rating: -1 } }, { $skip: 0 }, { $limit: 20 }, { $project: { name: 1, price: 1 } }],
      byTag:      [{ $unwind: '$tags' }, { $group: { _id: '$tags', count: { $sum: 1 } } }, { $sort: { count: -1 } }],
      priceRange: [{ $bucket: { groupBy: '$price', boundaries: [0, 10, 20, 31], default: 'other' } }],
      total:      [{ $count: 'value' }],
  } },
]).toArray();

Materialised views with $merge. Expensive reports can be computed periodically and stored, so the API reads precomputed results:

// Nightly job: upsert daily sales per product into a summary collection
await db.collection('orders').aggregate([
  { $match: { status: 'paid', createdAt: { $gte: startOfYesterday, $lt: startOfToday } } },
  { $unwind: '$items' },
  { $group: { _id: { day: startOfYesterday, sku: '$items.sku' }, units: { $sum: '$items.qty' } } },
  { $merge: { into: 'dailySales', on: '_id', whenMatched: 'replace', whenNotMatched: 'insert' } },
]).toArray();

Performance rules.

• Put $match (and $sort + $limit) as early as possible: only stages at the start of the pipeline can use indexes, and every later stage processes fewer documents.
• Project away large fields you do not need before $group or $lookup.
• Make the foreignField of $lookup indexed.
• Each stage may use up to 100 MB of RAM; beyond that it spills to temporary files on disk if allowed (allowDiskUse, enabled by default in recent versions), which is slower.
• Use explain on aggregations too: db.orders.explain('executionStats').aggregate([...]).

Key Takeaways

• An aggregation pipeline is a sequence of stages, each transforming the document stream; it answers analytical questions that find cannot.
• $match, $group, $project/$set, $sort, $unwind and $lookup cover most reports; accumulators compute sums, averages and collections.
• $facet runs several sub-pipelines on the same input; $merge writes results to a collection for materialised views.
• Filter and limit early, so indexes are used and later stages process fewer documents.

Practice

1. (Beginner) Count books per tag and return the five most common tags.
2. (Intermediate) From an orders collection, compute for each customer the number of orders, total spent and date of last order, and return the top 10 customers by total spent with their names joined from users.
3. (Interview) "An aggregation runs in 40 seconds. How do you investigate?" Hint: run explain, check whether the first stage uses an index, look at stage order, the size of documents flowing through $group and $lookup, and whether results could be precomputed with $merge.

Transactions and Change Streams

Single-document atomicity first. Every write to a single document in MongoDB is atomic, including updates to embedded arrays and sub-documents. A well-designed schema (store together what changes together) makes most operations single-document, which is why MongoDB applications need transactions less often than relational ones.

When you need a multi-document transaction. Some operations inherently touch several documents or collections and must succeed or fail as a unit: moving money between two accounts, creating an order while decrementing stock for each item, or transferring ownership of a resource. Since version 4.0 (replica sets) and 4.2 (sharded clusters), MongoDB supports multi-document ACID transactions with snapshot isolation.

Requirements and limits.

• A replica set or sharded cluster (a single-member replica set works for development).
• Operations must pass the same session object.
• A transaction should be short: by default it is aborted after 60 seconds, and it holds resources while open.
• Two transactions writing the same document cause a write conflict; one of them aborts with a TransientTransactionError label and should be retried.

The withTransaction helper. The Node.js driver's withTransaction runs your callback, commits, and automatically retries the whole callback on transient errors and the commit on UnknownTransactionCommitResult:

// Create an order and reserve stock for every item, all or nothing
export async function placeOrder(client, userId, items) {
  const db = client.db('shop');
  const session = client.startSession();
  let orderId;
  try {
    await session.withTransaction(async () => {
      for (const { sku, qty } of items) {
        const res = await db.collection('products').updateOne(
          { sku, stock: { $gte: qty } },
          { $inc: { stock: -qty } },
          { session },                                          // every operation must carry the session
        );
        if (res.modifiedCount === 0) {
          throw new HttpError(409, `insufficient stock for ${sku}`); // a non-transient error aborts and is rethrown
        }
      }
      const { insertedId } = await db.collection('orders').insertOne(
        { userId, items, status: 'pending', createdAt: new Date() },
        { session },
      );
      orderId = insertedId;                                      // assigned on every attempt; the last one committed
    }, {
      readConcern: { level: 'snapshot' },
      writeConcern: { w: 'majority' },                           // committed on a majority of replicas
    });
    return orderId;
  } finally {
    await session.endSession();
  }
}

Important caveats.

• The callback may run more than once. Do not perform side effects outside the database inside it (sending emails, calling payment APIs); do them after the transaction commits.
• Do not run operations of the same transaction in parallel with Promise.all; run them sequentially.
• Transactions are slower than single-document writes. If you find yourself using one for every request, revisit the schema.

Change streams: reacting to changes. Many features need to react when data changes: invalidate a cache, push an update to browsers over WebSockets (10.5), sync a search index, or trigger a workflow. Polling ("anything new since my last check?") is wasteful and slow. A change stream lets an application subscribe to inserts, updates, replaces and deletes on a collection, database or whole deployment. It is built on the replica set's oplog (operation log), so it requires a replica set.

// Push stock changes to connected clients (e.g. with Socket.IO, 10.5)
const products = client.db('shop').collection('products');

const stream = products.watch(
  [{ $match: { operationType: 'update', 'updateDescription.updatedFields.stock': { $exists: true } } }], // server-side filter
  { fullDocument: 'updateLookup' },                // include the current full document, not just the delta
);

for await (const change of stream) {
  // change: { _id: <resume token>, operationType: 'update', documentKey: { _id }, fullDocument, updateDescription, ... }
  io.to(`product:${change.documentKey._id}`).emit('stock', { stock: change.fullDocument.stock });
  await saveResumeToken(change._id);               // persist progress so a restart can continue where it stopped
}

Resuming. Each event carries a resume token. If the process restarts, pass the last saved token with resumeAfter (or startAfter) and the stream continues without missing events, as long as the token is still within the oplog window (the oplog is a capped collection with limited history; size it for your longest expected outage).

const token = await loadResumeToken();
const stream = products.watch([], token ? { resumeAfter: token } : {});

Change stream use cases and alternatives.

Use case	How
Cache invalidation	Delete the Redis key for the changed document (13.3)
Real-time UI	Forward changes to WebSocket rooms
Search sync	Upsert changed documents into Elasticsearch/OpenSearch
Outbox / event publishing	Watch an outbox collection written in the same transaction as business data, then publish to a message broker
Auditing	Copy change events to an audit collection

In a horizontally scaled service, make sure only one instance processes a given stream (or that processing is idempotent), otherwise each event is handled once per instance.

Key Takeaways

• Single-document writes are always atomic; good schema design keeps most operations single-document.
• Multi-document transactions require a replica set, a shared session on every operation, and should be short.
• session.withTransaction retries on transient errors, so the callback must be free of external side effects.
• Change streams deliver real-time change events from the oplog; persist resume tokens to continue after restarts.
• Change streams power cache invalidation, real-time updates, search synchronisation and the outbox pattern.

Practice

1. (Beginner) Implement a money transfer between two accounts documents with withTransaction, and show that a thrown error halfway through leaves both balances unchanged.
2. (Intermediate) Build a small service that watches the products collection and deletes the corresponding Redis cache key on every update or delete. Stop it, update some products, restart it with the saved resume token and verify that no change was missed.
3. (Interview) "How do you reliably publish an event to a message queue when an order is created, without losing events or publishing events for orders that were rolled back?" Hint: the transactional outbox pattern with a change stream, and why "write to DB then publish" in application code is not enough.

Replica Sets and Sharding

MongoDB scales and survives failures with two complementary mechanisms: replication copies the same data to several servers (availability and read scaling), and sharding splits different data across servers (capacity and write scaling).

Replica sets. A replica set is a group of mongod processes holding the same data. One member is the primary and receives all writes; it records them in the oplog. The secondaries copy the oplog asynchronously and apply the same operations.

                     writes + reads (default)
   application  ----------------------------------->  PRIMARY
        |                                              | oplog
        |  reads with readPreference=secondary         v
        +-----------------------------------> SECONDARY  SECONDARY
                                                (copy and replay the primary's oplog)

   heartbeats every 2 s between all members; if the primary is unreachable for ~10 s,
   the secondaries hold an ELECTION and one becomes the new primary

• Automatic failover. If the primary fails, the remaining members elect a new one, usually within about 10–12 seconds. Drivers discover the new primary and retryable writes (on by default) resend a write that failed due to the failover once.
• Majority. Elections need a majority of voting members, so deploy an odd number (3 or 5). With 3 members, one can fail; with 5, two can. A partitioned minority cannot elect a primary, which is how MongoDB avoids two primaries (a CP behaviour, 13.1).
• Special members: hidden (not visible to clients, for backups or analytics), delayed (applies the oplog with a delay, protection against accidental deletes), arbiter (votes but holds no data; generally discouraged).

Write concern: how durable must a write be before it is acknowledged?

Write concern	Acknowledged when	Risk
{ w: 0 }	Not acknowledged at all	Errors are invisible
{ w: 1 }	The primary applied it	Lost if the primary fails before replicating (rolled back)
{ w: 'majority' } (default since 5.0)	A majority of members have it	Survives failover
{ w: 'majority', j: true }	... and written to the on-disk journal	Survives full power loss of the majority

Read preference: which member serves reads?

Read preference	Reads from	Consideration
primary (default)	Primary only	Always the latest committed data
primaryPreferred	Primary, or a secondary if none	Stays available during elections
secondary / secondaryPreferred	Secondaries	Offloads analytics, but data may be stale (replication lag)
nearest	Lowest-latency member	Multi-region deployments

Read concern controls which version of data a read sees: local (latest on that member, might be rolled back), majority (acknowledged by a majority, never rolled back), linearizable (strongest, slower), snapshot (for transactions). Causally consistent sessions guarantee read-your-own-writes even when reading from secondaries.

const client = new MongoClient(process.env.MONGO_URL, {
  readPreference: 'primary',
  writeConcern: { w: 'majority', wtimeoutMS: 5000 },
  retryWrites: true,
});

// Analytics query on a secondary: tolerates a few seconds of lag, protects the primary
const report = await db.collection('orders').aggregate(pipeline, { readPreference: 'secondaryPreferred' }).toArray();

Replication is not a backup. A deleteMany({}) replicates to every member within milliseconds. Keep real backups (snapshots, mongodump, Atlas continuous backup with point-in-time restore) and test restoring them.

Sharding. When data or write volume outgrows what one replica set can handle, MongoDB shards a collection: documents are distributed across several replica sets (shards) according to a shard key.

                  application (driver)
                          |
                  +-------v-------+        +-----------------------+
                  |    mongos     |<------>| config servers (RS)   |  metadata: which key ranges live where
                  |  (router)     |        +-----------------------+
                  +---+-------+---+
                      |       |  routes each query using the shard key
             +--------v-+   +-v--------+   +----------+
             | shard A  |   | shard B  |   | shard C  |     each shard is itself a replica set
             | keys a-h |   | keys i-q |   | keys r-z |
             +----------+   +----------+   +----------+
                 ^ the balancer moves key ranges between shards to keep them even

Choosing a shard key is the most important (and historically hardest to change) decision:

Property	Why	Bad example
High cardinality	Enough distinct values to spread across many shards	country (few values)
Even frequency	No single value receives most of the data or traffic	status where 95% are 'active'
Non-monotonic writes	New documents should not all go to the same shard	createdAt or ObjectId with ranged sharding (all inserts hit the "last" shard)
Matches queries	Queries that include the key go to one shard (targeted); others go to all shards (scatter-gather)	Sharding by _id while most queries filter by tenantId

Strategies: ranged sharding (good for range queries on the key), hashed sharding (spreads monotonically increasing keys evenly, but range queries become scatter-gather), and compound keys such as { tenantId: 1, _id: 1 }. Zone sharding pins key ranges to specific shards, for example to keep EU customers' data in EU data centres. Modern versions can reshard a collection online, but it is an expensive operation.

When to shard. Not early. A single well-indexed replica set on adequate hardware handles a very large workload. Shard when the data no longer fits comfortably on one set, when write throughput exceeds one primary, or for data-residency requirements.

Key Takeaways

• A replica set has one primary for writes and secondaries that replay its oplog; automatic elections provide failover.
• Use an odd number of voting members; a majority is needed to elect a primary.
• Write concern controls durability (majority is the default); read preference and read concern control where reads go and how fresh they are.
• Replication is not a backup: keep and test real backups.
• Sharding distributes data across replica sets by shard key; choose a key with high cardinality, even distribution, non-monotonic writes and alignment with queries.

Practice

1. (Beginner) Start a three-member replica set with Docker Compose, insert data on the primary, stop the primary container and observe the election with rs.status().
2. (Intermediate) Write a script that inserts a document with w: 1 and immediately reads it with readPreference: 'secondary'. Show that the read can miss the write, then fix it with a causally consistent session or by reading from the primary.
3. (Interview) "You are sharding an events collection that receives 50,000 inserts per second and is queried by userId and time range. Which shard key do you choose?" Hint: compare { createdAt: 1 }, { userId: 'hashed' } and { userId: 1, createdAt: 1 } for write distribution and query targeting.

The Node.js MongoDB Driver

The official driver. The mongodb npm package is the official, MongoDB-maintained driver for Node.js. It handles connection pooling, server discovery and monitoring in replica sets and sharded clusters, retryable reads and writes, BSON encoding and TypeScript types. Mongoose (next topics) is built on top of it.

npm install mongodb

One client per process. MongoClient manages a pool of connections to every server it uses (default maxPoolSize is 100 per server). Create it once at start-up, share it, and close it on shutdown (see "Connection Pooling and Drivers" in 13.1):

// src/db/mongo.js
import { MongoClient } from 'mongodb';

export const client = new MongoClient(process.env.MONGO_URL, {
  maxPoolSize: 20,                    // per server; size for this process's concurrency
  serverSelectionTimeoutMS: 5_000,    // fail fast if no suitable server is found
  appName: 'bookshelf-api',           // shows up in server logs and monitoring
});

export const db = client.db(process.env.MONGO_DB ?? 'shop');

export async function connectMongo() {
  await client.connect();                          // optional (operations auto-connect), but fails fast at start-up
  await db.command({ ping: 1 });                   // verify credentials and connectivity
  await ensureIndexes();
}

async function ensureIndexes() {                   // idempotent: does nothing if the index already exists
  await db.collection('books').createIndexes([
    { key: { isbn: 1 }, unique: true },
    { key: { 'author.name': 1, createdAt: -1 } },
  ]);
}

// src/server.js: start-up and graceful shutdown (11.5, 12.1)
import { client, connectMongo } from './db/mongo.js';
import { createApp } from './app.js';

await connectMongo();
const server = createApp().listen(process.env.PORT ?? 3000);

process.on('SIGTERM', () => {
  server.close(async () => {
    await client.close();                          // close pooled connections cleanly
    process.exit(0);
  });
});

ObjectIds from URLs. Route parameters are strings; _id values are ObjectIds. Convert and validate them, otherwise { _id: '66fe...' } matches nothing (a string is not an ObjectId):

import { ObjectId } from 'mongodb';

export function parseObjectId(value) {
  if (!/^[0-9a-f]{24}$/i.test(value)) {          // ObjectId.isValid() also accepts any 12-character string
    throw new HttpError(400, 'invalid id', { code: 'INVALID_ID' });
  }
  return new ObjectId(value);
}

A MongoDB repository for the bookshelf API. In 12.3 the service depended on a repository with list, findById, findByIsbn, insert, update and remove. A MongoDB implementation exposes the same interface, so the service and routes need no changes (apart from validating IDs as ObjectId strings instead of integers):

// src/books/books.mongo-repository.js
import { ObjectId } from 'mongodb';

const toBook = ({ _id, ...rest }) => ({ id: _id.toHexString(), ...rest });   // hide Mongo's _id from API clients

export function createMongoBookRepository(db) {
  const books = db.collection('books');

  return {
    async list({ offset, limit, author }) {
      const filter = author
        ? { 'author.name': { $regex: `^${escapeRegex(author)}`, $options: 'i' } }   // escape user input in regexes
        : {};
      const [items, total] = await Promise.all([
        books.find(filter).sort({ createdAt: -1 }).skip(offset).limit(limit).toArray(),
        books.countDocuments(filter),
      ]);
      return { items: items.map(toBook), total };
    },

    async findById(id) {
      const doc = await books.findOne({ _id: new ObjectId(id) });
      return doc && toBook(doc);
    },

    async findByIsbn(isbn) {
      const doc = await books.findOne({ isbn });
      return doc && toBook(doc);
    },

    async insert(data) {
      const doc = { tags: [], ...data, createdAt: new Date() };
      const { insertedId } = await books.insertOne(doc);    // the driver also sets doc._id
      return toBook({ ...doc, _id: insertedId });
    },

    async update(id, changes) {
      const doc = await books.findOneAndUpdate(
        { _id: new ObjectId(id) },
        { $set: { ...changes, updatedAt: new Date() } },
        { returnDocument: 'after' },
      );
      return doc && toBook(doc);
    },

    async remove(id) {
      const { deletedCount } = await books.deleteOne({ _id: new ObjectId(id) });
      return deletedCount === 1;
    },
  };
}

const escapeRegex = (s) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');

Handling driver errors. Server errors are MongoServerError instances with a numeric code. The most common one in web applications is 11000, duplicate key, raised by unique indexes. Map it to 409 Conflict in the error handler (12.2). This also closes the race in the service's "check ISBN, then insert" logic: two concurrent requests can both pass the check, but the unique index rejects the second insert.

import { MongoServerError } from 'mongodb';

// in the central error handler (12.2)
if (err instanceof MongoServerError && err.code === 11000) {
  return res.status(409).json({ error: 'duplicate value', fields: Object.keys(err.keyValue ?? {}) });
}

Network errors (MongoNetworkError) and server selection timeouts (MongoServerSelectionError) indicate infrastructure problems: return 503 and make your health check report the database as down.

TypeScript. Collections accept a document type, which types filters, updates and results:

import type { ObjectId } from 'mongodb';

interface BookDoc {
  _id: ObjectId;
  title: string;
  isbn: string;
  author: { name: string };
  tags: string[];
  createdAt: Date;
}

const books = db.collection<BookDoc>('books');
await books.updateOne({ isbn: '9780441172719' }, { $set: { title: 'Dune' } });   // checked against BookDoc
// await books.updateOne({}, { $set: { titel: 'x' } });                          // compile-time error: unknown field

Note that these types are only compile-time hints: the database does not know about them, so validate input at the boundary and consider $jsonSchema on the server.

Testing. Integration tests should run against a real MongoDB: a Docker container in CI, Testcontainers, or the mongodb-memory-server package, which downloads and starts a real mongod binary for the test run. Use a separate database per test file and drop it afterwards.

Key Takeaways

• Use the official mongodb driver with one MongoClient per process; connect and create indexes at start-up and close the client on shutdown.
• Convert route parameters to ObjectId after validating them; strings do not match ObjectIds.
• A repository with the same interface as the in-memory one lets services and routes stay unchanged.
• Map duplicate key errors (code 11000) to 409 and network errors to 503; rely on unique indexes for uniqueness under concurrency.
• Typed collections catch mistakes at compile time but do not validate data at runtime.

Practice

1. (Beginner) Connect to MongoDB from a Node.js script, insert three books and print them sorted by title, closing the client at the end.
2. (Intermediate) Replace the in-memory repository of the 12.3 bookshelf API with createMongoBookRepository. Update the ID validation schema, map error 11000 to 409, and run the existing Supertest suite against mongodb-memory-server.
3. (Interview) "Why can the 'check if exists, then insert' pattern create duplicates, and how do you prevent it?" Hint: think about two concurrent requests between the check and the insert, and what a unique index guarantees.

Object Modeling with Mongoose

What Mongoose adds. The driver is deliberately thin: it stores whatever you give it. Mongoose is an ODM (Object Document Mapper) that adds an application-level schema on top of the driver: declared field types with casting, defaults, validation, middleware (hooks), relationships with populate, virtual properties, and model classes with methods. It trades some performance and transparency for structure and convenience, similar to an ORM in the relational world.

Concern	Native driver	Mongoose
Schema	Optional ($jsonSchema on the server)	Declared in code, applied to every document
Type casting	None: '10' stays a string	'10' cast to 10 for a Number path
Validation	Your own code or server validator	Built-in and custom validators
Hooks	None	pre/post middleware
Relationships	Manual or $lookup	ref + populate()
Result objects	Plain objects	Document instances with methods and change tracking (or plain objects with lean())
Overhead	Minimal	Higher (document hydration, change tracking)

Connecting.

import mongoose from 'mongoose';

await mongoose.connect(process.env.MONGO_URL, { maxPoolSize: 20, serverSelectionTimeoutMS: 5_000 });
mongoose.connection.on('disconnected', () => console.warn('MongoDB disconnected'));

// on shutdown
await mongoose.disconnect();

Mongoose buffers model operations until the connection is open, which can hide connection problems as hanging requests; set mongoose.set('bufferCommands', false) or connect before starting the HTTP server.

Schema and model.

// src/books/book.model.js
import mongoose from 'mongoose';
const { Schema, model } = mongoose;

const bookSchema = new Schema(
  {
    title:  { type: String, required: true, trim: true },
    isbn:   { type: String, required: true, unique: true },     // creates a unique INDEX (not a validator)
    price:  { type: Number, required: true, min: 0 },
    author: { type: Schema.Types.ObjectId, ref: 'Author', required: true },  // reference for populate()
    tags:   { type: [String], default: [] },
    status: { type: String, enum: ['draft', 'published'], default: 'draft' },
  },
  {
    timestamps: true,                                              // adds createdAt and updatedAt automatically
    toJSON: {                                                      // shape of res.json(book)
      virtuals: true,
      transform(_doc, ret) { delete ret._id; delete ret.__v; return ret; },
    },
  },
);

bookSchema.index({ author: 1, createdAt: -1 });                    // compound index declared on the schema

// Virtual: computed, not stored
bookSchema.virtual('priceWithVat').get(function () { return Math.round(this.price * 122) / 100; });

// Instance method: available on each document
bookSchema.methods.publish = function () { this.status = 'published'; return this.save(); };

// Static: available on the model
bookSchema.statics.findByIsbn = function (isbn) { return this.findOne({ isbn }); };

export const Book = model('Book', bookSchema);                     // collection name: 'books' (lowercased, pluralised)

Using models.

// Create: validation and casting run before the insert
const book = await Book.create({ title: ' Dune ', isbn: '9780441172719', price: '9.99', author: authorId });
book.title;   // 'Dune' (trimmed)
book.price;   // 9.99 (cast from string)

// Read
const one = await Book.findById(id);                             // casts the string id to ObjectId for you
const list = await Book.find({ tags: 'sci-fi' }).sort({ createdAt: -1 }).limit(20).select('title price');

// Update a document instance: change tracking saves only modified paths
one.price = 8.99;
await one.save();                                                // runs validators and save middleware

// Update directly in the database (faster, but see the validation caveats in the next topic)
await Book.updateOne({ _id: id }, { $inc: { price: 1 } });

await Book.deleteOne({ _id: id });

Queries are thenables. Book.find(...) returns a Query object that you can keep chaining; it executes when awaited. Call .exec() to get a real promise with better stack traces. Since Mongoose 7, callback APIs are gone: everything is promise-based.

lean() for read-heavy endpoints. By default each result is hydrated into a full Mongoose document with getters, setters and change tracking, which costs CPU and memory. For read-only API responses, lean() returns plain objects and is several times faster:

const books = await Book.find({ status: 'published' }).select('title price').lean();   // plain objects

Lean results do not have virtuals, methods or defaults applied (plugins exist to add virtuals back).

populate: following references. populate replaces an ObjectId with the referenced document, by running a second query:

const book = await Book.findById(id).populate('author', 'name country').lean();
// { title: 'Dune', author: { _id: ..., name: 'Frank Herbert', country: 'US' }, ... }

populate is convenient, but it is an application-side join: populating a list of 100 books issues one extra query with $in for all authors (not 100), but nested populates and large lists multiply queries. For hot paths, consider embedding (extended reference pattern) or $lookup.

Strictness. By default schemas are strict: fields not in the schema are silently dropped on save. That protects against mass assignment (a client sending { role: 'admin' }), but validating request bodies at the boundary (12.3) is still the primary defence.

Key Takeaways

• Mongoose is an ODM that adds schemas, casting, validation, middleware, virtuals, methods and populate on top of the driver.
• Define a Schema with types and options, then compile it into a Model used for all queries.
• timestamps, toJSON transforms, indexes, virtuals, instance methods and statics are declared on the schema.
• Use lean() for read-only queries to avoid document hydration overhead.
• populate performs extra queries; use it with care on hot paths and large lists.

Practice

1. (Beginner) Define Author and Book models with a reference from book to author, create two authors and four books, and list books with populated author names.
2. (Intermediate) Measure the time to load 10,000 books with and without lean() and compare heap usage with process.memoryUsage(). Explain the difference.
3. (Interview) "When would you choose the native driver over Mongoose?" Hint: weigh performance, transparency of the generated queries, team familiarity, the need for flexible document shapes, and where validation already happens (zod, $jsonSchema).

Mongoose Schemas, Validation and Middleware

Validation in Mongoose. Before a document is saved, Mongoose casts each value to its schema type and then runs validators. If anything fails, the save is rejected with a ValidationError and nothing reaches the database.

Built-in validators.

Type	Validators
All	required
Number	min, max, enum
String	minLength, maxLength, match (regex), enum
Date	min, max
Array	custom validators on the array or on its items

Each validator accepts a custom message, either as an array [value, message] or an object:

const userSchema = new Schema({
  email: {
    type: String,
    required: [true, 'email is required'],
    lowercase: true,                                               // setter: normalise before validation
    trim: true,
    match: [/^[^@\s]+@[^@\s]+\.[^@\s]+$/, 'invalid email address'],
    unique: true,                                                  // index, NOT a validator (see below)
  },
  username: { type: String, required: true, minLength: 3, maxLength: 30 },
  age: { type: Number, min: [13, 'must be at least 13'], max: 130 },
  role: { type: String, enum: { values: ['user', 'editor', 'admin'], message: '{VALUE} is not a valid role' }, default: 'user' },
  passwordHash: { type: String, required: true, select: false },   // excluded from query results unless asked for
});

Custom and async validators.

const eventSchema = new Schema({
  startsAt: { type: Date, required: true },
  endsAt: {
    type: Date,
    required: true,
    validate: {
      validator(value) { return value > this.startsAt; },          // 'this' is the document during save()
      message: 'endsAt must be after startsAt',
    },
  },
  venueId: {
    type: Schema.Types.ObjectId,
    validate: {
      async validator(id) { return !!(await Venue.exists({ _id: id })); }, // async validators return a promise
      message: 'venue does not exist',
    },
  },
});

Handling validation errors. A ValidationError has an errors object keyed by path; map it to 422 in the error handler, the same status used for schema validation in 12.3. Values that cannot be cast (for example 'abc' for a Number, or an invalid ObjectId in findById) produce a CastError.

import mongoose from 'mongoose';

// central error handler (12.2)
if (err instanceof mongoose.Error.ValidationError) {
  const details = Object.values(err.errors).map((e) => ({ path: e.path, message: e.message }));
  return res.status(422).json({ error: 'validation failed', details });
}
if (err instanceof mongoose.Error.CastError) {
  return res.status(400).json({ error: `invalid value for ${err.path}` });
}
if (err.code === 11000) {
  return res.status(409).json({ error: 'duplicate value', fields: Object.keys(err.keyValue) });
}

Three caveats that cause real bugs.

1. unique is not a validator. It only creates a unique index. Validation passes, and the duplicate is rejected later by the database with error 11000. If the collection already contains duplicates, the index build fails and uniqueness is not enforced at all.
2. Update queries skip validators by default. save() and create() validate; updateOne, updateMany and findOneAndUpdate do not, unless you pass runValidators: true. Even then, update validators only check the paths in the update, and in custom validators this is the query, not the document.
3. Update queries skip document middleware. pre('save') hooks do not run for updateOne or findOneAndUpdate.

await User.updateOne({ _id: id }, { $set: { age: -5 } });                          // accepted!
await User.updateOne({ _id: id }, { $set: { age: -5 } }, { runValidators: true });  // ValidationError
// Or enable globally: mongoose.set('runValidators', true)

Middleware (hooks). Middleware are functions that run before (pre) or after (post) certain operations, like Express middleware for database operations. There are four kinds, and this refers to something different in each:

Kind	Operations	this is
Document middleware	validate, save, updateOne/deleteOne called on a document	The document
Query middleware	find, findOne, findOneAndUpdate, updateOne, updateMany, deleteOne, deleteMany, countDocuments ...	The query
Aggregate middleware	aggregate	The aggregation object
Model middleware	insertMany, bulkWrite	The model

Example: hashing passwords on save.

import bcrypt from 'bcrypt';

userSchema.pre('save', async function () {
  if (!this.isModified('passwordHash')) return;                // only when the password changed
  this.passwordHash = await bcrypt.hash(this.passwordHash, 12); // async hooks: no need to call next()
});

userSchema.methods.checkPassword = function (plain) {
  return bcrypt.compare(plain, this.passwordHash);
};

// Usage: set the plain value, the hook replaces it with a hash before writing
const user = new User({ email, username, passwordHash: plainPassword });
await user.save();

// Login: passwordHash has select: false, so request it explicitly
const found = await User.findOne({ email }).select('+passwordHash');

Because of caveat 3, a password changed with User.updateOne({ $set: { passwordHash: 'new' } }) would be stored in plain text. Either always change passwords through save(), or add a pre('findOneAndUpdate')/pre('updateOne') hook that hashes the value in this.getUpdate().

Example: query middleware for soft delete.

// Hide soft-deleted documents from every find query automatically
userSchema.pre(/^find/, function () {                          // regex: find, findOne, findOneAndUpdate, ...
  if (this.getOptions().withDeleted) return;                     // escape hatch for admin tools
  this.where({ deletedAt: null });
});

userSchema.methods.softDelete = function () {
  this.deletedAt = new Date();
  return this.save();
};

Post hooks and error-handling middleware. A post hook receives the result. A post hook with three parameters (error, res, next) is error-handling middleware and can translate errors:

userSchema.post('save', function (doc) {
  logger.info({ userId: doc._id }, 'user saved');
});

userSchema.post('save', function (error, _doc, next) {
  if (error.code === 11000) next(new HttpError(409, 'email already registered'));
  else next(error);
});

Plugins: reusable schema behaviour. A plugin is a function that adds fields, hooks or methods to any schema:

// src/db/plugins/audit.js
export function auditPlugin(schema) {
  schema.add({ createdBy: Schema.Types.ObjectId, updatedBy: Schema.Types.ObjectId });
  schema.pre('save', function () {
    const actor = this.$locals.actorId;                          // set by the service before save
    if (this.isNew) this.createdBy = actor;
    this.updatedBy = actor;
  });
}

bookSchema.plugin(auditPlugin);        // or mongoose.plugin(auditPlugin) for all schemas

Where to validate? In a layered Express application, validation happens in several places with different purposes. Request schemas (zod, 12.3) reject malformed HTTP input early with clear messages. Mongoose validators protect the data's invariants whichever code path writes it. Unique indexes and $jsonSchema validators are the last line of defence, enforced by the database itself under concurrency.

Key Takeaways

• Mongoose casts values, then runs built-in (required, min, enum, match, ...) and custom (sync or async) validators before saving.
• Map ValidationError to 422, CastError to 400 and duplicate key errors (11000) to 409.
• unique is an index, not a validator; update queries skip validators unless runValidators is set, and skip save middleware entirely.
• Document, query, aggregate and model middleware run before or after operations; this differs for each kind.
• Plugins package reusable fields and hooks; validation belongs at the request boundary, in the model and in the database.

Practice

1. (Beginner) Add validators to a Product schema: required name of 2–100 characters, non-negative price, category from a fixed list. Show the errors object returned for an invalid product.
2. (Intermediate) Implement a user model with password hashing in pre('save'), then demonstrate the plain-text bug with findOneAndUpdate and fix it with a query hook.
3. (Interview) "A Mongoose schema declares email: { unique: true }, yet the database contains duplicate emails. How is that possible?" Hint: consider when the index is built, what happens if duplicates already existed, autoIndex settings in production, and the fact that unique is not a validator.

<a id="133-redis"></a>

13.3 Redis

Redis is an in-memory data structure server: it keeps strings, lists, hashes, sets, sorted sets and more in RAM and manipulates them with atomic commands in well under a millisecond. Earlier chapters already relied on it for sessions (10.2, 12.3), distributed rate limiting (10.6, 12.3) and WebSocket fan-out (10.5). This subchapter explains how it works and how to use it well: data types and expiration, atomicity and transactions, pub/sub, caching, sessions, rate limiting and queues, persistence, high availability and the Node.js clients.

In-Memory Data Store Concepts

Why keep data in memory? Reading from RAM takes about 100 nanoseconds; a random read from an SSD takes about 100 microseconds, and a database query including network, parsing, planning and disk is typically 1–10 milliseconds. An in-memory store removes disk access from the request path, so a whole command round trip (mostly network time) completes in a fraction of a millisecond. That makes it ideal for data that is read or updated on almost every request: sessions, counters, caches, rate limits.

An analogy: the database is a warehouse, safe and enormous but a drive away. Redis is the shelf behind the counter: small and quick to reach, holding the items asked for most often.

What Redis is. Redis (REmote DIctionary Server), created by Salvatore Sanfilippo in 2009, is a server that maps keys to typed values and exposes commands that operate on those values on the server side:

 client                      Redis server (one process, data in RAM)
 ---------------------       ------------------------------------------------
 INCR page:views    ---->    key "page:views" -> integer 41 -> becomes 42
                    <----    (integer) 42
 LPUSH jobs "email:17" -->   key "jobs" -> list ["email:16"] -> ["email:17","email:16"]
 ZADD lb 870 "ada"  ---->    key "lb" -> sorted set {ada:870, bob:640, ...}

Because operations like "increment", "append to list" or "add to sorted set" run inside Redis, many clients can share the same structures without race conditions (see "Atomic Operations and Transactions").

The single-threaded execution model. Redis executes commands one at a time on a single main thread, using an event loop much like Node.js (11.3). There are no locks, and every individual command is atomic: no other command can observe a half-applied change. Since Redis 6, network reading and writing can use extra I/O threads, but command execution remains sequential. The consequence: one slow command blocks everyone. Commands that touch every element of a large structure (KEYS *, SMEMBERS on a set of millions, HGETALL on a huge hash, FLUSHALL without ASYNC) can freeze the server for seconds. Use SCAN-family commands to iterate incrementally.

Memory is the limit. All data must fit in RAM. Configure a ceiling and an eviction policy, which decides what happens when the ceiling is reached:

Policy	Behaviour when maxmemory is reached	Good for
noeviction (default)	Writes fail with an OOM error	Data that must not disappear (queues, sessions)
allkeys-lru	Evict the least recently used key	General-purpose cache
allkeys-lfu	Evict the least frequently used key	Cache with stable popular items
volatile-lru / volatile-lfu	Same, but only among keys with a TTL	Mixed cache + persistent data in one instance
volatile-ttl	Evict keys closest to expiring	Caches with meaningful TTLs
allkeys-random / volatile-random	Evict randomly	Rarely the best choice

# redis.conf
maxmemory 2gb
maxmemory-policy allkeys-lru

Mixing a cache (evictable) with sessions or queues (not evictable) in one instance is risky: under memory pressure the cache policy may evict the data you needed to keep. Use separate instances or logical separation by policy.

Durability. "In memory" does not mean "lost on restart": Redis can write snapshots and an append-only log to disk (see "Persistence with RDB and AOF"). But Redis is usually not the system of record (13.1): it holds data that can be recomputed or that is acceptable to lose in a disaster.

The ecosystem and licensing. In 2024 Redis changed its licence from BSD to source-available licences; the Linux Foundation then forked the last BSD version as Valkey, which major cloud providers adopted. Redis 8 (2025) added the open-source AGPLv3 as a licensing option and bundled former modules (JSON, search, time series, probabilistic structures). Valkey and Redis remain protocol-compatible for everything in this chapter, and the same clients work with both. Managed offerings include Amazon ElastiCache and MemoryDB, Google Memorystore, Azure Managed Redis, Redis Cloud and Upstash.

Trying it out.

docker run -d --name redis -p 6379:6379 redis:8          # or: valkey/valkey:8
docker exec -it redis redis-cli                          # interactive client

127.0.0.1:6379> SET greeting "hello"
OK
127.0.0.1:6379> GET greeting
"hello"
127.0.0.1:6379> INFO memory                              # used_memory_human, maxmemory, fragmentation
127.0.0.1:6379> exit

docker exec -it redis redis-cli --latency                # measure round-trip latency continuously

Key Takeaways

• Redis keeps typed data structures in RAM and serves commands in well under a millisecond.
• Commands execute one at a time on a single thread: each command is atomic, but slow commands block everyone; iterate with SCAN.
• All data must fit in memory; set maxmemory and an eviction policy that matches the data's purpose.
• Redis can persist to disk but is usually not the system of record.
• Valkey is a protocol-compatible open-source fork; clients and commands in this chapter work with both.

Practice

1. (Beginner) Start Redis with Docker, set and read a few keys with redis-cli, and inspect INFO memory and INFO stats.
2. (Intermediate) Set maxmemory 10mb with allkeys-lru, write keys in a loop until memory is full, and observe evictions in INFO stats (evicted_keys). Repeat with noeviction and observe the error.
3. (Interview) "Redis is single-threaded. How can it handle hundreds of thousands of operations per second?" Hint: consider in-memory data, non-blocking I/O multiplexing, no locking overhead, simple O(1) operations and pipelining.

Keys, Expiration and TTL

Keys. A Redis key is a binary-safe string (up to 512 MB, but keep keys short). There are no tables or collections, so structure comes from naming conventions. The common convention uses colons to build a hierarchy:

 <app>:<object-type>:<id>[:<field>]

 shop:user:42                 hash with the user's profile
 shop:user:42:cart            hash sku -> quantity
 shop:product:9:views         counter
 sess:Yp3x...                 session (prefix set by connect-redis)
 ratelimit:login:203.0.113.7  rate-limit counter
 cache:v3:book:66fe1c...      cached JSON, with a version for mass invalidation

A consistent prefix per feature makes keys discoverable (SCAN MATCH shop:user:*), allows per-feature memory analysis, and enables ACL rules that restrict a client to some prefixes.

Generic key commands.

Command	Effect
EXISTS key [key ...]	Number of the given keys that exist
DEL key [key ...]	Delete synchronously
UNLINK key [key ...]	Delete, freeing memory in a background thread (prefer for large values)
TYPE key	string, list, hash, set, zset, stream
RENAME key newkey	Rename (overwrites newkey)
SCAN cursor MATCH pattern COUNT n	Iterate keys incrementally without blocking
KEYS pattern	List matching keys: blocks the server, never use in production
OBJECT ENCODING key, MEMORY USAGE key	Inspect internal representation and size

# Iterate safely: SCAN returns a new cursor and a batch; stop when the cursor is 0
127.0.0.1:6379> SCAN 0 MATCH shop:user:* COUNT 100
1) "1792"                       # next cursor
2) 1) "shop:user:42"
   2) "shop:user:7"
127.0.0.1:6379> SCAN 1792 MATCH shop:user:* COUNT 100
...

Expiration (time to live). Any key can be given a TTL, after which Redis deletes it automatically. This is one of Redis's most valuable features: caches refresh themselves, sessions end, rate-limit windows reset, and one-time tokens disappear, all without cleanup jobs.

SET otp:42 "731904" EX 300        # set value and expire in 300 seconds, atomically
TTL otp:42                        # (integer) 297: seconds remaining
PTTL otp:42                       # milliseconds remaining
EXPIRE cart:42 86400              # set/replace the TTL of an existing key (1 day)
EXPIREAT promo:summer 1790000000  # expire at a Unix timestamp
PERSIST cart:42                   # remove the TTL: the key lives forever
TTL no-such-key                   # (integer) -2: key does not exist
SET counter 0
TTL counter                       # (integer) -1: key exists, no TTL

SET options worth memorising.

Option	Meaning
EX seconds / PX milliseconds	Set a TTL together with the value
EXAT / PXAT timestamp	Expire at an absolute Unix time
NX	Only set if the key does not exist (locks, "first writer wins")
XX	Only set if the key already exists
KEEPTTL	Keep the existing TTL instead of clearing it
GET	Return the old value

EXPIRE also accepts conditions (Redis 7+): NX (only if no TTL), XX (only if a TTL exists), GT / LT (only if the new TTL is greater/less).

TTL rules that cause bugs.

• Overwriting a key with SET removes its TTL (unless KEEPTTL). A cached value refreshed without EX becomes permanent.
• Commands that modify a value (INCR, LPUSH, HSET, SADD) keep the existing TTL.
• RENAME transfers the TTL to the new name.
• The TTL applies to the whole key. A hash cannot normally expire individual fields; Redis 7.4 added per-field TTLs for hashes (HEXPIRE), but separate keys remain the portable approach.

How expiration works. Redis combines two mechanisms:

 passive (lazy):  when a client accesses a key, Redis checks its expiry and deletes it if past due
 active:          about 10 times per second, Redis samples random keys with a TTL and deletes the expired ones;
                  if many sampled keys were expired, it repeats immediately

So expired keys are never returned, but they may occupy memory briefly. Replicas do not expire keys on their own: they wait for the primary to send a DEL, which keeps replicas consistent.

Avoiding synchronized expiry. If 10,000 cache keys are written at the same moment with the same TTL, they also expire at the same moment and the database receives 10,000 queries at once. Add jitter:

const ttl = 3600 + Math.floor(Math.random() * 300);   // 60-65 minutes
await redis.set(key, value, { EX: ttl });

Key Takeaways

• Keys are strings; use a consistent app:type:id:field naming convention and keep keys short.
• Iterate with SCAN, never KEYS, and delete large values with UNLINK.
• SET ... EX/PX sets a value and TTL atomically; NX/XX make writes conditional; TTL returns -1 (no expiry) or -2 (missing).
• SET without KEEPTTL clears an existing TTL; modifying commands keep it.
• Expired keys are removed lazily and by active sampling; add jitter to TTLs to avoid mass expiry.

Practice

1. (Beginner) Create a one-time password key valid for 5 minutes, check its TTL, overwrite it with a plain SET and check the TTL again. Explain the result.
2. (Intermediate) Write a Node.js script that deletes all keys matching cache:v2:* using SCAN and UNLINK in batches, and measure how long it takes for 1 million keys.
3. (Interview) "Why is KEYS * dangerous in production when SCAN is not?" Hint: relate the single-threaded execution model to an O(N) command, and explain the guarantees SCAN gives (and does not give) about duplicates and concurrent changes.

Data Types: Strings, Lists, Hashes and Sets

Why typed values? A plain key-value store would make you GET a JSON array, modify it in your application and SET it back: two round trips and a race condition. Redis values have types with their own commands, so the change happens atomically on the server. Choosing the right type is the most important Redis design decision.

Strings. The basic type: a byte sequence up to 512 MB. It holds text, serialised JSON, binary data, or a number that Redis can increment.

SET user:42:name "Ada"
GET user:42:name                       # "Ada"
MSET a 1 b 2                           # several keys in one command
MGET a b missing                       # 1) "1" 2) "2" 3) (nil)
INCR page:home:views                   # (integer) 1: atomic, creates the key at 0 if missing
INCRBY page:home:views 10              # (integer) 11
INCRBYFLOAT balance:42 2.5
APPEND log:today "line\n"
SETRANGE / GETRANGE                    # byte ranges

Strings also power two compact structures: bitmaps (SETBIT, BITCOUNT: one bit per user ID, for example daily active users in 12 MB for 100 million users) and HyperLogLog (PFADD, PFCOUNT: approximate count of distinct items with about 0.81% error in at most 12 KB).

Lists. Ordered sequences of strings, implemented so that pushing and popping at both ends is O(1). Use them for queues, stacks, recent-items feeds and capped logs.

LPUSH notifications:42 "n3" "n2"       # push to the head (left)
RPUSH notifications:42 "n4"            # push to the tail (right)
LRANGE notifications:42 0 -1           # whole list: n2 n3 n4  (indexes; -1 = last)
LLEN notifications:42
LPOP notifications:42                  # remove and return the head
LTRIM notifications:42 0 99            # keep only the first 100 elements (capped list)

# Blocking pop: wait up to 5 s for an element (a worker waiting for jobs)
BLPOP jobs 5
# Atomically move an element to another list (reliable queue, see "Rate Limiting and Job Queues")
LMOVE jobs jobs:processing LEFT RIGHT

Accessing the middle of a long list (LINDEX, LINSERT) is O(N); lists are not arrays.

Hashes. A map of field-value pairs stored under one key, like a small object. Ideal for records whose fields are read or updated individually, and very memory-efficient when small.

HSET user:42 name "Ada" email "ada@example.com" logins 0
HGET user:42 email                     # "ada@example.com"
HMGET user:42 name email
HINCRBY user:42 logins 1               # atomic per-field counter
HGETALL user:42                        # all fields (fine for small hashes)
HDEL user:42 email
HEXISTS user:42 email                  # 0
HLEN user:42

HINCRBY cart:42 sku-9 2                # shopping cart: field = SKU, value = quantity

Hash or JSON string? Store an object as a hash when fields are updated independently (counters, partial updates) or read partially. Store it as a JSON string when it is always read and written whole and has nested structure. (Redis 8 also includes a native JSON type with path-level commands.)

Sets. Unordered collections of unique strings with O(1) add, remove and membership test, plus server-side set algebra.

SADD post:7:likes "u42" "u17"          # (integer) 2: number of NEW members
SADD post:7:likes "u42"                # (integer) 0: already present
SISMEMBER post:7:likes "u42"           # 1
SCARD post:7:likes                     # 2: cardinality
SREM post:7:likes "u17"
SMEMBERS post:7:likes                  # all members (use SSCAN for big sets)

SADD user:42:follows u1 u2 u3
SADD user:17:follows u2 u3 u9
SINTER user:42:follows user:17:follows # u2 u3: follows in common
SDIFF user:17:follows user:42:follows  # u9: suggestion for user 42
SRANDMEMBER giveaway:entrants 3        # pick 3 random winners (SPOP also removes them)

Choosing a type.

Need	Type	Example key
Single value, counter, cached JSON	String	cache:book:9, views:post:7
Queue, recent N items, timeline	List	queue:emails, feed:42
Object with independently updated fields	Hash	user:42, cart:42
Unique membership, tags, "who liked"	Set	post:7:likes, online:users
Ranking, ordering by score or time	Sorted set (next topic)	leaderboard:2026-10
Append-only event log with consumer groups	Stream	events:orders
Distinct counts at huge scale	HyperLogLog	uniques:2026-10-03

Complexity matters. Redis documentation lists the time complexity of every command. Commands that are O(1) or O(log N) are safe at any size. O(N) commands on large values (SMEMBERS, HGETALL, LRANGE 0 -1) should be replaced by their SCAN variants (SSCAN, HSCAN, ZSCAN) or bounded ranges.

Key Takeaways

• Typed values let Redis modify data atomically on the server instead of read-modify-write in the application.
• Strings hold text, JSON or numbers (INCR); bitmaps and HyperLogLog are compact counting structures built on strings.
• Lists are fast at both ends: queues, feeds and capped logs with LTRIM; blocking pops wait for data.
• Hashes store objects with independently updated fields; sets store unique members with fast membership and set algebra.
• Check command complexity and avoid O(N) commands on large values.

Practice

1. (Beginner) Model a shopping cart with a hash: add items, change a quantity with HINCRBY, remove an item and list the cart.
2. (Intermediate) Implement "recently viewed products" (last 10, no duplicates, most recent first) with a list, using LREM, LPUSH and LTRIM in one MULTI. Then discuss why a sorted set might be better.
3. (Interview) "How would you count unique daily visitors for a site with 50 million users using little memory?" Hint: compare a set, a bitmap keyed by user ID, and HyperLogLog in terms of memory, accuracy and the ability to list visitors.

Sorted Sets and Ranking Use Cases

The idea. A sorted set (zset) is a set of unique members where each member has a numeric score, and members are kept ordered by score. You can add or update a member in O(log N), get its rank in O(log N), and read any range by rank or score. Doing that in a relational database for millions of rows on every request ("what is my position on the leaderboard?") is expensive; in Redis it is a single fast command.

Internally, large sorted sets combine a skip list (for ordered traversal and ranges) with a hash table (for O(1) score lookup by member); small ones use a compact list.

 leaderboard (ordered by score, ascending internally)
 rank (ZRANK)    0       1        2        3
 member         bob     carol    ada      dan
 score          640     720      870      910
 rank (ZREVRANK) 3       2        1        0     <- "position" in a high-score table

Core commands.

ZADD lb 870 ada 640 bob 720 carol 910 dan   # add members with scores
ZINCRBY lb 50 bob                            # bob: 690 (atomic, creates member if missing)
ZSCORE lb ada                                # "870"
ZREVRANK lb ada                              # (integer) 1: second place (0-based)
ZCARD lb                                     # 4

# Top 3 (highest first), with scores
ZRANGE lb 0 2 REV WITHSCORES                 # dan 910, ada 870, carol 720

# By score range: players between 700 and 900
ZRANGE lb 700 900 BYSCORE WITHSCORES
ZCOUNT lb 700 900                            # how many

# Remove
ZREM lb bob
ZREMRANGEBYSCORE lb -inf 100                 # drop low scores
ZREMRANGEBYRANK lb 0 -1001                   # keep only the top 1000

# Conditional updates (ZADD flags)
ZADD lb GT 900 ada                           # only update if the new score is GREATER (keep personal best)
ZADD lb NX 0 newbie                          # only add new members, never update existing ones

When scores are equal, members are ordered lexicographically. Scores are 64-bit floating-point numbers, so integers up to 2^53 are exact.

Use case 1: leaderboards.

// node-redis
export async function recordScore(gameId, userId, score) {
  // GT flag: only raise the score (keep personal best); raw command because option syntax differs across client versions
  await redis.sendCommand(['ZADD', `lb:${gameId}`, 'GT', String(score), userId]);
}

export async function topTen(gameId) {
  return redis.zRangeWithScores(`lb:${gameId}`, 0, 9, { REV: true });        // [{ value, score }, ...]
}

export async function myPosition(gameId, userId) {
  const rank = await redis.zRevRank(`lb:${gameId}`, userId);
  return rank === null ? null : rank + 1;                                     // 1-based for display
}

// "Players around me": 2 above and 2 below
export async function neighbours(gameId, userId) {
  const rank = await redis.zRevRank(`lb:${gameId}`, userId);
  if (rank === null) return [];
  return redis.zRangeWithScores(`lb:${gameId}`, Math.max(0, rank - 2), rank + 2, { REV: true });
}

Periodic leaderboards use one key per period (lb:weekly:2026-W40) with a TTL, and ZUNIONSTORE can combine them (for example, a monthly board from weekly ones, summing scores).

Use case 2: time-ordered data (score = timestamp).

# Recent activity per user, scored by Unix milliseconds
ZADD activity:42 1790000000123 "login"
ZADD activity:42 1790000050000 "order:991"
ZRANGE activity:42 +inf 1789990000000 BYSCORE REV LIMIT 0 20   # latest 20 since a time
ZREMRANGEBYSCORE activity:42 -inf 1787400000000                # drop entries older than 30 days

Use case 3: delayed jobs and scheduling. Store job IDs scored by the time they should run; a worker periodically takes the jobs whose score is in the past. ZPOPMIN removes the earliest atomically.

await redis.zAdd('delayed', { score: Date.now() + 15 * 60_000, value: 'reminder:991' });  // run in 15 min

// worker loop (simplified; a Lua script makes "read due jobs and remove them" atomic across workers)
const due = await redis.zRangeByScore('delayed', 0, Date.now(), { LIMIT: { offset: 0, count: 100 } });

Use case 4: trending and top-N with decay. Increment a score per view (ZINCRBY trending:2026-10-03 1 post:7) in hourly keys, then ZUNIONSTORE the last 24 hours with WEIGHTS that give older hours less weight.

Use case 5: autocomplete. When all members have the same score, sorted sets order them lexicographically, and ZRANGE ... BYLEX returns members with a given prefix:

ZADD autocomplete 0 "redis" 0 "react" 0 "relational" 0 "rest"
ZRANGE autocomplete "[re" "[re\xff" BYLEX LIMIT 0 10      # react, redis, relational, rest

Key Takeaways

• A sorted set keeps unique members ordered by a floating-point score; updates, ranks and range reads are O(log N).
• ZADD (with GT, LT, NX, XX flags), ZINCRBY, ZRANGE (with REV, BYSCORE, BYLEX, LIMIT) and ZREVRANK cover most needs.
• Leaderboards, "around me" views, time-ordered feeds, delayed jobs, trending lists and prefix autocomplete are natural fits.
• Use one key per period with a TTL, and combine periods with ZUNIONSTORE.

Practice

1. (Beginner) Build a leaderboard for five players, update two scores, and print the top three and one player's position.
2. (Intermediate) Implement a sliding "last 24 hours" trending list for blog posts using hourly sorted sets and ZUNIONSTORE with weights, and expose it as GET /api/trending.
3. (Interview) "How would you show a user's global rank among 100 million players in real time?" Hint: compare COUNT(*) WHERE score > ? in SQL with ZREVRANK, and discuss memory per member and sharding by score range when one instance is not enough.

Atomic Operations and Transactions

The concurrency problem. A web application usually runs several processes (11.6, cluster and containers), each handling many requests concurrently. If two of them read a value, modify it in JavaScript and write it back, one update can be lost:

 process A: GET stock -> 1                     process B: GET stock -> 1
 process A: SET stock 0  (sold one)            process B: SET stock 0  (sold one)
 result: 2 items sold, stock says 0, real stock -1

Redis provides four tools to avoid this, from simplest to most powerful.

1. Single commands are atomic. Because commands run one at a time, a command that reads and modifies in one step is safe:

DECRBY stock:sku-9 1            # atomic decrement: returns the new value
SET lock:report "worker-3" NX PX 30000   # atomic "set if absent": only one worker gets the lock
HINCRBY cart:42 sku-9 1
SADD seen:emails "a@b.c"        # returns 1 if new, 0 if already present: atomic de-duplication

Prefer designs that need only one command.

2. MULTI / EXEC transactions. Commands sent between MULTI and EXEC are queued and then executed sequentially without any other client's command in between:

MULTI
INCR user:42:orders
LPUSH user:42:recent "order:991"
LTRIM user:42:recent 0 9
EXEC
# 1) (integer) 8
# 2) (integer) 11
# 3) OK

Redis transactions differ from SQL transactions in two important ways:

• No rollback. If a command fails at runtime (for example INCR on a non-numeric value), the others still execute. Only errors detected while queueing (syntax errors) abort the whole transaction.
• No reads inside. Commands are queued, so you cannot use a value read inside the transaction to decide what to write later in the same transaction.

3. WATCH: optimistic locking. For read-modify-write logic, WATCH keys before reading. If any watched key changes before EXEC, the transaction is discarded (EXEC returns null) and you retry:

// node-redis: transfer loyalty points without going negative
async function transferPoints(from, to, amount) {
  for (let attempt = 0; attempt < 5; attempt++) {
    const conn = redis.duplicate();                          // WATCH state is per connection: use a dedicated one
    await conn.connect();
    try {
      await conn.watch(`points:${from}`);
      const balance = Number(await conn.get(`points:${from}`) ?? 0);
      if (balance < amount) throw new Error('insufficient points');
      const result = await conn.multi()
        .decrBy(`points:${from}`, amount)
        .incrBy(`points:${to}`, amount)
        .exec();                                             // aborted if points:{from} changed since WATCH
      if (result !== null) return;                           // committed
    } catch (err) {
      if (err.name !== 'WatchError') throw err;              // node-redis throws WatchError on abort
    } finally {
      await conn.close();                                    // node-redis v5 (v4: quit())
    }
  }
  throw new Error('too much contention, try again');
}

WATCH is per connection, which is why this example uses a dedicated connection; in production, node-redis's executeIsolated (v4) or a client pool (v5) avoid creating a connection per call. This complexity is one reason Lua scripts are often preferred.

4. Lua scripts and functions. A Lua script runs atomically on the server: no other command executes while it runs. It can read, branch and write, which makes it the most flexible tool for atomic logic:

-- transfer-points.lua
-- KEYS[1] = points:from, KEYS[2] = points:to, ARGV[1] = amount
local balance = tonumber(redis.call('GET', KEYS[1]) or '0')
local amount = tonumber(ARGV[1])
if balance < amount then
  return redis.error_reply('INSUFFICIENT_POINTS')
end
redis.call('DECRBY', KEYS[1], amount)
redis.call('INCRBY', KEYS[2], amount)
return balance - amount

// node-redis: run the script with its keys and arguments (all arguments travel as strings)
import { readFile } from 'node:fs/promises';

const transferLua = await readFile(new URL('./transfer-points.lua', import.meta.url), 'utf8');

const remaining = await redis.eval(transferLua, {
  keys: ['points:42', 'points:17'],
  arguments: ['100'],
});

// Optimisation: load once, then call by SHA1 digest to avoid resending the source
const sha = await redis.scriptLoad(transferLua);
await redis.evalSha(sha, { keys: ['points:42', 'points:17'], arguments: ['100'] });

If the server restarts, its script cache is emptied and EVALSHA fails with NOSCRIPT; reload the script and retry. node-redis's scripts option and ioredis's defineCommand (see "Redis Clients in Node.js") handle this automatically.

Rules for scripts: pass every key the script touches in KEYS (required for Redis Cluster routing), keep scripts short (they block the server while running), and make them deterministic. Redis 7 also offers functions (FUNCTION LOAD, FCALL), which are named, persisted libraries of scripts.

Pipelining is not a transaction. Pipelining sends many commands without waiting for each reply, saving network round trips. It improves throughput dramatically but does not make the commands atomic: other clients' commands can interleave.

 without pipelining:  cmd1 -> reply1 -> cmd2 -> reply2 -> cmd3 -> reply3   (3 round trips)
 with pipelining:     cmd1, cmd2, cmd3 -> reply1, reply2, reply3            (1 round trip)

node-redis pipelines automatically: commands issued in the same event-loop tick are sent together, so await Promise.all([redis.get(a), redis.get(b)]) uses one round trip.

Distributed locks. SET lock:name <random-token> NX PX 30000 acquires a lock that expires automatically if the holder crashes. Release it only if you still own it, which requires a script (a plain DEL could delete a lock another process acquired after yours expired):

-- release: delete only if the token matches
if redis.call('GET', KEYS[1]) == ARGV[1] then
  return redis.call('DEL', KEYS[1])
end
return 0

A Redis lock is fine for efficiency (avoid doing the same work twice). For correctness-critical mutual exclusion, remember that a paused process can outlive its lock's TTL; protect the resource itself with a fencing token or database constraint.

Key Takeaways

• Each Redis command is atomic; design operations to need a single command where possible (INCR, SET NX, HINCRBY, SADD).
• MULTI/EXEC executes queued commands without interleaving, but has no rollback and cannot read inside the transaction.
• WATCH adds optimistic locking: the transaction aborts if a watched key changed, and the client retries.
• Lua scripts (and Redis 7 functions) run atomically with full read-branch-write logic; pass all keys in KEYS.
• Pipelining reduces round trips but is not atomic; distributed locks need tokens, TTLs and safe release.

Practice

1. (Beginner) Use MULTI/EXEC to increment a counter and append to a capped list in one transaction. Then make one command fail at runtime and observe that the other still applies.
2. (Intermediate) Implement a "reserve stock" Lua script that decrements stock:<sku> only if enough units remain and records the reservation in a hash, and test it with 100 concurrent requests for 10 units.
3. (Interview) "What is the difference between a Redis transaction and a SQL transaction?" Hint: compare atomicity, isolation, rollback behaviour and the ability to read inside the transaction; mention WATCH and Lua.

Publish and Subscribe Messaging

The idea. Publish/subscribe (pub/sub) decouples senders from receivers: publishers send messages to a named channel without knowing who is listening, and every client currently subscribed to that channel receives a copy. It is like a radio broadcast: the station transmits, anyone tuned in hears it, and anyone who was not tuned in misses it.

                         channel "chat:room:7"
 publisher (instance A) ---PUBLISH---> [ Redis ] ---> subscriber: instance A
                                           |-------> subscriber: instance B
                                           +-------> subscriber: instance C

Commands.

# terminal 1
SUBSCRIBE chat:room:7 notifications           # the connection enters subscriber mode
PSUBSCRIBE chat:room:*                        # pattern subscription

# terminal 2
PUBLISH chat:room:7 '{"user":"ada","text":"hi"}'
# (integer) 2   <- number of clients that received it (0 if nobody is listening)

Delivery semantics. Redis pub/sub is fire-and-forget (at-most-once):

• Messages are not stored. A subscriber that is disconnected, restarting or slow simply misses them.
• There are no acknowledgements or retries.
• A subscriber that cannot keep up has its output buffer grow until Redis disconnects it.

That is acceptable for ephemeral real-time signals (chat messages already persisted in a database, "user is typing", cache invalidation broadcasts, live dashboard updates) and unacceptable for work that must not be lost (orders, emails to send). For those, use lists or streams (see "Rate Limiting and Job Queues").

Subscriber connections are special. A connection in subscribe mode can only issue subscription commands (with the classic RESP2 protocol), so applications use a dedicated connection for subscribing:

// node-redis: fan out WebSocket messages across several server instances (10.5)
import { createClient } from 'redis';

const pub = createClient({ url: process.env.REDIS_URL });
const sub = pub.duplicate();                         // second connection, same options
await Promise.all([pub.connect(), sub.connect()]);

// Every instance subscribes; each delivers to ITS OWN connected sockets
await sub.subscribe('chat:room:7', (message, channel) => {
  const event = JSON.parse(message);
  for (const ws of localSocketsInRoom('7')) ws.send(JSON.stringify(event));
});

// Pattern subscriptions receive the concrete channel name too
await sub.pSubscribe('chat:room:*', (message, channel) => {
  const roomId = channel.split(':').at(-1);
  broadcastLocally(roomId, message);
});

// Any instance can publish when one of its clients sends a message
export async function sendToRoom(roomId, event) {
  await saveMessage(roomId, event);                  // persist first: pub/sub is not storage
  await pub.publish(`chat:room:${roomId}`, JSON.stringify(event));
}

This is exactly what the Socket.IO Redis adapter (10.5) does internally.

Cache invalidation across instances. Applications that keep a small in-process cache (an LRU map in each Node.js instance) can broadcast "key X changed" on a channel so every instance drops its local copy. Missed messages are tolerable if local entries also have a short TTL.

Keyspace notifications. Redis can publish events about its own keys (set, expired, evicted, ...) on special channels, enabled with notify-keyspace-events:

CONFIG SET notify-keyspace-events Ex          # E: keyevent channels, x: expired events
SUBSCRIBE __keyevent@0__:expired              # receive the names of keys as they expire

They are useful for debugging and some automation, but they inherit pub/sub's at-most-once semantics, and expiry events fire when Redis actually deletes the key, not at the exact TTL instant.

Sharded pub/sub. In Redis Cluster, classic PUBLISH is broadcast to every node, which limits scalability. Redis 7 added sharded pub/sub (SPUBLISH, SSUBSCRIBE), where a channel lives on the node owning its hash slot.

Pub/sub vs streams vs lists.

Feature	Pub/sub	List (queue)	Stream
Delivery	To all current subscribers	To one consumer	To each consumer group; one consumer per group
Persistence	None	Until popped	Until trimmed
Missed while offline	Lost	Kept	Kept; consumers resume from their last ID
Acknowledgement	No	Manual pattern (LMOVE)	Built in (XACK, pending list)
Typical use	Real-time fan-out, notifications	Simple work queues	Event logs, reliable queues, multiple consumer groups

Key Takeaways

• Pub/sub delivers each message to every client currently subscribed to the channel; patterns subscribe to many channels at once.
• Delivery is at-most-once with no storage or acknowledgements: offline or slow subscribers miss messages.
• Subscribers need a dedicated connection; publishers can use the normal client.
• Use pub/sub for ephemeral fan-out (WebSocket scaling, cache invalidation), and lists or streams for work that must not be lost.
• Keyspace notifications publish key events; sharded pub/sub scales channels in Redis Cluster.

Practice

1. (Beginner) Open two redis-cli sessions, subscribe in one and publish in the other. Then publish while the subscriber is disconnected and observe that the message is lost.
2. (Intermediate) Run two instances of a WebSocket chat server on different ports and use Redis pub/sub so that messages sent to one instance reach clients of the other.
3. (Interview) "Would you use Redis pub/sub to send order confirmation emails?" Hint: consider what happens when the email worker restarts, scaling to several workers (does each get a copy?), and how streams with consumer groups or a job queue solve both problems.

Caching Strategies and Invalidation

Why cache. A cache stores the result of an expensive operation (a database query, an external API call, a rendered fragment) so that later requests can reuse it. When the same data is read far more often than it changes, a cache reduces latency (Redis answers in under a millisecond), lowers load on the database, and absorbs traffic spikes. Phil Karlton's famous quip applies: "There are only two hard things in computer science: cache invalidation and naming things." Adding a cache is easy; keeping it correct is the hard part.

Where caches live. Redis is one layer among several (Chapter 10 covered HTTP caching):

 browser cache --> CDN / reverse proxy --> application (in-process LRU) --> Redis --> database
 (Cache-Control)   (shared HTTP cache)     (per instance, fastest)          (shared)  (system of record)

Caching patterns.

Pattern	How it works	Pros	Cons
Cache-aside (lazy loading)	App checks cache; on miss, reads DB and fills cache	Simple; caches only what is used; cache failure is non-fatal	First request is slow; stale until TTL or invalidation
Read-through	Cache library loads from DB itself on miss	Clean application code	Needs a loader abstraction
Write-through	Every write goes to the cache and the DB synchronously	Cache always fresh for written keys	Write latency; caches data never read
Write-behind (write-back)	Writes go to cache; flushed to DB asynchronously	Very fast writes	Data loss risk; complex
Refresh-ahead	Popular keys refreshed before they expire	No miss latency for hot keys	Wasted work on keys no longer needed

Cache-aside in practice. Cache-aside is the default choice for web applications:

 GET /books/9
   1. GET cache:book:9 from Redis
   2. hit?  -> return it
   3. miss  -> query MongoDB, SET cache:book:9 <json> EX 600, return it

 PATCH /books/9
   1. update MongoDB (system of record)
   2. DEL cache:book:9   (next read repopulates it)

// src/lib/cache.js: a small cache-aside helper
export function createCache(redis, { prefix = 'cache:v1:', defaultTtl = 600 } = {}) {
  return {
    async getOrLoad(key, loader, ttl = defaultTtl) {
      const fullKey = prefix + key;
      try {
        const cached = await redis.get(fullKey);
        if (cached !== null) return JSON.parse(cached);           // hit
      } catch (err) {
        logger.warn({ err }, 'cache read failed');                // Redis down: degrade to the database
      }
      const value = await loader();                               // miss: load from the system of record
      if (value !== undefined) {
        const jitter = Math.floor(Math.random() * ttl * 0.1);     // avoid synchronized expiry
        redis.set(fullKey, JSON.stringify(value), { EX: ttl + jitter })
          .catch((err) => logger.warn({ err }, 'cache write failed'));  // do not fail the request
      }
      return value;
    },
    invalidate: (key) => redis.unlink(prefix + key),
  };
}

// Usage in the books service (12.3)
const cache = createCache(redis);

export const bookService = {
  get: (id) => cache.getOrLoad(`book:${id}`, () => repo.findById(id)),
  async update(id, changes) {
    const book = await repo.update(id, changes);
    await cache.invalidate(`book:${id}`);                          // after the DB write succeeded
    return book;
  },
};

Invalidation strategies.

1. TTL only. Accept data up to N seconds stale. Simple and robust; pick N per data type (exchange rates: 60 s; product descriptions: 1 h).
2. Delete on write. After updating the database, delete the cache key. Delete rather than overwrite: overwriting races with concurrent writers, and deleting lets the next read load the true value.
3. Versioned keys. Include a version in the key prefix (cache:v3:). Bumping the version (on deploys that change the cached shape, or to flush a whole category) makes old keys unreachable; they expire on their own.
4. Tag-based invalidation. Keep a set of keys per tag (SADD tag:author:17 cache:v1:book:9 ...) and delete all of them when the author changes.
5. Event-driven. A change stream (13.2) or message triggers invalidation, which also catches writes made by other services or scripts.

The race in cache-aside. Even "update DB, then delete cache" has a rare race: a reader misses, reads the old value from the DB, the writer updates the DB and deletes the key, then the reader writes the old value into the cache. A TTL bounds how long that stale value survives; for stricter needs, delay a second delete (delete again a few hundred milliseconds later) or use versioned values.

Cache stampede (dogpile). When a popular key expires, hundreds of concurrent requests miss at once and all hit the database with the same expensive query. Mitigations:

Technique	Idea
Request coalescing (single-flight)	In each process, concurrent misses for the same key share one loader promise
Lock on miss	SET lock:key token NX PX 5000; only the winner recomputes, others wait briefly or serve stale data
Stale-while-revalidate	Store the value with a "soft" expiry; after it passes, serve the stale value and refresh in the background
Probabilistic early refresh	Each request refreshes early with a probability that grows as expiry approaches
TTL jitter	Spread expiries of keys written together

// Single-flight: concurrent misses in this process share one database query
const inFlight = new Map();
async function loadOnce(key, loader) {
  if (inFlight.has(key)) return inFlight.get(key);
  const promise = loader().finally(() => inFlight.delete(key));
  inFlight.set(key, promise);
  return promise;
}

Negative caching. If many requests ask for something that does not exist (/books/does-not-exist, or an attacker probing IDs), every request misses and hits the database. Cache the "not found" result too, with a short TTL, using a sentinel value.

What not to cache. Data that must be exactly current (balances during a payment), data unique to one request, values that are cheaper to compute than to fetch, and personal data without thinking about who can read the cache and when it is deleted.

Measure. Track hit ratio (INFO stats: keyspace_hits, keyspace_misses, or your own metrics per key prefix), latency with and without the cache, and memory. A cache with a 20% hit ratio adds a network round trip to 80% of requests for little benefit.

Key Takeaways

• Caches cut latency and database load for data read far more often than it changes; invalidation is the hard part.
• Cache-aside is the default pattern: read the cache, load from the database on a miss, and invalidate after writes.
• Prefer deleting keys on write over overwriting; use TTLs as a safety net, versioned keys for mass invalidation, and events for writes from other systems.
• Protect hot keys against stampedes with single-flight, locks, stale-while-revalidate and TTL jitter; cache negative results.
• A Redis failure should degrade performance, not availability: fall back to the database.

Practice

1. (Beginner) Add cache-aside caching to GET /api/v1/books/:id and measure latency with and without the cache using autocannon.
2. (Intermediate) Simulate a stampede: give a slow query (500 ms) a 5-second TTL and send 200 concurrent requests right after expiry. Count database queries, then add single-flight and a Redis lock and count again.
3. (Interview) "Users report seeing an old product price for a few minutes after it was changed. How would you investigate and fix it?" Hint: consider every cache layer (browser, CDN, Redis, in-process), the invalidation path after the write, the delete-vs-read race, and TTLs.

Redis as a Session Store

Why Redis for sessions. Server-side sessions (10.2) keep user state on the server and send the browser only a random session ID in a cookie. Where should that state live?

Store	Problem
Process memory (MemoryStore)	Lost on restart, not shared between instances, leaks memory; for development only
Relational/document database	Works, but adds a database read (and often a write) to every request; needs a cleanup job for expired sessions
Redis	Shared by all instances, sub-millisecond reads, built-in TTL deletes expired sessions automatically

Sessions are read on almost every request, are small, and have a natural expiry: a perfect fit for Redis.

How it fits together.

 browser                              Express instance (any of N)                 Redis
 Cookie: sid=s%3AYp3x...signature --> express-session verifies the signature
                                      GET sess:Yp3x...                    -------> {"cookie":{...},"userId":42,"role":"user"}
                                      req.session = parsed object
                                      ... route handler runs ...
                                      if modified: SET sess:Yp3x... <json> EX <maxAge>
                                      else (rolling/touch): EXPIRE sess:Yp3x... <maxAge>

Configuration. 10.2 and 12.3 showed the basic set-up; here it is with the decisions explained:

import session from 'express-session';
import { RedisStore } from 'connect-redis';
import { createClient } from 'redis';

const redisClient = createClient({ url: process.env.REDIS_URL });
redisClient.on('error', (err) => logger.error({ err }, 'redis error'));
await redisClient.connect();

app.set('trust proxy', 1);                       // Secure cookies behind a TLS-terminating proxy (12.1)
app.use(session({
  store: new RedisStore({
    client: redisClient,
    prefix: 'sess:',                             // key namespace: sess:<session id>
    ttl: 60 * 60 * 8,                            // seconds; used when the cookie has no maxAge
  }),
  name: 'sid',
  secret: process.env.SESSION_SECRETS.split(','), // array: first signs, all verify (key rotation)
  resave: false,                                 // do not rewrite unchanged sessions
  saveUninitialized: false,                      // no session (and no cookie) until something is stored
  rolling: true,                                 // reset the expiry on every response: idle timeout
  cookie: {
    httpOnly: true,
    secure: true,
    sameSite: 'lax',
    maxAge: 1000 * 60 * 30,                      // 30 min idle timeout; Redis TTL follows this value
  },
}));

The Redis key's TTL is derived from the cookie's maxAge, so the server-side session and the cookie expire together. With resave: false, unchanged sessions are not rewritten; the store's touch method only refreshes the TTL (EXPIRE), which is cheaper.

Idle and absolute timeouts. rolling gives an idle timeout. Security guidelines (10.2) also recommend an absolute timeout regardless of activity. Store the login time in the session and enforce it in middleware:

app.use((req, res, next) => {
  const MAX_SESSION_AGE = 12 * 60 * 60 * 1000;                     // 12 hours
  if (req.session.userId && Date.now() - req.session.loginAt > MAX_SESSION_AGE) {
    return req.session.destroy(() => res.status(401).json({ error: 'session expired' }));
  }
  next();
});

Login, regeneration and logout.

app.post('/login', async (req, res, next) => {
  const user = await authenticate(req.body.email, req.body.password);
  if (!user) return res.status(401).json({ error: 'invalid credentials' });

  req.session.regenerate(async (err) => {                         // new ID: prevents session fixation (10.2)
    if (err) return next(err);
    req.session.userId = user.id;
    req.session.loginAt = Date.now();
    await redisClient.sAdd(`user:${user.id}:sessions`, req.session.id);   // index sessions per user
    req.session.save((err2) => (err2 ? next(err2) : res.json({ ok: true })));
  });
});

app.post('/logout', (req, res, next) => {
  const { userId } = req.session;
  const sid = req.session.id;
  req.session.destroy(async (err) => {                            // DEL sess:<id> in Redis
    if (err) return next(err);
    if (userId) await redisClient.sRem(`user:${userId}:sessions`, sid);
    res.clearCookie('sid').status(204).end();
  });
});

"Log out everywhere". After a password change or a suspected compromise, all of a user's sessions must end. The set user:<id>:sessions maintained above makes that possible without scanning all keys:

export async function destroyAllSessions(userId) {
  const key = `user:${userId}:sessions`;
  const sids = await redisClient.sMembers(key);
  if (sids.length) await redisClient.unlink([...sids.map((sid) => `sess:${sid}`), key]);
}

Entries in the set can outlive their sessions (sessions expire by TTL without anyone removing them from the set). Clean the set at login (SMISMEMBER/EXISTS checks), or give it a TTL as long as the absolute session timeout.

Operational concerns.

• Eviction. Sessions must not be evicted to make room for cache entries. Use noeviction or a volatile-* policy on an instance dedicated to sessions, or a separate instance from the cache.
• Persistence. Without persistence, a Redis restart logs everyone out. Enable AOF (see "Persistence with RDB and AOF") if that is unacceptable.
• Availability. If Redis is down, no request can load its session. Use a replicated, managed deployment, and decide whether authenticated routes fail closed (they should).
• Size. Keep sessions small: user ID, role, CSRF token, a few flags. Do not store whole user profiles or carts that can be loaded from the database; large sessions cost bandwidth on every request.
• Security. Session data is sensitive. Require authentication on Redis (ACL users), use TLS across networks, and never expose Redis to the internet.

Sessions or tokens? Redis sessions give instant revocation and small cookies, at the cost of a shared store. Stateless JWTs (10.3) avoid the store but make revocation hard, which is why deny lists for revoked tokens also end up in Redis.

Key Takeaways

• Redis is the standard session store: shared across instances, sub-millisecond, and its TTLs expire sessions automatically.
• connect-redis stores sessions as sess:<id> with a TTL derived from the cookie's maxAge; rolling gives an idle timeout.
• Regenerate the session ID at login, destroy it at logout, and enforce an absolute timeout in middleware.
• Index sessions per user in a set to support "log out everywhere".
• Protect session data from eviction and restarts with the right eviction policy, persistence and replication.

Practice

1. (Beginner) Configure express-session with a Redis store, log in, and inspect the session key and its TTL with redis-cli. Make a request and watch the TTL reset.
2. (Intermediate) Implement POST /api/me/sessions/revoke-all that ends every session of the current user except the current one, and an endpoint that lists active sessions with their login time and user agent.
3. (Interview) "Your application stores sessions and cache entries in the same Redis instance with allkeys-lru. What can go wrong?" Hint: consider memory pressure from a cache-heavy traffic spike and which keys the eviction policy may choose.

Rate Limiting and Job Queues

Two very common Redis workloads have the same shape: many application instances coordinate through shared, atomic state. A rate limiter counts requests per client across instances; a job queue hands work from the web tier to background workers.

Rate limiting. 10.6 introduced the algorithms and 12.3 used express-rate-limit. With several instances behind a load balancer, counters must be shared, or each instance enforces its own limit and the effective limit is multiplied (12.3). Redis implements every algorithm with atomic commands.

Fixed window. Count requests per key per time window. Simple and memory-cheap; allows bursts of up to twice the limit at window boundaries.

// 100 requests per minute per client
export async function fixedWindow(redis, clientId, limit = 100, windowSec = 60) {
  const window = Math.floor(Date.now() / 1000 / windowSec);
  const key = `rl:fw:${clientId}:${window}`;
  const [count] = await redis.multi()
    .incr(key)                                   // atomic increment (creates the key at 1)
    .expire(key, windowSec, 'NX')                // set the TTL only the first time (Redis 7+)
    .exec();
  return { allowed: count <= limit, remaining: Math.max(0, limit - count) };
}

The MULTI ensures the TTL is always set together with the first increment. Without it, a crash between INCR and EXPIRE would leave a counter that never expires and blocks the client forever.

Sliding window log. Store each request's timestamp in a sorted set and count those in the last N seconds. Exact, but memory grows with the limit:

-- KEYS[1] = rl:sw:<client>, ARGV[1] = now (ms), ARGV[2] = window (ms), ARGV[3] = limit, ARGV[4] = unique request id
redis.call('ZREMRANGEBYSCORE', KEYS[1], 0, ARGV[1] - ARGV[2])   -- drop timestamps outside the window
local count = redis.call('ZCARD', KEYS[1])
if count >= tonumber(ARGV[3]) then
  return 0                                                       -- rejected
end
redis.call('ZADD', KEYS[1], ARGV[1], ARGV[4])
redis.call('PEXPIRE', KEYS[1], ARGV[2])
return 1                                                         -- allowed

Token bucket (smooth rate with bursts) is also usually implemented as a short Lua script storing tokens and lastRefill in a hash. In practice, use a library rather than writing your own:

import { rateLimit } from 'express-rate-limit';
import { RedisStore } from 'rate-limit-redis';

app.use('/api/auth/login', rateLimit({
  windowMs: 15 * 60 * 1000,
  limit: 10,                                     // 10 attempts per 15 minutes per IP
  standardHeaders: 'draft-8',                    // RateLimit headers for clients
  store: new RedisStore({
    prefix: 'rl:login:',
    sendCommand: (...args) => redisClient.sendCommand(args),   // adapter for node-redis
  }),
}));

Choosing a key. Limit by API key or user ID when the client is authenticated, and by IP otherwise (with trust proxy configured correctly, 12.1). For login endpoints, limit both per IP and per account to slow down credential stuffing and targeted guessing.

Job queues: why. Some work should not happen during the HTTP request: sending emails, resizing images, generating PDFs, calling slow third-party APIs. Doing it inline makes responses slow and loses the work if the process crashes. A job queue lets the web tier record "this needs doing" in milliseconds and return, while separate worker processes do the work, with retries.

 HTTP request --> API: validate, save order, ENQUEUE "send-confirmation" --> 202/201 response
                                                  |
                                             [ Redis queue ]
                                                  |
                         worker 1 <---------------+---------------> worker 2
                         sends email, ACK                           resizes image, ACK
                         on failure: retry with backoff, then move to "failed"

A minimal list-based queue. LPUSH enqueues, a blocking pop dequeues. To avoid losing a job if a worker crashes mid-task, move it atomically to a processing list and remove it only when done (the reliable queue pattern):

// producer
await redis.lPush('queue:emails', JSON.stringify({ id: crypto.randomUUID(), to, template: 'order-confirmation', orderId }));

// worker (dedicated connection: blocking commands hold the connection while waiting)
const worker = redis.duplicate();
await worker.connect();
for (;;) {
  const raw = await worker.blMove('queue:emails', 'queue:emails:processing', 'RIGHT', 'LEFT', 5); // wait up to 5 s
  if (!raw) continue;
  try {
    await sendEmail(JSON.parse(raw));
    await worker.lRem('queue:emails:processing', 1, raw);      // done: acknowledge
  } catch (err) {
    logger.error({ err }, 'job failed');                       // left in processing: a reaper re-queues stale jobs
  }
}

Streams with consumer groups. Redis streams are append-only logs with IDs, designed for this job: each consumer group tracks which entries were delivered to which consumer, and unacknowledged entries stay in a pending list where they can be claimed by another consumer after a timeout.

XADD stream:orders * type created orderId 991                 # append; * = auto-generated ID
XGROUP CREATE stream:orders mailer $ MKSTREAM                 # consumer group "mailer"
XREADGROUP GROUP mailer worker-1 COUNT 10 BLOCK 5000 STREAMS stream:orders >   # new entries for this consumer
XACK stream:orders mailer 1790000000123-0                     # acknowledge after processing
XAUTOCLAIM stream:orders mailer worker-2 60000 0              # take over entries pending for > 60 s

BullMQ: a production queue library. Retries with backoff, delayed and repeatable jobs, priorities, concurrency, rate limiting, progress reporting and dashboards take a lot of code to get right. BullMQ implements them on Redis (using Lua scripts and the ioredis client) and is the common choice in Node.js:

// src/queues/email.js
import { Queue, Worker } from 'bullmq';
import IORedis from 'ioredis';

const connection = new IORedis(process.env.REDIS_URL, { maxRetriesPerRequest: null }); // required by BullMQ workers

export const emailQueue = new Queue('emails', { connection });

// In a route handler: enqueue and respond immediately
await emailQueue.add('order-confirmation', { orderId: order.id, to: user.email }, {
  jobId: `order-confirmation-${order.id}`,     // idempotency: the same order is never enqueued twice (no ':' allowed)
  attempts: 5,
  backoff: { type: 'exponential', delay: 2_000 },   // 2 s, 4 s, 8 s, ...
  removeOnComplete: 1_000,                     // keep the last 1,000 completed jobs
  removeOnFail: 5_000,
});

// In a separate worker process
new Worker('emails', async (job) => {
  const { orderId, to } = job.data;
  await sendOrderConfirmation(to, orderId);    // throwing marks the attempt as failed and triggers a retry
}, { connection, concurrency: 10 });

Queue design rules.

• Jobs may run more than once (a worker can crash after doing the work but before acknowledging), so make handlers idempotent: check whether the email was already sent, use idempotency keys with external APIs.
• Put IDs in jobs, not whole objects: the worker loads current data from the database.
• Use noeviction and persistence (AOF) on the Redis instance holding queues: an evicted or lost job is lost work.
• Monitor queue length, job age and failure counts; a growing queue means workers cannot keep up.

Key Takeaways

• Shared Redis counters make rate limits correct across instances; fixed windows use INCR + EXPIRE atomically, sliding windows use sorted sets, token buckets use Lua.
• Prefer maintained libraries (rate-limit-redis, BullMQ) over hand-written algorithms.
• Job queues move slow or unreliable work out of the request path into retrying background workers.
• Lists give simple queues (reliable with BLMOVE to a processing list); streams add consumer groups, acknowledgements and pending lists.
• Jobs can run more than once: make workers idempotent, and protect queue data from eviction and restarts.

Practice

1. (Beginner) Implement the fixed-window limiter as Express middleware that sets RateLimit-* headers and returns 429 with Retry-After when exceeded.
2. (Intermediate) Move image thumbnail generation from an upload route (12.3) into a BullMQ worker. The route should return 202 Accepted with a job ID, and GET /api/jobs/:id should report the job's state and progress.
3. (Interview) "How do you guarantee that a customer is charged exactly once if your payment job can be retried?" Hint: at-least-once delivery plus idempotency: a job ID, an idempotency key sent to the payment provider, and a unique constraint on the payment record.

Persistence with RDB and AOF

Why persistence matters for an in-memory store. If Redis only held data in RAM, a restart (deployment, crash, host maintenance) would empty it. For a pure cache that may be acceptable; for sessions, queues, rate limits or leaderboards it usually is not. Redis offers two persistence mechanisms, which can be combined.

RDB: point-in-time snapshots. Redis periodically writes the entire dataset to a compact binary file (dump.rdb). To avoid blocking, it forks a child process; thanks to the operating system's copy-on-write, the child sees a frozen copy of memory and writes it to disk while the parent keeps serving requests.

 time ---------------------------------------------------------------->
 parent:  serves commands ......................................................
 fork  ---+
 child:    writes snapshot of memory as it was at fork time --> dump.rdb.tmp --> rename to dump.rdb
          (pages modified by the parent after the fork are copied, so memory use can grow)

# redis.conf: snapshot if at least N changes happened in M seconds (any rule triggers)
save 3600 1          # after 1 hour if at least 1 key changed
save 300 100         # after 5 minutes if at least 100 keys changed
save 60 10000        # after 1 minute if at least 10,000 keys changed
dbfilename dump.rdb
dir /data

Manual commands: BGSAVE (background snapshot), SAVE (blocking, avoid in production), LASTSAVE.

Strengths: compact single file, ideal for backups and copying to other environments, fast restarts (loading an RDB file is much faster than replaying a log). Weakness: everything written since the last snapshot is lost on a crash, potentially minutes of data. Forking a large dataset can also cause a latency spike, and copy-on-write may need significant extra memory under heavy writes.

AOF: append-only file. Redis appends every write command to a log. On restart, it replays the log to rebuild the dataset. How often the log is flushed to disk with fsync determines how much can be lost:

appendfsync	Data at risk on a crash	Performance
always	Practically none (each write is fsynced before replying)	Slowest; limited by disk latency
everysec (default)	Up to about 1 second of writes	Good: the usual choice
no	Whatever the OS has not flushed yet (often ~30 s)	Fastest

# redis.conf
appendonly yes
appendfsync everysec
appenddirname "appendonlydir"
auto-aof-rewrite-percentage 100     # rewrite when the AOF has doubled since the last rewrite
auto-aof-rewrite-min-size 64mb
aof-use-rdb-preamble yes            # the rewritten base file is in RDB format: faster to load

AOF rewriting. The log would grow forever (a counter incremented a million times produces a million entries), so Redis periodically rewrites it in the background: a forked child writes the minimal set of commands (or an RDB-format base) that recreate the current state. Since Redis 7, the AOF is multi-part: a base file plus incremental files listed in a manifest inside appendonlydir, which makes rewrites safer.

Comparison and choice.

Aspect	RDB	AOF (everysec)
Data loss on crash	Since the last snapshot (minutes)	About 1 second
File size	Compact	Larger (until rewritten)
Restart speed	Fast	Slower (replay), much improved with RDB preamble
Backups	Excellent (single file, copy anywhere)	Possible, less convenient
Write overhead	Periodic forks	Continuous appends plus periodic rewrites

Workload	Recommended persistence
Pure cache, rebuildable	None (save "", appendonly no), or RDB for warm restarts
Sessions, rate limits	AOF everysec (+ RDB for backups)
Queues, data you cannot easily rebuild	AOF everysec or always, replication, and backups
Analytics counters, leaderboards	RDB, or AOF if minutes of loss are unacceptable

The Redis documentation's general recommendation for durability comparable to a database is to enable both: AOF for minimal loss, RDB snapshots for backups and disaster recovery.

Backups. Copy the RDB file (or the whole AOF directory) off the machine regularly, for example hourly snapshots to object storage, and test restores. Managed services provide automatic snapshots.

Persistence is not replication. Persistence protects against restarts; it does not keep the service available while a machine is down. That is the job of replication and failover (next topic).

Operational notes.

• Leave memory headroom for forks: on a write-heavy instance, copy-on-write can temporarily require much more memory than the dataset.
• On Linux, set vm.overcommit_memory = 1 so fork() does not fail on large instances, and disable transparent huge pages (they make copy-on-write expensive).
• Use fast local disks for AOF with always or high write volumes.
• In containers, mount a volume on /data, otherwise the persisted files disappear with the container.

Key Takeaways

• RDB writes periodic snapshots from a forked child: compact, fast to load, great for backups, but loses writes since the last snapshot.
• AOF logs every write; with appendfsync everysec at most about a second is lost; background rewrites keep the log compact.
• Enable both for important data; no persistence (or RDB only) for rebuildable caches.
• Forking needs memory headroom and correct kernel settings; persisted files must live on durable volumes.
• Persistence survives restarts but does not provide availability; combine with replication and off-site backups.

Practice

1. (Beginner) Run Redis in Docker with AOF enabled and a volume, write keys, restart the container and verify the data survived. Repeat without the volume.
2. (Intermediate) Write 1 million keys, trigger BGSAVE and BGREWRITEAOF, and compare file sizes and restart times with RDB only, AOF without preamble and AOF with RDB preamble.
3. (Interview) "A Redis instance holding job queues crashed and some jobs were lost. Which configuration choices could have prevented it?" Hint: consider appendfsync, eviction policy, replication with WAIT, and whether the jobs' source of truth should live in the database.

Replication, Sentinel and Cluster

Why. A single Redis server is a single point of failure, and it is bounded by one machine's memory and one main thread. Redis provides three building blocks: replication for copies, Sentinel for automatic failover, and Cluster for partitioning data across several primaries.

Replication. A replica connects to a primary (historically "master") and maintains an exact copy of its data. Replication is asynchronous: the primary acknowledges writes to the client without waiting for replicas.

                 writes
 clients  ----------------->  PRIMARY  --- replication stream (async) --->  REPLICA 1  (read-only)
     |                                                       \------------>  REPLICA 2  (read-only)
     +-- reads that tolerate slight lag ------------------------------------>  replicas

# On the replica (redis.conf or at runtime)
replicaof 10.0.0.5 6379
masterauth <password>
replica-read-only yes

• On first connection, the primary sends an RDB snapshot (full sync), then streams every write. After short disconnections, the replica asks for only the missing part from the primary's replication backlog (partial resync, PSYNC).
• Replicas can serve reads to spread load, but those reads may be slightly stale.
• Because replication is asynchronous, writes acknowledged by a primary that crashes before replicating them are lost. WAIT numreplicas timeout makes the client wait until N replicas have the write (it reduces, but does not eliminate, the risk). min-replicas-to-write makes the primary refuse writes when too few replicas are connected.

Sentinel: automatic failover. Replication alone does not promote a replica when the primary dies; someone must do it. Redis Sentinel is a separate process (run at least three, on different machines) that:

1. monitors the primary and replicas;
2. notifies administrators or other systems of problems;
3. performs automatic failover: when enough Sentinels (the quorum) agree the primary is down, they elect a leader Sentinel, which promotes the best replica and reconfigures the others;
4. acts as a configuration provider: clients ask Sentinel for the current primary's address instead of hard-coding it.

 sentinel 1    sentinel 2    sentinel 3         quorum 2: two must agree the primary is unreachable
      \             |             /
       +---- monitor (PING) -----+
       |             |            |
    PRIMARY  --->  REPLICA A    REPLICA B
       X  (fails)
    sentinels agree -> promote REPLICA A -> REPLICA B replicates from A -> clients rediscover the primary

# sentinel.conf
sentinel monitor mymaster 10.0.0.5 6379 2      # name, address, quorum
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000

// ioredis connects through Sentinel and follows failovers automatically
import IORedis from 'ioredis';
const redis = new IORedis({
  sentinels: [{ host: 'sentinel-1', port: 26379 }, { host: 'sentinel-2', port: 26379 }, { host: 'sentinel-3', port: 26379 }],
  name: 'mymaster',
  password: process.env.REDIS_PASSWORD,
});

Sentinel gives high availability but no scale-out: all writes still go to one primary, and the dataset must fit on one machine.

Redis Cluster: sharding. Redis Cluster partitions the keyspace across several primaries, each with its own replicas, and handles failover internally (no Sentinel needed). The keyspace is divided into 16,384 hash slots; each key belongs to slot CRC16(key) mod 16384, and each primary owns a range of slots.

 slot = CRC16("user:42") mod 16384 = 5649    (example)

 +-------------------+  +-------------------+  +-------------------+
 | primary A         |  | primary B         |  | primary C         |
 | slots 0 - 5460    |  | slots 5461 - 10922|  | slots 10923-16383 |
 | + replica A1      |  | + replica B1      |  | + replica C1      |
 +-------------------+  +-------------------+  +-------------------+
        ^ nodes gossip over a cluster bus (port + 10000) and detect failures; a replica is promoted if its primary fails

Redirections. Any node can receive any command. If the key's slot lives elsewhere, the node replies MOVED 5649 10.0.0.6:6379; cluster-aware clients learn the slot map and send subsequent commands directly to the right node. During slot migration, an ASK redirection handles keys that are moving.

Multi-key operations. Commands touching several keys (MGET, SINTER, MULTI transactions, Lua scripts) only work if all keys are in the same slot, otherwise Redis returns CROSSSLOT. Hash tags control placement: only the part inside {...} is hashed.

SET {user:42}:profile "..."      # both keys hash on "user:42" -> same slot
SET {user:42}:cart "..."
MULTI                            # allowed: same slot
...

Overusing a single hash tag puts too much data on one node (a hot shard), so tag by entity, not globally.

Other Cluster limitations. Only database 0 exists (no SELECT). Classic pub/sub messages are broadcast to all nodes (use sharded pub/sub). Clients must support the cluster protocol.

Choosing a topology.

Topology	Availability	Scale	Complexity	Use when
Single instance	None (restart = downtime)	One machine	Lowest	Development, non-critical caches
Primary + replicas	Manual failover; read scaling	Reads only	Low	Read-heavy, manual recovery acceptable
Sentinel	Automatic failover	Reads only	Medium	Production data fits on one machine
Cluster	Automatic failover per shard	Reads, writes and memory	Highest	Data or write throughput exceeds one machine
Managed service	Provider-managed failover	Configurable	Lowest operational effort	Most production teams

Key Takeaways

• Replication keeps asynchronous copies on replicas; acknowledged writes can be lost on failover, and WAIT reduces that risk.
• Sentinel monitors instances, elects a new primary automatically with a quorum, and tells clients where the primary is; run at least three.
• Redis Cluster shards keys across primaries using 16,384 hash slots; clients follow MOVED/ASK redirections.
• Multi-key commands, transactions and scripts need all keys in one slot; use hash tags such as {user:42}.
• Choose the simplest topology that meets availability and scale needs; managed services remove most of the operational work.

Practice

1. (Beginner) With Docker Compose, run one primary and two replicas, write on the primary, read on a replica, and try writing on a replica.
2. (Intermediate) Add three Sentinels, connect a Node.js app through them with ioredis, kill the primary container while the app writes in a loop, and measure how long writes fail during failover.
3. (Interview) "Your Lua rate-limit script fails with CROSSSLOT after moving to Redis Cluster. Why, and how do you fix it?" Hint: look at which keys the script touches, how slots are computed, and how hash tags group keys.

Redis Clients in Node.js

Two main clients.

Client	Maintained by	Characteristics
redis (node-redis)	Redis	Official client; promise-based, camelCase command methods, TypeScript types, RESP3, modules (JSON, search); recommended for new projects
ioredis	Community	Mature, built-in Sentinel and Cluster support, Lua defineCommand; used by BullMQ and many libraries

Both work with Redis and Valkey. Earlier chapters (10.2, 10.5, 12.3) used node-redis, as does this topic; ioredis examples appear where libraries require it.

Connecting with node-redis.

// src/db/redis.js
import { createClient } from 'redis';

export const redis = createClient({
  url: process.env.REDIS_URL,                     // redis://user:pass@host:6379/0 or rediss:// for TLS
  socket: {
    connectTimeout: 5_000,
    reconnectStrategy: (retries) => Math.min(retries * 100, 3_000),   // back off up to 3 s between attempts
  },
});

redis.on('error', (err) => logger.error({ err }, 'redis client error'));  // REQUIRED: without a listener, errors crash the process
redis.on('reconnecting', () => logger.warn('redis reconnecting'));
redis.on('ready', () => logger.info('redis ready'));

export async function connectRedis() {
  await redis.connect();
  await redis.ping();                             // 'PONG'
}

export async function closeRedis() {
  await redis.close();                            // node-redis v5; in v4: await redis.quit()
}

Like the MongoDB client, create one client per process at start-up and reuse it (13.1). A single connection is enough for most workloads because commands are pipelined over it. Use extra connections only for: subscribers (pub/sub), blocking commands (BLPOP, BLMOVE, XREADGROUP BLOCK), and WATCH-based transactions. Create them with redis.duplicate().

Command API. Methods are named after commands in camelCase, take arguments in order, and return promises of JavaScript values:

await redis.set('greeting', 'hello', { EX: 60 });          // SET greeting hello EX 60
await redis.get('greeting');                               // 'hello' (or null)
await redis.incrBy('views:home', 5);                       // number
await redis.hSet('user:42', { name: 'Ada', logins: 0 });   // object form: several fields
await redis.hGetAll('user:42');                            // { name: 'Ada', logins: '0' }: values come back as STRINGS
await redis.sAdd('tags', ['a', 'b']);
await redis.zAdd('lb', [{ score: 870, value: 'ada' }]);
await redis.expire('user:42', 3600);

// Transactions: chain commands, exec() returns an array of replies
const [count, ttl] = await redis.multi().incr('c').ttl('c').exec();

// Anything not wrapped yet: raw command
await redis.sendCommand(['OBJECT', 'ENCODING', 'user:42']);

Values are strings on the wire: numbers sent to Redis are converted to strings, and hash field values and GET results come back as strings (Number(...) or JSON.parse(...) them). Buffers are supported for binary data.

Iterating keys safely.

// SCAN in batches; never KEYS in production
for await (const keys of redis.scanIterator({ MATCH: 'cache:v1:*', COUNT: 500 })) {   // v5 yields arrays (v4: one key at a time)
  if (keys.length) await redis.unlink(keys);
}

The same with ioredis. Commands are lowercase methods with positional arguments, and the client connects automatically:

import IORedis from 'ioredis';

const redis = new IORedis(process.env.REDIS_URL);          // connects lazily and reconnects automatically
await redis.set('greeting', 'hello', 'EX', 60, 'NX');      // arguments mirror the raw command
const pipeline = redis.pipeline();                         // explicit pipeline (non-atomic batch)
pipeline.incr('a').incr('b');
const results = await pipeline.exec();                     // [[err, value], [err, value]]

redis.defineCommand('transferPoints', { numberOfKeys: 2, lua: transferLua });
await redis.transferPoints('points:42', 'points:17', 100);

const cluster = new IORedis.Cluster([{ host: 'redis-1', port: 6379 }, { host: 'redis-2', port: 6379 }]);

Wiring Redis into an Express application. Keep Redis behind small modules (cache, sessions, rate limiter, queues), inject the client into them, and include Redis in the health check and graceful shutdown:

// src/server.js
import { connectMongo, client as mongo } from './db/mongo.js';
import { connectRedis, closeRedis, redis } from './db/redis.js';
import { createApp } from './app.js';

await Promise.all([connectMongo(), connectRedis()]);
const app = createApp({ redis });                            // dependencies injected (12.3)

app.get('/healthz', async (req, res) => {                    // readiness: can we serve traffic?
  try {
    await Promise.all([redis.ping(), mongo.db('admin').command({ ping: 1 })]);
    res.json({ status: 'ok' });
  } catch {
    res.status(503).json({ status: 'degraded' });
  }
});

const server = app.listen(process.env.PORT ?? 3000);
process.on('SIGTERM', () => {
  server.close(async () => {
    await Promise.allSettled([closeRedis(), mongo.close()]);
    process.exit(0);
  });
});

Resilience. When Redis is unreachable, node-redis queues commands while reconnecting (the offline queue) by default, so requests can hang. Decide per use:

• Cache: fail fast and fall back to the database (disableOfflineQueue: true, or a short timeout around the call).
• Sessions and rate limits on sensitive routes: fail closed with 503 rather than letting requests through unauthenticated or unlimited.
• Queues: fail the request so the client can retry, or write to an outbox table and enqueue later.

Security. Use ACL users with the minimum permissions (ACL SETUSER app on >password ~cache:* ~sess:* +@read +@write -@dangerous), TLS (rediss://) across networks, private networking only, and secrets from the environment. Redis's default configuration was designed for trusted networks; an exposed instance without a password can be taken over.

Testing. Run a real Redis in tests (Docker or Testcontainers), use a separate logical database or key prefix per test run, and FLUSHDB between test files. For pure unit tests, inject a fake implementing the few methods your module uses.

Key Takeaways

• node-redis (official, camelCase methods, promises) and ioredis (Sentinel/Cluster built in, required by BullMQ) are the two main clients; both work with Redis and Valkey.
• Create one client per process, attach an error listener, and use dedicated connections only for subscribers, blocking commands and WATCH.
• Replies are mostly strings; convert numbers and JSON explicitly.
• Iterate with scanIterator, include Redis in health checks and graceful shutdown, and decide per feature whether to fail open or closed when Redis is down.
• Secure Redis with ACL users, TLS and private networks, and test against a real instance.

Practice

1. (Beginner) Write a Node.js script with node-redis that stores a user as a hash, increments a counter field, reads the hash back and converts the numeric fields to numbers.
2. (Intermediate) Add Redis to the bookshelf API from 12.3: a cache-aside layer for GET /books/:id, a Redis-backed rate limiter for POST routes, and /healthz checking both MongoDB and Redis. Stop the Redis container and verify that reads still work (from MongoDB) while writes are rejected with 503.
3. (Interview) "Why does a subscriber or a worker using BLPOP need its own Redis connection?" Hint: think about what the connection can do while it is in subscribe mode or blocked waiting, and how pipelining multiplexes other commands over a shared connection.

<a id="14-front-end-development-with-react"></a>